Commit Graph

149 Commits

Author SHA1 Message Date
overtrue c43e415b09 feat(server): serve ListBuckets and HeadBucket on the gateway stack (#9296)
The walking skeleton of the gateway edge, behind the gateway-stack cargo
feature and RUSTFS_S3_STACK=gateway. rustfs-gateway inside the RustFS binary
serves ListBuckets and HeadBucket end to end and answers every other S3
operation 501 NotImplemented. A default build carries neither the stack
selection nor the wrapper: its request path is unchanged.

- One per-operation unit: impl Served for dto::Op with a metric label, an
  authorize without a default body, and an execute. The only impl Handler is
  the blanket one over Served (access phase, authorize, count, execute, one
  error mapping). authorize calls the function the legacy access hook calls.
- One table laid out per family (ten family files named after the migration
  batches); it registers the handlers and is the whole early gate.
- A shared access phase used by both edges, fed on the gateway edge from the
  gateway's verdict.
- Stored bucket names are materialised under the gateway floor only, so names
  the AWS grammar refuses are listed as before; HeadBucket's legacy edge builds
  its input the same way until the gateway re-pin (rustfs/gateway#1610).
- With the gateway selected the legacy service receives exactly what the
  legacy router claims (extension routes, object lambda, control plane).
- A start-up line names the stack the mounted service answers with.
- The release-boundary guard reads feature closures from cargo metadata.

Squashed from gw/T2.5-gateway-stack at aca434fdf onto ce9d0b153, with two
changes made while landing: the new CI job reads the ci-dev cache under its
current digest key, and the cache-reader count in scripts/test_ci_plan.py is
7 (this job is the seventh reader).

Checked: cargo clippy --locked -p rustfs --all-targets -- -D warnings in both
builds on aca434fdf (main up to eca95f297); on the squashed tree the Quick
Checks guards, the spell check, ci_gate.py --self-test and --check-workflow,
and the release-boundary tests. Not run on the squashed tree: a compile of
the rustfs crate (six small commits landed on main since the clippy run) and
any test. The legacy HeadBucket name table carries four provisional statuses
for names no store holds; they are corrected by a follow-up when the probe
of main reports.
2026-11-13 11:31:50 +08:00
overtrue ef79dcbe19 ci: conformance and client-matrix lanes against both S3 stacks
Acceptance lanes T3.4 (rustfs/backlog#2775) and T3.5 (rustfs/backlog#2776):
.github/workflows/gateway-acceptance.yml runs the rustfs/gateway conformance
corpus (--profile rustfs) and client matrix (ci/compat/run_matrix.sh
--external) against the candidate binary on RUSTFS_S3_STACK=legacy and
=gateway, one server per stack. The suites are checked out at the gateway rev
Cargo.toml pins. Nightly schedule and workflow_dispatch only.

- Each stack must be observed: the start step writes `ready` only when
  /health/ready answers while its own process is alive and the server's
  server_starting event names the requested stack (scripts/dual_stack.py
  stack); otherwise `did not start: ...`. Summaries print the observed stack.
- Verdicts come from files the run leaves on disk (startup record, stack
  evidence, liveness after the corpus, exit codes, report.json, matrix.json),
  never from an earlier step's exit status.
- Conformance: the rustfs profile describes the gateway edge, so only the
  gateway stack is judged (scripts/check_conformance_rulings.py against
  .config/conformance-rulings.toml). Red: an unruled failure, a ruled case
  failing differently from its ruling's `failure`, a stale ruling, and a skip
  on the server's own answer (endpoint not answering, refused fixture). The
  legacy run is an unjudged reference: it must serve, stay up and complete.
- Client matrix (scripts/compat_matrix_report.py): per stack, red on a `fail`
  or `sut-unregistered` cell, a client that could not be installed, or an
  unfinished matrix; legacy is the baseline and every legacy pass must pass on
  gateway.
- The gateway leg and the stack comparison are continue-on-error, report-only
  until T2.10.

Not built: no re-pin (write cases need a gateway rev with
--allow-external-writes plus that flag on the lane's run line); no legacy
rulings; no use of the runner's own --rulings ledger (stale rulings do not
block there); no call to the gateway's check_compat_matrix.sh (it does not
fail `fail` cells or clients that never ran); no JUnit (the pinned runner
writes a raw U+0001); no install cache; toolchain versions copied from the
gateway's client-matrix.yml at 7915d069.

Tests: --self-test of both checkers in make script-tests, 40 tests, 31
negative. Mutations, each red: unruled failure ignored
(test_an_unruled_failure_is_red); stale only when passed
(test_a_ruling_on_a_skipped_or_absent_case_is_stale); did-not-start green
(test_a_server_that_did_not_start_is_red_and_says_why); swallowed runner exit
(test_a_swallowed_exit_cannot_hide_a_failed_case); exit 3 accepted
(test_an_environment_exit_is_red); legacy judged or tallied
(test_the_legacy_reference_is_counted_never_judged); dead server green
(test_a_server_that_died_during_the_corpus_is_red); endpoint and fixture
skips not red (test_a_skip_on_the_endpoint_not_answering_is_red,
test_a_skip_on_a_refused_fixture_is_red); ruling on id only
(test_a_ruled_case_failing_differently_is_red); unknown verdict
(test_an_unknown_verdict_is_refused); stack not compared
(test_a_server_of_the_other_stack_is_red); no stack evidence
(test_ready_without_stack_evidence_is_red); failed cell, skipped client,
swallowed matrix exit (test_a_failed_cell_is_red,
test_a_client_that_could_not_be_installed_is_skipped_never_passed,
test_a_missing_manifest_is_red_even_when_the_exit_was_swallowed); superset
gate (test_a_legacy_pass_lost_or_absent_on_gateway_is_red,
test_a_gateway_superset_of_the_baseline_is_clean_and_its_extra_passes_are_listed,
test_diff_without_both_manifests_is_red); diffs
(test_diff_lists_changed_and_absent_cases_only,
test_diff_without_both_reports_compares_nothing).

Trial runs on rustfs/rustfs#9368: 43264289490, 43325101234. Gateway stack did
not start (gateway stack is not assembled yet, T2.5); legacy reference
completed, 140 of 890 cases executable against an external endpoint; legacy
matrix 13 x 14: pass 121, fail 0, client-unsupported 61, sut-unregistered 0.

Not verified: the stack observation and liveness record in CI (exercised
locally under bash -eo pipefail with a stand-in server and main's
dual_stack.py, both directions); a scheduled run; a diff with gateway content.
2026-11-13 10:31:46 +08:00
hector 5a26862e2e ci: build macOS and Windows on a weekly schedule (#9286)
* ci: build macOS and Windows on a weekly schedule

Keep the 01:13 UTC cron on Linux and add a Sunday 16:00 UTC cron
for macOS and Windows. Select that subset from the firing cron so a
delayed runner cannot guess the weekday.

Co-authored-by: hector <majinghe@users.noreply.github.com>

* ci: move the weekly desktop cron off minute zero

---------

Co-authored-by: hector <majinghe@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-11-12 20:51:08 -05:00
overtrue 704b984c1f ci: both-stack e2e diff in the source workflows and a strict mint stack gate
T3.1/T3.3 (rustfs/backlog#2772, #2774). The existing nightly lanes get a stack
dimension and their legacy leg is the legacy side: e2e-replication-nightly (repl,
cluster, protocols), e2e-distributed and the ODM minio-source job. A gateway leg
runs scripts/dual_stack.py preflight after its build and stops there when no gateway
start is proven; the summary says 'skipped: gateway stack refuses startup: <why>'.
One summary job per workflow diffs the legs of the same run with junit_diff.py.
dual-stack-nightly.yml covers only e2e-smoke and e2e-full: one build, both stacks
from that binary, gateway legs only when the preflight proved a start.
junit_diff.py reports a missing, empty or case-less side as 'no result for <label>'.
mint.yml stays weekly, builds the image once and runs it per stack; mint_stack_diff.py
gates that every case legacy passes is present and passing on gateway with no new
failure (missing, NA, unknown status and truncated lines never pass). Gateway legs and
the compare job are report-only until T2.10.

Not built: a shared-binary preflight job in the three source workflows (one preflight
step per gateway leg); harness-only filtering for their five profiles; mc/Vault/iptables
in the e2e-full leg; an s3s-e2e both-stack run; rustfs_gateway=debug.
Tests: dual_stack 30 (preflight with fake binaries), mint_stack_diff 16, junit_diff 27;
negatives outnumber positives. Mutations (red test named) are in the PR 9367 description.
Not verified: the workflows have not run in CI in this shape (composites, stack matrices,
image artifact round trip, needs result of a red continue-on-error leg).
2026-11-13 09:21:25 +08:00
Chris An b42f444166 ci: stop rebuilding what the cache already built (#9371)
* ci: stop rebuilding what the cache already built

cache-warm.yml built `--workspace --all-targets`, a selection no ci-dev
reader builds. Feature resolution differs per selection, so the test
archive recompiled 123 external units and the E2E archive about 290 on
every exact cache hit. warm-ci-dev now runs the readers' own builds:
`ci_test_artifact.py cargo warm` (the Cargo build of `nextest archive`)
for both archive configurations, the hooks server and the lint command.
Rust-cache never saves over an exact key match, so the warm jobs skip
their builds after one; ci_gate.py --check-workflow pins both rules.

The manual shard rehearsal ran only after a full execution, saved
nothing, and could not run since #9071 renamed the archive it downloads.
It is removed with its plan field, jobs, script and tests; the plan
contract moves to 5.

* ci: key ci-dev by its warm commands and close the warm guard's gaps

The ci-dev key now ends in a digest of warm-ci-dev's build steps, so the
reader selections this branch warms are written under a new key instead of
being skipped as an exact hit of the old one. ci_gate.py --check-workflow
derives the digest, requires the writer and every reader in ci.yml and
reference-parity.yml to name that key, and prints the key a change needs.

The presence check now reads only the steps that run whenever the key is
written, so a selection moved into the on-request --timings step no longer
counts. Configuration names are compared as sets in both directions and
accept either option spelling.

The plan contract moves to 6: #9369 already used 5 for a different plan.
2026-11-12 18:29:34 -05:00
Chris An 6b27fa4756 ci: accept the proving merge-group run as release evidence (#9376)
* ci: accept the proving merge-group run as release evidence

A commit that lands through the merge queue gets a push run of ci.yml in
mode proven: Quick Checks and Test and Lint succeed, and every validation
job, End-to-End Tests (full merge gate) included, is skipped. source_ci
refused such a commit, so no release candidate could be cut from it.

source_ci now accepts that push run when Quick Checks and Test and Lint
succeeded and a successful merge_group run of ci.yml on that branch's
queue has the same head tree and passed all three required jobs itself;
that run is recorded as the candidate's exact-source CI identity. The
match is ci_merge_group_proof.proves, the same predicate the push plan
uses.

Also: a test that a full merge_group run requires every job a full
integration push requires; the sibling queue-branch case
(gh-readonly-queue/mainline/...); select_plan's proof is keyword-only;
the release, CI gate and workflow docs describe proven pushes.

* test(release): pin the proving run's position, attempt and branch

The release test now lists the matching merge-group run among a run of
another tree and a run of another branch's queue, in both orders; gives
it attempt 3 and reads jobs by run and attempt; and accepts a release
branch's own queue run while refusing a main queue run for it.
2026-11-12 16:14:04 -05:00
Chris An 4a34e512cd ci: skip the push run for a tree the merge queue already proved (#9369)
A merge-queue landing fast-forwards main to the group's commit, so the
push run re-validates a tree a successful merge_group run of ci.yml has
just passed. Select CI scope now looks that run up with the job's
read-only Actions token (scripts/ci_merge_group_proof.py) and plans
mode `proven`: Select CI scope, Typos and Quick Checks run, every
validation job is skipped, and Test and Lint requires exactly that.
The plan names the proving run, its not-applicable reasons and the
aggregate's summary say the jobs were skipped as proven.

Any doubt runs everything: no token, an API or network error, a rate
limit, an unexpected payload, no successful queue run of the same tree,
or a proof on another event. A direct push has no such run. The plan
contract moves to 5.
2026-11-13 03:58:53 +08:00
Chris An 55d9bf08a2 ci: run workspace clippy beside the test binary build (#9345)
Clippy ran in front of the test-binary archive inside Workspace Lint and Build, so every workspace test waited 13 to 17 minutes for a lint that shares no compiler output with the archive.

Move the unchanged Clippy step into its own workspace-clippy job with the same condition, needs, runner and cache key. Workspace Lint and Build keeps its ID and name as the archive producer. The aggregate and the scheduled alert depend on the new job, and the plan contract moves to 4 so a base planner that predates the job set falls back to the full matrix.

ci_gate.py now rejects a workflow where the lint is conditional, duplicated, outside the aggregate, scoped differently from the archive producer, or back on the archive consumers' dependency chain, and where the workflow's plan-contract marker differs from the planner's.
2026-11-12 10:09:05 -05:00
Chris An 83ce509ff3 ci: run the gateway smoke lane only where the gateway stack is touched (#9344)
* ci: run the gateway smoke lane only where the gateway stack is touched

The report-only `e2e-smoke (gateway, report-only until T2.10)` lane ran on
every pull request and failed on every one of them: the gateway S3 stack
does not serve the whole e2e-smoke suite before T2.10. A pull request that
touches none of the lane's inputs can neither cause nor fix that failure,
yet it was shown a red check and occupied an sm-standard-2 runner.

Select CI scope now publishes one more output, `gateway_smoke`, computed by
`scripts/ci_gate.py select` from the change set: true when a path in
GATEWAY_SMOKE_PATHS changed or a `rustfs-gateway*` pin in Cargo.toml or
Cargo.lock differs, and on every event that is not a pull request. The lane
is skipped only on a pull request whose selection is the literal `false`; a
missing or refused selection runs it. Job-level continue-on-error stays, and
no step may carry it, so a failed suite is never rendered green.

`ci_gate.py --check-workflow` pins the condition by truth table: the lane
must run wherever End-to-End Tests runs except on unselected pull requests,
must need classify-changes, and the output must be published.

* ci: keep the e2e suite's own files out of the gateway smoke selection

Feature pull requests add e2e tests and touch the smoke selection all the
time, and until T2.10 the gateway lane cannot pass on any of them, so
selecting on `crates/e2e_test/*`, `.config/nextest.toml` and
`.config/e2e-smoke-selection.txt` kept producing the red check this change
exists to remove. What the suite contains carries no signal while the stack
cannot serve it. The gateway code paths, the pin comparison, the workflow,
the test-runtime action and the two scripts stay.

An e2e-only pull request being skipped is now a pinned self-test case. At
T2.10 the selection is deleted, not extended.
2026-11-11 18:52:15 -05:00
Chris An 31e1e42ad7 refactor: land the s3s decoupling groundwork on main (#9276)
Lands gateway/integration on main: the guard rails and the domain
decoupling for replacing s3s with rustfs/gateway (rustfs/backlog#2734,
Phase 0 and Phase 1). The S3 edge is still s3s and no wire behaviour
change is intended.

- Guard rails: lower-only s3s footprint baseline and edge allow-list,
  per-crate test-count floor, 39 byte-level goldens for the persisted
  bucket-config XML families, a two-server differential harness over
  all 73 S3 operations, a JUnit differ, and a reference-parity lane
  against the sha256-pinned 1.0.5 binary.
- Owned types: rustfs-s3-types (S3Error, S3ErrorCode, Body),
  RequestEnvelope, AuthzFacts.
- Engine crates off s3s: ecstore, lifecycle, replication, scanner,
  protocols, s3select, s3-client, rpc.
- Admin: error types moved; 300 of 314 Operation impls take the owned
  AdminRequest.
- Server: the s3s-compat tower layers grouped under
  server/legacy_compat; RUSTFS_S3_STACK=legacy|gateway selector, where
  gateway refuses to start until the gateway edge exists.

Fixes made on the branch that change ecstore and heal behaviour are
listed in the pull request description. The per-commit history stays
on gateway/integration.
2026-11-11 02:39:52 -05:00
Chris An 95e128336d ci: narrow checks for stable installation metadata (#9269)
* ci: narrow checks for stable installation metadata

* fix: reject release metadata marker collisions
2026-11-10 16:10:04 -05:00
Hauser Zhi 134820c1e0 test(storage): reproduce HTTPS rejoin readiness in Docker (#9206)
* test(storage): reproduce HTTPS rejoin readiness in Docker

Compare cluster-only and node-ready/write gates on immutable native binaries. Preserve first-failure and latest-acknowledged-content evidence across recovery, capture lock diagnostics before restart, and limit cleanup to verified owned resources.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(testing): unify Docker repro ownership and controls

Use typed resource absence and one verified container inventory for staging, partial setup and retained nodes. Reject inherited experiment overrides before creating resources and remove duplicated state.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(testing): guard staging volumes and interrupted capture

Verify volume ownership before privileged staging and disable implicit image-content copying. Preserve failed evidence and continue owned finalization when capture or cleanup is interrupted.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(testing): capture live rejoin logs and verify trace coverage

Read live tmpfs logs inside the verified container and use one explicit stdout sink for diagnostics. Record bounded capture provenance and fail diagnostic coverage when acquisition, guard-drop or physical-release records are missing, without masking HTTP failures.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(testing): reject inherited aliases and finalize on SIGTERM

Reject inherited MINIO compatibility settings before Docker resource creation. Route SIGTERM through interrupt-aware finalization and restore the previous handler after each run. Cover real process signals and failure preservation.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(testing): normalize legacy controls and termination

Reject inherited MINIO compatibility aliases in neutral runtime images. Bridge run-mode SIGTERM into the existing interruption finalization and restore the caller signal handler on return or exception.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-07 08:04:28 +08:00
Chris An 158aa276e9 fix(storage): bound heal retries and recover pool metadata (#9164)
* fix(ecstore): retry metadata writes rejected before staging

* fix(ecstore): reuse registered disks during startup

* fix(ecstore): retain remote health monitors during reconnect

* test(heal): release the commit barrier during graceful shutdown

* test(e2e): register graceful shutdown barrier regressions

* fix(ecstore): roll back interrupted pool metadata commits

* test(upgrade): honor the 1.0.4 delete recovery protocol

* fix(heal): bound admission during object rate pressure

* test(e2e): handle rebalance completion before stop

* test(e2e): observe rebalance activity before the start reply

* test(rebalance): hold the first run until stop cancels admission
2026-11-05 21:42:19 +08:00
hector 965d805a4f ci: run distributed and upgrade e2e on schedule only (#9126)
Remove pull_request from the distributed and upgrade workflows. Distributed
stays on the nightly cron, and upgrade stays on the Monday cron plus
version tags. Both remain manual-dispatch lanes and stay outside the
required merge checks.

Co-authored-by: majinghe <majinghe@users.noreply.github.com>
2026-11-02 23:03:36 +08:00
Chris An 1cb4c0ebe2 ci: stamp the plan contract and rename the workspace lint-and-build key (#9095)
* ci: gate the plan on a contract marker

A base planner predating the NIGHTLY_JOBS split still emits the old required
set, and the strict verifier would reject it. Select now stamps plan_contract=2
and the workflow refuses any plan without the current marker, falling back to
the full matrix, so a stale base planner can neither authorize the new narrower
selection nor force a job the new workflow no longer schedules.

* ci: rename the workspace lint-and-build job key and evidence path

The job key test-and-lint named a job that runs no tests: it does Clippy, the
test-binary archive, doctests and the log-rule anchors, while the tests live in
workspace-tests. Anyone reading needs: [test-and-lint] would conclude tests had
already gated that dependency. The key becomes workspace-lint-and-build to
match its display name, and the shared evidence directory becomes
artifacts/workspace so the path stops asserting the same thing. The aggregate's
display name stays Test and Lint because that is the live ruleset check
context. ci-gates.md also stops attributing the ECStore test run to the wrong
job.
2026-10-31 17:41:02 +08:00
Chris An e5563e6004 feat(release): gate publication on candidate certification (R01) (#9090)
Add a fail-closed release certification path so incomplete or substituted
evidence blocks every official publication exit. scripts/release/release_gate.py
validates the versioned release-validation plan, the candidate identity and the
G01 evidence-envelope set, then re-checks promoted bytes against the passed
list. build/docker/package/helm publication jobs require that certification and
stage artifacts before promoting identical bytes.
2026-10-31 16:53:18 +08:00
Chris An 51275cb7d8 ci: move specialized lanes off the pull-request path (#9093)
Connect Feature Boundaries, real io_uring and the rio-v2, Swift, SFTP and
HTTP/3 feature lanes no longer gate a merge. They run only in the scheduled
and manual full-validation runs, which report a regression within a day; on
the pull-request path they bought no earlier signal worth the runner
occupancy (Connect alone measured 53.5 median minutes and occasionally set
the wall). Workspace lint and tests, the short core contract, the debug
build, E2E smoke and the S3 lanes stay required on PRs and merge groups.
2026-10-31 16:52:03 +08:00
Chris An 3bf270b49f ci: record CI01 measurements, drill evidence and fix the OIDC live gate (#9091)
* ci: retain CI01 timing sample and fix the OIDC live gate tool

Record the measured PR feedback and runner occupancy for the CI plan's CI01
criterion, including the honest gap between the experimental targets and the
sampled medians, and replace the awscurl dependency in the live Keycloak gate
with the AWS CLI so the security suite runs on PEP 668 managed fleets.

* docs(testing): attach live run evidence to the event-drill index
2026-10-31 05:10:38 +00:00
Chris An d8d064bd18 docs(testing): index event-class drills and retained CI evidence (#9088) 2026-10-31 11:21:46 +08:00
Chris An b8ee0301e9 feat(scanner): fair bucket service across restarts and budgeted first enumeration (#9083)
* feat(scanner): bound first-build enumeration with the cycle budget

Charge raw directory entries and their name bytes, and let one bucket scan hold a child budget that stops its own token while charging the cycle. A count budget also bounds one directory listing, so a wide first build can no longer finish a full read_dir before the first object is charged.

The new stop reason maps onto the persisted Directories variant so older readers keep decoding checkpoints. A directory with queued children keeps its index until they end, and every index on the active walk path is saved as an optional map field: losing either one would recharge its replay and stop a fixed budget from advancing.

* feat(scanner): keep bucket service order across restarts

Save each set's first-in-first-out bucket order as an optional map field of its usage cache, including after an incomplete cycle, which otherwise writes nothing. Durable age now outranks the dirty-first and leader-local cohort orders, which only break its ties, so a restart cannot hand the budget back to the same first buckets. Failed turns add visible debt without moving a bucket forward; a budget stop or a resource veto adds none. A bucket that spends its quantum and saves forward motion returns to the queue tail within the cycle.

The order authorizes nothing: another movement epoch or a corrupt value starts a new one, and a recreated bucket joins as a new arrival.

* test(scanner): cover quantum handoff and saved service order

Run the real worker queue with injected bucket shares and verify a spent bucket yields to waiters, a share without durable motion ends its turn, and the saved order keeps certified coverage while rejecting older owners.

* docs(scanner): describe budgeted first build and service order

* test(scanner): pin index retention and requeue motion

Bound the nested restart fixture tightly enough that dropping a directory index fails it, and cover a repeated share whose frontier never moves.

* fix(scanner): keep service generations and owner stops intact

Queue a new bucket behind waiters but ahead of buckets that already had a turn, matching the leader-local cohort, and keep the set cache untouched when the owner was cancelled so an interrupted cycle cannot advance its revision. The order holds one record per inventory bucket, like the set cache's entries, so a truncating bound that could starve the long-waiting buckets is removed. Publish the incomplete scope before saving so a contended lock cannot delay it.

* ci(e2e): refresh the full selection digests

* fix(scanner): correct a comment spelling

* fix(scanner): separate service debt from vetoes and stale owners

Keep a resource veto and a remote count-budget stop out of failure debt,
settle the claim before a requeue exposes the bucket to another worker, and
stamp the service order with its leader so an older owner cannot replace it
after an incomplete save. The checkpoint fixture's budget now covers both
service orders.

* fix(ecstore): create parents before resource-scoped writes

#9064 moved parent creation into the lazy writer, so a write that holds a
resource unit reached the native open without its parent directory and
failed with FileNotFound on nested paths.

* fix(e2e): retry SlowDown codes during upgrade write readiness

The rolling-upgrade readiness probe only retried 503 ServiceUnavailable,
so a transient 503 SlowDown from a still-recovering node failed the
four_node_rolling_upgrade test immediately. Accept SlowDown and
SlowDownRead alongside ServiceUnavailable, matching the existing chaos
test transient-error set.
2026-10-31 07:57:18 +08:00
Chris An 191c95a469 ci: enforce affected validation and rehearse bounded shards (#9070)
* ci: enforce affected validation and rehearse bounded shards

* test(ci): keep regression output stable under forced color
2026-10-30 11:58:38 +08:00
Chris An dacb003856 ci: let failed-job reruns reuse the producer's test archives (#9071)
Since #9059 the shared test archives and debug binaries are named
with the run attempt, and the archive identity binds the attempt. A
"Re-run failed jobs" attempt does not rerun the successful producer,
so its consumers look for <name>-2, fail with "Artifact not found",
and every flaky test forces a full rerun.

Give those artifacts one name per run, overwrite them when a producer
does rerun, and bind the archive identity to the run instead of the
attempt. Source, tree, lock, toolchain and environment stay bound, and
an archive from another run is still rejected.
2026-10-30 11:16:10 +08:00
Chris An 4fcbf60f10 ci: unify validation planning, artifacts and suite orchestration (#9059)
* ci: unify validation planning, artifacts and suite orchestration

* fix(ci): preserve JSON diagnostics for release checks

* fix(ci): keep manual attempts without plans visible

* fix(ci): distinguish nextest setup scripts from test results

* ci: reuse extracted archives and consolidate duplicate checks
2026-10-30 02:02:38 +08:00
Chris An b7ff09e357 chore: strengthen agent delivery standards and validation (#9058) 2026-10-29 18:01:05 +08:00
cxymds 4a63e53161 test(validation): run canonical cases and build pinned Heal tools (#8455)
* test(validation): add pinned auto-testing manual lane

* Install pinned ILM client for canonical validation lane

* Prepare Python client for pool validation cases

* Install validation client for replication cases

* test(validation): provide the Python client for Heal cases

* test(validation): build the candidate-bound Heal inspector artifact

* test(validation): expose tool artifacts through the existing manual lane

* test(validation): verify pinned Heal tools before fleet execution

* test(validation): verify Heal tools against released shard captures

* ci: audit retained validation environments before finalization

* test(validation): add optional native Heal regression tooling

* test(validation): route native tools through the existing driver

* Run validation tool resolution on hosted Linux without acquiring the fleet lock

* test(validation): bind C07 to pinned native authority evidence

* test(validation): build optional C10 fixture codec on hosted Linux

* test(ci): bind C10 execution to the verified shared codec

* feat(ci): expose read-only validation fleet inventory

* docs(ci): describe supported validation recovery

* test(ci): add isolated manager native validation builds

* test(ci): expose manager regressions through tools-only dispatch

* ci(validation): consume pinned manager native prerequisites

* fix(heal): preserve typed bucket errors for bounded retries

* fix(heal): preserve typed errors and SSE-C response headers

* ci(validation): publish isolated immutable candidate packages

* test(validation): enforce private candidate manifest schema

* test(validation): verify isolated package publication failures

* ci(validation): dispatch isolated packages through the driver

* fix(heal): deeply verify read repair payload failures

* test(ecstore): verify incarnation fence at repair commit

* ci(validation): verify C11 native storage evidence

* fix(ci): isolate validation package publication credentials

* test(ci): archive scoped F07 directory fsync evidence

* fix(ecstore): retain typed quorum errors for missing parts

* chore(ecstore): shrink typed-error ratchet baseline

* fix(ecstore): retain below-quorum data during no-remove heal

* fix(ci): allow recovery of completed cancelled validation runs

* fix(heal): retain exhausted admin batch repair receipts

* fix(observability): retain typed object error diagnostics

* fix(s3): map typed RPC unavailability to service refusal

* fix(filemeta): classify supported inline framing damage as corrupt

* test(ecstore): exercise heal during committed overwrite cleanup

* ci(validation): admit source-bound G01 native evidence before fleet execution

* test(ecstore): fence version races and bind cleanup slots

* ci(validation): require complete race native runtime preflight

* test(ecstore): fence marker and batch deletes during heal

* ci(validation): require G04 native and runtime admission

* test(ecstore): fence multipart continue complete and abort during heal

* test(ecstore): cover default heal owners and copy metadata races

* ci(validation): admit source-bound owner and copy race proofs

* test(scanner): assert typed corruption after metadata truncation

* fix(ci): require complete native admission for multipart races

* test(validation): wire I05 queued-successor native prerequisites

* fix(scanner): bind object heals to the scanning disk scope

* test(scanner): reuse storage boundary for scope regression

* style: format RPC digest precedence assertion

* fix(heal): query automatic tasks on their owning node

* test(admin): reuse error facade in heal query regression

* fix(heal): retain cancellation after successful bucket storage I/O

* fix(ci): retain validation evidence outside runner temp

* fix(heal): retain resolved object identity on terminal failures

* test(scanner): adapt scope regression to admission receipts

* ci(validation): inspect shared runner without fleet access

* ci(validation): prepare Python before creating client environments

* ci(validation): isolate each client virtual environment

* ci(validation): retain interrupted evidence without fleet recovery

* ci(validation): bind recovery to retained runner evidence

* ci(validation): wire persistence and lock source plans

* fix(validation): align read-only dispatch defaults

---------

Co-authored-by: Chris <anzhengchao@gmail.com>
2026-10-29 08:50:42 +08:00
Hauser a9d3022f37 fix(ecstore): prove multipart part sizes before cleanup (#9037)
* fix(ecstore): prove multipart part sizes before cleanup

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(ecstore): merge strict multipart observation passes

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test: respect lifecycle architecture boundaries

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test: fix multipart E2E clippy warnings

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-29 00:14:32 +08:00
Hauser be485e6cab test(scanner): add memory acceptance runner and budget probes (#8988)
* test(scanner): add memory acceptance runner and budget probes

Add owned S3 workload phases with process, endpoint, cgroup and binary
identity checks for scanner memory and scanner-disabled request bursts.
Keep scale, restart, heap-attribution and overall-acceptance limits explicit.

Characterize healthy and seeded coverage-gap cache retention through the
real folder walker and production cache codec without changing runtime code.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): bind acceptance capacity to effective storage

Validate the serving process's complete local disk inventory before load
writes and during runtime sampling. Resolve each disk inside the owned
storage root, reject nested mounts and filesystem mismatches, and bind
storage observations to phase identity for container and native runs.

Cover the unused-mount bypass, CLI-overridden environment, native paths,
symlink escapes, named volumes and stable multi-disk inventory ordering.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): fence status observations before workload writes

Close each status API window with a second storage-bound process sample
before Phase entry can authorize bucket creation or PUT requests.

Add a fail-before regression for a process change during the initial status
request, asserting no ownership or workload writes occur.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-27 16:45:37 +08:00
Hauser fbd94e129d test(e2e): validate late multipart part transaction packets (#8989)
Retain original prepare and commit-settlement RPCs beyond failed replacement and Abort cleanup, with and without bucket recreation. Require typed refusal or a physically verified idempotent result, unchanged fresh-upload bytes and complete object reads.

Extend scoped proxy observations without rewriting signed frames or trailers, and preserve registered test membership and existing budgets.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-27 13:55:21 +08:00
Hauser aba9e2e9e6 test(e2e): cover partial EC:4 node recovery (#8983)
Verify a sixteen-drive set rejects reads and writes after two nodes stop,
then resumes complete S3 reads and new writes when only one node returns.
Keep the fourth node stopped throughout checks from all three live
coordinators and verify the original 12+4 shard geometry on disk.

Let the existing GET recovery probe retry the compatible SlowDownRead
response only with HTTP 503, retaining fatal status and byte checks.

Register the case in both platform selections and document the boundary
between process recovery and physical disk reattachment acceptance.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-27 12:38:38 +08:00
hector 26d55bb04e ci: remove never-green Performance A/B workflow (#8979) 2026-10-27 11:09:32 +08:00
Hauser c9f7e48a44 fix(observability): correlate physical namespace lock releases (#8974)
* fix(observability): correlate physical namespace lock releases

Map logical namespace guards to their actual granted peer lock IDs and local physical guard identities. Record physical completion only after successful manager release and registry removal, and record validated positive remote unlock responses without changing release behavior.

Exercise real distributed ownership, asynchronous cleanup, duplicate and disabled guards, stale-owner retirement, and explicit release followed by Drop. Document the required diagnostic filters and the remaining filesystem-tail boundaries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): remove unused lock manager test import

The diagnostic regression uses inherent manager methods and does not need the LockManager trait import. Remove it to pass strict warning checks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-27 10:10:17 +08:00
Chris An 1880bdca91 refactor(listing): list latest versions with MinIO's listing quorum (#8965)
* refactor(listing): list latest versions with MinIO's listing quorum

Latest-version SetDisks::list_path now follows MinIO's er.listPath:
ceil(asked/2) is the listing quorum for min_disks, directories and
objects, agreed entries are emitted as read, and a split sample goes
through MetaCacheEntries::resolve. One object can no longer fail a
page. A set with fewer than half of its drives online still fails the
page with ErasureReadQuorum.

Remove the write-quorum layer this replaces: ListingSupplement,
ListingReconciler and the LIST object read lock, the latest_listing_*
quorum helpers, resolve_agreed_entry, resolve_with_write_quorum and the
enforce_write_quorum / write_quorum_slack merge branches, and the
per-version listing quorum helpers in filemeta.

Make merge_file_meta_borrowed_streams pick a version id's most common
header regardless of reader order, so an interrupted null-version
overwrite no longer drops the committed generation from the merge.

* test(listing): do not assert sampled optimal listings are deterministic

optimal asks a random half of the drives, so an EC:8 object committed on nine
of sixteen drives can miss the four-copy listing quorum in roughly one sample
in six, exactly as in MinIO. Keep the deterministic assertion for strict and
for optimal when every online drive is asked.
2026-10-27 01:46:07 +08:00
Hauser fea5a584c3 fix(storage): fence multipart and metadata recovery (#8919)
* perf(ilm): back off idle tier-delete journal recovery

Use bounded idle backoff only after complete empty journal and dispatch scans, while preserving immediate notifications and the retry cadence for observed work.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): reconcile interrupted pool metadata publication

Bind pool metadata CAS and Undo to exact native predecessors. Recover an interrupted physical publication only from an ordinary committed authority and complete verified identity, current and backup evidence, then require normal V3 CAS and quorum readback.

Preserve known-conflict gates, startup and heal lock ordering, targeted admission and the original kill-recovery deadline. Wake proof-ready recovery immediately and bound parallel capacity sampling without removing mutation fences.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(ecstore): reuse proven pool metadata durability

Publish native rollback backups directly in their owned UUID directories. Reuse a one-shot data-durability receipt only for an exact hardlink to the pinned, fully synchronized head and root.

Preserve new-head and directory synchronization, native predecessor CAS, conditional Undo and uncertainty gates. Keep unknown, copied, raw-parent, cancelled and failed operations cold. Add bounded diagnostics, sync-call regressions and path-isolated test hooks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(multipart): exercise selective RPC delay and peer loss

Add authenticated method-entry barriers for metadata and lifecycle locks under e2e-test-hooks. Verify real dispatch, write quorum, prior part payloads, lock drain and incarnation isolation without changing production budgets.

Record cancellation separately from normal exits so late filesystem completion cannot overwrite Drop evidence.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* docs: define multipart delay gates and receipt invariants

Document RPC entry evidence, peer-loss and cancellation oracles, remaining incident attribution limits, and immutable head inodes required for durability receipts.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(readiness): expose missing runtime drive slots

Report configured slots without a runtime handle separately from unavailable inventoried drives. Preserve quorum and probe status semantics. Wait for every coordinator to regain its complete live inventory before multipart RPC fault windows and single-attempt recovery oracles.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): retry committed promotion convergence safely

Require peer reload convergence on every worker attempt, including unchanged retries. Recheck durable identity, replicas, namespace fence, generation and cancellation before allowing any data movement.

Keep known committed promotions active on a verified propagation pause. Release precise owners, retain queued successors and reuse the existing watchdog. Preserve uncertainty gates and terminal operation behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): guard native metadata mutation aliases

Reject protected RenamePart source and destination paths before I/O. Classify the protected namespace through the native path resolver, and require canonical conditional RenameData and Undo keys so aliases cannot create separate lease domains.

Preserve ordinary multipart and canonical CAS behavior, warm durability receipts and zero-allocation canonical fast paths. Cover real alias, byte identity, receipt and normal mutation behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): register Darwin multipart RPC delay gates

Update distributed membership from the actual Darwin listing. Preserve the full-profile digest and leave Linux unchanged until its own listing is produced, as required by the contributor guide.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(multipart): fence native publication and recovery

Publish multipart data through private hardlinks and atomically consume the
prepared source head. Bind exact compensation capabilities and native recovery
to the physical cohort, source generation and durable publication state. Require
fresh terminal confirmation for native replay and preserve indeterminate data
when no safe recovery witness exists.

Release logical guards only after validated irreversible quorum success while
retaining physical mutation leases, accounting and resource ownership through
actual callbacks. Preserve local instance proof authority and historical Copy
metadata without granting healed donors fresh publication authority.

Keep general movement identity strict while preserving directory capacity
compatibility in a separate checked comparison.

Cover native fault and cancellation boundaries, zero-byte objects, real Copy,
heal and migration, exact physical readback and distributed RPC delay. Retain
all quorum and durability requirements and document rollout limitations.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): fix ratchet scan and Windows fault hooks

Move native pool metadata inventory tests after production items so the
error-format ratchet sees the existing call site. Preserve the baseline
instead of accepting a false count decrease.

Make the directory-barrier failure hook available on every test platform
and exercise the parent-barrier failure/retry cases in Windows CI. Keep
production synchronization behavior and all existing assertions intact.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(protos): remove redundant recovery request clone

Move the baseline into its final struct update while retaining the earlier
clone needed to construct the first request variant.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): align RPC delay gates with native abort

* fix(storage): validate native source recovery and CI gates

Preserve wire layouts while bounding native response sizes and fix the
last-use recovery request clone. Avoid destination replay reads only for
fully observed initial native sources; keep upgraded and recovered sources
conservative. Cover real legacy publication after a cold restart.

Align upgrade acceptance with the native mixed-version mutation pause and
use exact upload-scoped test completion evidence before advancing time.
Keep all original quorum, ownership, byte and deadline assertions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): verify native abort under partial admission

Delay native cleanup at the exact upload root after all-participant closure.
Verify quorum-only admissions retain every source byte on refused Abort and
complete the same upload without creating a missing original participant.
Keep sub-quorum orphans and their discoverable identities isolated while a
fresh all-participant upload proves service recovery. Route shared constants
through the existing E2E storage facade.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): avoid formatting a fixed invalid upload path

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): align native MPU fault recovery assertions

Verify real source participants, preserved orphan bytes and same-upload completion after quorum-only admission. Require full physical inventories so an expected read-quorum error cannot hide unrelated leakage. Preserve ordinary quorum and recovery budgets, and document listing availability through the affected set.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): keep MPU tail deadlines on the wall clock

Global Tokio time advancement can expire drive-health probes while real
filesystem callbacks are pending. Use a short real drive deadline with the
existing physical-entry and cleanup barriers, retaining the byte, lock,
resource-permit and final-shard assertions.

Fixes #8937

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(storage): align native fixtures with instance ownership

Open independent disks for the simulated capacity-loss instance, scope four coordinator journal disks without rebinding source authority, and probe configured namespace keys while retaining physical I/O hooks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): isolate native multipart future construction

Construct optional native publication futures behind synchronous heap boundaries so ordinary rename callers do not carry their state machines. Preserve native guards and ownership without adding allocations to ordinary PUT, copy or heal branches.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(heal): assert coordinator journal scope after rejoin

Check the exact original four-member journal map after every rejoin and manager stop while preserving all sixteen native data slots. Reconnect expansion fails the test rather than being filtered away.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): bound object repair future construction

Construct the normal and FileCorrupt deep-retry state machines behind a synchronous heap boundary. Preserve guards and cancellation while bounding the shared rename and deletion call stacks without increasing test stacks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): qualify fixture disk identity checks

Call the same DiskAPI method explicitly so the independently reopened disk identity assertion compiles under every target and feature combination.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): isolate native inspection from mutation state

Select a dedicated read-only inspection future behind the existing single Box allocation. Preserve validation, source ownership and the canonical inspection body while keeping mutation state out of inline read callers. Reject misrouted operations with typed errors and retain source evidence.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): pause abort time after native cleanup

Wait for durable source deletion on the wall clock before advancing the
capacity lease clock at the post-delete barrier. Paused time can otherwise
expire the barrier while native filesystem cleanup is still pending.

Retain lease-loss injection, durable inflight-byte accounting and restart
reconciliation assertions. Document the test-util feature required to
select the native decommission fixtures in package-scoped regressions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): reclaim data from owned multipart undo

Retain exact runtime-only Undo ownership through deferred snapshot cleanup. Revalidate every target metadata version and the retirement anchor, keep the delete claim through the physical callback, and remove the anchor after the payload barrier. Preserve generic cleanup refusal when metadata is missing.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): preserve unformatted bootstrap routes

Keep explicitly unformatted local paths reachable by an in-flight elected formatter while quarantining their service slots and disk-ID membership. Own bounded legacy-read handles so the actual asynchronous creator future remains Send.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(s3): bound CopyObject tracing poll frames

Construct the business future behind an outlined single-Box boundary and lazily instrument its handle. Preserve Send, Unpin, span timing and cancellation without embedding concrete business-state construction in the tracing poll.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(mpu): observe recovery after direct legacy bootstrap

Seed all sixteen real previous-release sources before attaching the same-port method observers. Require exact per-peer disk-route Unimplemented responses with zero publication calls, retain byte and cleanup assertions, and remove superseded raw transport. Register the two new regressions from actual Linux and macOS listings.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(mpu): prepare the homogeneous previous-release fleet

Reuse the existing release baseline before creating the genuine legacy multipart sources. Restart old processes with their normal budgets and require every drive through the reduced-redundancy probes before installing the mixed-version observers and current fault budgets.

Keep standard EC:4, single-attempt calls, all mixed-version byte and reclamation assertions, and their original deadlines unchanged.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(mpu): isolate release compatibility from fault profiles

Keep normal component configuration for the observed mixed-release rollout, which injects no RPC delay or loss. Retain the short component profile in actual method-fault scenarios and preserve the single-attempt 45-second probe and every typed error, routing, source byte and reclamation assertion.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(mpu): use normal shutdown for planned release replacements

Use the existing bounded SIGTERM path during homogeneous release preparation and the observed mixed-version rollout. Preserve every multipart inventory, routing, quorum and byte assertion, and keep actual crash recovery on its original hard-kill path.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* revert: retain the verified release compatibility fixture

Restore the published fixture after the local configuration and lifecycle experiments failed to resolve the macOS startup failures. Keep their diagnostic records separate from the delivered implementation.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(copy): bound inherent handler polling stack

Build the inherent CopyObject future through a synchronous noinline
factory at the existing trait await. This keeps its large state out of
the trait wrapper's active polling frame during historical self-copy.
Preserve cancellation, tracing and post-write notifications, with one
additional allocation restricted to COPY.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): validate authoritative quota and native pool heals

Require actual authoritative quota usage in the existing readiness loop
instead of treating HTTP success or unknown usage as a numeric baseline.
Keep the original quota assertions and convergence budgets.

Allow the owning pool metadata CAS to republish into a fresh valid data
directory while retaining exact version, layout, completeness and shard
fingerprints. Require every node's real metadata/read/write readiness
before accepting recovery. Keep ordinary object manifests and the original
180-second recovery deadline unchanged, with negative oracle regressions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): distinguish certified native Abort recovery

Require native participants with a persisted full closure certificate to retire their heads and reclaim every surviving source on retry. Keep uncertified legacy authority below identity quorum conservative, while retaining actual partial-delete and committed-object byte checks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-26 12:39:54 +08:00
Chris An 4a96f06983 refactor(listing): remove env-gated ListObjects index provider (#8948)
* refactor(listing): keep paginated listing on ECStore only

`Sets` and `SetDisks` carried their own copies of `inner_list_objects_v2`,
`list_objects_generic`, `inner_list_object_versions`, `list_path` and
`list_merged`, plus a third copy of the task that converts merged raw entries
into the walk result stream. Only `ECStore` can answer a paginated listing,
since a page merges every set of every pool; the copies drifted (the `Sets`
copy still rejected `//` prefixes that #8605 made literal on `ECStore`) and
were reachable only from tests and one decommission helper.

Remove the `ListOperations` impls for `Sets` and `SetDisks` together with the
duplicated pagination code. `SetDisks::walk` stays as an inherent method for
set-scoped callers and now honors `versions_sort` like the store-wide walk;
`SetDisks::list_path_result` stays for the recursive delete lister. The walk
entry conversion moves into one `spawn_walk_entry_forwarder` shared by both
walks. The decommission receipt lister walks its pool's sets instead of
paginating through the removed `Sets` listing.

Tests that exercised the removed layers now exercise `ECStore` for pagination
or `SetDisks::walk` for per-pool version inspection. The architecture and
logging guard rows that pinned the removed `SetDisks`/`Sets` listing entry
points are dropped, and the error-other ratchet baseline shrinks by one.

* fix(listing): exclude synthetic directories from receipt walks

* refactor(listing): remove env-gated ListObjects index provider

Delete the experimental, default-off key-only / verified-page /
metadata-fast index provider from `ECStore::list_objects_generic` together
with everything only it drove: the `RUSTFS_LIST_OBJECTS_INDEX_*`,
`RUSTFS_LIST_OBJECTS_NAMESPACE_JOURNAL_*` and
`RUSTFS_LIST_OBJECTS_METADATA_FAST_*` readers, the persistent key-only
index and its on-disk format, the namespace mutation journal (state,
chaos injector, system/local backends, degraded set, global lock), the
index lifecycle/health/source-selection types, the verified-index
candidate pager, the `list-chaos` cargo feature, the
`rustfs_s3_list_objects_index_*` metrics and the archived bench script.

`list_objects_generic` now always walks. The write-path hooks
`observe_list_objects_mutation(s)` are gone; their callers bump the scanner
namespace generation directly through `observe_scanner_namespace_mutations`,
which is the sub-step the old hook performed before taking the journal
lock, so the generation observed by the scanner, the node RPC service and
the data-movement lease checks is unchanged while PUT/DELETE no longer take
a global tokio RwLock or read environment variables.

`ListSourceMode::{metadata_authority, requires_live_metadata_verification,
can_satisfy_strong_listing}` and `ListMetadataAuthority` were only reachable
from the provider and are removed as well; the enum, its variants, the
cursor encoding and `ListContinuationV2` are left untouched.

* docs(listing): define index retirement upgrade prerequisites

* fix(ci): retire the removed listing index test requirement
2026-10-26 12:02:15 +08:00
Hauser fab4bb0586 test(e2e): cover simultaneous multipart terminal contenders (#8943)
* test(e2e): assert Abort refusal below identity quorum

Keep exactly-quorum success and recoverable cleanup assertions. For
minority staging authority, require typed refusal, zero further Delete
RPCs, unchanged per-disk metadata and the previous object body. This
aligns the phase matrix with the existing conservative recovery contract.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): exercise simultaneous multipart terminal contenders

Retain the four pairwise resend cases and add live/cancelled three-way Complete, Abort and resend races. Observe distinct forwarded lock owners and verify terminal bytes and physical cleanup.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): freeze contender evidence before multipart cleanup

Retire the passive lock observer before terminal ListParts checks so the reported count identifies only the two contenders at the preparation barrier.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): require both multipart terminal race outcomes

Retain the free races and select both terminal winners with independently released, owner-scoped lock gates. Verify real contender admission, unchanged signed packets and exact terminal bytes for live and cancelled part waiters.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): isolate losing coordinator lock votes

Hold every remote vote for the losing coordinator while the same gate observes the winner. Prevent partial lock grants from delaying the requested terminal order and retain ordinary RPC caller lifetime.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-26 09:24:38 +08:00
Hauser c610d11e8f fix(storage): guard MPU recovery and startup RPC admission (#8934)
* fix(ecstore): preserve uncertainty after partial multipart abort

Keep recovery callers from reporting NoSuchUpload while minority primary metadata remains. Retain all quorum and deletion checks. Exercise actual partial Abort and explicitly retained late RPC packets after caller retirement, staging cleanup and bucket recreation.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): prevent retired multipart staging resurrection

Require the upload root to exist before preparing recovery publication. Preserve rooted directory walks for nested bases on non-Linux Unix and cover missing-source native work, configured aliases and pinned mixed-version coordinator recovery.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): use IPv4 peer aliases for pinned mixed recovery

Keep direct IPv6 S3 listeners behind IPv4 RPC proxies for the pinned previous release, whose peer health resolver rejects bracketed IPv6 hosts. Preserve signed authorities, ports and strict recovery assertions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): separate pinned startup from MPU fault budgets

Use the old fleet defaults during bootstrap, then apply the original short fault budgets to the current replacement. Keep direct S3 URLs aligned with IPv6 listeners and preserve all quorum and physical cleanup assertions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): seed legacy MPU through transparent transport

Retire raw proxy connections before enabling method faults so legacy streaming setup stays byte-transparent and every later scoped mutation must traverse the fault barrier. Verify raw and adapted protocol authority, bodies and trailers.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): retain direct listeners for legacy MPU peers

Advertise same-port localhost peers for the pinned resolver while retaining direct IPv4 S3 listeners. Exact scoped RPC hit counts still reject bypassed injection and all physical authority assertions remain unchanged.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(server): keep startup gRPC on the authenticated router

Use the hybrid protocol predicate for startup RPC admission so early control responses retain production authentication and owner validation. Keep REST FullReady gating and pinned replay capability rejection intact. Add deterministic regressions, scoped diagnostics and bounded legacy fixture resources.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ci): verify MPU recovery across upgrade source versions

Run the exact sixteen-drive mixed MPU case for every checksum-verified upgrade source, independently of generic compatibility failures. Require one listed test and retain per-version logs while preserving the historical distributed source pin and every recovery assertion.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ci): trigger upgrade MPU checks for fixture changes

Include distributed fixtures and the RPC fault proxy in the upgrade
workflow paths so test-only changes schedule the strict recovery gate.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(storage): verify resend owners across MPU finalization

Hold real part-transaction RPCs while another coordinator completes or
aborts the upload, with both live and cancelled client response waiters.
Require full object bytes, terminal upload state, and all staging roots
to remain consistent after the part owner drains.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-26 01:29:43 +08:00
Hauser 017b85a5bf fix(ecstore): preserve inactive multipart abort authority (#8920)
Retain per-disk recovery metadata through partial publication and require write-quorum proof for completion replay. Keep completed uploads terminal and strengthen RPC recovery acceptance.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-25 14:05:19 +08:00
Hauser 0f24511535 test(e2e): harden multipart fault and TLS acceptance (#8907)
* test(e2e): tighten scoped multipart fault acceptance

Target Abort root deletion instead of child transaction cleanup, require
actual metadata peer hits, and keep faulted APIs single-attempt. Validate
bounded ordinary recovery and safe quorum refusal for partial mutations,
with the recovery enhancement tracked in backlog#2940.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(e2e): drive concurrent TLS reads independently

Give each coordinator its own supervised task and cooperative polling
budget while retaining the existing workload, deadline and byte/hash
assertions. Preserve node logs after fixture teardown and report the
operation, response phase and received bytes when a transfer times out.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-25 04:54:35 +08:00
Hauser 882217221e fix(lock): reach quorum behind silent voters (#8895)
* test(e2e): add MPU RPC quorum and recovery acceptance

Exercise separate storage and lock RPC delays across multipart phases,
including quorum boundaries, cancellation, bucket recreation, and bounded
eight-node load. Honor TMPDIR for isolated cluster drive directories.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(lock): probe healthy voters behind silent peers

Spread additional probes across the existing acquisition attempt budget
when the initial quorum-plus-spare wave stalls. Preserve quorum thresholds,
fast-path fan-out, and existing ownership and late-response cleanup.

Cover every four-node read-voter ordering, a stalled eight-node write
quorum, and normal fan-out. Extend MPU acceptance to EC12+4 at twelve and
eleven metadata acknowledgements.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-24 22:30:31 +08:00
Chris 631cd0ceca fix(ecstore): preserve migrated reads and retry busy pool heals (#8821) 2026-10-23 12:09:38 +09:00
Chris d9ab8760ee test(e2e): accept retried rebalance deferrals; retry busy decom preflight (#8813)
test(e2e): accept retried rebalance deferrals and retry busy decommission preflight
2026-10-23 02:10:04 +09:00
Chris b06923beaf test(e2e): add scanner/heal topology protection matrix (#8705)
* test(e2e): add scanner/heal topology protection matrix

Add the S00 protection baseline for the Scanner/Heal program
(rustfs/backlog#2873, program rustfs/backlog#2872): seven
scanner_heal_topology_ cases in the distributed e2e lane that pin pool
expansion, rebalance and decommission behavior before the runtime changes.

The fixture seeds a deterministic inventory (empty, inline, external,
multipart, special and nested keys; a versioned bucket with an explicit
null version, three history versions and a delete marker per key; an
Object Lock bucket with legal hold and COMPLIANCE retention), records
every version with its requested and listed version id, length and
SHA-256, verifies the manifest through every node, reads physical
placement from xl.meta on the pool drives, and reads the strict
durability mode back from the admin API and the node logs.

Fault cases hold the mover at proven windows through file barriers that
compile only with the e2e-test-hooks feature: before the multipart target
commit and before source cleanup (shared by decommission and rebalance),
plus the existing pool-metadata before_publish cutpoint. An unreached
window fails the case. DistCluster gains capture_logs_under so a case can
read node startup events without a global environment variable.

docs/testing/scanner-heal-topology-gates.md maps the gates to the tests
and records the timeout basis.

* test(e2e): keep the barrier failure message static

The error(format!) ratchet rejects new formatted io::Error::other call
sites in ecstore. The reached observation already records the stage and
object, so the injected failure needs no per-call detail.

* test(e2e): require e2e-test-hooks for topology fault cases

The four scanner_heal_topology_ fault cases inject at server barriers that
exist only with the e2e-test-hooks feature. Against a default server they
waited for the barrier timeout and reported it as not reached. Check the
verified binary's features first so the missing prerequisite fails
immediately and is named.

* ci(e2e): build the distributed lane with e2e-test-hooks

Every other e2e lane builds and runs the fault-test hooks server. The
distributed lane alone used the default server, so the topology fault
cases could never reach their barriers there. Build and run with the same
feature so the receipt matches, and update the documented local commands.
The upgrade cases spawn the pinned previous release directly and do not
compare feature sets.

* test(e2e): keep topology artifacts under RUSTFS_E2E_LOG_DIR

The topology fixture captured node logs, barrier observations and evidence
in a temp dir removed on drop, so the distributed workflow uploaded nothing
for these cases. Fall back to RUSTFS_E2E_LOG_DIR, which the workflow
uploads, when no explicit evidence directory is set.

* test(e2e): remove retired pools before restart; race the pre-ACK start

The queue case restarted with decommissioned pools 0 and 1 still on the
command line, which the server rightly refuses. Add
remove_single_node_pools and restart without them, then expect the
remaining pools renumbered to 0 and 1.

The pre-ACK case now races the start request against the barrier. A start
that returns first, or stays in flight past the documented 60 s budget, is
reported as such instead of as an unreached barrier, and the evidence
records the reach latency.

capture_logs_under now overrides a log path assigned from
RUSTFS_E2E_LOG_DIR, so the seed node logs into the case directory too.

* test(e2e): bound fault-window waits by the movement budget

On Linux CI a busy target capacity gate delayed the first fault object by
about two minutes, past the 120 s wait borrowed from the server's release
poll. Wait for a movement window within the 240 s movement budget instead,
and fail at once when the decommission or rebalance ends without entering
the window, so an uninjected fault is never counted.

* test(e2e): bound status probes while waiting for a fault window

The admin decommission status request can wait on a mover held at a test
barrier until the barrier's own 120 s release timeout. The window wait
issued that request each loop, so a reached barrier went unnoticed until
the server timed it out and failed the decommission, which surfaced as a
false resume failure. Bound each status probe to 5 s and treat a slow probe
as still running.

* test(e2e): retry a busy decommission start

A decommission start that meets a busy movement gate can answer a
retryable 503 instead of waiting. The harness treated any non-proof
failure as fatal. Share one start helper that retries 503 within the
case's movement budget and the fleet-proof 500 for 45 s, and keeps every
other status fatal. The pre-ACK case retries 503 within its window budget.
2026-10-21 22:08:17 +09:00
Chris 473cddd7dc ci: gate pull requests on Clippy and warm its check artifacts (#8708)
Clippy ran only after merge, so lint regressions merged freely and then
failed every main run until fixed (10 of 12 main runs on 2026-10-17/18).
Run the workspace Clippy step on every event, warm its check-mode
artifacts in cache-warm with the identical invocation so pull requests
re-check only what they touch, and pin the unconditional step in the
ci_gate contract and its self-test.
2026-10-20 00:27:40 +09:00
Chris 90749fbb06 ci: run feature lanes nightly and warm caches on a schedule (#8683) 2026-10-19 16:18:22 +09:00
Chris 1ec620ac9c fix(ci): restore the release-profile short-credential guard (#8691)
#8682 replaced the cold `cargo check -p rustfs --release` with a
`cargo rustc ... -C debug-assertions=off` on the rustfs lib alone. That
leaves the dependencies on the dev-profile cfg, so the lib also fails on
items gated on one side of cfg(debug_assertions) only (E0599 on
Filenames::Embedded), and the step's "fails solely at the compile_error"
assertion can never hold; the lane has been red on main and every pull
request since. Restore the whole-graph release check, drop the two-job
limit that only a linking step needed, and record why the single-crate
variant is unsound.
2026-10-19 14:03:57 +08:00
Chris 392ccadb78 ci: build development packages nightly instead of on every main push (#8684) 2026-10-19 11:39:38 +09:00
Chris 4d7cd23f8e ci: reuse the dev cache in Connect boundary lanes and raise build jobs (#8682)
Raise CARGO_BUILD_JOBS to 3 on every event in the workspace test lane and
record the closed backlog#1601 experiment: the test-profile compile took
28m21s at 2 on a pull request and 16-23 min at 3 across three successful
main runs.

Merge the two Connect boundary jobs into connect-feature-boundaries. The
offline enrollment gate no longer exports a fresh CARGO_TARGET_DIR, so it
rebuilds only the rustfs crate on top of the restored ci-dev cache; the E2E
root is isolated by the feature gate and the dedicated binary name, not by
the target directory. The short-credential release guard is now
`cargo rustc -p rustfs --lib --features connect-e2e-short-credentials --
-C debug-assertions=off`, which trips the same cfg(not(debug_assertions))
compile_error against cached dev-profile dependencies instead of a cold
release check of the whole graph. Every assertion of both lanes is kept.

Update ci_gate.py and the ci-gates.md PR matrix for the merged job.
2026-10-19 09:16:12 +09:00
Chris ff70c06cef fix(auth): restrict root-derived accounts and credential logs (#8674) 2026-10-19 08:57:31 +09:00
Chris fca83c72e4 ci: move compile-free and prebuilt-binary lanes to hosted runners (#8679)
* ci: move compile-free and prebuilt-binary lanes to hosted runners

Standard GitHub-hosted runners are free for this public repository, while
every sm-standard minute is paid self-hosted capacity. Move the jobs that
never compile (quick checks, typos, scope selection, the required-checks
aggregate, audit, closed-PR cancellation) and the two s3-tests lanes that
only run the prebuilt debug binary to ubuntu-latest, drop the wget install
the custom image needed for typos, and move the upgrade-compatibility
workflow, whose build shape already runs on hosted runners in
e2e-distributed, with renamed cache keys so the hosted and self-hosted
images never share a rust-cache entry. Gate the package summary job on
resolve having run, so development builds stop starting a pod to report a
skipped job.

* docs(ci): document runner billing and placement rules

Record in .github/workflows/README.md which runner labels GitHub bills
(none for this public repository), which ones RustFS pays for, the rule for
when a job may leave ubuntu-latest, the rust-cache single-writer and
same-image constraints, the patterns that have leaked self-hosted minutes,
and a checklist for new or moved jobs. Point to it from .github/AGENTS.md,
docs/testing/ci-gates.md and the ci.yml placement comment so the rule has a
single home.
2026-10-19 08:55:29 +09:00
唐小鸭 78ea655d87 fix(startup): drive startup on an explicitly sized thread (#8630)
`run_process` called `block_on(async_main())` on the native main thread,
so the whole startup chain ran on whatever stack the platform hands the
main thread: 8 MiB from `ulimit -s` on Linux and macOS, but the 1 MiB PE
default on Windows. A Windows debug build overflowed before it listened.
`RUST_MIN_STACK` does not reach the native main thread.

Run the runtime on a `rustfs-main` thread with an explicit 8 MiB stack,
keeping the budget Linux and macOS already had and making it independent
of the linker, toolchain (msvc/gnu) and `ulimit`. Panics are re-raised
on the main thread so the exit status is unchanged.

Measured on macOS arm64 by lowering the main-thread stack with
`ulimit -s` before this change: debug startup overflows at 1792 KiB and
reaches /health/ready at 2048 KiB; release overflows at 512 KiB and is
ready at 768 KiB. After it, debug starts with a 256 KiB main thread.

Add a Windows `Startup Smoke` job that builds the debug server, runs
`--version`, and waits for /health/ready on a single volume.
2026-10-17 16:06:19 +08:00