* fix(ecstore)!: bind bitrot shards to immutable part identities
Verify part, coding-index, and block identity across write, GET, and Heal
paths. Preserve identities across metadata-only copies and repair, include
them in multipart quorum selection, and require payload proof for receipts.
Keep legacy decoding with conservative parity and target-digest validation,
and document its unsupported cases and additional verification I/O.
BREAKING CHANGE: New bound-v1 shards require compatible readers throughout
the fleet. Legacy objects without sufficient integrity evidence return an
error; binary rollback after new writes requires verified data migration.
Refs: rustfs/backlog#2497
* fix(ecstore): preserve shard framing with independent integrity
Commit immutable part-generation Merkle roots and replicated proof indexes without changing existing checksum frames. Verify reads, reconstruction and Deep Heal against metadata quorum; keep legacy reads and explicitly defer unproven legacy data repair.
Preserve multipart rollback generations, require acknowledged durable index publication, and add decoder compatibility and donor-shard regression coverage.
* fix(heal): verify protected partial-write replay
* fix(heal): rebuild truncated xl.meta from healthy quorum
* fix(test): pass topology to heal overlap RPC regression
* fix(test): drive heal admission alongside partial PUT
Poll the partial PUT and its mock heal receiver together, bound their handshake, and retain the existing repair-scope assertions.
* fix(test): prepare durable MRF fixtures and Linux heal stack
* fix(test): drive tier cleanup recovery after deferred attempts
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Allow a confirmed scanner root data-usage CAS write to prove its own publication when the follow-up root readback cannot provide a proof. Keep AlreadyDurable and all stale or companion paths on the existing readback-only proof boundary.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): persist and retry partial-write repairs
* test(heal): pass topology to overlap RPC tests
Use the existing coordinator endpoint fixture for the three overlap-test
calls to the endpoint-aware heal control executor. This repairs the E0061
test-build failure inherited from the release base.
Remove the superseded ServiceUnavailable-only PUT helper after the G14 outage PUT probe switched to the shared retry classifier.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Treat SlowDownRead as a retryable outage PUT probe response in the G14 multi-pool runner, and label terminal candidate failures with stage context.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Treat SlowDownRead as a bounded retryable deferred PUT response after the target pool rejoins, and label terminal deferred outage PUT failures with stage context.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Allow the Scanner/Heal interruption oracle to retry transient retryable GET failures after replacement recovery has converged. The readback still verifies exact object bytes and keeps a bounded timeout, so permanently unreadable objects continue to fail the case.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(storage): alias legacy meta bucket over internode rpc
Retry read-only internode RPC metadata access from legacy .minio.sys to .rustfs.sys when mixed-version peers report missing metadata during rolling upgrades.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): target durable ILM receipt quorum fixture
Use the actual durable ILM receipt object path when taking target disks offline so the test exercises receipt write quorum instead of whichever set owns the source record path.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): reuse API error boundary for decoding failures
* Update heal.rs
Signed-off-by: houseme <housemecn@gmail.com>
* test(ecstore): synchronize batch cleanup metadata reads
Acquire object read locks while observing unversioned and explicit-version batch cleanup, then release them before waiting for progress. This prevents snapshots from spanning per-disk marker removal while retaining the existing quorum, timeout, and remote delete count assertions.
Validation: cargo fmt --all --check and git diff --check passed. The focused nextest test passed 20 stress iterations each with test-util and test-util,rio-v2, with retries disabled.
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
* test(e2e): target multi-set outage heal candidate
Require the outage write used by EC8+4 multi-set root-heal evidence to miss the same erasure index owned by the selected replacement drive. This avoids accepting a candidate from a different set and turning a valid heal into a false negative.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(e2e): satisfy G14 heal lint gates
Remove clippy-only noise from the G14 multi-set heal evidence test and align the admin route policy inventory with the registered heal catch-all route.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* update
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(e2e): record EC8+4 drive restart set size
Include the erasure set drive count in the distributed EC8+4 drive restart oracle so Scanner/Heal release evidence matches the registry contract.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(s3): tighten s3s footprint ratchet
Replace release-merge s3_error! macro calls with equivalent S3Error constructors so the s3gate migration ratchet does not grow on the PR merge tree.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Keep the multi-pool evidence runner within the registry object budget while retaining deferred outage-write diagnostics for release-gate validation.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Include optional outage-write diagnostics in G14 evidence oracles so deferred multi-pool outage writes bind their down-window refusals and post-rejoin acceptance.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
test(e2e): keep multi-set restart graceful
Include the EC8+4 multi-set restart scenario in the graceful interruption lane so the unclean-shutdown marker assertion matches the scenario semantics.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Use the pool-wide namespace lock client domain for every set in a pool so degraded EC8+4 multi-set writes are gated by node-level lock quorum instead of the narrower per-set endpoint host slice.
Keep namespace-lock domain deduplication tied to both the pool namespace and shared clients, and add regression coverage for three-locker degraded writes plus cross-pool domain separation.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
fix(ecstore): admit metadata snapshots at read quorum
Allow guarded bucket metadata snapshot existence checks to use read quorum so degraded erasure sets can continue Object Lock snapshot reads without weakening bucket mutation or object write quorum.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
test(scanner): align G14 multi-pool outage evidence
Treat the localhost multi-pool topology as whole-pool loss when the target node is down. If that topology cannot admit the outage object while the pool is offline, defer that object write until the pool rejoins and record the oracle marker instead of failing before the real crash/restart evidence runs.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Choose the replacement disk from the target node by the presence of complete pool metadata instead of assuming the first configured drive is the scanner metadata holder. This keeps the G14 multi-set harness aligned with multi-drive EC layouts and adds clearer diagnostics when multi-pool outage writes fail closed.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Require lost scoped dirty-usage ACK reconciliation to observe a clean peer activity generation that covers the requested ACK generation. Apply the same guard in the scanner aggregation path so a stale clean activity snapshot cannot discharge pending maintenance after an uncertain ACK response.
Refs rustfs/backlog#2427
Refs rustfs/backlog#2281
Refs rustfs/backlog#2240
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bound native backlog fixture resources
Keep native scanner backlog restart fixtures within low file descriptor limits by reducing the native-only disk layout, making fixture shutdown drain background work, and avoiding long-lived ECStore retention from background loops.
Convert ECStore-backed background refresh/recovery/monitor tasks to upgrade weak owners only while doing work so completed test stores release their disk graph before the next native fixture starts.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(tier): refresh release-merged failure fixtures
Reuse the raw legacy-transition fixture helper so delete-all expiry actually exercises Unknown transitioned history metadata after the release merge.
Rewrite restore-failure disk fixtures through per-disk xl.meta snapshots so stale destination identities can be persisted without tripping ordinary metadata update guards.
Update the SSE KMS mismatch expectation to the InvalidRequest/context_mismatch behavior now returned by the rio-v2 path.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(nightly): publish packages as assets of the rolling 'nightly' release (sync from release) (#7593)
feat(nightly): publish packages as assets of the rolling 'nightly' release (#7592)
Replace the assets-branch scheme with a proper GitHub Release on
rustfs/auto-testing: a single 'nightly' release whose deb/rpm assets
are replaced in place on every build. This is the standard channel —
visible on the repo's Releases page, stable download URLs, no git
history growth (release assets live outside the repository).
- New scripts/release/publish_nightly_assets.sh: resolves-or-creates
the 'nightly' release via the REST API, deletes same-name assets,
uploads rustfs-nightly-latest.{deb,rpm}, then PATCHes the release
body with the build provenance (ref@sha, run link, sizes, SHA256).
Plain curl + python3, no gh CLI (the build fleet has none — #7586).
- The workflow step shrinks to invoking the script; full flow
exercised end-to-end against the real release with probe files
(create / upload / overwrite / download round-trip / body update).
* test(scanner): stabilize W13 release gate evidence
Treat only real raw-entry windows as replayed scanner enumeration in the restart diagnostic, so final completion rounds without raw entries are not fail-closed as raw replays.
Allow the EC8:4 multi-pool heal evidence case to select an outage object key that routes to an online pool while an entire target pool is down, preserving strict behavior for single-pool cases.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Classify a final remote scanner publication lease validation failure from a fresh activity snapshot instead of treating every validation error as data movement. If the peer session and movement generation still match the granted leases and publication is otherwise allowed, keep the publication rejected as an activity-baseline miss without creating pause backlog movement debt. Preserve DataMovement for expired leases, peer restarts, movement generation changes, and active publication blocks.
Tests cover namespace-only validation invalidation and the remote fence-loss cases that must still defer as DataMovement.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
cxymds