fix(ecstore): hedge data-only GET with parity
Route the opt-in data-shards-only lockstep path through the bounded parity race and preserve deferred parity reserves across canceled hedges.
Co-authored-by: heihutu <heihutu@gmail.com>
The transition_matrix_tests use #[serial_test::serial] which has no
effect under nextest (each test runs in a separate process). When running
alongside thousands of other ecstore tests, the shared metadata cache
generation counter can race, causing intermittent 'metadata read should
publish the generation under test' panics.
Add both tests to the ecstore-serial-flaky test group in both default
and ci nextest profiles so they run single-threaded.
Use the seqlock version as the publication fence and keep payload reads relaxed while validating the final version. This reduces ordering overhead in disk metric recording and snapshot collection without changing the rolling-window contract.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): classify system metadata volume failures
Preserve retryable quorum errors when system metadata reads or writes encounter missing volumes, and cover the create-bucket data-usage path with regressions.
Co-authored-by: heihutu <heihutu@gmail.com>
The check_preview_release_workflow.sh script uses exact line matching
(grep -Fxq) to verify the cleanup-preview-releases job contains:
gh release delete "$preview_tag" --yes
The extra --repo flag is unnecessary in GitHub Actions context since
gh auto-detects the repository from GITHUB_REPOSITORY, and it causes
the Workflow Pin Report check to fail on all PRs.
* fix(kms): restore persisted configuration after restart
* docs(kms): cover the reload route and startup load states
The admin contract matrix pins every dynamic KMS route for the rc and
console handoff, so the new POST /kms/reload needs a row there, and the
reload response reuses the configure snapshot shape rather than adding a
wire type. The observability runbook gains the operator procedure the
reload exists for: telling a load_failed startup apart from a server
that was never configured, and recovering without resubmitting secrets.
fix(replication): surface per-object failures and abort orphaned multipart uploads
Replication could mark an object FAILED with no server-log line naming
the object: the target-offline skip paths logged at debug without the
object key, and several failure branches omitted the key entirely. A
failed multipart transfer also leaked its incomplete upload on the
target, since nothing ever aborted it after CreateMultipartUpload
succeeded.
Log the offline skips at warn with the object key (they report the
object FAILED, matching the per-object put_object failure level), add
the object field to the remaining failure branches, and abort the
target-side multipart upload best-effort on any post-create failure
without masking the original transfer error.
Fixes#6825
fix(replication): never mint delete markers when replicating a version purge
Heal/resync/MRF rebuilds of a delete-marker version purge carry
delete_marker: true together with a purge-shaped entry. Passing that flag
straight into replication_delete_remove_options made the target DELETE
omit the versionId (marker-creation semantics), so a generic S3 target
that ignores the internal source-version headers minted a fresh delete
marker on every retry instead of purging one — the marker count on the
target grew monotonically (rustfs#6823).
- Gate marker-creation semantics on the new pure helper
delete_replication_creates_marker (delete_marker && !version purge) so
a purge always addresses the exact version.
- Stop falling through to the marker-creation send when the pre-send
source delete-marker verification fails with a transient error; fail
the entry instead so the MRF replay / heal scanner retries without
minting a marker on the target.
- Pin the purge-shape contract with unit tests in
crates/replication/src/delete.rs.
houseme