Add idempotent terminal transition handling for tier mutation intents and a compare-and-swap record update helper backed by config-object ETags. Cover commit and abort retries, conflicting terminal updates, stale ETag rejection, and exact-limit scan pagination.
Co-authored-by: heihutu <heihutu@gmail.com>
Add canonical record-object naming and crate-private save, load, and idempotent delete helpers for tier mutation intents. Cover malformed record keys, mismatched persisted mutation IDs, and an ECStore config-object round trip under test-util.
Co-authored-by: heihutu <heihutu@gmail.com>
Add an internal durable tier mutation intent model for the #1357 distributed fencing work. The new model validates schema, checksum, canonical targets, mutation-specific target identity shape, config ETags, expiry, and prepared-to-terminal state transitions without changing the production mutation path yet.
Co-authored-by: heihutu <heihutu@gmail.com>
Retry cleanup-pending transition transaction recovery after restart, keeping committed remote bodies when local metadata already references them and preserving records when remote cleanup fails.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ilm): accept null version transition sources
Treat nil/null source version IDs as null-version transition sources when building transition transaction source identity, while preserving fail-closed validation for truly missing IDs on versioned sources.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ilm): preserve multipart source versions
Propagate bucket versioning into CompleteMultipartUpload, assign a concrete version ID for versioned multipart completions, and classify unversioned FileInfo sources as null-version transition sources.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ilm): normalize multipart completion versions
Treat nil staged version IDs as missing for versioned multipart completion and clear staged version IDs when completion publishes a null-version object under suspended or unversioned semantics.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ilm): recover uploaded transition transactions
Persist transition transaction records through production transition uploads, use transaction-scoped remote object names, and start a conservative recovery loop for uploaded candidates.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ilm): reconcile committed transition records
Drop LocalCommitStarted transaction records only after object metadata confirms that the local transition commit already points at the same tier object and version.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* chore(deps): refresh workspace dependencies
Refresh compatible workspace dependencies while preserving the requested
version pins. Update async-nats and Hyper, and replace the temporary Hyper
Git patch with the released crate.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(deps): bump hotpath to 0.21.5
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
fix(site-replication): inject local deployment into replicate-add payload when omitted
The web console's "Set Up Site Replication" flow posts only the remote peer(s) to
/rustfs/admin/v3/site-replication/add and omits the local deployment. The add
preflight requires the local deployment to be present
(validate_add_preflight_topology), so the console's request failed with
"site replication add request must include the local deployment" and no
replication was configured from the web UI.
Inject the local site into the sites list when the payload does not already
include it (matched by endpoint identity), before validation. `mc admin
replicate add` always sends every site, so this is a no-op for the CLI; the
local site carries no credentials (validate_add_sites already skips credential
checks for it).
Adds unit tests for inject-when-missing and no-op-when-already-present.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tiering): make rejected upload cleanup durable
* fix(tiering): close transition upload cancellation gap
* test(tiering): cover failed upload without candidate
* test(tiering): synchronize cancelled cleanup recovery
* test(tiering): stabilize cancelled cleanup recovery
Prefer cancellation when the tier delete journal recovery worker is racing an immediate tick, and build the cancelled-cleanup regression store with an already-cancelled token so production recovery cannot consume the test journal.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Restore the workspace reqwest default feature stack for RustFS outbound HTTPS clients, while keeping per-crate extra APIs such as json, stream, and multipart explicit. Lazily initialize the notification runtime from admin target access when RUSTFS_NOTIFY_ENABLE=true is already effective, and add regression coverage for HTTPS webhook custom CA handling and target-list visibility.
Fixes#5052.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(site-replication): count replicated buckets as in-sync in replicate status
`mc admin replicate status` reported "0/N Buckets in sync" with a cross mark for
every bucket even when replication was healthy and objects had propagated.
build_sr_info stores each bucket's replication_config in the wire form as
base64-encoded XML (raw_config_to_base64), but site_replication_config_mismatch
XML-parsed that string directly without base64-decoding. The parse always
failed, so every bucket with a replication config was reported as a config
mismatch (replication_cfg_mismatch = true), which the status endpoint and mc
render as out-of-sync.
Decode the wire value (tolerant base64 decode, falling back to raw bytes) before
XML-parsing. Adds a regression test that feeds the base64 wire form used in
production; the existing tests only exercised raw XML, which masked the bug.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
fix(site-replication): report peer sync state in replicate info and surface pre-existing back-fill failures
BUG 1 — blank/Unknown Sync in `mc admin replicate info` and the console:
The info endpoint (SiteReplicationInfoHandler) serializes the persisted peer
map, whose sync_state is constructed as Unknown and never updated; the health
derivation added earlier only runs in build_status_info (the status endpoint).
Persist sync_state = Enable for peers on the add and join enable paths
(promote only Unknown, never clobber an explicit Disable). build_status_info
still refines live health for the status endpoint, so info/status/console and
peer_states all report a real sync state for a healthy peer.
BUG 2 — pre-existing-bucket back-fill failures were silently swallowed:
backfill_existing_buckets_after_add returned () and the add handler reported
success with an initial_sync_error_message that only ever carried
metadata-bootstrap errors, so a pre-existing bucket that failed to propagate
was invisible. Return per-bucket failures (including the previously silent
Ok(false) runtime-unavailable no-op and the versioning-failure drop), fold
them into the add response via compose_initial_sync_error_message, and log
reverse-direction gaps on the join path.
Adds unit tests for sync_state promotion (and Disable preservation) and for
back-fill failure surfacing.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Point the cla-bot config and the PR template at cla/v2.md so new
contributions are checked against the v2 Individual CLA.
- .github/cla.yml: document.version v1 -> v2, url -> cla/v2.md
- .github/pull_request_template.md: CLA link -> cla/v2.md
Commit aec2ee9ec (#5005) enforced that RPC secrets cannot be derived
from the public default credentials. This broke all multi-node cluster
E2E tests that relied on the constructor's DEFAULT_ACCESS_KEY /
DEFAULT_SECRET_KEY defaults.
Move the non-default credential and RUSTFS_RPC_SECRET blanking into
the RustFSTestClusterEnvironment constructor so every cluster test
starts with a valid RPC secret derivation base. Remove the per-test
overrides in cluster_multidrive_pool_test that were added as a partial
fix in #5005.
Add a deterministic transition matrix case for a stale prepared metadata snapshot followed by a committed transition, duplicate transition, and late GET. The test proves the stale metadata generation is physically reclaimed, duplicate transition does not upload a second remote candidate, and the late GET reads the committed remote body byte-for-byte instead of reopening the released local source.
Refs rustfs/backlog#1355
Co-authored-by: heihutu <heihutu@gmail.com>
Generate a unique Pulsar producer name for each producer instance while preserving the target type and target id context. This avoids reload-time collisions when an old producer is still connected while a new destination is initialized.
Co-authored-by: heihutu <heihutu@gmail.com>
houseme