* feat(odm): enable on-demand migration by default
The module switch RUSTFS_ON_DEMAND_MIGRATION_ENABLED now defaults to true,
so the feature is reachable without an opt-in; setting it to false still
keeps the module out of the read path entirely. A bucket without an
on-demand-migration.json is never resolved by the runtime and makes no
source call, so the flip changes nothing for unconfigured buckets.
The admin plane now reads the switch through the predicate published by
module_switches.rs instead of its own duplicated env constant; the
behaviour (an environment read per call) is unchanged.
* test(e2e): wire three on-demand migration cases into e2e-smoke
The PR smoke lane gains one case per user-visible contract: a GET miss
that pulls and persists, a HEAD miss that answers from the source and
stores nothing, and the admin config/status pair that must redact the
source secret. The HEAD case did not exist outside the nightly
real-source lane, so it is added to get_basic_test.
Measured on darwin: the lane goes from 168 tests in 101.98 s to 171
tests in 101.92 s, since the three cases overlap the lane's existing
work. The darwin selection digests for e2e-smoke and e2e-full are
regenerated; the e2e-full linux digest still needs a Linux runner.
* docs(changelog): record the on-demand migration feature
* feat(ecstore): add on-demand migration backfill job core
Add the background backfill job for on-demand migration
(rustfs/backlog#2159): a durable checkpoint under
buckets/<bucket>/on-demand-migration-backfill.json saved by If-Match
compare-and-set every 1000 keys or 10 s, a 60 s owner lease renewed by
every save, a recovery pass that takes over expired leases (or jobs this
node owned before a restart) and cancels jobs whose config changed, and a
main loop over the source ListObjectsV2 pages with the skip_existing
policy, dry runs, bounded outstanding pulls and wait-on-full enqueueing.
The pull queue gains per-job completion reports so the job can count
pulled/failed keys (hashes only), and pull permits become two-tier so an
online miss is never queued behind a backfill pull.
* feat(admin): expose on-demand migration backfill job
Wire the ODM-12 backfill job (rustfs/backlog#2159) to its operators:
POST /v3/on-demand-migration/{bucket}/backfill?op=start|cancel and
GET .../backfill return the checkpoint document, GET .../status gains a
backfill summary, and the recovery loop plus the process-wide runner are
installed at startup. Backfill control reuses
Set/GetBucketOnDemandMigrationAction and is recorded in the route policy,
the registration matrix and the admin route snapshot.
Add the rustfs-madmin wire types and client methods with golden fixtures
shared by the server tests, the backfill_* metric descriptors and their
collector, and three e2e scenarios: a full backfill across list pages,
cancellation, and resuming from the persisted continuation token after a
server restart.
The guard added in #7021 fails a >5 GiB single-PutObject replication up
front instead of streaming the body to a target that must reject it. Its
message asserted a conclusion: "was not written as multipart on the
source ... re-upload it with multipart". That text is only as right as
the transport decision feeding it, and until #7047 that decision was
wrong for multipart objects carrying a full-object checksum. On 1.0.0-rc.5
a 768-part object was misrouted to the single-PUT path, and the new
default-level error line told the operator to re-upload as multipart an
object whose own ETag ended in -768.
State the evidence instead of the conclusion. The message now quotes the
ETag the decision was read from and says what was read from it (no
part-count suffix), so an operator can check the line against the
object's listing. A misroute then reads as a visible contradiction --
a suffixed ETag on a single-PUT line -- and the message says that case is
a transport-selection defect to report, not something to fix by
re-uploading. A missing or empty ETag is printed as <none> rather than
hidden.
The routing itself is already fixed by #7047; this changes only what the
guard says when it fires.
* feat(ecstore): add on-demand migration pull queue and write-back pipeline
Background pull queue per bucket (bounded by pull_queue_capacity, concurrency via the state's pull slot), OdmWriteBack/PullSource traits, single-part and multipart write-back with a pumped body that enforces idle timeout, cancel and content length, retry policy for retryable source errors, inline commit helper, and stats accounting (rustfs/backlog#2153).
* feat(object): implement on-demand migration write-back over internal put
OdmWriteBack impl mapping source heads onto InternalPutContext (content-header allowlist, x-amz-meta copy, tags, dual-prefix odm-* provenance, ETag policy), injected into OnDemandMigrationSys at startup; removes the dead-code gates left by ODM-06a (rustfs/backlog#2153).
* feat(ecstore): add on-demand migration bucket config model
Introduce OnDemandMigrationConfig (deny_unknown_fields, version 1) with typed validation, credential redaction, a secret-free Debug impl, and the OnceLock publish hook the runtime registers into. Exported through the api facade.
* feat(ecstore): persist on-demand migration config in bucket metadata
Store the config as a RustFS extension entry (on-demand-migration.json) with its update time in .metadata.bin, add the typed BucketMetadataSys accessor, and publish the config through the hook on every cache-install path alongside the durability sync.
* refactor(ecstore): extract shared remote S3 client builder
Move the aws_sdk_s3 client construction out of bucket_target_sys into
bucket/remote_s3_client.rs: endpoint assembly, credential provider,
path-style selection, custom CA / skip-TLS transports and the outbound
SSRF gate now build from a neutral RemoteS3EndpointSpec so replication
targets and the upcoming on-demand migration source client share one
policy. Replication builds its client through From<&BucketTarget>; the
gate keeps its relaxed semantics (private allowed, loopback only behind
RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) verbatim. The builder also
gains optional connect/read timeouts and a User-Agent suffix
interceptor, both unset for replication.
Refs rustfs/backlog#2149
* feat(ecstore): add on-demand migration SourceClient
Add bucket/on_demand_migration/source_client.rs on top of the shared
remote S3 builder: HEAD, ranged streaming GET, ListObjectsV2 with
source-prefix mapping, GetObjectTagging and an admin probe. Every request
carries the x-rustfs-/x-minio-source-proxy-request anti-loop markers and
a RustFS-OnDemandMigration/<version> User-Agent suffix; SSE-C source
objects are rejected as unsupported. SourceError classifies SDK failures
(not found, access denied, throttled, timeout, connect, server error)
with retryability and a stable metrics label. Debug output redacts
credentials.
Refs rustfs/backlog#2149
* docs(operations): point outbound policy at shared remote S3 client builder
* chore: integrate ODM-01 and ODM-02 as B1 base (fix facade merge)
* feat(ecstore): add on-demand migration runtime OnDemandMigrationSys
Per-node runtime for On-Demand Migration (rustfs/backlog#2152): turns each
bucket's persisted config into a live SourceClient guarded by a three-state
circuit breaker, a TTL negative cache, per-key singleflight, a pull
concurrency semaphore and lock-free counters with a serializable snapshot.
- sys.rs: OnceLock singleton; `apply` installs/rebuilds/removes bucket state
(config compared by value, counters preserved across rebuilds, old
cancellation token fired); `publish` is the metadata publish-hook entry
(sync removal, spawned install, generation-ordered so a slow older install
cannot overwrite a newer one); `resolve(bucket, key)` judges module switch,
bucket state, prefix filter, client availability, negative cache, breaker.
- breaker.rs: Closed/Open/HalfOpen with fixed constants (5 failures / 30 s
window / 30 s open / 1 probe); NotFound resets, AccessDenied is neutral.
- negative_cache.rs: moka sync cache keyed by local key, ttl=0 disables.
- stats.rs: requests_total{op,outcome}, pulled_bytes/objects, pull_failures,
inflight/queue gauges, log-bucket latency histogram, last_source_error;
snake_case snapshot pinned by a golden JSON test.
- Anonymous sources surface as a typed `OdmStateError::AnonymousUnsupported`
until the shared client builder gains an anonymous mode.
- rustfs: `RUSTFS_ON_DEMAND_MIGRATION_ENABLED` module switch (default false)
published to module_switches and injected into ecstore before bucket
metadata loads; hook registered at the same point.
* feat(ecstore): add on-demand migration bucket config model
Introduce OnDemandMigrationConfig (deny_unknown_fields, version 1) with typed validation, credential redaction, a secret-free Debug impl, and the OnceLock publish hook the runtime registers into. Exported through the api facade.
* feat(ecstore): persist on-demand migration config in bucket metadata
Store the config as a RustFS extension entry (on-demand-migration.json) with its update time in .metadata.bin, add the typed BucketMetadataSys accessor, and publish the config through the hook on every cache-install path alongside the durability sync.
* test(e2e): rename stall timing variable flagged by typos
* test(storage): heap-pin the RestoreObject usecase future in the generation guard test
* refactor(ecstore): extract shared remote S3 client builder
Move the aws_sdk_s3 client construction out of bucket_target_sys into
bucket/remote_s3_client.rs: endpoint assembly, credential provider,
path-style selection, custom CA / skip-TLS transports and the outbound
SSRF gate now build from a neutral RemoteS3EndpointSpec so replication
targets and the upcoming on-demand migration source client share one
policy. Replication builds its client through From<&BucketTarget>; the
gate keeps its relaxed semantics (private allowed, loopback only behind
RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) verbatim. The builder also
gains optional connect/read timeouts and a User-Agent suffix
interceptor, both unset for replication.
Refs rustfs/backlog#2149
* feat(ecstore): add on-demand migration SourceClient
Add bucket/on_demand_migration/source_client.rs on top of the shared
remote S3 builder: HEAD, ranged streaming GET, ListObjectsV2 with
source-prefix mapping, GetObjectTagging and an admin probe. Every request
carries the x-rustfs-/x-minio-source-proxy-request anti-loop markers and
a RustFS-OnDemandMigration/<version> User-Agent suffix; SSE-C source
objects are rejected as unsupported. SourceError classifies SDK failures
(not found, access denied, throttled, timeout, connect, server error)
with retryability and a stable metrics label. Debug output redacts
credentials.
Refs rustfs/backlog#2149
* docs(operations): point outbound policy at shared remote S3 client builder
* fix(ilm): enqueue committed tier free versions
* fix(ilm): stabilize causal cleanup CI coverage
* test(ilm): make expire GET race deterministic
* test(ilm): synchronize expiry with active GET
A 6 GiB object uploaded to the source as a 768-part multipart upload was
replicated to a generic S3 target with a single PutObject, and the target
rejected the body with EntityTooLarge. No CreateMultipartUpload was ever
issued, so the multipart replication transport never ran for the object
it exists for.
`replication_put_object_options` seeded the transport from
`object_info.is_multipart()` and then overwrote it with the second
return value of `decrypt_checksums`. Those two booleans do not mean the
same thing: the first is the object's storage shape, read from the ETag,
while the second reports whether the stored *checksum record* carries
per-part data. A full-object checksum -- what `aws s3 cp` writes by
default for a CRC algorithm -- is serialized with no MULTIPART flag even
on a multipart upload, so the record reports false and the object was
routed as a single PUT. `decrypt_checksums` documents this in
object_api/types.rs: callers that need routing must consult
`is_multipart()`. Replication did the opposite.
Route on the object's own shape, and let the checksum record only add
multipart-ness, never take it away. Objects already stored with such a
record are fixed too: the ETag was always right.
This also repairs the diagnosis of rustfs#6825, where the single-PUT
5 GiB guard fired against an object that was multipart all along and
told the operator to re-upload it as multipart.
Tests cover the three shapes the router has to separate: a multipart
object with a full-object checksum record (the regression, which fails
without this change), a multipart object with a composite record, and a
single-part object that must not be promoted onto multipart.
Replication could fail an object with nothing in the server log an
operator could act on. Every failure branch in the resyncer is quieter
than `error` on purpose — most sit on the hot path and fire once per
object per ARN — but `DEFAULT_LOG_LEVEL` is `error`, so on a stock
deployment a failed object produced no line at all. Raising those
branches to `warn` (#6840) did not close this: the default filter still
dropped them.
Report the terminal outcome instead of the branches. `replicate_object_
with_outcome` and `replicate_delete_with_outcome` now emit one `error`
per failed (object, target) once the per-target results are merged,
carrying the object key, version id, target ARN and endpoint, and the
target's own error, redacted through `sanitize_resync_error_detail` so
an echoed credential cannot reach the log. Volume is bounded by objects
that actually fail rather than by attempts inside a transfer.
Also state the single-PutObject size limit instead of discovering it at
the target. Replication picks its transport from the source object's
storage shape, not its size, so an object written with one PutObject
replicates with one PutObject however large it is — and S3 caps that at
5 GiB. Such an object could never reach a generic S3 target, and only
found out after streaming the whole body. `replication_single_put_size_
error` fails it up front with a message naming the size, the limit, and
the remedy.
Version-identity drift moves to `error` on a 10-minute per-ARN throttle.
It was `warn` deduped once per ARN per process, so the one line
explaining why a purged version is still on the target was both filtered
out by default and gone for good after it first fired.
Fixes#6825
Refs #6822
`build_metrics_summary` emitted a single metric entry for the local
deployment with `online` hardcoded to `true` and `last_online` stamped
with the current time, so `mc admin replicate status` reported "I am
online" rather than whether the remote site was reachable. A peer could
be down for minutes with replication failing while the status page
stayed green, leaving operators with no signal that the link had
dropped.
Emit an entry for every peer instead, deriving `online` from the
`reachable_peers` set the handler already computes by probing each peer,
and take `total_downtime`/`last_online` from the replication heartbeat's
existing `EpHealth` tracking. Node-local replication counters stay on
the local entry so a two-site cluster does not double-count its own
traffic.
The new `BucketTargetSys::endpoint_health` accessor deliberately does not
call `init_hc`: unlike `is_offline` it must not create health entries as
a side effect, or merely rendering the status page would mark an unknown
peer online.
Failure counters (`Errors`) are unchanged and still read zero; that is a
separate defect in the bucket-level statistics path and is not addressed
here.
A replicated version purge reaches the peer without the governance
bypass header, so a GOVERNANCE-retained version deleted on the source
with x-amz-bypass-governance-retention was rejected by the peer's WORM
deletion gate forever: retryStats ended at a permanent failed count and
the sites stayed diverged (issue #6850).
The source is authoritative for such a purge: the same WORM gate
already ran there, and GOVERNANCE retention with an authorized bypass
is the only lock state it can purge through. The peer's commit-time
deletion gate now treats an authorized replication delete addressed to
an explicit version as carrying that judged bypass, reusing the same
trust judgment as the replication write exemption
(ObjectOptions::replication_request, set only after the handler
authorized ReplicateDeleteAction). COMPLIANCE retention and legal hold
keep blocking replicated purges, and a plain client delete without the
bypass header stays rejected.
fix(replication): surface per-object failures and abort orphaned multipart uploads
Replication could mark an object FAILED with no server-log line naming
the object: the target-offline skip paths logged at debug without the
object key, and several failure branches omitted the key entirely. A
failed multipart transfer also leaked its incomplete upload on the
target, since nothing ever aborted it after CreateMultipartUpload
succeeded.
Log the offline skips at warn with the object key (they report the
object FAILED, matching the per-object put_object failure level), add
the object field to the remaining failure branches, and abort the
target-side multipart upload best-effort on any post-create failure
without masking the original transfer error.
Fixes#6825
fix(replication): never mint delete markers when replicating a version purge
Heal/resync/MRF rebuilds of a delete-marker version purge carry
delete_marker: true together with a purge-shaped entry. Passing that flag
straight into replication_delete_remove_options made the target DELETE
omit the versionId (marker-creation semantics), so a generic S3 target
that ignores the internal source-version headers minted a fresh delete
marker on every retry instead of purging one — the marker count on the
target grew monotonically (rustfs#6823).
- Gate marker-creation semantics on the new pure helper
delete_replication_creates_marker (delete_marker && !version purge) so
a purge always addresses the exact version.
- Stop falling through to the marker-creation send when the pre-send
source delete-marker verification fails with a transient error; fail
the entry instead so the MRF replay / heal scanner retries without
minting a marker on the target.
- Pin the purge-shape contract with unit tests in
crates/replication/src/delete.rs.
Also replace deprecated Atomic::fetch_update calls with try_update so the
current Rust toolchain keeps lint and CI jobs warning-clean.
Co-authored-by: heihutu <heihutu@gmail.com>
Validate lifecycle tier references through the tier reference proof path, preserve S3 list CommonPrefix XML compatibility, and make GetObject audit completion use real S3 error status codes.
Co-authored-by: heihutu <heihutu@gmail.com>
`crates/lifecycle/src/tagging.rs` carried a byte-identical copy of the `form_urlencoded` tag decoder already owned by `rustfs-replication`, plus a duplicate of its test. Since `crates/lifecycle` already depends on `rustfs-replication`, replace the copy with a `pub(crate) use` re-export: no new crate edge, one parser, and no second implementation to drift from the replication contract. The `rule.rs` call site is unchanged.
Also drop `crates/ecstore/src/bucket/lifecycle/tagging_boundary.rs`, a migration-era boundary shim with zero call sites in the tree.
* Revert "perf(ecstore): use AHashMap for FileInfo metadata fields (#6738)"
This reverts commit 13a2ae212e.
* fix(filemeta): restore standard HashMap metadata (#6742)
Remove the direct ahash dependency added for FileInfo metadata and revert the affected filemeta/ecstore call sites back to std::collections::HashMap.
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* refactor(ecstore): drop the client shim, import rustfs-s3-client directly
Completes the migration window opened by the rustfs-s3-client extraction (rustfs/backlog#1842 PR3): every consumer now imports the client crate directly and the crate::client shim is deleted.
- All in-crate crate::client:: paths (tier warm backends, tier core, lifecycle tier_sweeper, replication storage boundary, set_disk) now import rustfs_s3_client::* directly; crates/ecstore/src/client/mod.rs and the lib.rs mod client declaration are gone.
- The two server-side modules historically misfiled under client/ move to their real homes: object_api_utils.rs to crates/ecstore/src/object_api/ (it builds engine-side object readers/writers), and object_handlers_common.rs to crates/ecstore/src/bucket/lifecycle/ (it is the lifecycle noncurrent-version cleanup helper). The latter now routes its replication calls through the lifecycle replication_sink boundary (schedule_delete wrapper and the sink's ReplicationObjectBridge re-export), as the lifecycle guard requires.
- The ecstore public facade drops api::client: object_api_utils is exposed as api::object_api_utils, and the rustfs crate takes admin_handler_utils (AdminError) from rustfs-s3-client directly (new dependency).
- Guard updates: the migration guard no longer pins mod client in ecstore's lib.rs or the admin_handler_utils facade module (it pins the new api::object_api_utils facade instead), and the module-lint register follows object_api_utils.rs to its new path.
Verification: cargo check -p rustfs-ecstore --all-targets and -p rustfs; cargo fmt --all; tier/transition/lifecycle-focused nextest (626 passed) and the decommission/rebalance/heal families in a filtered run (603 passed; the full-suite parallel run only fails on this machine's known decommission/rebalance baseline flakes, which pass in filtered reruns and fail identically on pristine origin/main); layer/migration/s3s/logging/error-format/doc-path guard scripts all pass.
* docs(architecture): record the S3 client extraction and reword invariant 4 (#6669)
Closes the documentation step of rustfs/backlog#1842. ARCHITECTURE.md invariant 4 now states the serving-vs-consuming distinction the adversarial ruling asked for: ecstore must not serve HTTP/S3 wire types, while consuming remote S3 endpoints is a legitimate engine capability that lives in the extracted rustfs-s3-client crate. The violation note is updated from the pre-extraction snapshot (58 files, embedded client) to the current ratcheted state (shrink-only S3S_ECSTORE_FILES_BASELINE in scripts/check_s3s_footprint.sh, object_lock converted first), and the crate map gains s3-client. ecstore-module-split-plan.md gets the client-directory entry the plan was missing: a Current Shape row and a completed-extraction section describing the pure-move + shim + direct-import sequence and the re-homing of the two misfiled server-side modules.
The object_lock module evaluated WORM state through s3s wire DTOs (ObjectLockRetention, ObjectLockLegalHold, DefaultRetention, Date) and s3s header constants, keeping the storage engine coupled to the serving protocol (rustfs/backlog#1842, ARCHITECTURE.md invariant 4). This PR gives the module its own storage-level vocabulary and pushes the DTO conversions to the boundaries that already speak s3s.
New crates/ecstore/src/bucket/object_lock/types.rs defines RetentionMode, LegalHoldStatus, ObjectRetention, ObjectLegalHold, and DefaultRetention with no s3s dependency. objectlock.rs parses persisted metadata into these types using the rustfs-utils lowercase header constants (the same literal keys as before, pinned by the existing g-key-002 test). objectlock_sys.rs evaluates retention/legal-hold/default-retention from them; the fail-closed error messages and decision logic are unchanged line for line where possible.
Boundary conversions:
- bucket/metadata_sys.rs gains default_retention_from_object_lock_config, converting the persisted s3s configuration into the storage-level DefaultRetention; a rule without a usable GOVERNANCE/COMPLIANCE mode converts to None exactly like the evaluation code always ignored it, and days/years pass through so an invalid period still fails closed at evaluation time.
- check_object_lock_for_deletion_with_config becomes check_object_lock_for_deletion_with_default_retention (it only ever read the default retention); the lifecycle object_lock_boundary keeps the old s3s-typed signature and converts.
- The ObjectLockApi / ObjectLockStatusExt trait impls for the s3s DTOs move next to the persisted configuration owner in bucket/metadata.rs; the traits stay in object_lock/mod.rs.
- check_retention_for_modification now takes Option<RetentionMode>. The serving-layer wrappers (rustfs storage_api, set_disk options path) convert the request string with the new RetentionMode::parse_exact, which accepts only the canonical spelling — preserving the historical literal comparison where a non-canonical requested mode reads as a mode change and stays blocked.
- rustfs app-layer wrappers return the storage types; the replication-overwrite gate in object_usecase.rs uses the typed API (legal_hold.is_on(), RetentionMode::Compliance).
Ratchet: the ecstore-scoped s3s counter drops 42 -> 39 and the repo-wide file counter 211 -> 208 in scripts/check_s3s_footprint.sh.
Verification: cargo check -p rustfs-ecstore --all-targets and -p rustfs (lib+bins); cargo clippy -p rustfs-ecstore --all-targets and -p rustfs --lib --bins (clean); cargo nextest run -p rustfs-ecstore --no-fail-fast (4534/4542; the 8 failures are the same store::rebalance / store::heal machine-baseline set that fails identically on pristine origin/main, plus one fencing flake that passes in isolation); all object_lock/retention/legal-hold tests pass; guard scripts (layer deps, migration rules, s3s footprint, logging, error-format ratchet, doc paths) pass.
Backlog#1845 step 7. The replication crate's hand-rolled, crate-generic Error type actually describes one thing: failures of the persisted resync/MRF state files. Rename it to ResyncStateError so the name says so, and stop collapsing io::Error into Other(String): a new Io(std::io::Error) variant keeps the kind and source chain, Display renders identically, and the ecstore boundary maps it to StorageError::Io so the kind survives into store-layer classification instead of degrading into a stringified other().
No thiserror introduced - the crate keeps its zero-internal-deps posture and hand-written impls.
Ref rustfs/backlog#1845
PR #6369 awaits record_durable_ilm_decommission_progress/terminal inline
from save/delete_transition_transaction_record. Their state machines are
large and sit on the already-deep transition worker poll chain
(worker -> transition -> transaction record -> delete_config -> full
store delete fanout), which overflowed the default 2 MiB tokio worker
stack in debug builds: app::lifecycle_transition_api_test::
compensation_driven_complete_multipart_upload_still_transitions died
with SIGABRT in under a second (first-bad commit via git bisect
1.0.0-rc.3..1ec1a8d90: 34bbc1adb, #6369).
41546dee5 already unblocked the test by moving it onto a dedicated
32 MiB thread; this change removes the underlying stack growth so every
caller of the transaction-record helpers keeps its previous headroom.
With it, the test also passes on a plain 2 MiB tokio worker.
* fix(replication): tolerate orphaned resync intents at startup
Since #5215 (1.0.0-beta.12) startup reconciles every pending/started
resync intent in resync.bin against the bucket's configured targets and
aborts the whole server when an intent has no matching target ARN. A
resync whose remote target was later removed leaves exactly such an
orphan on disk, so every later start fails with "accepted replication
resync target ... is not configured" regardless of the binary version.
Skip orphaned intents with a warning instead of failing startup; the
resync routine already settles them to ResyncFailed. Cancel the intent
when its remote target is removed so the orphan is not created again.
Fixes#4784
* fix(replication): cancel removed-target resync under the admission lock
Canceling through this node's cached whole-bucket status map could
persist a map that predates another node's admission, erasing that
node's durable restart intent. Reload resync.bin under the bucket
admission lock, publish the fresh map, and only then mark the removed
target's intent canceled. Two-node regression covers the clobber.
* fix(replication): persist resync status via ETag CAS merge
mark_status, the periodic saver, admission, and removed-target
cancellation all persisted their node's cached whole-bucket map, so any
one node's stale cache could resurrect states another node had already
finalized (a canceled intent flipping back to Pending, an admission
vanishing). All resync.bin writers now go through update_resync_status_cas:
load the freshest document with its ETag, apply a per-target mutation
with staleness and canceled-is-terminal guards re-checked against the
persisted entry, and save conditionally, retrying on concurrent writes.
The periodic saver merges per target, letting terminal states and newer
admissions recorded elsewhere win. Cache convergence stays per-target so
locally running resyncs keep their authoritative progress counters.
Regressions: stale_peer_status_write_cannot_resurrect_canceled_intent
(node B's pre-cancel cache marking its own run Started must not revive
node A's canceled intent) plus unit coverage for the periodic-save merge.
* test(ecstore): rename resync test helper off the guarded contract name
fn resync_target is on the architecture guard's reserved list for
crates/replication operation contracts; the merge-test helper now reads
resync_target_state.
* fix(replication): serialize resync status updates
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
cxymds