mirror of
https://github.com/rustfs/rustfs.git
synced 2026-09-04 11:15:39 +00:00
fix(replication): probe the version-identity contract in replication-check (#5881)
* test(replication): pin the version-fidelity probe contract (red) P1-19 (rustfs/backlog#1675 B2): the supported replication contract is targets that adopt the source version id — a target that mints its own ids silently breaks every version-addressed operation that follows (version deletes, heal re-drives never match), diverging the two sides with no signal. replication-check already captures the probe PUT's response version id but never compares it. Red evidence (current main): against a FakeS3Target with assign_own_version_ids enabled, ?replication-check returns Status "OK" — the drift is invisible. test_replication_check_flags_version_minting_target expects a VersionFidelity phase that fails with the machine-readable code BucketRemoteTargetVersionMismatch, skips the later mutation phases, and still cleans up the probe via the version id the target actually assigned. Test infra: FakeS3Target gains assign_own_version_ids (models a generic S3 service; validated-but-not-mirrored source version headers) and a prefix+max-keys ListObjectVersions implementation (the probe key allocation requires it); stored_versions accessor duplicated from the P1-21 branch (identical code, resolves clean on merge). * fix(replication): probe the version-identity contract in replication-check P1-19 (rustfs/backlog#1675 B2, plan B). Replication only converges on targets that adopt the source version id: version-addressed deletes and heal re-drives address the source id, so a target that mints its own ids silently diverges — nothing surfaced this. replication-check already captured the probe PUT's response version id but never compared it. - The probe PUT now carries the source version as `?versionId=` (the exact shape live replication uses since P0-5, and the only shape MinIO consumes; the internal source-version-id header alone would let the probe pass against targets the real data path drifts on). Reuses ecstore's append_version_id_query through the api facade. - New VersionFidelity phase: the probe PUT's response version id must equal the sent source id. On mismatch the phase fails with the machine-readable extension key `"Code": "BucketRemoteTargetVersionMismatch"` (new optional Code field on phase statuses; Go decoders ignore unknown keys), the overall target fails, the later version-addressed mutation phases are skipped, and cleanup still removes the probe via the id the target actually assigned (with the existing list-based sweep as backstop when the target returns no version id at all). - Runtime half: TargetClient::put_object now returns the assigned version id (mirroring remove_object), and the replication PUT path audits it — every drifting PUT increments rustfs_replication_version_identity_drift_total and the first drift per target ARN logs a structured warning pointing at ?replication-check. The drift judgment is a pure function with an exemption-matrix test (empty / literal "null" / nil-uuid sources carry no contract). - docs/operations/replication-check.md documents the phase and the code. Red -> green: test_replication_check_flags_version_minting_target (fake target with assign_own_version_ids; on main the check reported Status "OK"). The probe's query shape is pinned by a journal assertion (revert of the query hunk alone fails it), probe-level unit tests cover the mismatch/mirror matrix including cleanup addressing the minted id, and the existing success e2e now asserts VersionFidelity OK against a RustFS target. Adversarial review (seven roles): non-blocking; noted follow-ups are the multipart runtime audit (the probe phase already pins the contract) and per-target re-warning after reconfiguration. * fix(e2e): stop the fake target self-deadlocking on version-id minting The assign_own_version_ids flag was read with a fresh `lock(&self.store)` inside two paths that already hold that guard — delete_object's marker-creation branch and create_multipart_upload — and the store mutex is not reentrant, so both hung forever (CI: the fake target's own multipart and delete-marker tests ran >1560s until the job was cancelled). Read the flag from the live guard instead. The replication e2e paths did not catch this: a version-addressed purge DELETE never mints an id, and the probe PUT reads the flag before taking the guard. * chore(test): refresh the nextest replication count invariant The e2e-smoke/e2e-repl-nightly split comment is descriptive metadata (authority: `cargo nextest list`); refresh it to this branch's post-rebase total.
This commit is contained in:
@@ -1450,7 +1450,7 @@ fn resolve_put_api_version_id(source_version_id: &str) -> Option<&str> {
|
||||
/// member, so the query is spliced in via `map_request`, which runs at
|
||||
/// `modify_before_signing`: the parameter becomes part of the SigV4 canonical
|
||||
/// request.
|
||||
fn append_version_id_query(uri: &str, version_id: &str) -> String {
|
||||
pub fn append_version_id_query(uri: &str, version_id: &str) -> String {
|
||||
let separator = if uri.contains('?') { '&' } else { '?' };
|
||||
format!("{uri}{separator}versionId={}", urlencoding::encode(version_id))
|
||||
}
|
||||
@@ -1861,6 +1861,9 @@ impl TargetClient {
|
||||
}
|
||||
}
|
||||
|
||||
/// On success returns the version id the target assigned (from
|
||||
/// `x-amz-version-id`), letting callers audit the version-identity
|
||||
/// contract — a target that adopts the source version echoes it back.
|
||||
pub async fn put_object(
|
||||
&self,
|
||||
bucket: &str,
|
||||
@@ -1868,7 +1871,7 @@ impl TargetClient {
|
||||
size: i64,
|
||||
body: ByteStream,
|
||||
opts: &PutObjectOptions,
|
||||
) -> Result<(), S3ClientError> {
|
||||
) -> Result<Option<String>, S3ClientError> {
|
||||
let mut headers = opts.header();
|
||||
|
||||
let builder = self.client.put_object();
|
||||
@@ -1903,7 +1906,7 @@ impl TargetClient {
|
||||
.send()
|
||||
.await
|
||||
{
|
||||
Ok(_) => Ok(()),
|
||||
Ok(output) => Ok(output.version_id().map(ToOwned::to_owned)),
|
||||
Err(e) => match e {
|
||||
SdkError::ServiceError(service_err) => {
|
||||
let err = service_err.into_err();
|
||||
|
||||
Reference in New Issue
Block a user