mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-31 17:28:12 +00:00
feat(replication): split oversized hot-path functions, proxy unreplicated reads, and fail SSE-C passthrough closed (#6170)
* refactor(replication): split four oversized hot-path functions into focused helpers Pure-move decomposition of the four oversized functions flagged by the replication compatibility review (P1-18), unblocking migration milestone M2 which requires resyncer moves to stay mechanical: - resync_bucket (522 lines -> 61-line step sequence): leader lock, target resolution, walk/collector/worker spawning, and dispatch loop extracted into focused helpers; pure decision helpers (DTO builders, HEAD-result classification) separated from IO orchestration. - replicate_all (411 lines -> 113-line main body): initial target-info seeding, read/stat option builders, skip-path notes, target HEAD action resolution, and the multipart/single-put payload transport extracted as private free functions. - start_mrf_processor (306 lines -> 46-line spawn body): recovery guard, ledger load, per-entry replay (delete/object/metadata), and retained entry resolution extracted; retry bookkeeping semantics preserved exactly (inner continue-paths push inside helpers, outer Missed push stays in the loop). - apply_iam_item (255 lines -> match dispatch skeleton): one helper per IAM item type. No behavior change: log texts, error paths, event emissions, and metric counts are byte-identical; existing tests unchanged and green (238 ecstore replication/mrf/resync + 232 rustfs site-replication). * feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets (#6172) * feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets Implements the MinIO active-active read-proxy protocol (P1-5 of the replication compatibility review): when a GET/HEAD/GetObjectTagging/ PutObjectTagging/DeleteObjectTagging request fails locally with not-found and the bucket has replication targets, the request is proxied to the targets in rule order, mirroring bucket-replication.go proxyGetToReplicationTarget/proxyHeadToRepTarget/proxyTaggingToRepTarget. Protocol surface: - Anti-loop: inbound {x-rustfs-,x-minio-}source-proxy-request is parsed into ObjectOptions (proxy_request + proxy_header_set, matching MinIO ProxyRequest/ProxyHeaderSet); a request carrying the marker with ANY value is never re-proxied. Outbound client proxy calls send the marker as "true"; replication worker convergence HEADs send it as "false" so a peer's proxy layer cannot answer a convergence check by proxying back to the source (which would fake Completed without a PUT). - Target selection: new replication_proxy.rs get_proxy_targets — empty when the marker is set, versioning is suspended, or no replication config; otherwise filter_target_arns -> TargetClient lookup, skipping targets with proxying disabled. - TargetClient gains head_object_for_proxy/get_object (streaming) and the three tagging calls. Proxy calls never send the replication-check SSE-C exemption header; customer SSE-C keys are forwarded verbatim so the target performs real decryption. Conditional (If-*) headers are not forwarded (MinIO parity); Range and part_number are, with parts_count/tag_count/storage_class/expiration passed through. - Metrics: proxy counters now count only real client proxy traffic, MinIO-aligned (one total per proxied request, one failed when no target served it). The previous misattributed counters — replication worker HEAD/PUT (#2672) and local tagging operations (#2682) — are removed; ReplProxyMetric now maps the tagging counters instead of dropping them. e2e (fake_s3_target extended with tagging + header journaling): proxied GET body + outbound header contract (marker present, no replication-check, SSE-C passthrough), HEAD, anti-loop 404 with zero outbound requests, GetObjectTagging, and metric mapping unit tests. Rolling note: proxying only activates for buckets with replication targets; requests carrying the marker keep pre-upgrade behavior. Refs rustfs/backlog#1675 (P1-5) * fix(replication): fail SSE-C passthrough closed on targets that drop transport headers (#6178) SSE-C ciphertext passthrough replicates via X-Rustfs-Replication-* transport headers. A MinIO/generic-S3 target silently discards them, storing bare ciphertext with no decryption material — yet the PUT succeeded, so the object reported COMPLETED with a silently unreadable replica (backlog#1675 N2). Fail-closed design: - SsecPassthroughCapability {Unknown, Supported, Unsupported} cached in BucketTargetSys per target ARN with a recording timestamp. Entries reset whenever the target is rebuilt, edited, or removed (arn_remotes_map lifecycle) and expire after SSEC_PASSTHROUGH_CAPABILITY_TTL (10 minutes): an expired verdict in either direction is re-earned through the audit, so an Unsupported target recovers automatically after an upgrade (at most one wasted PUT+HEAD audit per bad target per TTL window) and a Supported verdict cannot outlive a backend swapped behind the same endpoint. - Replication worker (replicate_object and replicate_all): fresh Unsupported targets never receive the PUT — the attempt fails immediately into the normal MRF retry channel with a "run ?replication-check to re-probe" hint. Unknown or expired verdicts are audited: after the PUT the worker HEADs the replica back through the replication-check channel (source version id mapped through resolve_read_api_version_id, so null-version objects audit correctly) and requires SSE-C evidence (the echoed customer-algorithm header); missing evidence records Unsupported and fails the attempt. Convergence HEADs are audited the same way, so a broken ciphertext replica from an earlier attempt can never launder itself into COMPLETED via an ETag match. The gate/evidence policy is pure (replication_target_boundary, staleness folded in as an input) for the M2 worker migration. - replication-check grows an SsecPassthrough probe phase: a probe PUT carrying the live transport-header shape, HEAD-back for evidence, and a machine-readable Code BucketRemoteSsecPassthroughUnsupported on failure. The probe verdict is synced into the runtime capability cache. Unlike VersionFidelity, a failed SsecPassthrough phase does NOT fail the target overall — it is a capability limit, not a broken replication contract, and a plaintext-only deployment against such a target must not turn red. - fake_s3_target: default mode now models a RustFS target (stores the transport headers, echoes SSE-C evidence); the new drop_unlisted_replication_headers mode models MinIO. The journal records whether a request carried transport headers. Receiver-echo verification: the replication-check HEAD exemption only skips SSE-C key validation; the response has always built sse-customer-algorithm from stored metadata (rustfs/src/app/object_usecase.rs), so no receiver change was needed — pinned end to end by the replication-check e2e against a real RustFS target. Rolling-upgrade constraint: RustFS targets older than the replication-check HEAD exemption (#5898) answer the audit HEAD without SSE-C evidence (or fail it outright), so SSE-C replication to such targets reports FAILED. This is deliberate — FAILED-and-retryable beats a silently undecryptable replica — and self-heals: once the target is upgraded, the next TTL expiry (or a manual ?replication-check re-probe) re-audits and records Supported. Plaintext and managed-SSE replication are unaffected. The capability cache is per-node; each node audits independently. Known limitations: - The audit judges evidence from the echoed customer-algorithm header only. A hypothetical target that preserves that one header while dropping other transport headers (partial-drop) would pass the audit; no known target behaves this way — observed targets drop the whole unknown-header family. - A mixed-version target cluster can flap the verdict between audits routed to different target nodes until the rollout completes; the TTL bounds how long each stale verdict persists. New e2e (backlog#1675 C1 + N2, red-first): fail-closed against a header-dropping fake (FAILED + no second PUT via the capability cache, journal-asserted; red run showed the old COMPLETED), replication-check reports the SsecPassthrough phase Code while the target stays OK overall, SSE-C heal convergence after a real target outage, and SSE-C existing-object resync landing a REPLICA readable with the customer key. TTL expiry in both directions is pinned at the cache and gate seams. * refactor(replication): move resyncer pure decision logic into rustfs-replication (M2) (#6180) * refactor(replication): move resyncer pure decision logic into rustfs-replication (M2) Pure-move milestone M2 of the ECStore replication split (backlog#1675 P1-17): relocate the resyncer's IO-free decision helpers, with their unit tests, into the crates they already belong to by type ownership. No behavior change. Moved into crates/replication: - resync.rs: resync_status_duration - delete.rs: resync_existing_delete_replication_info, replicate_delete_outcome, target_delete_version_id, delete_marker_purge_version_id, delete_marker_purge_mrf_entry - object.rs: version_identity_drifted, is_replication_target_offline_error, SsecPassthroughCapability, SsecPassthroughGate, ssec_passthrough_gate, ssec_passthrough_evidence_present (param-demoted to the echoed customer-algorithm string; ECStore keeps the HeadObjectOutput adapter) - filemeta.rs: NULL_VERSION_ID wire literal (crate-owned copy per the filemeta-independence contract) ECStore rewiring (Rule #14: imports stay in *_boundary.rs): - resync/object-decision/target boundaries re-export the moved symbols; resyncer call sites are unchanged - bucket_target_sys keeps only the verdict cache + TTL and re-exports the capability enum so existing consumer paths keep compiling Not moved (signatures carry ECStore or aws-sdk types): verify_resync_head_result, resync_target_error_detail, the SdkError classifiers, the replicate_all_* option/info builders, and the env-coupled bounded_resync_max_jobs admission clamp. README milestone table updated. * chore(replication): retire the datatypes.rs relay early README sanctions retiring datatypes.rs ahead of M4. The module was a pure relay (resync boundary -> datatypes -> mod.rs facade) with no external consumer importing it directly, so the facade now re-exports ResyncStatusType from replication_resync_boundary and the relay file is deleted. Consumers stay behind the ECStore facade, keeping Migration Rule #15 intact — the original retirement wording ("consumers import through rustfs-replication directly") conflicted with that rule and is corrected in the README. * chore(arch): extend migration guards to the M2-moved decision contracts The adversarial review of the M2 move found the per-symbol ratchet in check_architecture_migration_rules.sh was not extended for the moved symbols, leaving them free to be redefined in ECStore or imported past their boundary without CI noticing: - resync definition pin + boundary fences gain resync_status_duration; - the object-decision boundary fences gain the five delete-family helpers (delete_marker_purge_mrf_entry, delete_marker_purge_version_id, replicate_delete_outcome, resync_existing_delete_replication_info, target_delete_version_id); - the target-boundary fence gains the SSE-C gate family, the offline classifier, and version_identity_drifted; - a new definition pin rejects ECStore redefinitions of the M2-moved fns/enums (ssec_passthrough_evidence_present deliberately excluded: ECStore keeps a thin HeadObjectOutput adapter under that name). Mutation-verified: a probe fn ssec_passthrough_gate under crates/ecstore/src/bucket/replication trips the new pin. Also anchors the intentionally-duplicated NULL_VERSION_ID wire literal from the filemeta side and tightens the M2 README note on bounded_resync_max_jobs.
This commit is contained in:
唐小鸭
@@ -32,7 +32,7 @@ pub mod bucket {
|
||||
pub mod bucket_target_sys {
|
||||
pub use crate::bucket::bucket_target_sys::{
|
||||
AdvancedPutOptions, BucketTargetError, BucketTargetSys, PutObjectOptions, RemoveObjectOptions, S3ClientError,
|
||||
TargetClient, append_version_id_query,
|
||||
SsecPassthroughCapability, TargetClient, append_version_id_query,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -198,12 +198,13 @@ pub mod bucket {
|
||||
ReplicationType, ResyncOpts, ResyncStatusType, RuntimeReplicationTargetBacklog, TargetReplicationResyncStatus,
|
||||
VersionPurgeStatusType, XferStats, commit_force_delete_intent, complete_force_delete_intent,
|
||||
delete_replication_state_from_config, delete_replication_version_id, get_global_replication_pool,
|
||||
get_global_replication_stats, init_background_replication, invalid_replication_config_status_field,
|
||||
persist_force_delete_intent, read_durable_mrf_backlog, replication_state_to_filemeta, replication_status_to_filemeta,
|
||||
replication_statuses_map, replication_target_arns, resync_start_conflict_id, should_remove_replication_target,
|
||||
should_schedule_delete_replication, should_use_existing_delete_replication_info,
|
||||
should_use_existing_delete_replication_source, unsupported_replication_config_field,
|
||||
validate_replication_config_structure, validate_replication_config_target_arns, version_purge_status_to_filemeta,
|
||||
get_global_replication_stats, get_proxy_targets, init_background_replication,
|
||||
invalid_replication_config_status_field, persist_force_delete_intent, read_durable_mrf_backlog,
|
||||
replication_state_to_filemeta, replication_status_to_filemeta, replication_statuses_map, replication_target_arns,
|
||||
resync_start_conflict_id, should_remove_replication_target, should_schedule_delete_replication,
|
||||
should_use_existing_delete_replication_info, should_use_existing_delete_replication_source,
|
||||
unsupported_replication_config_field, validate_replication_config_structure, validate_replication_config_target_arns,
|
||||
version_purge_status_to_filemeta,
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -27,10 +27,15 @@ use aws_sdk_s3::config::SharedHttpClient;
|
||||
use aws_sdk_s3::error::ProvideErrorMetadata;
|
||||
use aws_sdk_s3::error::SdkError;
|
||||
use aws_sdk_s3::operation::complete_multipart_upload::CompleteMultipartUploadOutput;
|
||||
use aws_sdk_s3::operation::delete_object_tagging::{DeleteObjectTaggingError, DeleteObjectTaggingOutput};
|
||||
use aws_sdk_s3::operation::get_object::{GetObjectError, GetObjectOutput};
|
||||
use aws_sdk_s3::operation::get_object_tagging::{GetObjectTaggingError, GetObjectTaggingOutput};
|
||||
use aws_sdk_s3::operation::head_bucket::HeadBucketError;
|
||||
use aws_sdk_s3::operation::head_object::HeadObjectError;
|
||||
use aws_sdk_s3::operation::put_object_tagging::{PutObjectTaggingError, PutObjectTaggingOutput};
|
||||
use aws_sdk_s3::operation::upload_part::UploadPartOutput;
|
||||
use aws_sdk_s3::primitives::ByteStream;
|
||||
use aws_sdk_s3::types::Tagging as SdkTagging;
|
||||
use aws_sdk_s3::types::{
|
||||
ChecksumMode, CompletedMultipartUpload, CompletedPart, ObjectLockLegalHoldStatus, ObjectLockRetentionMode,
|
||||
};
|
||||
@@ -57,8 +62,8 @@ use rustfs_utils::http::{
|
||||
is_rustfs_header, is_standard_header, is_storageclass_header,
|
||||
};
|
||||
use rustfs_utils::http::{
|
||||
SUFFIX_FORCE_DELETE, SUFFIX_SOURCE_DELETEMARKER, SUFFIX_SOURCE_ETAG, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_CHECK,
|
||||
SUFFIX_SOURCE_REPLICATION_LEGALHOLD_TIMESTAMP, SUFFIX_SOURCE_REPLICATION_REQUEST,
|
||||
SUFFIX_FORCE_DELETE, SUFFIX_SOURCE_DELETEMARKER, SUFFIX_SOURCE_ETAG, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_PROXY_REQUEST,
|
||||
SUFFIX_SOURCE_REPLICATION_CHECK, SUFFIX_SOURCE_REPLICATION_LEGALHOLD_TIMESTAMP, SUFFIX_SOURCE_REPLICATION_REQUEST,
|
||||
SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP, SUFFIX_SOURCE_REPLICATION_TAGGING_TIMESTAMP, SUFFIX_SOURCE_VERSION_ID,
|
||||
insert_header,
|
||||
};
|
||||
@@ -294,9 +299,41 @@ struct TargetClientBuildProbe {
|
||||
release: Arc<tokio::sync::Semaphore>,
|
||||
}
|
||||
|
||||
/// SSE-C passthrough capability verdicts (see the enum's own docs in
|
||||
/// `rustfs-replication`) are cached here per target ARN: entries follow the
|
||||
/// `arn_remotes_map` lifecycle (rebuilding or removing a target resets its
|
||||
/// capability to `Unknown`) and additionally expire after
|
||||
/// [`SSEC_PASSTHROUGH_CAPABILITY_TTL`], after which the next attempt
|
||||
/// re-audits. Re-exported so existing `bucket_target_sys` consumers keep
|
||||
/// their import path while the verdict vocabulary lives with the
|
||||
/// replication decision logic.
|
||||
pub use crate::bucket::replication::SsecPassthroughCapability;
|
||||
|
||||
/// How long an audited SSE-C passthrough verdict stays authoritative.
|
||||
///
|
||||
/// Trade-off: without a TTL a verdict is sticky for the process lifetime —
|
||||
/// an `Unsupported` target that gets upgraded (or re-probed only via
|
||||
/// replication-check) would keep failing SSE-C replication forever, and the
|
||||
/// fail-open twin: a `Supported` verdict would outlive a backend swapped
|
||||
/// behind the same endpoint/ARN. With the TTL, a bad target costs at most
|
||||
/// one wasted PUT+HEAD audit per TTL window, and a changed backend is
|
||||
/// re-discovered within the same window.
|
||||
pub const SSEC_PASSTHROUGH_CAPABILITY_TTL: Duration = Duration::from_secs(10 * 60);
|
||||
|
||||
/// A recorded SSE-C passthrough verdict plus when it was recorded, so reads
|
||||
/// can report staleness against [`SSEC_PASSTHROUGH_CAPABILITY_TTL`].
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
struct SsecPassthroughRecord {
|
||||
capability: SsecPassthroughCapability,
|
||||
recorded_at: Instant,
|
||||
}
|
||||
|
||||
#[derive(Debug, Default)]
|
||||
pub struct BucketTargetSys {
|
||||
pub arn_remotes_map: Arc<RwLock<HashMap<String, ArnTarget>>>,
|
||||
/// SSE-C passthrough capability verdicts keyed by target ARN. See
|
||||
/// [`SsecPassthroughCapability`]; reset alongside `arn_remotes_map`.
|
||||
ssec_passthrough_map: Arc<RwLock<HashMap<String, SsecPassthroughRecord>>>,
|
||||
pub targets_map: Arc<RwLock<HashMap<String, Vec<BucketTarget>>>>,
|
||||
pub h_mutex: Arc<RwLock<HashMap<String, EpHealth>>>,
|
||||
target_h_mutex: Arc<RwLock<HashMap<String, EpHealth>>>,
|
||||
@@ -317,6 +354,7 @@ impl BucketTargetSys {
|
||||
fn new() -> Self {
|
||||
Self {
|
||||
arn_remotes_map: Arc::new(RwLock::new(HashMap::new())),
|
||||
ssec_passthrough_map: Arc::new(RwLock::new(HashMap::new())),
|
||||
targets_map: Arc::new(RwLock::new(HashMap::new())),
|
||||
h_mutex: Arc::new(RwLock::new(HashMap::new())),
|
||||
target_h_mutex: Arc::new(RwLock::new(HashMap::new())),
|
||||
@@ -580,19 +618,59 @@ impl BucketTargetSys {
|
||||
let update_mutex = self.target_update_mutex(bucket).await;
|
||||
let _update_guard = update_mutex.lock().await;
|
||||
|
||||
// Lock order: targets_map, then arn_remotes_map, then target_h_mutex.
|
||||
// Lock order: targets_map, then arn_remotes_map, then target_h_mutex,
|
||||
// then ssec_passthrough_map (always last; also taken standalone by the
|
||||
// capability accessors).
|
||||
let mut targets_map = self.targets_map.write().await;
|
||||
let mut arn_remotes_map = self.arn_remotes_map.write().await;
|
||||
let mut health_map = self.target_h_mutex.write().await;
|
||||
|
||||
if let Some(targets) = targets_map.remove(bucket) {
|
||||
let mut ssec_map = self.ssec_passthrough_map.write().await;
|
||||
for target in targets {
|
||||
arn_remotes_map.remove(&target.arn);
|
||||
health_map.remove(&target.arn);
|
||||
ssec_map.remove(&target.arn);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Cached SSE-C passthrough capability for a target ARN, plus whether the
|
||||
/// verdict is older than [`SSEC_PASSTHROUGH_CAPABILITY_TTL`]. `(Unknown,
|
||||
/// false)` when no verdict has been recorded since the target was built.
|
||||
/// Staleness is computed here so the gate policy stays a pure function.
|
||||
pub async fn ssec_passthrough_capability(&self, arn: &str) -> (SsecPassthroughCapability, bool) {
|
||||
match self.ssec_passthrough_map.read().await.get(arn) {
|
||||
Some(record) => (record.capability, record.recorded_at.elapsed() >= SSEC_PASSTHROUGH_CAPABILITY_TTL),
|
||||
None => (SsecPassthroughCapability::Unknown, false),
|
||||
}
|
||||
}
|
||||
|
||||
/// Record an audited SSE-C passthrough verdict for a target ARN. Written by
|
||||
/// the replication worker's HEAD-back audit and by the replication-check
|
||||
/// SsecPassthrough probe phase.
|
||||
pub async fn record_ssec_passthrough_capability(&self, arn: &str, capability: SsecPassthroughCapability) {
|
||||
self.ssec_passthrough_map.write().await.insert(
|
||||
arn.to_string(),
|
||||
SsecPassthroughRecord {
|
||||
capability,
|
||||
recorded_at: Instant::now(),
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
/// Test hook: age an existing verdict so TTL expiry is observable without
|
||||
/// waiting out the real window.
|
||||
#[cfg(test)]
|
||||
pub(crate) async fn backdate_ssec_passthrough_capability(&self, arn: &str, age: Duration) {
|
||||
let backdated = Instant::now()
|
||||
.checked_sub(age)
|
||||
.expect("system uptime must exceed the backdate age");
|
||||
if let Some(record) = self.ssec_passthrough_map.write().await.get_mut(arn) {
|
||||
record.recorded_at = backdated;
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn set_target(
|
||||
&self,
|
||||
bucket: &str,
|
||||
@@ -948,15 +1026,21 @@ impl BucketTargetSys {
|
||||
}
|
||||
}
|
||||
|
||||
// Lock order: targets_map, then arn_remotes_map, then target_h_mutex.
|
||||
// Lock order: targets_map, then arn_remotes_map, then target_h_mutex,
|
||||
// then ssec_passthrough_map (always last; also taken standalone by the
|
||||
// capability accessors).
|
||||
let mut targets_map = self.targets_map.write().await;
|
||||
let mut arn_remotes_map = self.arn_remotes_map.write().await;
|
||||
let mut health_map = self.target_h_mutex.write().await;
|
||||
// Remove existing targets
|
||||
if let Some(existing_targets) = targets_map.remove(bucket) {
|
||||
let mut ssec_map = self.ssec_passthrough_map.write().await;
|
||||
for target in existing_targets {
|
||||
arn_remotes_map.remove(&target.arn);
|
||||
health_map.remove(&target.arn);
|
||||
// A rebuilt/edited target may point at a different service:
|
||||
// the SSE-C passthrough verdict must be re-audited from Unknown.
|
||||
ssec_map.remove(&target.arn);
|
||||
self.update_bandwidth_limit(bucket, &target.arn, 0);
|
||||
}
|
||||
}
|
||||
@@ -1446,6 +1530,43 @@ fn resolve_put_api_version_id(source_version_id: &str) -> Option<&str> {
|
||||
}
|
||||
}
|
||||
|
||||
/// Resolve the S3 `versionId` for a proxied read against a remote target.
|
||||
/// RustFS represents the null version internally as the nil UUID while the S3
|
||||
/// API addresses it as the literal "null" (same mapping as
|
||||
/// [`resolve_put_api_version_id`]); empty means "no version requested".
|
||||
pub(crate) fn resolve_read_api_version_id(version_id: Option<String>) -> Option<String> {
|
||||
let version_id = version_id?;
|
||||
let trimmed = version_id.trim();
|
||||
if trimmed.is_empty() {
|
||||
None
|
||||
} else if Uuid::parse_str(trimmed).is_ok_and(|uuid| uuid.is_nil()) {
|
||||
Some(rustfs_filemeta::NULL_VERSION_ID.to_string())
|
||||
} else {
|
||||
Some(trimmed.to_string())
|
||||
}
|
||||
}
|
||||
|
||||
/// Outbound header set for a proxied read: the caller-provided passthrough
|
||||
/// headers (client SSE-C key family, conditional headers) plus the anti-loop
|
||||
/// `source-proxy-request` marker in both the x-rustfs- and x-minio- prefixes
|
||||
/// (a MinIO target only understands the latter). Never adds
|
||||
/// `source-replication-check`: that exemption channel belongs exclusively to
|
||||
/// the replication worker's HEAD.
|
||||
fn proxy_outbound_headers(mut extra_headers: HeaderMap) -> HeaderMap {
|
||||
insert_header(&mut extra_headers, SUFFIX_SOURCE_PROXY_REQUEST, "true");
|
||||
extra_headers
|
||||
}
|
||||
|
||||
/// Copy `headers` onto an SDK request inside `customize().map_request` (runs
|
||||
/// before signing, so the headers join the SigV4 canonical request).
|
||||
fn apply_extra_headers(mut req: HttpRequest, headers: &HeaderMap) -> Result<HttpRequest, std::convert::Infallible> {
|
||||
for (k, v) in headers.iter() {
|
||||
req.headers_mut()
|
||||
.insert(k.as_str().to_string(), v.to_str().unwrap_or("").to_string());
|
||||
}
|
||||
Ok(req)
|
||||
}
|
||||
|
||||
/// Append `versionId=<id>` to an already-built request URI. aws-sdk-s3's
|
||||
/// `PutObjectInput` / `CreateMultipartUploadInput` expose no version id
|
||||
/// member, so the query is spliced in via `map_request`, which runs at
|
||||
@@ -1853,6 +1974,13 @@ impl TargetClient {
|
||||
// worker cannot hold; otherwise SSE-C replicas never converge on HEAD.
|
||||
let mut headers = HeaderMap::new();
|
||||
insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_CHECK, "true");
|
||||
// `source-proxy-request: false` (MinIO `ProxyHeaderSet` semantics):
|
||||
// the header's mere presence tells the receiver to answer LOCALLY
|
||||
// instead of proxying the miss back to us. Without it, a not-found on
|
||||
// the target gets read-proxied back to this source, echoes the source
|
||||
// object with an identical ETag, and the worker concludes the object
|
||||
// already converged — so it never actually replicates it.
|
||||
insert_header(&mut headers, SUFFIX_SOURCE_PROXY_REQUEST, "false");
|
||||
match self
|
||||
.client
|
||||
.head_object()
|
||||
@@ -1877,6 +2005,129 @@ impl TargetClient {
|
||||
}
|
||||
}
|
||||
|
||||
/// HEAD used by the read-proxy path (GET/HEAD of an object not yet
|
||||
/// replicated locally, MinIO `proxyHeadToRepTarget`).
|
||||
///
|
||||
/// Deliberately different from [`TargetClient::head_object`]: it must NOT
|
||||
/// send `source-replication-check` — that header is the replication
|
||||
/// worker's SSE-C metadata exemption channel. A proxied client request
|
||||
/// instead forwards the client's own SSE-C headers (`extra_headers`) so
|
||||
/// the target performs the real SSE-C validation/decryption. The
|
||||
/// `source-proxy-request` marker is always added so the target does not
|
||||
/// proxy the request onward (anti-loop).
|
||||
pub async fn head_object_for_proxy(
|
||||
&self,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
version_id: Option<String>,
|
||||
range: Option<String>,
|
||||
part_number: Option<i32>,
|
||||
extra_headers: HeaderMap,
|
||||
) -> Result<HeadObjectOutput, SdkError<HeadObjectError>> {
|
||||
let headers = proxy_outbound_headers(extra_headers);
|
||||
self.client
|
||||
.head_object()
|
||||
.bucket(bucket)
|
||||
.key(object)
|
||||
.set_version_id(resolve_read_api_version_id(version_id))
|
||||
.set_range(range)
|
||||
.set_part_number(part_number)
|
||||
.customize()
|
||||
.map_request(move |req| apply_extra_headers(req, &headers))
|
||||
.send()
|
||||
.await
|
||||
}
|
||||
|
||||
/// GET used by the read-proxy path (MinIO `proxyGetToReplicationTarget`).
|
||||
/// Returns the streaming SDK output; callers must forward the body without
|
||||
/// buffering it. Same header contract as [`Self::head_object_for_proxy`]:
|
||||
/// anti-loop marker on, replication-check never sent, client SSE-C /
|
||||
/// conditional headers forwarded verbatim via `extra_headers`.
|
||||
pub async fn get_object(
|
||||
&self,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
version_id: Option<String>,
|
||||
range: Option<String>,
|
||||
part_number: Option<i32>,
|
||||
extra_headers: HeaderMap,
|
||||
) -> Result<GetObjectOutput, SdkError<GetObjectError>> {
|
||||
let headers = proxy_outbound_headers(extra_headers);
|
||||
self.client
|
||||
.get_object()
|
||||
.bucket(bucket)
|
||||
.key(object)
|
||||
.set_version_id(resolve_read_api_version_id(version_id))
|
||||
.set_range(range)
|
||||
.set_part_number(part_number)
|
||||
.customize()
|
||||
.map_request(move |req| apply_extra_headers(req, &headers))
|
||||
.send()
|
||||
.await
|
||||
}
|
||||
|
||||
/// GetObjectTagging for the tagging read-proxy path
|
||||
/// (MinIO `proxyGetTaggingToRepTarget`). Anti-loop marker always added.
|
||||
pub async fn get_object_tagging(
|
||||
&self,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
version_id: Option<String>,
|
||||
) -> Result<GetObjectTaggingOutput, SdkError<GetObjectTaggingError>> {
|
||||
let headers = proxy_outbound_headers(HeaderMap::new());
|
||||
self.client
|
||||
.get_object_tagging()
|
||||
.bucket(bucket)
|
||||
.key(object)
|
||||
.set_version_id(resolve_read_api_version_id(version_id))
|
||||
.customize()
|
||||
.map_request(move |req| apply_extra_headers(req, &headers))
|
||||
.send()
|
||||
.await
|
||||
}
|
||||
|
||||
/// PutObjectTagging for the tagging proxy path
|
||||
/// (MinIO `proxyTaggingToRepTarget`). Anti-loop marker always added.
|
||||
pub async fn put_object_tagging(
|
||||
&self,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
version_id: Option<String>,
|
||||
tagging: SdkTagging,
|
||||
) -> Result<PutObjectTaggingOutput, SdkError<PutObjectTaggingError>> {
|
||||
let headers = proxy_outbound_headers(HeaderMap::new());
|
||||
self.client
|
||||
.put_object_tagging()
|
||||
.bucket(bucket)
|
||||
.key(object)
|
||||
.set_version_id(resolve_read_api_version_id(version_id))
|
||||
.tagging(tagging)
|
||||
.customize()
|
||||
.map_request(move |req| apply_extra_headers(req, &headers))
|
||||
.send()
|
||||
.await
|
||||
}
|
||||
|
||||
/// DeleteObjectTagging for the tagging proxy path
|
||||
/// (MinIO `proxyTaggingToRepTarget`). Anti-loop marker always added.
|
||||
pub async fn delete_object_tagging(
|
||||
&self,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
version_id: Option<String>,
|
||||
) -> Result<DeleteObjectTaggingOutput, SdkError<DeleteObjectTaggingError>> {
|
||||
let headers = proxy_outbound_headers(HeaderMap::new());
|
||||
self.client
|
||||
.delete_object_tagging()
|
||||
.bucket(bucket)
|
||||
.key(object)
|
||||
.set_version_id(resolve_read_api_version_id(version_id))
|
||||
.customize()
|
||||
.map_request(move |req| apply_extra_headers(req, &headers))
|
||||
.send()
|
||||
.await
|
||||
}
|
||||
|
||||
/// On success returns the version id the target assigned (from
|
||||
/// `x-amz-version-id`), letting callers audit the version-identity
|
||||
/// contract — a target that adopts the source version echoes it back.
|
||||
@@ -2506,6 +2757,57 @@ mod tests {
|
||||
assert_eq!(health.last_online, Some(now));
|
||||
}
|
||||
|
||||
/// N2 TTL contract, both flip directions: a recorded verdict is fresh
|
||||
/// until [`SSEC_PASSTHROUGH_CAPABILITY_TTL`], then reads as expired; a
|
||||
/// re-audit that records the OPPOSITE verdict replaces it as fresh. The
|
||||
/// worker gate maps expired verdicts to ProceedWithAudit (pinned in
|
||||
/// `replication_target_boundary`), so together this proves an Unsupported
|
||||
/// target recovers to Supported through the audit once its verdict ages
|
||||
/// out — and a stale Supported one is re-proven rather than trusted.
|
||||
#[tokio::test]
|
||||
async fn ssec_passthrough_capability_ttl_expires_and_reaudit_flips_verdict() {
|
||||
let sys = BucketTargetSys::default();
|
||||
let arn = "arn:rustfs:replication:us-east-1:bucket:ssec-ttl";
|
||||
let expired_age = SSEC_PASSTHROUGH_CAPABILITY_TTL + Duration::from_secs(1);
|
||||
|
||||
assert_eq!(
|
||||
sys.ssec_passthrough_capability(arn).await,
|
||||
(SsecPassthroughCapability::Unknown, false),
|
||||
"an unrecorded target must read Unknown and never expired"
|
||||
);
|
||||
|
||||
sys.record_ssec_passthrough_capability(arn, SsecPassthroughCapability::Unsupported)
|
||||
.await;
|
||||
assert_eq!(
|
||||
sys.ssec_passthrough_capability(arn).await,
|
||||
(SsecPassthroughCapability::Unsupported, false)
|
||||
);
|
||||
|
||||
sys.backdate_ssec_passthrough_capability(arn, expired_age).await;
|
||||
assert_eq!(
|
||||
sys.ssec_passthrough_capability(arn).await,
|
||||
(SsecPassthroughCapability::Unsupported, true),
|
||||
"an aged-out Unsupported verdict must read expired so the gate re-audits"
|
||||
);
|
||||
|
||||
// The re-audit against an upgraded target records Supported afresh.
|
||||
sys.record_ssec_passthrough_capability(arn, SsecPassthroughCapability::Supported)
|
||||
.await;
|
||||
assert_eq!(
|
||||
sys.ssec_passthrough_capability(arn).await,
|
||||
(SsecPassthroughCapability::Supported, false),
|
||||
"a fresh Supported verdict replaces the expired Unsupported one"
|
||||
);
|
||||
|
||||
// And the fail-open twin: Supported also ages out.
|
||||
sys.backdate_ssec_passthrough_capability(arn, expired_age).await;
|
||||
assert_eq!(
|
||||
sys.ssec_passthrough_capability(arn).await,
|
||||
(SsecPassthroughCapability::Supported, true),
|
||||
"an aged-out Supported verdict must read expired so the gate re-proves it"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn list_targets_applies_health_stats_by_arn_and_preserves_endpoint_port() {
|
||||
let sys = BucketTargetSys::default();
|
||||
|
||||
@@ -11,9 +11,9 @@ paths.
|
||||
| Module | Current role | Split blocker |
|
||||
|---|---|---|
|
||||
| `config.rs` | Replication config helpers, rule matching, and tag filtering. | Uses replication-local filemeta/tagging boundaries and S3 DTOs directly. |
|
||||
| `datatypes.rs` | ECStore compatibility re-export for resync status enums. | Re-exports `rustfs-replication` contracts while downstream facade consumers migrate. |
|
||||
| `replication_object_decision_boundary.rs` | Object replication option DTOs, resync target projection, delete replication decisions, and multipart planning helpers. | Keeps ECStore runtime modules from importing object decision contracts directly from `rustfs-replication`. |
|
||||
| `replication_pool.rs` | Replication queue, worker pool, MRF persistence, bucket stats, and delete/object scheduling. | Depends on bucket target sys, bucket metadata sys, metadata paths, queue contracts through the queue boundary, file metadata replication contracts through local boundaries, config storage, storage contracts through the replication storage boundary, runtime sources, and notification state. |
|
||||
| `replication_proxy.rs` | Proxy-target selection for GET/HEAD/Tagging reads of objects not yet replicated locally (MinIO `getProxyTargets` parity: anti-loop, version-suspended, and no-config empty branches). | Uses replication config lookup, rule matching, and target clients through local boundaries. |
|
||||
| `replication_queue_boundary.rs` | Queue/admission DTOs, heal queue DTOs, worker sizing, and backpressure helpers. | Keeps ECStore runtime modules from importing queue/backpressure contracts directly from `rustfs-replication`. |
|
||||
| `replication_resync_boundary.rs` | Resync DTOs, status classifiers, persisted resync/MRF codec wrappers, and ECStore error mapping. | Keeps ECStore runtime modules from importing resync contract helpers directly from `rustfs-replication`. |
|
||||
| `replication_resyncer.rs` | Object replication, delete replication, resync execution, target calls, and multipart target upload paths. | Depends on target calls and target config types through the replication target boundary, metadata paths and metadata systems through the replication metadata boundary, file metadata replication contracts through the filemeta boundary, object decisions and multipart planning through the object decision boundary, resync contracts through the resync boundary, queue DTOs through the queue boundary, error contracts through the error boundary, versioning systems, storage contracts through the replication storage boundary, config-derived storage class labels through the config store, runtime sources, notification events and local event host selection through the event sink, bandwidth reader wrapping, and SetDisks lock timing. |
|
||||
@@ -117,9 +117,12 @@ Target end state:
|
||||
their file names — so batch-merging them beforehand is explicitly rejected:
|
||||
it forces synchronized guard-script/mod/import churn with zero functional
|
||||
gain;
|
||||
- the only module that can retire early is `datatypes.rs`: delete it once its
|
||||
facade consumers import the resync status enums through `rustfs-replication`
|
||||
directly.
|
||||
- `datatypes.rs` retired early (its sanctioned exception): it was a pure
|
||||
relay (`boundary -> datatypes -> mod.rs`), so the facade now re-exports
|
||||
`ResyncStatusType` from the resync boundary directly and the relay file is
|
||||
deleted. Note the original retirement wording ("consumers import through
|
||||
`rustfs-replication` directly") conflicted with Migration Rule #15 —
|
||||
consumers stay behind the ECStore facade; only the relay hop dissolves.
|
||||
|
||||
## Milestones
|
||||
|
||||
@@ -127,9 +130,9 @@ Target end state:
|
||||
|---|---|---|
|
||||
| M0 | Record the completion criteria and end state (this section). | Done |
|
||||
| M1 | Contract extraction: resync/queue/stats/object-decision/filemeta/storage wire contracts owned by `crates/replication`; ECStore imports concentrated in `*_boundary.rs`; event sink and runtime access behind local contracts. | Done — see Required Contracts |
|
||||
| M2 | Move resyncer pure decision logic (no IO) into `crates/replication`. | Pending; sequence after splitting the oversized resyncer/pool functions (`resync_bucket`, `replicate_all`, `start_mrf_processor`) so moves stay mechanical |
|
||||
| M2 | Move resyncer pure decision logic (no IO) into `crates/replication`. | Done — moved the pure decision helpers with their unit tests: `resync_status_duration` (resync), `resync_existing_delete_replication_info` / `replicate_delete_outcome` / `target_delete_version_id` / `delete_marker_purge_version_id` / `delete_marker_purge_mrf_entry` (delete), `version_identity_drifted` / `is_replication_target_offline_error` / the SSE-C passthrough gate family incl. `SsecPassthroughCapability` (object; `ssec_passthrough_evidence_present` was param-demoted to the echoed customer-algorithm string, ECStore keeps the `HeadObjectOutput` adapter). ECStore imports them through the resync/object-decision/target boundaries; `bucket_target_sys` keeps only the verdict cache + TTL and re-exports the capability enum. Not moved (signatures carry ECStore or aws-sdk types): `verify_resync_head_result`, `resync_target_error_detail`, the `SdkError` classifiers (`has_raw_status`, `is_version_id_format_mismatch`), the `replicate_all_*` option/info builders, and `bounded_resync_max_jobs` (itself a pure clamp, but it forms one local configuration unit with the env-reading `configured_resync_max_jobs` and its ECStore-local constants — moving the clamp alone has negative value). |
|
||||
| M3 | Move the worker runtime (`replication_pool.rs`, the IO paths of `replication_resyncer.rs`, `replication_state.rs`) once the contract traits are stable. Highest-risk step of the whole plan; do it last. | Pending |
|
||||
| M4 | Retire the boundary modules together with their guard-script entries; delete `datatypes.rs`. | Pending |
|
||||
| M4 | Retire the boundary modules together with their guard-script entries. | Pending (`datatypes.rs` already retired early alongside M2) |
|
||||
|
||||
The original first code-bearing step (narrow `ReplicationEventSink` /
|
||||
`ReplicationRuntime` contracts) has landed — `replication_event_sink.rs`
|
||||
|
||||
@@ -1,15 +0,0 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
pub use super::replication_resync_boundary::ResyncStatusType;
|
||||
@@ -12,7 +12,6 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
pub mod datatypes;
|
||||
mod replication_bandwidth_boundary;
|
||||
mod replication_config_boundary;
|
||||
mod replication_config_store;
|
||||
@@ -29,6 +28,7 @@ mod replication_object_bridge;
|
||||
mod replication_object_config;
|
||||
mod replication_object_decision_boundary;
|
||||
pub(crate) mod replication_pool;
|
||||
mod replication_proxy;
|
||||
mod replication_queue_boundary;
|
||||
mod replication_resync_boundary;
|
||||
mod replication_resyncer;
|
||||
@@ -43,7 +43,6 @@ pub(crate) mod replication_timing;
|
||||
mod replication_versioning_boundary;
|
||||
mod runtime_boundary;
|
||||
|
||||
pub use datatypes::ResyncStatusType;
|
||||
pub use replication_config_boundary::{
|
||||
ObjectOpts, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION, REMOTE_TARGET_UNSUPPORTED_FIELDS, REMOTE_TARGET_WRITABLE_FIELDS,
|
||||
REPLICATION_CAPABILITY_CONTRACT_VERSION, REPLICATION_READ_ONLY_HISTORICAL_FIELDS, REPLICATION_WRITABLE_FIELDS,
|
||||
@@ -74,13 +73,16 @@ pub use replication_pool::{
|
||||
get_global_replication_pool, get_global_replication_stats, init_background_replication, persist_force_delete_intent,
|
||||
read_durable_mrf_backlog, resync_start_conflict_id,
|
||||
};
|
||||
pub use replication_proxy::get_proxy_targets;
|
||||
pub use replication_queue_boundary::{
|
||||
DeletedObjectReplicationInfo, ReplicationBatchAdmission, ReplicationHealQueueResult, ReplicationOperation,
|
||||
ReplicationPriority, ReplicationQueueAdmission,
|
||||
};
|
||||
pub use replication_resync_boundary::ResyncStatusType;
|
||||
pub use replication_resync_boundary::{BucketReplicationResyncStatus, ResyncOpts, TargetReplicationResyncStatus};
|
||||
pub use replication_scanner_bridge::ReplicationScannerBridge;
|
||||
pub use replication_state::{ReplicationStats, RuntimeReplicationTargetBacklog};
|
||||
pub use replication_stats_boundary::{BucketReplicationStat, BucketReplicationStats, BucketStats, InQueueMetric, XferStats};
|
||||
pub use replication_storage_boundary::{ReplicationObjectIO, ReplicationStorage};
|
||||
pub use replication_target_boundary::SsecPassthroughCapability;
|
||||
pub(crate) use replication_target_config_bridge::ReplicationTargetConfigBridge;
|
||||
|
||||
@@ -12,12 +12,11 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
pub(crate) use rustfs_filemeta::NULL_VERSION_ID;
|
||||
pub use rustfs_replication::{MrfOpKind, MrfReplicateEntry};
|
||||
pub(crate) use rustfs_replication::{
|
||||
REPLICATE_EXISTING, REPLICATE_EXISTING_DELETE, REPLICATE_HEAL_DELETE, ReplicateTargetDecision, ReplicatedInfos,
|
||||
ReplicatedTargetInfo, ReplicationAction, ReplicationWorkerOperation, ResyncDecision, get_replication_state,
|
||||
parse_replicate_decision, replicate_decision_for_admitted_targets, target_reset_header, version_purge_statuses_map,
|
||||
REPLICATE_EXISTING, REPLICATE_HEAL_DELETE, ReplicateTargetDecision, ReplicatedInfos, ReplicatedTargetInfo, ReplicationAction,
|
||||
ReplicationWorkerOperation, ResyncDecision, get_replication_state, parse_replicate_decision,
|
||||
replicate_decision_for_admitted_targets, target_reset_header, version_purge_statuses_map,
|
||||
};
|
||||
pub use rustfs_replication::{
|
||||
REPLICATE_INCOMING_DELETE, ReplicateDecision, ReplicateObjectInfo, ReplicationState, ReplicationStatusType, ReplicationType,
|
||||
|
||||
@@ -18,9 +18,10 @@ pub use rustfs_replication::{
|
||||
should_use_existing_delete_replication_source,
|
||||
};
|
||||
pub(crate) use rustfs_replication::{
|
||||
ReplicationDeleteSource, ReplicationMultipartPartInput, ReplicationResyncTargetObject,
|
||||
delete_replication_missing_source_decision, delete_replication_object_opts, heal_uses_delete_replication_path,
|
||||
is_retryable_delete_replication_head_error, is_version_delete_replication, replication_etags_match,
|
||||
replication_multipart_complete_actual_size, replication_multipart_part_plan, resync_target_for_object,
|
||||
should_retry_delete_marker_purge,
|
||||
ReplicationDeleteSource, ReplicationMultipartPartInput, ReplicationResyncTargetObject, delete_marker_purge_mrf_entry,
|
||||
delete_marker_purge_version_id, delete_replication_missing_source_decision, delete_replication_object_opts,
|
||||
heal_uses_delete_replication_path, is_retryable_delete_replication_head_error, is_version_delete_replication,
|
||||
replicate_delete_outcome, replication_etags_match, replication_multipart_complete_actual_size,
|
||||
replication_multipart_part_plan, resync_existing_delete_replication_info, resync_target_for_object,
|
||||
should_retry_delete_marker_purge, target_delete_version_id,
|
||||
};
|
||||
|
||||
@@ -667,6 +667,368 @@ async fn acknowledge_mrf_recovery<S: ReplicationStorage>(
|
||||
Err(EcstoreError::PreconditionFailed)
|
||||
}
|
||||
|
||||
/// Acquires the MRF recovery leader lock for the startup replay.
|
||||
/// Returns `None` (after logging) when the lock cannot be created or another
|
||||
/// node is already processing the backlog.
|
||||
async fn acquire_mrf_recovery_guard<S: ReplicationStorage>(storage: &Arc<S>) -> Option<rustfs_lock::NamespaceLockGuard> {
|
||||
let recovery_lock = match storage
|
||||
.new_ns_lock(
|
||||
ReplicationMetadataStore::rustfs_meta_bucket(),
|
||||
ReplicationMetadataStore::MRF_REPLICATION_RECOVERY_LOCK,
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(lock) => lock,
|
||||
Err(error) => {
|
||||
warn!(
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION,
|
||||
error = %error,
|
||||
"Failed to create the MRF recovery leader lock"
|
||||
);
|
||||
return None;
|
||||
}
|
||||
};
|
||||
match recovery_lock
|
||||
.get_write_lock_quiet(ReplicationLockTiming::acquire_timeout())
|
||||
.await
|
||||
{
|
||||
Ok(guard) => Some(guard),
|
||||
Err(_) => {
|
||||
debug!(
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION,
|
||||
"Another node is already processing the MRF recovery backlog"
|
||||
);
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Reads and decodes the on-disk MRF recovery file.
|
||||
/// Returns `None` when there is nothing to replay: missing file (publishes an
|
||||
/// empty available summary), read failure, or corrupt data (quarantined).
|
||||
async fn load_mrf_recovery_entries<S: ReplicationStorage>(storage: &Arc<S>) -> Option<Vec<MrfReplicateEntry>> {
|
||||
let data = match ReplicationConfigStore::read(storage.clone(), ReplicationMetadataStore::MRF_REPLICATION_FILE).await {
|
||||
Ok(d) => d,
|
||||
Err(EcstoreError::ConfigNotFound) => {
|
||||
set_durable_mrf_backlog_summary(DurableMrfBacklogSummary {
|
||||
available: true,
|
||||
buckets: Vec::new(),
|
||||
});
|
||||
return None;
|
||||
}
|
||||
Err(e) => {
|
||||
warn!(
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION,
|
||||
error = %e,
|
||||
"Failed to load MRF recovery file"
|
||||
);
|
||||
return None;
|
||||
}
|
||||
};
|
||||
|
||||
match decode_mrf_file(&data) {
|
||||
Ok(v) => Some(v),
|
||||
Err(e) => {
|
||||
warn!(
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION,
|
||||
error = %e,
|
||||
"Failed to decode MRF recovery file — preserving corrupt data"
|
||||
);
|
||||
quarantine_mrf_file(storage, &data).await;
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Replays one MRF recovery entry by operation kind.
|
||||
/// Returns `None` when the entry is skipped entirely (no admission outcome);
|
||||
/// entries that must be retried later are pushed onto `retry_entries`.
|
||||
async fn replay_mrf_entry<S: ReplicationStorage>(
|
||||
entry: &MrfReplicateEntry,
|
||||
storage: &Arc<S>,
|
||||
retry_entries: &mut Vec<MrfReplicateEntry>,
|
||||
) -> Option<ReplicationQueueAdmission> {
|
||||
match entry.op {
|
||||
MrfOpKind::Delete => replay_mrf_delete_entry(entry, storage, retry_entries).await,
|
||||
MrfOpKind::Object | MrfOpKind::Heal | MrfOpKind::ExistingObject => {
|
||||
replay_mrf_object_entry(entry, storage, retry_entries).await
|
||||
}
|
||||
MrfOpKind::Metadata => replay_mrf_metadata_entry(entry, storage, retry_entries).await,
|
||||
}
|
||||
}
|
||||
|
||||
/// Replays a delete-kind MRF entry: force-delete intents replay directly,
|
||||
/// stale force-delete generations are skipped, and plain deletes are
|
||||
/// reconstructed as heal deletes.
|
||||
async fn replay_mrf_delete_entry<S: ReplicationStorage>(
|
||||
entry: &MrfReplicateEntry,
|
||||
storage: &Arc<S>,
|
||||
retry_entries: &mut Vec<MrfReplicateEntry>,
|
||||
) -> Option<ReplicationQueueAdmission> {
|
||||
if should_replay_force_delete_intent(entry) {
|
||||
let operation_id = entry.force_delete_id?;
|
||||
let delete = force_delete_heal_replication_info(entry, operation_id);
|
||||
if replicate_delete_with_outcome(delete, storage.clone()).await {
|
||||
Some(ReplicationQueueAdmission::Queued)
|
||||
} else {
|
||||
Some(ReplicationQueueAdmission::Missed)
|
||||
}
|
||||
} else if entry.force_delete_id.is_some() {
|
||||
Some(ReplicationQueueAdmission::Skipped)
|
||||
} else {
|
||||
replay_mrf_reconstructed_delete(entry, storage, retry_entries).await
|
||||
}
|
||||
}
|
||||
|
||||
/// Pure DTO construction: heal replication info for a replayed force-delete intent.
|
||||
fn force_delete_heal_replication_info(entry: &MrfReplicateEntry, operation_id: uuid::Uuid) -> DeletedObjectReplicationInfo {
|
||||
DeletedObjectReplicationInfo {
|
||||
delete_object: ReplicationDeletedObject {
|
||||
object_name: entry.object.clone(),
|
||||
force_delete: true,
|
||||
force_delete_id: Some(operation_id),
|
||||
force_delete_target_arns: entry.target_arns.clone(),
|
||||
force_delete_generation: entry.force_delete_generation,
|
||||
..Default::default()
|
||||
},
|
||||
bucket: entry.bucket.clone(),
|
||||
op_type: ReplicationType::Heal,
|
||||
event_type: REPLICATE_HEAL_DELETE.to_string(),
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
/// Reconstruct a heal delete and re-queue it. We do NOT call
|
||||
/// get_object_info here because the delete-marker or version may
|
||||
/// already be absent from the local store — that is expected.
|
||||
async fn replay_mrf_reconstructed_delete<S: ReplicationStorage>(
|
||||
entry: &MrfReplicateEntry,
|
||||
storage: &Arc<S>,
|
||||
retry_entries: &mut Vec<MrfReplicateEntry>,
|
||||
) -> Option<ReplicationQueueAdmission> {
|
||||
let versioned = ReplicationVersioningStore::prefix_enabled(&entry.bucket, &entry.object).await;
|
||||
let oi = ObjectInfo {
|
||||
bucket: entry.bucket.clone(),
|
||||
name: entry.object.clone(),
|
||||
version_id: entry.version_id,
|
||||
delete_marker: entry.delete_marker,
|
||||
..Default::default()
|
||||
};
|
||||
let dsc = resolve_mrf_delete_replicate_decision(entry, &oi, versioned, retry_entries).await?;
|
||||
let dv = reconstructed_heal_delete_info(entry, &oi, &dsc);
|
||||
if replicate_delete_with_outcome(dv, storage.clone()).await {
|
||||
Some(ReplicationQueueAdmission::Queued)
|
||||
} else {
|
||||
Some(ReplicationQueueAdmission::Missed)
|
||||
}
|
||||
}
|
||||
|
||||
/// The MRF entry does not persist the replication decision and the
|
||||
/// source object is gone, so re-derive the decision from the live
|
||||
/// bucket config (mirroring get_heal_replicate_object_info) and set
|
||||
/// it on the reconstructed delete. Without this the decision string
|
||||
/// is empty and the delete replicates to zero targets — a silent
|
||||
/// no-op that leaves replicas diverged (backlog#858 / #799 B9).
|
||||
async fn resolve_mrf_delete_replicate_decision(
|
||||
entry: &MrfReplicateEntry,
|
||||
oi: &ObjectInfo,
|
||||
versioned: bool,
|
||||
retry_entries: &mut Vec<MrfReplicateEntry>,
|
||||
) -> Option<ReplicateDecision> {
|
||||
if entry.target_arns.is_empty() {
|
||||
match ReplicationMetadataStore::optional_replication_config(&entry.bucket).await {
|
||||
Ok(None) => None,
|
||||
Err(_) => {
|
||||
retry_entries.push(entry.clone());
|
||||
None
|
||||
}
|
||||
Ok(Some(_)) => match check_replicate_delete_strict(
|
||||
&entry.bucket,
|
||||
&ObjectToDelete {
|
||||
object_name: entry.object.clone(),
|
||||
version_id: entry.version_id,
|
||||
..Default::default()
|
||||
},
|
||||
oi,
|
||||
&ObjectOptions {
|
||||
versioned,
|
||||
..Default::default()
|
||||
},
|
||||
None,
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(dsc) => Some(dsc),
|
||||
Err(_) => {
|
||||
retry_entries.push(entry.clone());
|
||||
None
|
||||
}
|
||||
},
|
||||
}
|
||||
} else {
|
||||
Some(replicate_decision_for_admitted_targets(&entry.target_arns))
|
||||
}
|
||||
}
|
||||
|
||||
/// Pure DTO construction: reconstructed heal delete carrying the re-derived
|
||||
/// replication decision.
|
||||
fn reconstructed_heal_delete_info(
|
||||
entry: &MrfReplicateEntry,
|
||||
oi: &ObjectInfo,
|
||||
dsc: &ReplicateDecision,
|
||||
) -> DeletedObjectReplicationInfo {
|
||||
let mut rstate = oi.replication_state();
|
||||
rstate.replicate_decision_str = dsc.to_string();
|
||||
|
||||
let delete_marker_mtime = entry
|
||||
.delete_marker_mtime
|
||||
.and_then(|nanos| OffsetDateTime::from_unix_timestamp_nanos(i128::from(nanos)).ok());
|
||||
|
||||
DeletedObjectReplicationInfo {
|
||||
delete_object: ReplicationDeletedObject {
|
||||
object_name: entry.object.clone(),
|
||||
version_id: entry.version_id,
|
||||
delete_marker_version_id: entry.delete_marker_version_id,
|
||||
delete_marker: entry.delete_marker,
|
||||
delete_marker_mtime,
|
||||
force_delete: entry.force_delete,
|
||||
replication_state: Some(rstate),
|
||||
..Default::default()
|
||||
},
|
||||
bucket: entry.bucket.clone(),
|
||||
op_type: ReplicationType::Heal,
|
||||
event_type: REPLICATE_HEAL_DELETE.to_string(),
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
/// Replays an Object/Heal/ExistingObject MRF entry against the live source object.
|
||||
async fn replay_mrf_object_entry<S: ReplicationStorage>(
|
||||
entry: &MrfReplicateEntry,
|
||||
storage: &Arc<S>,
|
||||
retry_entries: &mut Vec<MrfReplicateEntry>,
|
||||
) -> Option<ReplicationQueueAdmission> {
|
||||
let opts = ObjectOptions {
|
||||
version_id: entry.version_id.map(|u| u.to_string()),
|
||||
..Default::default()
|
||||
};
|
||||
let oi = match storage.get_object_info(&entry.bucket, &entry.object, &opts).await {
|
||||
Ok(oi) => oi,
|
||||
Err(e) => {
|
||||
debug!(
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION,
|
||||
bucket = %entry.bucket,
|
||||
object = %entry.object,
|
||||
error = %e,
|
||||
"MRF recovery: source object lookup failed"
|
||||
);
|
||||
if should_retry_mrf_source_lookup(&e) {
|
||||
retry_entries.push(entry.clone());
|
||||
}
|
||||
return None;
|
||||
}
|
||||
};
|
||||
if entry.target_arns.is_empty() {
|
||||
// Legacy entries predate target admission persistence. They cannot
|
||||
// be safely attributed, so retain the old live-config fallback.
|
||||
Some(queue_replication_heal(&entry.bucket, oi, entry.retry_count.max(0) as u32).await)
|
||||
} else {
|
||||
let roi = admitted_mrf_replicate_object(oi, entry, entry.op.replication_type());
|
||||
if replicate_object_with_outcome(roi, storage.clone()).await.1 {
|
||||
Some(ReplicationQueueAdmission::Queued)
|
||||
} else {
|
||||
Some(ReplicationQueueAdmission::Missed)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Replays a metadata-kind MRF entry against the live source object.
|
||||
async fn replay_mrf_metadata_entry<S: ReplicationStorage>(
|
||||
entry: &MrfReplicateEntry,
|
||||
storage: &Arc<S>,
|
||||
retry_entries: &mut Vec<MrfReplicateEntry>,
|
||||
) -> Option<ReplicationQueueAdmission> {
|
||||
let opts = ObjectOptions {
|
||||
version_id: entry.version_id.map(|u| u.to_string()),
|
||||
..Default::default()
|
||||
};
|
||||
let oi = match storage.get_object_info(&entry.bucket, &entry.object, &opts).await {
|
||||
Ok(oi) => oi,
|
||||
Err(e) => {
|
||||
debug!(
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION,
|
||||
bucket = %entry.bucket,
|
||||
object = %entry.object,
|
||||
error = %e,
|
||||
"MRF metadata recovery: source object lookup failed"
|
||||
);
|
||||
if should_retry_mrf_source_lookup(&e) {
|
||||
retry_entries.push(entry.clone());
|
||||
}
|
||||
return None;
|
||||
}
|
||||
};
|
||||
if entry.target_arns.is_empty() {
|
||||
Some(queue_replication_metadata(&entry.bucket, oi, entry.retry_count.max(0) as u32).await)
|
||||
} else {
|
||||
let roi = admitted_mrf_replicate_object(oi, entry, ReplicationType::Metadata);
|
||||
if replicate_object_with_outcome(roi, storage.clone()).await.1 {
|
||||
Some(ReplicationQueueAdmission::Queued)
|
||||
} else {
|
||||
Some(ReplicationQueueAdmission::Missed)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Pure DTO construction: replicate-object info for an entry with persisted
|
||||
/// admitted targets, carrying over the entry's retry count.
|
||||
fn admitted_mrf_replicate_object(oi: ObjectInfo, entry: &MrfReplicateEntry, op_type: ReplicationType) -> ReplicateObjectInfo {
|
||||
let dsc = replicate_decision_for_admitted_targets(&entry.target_arns);
|
||||
let mut roi = replicate_object_info_from_object_info(oi, dsc, op_type);
|
||||
roi.retry_count = entry.retry_count.max(0) as u32;
|
||||
roi
|
||||
}
|
||||
|
||||
/// Acknowledges the replayed MRF prefix and returns the retained backlog.
|
||||
/// On acknowledgement failure the backlog is preserved for the next startup and
|
||||
/// re-read (falling back to the replayed snapshot) so the published summary stays accurate.
|
||||
async fn resolve_retained_mrf_entries<S: ReplicationStorage>(
|
||||
storage: &Arc<S>,
|
||||
recovery_guard: &rustfs_lock::NamespaceLockGuard,
|
||||
entries: &[MrfReplicateEntry],
|
||||
retry_entries: &[MrfReplicateEntry],
|
||||
) -> Vec<MrfReplicateEntry> {
|
||||
match acknowledge_mrf_recovery(storage.clone(), recovery_guard, entries, retry_entries).await {
|
||||
Ok(retained) => retained,
|
||||
Err(error) => {
|
||||
warn!(
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION,
|
||||
error = %error,
|
||||
"Failed to acknowledge the MRF recovery prefix; preserving it for the next startup"
|
||||
);
|
||||
match read_mrf_entries(storage.clone()).await {
|
||||
Ok(current) => current,
|
||||
Err(read_error) => {
|
||||
warn!(
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION,
|
||||
error = %read_error,
|
||||
"Failed to refresh the MRF backlog after acknowledgement failure"
|
||||
);
|
||||
entries.to_vec()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
#[error("replication resync {active_resync_id} is already active for {bucket}/{arn}")]
|
||||
struct ResyncActiveConflictError {
|
||||
@@ -1221,71 +1583,12 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
|
||||
let storage = self.storage.clone();
|
||||
|
||||
let handle = tokio::spawn(async move {
|
||||
let recovery_lock = match storage
|
||||
.new_ns_lock(
|
||||
ReplicationMetadataStore::rustfs_meta_bucket(),
|
||||
ReplicationMetadataStore::MRF_REPLICATION_RECOVERY_LOCK,
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(lock) => lock,
|
||||
Err(error) => {
|
||||
warn!(
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION,
|
||||
error = %error,
|
||||
"Failed to create the MRF recovery leader lock"
|
||||
);
|
||||
return;
|
||||
}
|
||||
};
|
||||
let recovery_guard = match recovery_lock
|
||||
.get_write_lock_quiet(ReplicationLockTiming::acquire_timeout())
|
||||
.await
|
||||
{
|
||||
Ok(guard) => guard,
|
||||
Err(_) => {
|
||||
debug!(
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION,
|
||||
"Another node is already processing the MRF recovery backlog"
|
||||
);
|
||||
return;
|
||||
}
|
||||
let Some(recovery_guard) = acquire_mrf_recovery_guard(&storage).await else {
|
||||
return;
|
||||
};
|
||||
|
||||
let data = match ReplicationConfigStore::read(storage.clone(), ReplicationMetadataStore::MRF_REPLICATION_FILE).await {
|
||||
Ok(d) => d,
|
||||
Err(EcstoreError::ConfigNotFound) => {
|
||||
set_durable_mrf_backlog_summary(DurableMrfBacklogSummary {
|
||||
available: true,
|
||||
buckets: Vec::new(),
|
||||
});
|
||||
return;
|
||||
}
|
||||
Err(e) => {
|
||||
warn!(
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION,
|
||||
error = %e,
|
||||
"Failed to load MRF recovery file"
|
||||
);
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
let entries = match decode_mrf_file(&data) {
|
||||
Ok(v) => v,
|
||||
Err(e) => {
|
||||
warn!(
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION,
|
||||
error = %e,
|
||||
"Failed to decode MRF recovery file — preserving corrupt data"
|
||||
);
|
||||
quarantine_mrf_file(&storage, &data).await;
|
||||
return;
|
||||
}
|
||||
let Some(entries) = load_mrf_recovery_entries(&storage).await else {
|
||||
return;
|
||||
};
|
||||
set_durable_mrf_backlog_snapshot(durable_mrf_backlog_summary_from_entries(&entries));
|
||||
|
||||
@@ -1294,187 +1597,8 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
|
||||
let mut retry_entries = Vec::new();
|
||||
|
||||
for entry in entries.iter() {
|
||||
let admission = match entry.op {
|
||||
MrfOpKind::Delete => {
|
||||
if should_replay_force_delete_intent(entry) {
|
||||
let Some(operation_id) = entry.force_delete_id else {
|
||||
continue;
|
||||
};
|
||||
let delete = DeletedObjectReplicationInfo {
|
||||
delete_object: ReplicationDeletedObject {
|
||||
object_name: entry.object.clone(),
|
||||
force_delete: true,
|
||||
force_delete_id: Some(operation_id),
|
||||
force_delete_target_arns: entry.target_arns.clone(),
|
||||
force_delete_generation: entry.force_delete_generation,
|
||||
..Default::default()
|
||||
},
|
||||
bucket: entry.bucket.clone(),
|
||||
op_type: ReplicationType::Heal,
|
||||
event_type: REPLICATE_HEAL_DELETE.to_string(),
|
||||
..Default::default()
|
||||
};
|
||||
if replicate_delete_with_outcome(delete, storage.clone()).await {
|
||||
ReplicationQueueAdmission::Queued
|
||||
} else {
|
||||
ReplicationQueueAdmission::Missed
|
||||
}
|
||||
} else if entry.force_delete_id.is_some() {
|
||||
ReplicationQueueAdmission::Skipped
|
||||
} else {
|
||||
// Reconstruct a heal delete and re-queue it. We do NOT call
|
||||
// get_object_info here because the delete-marker or version may
|
||||
// already be absent from the local store — that is expected.
|
||||
//
|
||||
// The MRF entry does not persist the replication decision and the
|
||||
// source object is gone, so re-derive the decision from the live
|
||||
// bucket config (mirroring get_heal_replicate_object_info) and set
|
||||
// it on the reconstructed delete. Without this the decision string
|
||||
// is empty and the delete replicates to zero targets — a silent
|
||||
// no-op that leaves replicas diverged (backlog#858 / #799 B9).
|
||||
let versioned = ReplicationVersioningStore::prefix_enabled(&entry.bucket, &entry.object).await;
|
||||
let oi = ObjectInfo {
|
||||
bucket: entry.bucket.clone(),
|
||||
name: entry.object.clone(),
|
||||
version_id: entry.version_id,
|
||||
delete_marker: entry.delete_marker,
|
||||
..Default::default()
|
||||
};
|
||||
let dsc = if entry.target_arns.is_empty() {
|
||||
match ReplicationMetadataStore::optional_replication_config(&entry.bucket).await {
|
||||
Ok(None) => continue,
|
||||
Err(_) => {
|
||||
retry_entries.push(entry.clone());
|
||||
continue;
|
||||
}
|
||||
Ok(Some(_)) => match check_replicate_delete_strict(
|
||||
&entry.bucket,
|
||||
&ObjectToDelete {
|
||||
object_name: entry.object.clone(),
|
||||
version_id: entry.version_id,
|
||||
..Default::default()
|
||||
},
|
||||
&oi,
|
||||
&ObjectOptions {
|
||||
versioned,
|
||||
..Default::default()
|
||||
},
|
||||
None,
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(dsc) => dsc,
|
||||
Err(_) => {
|
||||
retry_entries.push(entry.clone());
|
||||
continue;
|
||||
}
|
||||
},
|
||||
}
|
||||
} else {
|
||||
replicate_decision_for_admitted_targets(&entry.target_arns)
|
||||
};
|
||||
let mut rstate = oi.replication_state();
|
||||
rstate.replicate_decision_str = dsc.to_string();
|
||||
|
||||
let delete_marker_mtime = entry
|
||||
.delete_marker_mtime
|
||||
.and_then(|nanos| OffsetDateTime::from_unix_timestamp_nanos(i128::from(nanos)).ok());
|
||||
|
||||
let dv = DeletedObjectReplicationInfo {
|
||||
delete_object: ReplicationDeletedObject {
|
||||
object_name: entry.object.clone(),
|
||||
version_id: entry.version_id,
|
||||
delete_marker_version_id: entry.delete_marker_version_id,
|
||||
delete_marker: entry.delete_marker,
|
||||
delete_marker_mtime,
|
||||
force_delete: entry.force_delete,
|
||||
replication_state: Some(rstate),
|
||||
..Default::default()
|
||||
},
|
||||
bucket: entry.bucket.clone(),
|
||||
op_type: ReplicationType::Heal,
|
||||
event_type: REPLICATE_HEAL_DELETE.to_string(),
|
||||
..Default::default()
|
||||
};
|
||||
if replicate_delete_with_outcome(dv, storage.clone()).await {
|
||||
ReplicationQueueAdmission::Queued
|
||||
} else {
|
||||
ReplicationQueueAdmission::Missed
|
||||
}
|
||||
}
|
||||
}
|
||||
MrfOpKind::Object | MrfOpKind::Heal | MrfOpKind::ExistingObject => {
|
||||
let opts = ObjectOptions {
|
||||
version_id: entry.version_id.map(|u| u.to_string()),
|
||||
..Default::default()
|
||||
};
|
||||
let oi = match storage.get_object_info(&entry.bucket, &entry.object, &opts).await {
|
||||
Ok(oi) => oi,
|
||||
Err(e) => {
|
||||
debug!(
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION,
|
||||
bucket = %entry.bucket,
|
||||
object = %entry.object,
|
||||
error = %e,
|
||||
"MRF recovery: source object lookup failed"
|
||||
);
|
||||
if should_retry_mrf_source_lookup(&e) {
|
||||
retry_entries.push(entry.clone());
|
||||
}
|
||||
continue;
|
||||
}
|
||||
};
|
||||
if entry.target_arns.is_empty() {
|
||||
// Legacy entries predate target admission persistence. They cannot
|
||||
// be safely attributed, so retain the old live-config fallback.
|
||||
queue_replication_heal(&entry.bucket, oi, entry.retry_count.max(0) as u32).await
|
||||
} else {
|
||||
let dsc = replicate_decision_for_admitted_targets(&entry.target_arns);
|
||||
let mut roi = replicate_object_info_from_object_info(oi, dsc, entry.op.replication_type());
|
||||
roi.retry_count = entry.retry_count.max(0) as u32;
|
||||
if replicate_object_with_outcome(roi, storage.clone()).await.1 {
|
||||
ReplicationQueueAdmission::Queued
|
||||
} else {
|
||||
ReplicationQueueAdmission::Missed
|
||||
}
|
||||
}
|
||||
}
|
||||
MrfOpKind::Metadata => {
|
||||
let opts = ObjectOptions {
|
||||
version_id: entry.version_id.map(|u| u.to_string()),
|
||||
..Default::default()
|
||||
};
|
||||
let oi = match storage.get_object_info(&entry.bucket, &entry.object, &opts).await {
|
||||
Ok(oi) => oi,
|
||||
Err(e) => {
|
||||
debug!(
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION,
|
||||
bucket = %entry.bucket,
|
||||
object = %entry.object,
|
||||
error = %e,
|
||||
"MRF metadata recovery: source object lookup failed"
|
||||
);
|
||||
if should_retry_mrf_source_lookup(&e) {
|
||||
retry_entries.push(entry.clone());
|
||||
}
|
||||
continue;
|
||||
}
|
||||
};
|
||||
if entry.target_arns.is_empty() {
|
||||
queue_replication_metadata(&entry.bucket, oi, entry.retry_count.max(0) as u32).await
|
||||
} else {
|
||||
let dsc = replicate_decision_for_admitted_targets(&entry.target_arns);
|
||||
let mut roi = replicate_object_info_from_object_info(oi, dsc, ReplicationType::Metadata);
|
||||
roi.retry_count = entry.retry_count.max(0) as u32;
|
||||
if replicate_object_with_outcome(roi, storage.clone()).await.1 {
|
||||
ReplicationQueueAdmission::Queued
|
||||
} else {
|
||||
ReplicationQueueAdmission::Missed
|
||||
}
|
||||
}
|
||||
}
|
||||
let Some(admission) = replay_mrf_entry(entry, &storage, &mut retry_entries).await else {
|
||||
continue;
|
||||
};
|
||||
|
||||
if admission == ReplicationQueueAdmission::Missed {
|
||||
@@ -1484,29 +1608,7 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
|
||||
}
|
||||
}
|
||||
|
||||
let retained = match acknowledge_mrf_recovery(storage.clone(), &recovery_guard, &entries, &retry_entries).await {
|
||||
Ok(retained) => retained,
|
||||
Err(error) => {
|
||||
warn!(
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION,
|
||||
error = %error,
|
||||
"Failed to acknowledge the MRF recovery prefix; preserving it for the next startup"
|
||||
);
|
||||
match read_mrf_entries(storage.clone()).await {
|
||||
Ok(current) => current,
|
||||
Err(read_error) => {
|
||||
warn!(
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION,
|
||||
error = %read_error,
|
||||
"Failed to refresh the MRF backlog after acknowledgement failure"
|
||||
);
|
||||
entries.clone()
|
||||
}
|
||||
}
|
||||
}
|
||||
};
|
||||
let retained = resolve_retained_mrf_entries(&storage, &recovery_guard, &entries, &retry_entries).await;
|
||||
let retained_count = retained.len();
|
||||
set_durable_mrf_backlog_snapshot(durable_mrf_backlog_summary_from_entries(&retained));
|
||||
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Proxy-target selection for reads of objects not yet replicated locally
|
||||
//! (MinIO `getProxyTargets`, bucket-replication.go).
|
||||
//!
|
||||
//! During the active-active replication lag window a GET/HEAD/Tagging request
|
||||
//! for an object the local site does not have yet may be served by proxying to
|
||||
//! a replication target. This module only *selects* the candidate targets; the
|
||||
//! request-path callers perform the remote calls and response translation.
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use tracing::debug;
|
||||
|
||||
use super::replication_config_boundary::{ObjectOpts, ReplicationConfigurationExt as _};
|
||||
use super::replication_object_config::get_replication_config;
|
||||
use super::replication_storage_boundary::ObjectOptions;
|
||||
use super::replication_target_boundary::{ReplicationTargetStore, TargetClient};
|
||||
|
||||
/// Returns the replication-target clients eligible to serve a proxied read of
|
||||
/// `bucket/object`, in rule order. Mirrors MinIO's `getProxyTargets`:
|
||||
///
|
||||
/// - the `source-proxy-request` header family was present at all
|
||||
/// (`opts.proxy_request` / `opts.proxy_header_set`, MinIO `ProxyRequest` /
|
||||
/// `ProxyHeaderSet`) -> empty. "true" is the anti-loop marker of an
|
||||
/// already-proxied client read; "false" is what a peer's replication
|
||||
/// worker sends on convergence HEADs so the receiver answers locally —
|
||||
/// proxying that miss back would echo the source object and fake
|
||||
/// convergence, permanently skipping replication;
|
||||
/// - the bucket's versioning is suspended for the object -> empty;
|
||||
/// - no replication configuration / no matching rule -> empty;
|
||||
/// - otherwise every distinct target ARN whose rules match the object,
|
||||
/// resolved through the bucket target system, skipping targets that opted
|
||||
/// out of proxying (`disable_proxy`).
|
||||
pub async fn get_proxy_targets(bucket: &str, object: &str, opts: &ObjectOptions) -> Vec<Arc<TargetClient>> {
|
||||
if opts.proxy_request || opts.proxy_header_set {
|
||||
return Vec::new();
|
||||
}
|
||||
if opts.version_suspended {
|
||||
return Vec::new();
|
||||
}
|
||||
|
||||
let cfg = match get_replication_config(bucket).await {
|
||||
Ok(Some(cfg)) => cfg,
|
||||
Ok(None) => return Vec::new(),
|
||||
Err(err) => {
|
||||
debug!(bucket, object, error = %err, "read proxy: failed to load replication config; not proxying");
|
||||
return Vec::new();
|
||||
}
|
||||
};
|
||||
|
||||
let arns = cfg.filter_target_arns(&ObjectOpts {
|
||||
name: object.to_string(),
|
||||
..Default::default()
|
||||
});
|
||||
|
||||
let mut targets = Vec::with_capacity(arns.len());
|
||||
for arn in arns {
|
||||
let Some(client) = ReplicationTargetStore::remote_target_client(bucket, &arn).await else {
|
||||
debug!(bucket, object, arn, "read proxy: no client for replication target ARN");
|
||||
continue;
|
||||
};
|
||||
if client.disable_proxy {
|
||||
continue;
|
||||
}
|
||||
targets.push(client);
|
||||
}
|
||||
|
||||
targets
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn opts() -> ObjectOptions {
|
||||
ObjectOptions::default()
|
||||
}
|
||||
|
||||
/// Anti-loop: a request that was already proxied by a peer must never be
|
||||
/// proxied onward, regardless of replication configuration.
|
||||
#[tokio::test]
|
||||
async fn proxy_request_yields_no_targets() {
|
||||
let targets = get_proxy_targets(
|
||||
"bucket",
|
||||
"object",
|
||||
&ObjectOptions {
|
||||
proxy_request: true,
|
||||
..opts()
|
||||
},
|
||||
)
|
||||
.await;
|
||||
assert!(targets.is_empty());
|
||||
}
|
||||
|
||||
/// MinIO `ProxyHeaderSet` parity: the header family being present at all
|
||||
/// disables proxying, even with the value "false" — that is what a
|
||||
/// peer's replication worker sends on convergence HEADs.
|
||||
#[tokio::test]
|
||||
async fn proxy_header_set_yields_no_targets() {
|
||||
let targets = get_proxy_targets(
|
||||
"bucket",
|
||||
"object",
|
||||
&ObjectOptions {
|
||||
proxy_header_set: true,
|
||||
proxy_request: false,
|
||||
..opts()
|
||||
},
|
||||
)
|
||||
.await;
|
||||
assert!(targets.is_empty());
|
||||
}
|
||||
|
||||
/// Suspended versioning disables proxying (MinIO parity): the local null
|
||||
/// version is authoritative and a remote read could resurrect data.
|
||||
#[tokio::test]
|
||||
async fn version_suspended_yields_no_targets() {
|
||||
let targets = get_proxy_targets(
|
||||
"bucket",
|
||||
"object",
|
||||
&ObjectOptions {
|
||||
version_suspended: true,
|
||||
..opts()
|
||||
},
|
||||
)
|
||||
.await;
|
||||
assert!(targets.is_empty());
|
||||
}
|
||||
|
||||
/// A bucket without replication configuration has nothing to proxy to.
|
||||
/// (No metadata system is running in unit tests, so the config lookup
|
||||
/// resolves to "no configuration" — the same empty-result contract.)
|
||||
#[tokio::test]
|
||||
async fn missing_replication_config_yields_no_targets() {
|
||||
let targets = get_proxy_targets("bucket-without-replication", "object", &opts()).await;
|
||||
assert!(targets.is_empty());
|
||||
}
|
||||
}
|
||||
@@ -15,10 +15,15 @@
|
||||
use super::replication_error_boundary::{Error, Result};
|
||||
use super::replication_filemeta_boundary::MrfReplicateEntry;
|
||||
|
||||
/// Kept test-only: the runtime consumer was the worker HEAD's fake proxy
|
||||
/// counting (removed in backlog#1675 P1-5); the resyncer tests still pin the
|
||||
/// classifier's semantics for the real client read-proxy failure accounting.
|
||||
#[cfg(test)]
|
||||
pub(crate) use rustfs_replication::should_count_head_proxy_failure;
|
||||
pub use rustfs_replication::{BucketReplicationResyncStatus, ResyncOpts, ResyncStatusType, TargetReplicationResyncStatus};
|
||||
pub(crate) use rustfs_replication::{
|
||||
is_version_id_mismatch, resync_state_accepts_update, sanitize_resync_error_detail, should_auto_resume_resync,
|
||||
should_count_head_proxy_failure,
|
||||
is_version_id_mismatch, resync_state_accepts_update, resync_status_duration, sanitize_resync_error_detail,
|
||||
should_auto_resume_resync,
|
||||
};
|
||||
|
||||
#[allow(
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1161,6 +1161,31 @@ mod tests {
|
||||
assert!(all.contains_key("proxy-only-bucket"));
|
||||
}
|
||||
|
||||
/// Pins the read-proxy metric contract (backlog#1675 P1-5): the API
|
||||
/// strings the GET/HEAD/Tagging proxy paths record map onto the
|
||||
/// get/head/tagging totals, and only unexpected failures raise the
|
||||
/// failed counters.
|
||||
#[tokio::test]
|
||||
async fn test_proxy_stats_map_read_proxy_apis_to_totals() {
|
||||
let stats = ReplicationStats::new();
|
||||
stats.inc_proxy("proxy-bucket", "GetObject", false).await;
|
||||
stats.inc_proxy("proxy-bucket", "GetObject", true).await;
|
||||
stats.inc_proxy("proxy-bucket", "HeadObject", false).await;
|
||||
stats.inc_proxy("proxy-bucket", "GetObjectTagging", false).await;
|
||||
stats.inc_proxy("proxy-bucket", "PutObjectTagging", false).await;
|
||||
stats.inc_proxy("proxy-bucket", "DeleteObjectTagging", true).await;
|
||||
|
||||
let metric = stats.get_proxy_stats("proxy-bucket").await;
|
||||
assert_eq!(metric.get_total, 2);
|
||||
assert_eq!(metric.get_failed, 1);
|
||||
assert_eq!(metric.head_total, 1);
|
||||
assert_eq!(metric.head_failed, 0);
|
||||
assert_eq!(metric.get_tag_total, 1);
|
||||
assert_eq!(metric.put_tag_total, 1);
|
||||
assert_eq!(metric.delete_tag_total, 1);
|
||||
assert_eq!(metric.delete_tag_failed, 1);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_calculate_bucket_replication_stats_merges_resync_metrics() {
|
||||
let stats = ReplicationStats::new();
|
||||
|
||||
@@ -36,11 +36,15 @@ use time::OffsetDateTime;
|
||||
use time::format_description::well_known::Rfc3339;
|
||||
|
||||
pub(crate) use crate::bucket::bucket_target_sys::{
|
||||
AdvancedPutOptions, PutObjectOptions, PutObjectPartOptions, RemoveObjectOptions, TargetClient,
|
||||
AdvancedPutOptions, PutObjectOptions, PutObjectPartOptions, RemoveObjectOptions, TargetClient, resolve_read_api_version_id,
|
||||
};
|
||||
#[cfg(test)]
|
||||
pub(crate) use crate::bucket::target::BucketTarget;
|
||||
pub(crate) use crate::bucket::target::BucketTargets;
|
||||
pub use rustfs_replication::SsecPassthroughCapability;
|
||||
pub(crate) use rustfs_replication::{
|
||||
SsecPassthroughGate, is_replication_target_offline_error, ssec_passthrough_gate, version_identity_drifted,
|
||||
};
|
||||
|
||||
use super::replication_config_store::ReplicationConfigStore;
|
||||
use super::replication_error_boundary::{Error, Result};
|
||||
@@ -65,6 +69,8 @@ static STANDARD_HEADERS: &[&str] = &[
|
||||
];
|
||||
|
||||
const ERR_REPLICATION_ENCRYPTION_METADATA_UNSUPPORTED: &str = "replication source contains unsupported encryption metadata";
|
||||
pub(crate) const ERR_REPLICATION_SSEC_PASSTHROUGH_UNSUPPORTED: &str = "replication target does not support SSE-C passthrough: the replica would lose its decryption material \
|
||||
(run ?replication-check to re-probe)";
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
enum ReplicationSourceEncryption {
|
||||
@@ -146,6 +152,13 @@ pub(crate) fn replication_object_is_ssec_encrypted(user_defined: &HashMap<String
|
||||
rustfs_replication::is_ssec_encrypted(user_defined)
|
||||
}
|
||||
|
||||
/// HeadObjectOutput adapter over the pure SSE-C passthrough evidence
|
||||
/// judgment owned by `rustfs-replication`: extract the echoed
|
||||
/// customer-algorithm header and let the crate-owned policy decide.
|
||||
pub(crate) fn ssec_passthrough_evidence_present(head: &HeadObjectOutput) -> bool {
|
||||
rustfs_replication::ssec_passthrough_evidence_present(head.sse_customer_algorithm.as_deref())
|
||||
}
|
||||
|
||||
pub(crate) struct ReplicationTargetStore;
|
||||
|
||||
impl ReplicationTargetStore {
|
||||
@@ -165,6 +178,17 @@ impl ReplicationTargetStore {
|
||||
BucketTargetSys::get().mark_target_offline(target_client).await
|
||||
}
|
||||
|
||||
/// Returns the cached verdict and whether it has outlived its TTL.
|
||||
pub(crate) async fn ssec_passthrough_capability(arn: &str) -> (SsecPassthroughCapability, bool) {
|
||||
BucketTargetSys::get().ssec_passthrough_capability(arn).await
|
||||
}
|
||||
|
||||
pub(crate) async fn record_ssec_passthrough_capability(arn: &str, capability: SsecPassthroughCapability) {
|
||||
BucketTargetSys::get()
|
||||
.record_ssec_passthrough_capability(arn, capability)
|
||||
.await
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) async fn register_test_target(target_client: &Arc<TargetClient>) {
|
||||
BucketTargetSys::get().arn_remotes_map.write().await.insert(
|
||||
@@ -898,6 +922,27 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
/// Pins the HeadObjectOutput field extraction feeding the crate-owned
|
||||
/// evidence judgment (the gate/evidence policy matrix itself is pinned in
|
||||
/// `rustfs-replication`'s object tests).
|
||||
#[test]
|
||||
fn ssec_passthrough_evidence_requires_customer_algorithm_echo() {
|
||||
let with_evidence = HeadObjectOutput::builder().sse_customer_algorithm("AES256").build();
|
||||
assert!(ssec_passthrough_evidence_present(&with_evidence));
|
||||
|
||||
let empty_algorithm = HeadObjectOutput::builder().sse_customer_algorithm("").build();
|
||||
assert!(
|
||||
!ssec_passthrough_evidence_present(&empty_algorithm),
|
||||
"an empty echo is not evidence of preserved SSE-C material"
|
||||
);
|
||||
|
||||
let without_evidence = HeadObjectOutput::builder().e_tag("\"abc\"").content_length(8).build();
|
||||
assert!(
|
||||
!ssec_passthrough_evidence_present(&without_evidence),
|
||||
"a plain HEAD response must classify the target as having dropped the material"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replication_put_options_adds_ssec_checksum_metadata() {
|
||||
let metadata = HashMap::from([(SSEC_ALGORITHM_HEADER.to_string(), "AES256".to_string())]);
|
||||
|
||||
@@ -277,6 +277,20 @@ pub struct ObjectOptions {
|
||||
/// fence avoids recursively acquiring the read lock behind a queued writer.
|
||||
pub bucket_lifecycle_lock_fence: Option<NamespaceLockFence>,
|
||||
pub replication_request: bool,
|
||||
/// True when the inbound request carried the
|
||||
/// `{x-rustfs-,x-minio-}source-proxy-request` header family with the
|
||||
/// value "true": the request was already proxied by a replication peer,
|
||||
/// so this server must not proxy a local miss onward (anti-loop,
|
||||
/// MinIO-compatible). The header only disables proxying — it grants no
|
||||
/// capability — so no authorization gate is required to honor it.
|
||||
pub proxy_request: bool,
|
||||
/// True when the `source-proxy-request` header family was present at
|
||||
/// all, regardless of value (MinIO's `ProxyHeaderSet`). A replication
|
||||
/// peer sends `source-proxy-request: false` on its worker convergence
|
||||
/// HEADs precisely so the receiver answers locally instead of proxying
|
||||
/// back — otherwise a proxied 404->200 echo makes the worker believe the
|
||||
/// object already converged and it never replicates it.
|
||||
pub proxy_header_set: bool,
|
||||
/// Source-cluster LWW timestamps carried by an authorized replication
|
||||
/// request; None when the source never modified the category. Only the
|
||||
/// replication-authorized options builders may set these.
|
||||
|
||||
Reference in New Issue
Block a user