mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-18 18:46:17 +00:00
1cf0f7af15
* refactor(replication): split four oversized hot-path functions into focused helpers Pure-move decomposition of the four oversized functions flagged by the replication compatibility review (P1-18), unblocking migration milestone M2 which requires resyncer moves to stay mechanical: - resync_bucket (522 lines -> 61-line step sequence): leader lock, target resolution, walk/collector/worker spawning, and dispatch loop extracted into focused helpers; pure decision helpers (DTO builders, HEAD-result classification) separated from IO orchestration. - replicate_all (411 lines -> 113-line main body): initial target-info seeding, read/stat option builders, skip-path notes, target HEAD action resolution, and the multipart/single-put payload transport extracted as private free functions. - start_mrf_processor (306 lines -> 46-line spawn body): recovery guard, ledger load, per-entry replay (delete/object/metadata), and retained entry resolution extracted; retry bookkeeping semantics preserved exactly (inner continue-paths push inside helpers, outer Missed push stays in the loop). - apply_iam_item (255 lines -> match dispatch skeleton): one helper per IAM item type. No behavior change: log texts, error paths, event emissions, and metric counts are byte-identical; existing tests unchanged and green (238 ecstore replication/mrf/resync + 232 rustfs site-replication). * feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets (#6172) * feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets Implements the MinIO active-active read-proxy protocol (P1-5 of the replication compatibility review): when a GET/HEAD/GetObjectTagging/ PutObjectTagging/DeleteObjectTagging request fails locally with not-found and the bucket has replication targets, the request is proxied to the targets in rule order, mirroring bucket-replication.go proxyGetToReplicationTarget/proxyHeadToRepTarget/proxyTaggingToRepTarget. Protocol surface: - Anti-loop: inbound {x-rustfs-,x-minio-}source-proxy-request is parsed into ObjectOptions (proxy_request + proxy_header_set, matching MinIO ProxyRequest/ProxyHeaderSet); a request carrying the marker with ANY value is never re-proxied. Outbound client proxy calls send the marker as "true"; replication worker convergence HEADs send it as "false" so a peer's proxy layer cannot answer a convergence check by proxying back to the source (which would fake Completed without a PUT). - Target selection: new replication_proxy.rs get_proxy_targets — empty when the marker is set, versioning is suspended, or no replication config; otherwise filter_target_arns -> TargetClient lookup, skipping targets with proxying disabled. - TargetClient gains head_object_for_proxy/get_object (streaming) and the three tagging calls. Proxy calls never send the replication-check SSE-C exemption header; customer SSE-C keys are forwarded verbatim so the target performs real decryption. Conditional (If-*) headers are not forwarded (MinIO parity); Range and part_number are, with parts_count/tag_count/storage_class/expiration passed through. - Metrics: proxy counters now count only real client proxy traffic, MinIO-aligned (one total per proxied request, one failed when no target served it). The previous misattributed counters — replication worker HEAD/PUT (#2672) and local tagging operations (#2682) — are removed; ReplProxyMetric now maps the tagging counters instead of dropping them. e2e (fake_s3_target extended with tagging + header journaling): proxied GET body + outbound header contract (marker present, no replication-check, SSE-C passthrough), HEAD, anti-loop 404 with zero outbound requests, GetObjectTagging, and metric mapping unit tests. Rolling note: proxying only activates for buckets with replication targets; requests carrying the marker keep pre-upgrade behavior. Refs rustfs/backlog#1675 (P1-5) * fix(replication): fail SSE-C passthrough closed on targets that drop transport headers (#6178) SSE-C ciphertext passthrough replicates via X-Rustfs-Replication-* transport headers. A MinIO/generic-S3 target silently discards them, storing bare ciphertext with no decryption material — yet the PUT succeeded, so the object reported COMPLETED with a silently unreadable replica (backlog#1675 N2). Fail-closed design: - SsecPassthroughCapability {Unknown, Supported, Unsupported} cached in BucketTargetSys per target ARN with a recording timestamp. Entries reset whenever the target is rebuilt, edited, or removed (arn_remotes_map lifecycle) and expire after SSEC_PASSTHROUGH_CAPABILITY_TTL (10 minutes): an expired verdict in either direction is re-earned through the audit, so an Unsupported target recovers automatically after an upgrade (at most one wasted PUT+HEAD audit per bad target per TTL window) and a Supported verdict cannot outlive a backend swapped behind the same endpoint. - Replication worker (replicate_object and replicate_all): fresh Unsupported targets never receive the PUT — the attempt fails immediately into the normal MRF retry channel with a "run ?replication-check to re-probe" hint. Unknown or expired verdicts are audited: after the PUT the worker HEADs the replica back through the replication-check channel (source version id mapped through resolve_read_api_version_id, so null-version objects audit correctly) and requires SSE-C evidence (the echoed customer-algorithm header); missing evidence records Unsupported and fails the attempt. Convergence HEADs are audited the same way, so a broken ciphertext replica from an earlier attempt can never launder itself into COMPLETED via an ETag match. The gate/evidence policy is pure (replication_target_boundary, staleness folded in as an input) for the M2 worker migration. - replication-check grows an SsecPassthrough probe phase: a probe PUT carrying the live transport-header shape, HEAD-back for evidence, and a machine-readable Code BucketRemoteSsecPassthroughUnsupported on failure. The probe verdict is synced into the runtime capability cache. Unlike VersionFidelity, a failed SsecPassthrough phase does NOT fail the target overall — it is a capability limit, not a broken replication contract, and a plaintext-only deployment against such a target must not turn red. - fake_s3_target: default mode now models a RustFS target (stores the transport headers, echoes SSE-C evidence); the new drop_unlisted_replication_headers mode models MinIO. The journal records whether a request carried transport headers. Receiver-echo verification: the replication-check HEAD exemption only skips SSE-C key validation; the response has always built sse-customer-algorithm from stored metadata (rustfs/src/app/object_usecase.rs), so no receiver change was needed — pinned end to end by the replication-check e2e against a real RustFS target. Rolling-upgrade constraint: RustFS targets older than the replication-check HEAD exemption (#5898) answer the audit HEAD without SSE-C evidence (or fail it outright), so SSE-C replication to such targets reports FAILED. This is deliberate — FAILED-and-retryable beats a silently undecryptable replica — and self-heals: once the target is upgraded, the next TTL expiry (or a manual ?replication-check re-probe) re-audits and records Supported. Plaintext and managed-SSE replication are unaffected. The capability cache is per-node; each node audits independently. Known limitations: - The audit judges evidence from the echoed customer-algorithm header only. A hypothetical target that preserves that one header while dropping other transport headers (partial-drop) would pass the audit; no known target behaves this way — observed targets drop the whole unknown-header family. - A mixed-version target cluster can flap the verdict between audits routed to different target nodes until the rollout completes; the TTL bounds how long each stale verdict persists. New e2e (backlog#1675 C1 + N2, red-first): fail-closed against a header-dropping fake (FAILED + no second PUT via the capability cache, journal-asserted; red run showed the old COMPLETED), replication-check reports the SsecPassthrough phase Code while the target stays OK overall, SSE-C heal convergence after a real target outage, and SSE-C existing-object resync landing a REPLICA readable with the customer key. TTL expiry in both directions is pinned at the cache and gate seams. * refactor(replication): move resyncer pure decision logic into rustfs-replication (M2) (#6180) * refactor(replication): move resyncer pure decision logic into rustfs-replication (M2) Pure-move milestone M2 of the ECStore replication split (backlog#1675 P1-17): relocate the resyncer's IO-free decision helpers, with their unit tests, into the crates they already belong to by type ownership. No behavior change. Moved into crates/replication: - resync.rs: resync_status_duration - delete.rs: resync_existing_delete_replication_info, replicate_delete_outcome, target_delete_version_id, delete_marker_purge_version_id, delete_marker_purge_mrf_entry - object.rs: version_identity_drifted, is_replication_target_offline_error, SsecPassthroughCapability, SsecPassthroughGate, ssec_passthrough_gate, ssec_passthrough_evidence_present (param-demoted to the echoed customer-algorithm string; ECStore keeps the HeadObjectOutput adapter) - filemeta.rs: NULL_VERSION_ID wire literal (crate-owned copy per the filemeta-independence contract) ECStore rewiring (Rule #14: imports stay in *_boundary.rs): - resync/object-decision/target boundaries re-export the moved symbols; resyncer call sites are unchanged - bucket_target_sys keeps only the verdict cache + TTL and re-exports the capability enum so existing consumer paths keep compiling Not moved (signatures carry ECStore or aws-sdk types): verify_resync_head_result, resync_target_error_detail, the SdkError classifiers, the replicate_all_* option/info builders, and the env-coupled bounded_resync_max_jobs admission clamp. README milestone table updated. * chore(replication): retire the datatypes.rs relay early README sanctions retiring datatypes.rs ahead of M4. The module was a pure relay (resync boundary -> datatypes -> mod.rs facade) with no external consumer importing it directly, so the facade now re-exports ResyncStatusType from replication_resync_boundary and the relay file is deleted. Consumers stay behind the ECStore facade, keeping Migration Rule #15 intact — the original retirement wording ("consumers import through rustfs-replication directly") conflicted with that rule and is corrected in the README. * chore(arch): extend migration guards to the M2-moved decision contracts The adversarial review of the M2 move found the per-symbol ratchet in check_architecture_migration_rules.sh was not extended for the moved symbols, leaving them free to be redefined in ECStore or imported past their boundary without CI noticing: - resync definition pin + boundary fences gain resync_status_duration; - the object-decision boundary fences gain the five delete-family helpers (delete_marker_purge_mrf_entry, delete_marker_purge_version_id, replicate_delete_outcome, resync_existing_delete_replication_info, target_delete_version_id); - the target-boundary fence gains the SSE-C gate family, the offline classifier, and version_identity_drifted; - a new definition pin rejects ECStore redefinitions of the M2-moved fns/enums (ssec_passthrough_evidence_present deliberately excluded: ECStore keeps a thin HeadObjectOutput adapter under that name). Mutation-verified: a probe fn ssec_passthrough_gate under crates/ecstore/src/bucket/replication trips the new pin. Also anchors the intentionally-duplicated NULL_VERSION_ID wire literal from the filemeta side and tightens the M2 README note on bounded_resync_max_jobs.
151 lines
5.4 KiB
Rust
151 lines
5.4 KiB
Rust
// Copyright 2024 RustFS Team
|
|
//
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
//! Proxy-target selection for reads of objects not yet replicated locally
|
|
//! (MinIO `getProxyTargets`, bucket-replication.go).
|
|
//!
|
|
//! During the active-active replication lag window a GET/HEAD/Tagging request
|
|
//! for an object the local site does not have yet may be served by proxying to
|
|
//! a replication target. This module only *selects* the candidate targets; the
|
|
//! request-path callers perform the remote calls and response translation.
|
|
|
|
use std::sync::Arc;
|
|
|
|
use tracing::debug;
|
|
|
|
use super::replication_config_boundary::{ObjectOpts, ReplicationConfigurationExt as _};
|
|
use super::replication_object_config::get_replication_config;
|
|
use super::replication_storage_boundary::ObjectOptions;
|
|
use super::replication_target_boundary::{ReplicationTargetStore, TargetClient};
|
|
|
|
/// Returns the replication-target clients eligible to serve a proxied read of
|
|
/// `bucket/object`, in rule order. Mirrors MinIO's `getProxyTargets`:
|
|
///
|
|
/// - the `source-proxy-request` header family was present at all
|
|
/// (`opts.proxy_request` / `opts.proxy_header_set`, MinIO `ProxyRequest` /
|
|
/// `ProxyHeaderSet`) -> empty. "true" is the anti-loop marker of an
|
|
/// already-proxied client read; "false" is what a peer's replication
|
|
/// worker sends on convergence HEADs so the receiver answers locally —
|
|
/// proxying that miss back would echo the source object and fake
|
|
/// convergence, permanently skipping replication;
|
|
/// - the bucket's versioning is suspended for the object -> empty;
|
|
/// - no replication configuration / no matching rule -> empty;
|
|
/// - otherwise every distinct target ARN whose rules match the object,
|
|
/// resolved through the bucket target system, skipping targets that opted
|
|
/// out of proxying (`disable_proxy`).
|
|
pub async fn get_proxy_targets(bucket: &str, object: &str, opts: &ObjectOptions) -> Vec<Arc<TargetClient>> {
|
|
if opts.proxy_request || opts.proxy_header_set {
|
|
return Vec::new();
|
|
}
|
|
if opts.version_suspended {
|
|
return Vec::new();
|
|
}
|
|
|
|
let cfg = match get_replication_config(bucket).await {
|
|
Ok(Some(cfg)) => cfg,
|
|
Ok(None) => return Vec::new(),
|
|
Err(err) => {
|
|
debug!(bucket, object, error = %err, "read proxy: failed to load replication config; not proxying");
|
|
return Vec::new();
|
|
}
|
|
};
|
|
|
|
let arns = cfg.filter_target_arns(&ObjectOpts {
|
|
name: object.to_string(),
|
|
..Default::default()
|
|
});
|
|
|
|
let mut targets = Vec::with_capacity(arns.len());
|
|
for arn in arns {
|
|
let Some(client) = ReplicationTargetStore::remote_target_client(bucket, &arn).await else {
|
|
debug!(bucket, object, arn, "read proxy: no client for replication target ARN");
|
|
continue;
|
|
};
|
|
if client.disable_proxy {
|
|
continue;
|
|
}
|
|
targets.push(client);
|
|
}
|
|
|
|
targets
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
|
|
fn opts() -> ObjectOptions {
|
|
ObjectOptions::default()
|
|
}
|
|
|
|
/// Anti-loop: a request that was already proxied by a peer must never be
|
|
/// proxied onward, regardless of replication configuration.
|
|
#[tokio::test]
|
|
async fn proxy_request_yields_no_targets() {
|
|
let targets = get_proxy_targets(
|
|
"bucket",
|
|
"object",
|
|
&ObjectOptions {
|
|
proxy_request: true,
|
|
..opts()
|
|
},
|
|
)
|
|
.await;
|
|
assert!(targets.is_empty());
|
|
}
|
|
|
|
/// MinIO `ProxyHeaderSet` parity: the header family being present at all
|
|
/// disables proxying, even with the value "false" — that is what a
|
|
/// peer's replication worker sends on convergence HEADs.
|
|
#[tokio::test]
|
|
async fn proxy_header_set_yields_no_targets() {
|
|
let targets = get_proxy_targets(
|
|
"bucket",
|
|
"object",
|
|
&ObjectOptions {
|
|
proxy_header_set: true,
|
|
proxy_request: false,
|
|
..opts()
|
|
},
|
|
)
|
|
.await;
|
|
assert!(targets.is_empty());
|
|
}
|
|
|
|
/// Suspended versioning disables proxying (MinIO parity): the local null
|
|
/// version is authoritative and a remote read could resurrect data.
|
|
#[tokio::test]
|
|
async fn version_suspended_yields_no_targets() {
|
|
let targets = get_proxy_targets(
|
|
"bucket",
|
|
"object",
|
|
&ObjectOptions {
|
|
version_suspended: true,
|
|
..opts()
|
|
},
|
|
)
|
|
.await;
|
|
assert!(targets.is_empty());
|
|
}
|
|
|
|
/// A bucket without replication configuration has nothing to proxy to.
|
|
/// (No metadata system is running in unit tests, so the config lookup
|
|
/// resolves to "no configuration" — the same empty-result contract.)
|
|
#[tokio::test]
|
|
async fn missing_replication_config_yields_no_targets() {
|
|
let targets = get_proxy_targets("bucket-without-replication", "object", &opts()).await;
|
|
assert!(targets.is_empty());
|
|
}
|
|
}
|