fix(replication): madmin reset/diff wire compat and config validation (#5799)

* fix(admin): align replication-reset responses with madmin ResyncTargetsInfo shape

The replication-reset and replication-reset-status responses serialized
their shell as "Targets" and per-target fields in PascalCase, while
madmin-go ResyncTargetsInfo/ResyncTarget expect the "target" shell key
and lowercase field tags (arn/resetid/resyncStatus/replicationCount/
completedReplicationSize/failedReplicationCount/failedReplicationSize).
Go json decoding is case-insensitive per field, but Targets vs target,
Status vs resyncStatus and the size/count key names cannot match, so
mc replicate resync decoded empty results.

Rename the serde tags to the exact madmin wire shape, keep the
ResetBeforeDate/Error RustFS extension keys (unknown keys are ignored
by Go decoders), pin the shape with a snapshot unit test, and update
the e2e client DTO to decode the madmin shape.

* fix(admin): stream bare madmin DiffInfo documents from replication diff

POST /v3/replication/diff returned a single enveloped object
({Entries, IsTruncated, ScannedVersions}) while madmin-go
BucketReplicationDiff decodes the body with a json.Decoder loop over
bare DiffInfo documents. The envelope decoded as exactly one DiffInfo
with an empty object, so mc replicate diff printed a phantom empty row
instead of the real backlog.

Emit one DiffInfo JSON document per line by default, using the exact
madmin json tags (object/versionId/rStatus/deletemarker/lastModified;
Size stays as a RustFS extension key that Go decoders ignore). The
enveloped shape moves to the opt-in ?aggregate=true RustFS extension,
which remains the only carrier of scan-coverage metadata; a truncated
default-mode scan is surfaced via a warn tracing event instead of
in-stream. Pin both shapes with unit tests and tighten the e2e helper
to reject any envelope in the stream.

* feat(replication): validate replication config structure before persisting

PutBucketReplication accepted structurally invalid configurations that
MinIO's replication.Config.Validate rejects: empty or oversized rule
lists, duplicate or negative rule priorities, over-long rule IDs,
filters carrying more than one of Prefix/Tag/And, and delete marker
replication enabled on tag-filtered rules. Such configs persisted
silently and later produced undefined routing (e.g. ambiguous priority
ties) instead of failing the PUT.

Add validate_replication_config_structure as a pure function in
rustfs-replication (limits documented as constants), surface it through
the ecstore api facade, and run it first in the PUT capability gate so
defects are named before any metadata write. Missing Priority counts as
zero for the uniqueness check, matching Go's zero-value semantics. The
self-target rejection deliberately stays at set-remote-target, where the
endpoint is known; a config can never reference a self-pointing ARN.
Document the rule-level Destination.StorageClass contract (use the
remote target's storage_class instead) and renumber the acceptance
matrix e2e to unique priorities, which MinIO would also require.

* test(replication): pin duplicated wire types with boundary reconciliation tests

rustfs-filemeta (xl.meta disk format) and rustfs-replication (MRF/resync
persistence format) deliberately each own ReplicationStatusType,
VersionPurgeStatusType and ReplicationState; the boundary converts
between them via as_str(), whose From<&str> impls fall back to Empty on
unknown tokens — a variant added on one side silently degrades to Empty
on the other.

Add reconciliation tests in replication_filemeta_boundary: exhaustive
matches with no wildcard arm on both sides of both enums (a new variant
fails compilation until the mapping is reconsidered), string-token
round-trip asserts (a token the other side does not recognize fails
instead of quietly becoming Empty), and a full-field ReplicationState
round-trip. Cross-reference the tests from both type definitions.
Struct drift was already compile-guarded by the exhaustive struct
literals in the conversion functions.

* docs(replication): define split completion criteria and milestone sequence

The ecstore replication split plan had no completion measure — the
boundary scaffolding risked ossifying because nothing said when the
migration counts as done. Record the criteria in the module inventory:
done means the Required Contracts table's 'Current dependency to
remove' column is empty; the end state moves pool/resyncer/state into
crates/replication, with the boundary micro-files dissolving as code
crosses the crate line (batch-merging them beforehand is explicitly
rejected — the guard scripts anchor on their file names, so merging is
churn with zero functional gain; only datatypes.rs can retire early).

Sequence the remaining work as M2 (resyncer pure decision logic, after
the oversized function splits) → M3 (worker runtime, highest risk,
last) → M4 (retire boundaries and guard entries). Refresh the stale
first-step text — the event sink / runtime contracts already landed —
and update the split-plan status table accordingly.

* fix(replication): align structural validator with MinIO semantics after adversarial review

Three interop corrections found by adversarial review of the new
structural validator, plus review fallout fixes:

- Delete-marker replication is now rejected only for a direct Filter.Tag,
  not for tags inside Filter.And — MinIO's validator only inspects the
  direct tag, and mc replicate add --tags "k1=v1&k2=v2" (delete-marker
  replication on by default) puts multiple tags into And.Tags, so the
  stricter check rejected mc-generated configs MinIO accepts.
- Rule ID length is measured in bytes (Go len semantics), not chars —
  a 255-char multibyte ID must not round-trip into a config MinIO
  rejects.
- An empty <Tag/> element (no key) counts as absent, matching MinIO's
  Tag.IsEmpty(); console form serializers emit empty tags, which would
  otherwise trip the exactly-one-of and delete-marker checks.

Also: repair the store-uninitialized PUT test whose empty-rules fixture
now (correctly) fails structural validation before reaching the store
lookup; pin the previously untested startTime madmin key in the
reset-status shape test; and signal a truncated default-mode diff scan
via the x-rustfs-replication-diff-truncated response header — the bare
madmin stream has no envelope, so a truncated scan was otherwise
indistinguishable from a complete healthy one (madmin/mc ignore unknown
headers).

* test(e2e): activate SSE-S3 replication contract and pin resync fail-closed path

The SSE-S3 replication contract e2e was ignored under backlog#1291
(silent plaintext replication); the fail-closed gate in
replication_target_boundary.rs closed that hole, so the ignore reason
expired. Un-ignore the test — it now pins the current fail-closed
contract (FAILED status, failure event, readable encrypted source,
stable absence of all target versions), verified green.

Add test_bucket_replication_sse_s3_resync_stays_fail_closed: drives the
existing-object resync path (PUT ?replication-reset) over a FAILED
SSE-S3 object and asserts the resync generation reaches a terminal
state without ever materializing a target version, with the
stays-absent window also spanning fast-scanner heal cycles. The new
start_bucket_replication_reset helper doubles as the madmin
ResyncTargetsInfo shape assertion (target[0].arn/resetid) for the
reset-start response.

Refresh the stale nextest count commentary (the module is at 20 fast +
36 nightly = 56 tests by cargo nextest list; the SSE-S3-ignored note no
longer holds).
This commit is contained in:
唐小鸭
2026-08-07 22:30:12 +08:00
committed by GitHub
parent 7553715f62
commit 3792fed827
16 changed files with 861 additions and 97 deletions
+315
View File
@@ -32,6 +32,12 @@ pub const REPLICATION_CAPABILITY_CONTRACT_VERSION: u32 = 1;
// clients should keep omitting it, but the validator tolerates an explicit
// `STANDARD` as a no-op (see `unsupported_replication_config_field`) because the
// console's rule form always sends it.
//
// Contract note: rule-level `Destination.StorageClass` is never consumed by the
// replication engine (MinIO's engine likewise reads only the target-level
// storage class). To control the storage class of replicated objects, set the
// remote target's `storage_class` field (set-remote-target API), which RustFS
// does apply on replication PUTs.
pub const REPLICATION_WRITABLE_FIELDS: &[&str] = &[
"Role",
"Rule.ID",
@@ -297,6 +303,119 @@ pub fn should_remove_replication_target(
is_replication_target && config_target_arns.contains(target_arn)
}
/// Maximum number of rules accepted in one replication configuration,
/// matching MinIO's `replication.Config.Validate` limit.
pub const REPLICATION_CONFIG_MAX_RULES: usize = 1000;
/// Maximum length of a replication rule ID, matching the S3 schema.
pub const REPLICATION_CONFIG_MAX_RULE_ID_LEN: usize = 255;
/// A structural defect in a replication configuration, detected before the
/// configuration is persisted.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ReplicationConfigStructureError {
NoRules,
TooManyRules,
NegativeRulePriority,
DuplicateRulePriority,
RuleIdTooLong,
AmbiguousRuleFilter,
TagFilterWithDeleteMarkerReplication,
}
impl ReplicationConfigStructureError {
pub fn message(self) -> &'static str {
match self {
Self::NoRules => "replication configuration must contain at least one rule",
Self::TooManyRules => "replication configuration cannot contain more than 1000 rules",
Self::NegativeRulePriority => "replication rule Priority must be zero or a positive integer",
Self::DuplicateRulePriority => "replication rule Priority must be unique across rules",
Self::RuleIdTooLong => "replication rule ID cannot be longer than 255 characters",
Self::AmbiguousRuleFilter => "replication rule Filter must specify only one of Prefix, Tag or And",
Self::TagFilterWithDeleteMarkerReplication => {
"delete marker replication cannot be enabled on a rule with a Tag filter"
}
}
}
}
fn filter_and_operator_is_set(and: &s3s::dto::ReplicationRuleAndOperator) -> bool {
and.prefix.as_ref().is_some_and(|prefix| !prefix.is_empty()) || and.tags.as_ref().is_some_and(|tags| !tags.is_empty())
}
/// Structural validation of a replication configuration, mirroring the checks
/// MinIO's `replication.Config.Validate` performs before persisting: at least
/// one rule, at most [`REPLICATION_CONFIG_MAX_RULES`], non-negative and unique
/// per-rule priorities (a missing Priority counts as 0, like Go's zero value),
/// rule IDs within [`REPLICATION_CONFIG_MAX_RULE_ID_LEN`] bytes, a Filter
/// carrying only one of Prefix/Tag/And, and delete marker replication
/// disabled on rules with a direct `Filter.Tag`. Tags inside `Filter.And` do
/// NOT trigger the delete-marker check — MinIO only inspects the direct tag,
/// and `mc replicate add --tags "k1=v1&k2=v2"` (delete-marker replication on
/// by default) puts multiple tags into `And.Tags`, so rejecting that shape
/// would break mc-generated configs that MinIO accepts.
///
/// This is shape-only validation: capability gating lives in
/// [`unsupported_replication_config_field`]/[`invalid_replication_config_status_field`],
/// and the self-target ("same target") rejection is enforced when the remote
/// target itself is created, so a config can never reference a self-pointing
/// ARN.
pub fn validate_replication_config_structure(
config: &ReplicationConfiguration,
) -> std::result::Result<(), ReplicationConfigStructureError> {
if config.rules.is_empty() {
return Err(ReplicationConfigStructureError::NoRules);
}
if config.rules.len() > REPLICATION_CONFIG_MAX_RULES {
return Err(ReplicationConfigStructureError::TooManyRules);
}
let mut priorities = HashSet::new();
for rule in &config.rules {
let priority = rule.priority.unwrap_or(0);
if priority < 0 {
return Err(ReplicationConfigStructureError::NegativeRulePriority);
}
if !priorities.insert(priority) {
return Err(ReplicationConfigStructureError::DuplicateRulePriority);
}
// Byte length, matching Go's `len(r.ID) > 255` in MinIO.
if rule
.id
.as_ref()
.is_some_and(|id| id.len() > REPLICATION_CONFIG_MAX_RULE_ID_LEN)
{
return Err(ReplicationConfigStructureError::RuleIdTooLong);
}
if let Some(filter) = &rule.filter {
let has_and = filter.and.as_ref().is_some_and(filter_and_operator_is_set);
let has_prefix = filter.prefix.as_ref().is_some_and(|prefix| !prefix.is_empty());
// An empty <Tag/> element (no key) counts as absent, matching
// MinIO's Tag.IsEmpty(); console form serializers emit empty tags.
let has_tag = filter
.tag
.as_ref()
.is_some_and(|tag| tag.key.as_ref().is_some_and(|key| !key.is_empty()));
if usize::from(has_and) + usize::from(has_prefix) + usize::from(has_tag) > 1 {
return Err(ReplicationConfigStructureError::AmbiguousRuleFilter);
}
let delete_marker_replication_enabled = rule
.delete_marker_replication
.as_ref()
.and_then(|delete_marker| delete_marker.status.as_ref())
.is_some_and(|status| status.as_str() == DeleteMarkerReplicationStatus::ENABLED);
if delete_marker_replication_enabled && has_tag {
return Err(ReplicationConfigStructureError::TagFilterWithDeleteMarkerReplication);
}
}
}
Ok(())
}
impl ReplicationConfigurationExt for ReplicationConfiguration {
/// Check whether any object-replication rules exist
fn has_existing_object_replication(&self, arn: &str) -> (bool, bool) {
@@ -565,6 +684,202 @@ mod tests {
}
}
fn structure_config(rules: Vec<ReplicationRule>) -> ReplicationConfiguration {
ReplicationConfiguration {
role: String::new(),
rules,
}
}
fn tag_filter() -> s3s::dto::ReplicationRuleFilter {
s3s::dto::ReplicationRuleFilter {
tag: Some(s3s::dto::Tag {
key: Some("k".to_string()),
value: Some("v".to_string()),
}),
..Default::default()
}
}
#[test]
fn structure_validation_accepts_multi_rule_config_with_unique_priorities() {
let mut second = replication_rule("rule-2", "arn:target:a");
second.priority = Some(2);
second.filter = Some(s3s::dto::ReplicationRuleFilter {
and: Some(s3s::dto::ReplicationRuleAndOperator {
prefix: Some("photos/".to_string()),
tags: Some(vec![s3s::dto::Tag {
key: Some("k".to_string()),
value: Some("v".to_string()),
}]),
}),
..Default::default()
});
let config = structure_config(vec![replication_rule("rule-1", "arn:target:a"), second]);
assert_eq!(validate_replication_config_structure(&config), Ok(()));
}
#[test]
fn structure_validation_rejects_empty_rule_list() {
let config = structure_config(Vec::new());
assert_eq!(
validate_replication_config_structure(&config),
Err(ReplicationConfigStructureError::NoRules)
);
}
#[test]
fn structure_validation_rejects_more_than_max_rules() {
let rules = (0..=REPLICATION_CONFIG_MAX_RULES as i32)
.map(|priority| {
let mut rule = replication_rule(&format!("rule-{priority}"), "arn:target:a");
rule.priority = Some(priority);
rule
})
.collect();
assert_eq!(
validate_replication_config_structure(&structure_config(rules)),
Err(ReplicationConfigStructureError::TooManyRules)
);
}
#[test]
fn structure_validation_rejects_duplicate_priorities() {
let config = structure_config(vec![
replication_rule("rule-1", "arn:target:a"),
replication_rule("rule-2", "arn:target:a"),
]);
assert_eq!(
validate_replication_config_structure(&config),
Err(ReplicationConfigStructureError::DuplicateRulePriority)
);
}
#[test]
fn structure_validation_treats_missing_priority_as_zero_for_uniqueness() {
let mut first = replication_rule("rule-1", "arn:target:a");
first.priority = None;
let mut second = replication_rule("rule-2", "arn:target:a");
second.priority = None;
assert_eq!(
validate_replication_config_structure(&structure_config(vec![first, second])),
Err(ReplicationConfigStructureError::DuplicateRulePriority)
);
}
#[test]
fn structure_validation_rejects_negative_priority() {
let mut rule = replication_rule("rule-1", "arn:target:a");
rule.priority = Some(-1);
assert_eq!(
validate_replication_config_structure(&structure_config(vec![rule])),
Err(ReplicationConfigStructureError::NegativeRulePriority)
);
}
#[test]
fn structure_validation_rejects_rule_id_longer_than_255_chars() {
let mut rule = replication_rule(&"x".repeat(REPLICATION_CONFIG_MAX_RULE_ID_LEN + 1), "arn:target:a");
rule.priority = Some(1);
assert_eq!(
validate_replication_config_structure(&structure_config(vec![rule])),
Err(ReplicationConfigStructureError::RuleIdTooLong)
);
}
#[test]
fn structure_validation_rejects_filter_with_both_prefix_and_tag() {
let mut rule = replication_rule("rule-1", "arn:target:a");
let mut filter = tag_filter();
filter.prefix = Some("photos/".to_string());
rule.filter = Some(filter);
assert_eq!(
validate_replication_config_structure(&structure_config(vec![rule])),
Err(ReplicationConfigStructureError::AmbiguousRuleFilter)
);
}
#[test]
fn structure_validation_rejects_delete_marker_replication_on_tag_filtered_rule() {
let mut rule = replication_rule("rule-1", "arn:target:a");
rule.delete_marker_replication = Some(DeleteMarkerReplication {
status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)),
});
rule.filter = Some(tag_filter());
assert_eq!(
validate_replication_config_structure(&structure_config(vec![rule])),
Err(ReplicationConfigStructureError::TagFilterWithDeleteMarkerReplication)
);
}
#[test]
fn structure_validation_treats_empty_tag_element_as_absent() {
// MinIO's Tag.IsEmpty() ignores an empty <Tag/> element; the console's
// form serializer emits them, so prefix + empty tag must stay valid
// and an empty tag must not trip the delete-marker check.
let mut rule = replication_rule("rule-1", "arn:target:a");
rule.delete_marker_replication = Some(DeleteMarkerReplication {
status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)),
});
rule.filter = Some(s3s::dto::ReplicationRuleFilter {
prefix: Some("photos/".to_string()),
tag: Some(s3s::dto::Tag { key: None, value: None }),
..Default::default()
});
assert_eq!(validate_replication_config_structure(&structure_config(vec![rule])), Ok(()));
}
#[test]
fn structure_validation_allows_delete_marker_replication_with_and_tags() {
// mc `replicate add --tags "k1=v1&k2=v2"` puts multiple tags into
// Filter.And.Tags and enables delete-marker replication by default;
// MinIO's validator only inspects the direct Filter.Tag, so this
// shape must stay accepted for mc interop.
let mut rule = replication_rule("rule-1", "arn:target:a");
rule.delete_marker_replication = Some(DeleteMarkerReplication {
status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)),
});
rule.filter = Some(s3s::dto::ReplicationRuleFilter {
and: Some(s3s::dto::ReplicationRuleAndOperator {
prefix: None,
tags: Some(vec![
s3s::dto::Tag {
key: Some("k1".to_string()),
value: Some("v1".to_string()),
},
s3s::dto::Tag {
key: Some("k2".to_string()),
value: Some("v2".to_string()),
},
]),
}),
..Default::default()
});
assert_eq!(validate_replication_config_structure(&structure_config(vec![rule])), Ok(()));
}
#[test]
fn structure_validation_allows_tag_filter_when_delete_marker_replication_disabled() {
let mut rule = replication_rule("rule-1", "arn:target:a");
rule.delete_marker_replication = Some(DeleteMarkerReplication {
status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::DISABLED)),
});
rule.filter = Some(tag_filter());
assert_eq!(validate_replication_config_structure(&structure_config(vec![rule])), Ok(()));
}
#[test]
fn filter_target_arns_uses_role_when_role_is_present() {
let config = ReplicationConfiguration {
+8
View File
@@ -48,6 +48,14 @@ pub const REPLICATE_HEAL: &str = "replicate:heal";
pub const REPLICATE_HEAL_DELETE: &str = "replicate:heal:delete";
/// StatusType of Replication for x-amz-replication-status header
///
/// NOTE: `rustfs-filemeta` owns a sibling copy of this enum (plus
/// `VersionPurgeStatusType` and `ReplicationState`) bound to the xl.meta disk
/// format, while this copy is bound to the MRF/resync persistence format.
/// When adding or renaming a variant here, reconcile the sibling and the
/// conversion layer — the reconciliation tests in
/// `crates/ecstore/src/bucket/replication/replication_filemeta_boundary.rs`
/// fail to compile until both sides agree.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, Default, Hash)]
pub enum ReplicationStatusType {
/// Pending - replication is pending.
+4 -3
View File
@@ -31,9 +31,10 @@ pub mod tagging;
pub use config::{
ObjectOpts, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION, REMOTE_TARGET_UNSUPPORTED_FIELDS, REMOTE_TARGET_WRITABLE_FIELDS,
REPLICATION_CAPABILITY_CONTRACT_VERSION, REPLICATION_READ_ONLY_HISTORICAL_FIELDS, REPLICATION_WRITABLE_FIELDS,
ReplicationConfigurationExt, ReplicationTargetValidationError, active_replication_rule_destination_arns,
invalid_replication_config_status_field, replication_target_arns, should_remove_replication_target,
unsupported_replication_config_field, validate_replication_config_target_arns,
ReplicationConfigStructureError, ReplicationConfigurationExt, ReplicationTargetValidationError,
active_replication_rule_destination_arns, invalid_replication_config_status_field, replication_target_arns,
should_remove_replication_target, unsupported_replication_config_field, validate_replication_config_structure,
validate_replication_config_target_arns,
};
pub use delete::{
DeletedObjectReplicationInfo, is_retryable_delete_replication_head_error, is_version_delete_replication,