* fix(admin): align replication-reset responses with madmin ResyncTargetsInfo shape
The replication-reset and replication-reset-status responses serialized
their shell as "Targets" and per-target fields in PascalCase, while
madmin-go ResyncTargetsInfo/ResyncTarget expect the "target" shell key
and lowercase field tags (arn/resetid/resyncStatus/replicationCount/
completedReplicationSize/failedReplicationCount/failedReplicationSize).
Go json decoding is case-insensitive per field, but Targets vs target,
Status vs resyncStatus and the size/count key names cannot match, so
mc replicate resync decoded empty results.
Rename the serde tags to the exact madmin wire shape, keep the
ResetBeforeDate/Error RustFS extension keys (unknown keys are ignored
by Go decoders), pin the shape with a snapshot unit test, and update
the e2e client DTO to decode the madmin shape.
* fix(admin): stream bare madmin DiffInfo documents from replication diff
POST /v3/replication/diff returned a single enveloped object
({Entries, IsTruncated, ScannedVersions}) while madmin-go
BucketReplicationDiff decodes the body with a json.Decoder loop over
bare DiffInfo documents. The envelope decoded as exactly one DiffInfo
with an empty object, so mc replicate diff printed a phantom empty row
instead of the real backlog.
Emit one DiffInfo JSON document per line by default, using the exact
madmin json tags (object/versionId/rStatus/deletemarker/lastModified;
Size stays as a RustFS extension key that Go decoders ignore). The
enveloped shape moves to the opt-in ?aggregate=true RustFS extension,
which remains the only carrier of scan-coverage metadata; a truncated
default-mode scan is surfaced via a warn tracing event instead of
in-stream. Pin both shapes with unit tests and tighten the e2e helper
to reject any envelope in the stream.
* feat(replication): validate replication config structure before persisting
PutBucketReplication accepted structurally invalid configurations that
MinIO's replication.Config.Validate rejects: empty or oversized rule
lists, duplicate or negative rule priorities, over-long rule IDs,
filters carrying more than one of Prefix/Tag/And, and delete marker
replication enabled on tag-filtered rules. Such configs persisted
silently and later produced undefined routing (e.g. ambiguous priority
ties) instead of failing the PUT.
Add validate_replication_config_structure as a pure function in
rustfs-replication (limits documented as constants), surface it through
the ecstore api facade, and run it first in the PUT capability gate so
defects are named before any metadata write. Missing Priority counts as
zero for the uniqueness check, matching Go's zero-value semantics. The
self-target rejection deliberately stays at set-remote-target, where the
endpoint is known; a config can never reference a self-pointing ARN.
Document the rule-level Destination.StorageClass contract (use the
remote target's storage_class instead) and renumber the acceptance
matrix e2e to unique priorities, which MinIO would also require.
* test(replication): pin duplicated wire types with boundary reconciliation tests
rustfs-filemeta (xl.meta disk format) and rustfs-replication (MRF/resync
persistence format) deliberately each own ReplicationStatusType,
VersionPurgeStatusType and ReplicationState; the boundary converts
between them via as_str(), whose From<&str> impls fall back to Empty on
unknown tokens — a variant added on one side silently degrades to Empty
on the other.
Add reconciliation tests in replication_filemeta_boundary: exhaustive
matches with no wildcard arm on both sides of both enums (a new variant
fails compilation until the mapping is reconsidered), string-token
round-trip asserts (a token the other side does not recognize fails
instead of quietly becoming Empty), and a full-field ReplicationState
round-trip. Cross-reference the tests from both type definitions.
Struct drift was already compile-guarded by the exhaustive struct
literals in the conversion functions.
* docs(replication): define split completion criteria and milestone sequence
The ecstore replication split plan had no completion measure — the
boundary scaffolding risked ossifying because nothing said when the
migration counts as done. Record the criteria in the module inventory:
done means the Required Contracts table's 'Current dependency to
remove' column is empty; the end state moves pool/resyncer/state into
crates/replication, with the boundary micro-files dissolving as code
crosses the crate line (batch-merging them beforehand is explicitly
rejected — the guard scripts anchor on their file names, so merging is
churn with zero functional gain; only datatypes.rs can retire early).
Sequence the remaining work as M2 (resyncer pure decision logic, after
the oversized function splits) → M3 (worker runtime, highest risk,
last) → M4 (retire boundaries and guard entries). Refresh the stale
first-step text — the event sink / runtime contracts already landed —
and update the split-plan status table accordingly.
* fix(replication): align structural validator with MinIO semantics after adversarial review
Three interop corrections found by adversarial review of the new
structural validator, plus review fallout fixes:
- Delete-marker replication is now rejected only for a direct Filter.Tag,
not for tags inside Filter.And — MinIO's validator only inspects the
direct tag, and mc replicate add --tags "k1=v1&k2=v2" (delete-marker
replication on by default) puts multiple tags into And.Tags, so the
stricter check rejected mc-generated configs MinIO accepts.
- Rule ID length is measured in bytes (Go len semantics), not chars —
a 255-char multibyte ID must not round-trip into a config MinIO
rejects.
- An empty <Tag/> element (no key) counts as absent, matching MinIO's
Tag.IsEmpty(); console form serializers emit empty tags, which would
otherwise trip the exactly-one-of and delete-marker checks.
Also: repair the store-uninitialized PUT test whose empty-rules fixture
now (correctly) fails structural validation before reaching the store
lookup; pin the previously untested startTime madmin key in the
reset-status shape test; and signal a truncated default-mode diff scan
via the x-rustfs-replication-diff-truncated response header — the bare
madmin stream has no envelope, so a truncated scan was otherwise
indistinguishable from a complete healthy one (madmin/mc ignore unknown
headers).
* test(e2e): activate SSE-S3 replication contract and pin resync fail-closed path
The SSE-S3 replication contract e2e was ignored under backlog#1291
(silent plaintext replication); the fail-closed gate in
replication_target_boundary.rs closed that hole, so the ignore reason
expired. Un-ignore the test — it now pins the current fail-closed
contract (FAILED status, failure event, readable encrypted source,
stable absence of all target versions), verified green.
Add test_bucket_replication_sse_s3_resync_stays_fail_closed: drives the
existing-object resync path (PUT ?replication-reset) over a FAILED
SSE-S3 object and asserts the resync generation reaches a terminal
state without ever materializing a target version, with the
stays-absent window also spanning fast-scanner heal cycles. The new
start_bucket_replication_reset helper doubles as the madmin
ResyncTargetsInfo shape assertion (target[0].arn/resetid) for the
reset-start response.
Refresh the stale nextest count commentary (the module is at 20 fast +
36 nightly = 56 tests by cargo nextest list; the SSE-S3-ignored note no
longer holds).
Introduce RUSTFS_REPL_HEALTH_CHECK_INTERVAL_MS, RUSTFS_REPL_MRF_FLUSH_INTERVAL_MS
and RUSTFS_REPL_RESYNC_POLL_MAX_MS so tests and operators can shorten the
replication background loops. Defaults are unchanged; invalid values fall
back with a warn and values below 10ms are clamped to avoid busy-spin.
Add replication_fast_env() e2e helper (backlog#1147 repl-4).
This commit introduces a significant reorganization of the project structure to improve maintainability and clarity.
Key changes include:
- Adjusted the directory layout for a more logical module organization.
- Removed unused crate dependencies, reducing the overall project size and potentially speeding up build times.
- Updated import paths and configuration files to reflect the structural changes.
唐小鸭