* test(heal): add privileged replacement rebuild e2e
Add ignored Linux-only 3x4 automatic replacement coverage for EC8+4 and EC6+6. The tests use real tmpfs mounts in an isolated mount namespace, wait for scanner-driven replacement recovery status, and verify the replacement target with per-version xl.meta and part.N physical census without invoking Admin deep heal.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(test): avoid unsafe in privileged replacement e2e
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(heal): harden privileged replacement e2e
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(heal): prove absent replacement recovery witness
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): prove absent replacement observation
Stop the target node before detaching the test mount so RustFS releases its mount lease instead of continuing to serve the old tmpfs through an open fd. Restart the node with the endpoint absent and wait for the scanner's real readiness rejection in that node's log.
Assert the absent window has no replacement intent, completion proof, checkpoint, healing marker, or Admin v4 durable record for the target before mounting the blank replacement and waiting for automatic recovery plus physical shard census.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): streamline cluster log capture
Move cluster-node log capture out of ClusterNode and into per-node cluster launch configuration so the privileged replacement E2E uses an explicit harness API instead of mutating node identity data.
Reuse the same stdout/stderr capture helper for single-node and cluster processes, and pin the per-node capture behavior with a focused common test.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): harden privileged replacement proof
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
P1-20 (rustfs/backlog#1675 B2, test-only). No prior test wrote objects
BEFORE the replication rule arrived, leaving the scanner's existing-object
resync pass — the only channel for such objects — without end-to-end
coverage, and the enqueue truth table partially unpinned at unit level.
e2e (both negative cells are contracts, asserted over multiple fast-scanner
cycles next to a replicated control key that proves the scanner and the
live path are running):
- test_scanner_compensates_existing_objects_across_write_paths: plain PUT,
CopyObject and Snowball auto-extract products written pre-rule all
converge via scanner compensation; a null-version object (PUT before the
bucket became versioned) is pinned as never compensated (the scanner heal
gate skips nil-version objects).
- test_scanner_never_compensates_when_existing_object_replication_disabled:
ExistingObjectReplication=Disabled is a contract, not a delay — existing
keys stay absent while post-rule writes replicate normally.
Unit truth-table pins (crates/replication):
- queue.rs: an empty replicate decision (Disabled existing-object, inbound
REPLICA) skips heal queueing for every status; Completed without a resync
decision skips.
- operation.rs: existing-object resync without a reset replicates exactly
the never-replicated (Empty) objects.
Helper: put_bucket_replication_with_statuses parameterizes the previously
hardcoded ExistingObjectReplication status; the nextest count comments are
refreshed to the post-rebase totals.
* test(replication): pin the version-fidelity probe contract (red)
P1-19 (rustfs/backlog#1675 B2): the supported replication contract is
targets that adopt the source version id — a target that mints its own ids
silently breaks every version-addressed operation that follows (version
deletes, heal re-drives never match), diverging the two sides with no
signal. replication-check already captures the probe PUT's response version
id but never compares it.
Red evidence (current main): against a FakeS3Target with
assign_own_version_ids enabled, ?replication-check returns Status "OK" —
the drift is invisible.
test_replication_check_flags_version_minting_target expects a
VersionFidelity phase that fails with the machine-readable code
BucketRemoteTargetVersionMismatch, skips the later mutation phases, and
still cleans up the probe via the version id the target actually assigned.
Test infra: FakeS3Target gains assign_own_version_ids (models a generic S3
service; validated-but-not-mirrored source version headers) and a
prefix+max-keys ListObjectVersions implementation (the probe key allocation
requires it); stored_versions accessor duplicated from the P1-21 branch
(identical code, resolves clean on merge).
* fix(replication): probe the version-identity contract in replication-check
P1-19 (rustfs/backlog#1675 B2, plan B). Replication only converges on
targets that adopt the source version id: version-addressed deletes and
heal re-drives address the source id, so a target that mints its own ids
silently diverges — nothing surfaced this. replication-check already
captured the probe PUT's response version id but never compared it.
- The probe PUT now carries the source version as `?versionId=` (the exact
shape live replication uses since P0-5, and the only shape MinIO
consumes; the internal source-version-id header alone would let the
probe pass against targets the real data path drifts on). Reuses
ecstore's append_version_id_query through the api facade.
- New VersionFidelity phase: the probe PUT's response version id must
equal the sent source id. On mismatch the phase fails with the
machine-readable extension key `"Code": "BucketRemoteTargetVersionMismatch"`
(new optional Code field on phase statuses; Go decoders ignore unknown
keys), the overall target fails, the later version-addressed mutation
phases are skipped, and cleanup still removes the probe via the id the
target actually assigned (with the existing list-based sweep as backstop
when the target returns no version id at all).
- Runtime half: TargetClient::put_object now returns the assigned version
id (mirroring remove_object), and the replication PUT path audits it —
every drifting PUT increments
rustfs_replication_version_identity_drift_total and the first drift per
target ARN logs a structured warning pointing at ?replication-check.
The drift judgment is a pure function with an exemption-matrix test
(empty / literal "null" / nil-uuid sources carry no contract).
- docs/operations/replication-check.md documents the phase and the code.
Red -> green: test_replication_check_flags_version_minting_target (fake
target with assign_own_version_ids; on main the check reported Status
"OK"). The probe's query shape is pinned by a journal assertion (revert
of the query hunk alone fails it), probe-level unit tests cover the
mismatch/mirror matrix including cleanup addressing the minted id, and
the existing success e2e now asserts VersionFidelity OK against a RustFS
target. Adversarial review (seven roles): non-blocking; noted follow-ups
are the multipart runtime audit (the probe phase already pins the
contract) and per-target re-warning after reconfiguration.
* fix(e2e): stop the fake target self-deadlocking on version-id minting
The assign_own_version_ids flag was read with a fresh `lock(&self.store)`
inside two paths that already hold that guard — delete_object's
marker-creation branch and create_multipart_upload — and the store mutex
is not reentrant, so both hung forever (CI: the fake target's own
multipart and delete-marker tests ran >1560s until the job was
cancelled). Read the flag from the live guard instead.
The replication e2e paths did not catch this: a version-addressed purge
DELETE never mints an id, and the probe PUT reads the flag before taking
the guard.
* chore(test): refresh the nextest replication count invariant
The e2e-smoke/e2e-repl-nightly split comment is descriptive metadata
(authority: `cargo nextest list`); refresh it to this branch's
post-rebase total.
* test(replication): pin delayed delete-marker purge failure handling (red)
P1-21 (rustfs/backlog#1675 B2): two failing e2e tests that pin the missing
failure handling of the delayed delete-marker purge:
- test_delayed_delete_marker_purge_retries_after_transient_target_failure:
four scripted 503s outlast every existing channel (version-purge
replication + its in-process MRF fast retries + the watcher's single
attempt = 3 target DELETEs, all faulted in the recorded run); the
replicated marker is stranded on the target forever.
- test_delayed_delete_marker_purge_exhaustion_persists_to_mrf_and_replays_on_restart:
exhausted purge intents never reach the durable MRF journal, so a restart
replays nothing (recorded run: 3 faulted attempts, zero post-restart).
Red-light evidence (current main):
- Test A: FAILED, journal shows 3x DeleteObject fault=Status(503), no clean
attempt, target marker still present after 15s.
- Test B: FAILED after 468s, same 3 faulted attempts, no purge DELETE after
restart, marker still present.
Test infra: FakeS3Target::stored_versions() exposes per-key version state so
purge tests assert target state instead of inferring it from the journal;
nextest count comments 36->38 nightly / 56->58 total.
* fix(replication): retry, persist and replay failed delete-marker purges
P1-21 (rustfs/backlog#1675 B2). The delayed delete-marker purge was
fire-and-forget: the target DELETE discarded its result (`let _ =`), a
missing target client was silently skipped, and nothing recorded the intent
— one transient target error stranded the replicated marker on the target
forever. Separately, `replicate_delete_with_outcome` held its outcome
hostage to `!requires_delayed_purge`, pinning every delete-marker MRF entry
to Missed so the durable backlog retained them permanently.
Changes:
- `replicate_delete_marker_purge_to_targets` now reports per-target
results (warn + metrics on failure, including `target_client_missing`),
supports retrying only the failed targets, and treats a target-side
NoSuchKey/NoSuchVersion as purge success (strict-404 targets must not
retain the intent forever).
- The delayed watcher (`watch_and_purge_source_delete_marker`) retries
failed targets across its 5x1s watch window; on exhaustion it persists
the purge intent to the durable MRF journal via the new
`ReplicationPoolTrait::persist_mrf_entry` (journal-only on purpose: live
re-dispatch would loop unboundedly against a down target). Intent entries
are shaped as marker-creation deletes so replay funnels into the stale-
marker branch.
- The stale-marker branch (source marker already gone) now purges the
targets instead of silently returning success — closing a latent leak —
and reports the purge result as the replay outcome. Heal callers retry
for the full window (the startup MRF processor runs before target
clients initialize); live callers attempt once and fall back to a fresh
durable intent, so a down target cannot pin a replication worker.
- The outcome formula (extracted as `replicate_delete_outcome` and pinned
by a unit test) no longer includes the delayed purge, so successfully
replayed delete-marker entries are acknowledged instead of retained
forever.
Verification: red -> green e2e pair (transient-failure retry; exhaustion ->
durable MRF -> restart replay -> second-restart zero-replay ack) plus unit
tests; `make pre-commit`, logging guardrails, clippy (ecstore + e2e_test)
all clean; full ecstore lib suite 3729 passed (3 pre-existing local-DNS
kubernetes endpoint failures reproduce without this change).
Adversarial validation (7 roles): no blocking findings after adding the
outcome-formula guard test. Known residuals recorded in the PR: watcher
shutdown window (intent not yet persisted), rolling-downgrade replay acks
without purging (equals pre-fix behavior), and replay falling back to the
source version id on targets that mint their own version ids (P1-19).
* chore(test): refresh the nextest replication count invariant
The e2e-smoke/e2e-repl-nightly split comment is descriptive metadata
(authority: `cargo nextest list`); refresh it to this branch's
post-rebase total.
* fix(replication): purge the marker version the target actually assigned
Review follow-up (#5864), two real defects:
- The delayed purge watcher was spawned with the pre-merge `dobj`, so the
per-target marker version ids this round recorded were invisible to it.
Against a target that mints its own ids the purge fell back to a
source-derived id, the target answered the versioned DELETE with an
idempotent 204, and that "success" cleared the retry set while the real
marker stayed behind. The watcher now receives the merged replication
state (`drs`), which folds this round's target-assigned ids in.
- A target whose recorded version metadata is inconsistent was skipped
without entering `failed_arns`, so an empty result made both the watcher
and the MRF replay treat a purge that issued no DELETE as successful and
drop the intent. The refusal is now a per-target failure (own metric
label): the leak stays visible and the intent is retained instead of
being acknowledged. The version decision also moved ahead of the client
lookup, so the refusal is decided from metadata alone.
Tests: a new e2e drives a fake target with `assign_own_version_ids`, which
ignores the forwarded source-version header for both objects and delete
markers, and asserts the replicated marker is really gone; a unit test
pins the corrupt-metadata refusal as a failed outcome without any target
client registered. The detached-watcher shutdown window is documented at
the watcher as a known non-durable window with the write-ahead follow-up
spelled out.
Open the managed-SSE replication gate (backlog#1783, PR-B of 3, after
#5872): the replication reader already decrypts through the injected
object-encryption resolver, so the source sends plaintext plus an
encryption intent header (AES256 / aws:kms, never the source key id) and
the target re-encrypts on its normal PUT path with its own KMS. No DEK
crosses sites.
- replication_put_object_options: fail closed only on Unsupported;
insert the SSE intent after the strip loop.
- TargetClient::create_multipart_upload sends the full opts.header()
set, fixing multipart replicas losing content-type/user metadata
(plaintext included).
- Preserve source ETag and mtime on replicas (authorized replication
only): receiver wires x-rustfs-source-etag into preserve_etag for PUT
and CompleteMultipartUpload, resolve_complete_etag consumes it, and
complete options carry source_etag/source_mtime (absent mtime
degrades to epoch, not now_utc). Without this every replication HEAD
comparison re-drives re-encrypted objects forever.
- e2e: managed SSE contracts flip to success on an independent-KMS
dual-process pair (byte-identical plain GET proves target-owned
envelopes; ETag/mtime preserved; version stable across scanner
cycles; resync converges; multipart keeps structure and metadata);
new target-without-KMS fail-closed contract; SSE-C stays FAILED.
Co-authored-by: houseme <housemecn@gmail.com>
* fix(admin): align replication-reset responses with madmin ResyncTargetsInfo shape
The replication-reset and replication-reset-status responses serialized
their shell as "Targets" and per-target fields in PascalCase, while
madmin-go ResyncTargetsInfo/ResyncTarget expect the "target" shell key
and lowercase field tags (arn/resetid/resyncStatus/replicationCount/
completedReplicationSize/failedReplicationCount/failedReplicationSize).
Go json decoding is case-insensitive per field, but Targets vs target,
Status vs resyncStatus and the size/count key names cannot match, so
mc replicate resync decoded empty results.
Rename the serde tags to the exact madmin wire shape, keep the
ResetBeforeDate/Error RustFS extension keys (unknown keys are ignored
by Go decoders), pin the shape with a snapshot unit test, and update
the e2e client DTO to decode the madmin shape.
* fix(admin): stream bare madmin DiffInfo documents from replication diff
POST /v3/replication/diff returned a single enveloped object
({Entries, IsTruncated, ScannedVersions}) while madmin-go
BucketReplicationDiff decodes the body with a json.Decoder loop over
bare DiffInfo documents. The envelope decoded as exactly one DiffInfo
with an empty object, so mc replicate diff printed a phantom empty row
instead of the real backlog.
Emit one DiffInfo JSON document per line by default, using the exact
madmin json tags (object/versionId/rStatus/deletemarker/lastModified;
Size stays as a RustFS extension key that Go decoders ignore). The
enveloped shape moves to the opt-in ?aggregate=true RustFS extension,
which remains the only carrier of scan-coverage metadata; a truncated
default-mode scan is surfaced via a warn tracing event instead of
in-stream. Pin both shapes with unit tests and tighten the e2e helper
to reject any envelope in the stream.
* feat(replication): validate replication config structure before persisting
PutBucketReplication accepted structurally invalid configurations that
MinIO's replication.Config.Validate rejects: empty or oversized rule
lists, duplicate or negative rule priorities, over-long rule IDs,
filters carrying more than one of Prefix/Tag/And, and delete marker
replication enabled on tag-filtered rules. Such configs persisted
silently and later produced undefined routing (e.g. ambiguous priority
ties) instead of failing the PUT.
Add validate_replication_config_structure as a pure function in
rustfs-replication (limits documented as constants), surface it through
the ecstore api facade, and run it first in the PUT capability gate so
defects are named before any metadata write. Missing Priority counts as
zero for the uniqueness check, matching Go's zero-value semantics. The
self-target rejection deliberately stays at set-remote-target, where the
endpoint is known; a config can never reference a self-pointing ARN.
Document the rule-level Destination.StorageClass contract (use the
remote target's storage_class instead) and renumber the acceptance
matrix e2e to unique priorities, which MinIO would also require.
* test(replication): pin duplicated wire types with boundary reconciliation tests
rustfs-filemeta (xl.meta disk format) and rustfs-replication (MRF/resync
persistence format) deliberately each own ReplicationStatusType,
VersionPurgeStatusType and ReplicationState; the boundary converts
between them via as_str(), whose From<&str> impls fall back to Empty on
unknown tokens — a variant added on one side silently degrades to Empty
on the other.
Add reconciliation tests in replication_filemeta_boundary: exhaustive
matches with no wildcard arm on both sides of both enums (a new variant
fails compilation until the mapping is reconsidered), string-token
round-trip asserts (a token the other side does not recognize fails
instead of quietly becoming Empty), and a full-field ReplicationState
round-trip. Cross-reference the tests from both type definitions.
Struct drift was already compile-guarded by the exhaustive struct
literals in the conversion functions.
* docs(replication): define split completion criteria and milestone sequence
The ecstore replication split plan had no completion measure — the
boundary scaffolding risked ossifying because nothing said when the
migration counts as done. Record the criteria in the module inventory:
done means the Required Contracts table's 'Current dependency to
remove' column is empty; the end state moves pool/resyncer/state into
crates/replication, with the boundary micro-files dissolving as code
crosses the crate line (batch-merging them beforehand is explicitly
rejected — the guard scripts anchor on their file names, so merging is
churn with zero functional gain; only datatypes.rs can retire early).
Sequence the remaining work as M2 (resyncer pure decision logic, after
the oversized function splits) → M3 (worker runtime, highest risk,
last) → M4 (retire boundaries and guard entries). Refresh the stale
first-step text — the event sink / runtime contracts already landed —
and update the split-plan status table accordingly.
* fix(replication): align structural validator with MinIO semantics after adversarial review
Three interop corrections found by adversarial review of the new
structural validator, plus review fallout fixes:
- Delete-marker replication is now rejected only for a direct Filter.Tag,
not for tags inside Filter.And — MinIO's validator only inspects the
direct tag, and mc replicate add --tags "k1=v1&k2=v2" (delete-marker
replication on by default) puts multiple tags into And.Tags, so the
stricter check rejected mc-generated configs MinIO accepts.
- Rule ID length is measured in bytes (Go len semantics), not chars —
a 255-char multibyte ID must not round-trip into a config MinIO
rejects.
- An empty <Tag/> element (no key) counts as absent, matching MinIO's
Tag.IsEmpty(); console form serializers emit empty tags, which would
otherwise trip the exactly-one-of and delete-marker checks.
Also: repair the store-uninitialized PUT test whose empty-rules fixture
now (correctly) fails structural validation before reaching the store
lookup; pin the previously untested startTime madmin key in the
reset-status shape test; and signal a truncated default-mode diff scan
via the x-rustfs-replication-diff-truncated response header — the bare
madmin stream has no envelope, so a truncated scan was otherwise
indistinguishable from a complete healthy one (madmin/mc ignore unknown
headers).
* test(e2e): activate SSE-S3 replication contract and pin resync fail-closed path
The SSE-S3 replication contract e2e was ignored under backlog#1291
(silent plaintext replication); the fail-closed gate in
replication_target_boundary.rs closed that hole, so the ignore reason
expired. Un-ignore the test — it now pins the current fail-closed
contract (FAILED status, failure event, readable encrypted source,
stable absence of all target versions), verified green.
Add test_bucket_replication_sse_s3_resync_stays_fail_closed: drives the
existing-object resync path (PUT ?replication-reset) over a FAILED
SSE-S3 object and asserts the resync generation reaches a terminal
state without ever materializing a target version, with the
stays-absent window also spanning fast-scanner heal cycles. The new
start_bucket_replication_reset helper doubles as the madmin
ResyncTargetsInfo shape assertion (target[0].arn/resetid) for the
reset-start response.
Refresh the stale nextest count commentary (the module is at 20 fast +
36 nightly = 56 tests by cargo nextest list; the SSE-S3-ignored note no
longer holds).
* fix: address rc.1 release blockers
* fix: route release guards through architecture boundaries
* fix: close remaining rc.1 regression gaps
* refactor: group multipart listing options
* fix: resolve rc.1 CI regressions
* fix(ecstore): keep bucket-config writes off the caller's stack
A bucket-config write nests incarnation resolution (which can drive legacy
migration and a peer fan-out), a full metadata load, and `save` — itself an
object PUT that pulls in the whole erasure write path. Every request that
mutates bucket config is already several futures deep, so inlining all of
that into one state machine overflows the 2MiB worker stack in debug builds.
Two CI lanes aborted with SIGABRT on this:
ILM Integration (serial)
rustfs app::lifecycle_transition_api_test::
compensation_driven_complete_multipart_upload_still_transitions
Test and Lint (swift)
rustfs-protocols::swift_metadata_persistence::
swift_metadata_writes_are_durable
Neither test file is touched by this branch and both lanes are green on
main. Stack-pointer probing showed ~780KiB consumed between
`metadata_sys::update` and the config read alone, with single hops of
363KiB (`update` -> `acquire_config_write_guard_for_incarnation`), 125KiB
and 105KiB.
Box the deep sub-futures on both read-modify-write paths (`update` /
`update_checked` and `update_config_with` / `update_config_with_checked`)
so each guard's own state machine stays small. Behaviour is unchanged;
`update` -> guard drops to 253KiB and both tests pass on the default stack.
* fix(lifecycle): unbreak restore under the bucket generation fence
The ILM lane aborted on a stack overflow before reaching these, so they
were never reported; with that fixed, four restore tests fail. All four
are green on main and none of their test files are touched by this branch.
1. RestoreObject and ListMultipartUploads hard-required
`opts.expected_bucket_incarnation_id`, but `apply_bucket_generation_guard`
deliberately leaves it unset when no guard extension is present — only the
S3 access layer installs one. Every direct caller therefore got
`InternalError: ... bucket generation guard is missing`. Resolve the
current generation instead, the way the copy path already does. The fence
is unaffected: RestoreObject still re-reads the incarnation from disk and
compares before admitting the restore, and the multipart listing is
filtered by the value it resolves.
2. `restore_expiry_snapshot_matches` (new on this branch) rejected every
restored-copy expiry whose `restore_expires` had not already elapsed.
Whether the restored copy is due to expire is the ILM evaluator's
decision, made when it emitted DeleteRestoredAction; re-deriving it in
the set layer only adds a way for a legitimate action to be rejected.
The stale-event risk it appears to guard is already covered by the
surrounding snapshot match — a re-restore rewrites `restore_expires`,
so a replayed event fails the equality check. Drop the clause; the
fifteen identity clauses are unchanged.
Fixed:
rustfs app::lifecycle_transition_api_test::
restore_object_usecase_accepts_exactly_one_of_two_concurrent_restores
restore_object_usecase_completes_suspended_null_version_in_place
restore_object_usecase_reports_ongoing_conflict
rustfs-scanner::lifecycle_integration_test serial_tests::
test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore
Verification: the CI ILM lane filter now runs 53/53 green locally.
* chore: address review follow-ups on this branch
Four items from the adversarial review that were still open.
- Restore the assertion `test_bucket_replication_replayed_delete_marker_
preserves_source_mtime_without_source_restart` is named for. The branch
had replaced the backlog#867 mtime check with `assert_replication_
converged`, which any successful replication satisfies, and deleted the
two helpers it needed — so the regression the test exists to catch would
now pass. This matters here specifically because the branch changes the
flag feeding `replication_delete_remove_options` and routes replay
through a new file and ordering.
- Drop `read_config_no_lock_preserve_empty`: zero production callers (the
one real consumer calls the `_with_metadata` variant directly). Its test
stanza now exercises that variant, so the coverage moves to live code
rather than being deleted.
- Revert the `bytesize` bump. It is a no-op: `Cargo.lock` already pinned
2.7.0 before this branch and is untouched, so the caret range already
resolved there. Nothing in the diff uses the crate.
- Split the AGENTS.md "Adversarial Validation" policy change out of this
branch. The edit is defensible on its own, but it relaxes the review gate
that this branch has to pass, so it should land as its own PR reviewed on
its own merits rather than bundled with the change that benefits from it.
The reverted hunks are unchanged and ready to re-apply.
Not changed, deliberately: the missing-sidecar path still fails closed.
`missing_bucket_incarnation_sidecar_for_new_metadata_fails_closed` pins
that on purpose, and serving a non-authoritative Object Lock state would
be the wrong trade. The residual concern stands and is recorded in review
— a crash between the two writes in `persist_new_and_set` leaves the
bucket unloadable until DeleteBucket+CreateBucket, and the repair branches
in `migrate_legacy_metadata` and `make_bucket` are unreachable dead code
for that case. Resolving it needs the read path and the (transaction-lock
holding) repair path to be separated, which is more than a follow-up edit.
* test(ci): serialize the new bucket-incarnation tests
The five tests this branch adds around the incarnation / lifecycle fence
drive `init_bucket_metadata_sys` and `bucket_metadata_sys_of` — process-global
OnceLock state that `serial_test`'s `#[serial]` cannot protect across
nextest's process boundary — and they delete+recreate buckets, the shape that
raced into InsufficientWriteQuorum in backlog#937.
Add them to the `ecstore-serial-flaky` group in both the default and ci
profiles (nextest evaluates a named profile's own overrides list, so the
ci mirror is required). Preventive serialization only, no retries.
Not a full fix for the review comment: `bucket_delete_waits_for_config_
mutation_fence` still proves liveness with a fixed 200ms sleep plus
`assert!(!delete.is_finished())`. Turning that into readiness polling needs
a production-side signal to wait on — asserting "still blocked" is inherently
a negative. Serializing the group removes the parallel-load pressure that
makes the window fragile; the sleep itself is left for a follow-up.
* test(ecstore): pin that a drained bucket is actually deletable
`DeleteBucket`'s emptiness check is `has_xlmeta_files`, a raw scan of the
bucket directory on local disks — not an S3-level listing. So "the client
drained the bucket" and "the bucket is deletable" are two different
contracts, and only the first one was covered.
That gap is what the `S3 Implemented Tests` lane is failing on: 219 cases,
all `BucketNotEmpty` on `nuke_prefixed_buckets`, with every test body
passing. The first one is `test_versioning_obj_suspend_versions`, reported
by pytest as PASSED followed by ERROR at teardown.
Add the missing assertion for the unversioned path: PUT, client DELETE,
then assert no `xl.meta` survives and `DeleteBucket` succeeds. It passes —
which is itself a result: the plain delete path leaves no residue, so the
s3-tests failure is not there.
The versioning-suspended path is the remaining suspect (the client DELETE
leaves a null delete marker, and draining means purging it by
`versionId=null`). It is not covered here: `BucketVersioningSys` resolves
through the ambient `get_bucket_metadata_sys()` OnceLock, which this unit
env cannot set, so the bucket never actually reports as suspended. That
repro belongs at the e2e layer where a real server owns the versioning
state.
* fix(ecstore): let an explicit null-version delete purge its delete marker
Root cause of the `S3 Implemented Tests` lane: 219 cases, all
`BucketNotEmpty` on `nuke_prefixed_buckets`, every test body passing.
On a versioning-suspended bucket a client DELETE leaves a null delete
marker — correct S3 semantics, and an `xl.meta` on disk. Draining the
bucket therefore means purging that marker as `?versionId=null`, which is
what `nuke_bucket` does before `DeleteBucket`. That purge was rejected:
explicit null-version purge of the null delete marker must succeed,
got [Some(MethodNotAllowed)]
so the marker survived, and `DeleteBucket`'s emptiness check — a raw
`has_xlmeta_files` scan of the bucket directory, not an S3 listing — kept
reporting the bucket as non-empty.
The two sides of the version comparison in the batch delete loop are in
different namespaces. `goi.version_id` is the client-facing identity, where
`from_file_info` synthesizes `Some(Uuid::nil())` for a null version on a
versioned *or versioning-suspended* bucket. `version_id` is the storage
identity, where `delete_file_info_version_id` maps an explicit
`?versionId=null` to `None`. Comparing them raw makes the purge look like a
version mismatch, so `explicit_delete_marker` is false and the
`MethodNotAllowed` from the lookup is recorded as a delete failure.
This only became reachable on this branch: previously `check_opts` did not
carry `dobj.version_id`, so `set_disk_delete_creates_delete_marker` was
true, `object_lock_check_required` was false, and the lookup that produces
`MethodNotAllowed` never ran. Adding the version id to `check_opts` lit up
a comparison that was already wrong.
Normalize both sides through `delete_file_info_version_id`.
The regression test injects a real Suspended bucket-config snapshot — the
delete path reads versioned/suspended from that snapshot, not from `opts`,
so without it `from_file_info` never synthesizes the null version id and
the branch is not reached. Mutation-checked: restoring the raw comparison
fails the test with the exact `MethodNotAllowed` above.
* fix(app): drop the now-needless struct update
Reverting `crates/replication` to main removed the extra `MrfReplicateEntry`
fields, so this literal specifies every field again and `..Default::default()`
trips `clippy::needless_update` under `-D warnings`.
Caught by CI, not locally: I had run `cargo check --workspace --all-targets`,
which does not see clippy-only lints. Ran `cargo clippy --workspace
--all-targets -- -D warnings` here — clean.
* test(e2e): assert the fresh-volume classification
four_node_empty_legacy_volumes_start_as_fresh only started the cluster and
listed buckets — no assertion, so any classification path that still permits
startup left it green without proving the pre-created empty `.minio.sys`
directories were treated as fresh volumes.
Pin what that classification actually leaves behind: no buckets adopted into
the namespace, `.rustfs.sys/format.json` written on every drive, and the empty
legacy directory left untouched rather than migrated into.
* fix(bucket): apply the requested Object Lock to existing buckets
Site replication replays make-with-versioning against the destination,
carrying the source's `lockEnabled`. When the destination bucket already
exists it takes `force_create`, and the whole option-application block was
gated on `confirmed_missing` — so the call returned success while the replica
stayed unlocked. Replicated versions could then be deleted without the
retention the source enforces.
Object Lock enable is one-way, so applying it to an existing bucket is safe:
move it out of the creation-only gate, keeping `created` and versioning-only
options creation-scoped as before.
An existing authoritative bucket takes the `cache_bucket_metadata_in` branch,
which only caches, so the enable would have been dropped on restart. Persist
instead when the enable actually changed something.
Mutation-checked: restoring the creation-only gate fails the new
`force_create_enables_object_lock_on_an_existing_bucket` with "Object Lock
must be enabled on the existing bucket".
cargo nextest run -p rustfs-ecstore --lib: 3633 passed.
* fix(ecstore): box the generation-checked config mutation paths too
The earlier stack fix boxed `update` and `delete`, but an authorized
bucket-config mutation carrying an incarnation takes `update_if_incarnation`
/ `delete_if_incarnation` instead — which were still inlining the whole
resolve/load/save chain into an already-deep request future. Same overflow,
sibling path.
* fix(restore): keep the nil-version normalization the strip removed
Reverting the replication subsystem to main took `set_disk/replication.rs`
with it, but one line in that file was this branch's own fix rather than
replication work:
- self.version_id.filter(|v| !v.is_nil()) == fi.version_id.filter(|v| !v.is_nil())
+ self.version_id == fi.version_id
For a versioning-suspended object the expected version is `Some(Uuid::nil())`
while the read-back `FileInfo` carries `None`, so the raw compare reports
every suspended restore as "restored object changed before restore metadata
finalization" and the copy-back never commits. Same nil-vs-None mismatch as
the null delete-marker purge fixed earlier on this branch.
Caught by `Test and Lint (rio-v2)`, not by my local runs: the test lives in
`transition_commit_failure_tests`, gated behind `feature = "test-util"`, so
the 3633-test suite I had been running never included it. Re-ran with
`--features rio-v2,test-util`: 3722 passed.
* fix(site-replication): keep reverse direction after config broadcast
`site-repl-*` rules encode the sender's outbound direction: their
destination ARN names the receiver. `apply_bucket_meta_item` wrote an
incoming rule set verbatim over the receiver's, leaving the receiver with
a rule whose ARN is its own deployment ID. `reconcile_site_replication_bucket_targets`
skips the local peer, so no bucket target can back that ARN and every
object was dropped; the follow-up call reconciled targets only, so
nothing rebuilt the lost reverse rule. Replication went one-directional
after any PutBucketReplication broadcast — the console's Save button,
`mc replicate import`, a metadata import, or `/site-replication/repair`.
Only operator-authored rules now travel between sites; each site owns its
`site-repl-*` rules and rebuilds them from the current peer set.
Four defects kept that invisible or unrecoverable:
- `update_all_targets` discarded target-client build errors silently, and
`replicate_object` logged the resulting missing-target drop at debug
while every other failure there logs at error. Both now report.
- `site_replication_rule_complete` never checked that a rule's
destination named a remote site, so two sites holding identical
configs — the post-clobber state — passed as in sync.
- `update_service_account` cannot rewrite `parent_user`, and IAM records
encrypted with a previous root secret decode as "no such account".
Startup now reconciles the account, reseeding from the secret every
site-replication bucket target already stores, and refuses the
delete-then-create sequence when the parent cannot back an account.
Bucket rules are reconciled too, so an already-broken site heals on
upgrade.
- A joined site never verified it could reach the initiator, whose
endpoint is derived from the Host header of the admin request that
created the topology. The join now probes each peer and reports through
`initial_sync_error_message`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(site-replication): report unreachable targets and reconcile on a timer
Rule-shape checking cannot see an unreachable peer. A `site-repl-*` rule
can be perfectly formed while the endpoint recorded for its peer is one
this site cannot reach: `update_all_targets` then builds no client and
`replicate_object` drops every object against that ARN, yet the rule set
still reads as correct and the bucket reports in sync.
Each site now reports whether all of its `site-repl-*` rules resolve to a
live target (`SRBucketInfo.replicationTargetsOnline`, read from the
already-resolved client map so the status path stays cheap), and the
status aggregation treats an offline report as a mismatch. The field is
additive and optional: peers that omit it are "unknown", never a fault,
so a mixed-version topology does not flip every bucket to out of sync.
The reconcilers also run on a 10-minute timer instead of at startup only,
so drift is repaired without waiting for a restart. Both are no-ops when
the wiring already matches — the bucket pass compares serialized targets
and the rule set before writing. The tick takes the site-replication
lifecycle lock with a non-blocking try_acquire and skips the round when
an add/remove/endpoint-refresh holds it: those run in phases, and
rebuilding rules between two of them would resurrect what the operation
just tore down.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(site-replication): invert reconcile dependency to satisfy layers
`startup_services.rs` sits in the infra layer and was calling the
reconcilers in `admin::handlers::site_replication`, which is interface —
a reverse dependency that check_layer_dependencies.sh rejects.
Moving the reconcilers down is not viable in this change: they rest on
the site-replication state core (`SiteReplicationState` alone has 107
in-file uses, `load_site_replication_state` 38, the state lock 33), so
relocating it would move ~2000 lines and ~200 call sites through a
bug-fix PR.
Invert the direction instead. A new infra module owns the contract and
the schedule; the admin layer registers its reconciler from
`register_site_replication_route`, which runs while the admin router is
built — `init_startup_http_servers` awaits that before
`init_startup_runtime_services` reconciles, so the hook is always
installed in time. No logic moves and no baseline entry is added: the
dependency genuinely reverses.
The lifecycle guard now wraps both reconcilers in one round rather than
each separately, closing the window between them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(site-replication): harden reconcile per review feedback
Addresses the automated review on #5292.
Security: secret recovery from bucket targets accepted any target carrying
the `site-replicator-0` access key. Bucket targets are writable by anyone
holding `admin:SetBucketTarget`, so such a principal could plant a secret
and have reconciliation recreate the broadly privileged replication
account with it. A target must now name a peer in the persisted state and
point at that peer's recorded endpoint, disagreeing targets abort the
recovery, and only missing/unreadable-account errors may trigger it at
all — a transient store failure no longer rewrites a live account.
Durability: the repair no longer deletes before creating. A readable
account is rebound in place through a new `parent_user` field on
`UpdateServiceAccountOpts`, gated to `site-replicator-0` under
`allow_site_replicator_account` exactly as the account itself is. The
parent also lives in the session-token claims, and
`prepare_service_account_auth` denies the account when the two disagree,
so both move together.
Availability: the reconcile scheduler no longer requires an inline IAM
bootstrap. Deferred IAM recovers in the background with no callback into
the scheduler, which left a recovered node with self-pointing rules until
the next restart. It now starts unconditionally and returns early while
IAM or the object store are unavailable. Its first pass runs inside the
task, so walking every bucket no longer delays startup.
Correctness: an endpoint refresh commits bucket targets and peer state in
separate steps without holding the lifecycle lock, so a tick landing
between them rewrote targets from the stale endpoint; the reconciler now
also skips while any pending marker is set. Rule repair preserves an
operator-authored `role` and clears only sender-owned site-replication
ARNs, matching the merge path.
Hot path: the per-object missing-target message returns to debug. The
condition is reported once per bucket per reconcile pass instead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>
* fix(sse): surface unconfigured managed SSE as 400, fix POST SSE-S3 e2e
Managed SSE (SSE-S3 / bucket-default) on a server without KMS and without RUSTFS_SSE_S3_MASTER_KEY fails closed by design since #3564, but surfaced as 500 InternalError. Map the misconfiguration to InvalidRequest (400) and seed the local SSE master key in the anonymous POST-object SSE e2e tests; align the missing-from-policy test with the MinIO-compatible SSE field exemption (s3s#608). Re-admit the tests to the e2e-full profile (rustfs#4844).
* fix(auth): POST-object lock fields and signed metadata=true listing 403s
Two authorization surfaces returned 403 for allowed requests (rustfs#4845): the PutObject access hook demanded PutObjectRetention/PutObjectLegalHold IAM actions for POST-object form uploads whose lock fields are governed by the validated POST policy, and the metadata=true listing route never resolved SigV4-verified credentials into ReqInfo so signed requests were evaluated as anonymous. Skip the PUT-header lock actions for POST form uploads and resolve credentials in the metadata route; re-admit the four e2e tests to the e2e-full profile.
fix(sse): surface unconfigured managed SSE as 400, fix POST SSE-S3 e2e
Managed SSE (SSE-S3 / bucket-default) on a server without KMS and without RUSTFS_SSE_S3_MASTER_KEY fails closed by design since #3564, but surfaced as 500 InternalError. Map the misconfiguration to InvalidRequest (400) and seed the local SSE master key in the anonymous POST-object SSE e2e tests; align the missing-from-policy test with the MinIO-compatible SSE field exemption (s3s#608). Re-admit the tests to the e2e-full profile (rustfs#4844).
An extracted entry whose tar header mtime is 0 was stored with
mod_time = UNIX_EPOCH, which xl.meta encodes as 0 nanos (= no
mod_time). On read-back the version failed valid() and parsing fell
into the legacy rmp_serde fallback, so every read of the object
returned 500 (invalid type: integer 0, expected an OffsetDateTime).
Treat mtime 0 as unset (tar convention) and fall back to the upload
time. Also fix two test-side issues uncovered behind the 500: the pax
fixture must use a ustar header for its XHeader entry, and the SSE-S3
extract tests must provision RUSTFS_SSE_S3_MASTER_KEY. Re-admit the 19
quarantined tests to the e2e-full merge gate.
Fixes#4842
test(e2e): add drivesPerNode and 2-pool cluster harness topology (backlog #1325)
Extend the e2e cluster harness so tests can declare per-node multiple drives (drivesPerNode) and a two-pool topology, alongside a smoke suite that boots such a cluster and round-trips a PUT/GET.
A new `ClusterTopology` describes node_count, drives_per_node, and pool membership, and `RustFSTestClusterEnvironment::with_topology` builds the matching on-disk drive directories and `RUSTFS_VOLUMES` string. `new(node_count)` now delegates to a single-pool single-drive topology, so existing single-drive cluster tests are byte-for-byte unchanged. `ClusterNode` gains `data_dirs` (all drives, `data_dirs[0] == data_dir`) and `pool_idx`; multi-drive/multi-pool clusters automatically add `RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true` because their drives share the same temp filesystem.
The `RUSTFS_VOLUMES` assembly matches the two forms the server parser accepts on a single localhost machine (verified against ecstore `DisksLayout::from_volumes`): a single pool is the explicit enumeration of every `(node, drive)` endpoint (no ellipses, one legacy DistErasure pool), while multiple pools each contribute one ellipses argument `http://<addr><node-base>/drive{0...N-1}`. A pool argument is a single URL template, so it cannot enumerate multiple distinct-port hosts; a pool striped across several localhost nodes would need a host ellipses that forces a shared on-disk path and collides physically. The topology validator therefore requires every pool in a multi-pool layout to own exactly one node and requires drives_per_node >= 2 (the parser rejects a single-drive `drive{0...0}` ellipses pool). Genuine multi-node pools need real multi-host infrastructure and are deferred to the nightly cluster lane (backlog #1313/#1314).
Unit tests assert the volumes-string layout for single-pool single-drive (backward compatible), single-pool multi-drive (8 explicit endpoints), and two-pool (one ellipses arg per pool), plus the validator rejections. The smoke suite `cluster_multidrive_pool_test` boots a 4-node x 2-drive single pool and a 2-node/2-pool cluster, asserts the volumes layout, and round-trips a PUT/GET using the harness readiness handshake (no fixed sleeps). It joins the six existing RustFSTestClusterEnvironment suites excluded from the merge-gate `e2e-full` profile so it runs in the nightly 4-node lane.
Network fault injection (toxiproxy / socket proxy) and 5GiB large-object budgets remain out of scope for this block.
First systematic HTTP e2e batch for the admin management surface
(backlog#1154 peri-2): full user -> canned-policy -> attach ->
service-account lifecycle with data-plane effect assertions (the
credential really gains and loses S3 access), plus a non-admin 403
probe per management endpoint targeting a different principal so
deny_only self-access semantics do not mask the gate. Wired into the
e2e-smoke nextest profile (ci-4 mechanism).
Pin the console listener's wire contract with a real binary on a random
port (backlog#1154 peri-4): unauthenticated version/license endpoints
answer complete JSON without credential material, the SPA prefix always
dispatches to the console static handler (never the S3 API), the admin
API and S3 root on the console listener stay 403 for anonymous callers,
and a console-disabled listener serves no console surface. Wired into
the e2e-smoke nextest profile (ci-4 mechanism).
Prove the swap-certificates-without-restart promise over real TLS
handshakes (backlog#1154 peri-5): new connections pick up the rotated
certificate within the reload interval, a session opened under the old
certificate survives the swap, and garbage material is fail-safe (the
old certificate keeps serving, the failure leaves a tls_reload_failed
log event, the process stays up). Wired into the e2e-smoke nextest
profile (ci-4 mechanism).
* ci: add e2e-full merge-gate job (single-node full e2e via nextest)
Adds the [profile.e2e-full] nextest profile and a matching e2e-full job in
ci.yml (push main + merge_group + workflow_dispatch) that runs the
never-automated user-visible e2e suites (KMS, object_lock, multipart_auth,
quota, checksum, encryption, security-boundary, ...) the fast PR e2e-smoke
subset skips.
The filter is the whole e2e_test crate minus the sets other lanes own:
protocols:: (ci-6/ci-7), the 6 RustFSTestClusterEnvironment cluster suites
(ci-7 nightly), replication_extension_test (repl-1's smoke + repl-nightly
lanes), and #[ignore]d tests (ci-13). The 4-disk reliability tests are
serialized, mirroring the ci profile. Reuses the downloaded debug binary and
uploads junit. backlog#1149 ci-5.
* ci: exclude characterized known-failures from e2e-full with tracked issues
First-ever automated run of the full suite (dispatch 29381309848: 341 ran /
32 failed / 460s) surfaced five real product-bug families, each now tracked:
rustfs#4842 (extract 500s, mtime=0 OffsetDateTime), rustfs#4843 (path-limit
vs ignore-errors), rustfs#4844 (anonymous POST SSE-S3 500), rustfs#4845
(object-lock POST + list metadata=true 403), rustfs#4846 (lock quorum
misclassified as timeout under load). Deterministic failures cannot be
retried away, so each family is excluded with its OPEN issue cited and the
fixing PR contract to delete the exclusion — passing negative-path siblings
stay in as regression guards.
The header-SigV4 (sec-1), presigned-URL (sec-2), and admin-gate (sec-4) negative
auth-rejection e2e suites merged earlier but only presigned_negative was actually
selected by any CI profile; negative_sigv4_test and admin_auth_test compiled and
sat unrun. Add both to the e2e-smoke default-filter so all three attacker-facing
S3 auth-rejection suites execute on every PR. They already meet the smoke
admission criteria (RustFSTestEnvironment, random ports, parallel-safe, no
#[ignore], no feature gates), so this is a pure filterset change — the single
e2e-in-CI wiring mechanism (backlog#1149 ci-4), not a new job.
Because the filter selects by module name, a rename or deletion could silently
drop a suite out of the security gate with no CI signal. Add
scripts/check_security_smoke_count.sh (infra-12 count-floor mechanism): it lists
what the e2e-smoke profile selects and fails if the count of security
auth-rejection tests drops below the committed floor in
.config/security-smoke-floor.txt (16). Invoked from the e2e-tests job, before the
smoke run, so the nextest list compiles the binaries the run reuses.
The GHSA-3p3x FTPS/WebDAV constant-time e2e (protocols::test_protocol_core_suite)
stays out by topology: it binds fixed ports, needs the ftps,webdav features, and
is #[serial], so it cannot join the random-port default-feature smoke profile as
a filterset change (global ruling G5). Its GHSA-r5qv sibling is a unit test that
already runs in the default CI pass. docs/testing/security-regressions.md now
carries the full CI-execution map and flags the GHSA-3p3x e2e CI-lane gap as a
ci-domain follow-up.
Refs: backlog#1151 (sec-5)
Convert the two reliant lifecycle expiry tests from #[ignore] + a hardcoded
localhost:9000 server to self-managed RustFSTestEnvironment tests (random port,
isolated temp dir) and wire them into the PR e2e-smoke subset.
Time control is chosen per test and documented in-module:
- test_lifecycle_expiry_backdated_mtime: mod_time back-dating via the internal
source-replication backdoor (new put_object_with_backdated_mtime helper) so a
Days=1 rule is already due, with RUSTFS_ILM_PROCESS_TIME=1 shrinking the
rounding boundary. Asserts the backdated matching-prefix object expires and a
non-matching-prefix object with the same backdating survives (isolates the
prefix filter). Proves the backdoor does not trip the replication gate.
- test_lifecycle_versioned_current_version_expiry_creates_delete_marker:
RUSTFS_ILM_DEBUG_DAY_SECS=2 (ilm-5) accelerates the day length; asserts a
latest delete marker is created, the original data version is retained and
still readable by version id.
- test_lifecycle_zero_day_expiry: Days=0 immediate expiry; matching object
deleted, non-matching survives.
All three use RUSTFS_SCANNER_CYCLE=1 so the scanner runs every second; tests poll
for the terminal state instead of sleeping fixed wall-clock. Ignore count for
reliant/lifecycle.rs goes 2 -> 0.
CI wiring: added a distinct 'test(/^reliant::lifecycle::/)' clause to the
existing profile.e2e-smoke default-filter in .config/nextest.toml (the single
sanctioned e2e wiring mechanism per crates/e2e_test/README.md; no new e2e job).
Refs rustfs/backlog#1148 (ilm-3), rustfs/backlog#1155
test(e2e): negative presigned-URL SigV4 suite (backlog#1151 sec-2)
Add crates/e2e_test/src/presigned_negative_test.rs, the query-string
SigV4 sibling of the header-SigV4 suite (sec-1, #4708). Presigned URLs
were only exercised on the happy path, so nothing pinned our end-to-end
enforcement of expiry or query-signature verification.
Against a live single-node RustFSTestEnvironment (random port, isolated
temp dir), the suite asserts HTTP status + S3 error <Code> for:
- expired presigned GET -> 403 AccessDenied (Request has expired)
- tampered X-Amz-Signature -> 403 SignatureDoesNotMatch
- wrong secret key -> 403 SignatureDoesNotMatch
- tampered target key (swapped after signing) -> 403 SignatureDoesNotMatch
- tampered presigned PUT -> 403 SignatureDoesNotMatch + object not stored
- positive controls: valid presigned GET (body match) and PUT (HEAD verify)
Expiry is controlled without real waiting via the AWS SDK presigner's
start_time (sign as of one hour ago with a 60s window). s3s checks
expiry before the signature, so the expired case surfaces AccessDenied.
Wired into the e2e-smoke nextest profile (fast, single-node, no external
deps) so it runs on every PR per the crates/e2e_test/README.md admission
criteria.
Refs rustfs/backlog#1151 (sec-2), rustfs/backlog#1155.
* fix(replication): allow loopback replication targets under an explicit test opt-in
Commit 5c7c757a3 (#4712) activated the previously-dormant replication e2e
suite (they had never run anywhere). All 9 fast tests then failed on main
because the SSRF egress guard rejects the 127.0.0.1 targets the e2e harness
configures: `target endpoint is not allowed: outbound URL host '127.0.0.1'
is not allowed: loopback address`. The whole harness runs on loopback, so
every replication test hit this before reaching its actual assertion.
Loopback is a genuine SSRF vector and must stay rejected in production, so
this does not relax the guard. Instead `validate_replication_target_endpoint`
gains an off-by-default opt-in (`RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET`)
that re-enables loopback targets (127.0.0.1 / ::1 / localhost) for single-host
multi-instance dev and the e2e harness. Private addresses stay unconditionally
allowed as before; the opt-in does not widen into link-local or the cloud
metadata endpoint. The e2e harness sets the env for every server it spawns
(single-node and cluster paths), overridable via extra_env.
Verified end-to-end: all 9 previously-failing replication_extension_test
smoke tests pass against a locally built binary. New unit tests in
bucket_target_sys pin the matrix — public/private always allowed, loopback
gated on the opt-in in both IP and hostname forms, and metadata/link-local
still rejected even with the opt-in on.
Refs: backlog#1147
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(replication): rename optin -> opt_in to satisfy typos check
Pure rename of three unit-test function names; no behaviour change.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
ci: pull replication tests out of e2e-smoke — loopback targets fail every PR
repl-1 (#4712) added 20 bucket-replication admin-path tests to the
e2e-smoke PR lane. They set a remote replication target at a loopback
endpoint (127.0.0.1, a second local server), which RustFS's target SSRF
guard rejects by default ('outbound URL host 127.0.0.1 is not allowed:
loopback address'). repl-1's local verification was inconclusive under
machine load, so this shipped broken and failed End-to-End Tests on
every PR based on post-repl-1 main (#4707 etc).
Move all replication e2e tests to the e2e-repl-nightly lane (now the full
replication_extension_test set, no allowlist split) to un-block PRs. The
nightly job needs loopback-target-allow server config for these to pass;
tracked as a repl-1 follow-up (backlog#1147). e2e-smoke returns to the
17 functional modules from ci-4 (63 tests).
houseme