mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-16 09:58:21 +00:00
728b0488c4a1a55656ccac1b20fbfb1436d2119f
90 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Zhengchao An
|
d668a9293f |
chore(ecstore): remove test-only BitrotErrorType and pin wire-only disk variants (#6032)
BitrotErrorType (disk/error.rs) was constructed only by its own unit test: production bitrot mismatches never flow through it (they surface as DiskError::other strings). Delete the enum, its From<BitrotErrorType> for DiskError impl, the self-test, and the api facade re-export. The facade inventory doc does not name the type, so no doc change is needed. DiskError::SourceStalled and DiskError::CrossDeviceLink are never constructed locally — they are reachable only through wire decoding and no current node sends them. Their decode arms stay per the cross-version compatibility constraint; each variant now carries a doc comment saying exactly that so the next dead-code sweep does not re-litigate them. Their consumer arms (heal classifier, batch processor) are left untouched — the values cannot appear, so removing the arms would be unobservable, and the heal classifier is pinned by the issue as do-not-touch. Ref rustfs/backlog#1831 (PR4). |
||
|
houseme
|
398d2d87c8 |
fix(ecstore): retry manual ILM job CAS updates (#6012)
Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
cxymds
|
6cce3d60bb |
fix(quota): reject oversized multipart completion (#5958)
* fix(quota): reject oversized multipart completion * fix(arch): route quota test through app facade |
||
|
唐小鸭
|
603bdea516 |
fix(site-replication): route state RMW through one locked transaction (#5882)
* test(site-replication): pin retry-event lost-update against locked RMW (red) P1-15 (rustfs/backlog#1675 B2): the site-replication retry-event writers (enqueue/dequeue, which hang off every hook broadcast path) perform a load -> mutate -> persist without taking SITE_REPLICATION_STATE_LOCK, so a single process can lose a concurrent lock-holding writer's update; the service-side reload path is equally unlocked, and no writer holds a distributed lock across the read-modify-write, so multi-node RMW loses updates even where the process lock is held. Red evidence (current main): replaying enqueue's exact three steps around a completed mark_pending_rotation_peer_acked commit wipes the rotation ack — the final state holds the retry event but not the ack. * fix(site-replication): route state RMW through one locked transaction P1-15 PR1 (rustfs/backlog#1675 B2). The site-replication state object (config/site-replication/state.json, which also carries the retry-event queue) was mutated through read-modify-write sequences with inconsistent locking: the retry-event writers on every hook broadcast path and the RPC-driven service reload took no lock at all (single-process lost updates, pinned by the red commit), and no writer held a distributed lock across the whole RMW (cross-node lost updates everywhere). - New admin/site_replication_state module: the state transaction boundary `with_site_replication_state_lock[_on]` — process mutex plus the distributed config-object write lock (the pattern proven by the repair state), with the shared path constant. The process mutex is transitional until PR2 migrates the remaining ~26 call sites. - handlers: typed `update_site_replication_state` (no-lock load / persist-or-clear inside the boundary; normalizes the peer map exactly once, retiring the double-clone/double-normalize persist path, P2-22). Migrated: retry-event enqueue (always-write), dequeue (lock-free probe, transaction on hit), mark_pending_rotation/remove_peer_acked. - service reload: the tolerant byte-level read->normalize->save now runs inside the same boundary via no-lock IO — a cluster-wide reload fan-out can no longer overwrite a concurrent state writer. Normalization semantics untouched (all six service-side tests unchanged and green). - Add/PeerJoin/Edit handlers release the state guard before their peer fan-out: the transport helpers' retry-event bookkeeping now re-enters the state transaction and must not nest inside the guard (the adversarial review caught this as a re-entrancy deadlock; the fix mirrors the Remove/Rotate handlers' existing scope). The Edit non- refresh branch commits before fanning out — the old fanout-first order recorded retry events pointing at a state the local site had not saved. - ecstore: delete_config_no_lock (+ facade/bridge exports) so the clear half of persist-or-clear works under the held object lock. Red -> green: the red commit pinned the deterministic lost-update interleaving (stale retry-event persist wiping a committed rotation ack); the test now drives the real functions concurrently for 8 rounds and asserts every retry event and every ack survives. Full handlers/service site-replication unit suites green (171 + 6); dual-node site-replication e2e (state edit fresh/stale, object replication) green; fmt / clippy / logging guardrails clean. Adversarial review: one blocking finding (the re-entrancy deadlock above) fixed and re-verified by a full second pass over all 30 lock sites and the Add/Join/Edit call graphs. Non-blocking notes recorded for PR2: mark_* now persists on miss (persist-or-clear semantics; a miss-skip return is a cheap follow-up), Add still holds the guard across the peer join probe (pre-existing availability debt), and a timeout-guarded unreachable-peer regression test for the fan-out paths. * fix(site-replication): keep the state mutex behind an owner helper CI's architecture migration guard lists SITE_REPLICATION_STATE_LOCK as an owner-local static, so it may not be `pub(crate)`. Keep it private to the new module and let the not-yet-migrated RMW call sites take it through `site_replication_state_process_guard()` — the sanctioned owner-helper pattern; the helper disappears with the mutex in PR2. * fix(site-replication): keep peer-edit delivery under the state guard Review follow-up (#5882). Releasing the guard before the fan-out (my deadlock fix) traded the ordering the guard used to provide: edit A could commit and stall while edit B committed and reached a peer first, then A arrived last and won. The peer edit handler applies whatever arrives — it has no generation or updated-at fence — and a successful stale delivery is not repaired by the retry queue, so the sites diverge silently. The fan-out is back under the guard. What actually could not run there is the retry-event bookkeeping, which re-enters the state transaction, so the edit branch now delivers with the plain transport and settles the retry queue after the guard is released: successes dequeue, the first failure enqueues and is returned. Ordering and bookkeeping both preserved. The add handler keeps its peer-edit finalize fan-out under the guard for the same reason and releases only before bootstrap/back-fill, which send bucket-ops (not peer edits) through retry-event transports. The concurrency test could not tell the two guards apart — both writers took both locks, so it passed with either removed. Replaced by two tests that isolate one guard each, both verified by mutation: - a process-only legacy writer (the shape the not-yet-migrated call sites still use) racing the transaction: fails when the transaction stops taking the process mutex; - two writers that bypass the process mutex, as separate nodes do, driving the production object-lock path (`with_site_replication_state_object_lock` factored out for exactly this): fails when the distributed lock is removed. Verification: handlers 173 + service 6 unit tests green; site-replication dual-node and three-node edit e2e green; arch/layer/logging guardrails, fmt and clippy clean. * fix(site-replication): fence peer-edit delivery by generation Review follow-up on the two remaining holes in the edit path. Ordering was only process-local. `SITE_REPLICATION_STATE_LOCK` is per node, so holding it across the fan-out orders the edits ONE node accepts and nothing else: two nodes of the same site can both commit and reach a peer in the opposite order, and the peer edit handler applied whatever arrived last. Each edit now takes a generation from `SiteReplicationState::edit_generation`, allocated in the same commit as the edit itself — i.e. under the distributed state-object lock, so two nodes can never share one. The generation rides the peer-edit request as query parameters and the receiver rejects (acks without applying) a delivery at or below the mark it already applied for that origin site, recording the mark in the same commit as the edit it fences. Peers that predate the fence send no parameters and are applied as before. Retry settlement could discard a newer failure. After the guard is released, a success for edit A removed every retry event for (peer, peer-edit): if edit B committed, failed its own delivery and enqueued while A was in flight, A erased it — local state B, peer on A, nothing queued to converge them. Settlement now only removes events whose recorded generation is not newer than the one being settled, and a later failure never lowers the fence. Broadcast paths carry no generation and settle unconditionally as before; their events live under their own paths and cannot collide with a peer-edit delivery. A departed peer's mark is dropped on load: a site that leaves drops below two peers, which clears its state object and restarts its counter at zero, so a leftover mark would reject every edit it sends after it rejoins. Tests: two-node generation uniqueness (drop the object lock and the two nodes collide), the receiver's staleness predicate and its wiring, the settlement interleaving (drop the fence and B's retry is erased), and the rejoin reset. Refs: rustfs/backlog#1675 (P1-15) |
||
|
唐小鸭
|
2ecf6b4575 |
fix(replication): probe the version-identity contract in replication-check (#5881)
* test(replication): pin the version-fidelity probe contract (red) P1-19 (rustfs/backlog#1675 B2): the supported replication contract is targets that adopt the source version id — a target that mints its own ids silently breaks every version-addressed operation that follows (version deletes, heal re-drives never match), diverging the two sides with no signal. replication-check already captures the probe PUT's response version id but never compares it. Red evidence (current main): against a FakeS3Target with assign_own_version_ids enabled, ?replication-check returns Status "OK" — the drift is invisible. test_replication_check_flags_version_minting_target expects a VersionFidelity phase that fails with the machine-readable code BucketRemoteTargetVersionMismatch, skips the later mutation phases, and still cleans up the probe via the version id the target actually assigned. Test infra: FakeS3Target gains assign_own_version_ids (models a generic S3 service; validated-but-not-mirrored source version headers) and a prefix+max-keys ListObjectVersions implementation (the probe key allocation requires it); stored_versions accessor duplicated from the P1-21 branch (identical code, resolves clean on merge). * fix(replication): probe the version-identity contract in replication-check P1-19 (rustfs/backlog#1675 B2, plan B). Replication only converges on targets that adopt the source version id: version-addressed deletes and heal re-drives address the source id, so a target that mints its own ids silently diverges — nothing surfaced this. replication-check already captured the probe PUT's response version id but never compared it. - The probe PUT now carries the source version as `?versionId=` (the exact shape live replication uses since P0-5, and the only shape MinIO consumes; the internal source-version-id header alone would let the probe pass against targets the real data path drifts on). Reuses ecstore's append_version_id_query through the api facade. - New VersionFidelity phase: the probe PUT's response version id must equal the sent source id. On mismatch the phase fails with the machine-readable extension key `"Code": "BucketRemoteTargetVersionMismatch"` (new optional Code field on phase statuses; Go decoders ignore unknown keys), the overall target fails, the later version-addressed mutation phases are skipped, and cleanup still removes the probe via the id the target actually assigned (with the existing list-based sweep as backstop when the target returns no version id at all). - Runtime half: TargetClient::put_object now returns the assigned version id (mirroring remove_object), and the replication PUT path audits it — every drifting PUT increments rustfs_replication_version_identity_drift_total and the first drift per target ARN logs a structured warning pointing at ?replication-check. The drift judgment is a pure function with an exemption-matrix test (empty / literal "null" / nil-uuid sources carry no contract). - docs/operations/replication-check.md documents the phase and the code. Red -> green: test_replication_check_flags_version_minting_target (fake target with assign_own_version_ids; on main the check reported Status "OK"). The probe's query shape is pinned by a journal assertion (revert of the query hunk alone fails it), probe-level unit tests cover the mismatch/mirror matrix including cleanup addressing the minted id, and the existing success e2e now asserts VersionFidelity OK against a RustFS target. Adversarial review (seven roles): non-blocking; noted follow-ups are the multipart runtime audit (the probe phase already pins the contract) and per-target re-warning after reconfiguration. * fix(e2e): stop the fake target self-deadlocking on version-id minting The assign_own_version_ids flag was read with a fresh `lock(&self.store)` inside two paths that already hold that guard — delete_object's marker-creation branch and create_multipart_upload — and the store mutex is not reentrant, so both hung forever (CI: the fake target's own multipart and delete-marker tests ran >1560s until the job was cancelled). Read the flag from the live guard instead. The replication e2e paths did not catch this: a version-addressed purge DELETE never mints an id, and the probe PUT reads the flag before taking the guard. * chore(test): refresh the nextest replication count invariant The e2e-smoke/e2e-repl-nightly split comment is descriptive metadata (authority: `cargo nextest list`); refresh it to this branch's post-rebase total. |
||
|
houseme
|
f17ea7f146 |
fix(heal): harden replacement rebuild tracking (#5892)
* fix(heal): gate auto replacement formatting Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): require replacement target outcomes Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): bind resumes to replacement targets Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): fence healing marker ownership Co-Authored-By: heihutu <heihutu@gmail.com> * test(heal): cover replacement target completion Co-Authored-By: heihutu <heihutu@gmail.com> * docs(heal): clarify replacement recovery status Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): canonicalize replacement target checks Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): satisfy marker test module lint Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): scope automatic replacement format Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): require a mounted replacement target Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): avoid cloned ref slice in test Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): revalidate replacement before scanning Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): reset stale resume checkpoints Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): release scanner disk map before probing Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): persist replacement intent before format Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): fail closed on mountinfo read errors Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): fence replacement target identity Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): order replacement completion cleanup Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): atomically seal replacement completion Co-Authored-By: heihutu <heihutu@gmail.com> * test(heal): census replacement target shards Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): fence replacement recovery ownership Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): preserve replacement recovery anchors Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): satisfy replacement recovery lint gates Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): bind replacement identity to mount lease Co-Authored-By: heihutu <heihutu@gmail.com> * test(heal): cover durable replacement recovery states Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): validate persisted resume task identifiers Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): avoid blocking replacement marker CAS Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): report failed marker rollback Co-Authored-By: heihutu <heihutu@gmail.com> * test(heal): pin replacement resume schema compatibility Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): preserve durable recovery anchors Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): preserve public disk path semantics Co-Authored-By: heihutu <heihutu@gmail.com> * test(heal): use canonical replacement task ids Co-Authored-By: heihutu <heihutu@gmail.com> * test(heal): cover automatic replacement in 3x4 cluster Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): verify replacement target commits Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): persist replacement completion proof Co-Authored-By: heihutu <heihutu@gmail.com> * feat(heal): expose durable replacement status Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): bound durable replacement discovery Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): remove replacement readiness bypass Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): retry terminal replacement cleanup Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): isolate replacement intents from legacy resume Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): migrate legacy replacement intents at startup Co-Authored-By: heihutu <heihutu@gmail.com> * style(heal): apply strict clippy fix Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): prioritize active replacement recovery state Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): bind readiness to the admitted mount lease Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): atomically publish replacement intents Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): isolate replacement recovery directory Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): tolerate an empty recovery directory Co-Authored-By: heihutu <heihutu@gmail.com> * style(heal): remove redundant disk bytes conversion Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): reconcile proof-first replacement recovery Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): fence torn intent recovery Co-Authored-By: heihutu <heihutu@gmail.com> * test(heal): cover replacement migration conflicts Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): fence replacement lease mount identity Co-Authored-By: heihutu <heihutu@gmail.com> * test(heal): cover missing replacement path admission Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): reject conflicting legacy completion proof Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): fall back to proc mount identity Co-Authored-By: heihutu <heihutu@gmail.com> * feat(admin): expose replacement recovery status Surface the local durable replacement recovery snapshot in the background heal status response so operators can tell whether replacement cleanup is definitive or still pending. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): keep replacement status compatible Keep the existing background heal status response wire-compatible while retaining the Linux mount lease cleanup needed for the replacement recovery branch. Co-Authored-By: heihutu <heihutu@gmail.com> * style(ecstore): match linux mount lease formatting Keep Linux rustfmt output stable for the replacement mount lease comparison. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): qualify mount lease test constant Use the disk module path for the format config constant in the Linux mount lease regression test. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): keep procfd mount roots directory-safe Use a procfd path with an explicit directory component so Unix directory guards can open the replacement mount lease root with O_NOFOLLOW while preserving handle-relative I/O semantics. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): delete empty leased buckets via dirfd Use the held mount lease fd as the parent for non-force empty bucket deletion on Linux so procfd-rooted paths do not get rejected as BucketNotEmpty. Also make the download-part OpenOptions truncate behavior explicit and keep fsync test recording stable across procfd canonicalization. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): scan leased bucket paths for emptiness Use the local disk I/O root for bucket emptiness probes before non-force bucket deletion and table-bucket metadata checks. This keeps validation on the same mount instance as the subsequent local disk delete path. Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): align lease path test probes Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): block unsafe replacement recovery restarts Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): defer blocked replacement candidates Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): retry transient replacement discovery Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): keep transient recovery errors retryable Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): block corrupt legacy replacement state Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): classify flat replacement intent corruption Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): keep transient resume loads retryable Classify malformed legacy replacement state as blocking corruption while preserving disk and transient load failures for retry. This avoids permanently blocking replacement recovery on temporary storage errors. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): avoid latching transient legacy publishes Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): retry blocked legacy migrations Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): defer blocked startup recoveries Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): preserve disk sync limiter across lease roots Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Zhengchao An
|
be0cea83b7 | test(ecstore): pin persisted metadata key literals and bucket config goldens (#5904) | ||
|
cxymds
|
8f9633ee83 |
fix(rpc): negotiate authenticated file writes (#5880)
* fix(rpc): negotiate authenticated file writes * fix(rpc): share capability probe failures * test(rpc): cover dedicated capability route * fix(rpc): satisfy capability cache lints * fix(rpc): retry timed out capability probes Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: houseme <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
GatewayJ
|
70deb3284b | fix(select): pin object snapshot for query lifetime (#5835) | ||
|
cxymds
|
3b9c67e79b | fix(rpc): authenticate internode put file bodies (#5868) | ||
|
唐小鸭
|
3792fed827 |
fix(replication): madmin reset/diff wire compat and config validation (#5799)
* fix(admin): align replication-reset responses with madmin ResyncTargetsInfo shape
The replication-reset and replication-reset-status responses serialized
their shell as "Targets" and per-target fields in PascalCase, while
madmin-go ResyncTargetsInfo/ResyncTarget expect the "target" shell key
and lowercase field tags (arn/resetid/resyncStatus/replicationCount/
completedReplicationSize/failedReplicationCount/failedReplicationSize).
Go json decoding is case-insensitive per field, but Targets vs target,
Status vs resyncStatus and the size/count key names cannot match, so
mc replicate resync decoded empty results.
Rename the serde tags to the exact madmin wire shape, keep the
ResetBeforeDate/Error RustFS extension keys (unknown keys are ignored
by Go decoders), pin the shape with a snapshot unit test, and update
the e2e client DTO to decode the madmin shape.
* fix(admin): stream bare madmin DiffInfo documents from replication diff
POST /v3/replication/diff returned a single enveloped object
({Entries, IsTruncated, ScannedVersions}) while madmin-go
BucketReplicationDiff decodes the body with a json.Decoder loop over
bare DiffInfo documents. The envelope decoded as exactly one DiffInfo
with an empty object, so mc replicate diff printed a phantom empty row
instead of the real backlog.
Emit one DiffInfo JSON document per line by default, using the exact
madmin json tags (object/versionId/rStatus/deletemarker/lastModified;
Size stays as a RustFS extension key that Go decoders ignore). The
enveloped shape moves to the opt-in ?aggregate=true RustFS extension,
which remains the only carrier of scan-coverage metadata; a truncated
default-mode scan is surfaced via a warn tracing event instead of
in-stream. Pin both shapes with unit tests and tighten the e2e helper
to reject any envelope in the stream.
* feat(replication): validate replication config structure before persisting
PutBucketReplication accepted structurally invalid configurations that
MinIO's replication.Config.Validate rejects: empty or oversized rule
lists, duplicate or negative rule priorities, over-long rule IDs,
filters carrying more than one of Prefix/Tag/And, and delete marker
replication enabled on tag-filtered rules. Such configs persisted
silently and later produced undefined routing (e.g. ambiguous priority
ties) instead of failing the PUT.
Add validate_replication_config_structure as a pure function in
rustfs-replication (limits documented as constants), surface it through
the ecstore api facade, and run it first in the PUT capability gate so
defects are named before any metadata write. Missing Priority counts as
zero for the uniqueness check, matching Go's zero-value semantics. The
self-target rejection deliberately stays at set-remote-target, where the
endpoint is known; a config can never reference a self-pointing ARN.
Document the rule-level Destination.StorageClass contract (use the
remote target's storage_class instead) and renumber the acceptance
matrix e2e to unique priorities, which MinIO would also require.
* test(replication): pin duplicated wire types with boundary reconciliation tests
rustfs-filemeta (xl.meta disk format) and rustfs-replication (MRF/resync
persistence format) deliberately each own ReplicationStatusType,
VersionPurgeStatusType and ReplicationState; the boundary converts
between them via as_str(), whose From<&str> impls fall back to Empty on
unknown tokens — a variant added on one side silently degrades to Empty
on the other.
Add reconciliation tests in replication_filemeta_boundary: exhaustive
matches with no wildcard arm on both sides of both enums (a new variant
fails compilation until the mapping is reconsidered), string-token
round-trip asserts (a token the other side does not recognize fails
instead of quietly becoming Empty), and a full-field ReplicationState
round-trip. Cross-reference the tests from both type definitions.
Struct drift was already compile-guarded by the exhaustive struct
literals in the conversion functions.
* docs(replication): define split completion criteria and milestone sequence
The ecstore replication split plan had no completion measure — the
boundary scaffolding risked ossifying because nothing said when the
migration counts as done. Record the criteria in the module inventory:
done means the Required Contracts table's 'Current dependency to
remove' column is empty; the end state moves pool/resyncer/state into
crates/replication, with the boundary micro-files dissolving as code
crosses the crate line (batch-merging them beforehand is explicitly
rejected — the guard scripts anchor on their file names, so merging is
churn with zero functional gain; only datatypes.rs can retire early).
Sequence the remaining work as M2 (resyncer pure decision logic, after
the oversized function splits) → M3 (worker runtime, highest risk,
last) → M4 (retire boundaries and guard entries). Refresh the stale
first-step text — the event sink / runtime contracts already landed —
and update the split-plan status table accordingly.
* fix(replication): align structural validator with MinIO semantics after adversarial review
Three interop corrections found by adversarial review of the new
structural validator, plus review fallout fixes:
- Delete-marker replication is now rejected only for a direct Filter.Tag,
not for tags inside Filter.And — MinIO's validator only inspects the
direct tag, and mc replicate add --tags "k1=v1&k2=v2" (delete-marker
replication on by default) puts multiple tags into And.Tags, so the
stricter check rejected mc-generated configs MinIO accepts.
- Rule ID length is measured in bytes (Go len semantics), not chars —
a 255-char multibyte ID must not round-trip into a config MinIO
rejects.
- An empty <Tag/> element (no key) counts as absent, matching MinIO's
Tag.IsEmpty(); console form serializers emit empty tags, which would
otherwise trip the exactly-one-of and delete-marker checks.
Also: repair the store-uninitialized PUT test whose empty-rules fixture
now (correctly) fails structural validation before reaching the store
lookup; pin the previously untested startTime madmin key in the
reset-status shape test; and signal a truncated default-mode diff scan
via the x-rustfs-replication-diff-truncated response header — the bare
madmin stream has no envelope, so a truncated scan was otherwise
indistinguishable from a complete healthy one (madmin/mc ignore unknown
headers).
* test(e2e): activate SSE-S3 replication contract and pin resync fail-closed path
The SSE-S3 replication contract e2e was ignored under backlog#1291
(silent plaintext replication); the fail-closed gate in
replication_target_boundary.rs closed that hole, so the ignore reason
expired. Un-ignore the test — it now pins the current fail-closed
contract (FAILED status, failure event, readable encrypted source,
stable absence of all target versions), verified green.
Add test_bucket_replication_sse_s3_resync_stays_fail_closed: drives the
existing-object resync path (PUT ?replication-reset) over a FAILED
SSE-S3 object and asserts the resync generation reaches a terminal
state without ever materializing a target version, with the
stays-absent window also spanning fast-scanner heal cycles. The new
start_bucket_replication_reset helper doubles as the madmin
ResyncTargetsInfo shape assertion (target[0].arn/resetid) for the
reset-start response.
Refresh the stale nextest count commentary (the module is at 20 fast +
36 nightly = 56 tests by cargo nextest list; the SSE-S3-ignored note no
longer holds).
|
||
|
anthonymartin
|
706a8b6061 |
fix(scanner): publish bounded observational usage (#5742)
* fix(scanner): publish bounded observational usage * test(ci): serialize embedded integration ports * test(cache): isolate generation-change timeout * fix(scanner): address observational usage review Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: Anthony Martin <949506+anthonymartin@users.noreply.github.com> Co-authored-by: houseme <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
fc0de983d8 |
perf: add RPC auth profiling diagnostics (#5775)
perf: add rpc auth profiling diagnostics Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
唐小鸭
|
733c7b0f67 |
fix(replication): accept remote target healthCheckDuration nanoseconds (#5754)
* test(replication): accept madmin nanosecond healthCheckDuration payloads Red-phase TDD tests for P0-7: mc 'replicate add' sends the madmin default healthCheckDuration=60s as a Go time.Duration nanosecond integer (60000000000), which RustFS currently rejects as an unsupported field and would misread as seconds. Also pins the defensive seconds-or-nanos read for persisted bucket-targets metadata and the capability contract listing healthCheckDuration as writable. Currently failing (red): - remote_target_request_accepts_go_duration_wire_values - remote_target_request_accepts_legacy_seconds_health_check - remote_target_health_check_duration_is_declared_writable - bucket_target_reads_go_nanosecond_durations_defensively - runtime_capabilities_response_reports_missing_topology_before_storage_init * fix(replication): accept remote target healthCheckDuration nanoseconds mc 'replicate add' always sends the madmin default healthcheck-seconds=60 serialized as a Go time.Duration nanosecond integer (60000000000), so the default mc link-creation path (and 'mc replicate update') failed with InvalidRequest. Move healthCheckDuration from the unsupported to the writable remote-target field list; the capability contract in the runtime capabilities response follows the constants automatically. Fix the unit mismatch in both directions: - Request parsing and persisted bucket-targets reads decode the value defensively: below 10^7 it is legacy RustFS seconds, otherwise Go time.Duration nanoseconds (also covers MinIO-written metadata). totalDowntime shares the same wire shape and gets the same handling. - The list-remote-targets admin response re-encodes only these two fields as nanoseconds via a dedicated serialization path, leaving the persisted seconds-based wire format untouched for existing readers. The per-target health-check interval is accepted for mc compatibility but not yet applied; the heartbeat keeps its global env-configured interval, and the explicit 'healthcheck' update op stays rejected. disableProxy, edge, and edgeSyncBeforeExpiry remain explicitly rejected. |
||
|
Zhengchao An
|
98d3619613 |
fix: address rc.1 release blockers (#5648)
* fix: address rc.1 release blockers
* fix: route release guards through architecture boundaries
* fix: close remaining rc.1 regression gaps
* refactor: group multipart listing options
* fix: resolve rc.1 CI regressions
* fix(ecstore): keep bucket-config writes off the caller's stack
A bucket-config write nests incarnation resolution (which can drive legacy
migration and a peer fan-out), a full metadata load, and `save` — itself an
object PUT that pulls in the whole erasure write path. Every request that
mutates bucket config is already several futures deep, so inlining all of
that into one state machine overflows the 2MiB worker stack in debug builds.
Two CI lanes aborted with SIGABRT on this:
ILM Integration (serial)
rustfs app::lifecycle_transition_api_test::
compensation_driven_complete_multipart_upload_still_transitions
Test and Lint (swift)
rustfs-protocols::swift_metadata_persistence::
swift_metadata_writes_are_durable
Neither test file is touched by this branch and both lanes are green on
main. Stack-pointer probing showed ~780KiB consumed between
`metadata_sys::update` and the config read alone, with single hops of
363KiB (`update` -> `acquire_config_write_guard_for_incarnation`), 125KiB
and 105KiB.
Box the deep sub-futures on both read-modify-write paths (`update` /
`update_checked` and `update_config_with` / `update_config_with_checked`)
so each guard's own state machine stays small. Behaviour is unchanged;
`update` -> guard drops to 253KiB and both tests pass on the default stack.
* fix(lifecycle): unbreak restore under the bucket generation fence
The ILM lane aborted on a stack overflow before reaching these, so they
were never reported; with that fixed, four restore tests fail. All four
are green on main and none of their test files are touched by this branch.
1. RestoreObject and ListMultipartUploads hard-required
`opts.expected_bucket_incarnation_id`, but `apply_bucket_generation_guard`
deliberately leaves it unset when no guard extension is present — only the
S3 access layer installs one. Every direct caller therefore got
`InternalError: ... bucket generation guard is missing`. Resolve the
current generation instead, the way the copy path already does. The fence
is unaffected: RestoreObject still re-reads the incarnation from disk and
compares before admitting the restore, and the multipart listing is
filtered by the value it resolves.
2. `restore_expiry_snapshot_matches` (new on this branch) rejected every
restored-copy expiry whose `restore_expires` had not already elapsed.
Whether the restored copy is due to expire is the ILM evaluator's
decision, made when it emitted DeleteRestoredAction; re-deriving it in
the set layer only adds a way for a legitimate action to be rejected.
The stale-event risk it appears to guard is already covered by the
surrounding snapshot match — a re-restore rewrites `restore_expires`,
so a replayed event fails the equality check. Drop the clause; the
fifteen identity clauses are unchanged.
Fixed:
rustfs app::lifecycle_transition_api_test::
restore_object_usecase_accepts_exactly_one_of_two_concurrent_restores
restore_object_usecase_completes_suspended_null_version_in_place
restore_object_usecase_reports_ongoing_conflict
rustfs-scanner::lifecycle_integration_test serial_tests::
test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore
Verification: the CI ILM lane filter now runs 53/53 green locally.
* chore: address review follow-ups on this branch
Four items from the adversarial review that were still open.
- Restore the assertion `test_bucket_replication_replayed_delete_marker_
preserves_source_mtime_without_source_restart` is named for. The branch
had replaced the backlog#867 mtime check with `assert_replication_
converged`, which any successful replication satisfies, and deleted the
two helpers it needed — so the regression the test exists to catch would
now pass. This matters here specifically because the branch changes the
flag feeding `replication_delete_remove_options` and routes replay
through a new file and ordering.
- Drop `read_config_no_lock_preserve_empty`: zero production callers (the
one real consumer calls the `_with_metadata` variant directly). Its test
stanza now exercises that variant, so the coverage moves to live code
rather than being deleted.
- Revert the `bytesize` bump. It is a no-op: `Cargo.lock` already pinned
2.7.0 before this branch and is untouched, so the caret range already
resolved there. Nothing in the diff uses the crate.
- Split the AGENTS.md "Adversarial Validation" policy change out of this
branch. The edit is defensible on its own, but it relaxes the review gate
that this branch has to pass, so it should land as its own PR reviewed on
its own merits rather than bundled with the change that benefits from it.
The reverted hunks are unchanged and ready to re-apply.
Not changed, deliberately: the missing-sidecar path still fails closed.
`missing_bucket_incarnation_sidecar_for_new_metadata_fails_closed` pins
that on purpose, and serving a non-authoritative Object Lock state would
be the wrong trade. The residual concern stands and is recorded in review
— a crash between the two writes in `persist_new_and_set` leaves the
bucket unloadable until DeleteBucket+CreateBucket, and the repair branches
in `migrate_legacy_metadata` and `make_bucket` are unreachable dead code
for that case. Resolving it needs the read path and the (transaction-lock
holding) repair path to be separated, which is more than a follow-up edit.
* test(ci): serialize the new bucket-incarnation tests
The five tests this branch adds around the incarnation / lifecycle fence
drive `init_bucket_metadata_sys` and `bucket_metadata_sys_of` — process-global
OnceLock state that `serial_test`'s `#[serial]` cannot protect across
nextest's process boundary — and they delete+recreate buckets, the shape that
raced into InsufficientWriteQuorum in backlog#937.
Add them to the `ecstore-serial-flaky` group in both the default and ci
profiles (nextest evaluates a named profile's own overrides list, so the
ci mirror is required). Preventive serialization only, no retries.
Not a full fix for the review comment: `bucket_delete_waits_for_config_
mutation_fence` still proves liveness with a fixed 200ms sleep plus
`assert!(!delete.is_finished())`. Turning that into readiness polling needs
a production-side signal to wait on — asserting "still blocked" is inherently
a negative. Serializing the group removes the parallel-load pressure that
makes the window fragile; the sleep itself is left for a follow-up.
* test(ecstore): pin that a drained bucket is actually deletable
`DeleteBucket`'s emptiness check is `has_xlmeta_files`, a raw scan of the
bucket directory on local disks — not an S3-level listing. So "the client
drained the bucket" and "the bucket is deletable" are two different
contracts, and only the first one was covered.
That gap is what the `S3 Implemented Tests` lane is failing on: 219 cases,
all `BucketNotEmpty` on `nuke_prefixed_buckets`, with every test body
passing. The first one is `test_versioning_obj_suspend_versions`, reported
by pytest as PASSED followed by ERROR at teardown.
Add the missing assertion for the unversioned path: PUT, client DELETE,
then assert no `xl.meta` survives and `DeleteBucket` succeeds. It passes —
which is itself a result: the plain delete path leaves no residue, so the
s3-tests failure is not there.
The versioning-suspended path is the remaining suspect (the client DELETE
leaves a null delete marker, and draining means purging it by
`versionId=null`). It is not covered here: `BucketVersioningSys` resolves
through the ambient `get_bucket_metadata_sys()` OnceLock, which this unit
env cannot set, so the bucket never actually reports as suspended. That
repro belongs at the e2e layer where a real server owns the versioning
state.
* fix(ecstore): let an explicit null-version delete purge its delete marker
Root cause of the `S3 Implemented Tests` lane: 219 cases, all
`BucketNotEmpty` on `nuke_prefixed_buckets`, every test body passing.
On a versioning-suspended bucket a client DELETE leaves a null delete
marker — correct S3 semantics, and an `xl.meta` on disk. Draining the
bucket therefore means purging that marker as `?versionId=null`, which is
what `nuke_bucket` does before `DeleteBucket`. That purge was rejected:
explicit null-version purge of the null delete marker must succeed,
got [Some(MethodNotAllowed)]
so the marker survived, and `DeleteBucket`'s emptiness check — a raw
`has_xlmeta_files` scan of the bucket directory, not an S3 listing — kept
reporting the bucket as non-empty.
The two sides of the version comparison in the batch delete loop are in
different namespaces. `goi.version_id` is the client-facing identity, where
`from_file_info` synthesizes `Some(Uuid::nil())` for a null version on a
versioned *or versioning-suspended* bucket. `version_id` is the storage
identity, where `delete_file_info_version_id` maps an explicit
`?versionId=null` to `None`. Comparing them raw makes the purge look like a
version mismatch, so `explicit_delete_marker` is false and the
`MethodNotAllowed` from the lookup is recorded as a delete failure.
This only became reachable on this branch: previously `check_opts` did not
carry `dobj.version_id`, so `set_disk_delete_creates_delete_marker` was
true, `object_lock_check_required` was false, and the lookup that produces
`MethodNotAllowed` never ran. Adding the version id to `check_opts` lit up
a comparison that was already wrong.
Normalize both sides through `delete_file_info_version_id`.
The regression test injects a real Suspended bucket-config snapshot — the
delete path reads versioned/suspended from that snapshot, not from `opts`,
so without it `from_file_info` never synthesizes the null version id and
the branch is not reached. Mutation-checked: restoring the raw comparison
fails the test with the exact `MethodNotAllowed` above.
* fix(app): drop the now-needless struct update
Reverting `crates/replication` to main removed the extra `MrfReplicateEntry`
fields, so this literal specifies every field again and `..Default::default()`
trips `clippy::needless_update` under `-D warnings`.
Caught by CI, not locally: I had run `cargo check --workspace --all-targets`,
which does not see clippy-only lints. Ran `cargo clippy --workspace
--all-targets -- -D warnings` here — clean.
* test(e2e): assert the fresh-volume classification
four_node_empty_legacy_volumes_start_as_fresh only started the cluster and
listed buckets — no assertion, so any classification path that still permits
startup left it green without proving the pre-created empty `.minio.sys`
directories were treated as fresh volumes.
Pin what that classification actually leaves behind: no buckets adopted into
the namespace, `.rustfs.sys/format.json` written on every drive, and the empty
legacy directory left untouched rather than migrated into.
* fix(bucket): apply the requested Object Lock to existing buckets
Site replication replays make-with-versioning against the destination,
carrying the source's `lockEnabled`. When the destination bucket already
exists it takes `force_create`, and the whole option-application block was
gated on `confirmed_missing` — so the call returned success while the replica
stayed unlocked. Replicated versions could then be deleted without the
retention the source enforces.
Object Lock enable is one-way, so applying it to an existing bucket is safe:
move it out of the creation-only gate, keeping `created` and versioning-only
options creation-scoped as before.
An existing authoritative bucket takes the `cache_bucket_metadata_in` branch,
which only caches, so the enable would have been dropped on restart. Persist
instead when the enable actually changed something.
Mutation-checked: restoring the creation-only gate fails the new
`force_create_enables_object_lock_on_an_existing_bucket` with "Object Lock
must be enabled on the existing bucket".
cargo nextest run -p rustfs-ecstore --lib: 3633 passed.
* fix(ecstore): box the generation-checked config mutation paths too
The earlier stack fix boxed `update` and `delete`, but an authorized
bucket-config mutation carrying an incarnation takes `update_if_incarnation`
/ `delete_if_incarnation` instead — which were still inlining the whole
resolve/load/save chain into an already-deep request future. Same overflow,
sibling path.
* fix(restore): keep the nil-version normalization the strip removed
Reverting the replication subsystem to main took `set_disk/replication.rs`
with it, but one line in that file was this branch's own fix rather than
replication work:
- self.version_id.filter(|v| !v.is_nil()) == fi.version_id.filter(|v| !v.is_nil())
+ self.version_id == fi.version_id
For a versioning-suspended object the expected version is `Some(Uuid::nil())`
while the read-back `FileInfo` carries `None`, so the raw compare reports
every suspended restore as "restored object changed before restore metadata
finalization" and the copy-back never commits. Same nil-vs-None mismatch as
the null delete-marker purge fixed earlier on this branch.
Caught by `Test and Lint (rio-v2)`, not by my local runs: the test lives in
`transition_commit_failure_tests`, gated behind `feature = "test-util"`, so
the 3633-test suite I had been running never included it. Re-ran with
`--features rio-v2,test-util`: 3722 passed.
|
||
|
cxymds
|
380ec74ece |
fix(replication): persist force-delete handoff state (#5641)
* fix(replication): persist force-delete handoff state * fix(arch): route force-delete config access through boundary * style: format force-delete imports --------- Co-authored-by: Zhengchao An <anzhengchao@gmail.com> |
||
|
houseme
|
035ce5d784 |
feat(obs): add bounded metrics dimensions (#5645)
* feat(obs): add drive topology detail metrics Expose additive drive info, topology, state, and per-drive API metrics while preserving the existing drive metric label sets. Backlog: rustfs/backlog#1655 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): preserve suspect drive runtime state Keep suspect as a bounded drive runtime state and avoid all-zero runtime_state samples for that storage health state. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): skip unknown drive inode samples Avoid exporting zero inode gauges for missing or stale drive snapshots and ignore zero-count API latency buckets. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add scanner source work detail metrics Expose additive scanner source and cycle work metrics with bounded server/source/state labels while leaving the existing aggregate scanner metrics unchanged. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add ilm action detail metrics Expose additive ILM action/state task metrics with a server label while preserving the existing aggregate ILM series. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add delivery target server metrics Expose additive audit and notification delivery target metrics with server labels and extend removed-target tombstones for the server-aware series. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add replication target flow metrics Expose additive bucket replication target sent and failed-flow metrics while preserving existing bucket aggregates and target backlog series. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add request server metrics Expose additive API request metrics with server labels while preserving the existing request and traffic metric label sets. Co-Authored-By: heihutu <heihutu@gmail.com> * style(obs): apply rustfmt to metrics changes Apply rustfmt output to the metrics dimension changes without altering behavior. Co-Authored-By: heihutu <heihutu@gmail.com> * style(obs): reuse audit target label constant Use the exported audit target_id label constant for legacy audit target metrics. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): populate drive disk metrics Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add scanner bucket drive result metrics Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add replication proxy server metrics Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): address metric liveness review Use checked division for drive API latency aggregation and keep recovered drive, scanner current-cycle, replication flow, audit target, and notification target series from retaining stale values. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): address metric dimension review Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): address additional metric review Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): count drive calls at start Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): address metrics dimension review Co-Authored-By: heihutu <heihutu@gmail.com> * fix(metrics): address dimension review gaps Co-Authored-By: heihutu <heihutu@gmail.com> * fix(metrics): address scanner review follow-ups Co-Authored-By: heihutu <heihutu@gmail.com> * fix(metrics): address runtime review follow-ups Co-Authored-By: heihutu <heihutu@gmail.com> * fix(metrics): reduce disk metric contention Co-Authored-By: heihutu <heihutu@gmail.com> * fix(metrics): address runtime review follow-ups Co-Authored-By: heihutu <heihutu@gmail.com> * fix(metrics): retire stale dimension series Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
cxymds
|
114b2420a2 |
feat(admin): expose versioned replication capabilities (#5631)
* feat(admin): expose replication capabilities * fix(admin): route replication capabilities through facades |
||
|
cxymds
|
2cc7443067 |
feat(replication): bound DeleteObjects queue admission (#5637)
feat(replication): batch DeleteObjects queue admission |
||
|
cxymds
|
c1955a8498 | fix(replication): harden live delete admission (#5599) | ||
|
houseme
|
fbec33bd29 |
Expose target-scoped durable MRF backlog metrics (#5584)
* feat(replication): expose target durable mrf backlog Add target ARN attribution to durable MRF entries and surface target-scoped durable backlog metrics without changing existing bucket-only metric labels. Keep legacy MRF files bucket-only by defaulting missing targetARNs to an empty list, and expose target snapshots through an additive API so existing DurableMrfBacklogSummary callers remain source-compatible. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(replication): expose runtime target backlog (#5586) Track runtime replication backlog by target ARN for regular, large, delete, and MRF admission paths while preserving the existing bucket-level backlog semantics. Add target-scoped current backlog metrics and merge them with durable target backlog snapshots for observability. Co-authored-by: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
唐小鸭
|
c8016cbcdb |
fix(replication): enforce bucket replication switches (#5449)
* fix(replication): enforce bucket replication switches * fix(replication): satisfy delete admission clippy lint * fix(replication): restore MinIO tag filter behavior --------- Co-authored-by: Zhengchao An <anzhengchao@gmail.com> Co-authored-by: cxymds <cxymds@gmail.com> |
||
|
houseme
|
da389c0e21 |
fix(replication): harden backlog observability (#5564)
Add RAII guards for replication runtime backlog tickets so active worker and queue counters unwind on every terminal path. Expose node-local MRF pending, dropped, missed, and flush-failure metrics through the bucket replication Prometheus collector while keeping the existing current backlog and durable MRF gauges additive. Update durable MRF summary maintenance to aggregate incrementally during the persister loop, avoiding repeated full-entry scans on each successful flush. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
houseme
|
62d44d10b8 |
Expose replication backlog gauges (#5557)
* fix(replication): count backlog at queue admission Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): expose bucket replication backlog gauges Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): report recent backlog from queued work Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): preserve legacy backlog metric semantics Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): expose durable MRF backlog gauges Co-Authored-By: heihutu <heihutu@gmail.com> * test(obs): cover replication backlog metric scope Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): keep backlog metrics API-compatible Co-Authored-By: heihutu <heihutu@gmail.com> * refactor(obs): streamline replication backlog metrics Keep MRF backlog accounting and OBS metric collection on a single, cheaper path. Co-Authored-By: heihutu <heihutu@gmail.com> * test(kms): update aws capability snapshot Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
4e34f97dd7 | feat(kms): converge KMS configuration across nodes after a runtime change (#5551) | ||
|
cxymds
|
c09d11ff3b |
fix(config): fence persisted config updates and reloads (#5512)
* fix(config): fence persisted config updates and reloads * fix(ci): unblock config and e2e checks Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: houseme <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
唐小鸭
|
ad7663afd1 |
refactor(sse): decouple ecstore and harden KMS lifecycle (#5435)
* refactor(sse): decouple encryption from ecstore * feat(kms): enhance KMS service manager with runtime state and persistence support * feat(kms): add local key export functionality for SSE-S3 migration tests * fix(kms): keep local key export narrowly scoped * fix(sse): validate copy source customer algorithm --------- Co-authored-by: Zhengchao An <anzhengchao@gmail.com> |
||
|
Zhengchao An
|
719c0d6ef0 | feat(rpc): add replay-scoped internode authentication (#5455) | ||
|
cxymds
|
0247c48ce0 |
fix(tiering): bind recovery to transaction metadata (#5409)
* fix(tiering): bind recovery to transaction metadata * fix(tier): add operator transition reconciliation (#5410) * fix(tier): add operator transition reconciliation * fix(tiering): require live fleet capability proof (#5423) |
||
|
Zhengchao An
|
88fa3877c1 |
fix(ecstore): serialize every bucket config write under the transaction lock (#5445)
Only replication-target writes took the bucket transaction lock. Every other
config write (policy, tagging, lifecycle, versioning, ...) went straight to
the process-local metadata-system guard, which serializes nothing across
nodes.
Each config write is a read-modify-write of one whole BucketMetadata blob:
load the blob, replace one field, save the blob back. The namespace locks
inside read_config/save_config are taken and released separately, so they do
not span that cycle. Two nodes updating different config files of the same
bucket therefore both load the same blob, each set their own field, and the
later save drops the other's -- with both clients already told 2xx. This is
not last-writer-wins on one document; an orthogonal config silently vanishes.
Route update(), delete() and update_config_with() through
acquire_config_write_guards(), which takes the cluster-wide transaction lock
first and the metadata-system write guard second. That order is load-bearing:
taking the process-local guard first would park every local reader and writer
of every bucket behind a lock whose holder may be another node, turning
remote contention into a local stall.
The lock is per bucket rather than per config file, since a per-file key
would let exactly the offending pair run concurrently. Rename the helper to
acquire_bucket_metadata_transaction_lock to match, but deliberately keep the
lock resource string as "bucket-targets/{bucket}/transaction.lock": the key
is what nodes agree on, so renaming it would leave a mixed-version cluster
with two disjoint keys and stop old and new nodes from excluding each other
on the very writes that are serialized today.
update_config_with() already narrowed its staleness window to a single load
and save, but its exclusion was explicitly process-local; it is now
cluster-wide, so its doc comment no longer disclaims cross-node races.
Also make update_and_parse load through self.api instead of the ambient store
handle, so the read and the write of one read-modify-write cannot resolve to
different instances.
The new tests drive two BucketMetadataSys instances over one ECStore -- the
in-process stand-in for two nodes, since they share no RwLock and can only be
serialized by the namespace lock. Verified the lost-update test has teeth by
removing the lock and confirming it fails on round 0, with the tagging config
clobbered to empty by the concurrent policy write.
|
||
|
Zhengchao An
|
a7f035a8c3 |
fix(ecstore): fail peer metadata reloads closed instead of caching fabricated defaults (#5396)
The LoadBucketMetadata peer-notification handler loaded bucket metadata with the fabricating loader (ConfigNotFound -> BucketMetadata::new) and unconditionally cached the result. On a transient read-quorum dip during a reload notification, a peer cached an authoritative "no Object Lock" default for a lock-enabled bucket, disabling the batch-delete retention gate (object_lock_delete_check_required) on that node until the next refresh, and wiping its bucket-target/durability sync state. Production changes: - New BucketMetadataSys::reload_from_store (metadata_sys:: reload_bucket_metadata): the peer reload path uses the presence-aware loader and installs only metadata actually read from persisted storage. A load miss returns an error (surfaced to the notifying peer as success=false) and leaves the cache untouched; deletion still propagates only through the dedicated DeleteBucketMetadata notification. - The reload runs under the outer metadata-sys write guard, load included, mirroring update(): every other cache installer holds that lock, so a reload snapshot can never land after - and roll back - a newer concurrent install (the stale-load lost-update from the review), and the install-plus-registry-sync sequence stays atomic against concurrent removes and reloads (previously only the set call was write-guarded, with the load outside any lock). - The peer-visible miss error is a fixed string: the notifying peer substring-matches error text against network-failure needles (is_network_like_error), so interpolating a bucket name (e.g. a legal bucket literally named "unavailable") could mark a healthy peer offline. - get_config's lazy insert routes through set(), picking up the negative-cache invalidation. An earlier draft instead guarded set() with a per-config updated_at freshness comparison. Adversarial validation rejected it (three roles independently): update_config stamps with the handling node's wall clock, so within the skew the cluster already tolerates (+/-300s RPC auth window) a config rewritten with an earlier stamp - e.g. revoking a public-read policy through a second node, or any same-field rewrite after an NTP step-back - would be skipped by every peer forever, silently pinning the revoked permissive config with no re-convergence path (the 15-minute refresh also routed through the guard). Race staleness is second-scale while skew is minute-scale, so no tolerance bound can separate them; the write-guard serialization closes the same race without clocks and preserves the refresh loop's unconditional converge-to-disk property, which is the cluster's self-healing mechanism. Startup audit (BucketMetadataSys::init): concurrent_load's insert-if-vacant still installs a fabricated default when a transient miss hits at boot - indistinguishable from a legacy bucket without a metadata file at this layer - bounded by the next successful persisted load. Making the object-lock gate fail closed on such entries is filed as a follow-up, alongside the bare "unavailable" needle in is_network_like_error and the Swift cache-only metadata writes. Tests: - bucket::metadata_sys::tests:: peer_reload_never_caches_fabricated_defaults_as_authoritative: miss installs nothing / miss keeps the existing entry intact (asserting the dedicated non-persisted error) / persisted reload converges the cache over a stale entry. - node_service::tests:: test_load_bucket_metadata_failure_skips_scanner_maintenance: a failed reload reports failure and does not advance scanner maintenance activity (previously recorded even on a miss). - The handler success path stays uncovered at the RPC layer (needs an isolated global object layer, like the pre-existing ignored test); the composition is pinned at the sys level instead. Verification: - cargo fmt --check and cargo clippy --lib --tests clean on rustfs-ecstore and rustfs. - Targeted suites green; full cargo test -p rustfs-ecstore --lib: 3198/3200 with two parallelism-sensitive lock-test flakes from the known baseline (pass in isolation; a different pair flakes per run). - Adversarial validation (high-risk tier, all seven roles as independent parallel reviewers) run per AGENTS.md; all findings fixed or rebutted with evidence, three out-of-scope findings filed as follow-up tasks. Co-authored-by: houseme <housemecn@gmail.com> |
||
|
cxymds
|
d9efd6b853 |
feat(ecstore): add remote snapshot lease RPCs (#5389)
* feat(ecstore): add local snapshot leases * feat(ecstore): add remote snapshot lease RPCs * fix(rpc): keep snapshot lease checks CI-compatible |
||
|
Zhengchao An
|
957080bea5 |
fix(swift): persist container and account metadata writes (#5398)
Swift container and account metadata handlers cloned the cached BucketMetadata, set the tagging fields, and called set_bucket_metadata, which only updates the in-memory cache map. Nothing reached .metadata.bin, so every Swift metadata POST was lost on restart and silently overwritten by the next disk-truth reload (a peer LoadBucketMetadata notification or the 15-minute refresh loop) — while the client had already been told 2xx. Route these writes through a new metadata_sys::update_config_with: a read-modify-write that loads the on-disk metadata and persists the result under the same write guard metadata_sys::update uses, so the rewrite merges against disk truth instead of a possibly stale cache and cannot clobber a concurrent update to another config file. Peers are notified afterwards, matching the S3 config handlers. Persisting these writes required hardening the paths that now produce durable state: - Account metadata writes validate account ownership. This metadata holds the account's TempURL signing key, so an unauthenticated write for someone else's account would have become a durable, cluster-wide takeover of that account's pre-signed URLs. Reads stay open because TempURL signature validation runs before credentials exist. - disable_versioning verifies the container exists. Without it the metadata loader's "no metadata on disk" default would be persisted, creating an orphan metadata file and caching a fabricated default as authoritative. - Container and account metadata are size- and count-limited, reusing the Swift limits object metadata already enforces; these tags land in the bucket metadata file that every later config write rewrites whole. - A rewrite refuses to run when the persisted tagging config is unreadable, instead of merging onto an empty set and wiping the container ACL and versioning tags. It reports 409 naming the remedy. - Storage errors are logged in full and reported generically, since they now carry real disk and quorum detail. The tagging arm of BucketMetadata::update_config also clears the parsed config, as the lifecycle arm does: parse_all_configs skips empty XML rather than clearing, so a cleared config kept serving the old tags. Tagging is serialized with the S3 XML serializer the loader can parse back, not quick_xml, whose output was never round-trippable. |
||
|
cxymds
|
b432f31c2c |
fix(multipart): preserve retried parts on quorum failure (#5363)
* fix(multipart): preserve retried parts on quorum failure * style(multipart): format transaction rollback * fix(proto): regenerate multipart transaction RPCs * fix(multipart): import rollback marker constant * fix(multipart): export transaction action * fix: import multipart transaction test requests |
||
|
Henry Guo
|
d5df66ac4f | fix(scanner): require authoritative usage snapshots (#5333) | ||
|
Zhengchao An
|
61d4e04d65 |
feat(rpc): bind canonical body digest into internode mutating disk RPC signatures (#5234)
* feat(rpc): bind canonical body digest into internode mutating disk RPC signatures Binds a domain-separated, length-prefixed canonical request-body digest into the v2 HMAC signature scope for every mutating NodeService disk RPC, so an on-path attacker on the default-plaintext internode channel can no longer tamper with a mutation payload (or strip the msgpack `_bin` field to force the JSON fallback decode) without invalidating the signature. Covers 13 mutating disk RPCs: RenameData, DeleteVersion, DeleteVersions, WriteMetadata, UpdateMetadata, WriteAll, Delete, DeletePaths, RenameFile, RenamePart, DeleteVolume, MakeVolume, MakeVolumes. The digest covers both the msgpack `_bin` payloads and their JSON compatibility copies. Gated fail-open by default (RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT) with a convergence counter, so rolling upgrades are byte-for-byte unaffected; the replay-cache capacity is now configurable and overflow fails closed with a metric. Refs https://github.com/rustfs/backlog/issues/1327 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(rpc): satisfy architecture-migration compat-marker guard Put the removal condition on the RUSTFS_COMPAT_TODO marker line itself, and stop backticking env-var/metric names in the cleanup-register entry so the guard's id extractor only sees the task-id. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
houseme
|
6974963e20 |
feat(ilm): add durable manual transition job store (#5229)
* feat(ilm): add manual transition job route contract Refs #1479 Co-Authored-By: heihutu <heihutu@gmail.com> * feat(ilm): add durable manual transition job store Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ilm): harden durable transition job cancellation Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Henry Guo
|
a63b79004c |
fix(scanner): make distributed usage convergence authoritative (#5151)
* fix(scanner): make distributed usage cycles authoritative * fix(scanner): close distributed refresh races * fix(config): align scanner reload integration * fix(admin): scope config test helpers * fix(scanner): harden distributed usage convergence * fix(scanner): preserve rolling activity compatibility * fix(admin): expose non-secret optional config values * fix(scanner): acknowledge distributed dirty usage * fix(ecstore): make bucket mutations cancellation safe * fix(scanner): preserve pending dirty acknowledgements * test(obs): account for superseded scanner metric * fix(api): reject excess detached bucket mutations * test: close scanner convergence coverage gaps * fix(scanner): make path tracking cleanup one-shot --------- Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com> Co-authored-by: houseme <housemecn@gmail.com> |
||
|
houseme
|
2ee111ad8b |
feat(ilm): add durable manual transition jobs (#5223)
Refs #1479 Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
cxymds
|
7320d7fab2 | fix(replication): make resync starts atomic (#5215) | ||
|
cxymds
|
45b675c641 | fix(replication): report authoritative backlog metrics (#5209) | ||
|
cxymds
|
358caa23cb | fix(storage): expose truthful storage class capabilities (#5172) | ||
|
houseme
|
6765aca3f9 |
feat(ilm): add manual transition run endpoint (#5171)
* feat(ilm): report manual transition backfill outcomes Add scoped lifecycle transition backfill reporting for backlog #1478 and expose enqueue outcomes needed by #1479 without changing the existing scanner/compensation bool API. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(admin): add manual transition run endpoint Add a bounded POST /rustfs/admin/v3/ilm/transition/run API for backlog #1477 and cover the route, policy, query parsing, and partial status contract needed by #1481. Console operations from #1480 are intentionally left for a later client integration. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(ilm): allow manual transition resume markers Accept additive marker and versionMarker parameters on the bounded manual transition run API so clients can continue from a partial report without changing the existing default scan behavior. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ilm): harden manual transition partial reports Preserve the null-version cursor contract, stop manual scans on enqueue pressure without skipping the failed object, and keep raw resume markers out of admin JSON responses. Co-Authored-By: heihutu <heihutu@gmail.com> * test(ilm): add manual transition e2e coverage Co-Authored-By: heihutu <heihutu@gmail.com> * perf(ilm): keep transition enqueue hot path direct Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
cxymds
|
1655f3192e |
fix(notify): unify runtime lifecycle coordination (#5088)
* fix(notify): unify runtime lifecycle coordination * fix(notify): repair lifecycle convergence checks * fix(admin): expose effective notify state (#5097) |
||
|
houseme
|
937b311316 |
fix(tier): lock tier config mutations (#5080)
* fix(tier): lock tier config mutations Co-Authored-By: heihutu <heihutu@gmail.com> * fix(tier): add mutation RPC auth contract (#5082) Co-authored-by: heihutu <heihutu@gmail.com> * fix(tier): add peer mutation handler core (#5084) Co-authored-by: heihutu <heihutu@gmail.com> * fix(tier): add mutation control rpc service (#5087) Co-authored-by: heihutu <heihutu@gmail.com> * fix(tier): recover prepared mutation drains (#5093) Recover prepared tier mutation intent records into the local tier runtime so a restarted peer fails closed before issuing new remote-tier operation leases or conflicting admin publishes. Reconcile the recovered block map on each scan so committed, aborted, or removed intents clear stale local blocks instead of wedging the peer until process restart. Co-authored-by: heihutu <heihutu@gmail.com> * fix(tier): prove zero references before tier removal (#5092) Signed-off-by: houseme <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> * fix(tier): clear peer mutation runtime blocks (#5094) Install prepared mutation runtime blocks when peer prepare requests are applied or replayed so followers fail closed immediately before restart recovery. Clear the in-memory block once peer commit or abort reaches a durable terminal state, including delayed duplicate prepare requests that observe a committed or aborted record. Co-authored-by: heihutu <heihutu@gmail.com> --------- Signed-off-by: houseme <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
cxymds
|
4f133eb95f | feat(tiering): add Wasabi lifecycle target support (#5057) | ||
|
cxymds
|
28fdcc87be |
fix(tiering): make rejected upload cleanup durable (#5059)
* fix(tiering): make rejected upload cleanup durable * fix(tiering): close transition upload cancellation gap * test(tiering): cover failed upload without candidate * test(tiering): synchronize cancelled cleanup recovery * test(tiering): stabilize cancelled cleanup recovery Prefer cancellation when the tier delete journal recovery worker is racing an immediate tick, and build the cancelled-cleanup regression store with an already-cancelled token so production recovery cannot consume the test journal. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: houseme <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
cxymds
|
908ca548bb |
feat(heal): gate control capability by cluster topology (#4994)
* fix(rpc): bind internode auth to exact targets * fix(heal): initialize the runtime atomically * fix(heal): aggregate status across cluster nodes * fix(heal): return canonical tokens for duplicate starts * feat(heal): add authenticated control RPC contract * feat(heal): gate control capability by cluster topology * fix(heal): return canonical tokens for duplicate starts (#4992) --------- Co-authored-by: Zhengchao An <anzhengchao@gmail.com> |
||
|
cxymds
|
133499c2d5 | fix(rpc): bind internode auth to exact targets (#4988) | ||
|
houseme
|
21049401fa |
fix(ilm): harden tier transition failure boundaries (#5031)
* fix(tier): fence generation-scoped operations Refs rustfs/backlog#1354 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ilm): verify transition upload streams Refs rustfs/backlog#1353 Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): expand transition fault matrix Refs rustfs/backlog#1355 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |