* feat(common): add MRF intent channel and Mrf request source (HS-01)
Introduce the producer-facing half of the mission repair feed: a global
bounded (8192) channel carrying lightweight MrfIntent values from IO
error paths, plus the RUSTFS_HEAL_MRF_ENABLE delivery kill-switch and
config constants for queue/journal sizing. Delivery is strictly
non-blocking (try_send, drop-on-full) so it can sit on decode-failure
and partial-write paths without adding latency. HealRequestSource grows
a 'mrf' variant so admission accounting can attribute replayed intents.
Part of backlog#1865 (option a: wire HealEvent-style intents with a
durable retry ledger).
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(heal): add MRF queue, durable journal, and intent consumer (HS-01)
Consumer half of the mission repair feed: a bounded pending queue
(100k intents / 8 MiB dual ceiling, drop-newest on overflow), a durable
journal at buckets/.heal/mrf/journal.bin holding the unaccepted pending
snapshot, and a consumer task that batches intents off the global
channel, translates them into prioritized heal requests (decode
failure -> Urgent ECDecode, metadata corruption -> High Metadata,
partial write -> Normal object heal), and retries full admissions with
a 5s backoff and a 3-attempt ceiling.
Durability: every journal record carries its own CRC32 and a
format/version header, so a torn tail truncates cleanly at replay; the
journal is deleted after a successful replay and when the pending set
drains (mirroring MinIO's post-replay list.bin unlink). Losing the last
500 ms flush window is acceptable: replayed duplicates merge via the
manager dedup key and read-repair remains the safety net.
Metrics: rustfs_heal_mrf_queue_depth/_queue_bytes, _dropped_total
{reason}, _replayed_total, _journal_bytes, _journal_fsync_total.
The consumer is wired at heal runtime bootstrap right after manager
start, honoring RUSTFS_HEAL_MRF_ENABLE (default on, rollback = off).
Tests: unit tests for the dual ceiling, record roundtrip, torn-tail
truncation, and the priority mapping; integration tests against a real
4-disk ECStore proving channel intents reach the manager queue as
Urgent/mrf-attributed requests and journal replay arms intents, drops
torn tails, and removes the file.
Part of backlog#1865 (option a).
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ecstore,scanner): deliver MRF intents from error paths (HS-01)
Wire the three production delivery points, each a single non-blocking
try_send next to the existing in-memory heal paths, which stay as the
fast path:
- read.rs decode-error branch: DecodeFailure intent beside the existing
read-repair submit, so an Urgent ECDecode request survives restarts
even when the Low-priority read-repair request was dropped or lost.
- add_partial: PartialWrite intent, giving partial-write recovery a
durable Normal-priority object heal across restarts.
- scanner_folder metadata-corruption classification: MetadataCorruption
intent beside the existing High-priority scanner heal request.
All three are on error paths only: zero cost on healthy IO.
Part of backlog#1865 (option a).
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix: include mrf heal source counts
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix: keep node heal status wire compatibility
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Add an experimental fixed-count foreground PutObject admission gate for #1882 Phase 0 validation. The gate is default-off, returns SlowDown before body ingest when saturated, and keeps the admission permit with the spawned store commit owner until store PUT returns.
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(heal): track erasure set progress baseline
Record erasure-set heal byte progress from per-object results and seed progress totals from complete usage-cache snapshots when available.
Keep usage-cache failures observational so heal execution continues without a baseline.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(heal): skip filtered erasure set versions
Skip erasure-set versions written after the durable heal start time, and queue lifecycle-expired versions for expiry before skipping them.
Track new-version and ILM-expired skips separately so progress can explain completed baseline work without treating these skips as retry-blocking failures.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(heal): wire abandoned data-dir cleanup check
Connect check_abandoned_parts through ECStore, pool, and set layers so heal can invoke the existing orphan data-dir reclaim path instead of returning NotImplemented.
Add dry-run support to the reclaim scan and cover dry-run plus scoped set behavior with regression tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(obs): add heal scanner trace bus
Introduce an in-process broadcast trace bus with typed heal and scanner events, lazy event construction, and bounded lagged-subscriber behavior.
Cover zero-subscriber publishing, subscription delivery, drop accounting, and lagged receivers with focused common-crate tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(obs): stream heal trace events from admin API
Wire the admin trace endpoint to the common trace bus for heal/scanner events, including kind, regex, and threshold filtering.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(obs): emit heal trace events
Publish heal task lifecycle and abandoned-parts cleanup events through the common trace bus so the admin trace stream has live heal diagnostics.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(obs): emit scanner trace events
Publish scanner folder, lifecycle action, and heal-candidate events through the common trace bus for live admin scanner diagnostics.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(heal): route data usage loader through storage api
Keep ECStore data-usage facade access behind the heal storage_api boundary so architecture migration guards can validate the heal progress path.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(heal): avoid lifecycle snapshots on ordinary heal pages
Only request lifecycle object snapshots when the heal pass has lifecycle expiry context. This keeps ordinary listing and disk-walk pages from cloning FileInfo/ObjectInfo payloads while preserving the skip path that queues expired versions.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(heal): update bug-fix mocks for lifecycle snapshots
Carry the lifecycle snapshot opt-in argument through the remaining heal bug-fix test mocks so all-targets clippy covers the updated storage trait.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rustfs): sync heal storage mock signature
Update the rustfs storage RPC test mock for the lifecycle snapshot opt-in argument and cover it with rustfs all-targets clippy.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): allocate smoke ports across nextest processes
Serialize E2E port selection with a small /tmp allocator so nextest workers do not reuse the same just-released ephemeral port before RustFS binds it.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
feat(scanner): expose prefix-level bucket usage via admin API
The scanner's per-bucket, per-set usage caches already hold a path-keyed
prefix tree, but dui() flattened it only to bucket names — consoles and
operators had no way to ask "what does this prefix hold" without an S3
listing sweep (rustfs/backlog#1872, MinIO loadPrefixUsageFromBackend
parity).
Add:
- data-usage: prefix_usage_in_cache — a shared aggregation over the
entry map (arbitrary prefix, full counters, one-level sub-prefix
breakdown with names recovered from the literal-path cache keys),
hardened like the scanner's checked flatten: cycles, dangling child
links, over-deep trees, and overflowing counters yield None rather
than unbounded recursion or wrapped totals.
- ecstore: ECStore::all_set_disks — iterate every erasure set so a
query can read each set's own cache copy; the hash-routed store path
would always land on one set.
- scanner: bucket_prefix_usage — per-set loads (5s budget each, a slow
set degrades to not-reporting instead of stalling the caller),
merged across sets with partial/compacted/truncated flags, served
from a bounded 30s cache (128 entries, hard-capped) that bucket
writes invalidate through the dirty-usage hook.
- admin: GET /rustfs/admin/v3/usage/{bucket}?prefix=&max-entries=
behind the same any-of gate as datausageinfo (DataUsageInfoAdminAction
OR ListBucketAction), rejecting unknown query parameters and
clamping max-entries to 1..=10000. Route registered in the policy
table (deferred MultipleActions, matching datausageinfo) and the
route matrix test.
Closesrustfs/backlog#1872.
Co-authored-by: heihutu <heihutu@gmail.com>
feat(ecstore): pin bitrot algorithms with a startup self-test
A drifted HighwayHash implementation fails silently: every shard reads
back corrupt, heal rewrites healthy data, and cross-platform clusters
disagree about which copy is good. Mirror MinIO's bitrotSelfTest by
verifying, once at process start:
- known-answer digests for HighwayHash256S / HighwayHash256SLegacy over
a deterministic 4096-byte xorshift64* payload, plus the externally
verifiable FIPS SHA-256 "abc" vector guarding the HashAlgorithm
plumbing itself;
- an end-to-end roundtrip per streaming variant (encode -> size formula
-> bitrot_verify -> BitrotReader read-back), over full blocks and a
partial tail;
- tamper detection: one flipped byte in the final data block and one in
the leading hash must both be rejected as a hash mismatch, not by an
incidental read error.
The check costs microseconds and runs inline in
init_background_service_runtime before any shard can be written or
verified. Outcome surfaces as one structured bitrot_selftest log event,
the rustfs_bitrot_selftest_status gauge (1=passed / 0=failed / 2=skipped),
a bitrotSelftest field on the admin server-info response, and
RUSTFS_BITROT_SELFTEST_STRICT=on turns a failure into a startup error
(MinIO Fatal parity; the default only degrades the status so a bad build
cannot brick an existing fleet on upgrade).
Closesrustfs/backlog#1873 (HS-11).
Co-authored-by: heihutu <heihutu@gmail.com>
Use MemoryTrackedBytesStream directly as an s3s ByteStream so in-memory GET bodies avoid the generic StreamingBlob::wrap adapter while preserving exact remaining length, request lifecycle tracking, and length-mismatch failure semantics.
Co-authored-by: heihutu <heihutu@gmail.com>
backlog#1834 PR5. Whether the scanner, heal, audit and notify modules are on gets read from infra (storage helpers, node-service RPC) and from interface (admin handlers), but the switches lived in startup_background (composition) and server (interface). Every one of those reads was an upward edge carried in the layer-dependency baseline.
The env-derived scanner/heal predicates and the audit/notify state cells now live in rustfs/src/module_switches.rs, at the bottom of the layer order, so the same reads are ordinary downward edges. startup_background and server import from there; server keeps re-exporting the getters for its own consumers.
The issue's plan was to move is/refresh_audit/notify_module_enabled as a group. Moving refresh_* wholesale would have dragged resolve_audit_module_state and resolve_notify_module_state — server-side configuration logic — down into infra, which breaks more layering than it fixes. State and resolution are split instead: module_switches owns the atomics plus is_*/set_* accessors, and server's refresh_* keeps the configuration logic and publishes through the setter.
That leaves storage/helper.rs's test module importing refresh_* from server, so two infra->interface edges stay. Those tests assert that a configuration change takes effect through refresh, which a plain setter would no longer exercise; the edges are worth more than the two baseline lines.
Baseline drops 44 -> 36 lines, deletions only:
- 4 interface/infra -> composition edges for ENV_SCANNER_ENABLED, scanner_enabled_from_env and heal_enabled_from_env
- 2 infra -> interface edges for is_audit_module_enabled and is_notify_module_enabled
- cycle|composition<->infra and cycle|composition<->interface
The two cycles were not expected to go until whole subsystems moved out; clearing composition's inbound upward edges dissolved both, leaving three of the original five.
Verification: scripts/check_layer_dependencies.sh passes, cargo check -p rustfs warning-free, make pre-commit exit 0.
* test(ecstore): pin madmin-compatible ARN partition contract
Red-light evidence for backlog#1675 P1-7: madmin-go's ParseARN
hard-rejects any ARN that does not start with 'arn:minio:', while RustFS
generates and only accepts 'arn:rustfs:'. mc/madmin tooling therefore
cannot decode RustFS remote-target listings, and MinIO-era replication
configs are rejected as StaleTarget when re-registered. The new tests
pin the target contract (generate arn:minio:, parse both partitions,
reject unknown partitions) and fail against the current single-partition
gate.
* fix(ecstore): mint bucket-target ARNs in the madmin arn:minio partition
madmin-go's ParseARN hard-rejects any partition other than 'arn:minio:',
so native mc/madmin tooling could not decode RustFS remote-target
listings, and re-registering a MinIO-era replication config failed its
StaleTarget check against freshly minted arn:rustfs: targets
(backlog#1675 P1-7, route A).
- ARN Display now emits 'arn:minio:'; FromStr accepts a {minio, rustfs}
partition whitelist (the legacy partition stays readable forever for
persisted bucket-targets.json / replication configs). The whitelist is
the only structural gate — BucketTargetType::from_str never fails —
so it deliberately rejects foreign partitions such as arn:aws:.
- No data migration: every runtime match between targets, rules and
stats keys is full-string equality, so existing arn:rustfs: targets
keep matching their persisted rules; site replication already
preserves MinIO-era ARNs on reconcile (pinned by existing tests).
- Rolling upgrade note: upgrade all cluster nodes before creating new
remote targets — a not-yet-upgraded node rejects remove-remote-target
for a freshly minted arn:minio: ARN with BucketRemoteArnInvalid.
- Out of scope: notification/SQS ARNs (crates/targets) keep the
arn:rustfs:sqs: partition; they have their own compatibility story.
* test(replication): pin missing LWW timestamp header transport
Red-light tests for the replication timestamp three-header contract:
- put_object_headers_carry_replication_timestamp_headers pins that
PutObjectOptions::header() must emit the
x-{rustfs,minio}-source-replication-{tagging,retention,legalhold}-timestamp
headers when the internal timestamps are set (currently missing).
- test_put_opts_from_headers_gates_replication_timestamp_persistence_on_authorization
and test_complete_multipart_opts_persist_replication_timestamps_when_authorized
pin that an authorized replication PUT / multipart complete must persist
the inbound timestamps into the internal metadata keys while unauthorized
requests must not (currently never persisted).
- fake_s3_target journals the three timestamp headers per request
(ReplicationTimestampHeaders on RequestRecord) so sender-side e2e
assertions can observe what a real target receives; self-test included.
* fix(replication): transport and persist LWW timestamps for tag, retention, and legal hold
Active-active conflict resolution for concurrent tag/retention/legal-hold
edits needs the source's per-category modification times on both sides of
the wire; the three AdvancedPutOptions timestamp fields were dead and the
headers were neither sent nor parsed.
- Emit x-{rustfs,minio}-source-replication-{tagging,retention,legalhold}-
timestamp from PutObjectOptions::header(); names and RFC3339 values
interoperate with MinIO (minio-go constants.go, object-api-options.go),
pinned by a header_compat wire-name test.
- Default the three AdvancedPutOptions timestamps to UNIX_EPOCH and skip
epoch values in header(), so "never modified" is not sent as a
modification made now.
- Parse the headers only on authorized replication PUTs and multipart
completes, expose them as Option<OffsetDateTime> on ObjectOptions, and
persist them into the dual-prefix internal metadata keys so the
outbound pass (replication_target_boundary) reads the source's
timestamps instead of the mod_time fallback.
- Record the local tagging timestamp in the PutObjectTagging and
DeleteObjectTagging eval metadata, mirroring the object-lock handlers;
without it the sender only ever had the mod_time fallback to offer.
Receiver-side LWW comparison (keep newer stored category metadata over a
stale inbound copy) is left as a TODO at the parse site.
* fix(replication): load the stored tagging timestamp independently of remaining tags
Review: DeleteObjectTagging persists the tagging-timestamp internal key
but leaves the object tagless, and the outbound mapper only loaded the
key inside the user_tags-nonempty branch — the deletion's LWW timestamp
stayed at the epoch and the header was omitted, so the deletion could
never win conflict resolution on the replica. The stored key is now
loaded unconditionally; the mod_time fallback still applies only while
tags exist (MinIO parity), and a tagless object without the key keeps
the epoch default (no header). Deletion-path regression test added.
* fix(storage): reserve replication transport names at metadata ingest
Second review round: a client PUT of
x-amz-meta-x-rustfs-source-replication-tagging-timestamp materialized
the bare transport key as stored user metadata. The outbound
replication header builder forwards user metadata verbatim on a
server-authorized request, so the receiver would persist the
attacker-chosen value as trusted internal LWW state — and for a
tagless object nothing later overwrites it.
The ingest namespacing guard now reserves the whole
x-rustfs-source- / x-minio-source- families (the new timestamps and
their siblings: source-mtime/-etag/-version-id/-replication-request),
folding forged keys back under x-amz-meta-. Forged-ingress regression
covers both prefixes and a sibling.
* fix(replication): harden timestamp replay
* fix(app): route retention helper through facade
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
* test(admin): pin madmin ReplicationMRF stream contract for /v3/replication/mrf
Red-light evidence for backlog#1675 P1-13 (mrf half): madmin's
BucketReplicationMRF decodes the response one ReplicationMRF document at
a time, so the current aggregate envelope decodes as a single phantom
row with an empty object in 'mc replicate backlog'. The new contract
tests assert the desired bare-document stream (exact madmin json tags,
empty body for an empty backlog) and fail against the current
render_mrf_backlog extraction, which preserves the envelope-only
behavior:
- mrf_stream_renders_bare_madmin_documents: envelope keys leak, no
per-entry documents
- mrf_stream_renders_empty_body_for_no_entries: empty backlog still
renders the envelope (phantom row)
- mrf_aggregate_envelope_retains_counters: PerObjectEntriesAvailable
never advertises the enumerable stream
* fix(admin): stream madmin ReplicationMRF documents from /v3/replication/mrf
The mrf endpoint returned a single aggregate envelope, which madmin's
json.Decoder loop decoded as one phantom row (empty object) in
'mc replicate backlog' (backlog#1675 P1-13, mrf half; the diff half was
fixed in #5799 and this mirrors its pattern).
- Default response is now a bare stream of ReplicationMRF documents
(exact madmin json tags; Size/TargetARNs as ignored extension keys)
built from the durable backlog ledger; an empty backlog renders an
empty body, so mc shows zero rows instead of a phantom row.
- The aggregate counter envelope moves behind ?aggregate=true (RustFS
extension) and now advertises PerObjectEntriesAvailable whenever the
durable backlog is readable.
- An unreadable backlog is signalled out-of-band via
x-rustfs-replication-mrf-backlog-unavailable (mirrors the diff
truncation header) plus a warn event, since the bare stream cannot
carry source health.
- The madmin node parameter is accepted but documented as a no-op: the
durable ledger is cluster-shared with no per-node attribution.
- Delete-marker purge entries fall back to the marker version id so
those rows keep a version identity.
* fix(admin): fail the mrf stream request when the durable ledger is unreadable
Review: madmin only decodes the body of a 200, so the out-of-band
unavailability header was invisible to it and an unreadable ledger read
as a clean zero-row backlog. Stream mode now returns 503; aggregate
mode keeps the availability fields.
* fix(admin): gate, bound, and null-map the mrf stream
Second review round:
- Authorization: the default stream enumerates object names and version
ids, which a metrics-only principal must not see — it now requires
admin:ReplicationDiff (MinIO parity, route policy updated);
?aggregate=true carries no object identities and keeps
admin:GetReplicationMetrics.
- The nil UUID is RustFS's in-memory null-version sentinel and now
leaves as the S3 wire token 'null' instead of a zero UUID (a
pre-versioning object scanned after versioning + existing-object
replication can persist it into the ledger).
- The durable ledger is not bounded by the in-memory pending cap and
the body is buffered before send; the stream now stops at 10,000
documents and signals truncation via
x-rustfs-replication-mrf-truncated (mirroring the diff endpoint)
plus a warn event, instead of staging an unbounded body.
* fix(admin): reject truncated MRF streams
---------
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
* test(site-replication): pin ILM expiry merge contract for incoming lc-config
Red-light evidence for backlog#1675 P1-1: the lc-config receiver
overwrites the whole local lifecycle config with whatever the peer
sends (and deletes it wholesale on peer delete), so an expiry-only
document erases the receiver's local tier/transition rules, and peer
transition rules get installed across sites. The new tests pin the
MinIO mergeWithCurrentLCConfig semantics plus RustFS hardening:
- incoming expiry documents merge with (never replace) local rules
- local transition sides are authoritative for same-id rules
- incoming transition fields are discarded at the trust boundary
- dropped expiry rules strip the expiry side but keep transitions;
pure-expiry rules are removed
- delete merges with the empty set instead of dropping the config
- disabled rules survive; abort-mpu-only rules stay site-local
- deterministic order (idempotent re-delivery) and expiry_updated_at
stamping for the staleness axis
All fail against the current overwrite implementation (identity
extraction of merge_incoming_lifecycle_config).
* fix(site-replication): merge incoming ILM expiry documents instead of overwriting
The lc-config receiver replaced the whole local lifecycle config with
the peer's document (and deleted it wholesale on peer delete), so an
expiry-only update erased the receiver's local tier/transition rules,
and a peer's transition rules were installed across sites
(backlog#1675 P1-1).
Receiver (apply_bucket_meta_item):
- lc-config now merges via merge_incoming_lifecycle_config, mirroring
MinIO's mergeWithCurrentLCConfig with a trust-boundary hardening:
incoming transition fields are discarded outright; the local
transition side of a same-id rule is authoritative. A peer delete
merges with the empty set — pure-expiry rules go away, transition
rules survive with their expiry side cleared, and only an empty
result deletes the config file.
- Staleness moves to the expiry axis (config.expiry_updated_at):
lifecycle_config_updated_at also moves on local transition-only
edits, which shadowed newer peer expiry updates.
- Receiver-side replicateILMExpiry gate, symmetric with the sender
hook (previously any peer could install expiry rules while the
option was off).
- Rule order is deterministic (local order, incoming-new appended), so
re-delivering the same document is byte-stable and does not rewrite
bucket metadata per broadcast.
Sender:
- Both admin choke points — the bucket-meta hook and the SRInfo bucket
entry feeding bootstrap/repair and consistency views — now emit only
the expiry subset (transition fields stripped, non-expiry rules
dropped). MinIO receivers install incoming rules verbatim, so
transition rules must never leave the site. An unparseable local
config is forwarded unfiltered rather than degraded to a delete.
Not covered here (follow-up): a two-site e2e with a real tier backend
to exercise transition-rule preservation end to end; receiver-side
validate_transition_tier for merged configs.
* fix(site-replication): close ILM merge review findings
Adversarial review of the lc-config merge surfaced four real defects,
all fixed here:
- Deletion tombstone regression: with the staleness axis moved to the
in-config expiry_updated_at, a deleted lifecycle config fell back to
UNIX_EPOCH and any delayed stale broadcast could resurrect deleted
expiry rules. The axis now falls back to the whole-config write time
(which survives deletion in bucket metadata as the deletion's lower
bound), also covering legacy configs that predate the axis field.
- MinIO zero-rule documents: MinIO's delete tombstone / transition-only
state marshals a lifecycle document with no <Rule>, which the strict
s3s deserializer rejects — the receiver now recognizes it as the 'no
expiry rules here' statement (delete semantics) instead of erroring
on every MinIO heal pass.
- Inflated expiry axis at the sender: PutBucketLifecycle stamped
expiry_updated_at unconditionally, so a transition-only edit advanced
the axis and let this site's stale expiry subset shadow and roll back
newer peer expiry edits fleet-wide. The stamp is now conditional
(expiry subset present before or after the edit, MinIO parity), the
hook item travels with the config's expiry axis (UNIX_EPOCH when the
site has none), and the SRInfo bucket entry feeds bootstrap/repair
the same axis instead of the whole-config write time.
- Del-marker parity: MinIO's CloneNonTransition never emits del-marker
or abort-mpu fields, so treating del_marker_expiration as traveling
expiry let a MinIO broadcast delete this site's del-marker-only
rules. Both fields are now site-local on every edge: stripped from
outbound subsets and inbound rules, restored from the local side on
same-id merges, and never a deletion criterion.
Receiver-side validation of merged configs (object-lock / tier
constraints, MinIO runs finalLcCfg.Validate) remains a follow-up.
* fix(site-replication): close the second ILM review round
- Missed-delete repair: a deleted expiry state now travels through
bootstrap/repair as an explicit timestamped lc-config delete item
(lifecycle_expiry_statement distinguishes deletion — whole-config
write time advanced past the created backfill — from never-configured
buckets and from transition-only configs without an expiry axis,
which say nothing). A peer that missed the live delete converges on
repair; the receiver's staleness guard protects newer peer state.
- Strict tombstone recognition: only a well-delimited zero-rule
<LifecycleConfiguration> document maps to delete semantics; truncated
or foreign payloads that fail the strict deserializer are rejected
instead of being treated as a delete that erases local expiry rules.
- Staleness fallback axis narrowed: the whole-config write time is used
only for deleted or legacy-with-expiry state. A present
transition-only config without an expiry axis compares at epoch — its
whole-config time moves on transition edits and must not shadow or
block independent peer expiry updates and same-timestamp repairs.
* fix(site-replication): validate tombstone children structurally
Second review round: a well-delimited root could still smuggle
malformed content — e.g. <LifecycleConfiguration><ExpiryUpdatedAt>
</LifecycleConfiguration> passed the no-<Rule check and was applied as
a delete. The tombstone body must now be a sequence of well-formed
simple children (matching open/close or self-closing, no nested markup,
no stray text, none named Rule); anything else surfaces InvalidRequest.
Malformed-child cases pinned in the recognition test.
* fix(site-replication): serialize lifecycle merges
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
* test(admin): pin minio-go Metrics/MetricsV2 wire contract for replication metrics
Red-light evidence for backlog#1675 P1-11: ?replication-metrics[=2]
serializes the internal snake_case BucketStats family straight onto the
wire, while minio-go's replication.Metrics/MetricsV2 expect camelCase
tags (currStats/queueStats/replicaCount/queued/...). Go's decoder is
case-insensitive but does not ignore underscores, so 'mc replicate
status' shows all zeros without any error. The rewritten snapshot tests
assert the minio-go tags (plus a synthesized queueStats node — the
aggregation path leaves queue_stats.nodes empty today) and fail against
the current pass-through serialization.
* fix(admin): serialize replication metrics in minio-go wire shapes
?replication-metrics[=2] and the admin replicationmetrics endpoint
serialized the internal snake_case BucketStats family straight onto the
wire, so 'mc replicate status' decoded all zeros without any error
(backlog#1675 P1-11). The internal structs cannot be renamed: they are
the intra-cluster peer-RPC wire format (rmp_serde to_vec_named in
node_service.rs), pinned by a new regression test.
- New admin/replication_metrics_wire.rs: Serialize-only projections onto
minio-go replication.Metrics (v1 body, currStats) and MetricsV2
(uptime/currStats/queueStats/downtimeInfo) with the exact json tags;
per-target failed becomes the TimedErrStats envelope fed from the
FailStats rolling window; the queue peak is dual-emitted as max
(MinIO server tag) and peak (minio-go tag).
- queueStats synthesizes one node from the bucket queue snapshot — the
aggregation path leaves queue_stats.nodes empty, and mc treats an
empty node list as 'no data' — and carries transfer summaries
(Large/Small/Total) derived from the per-target xfer rates.
- Both endpoints share the DTOs; source-health extension keys
(provider_available/cluster_complete/...) ride along and are ignored
by Go decoders.
- Widen the ecstore replication_stats_boundary re-exports
(BucketReplicationStat/InQueueMetric/XferStats) so the admin facade
chain can name the projected types.
* fix(replication): carry failure rolling windows through cluster aggregation
Review: both metrics endpoints aggregate first, and FailStats::merge
dropped the process-local samples (which also never cross the peer-RPC
wire — serde-skipped), so lastMinute/lastHour serialized as zero right
after a failure while totals was nonzero.
- FailStats gains serializable last_minute/last_hour window snapshots
(serde default: old nodes read zeros, new fields are ignored by old
decoders), recomputed on every add_size and re-stamped at the
per-node collection point (get_latest_replication_stats), and summed
by merge.
- The wire DTO takes the component-wise max of the live samples and the
snapshot, so both the single-node and the aggregated path report the
window.
- Regression test drives a stat through rmp round trip + merge before
serialization, as requested.
Also restore the #[allow(dead_code)] attribute to route_policy — the
new module declaration had been inserted between the attribute and its
item, which broke the -D warnings CI lanes.
* fix(replication): bin transfer summaries at 128 MiB and keep window refresh off the hot path
Second review round:
- update_xfer_rate split at 1 MiB while the minio-go transferSummary
labels (and RustFS's own worker-pool split) mean >= 128 MiB for
Large, so a 2 MiB replication reported under Large with Small stuck
at zero. The producer now bins on MIN_LARGE_OBJ_SIZE; a MetricsV2
assertion covers 2 MiB / 127 MiB / exactly 128 MiB.
- add_size no longer recomputes the rolling windows: two full
one-hour-deque scans per failure under the bucket-stats write lock
made failure bursts quadratic (30k events ~2.1s). The windows are
stamped only at the collection point (get_latest_replication_stats,
which serves both the local leg and the peer RPC); the aggregation
regression now drives that path explicitly before the RPC round trip
and merge.
* fix(replication): average transfer summaries
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
fix(site-replication): lift a rejoined site's restarted edit counter over stale fence marks
A site removed while unreachable (unilateral removal: the receiver never
dropped it from its peer map, so parse_site_replication_state's load-time
mark pruning never fired) that later rejoins recreates its state object
and restarts edit_generation at zero. The receiver's surviving high-water
mark then silently fences out every stamped delivery from that origin —
peer edits and the add finalize fan-out alike are acked without applying
— until the restarted counter catches up.
Allocate the generation as a hybrid logical clock instead:
max(wall clock in unix nanoseconds, previous + 1), still inside the state
transaction under the distributed state-object lock. Every value a
lifetime hands out is capped by the wall clock at its own allocation, so
a recreated lifetime's first allocation exceeds them all and clears the
stale mark, while a pre-removal delivery still in flight stays below the
new floor and remains correctly fenced. previous+1 keeps allocations
strictly increasing across same-tick allocations and mid-lifetime clock
regressions.
Nothing changes on the wire or in the persisted schema: editGeneration
stays the single fence param and edit_generation the single counter
field, so pre-hybrid receivers get the fix as soon as the sender
upgrades, old binaries preserve the field across rolling up/downgrades,
and marks recorded by plain-counter receivers (small values) are cleared
by any wall-clock allocation. A clock that regresses across a
delete/recreate degrades to a fence that self-heals once real time
passes the previous lifetime's last allocation, and introduces no
rollback window beyond what the plain counter already had.
An epoch-based design (editEpoch wire param + per-origin epoch marks)
was built first and rejected under adversarial review: old binaries
rewriting the state object drop the unknown epoch fields, which both
disarms the fix mid-rolling-upgrade and — because epoch adoption lowers
the generation mark — reopens the pre-restart rollback the fence exists
to prevent; a backwards clock also fences an origin permanently instead
of self-healing. The hybrid clock has none of these modes.
* fix(storage): restore multipart disk compression and make the legacy decompressor resumable
Multipart uploads have bypassed disk compression since #5169 removed the session marker as a stopgap for mid-stream GET failures. The actual root cause was never the multipart layout: the legacy DecompressReader reset its payload consumption state on every poll re-entry, so a Poll::Pending in the middle of a block payload (routine under the erasure duplex) desynchronized the block framing and surfaced as LZ4 frameType errors. This rewrites the decoder as a resumable state machine, restores the multipart session compression marker, reports logical part sizes in ListParts, and makes the rebalance migration read raw stored bytes so compressed and encrypted objects survive migration verbatim.
Fixes#5957. Internal tracking: backlog#1848, backlog#1850.
* feat(storage): stage multipart compression behind RUSTFS_COMPRESSION_MULTIPART_ENABLED
Review follow-up: a rolling-upgrade window must not create new compressed multipart objects while pre-fix nodes (whose decompressor is not resumable) may still serve reads. The session marker is now additionally gated on RUSTFS_COMPRESSION_MULTIPART_ENABLED, default off, so the restored capability stays dark until the operator confirms fleet convergence. The default flips per the multipart-compression-default-off-window entry in docs/architecture/compat-cleanup-register.md once the minimum supported direct-upgrade release ships the resumable decoder.
* chore(compat): satisfy the cleanup-register guard for the multipart compression switch
The architecture guard requires every backticked identifier in a register entry to carry a RUSTFS_COMPAT_TODO source marker: keep only the entry slug in backticks, and add the marker (with its literal Remove-after condition) at the switch definition.
* chore(rio): drop a dead store in the poison guard and note the end-block branch
Review follow-up: the poison gate re-assigned an already-true flag, and the COMPRESS_TYPE_END branch reads as dead without stating that the writer never emits an end block — that absence is exactly what lets concatenated per-part streams decode as one.
* fix(s3): report empty compressed multipart part size
* fix(s3): report empty encrypted multipart part size
* fix(site-replication): route every state RMW through the locked transaction
P1-15 PR2 (rustfs/backlog#1796, batch B2 of rustfs/backlog#1675), the
follow-up promised by rustfs/rustfs#5882.
PR1 left ~26 read-modify-write call sites on
config/site-replication/state.json in the pre-transaction shape: a
process-local mutex around load / mutate / save, each IO taking its own
object lock. Nothing held a distributed lock across the whole sequence, so
two nodes of one site still lost each other's updates, and the transitional
mutex kept the old shape available to copy.
Every remaining RMW now runs inside update_site_replication_state;
read-only sites use load_site_replication_state, whose object read comes
with the object-level read lock. SITE_REPLICATION_STATE_LOCK and its owner
helper are gone, together with their architecture-guard allowlist entry and
inventory row.
The multi-stage flows (add / edit / peer join / peer edit / remove / rotate)
keep their updated_at and pending-id CAS, but the CAS now runs inside the
transaction that writes, against the state that transaction loaded. Peer
probes, IAM work and fan-outs run between transactions and hold no lock at
all — the add no longer blocks every writer of the site across its peer join
round trips, and it re-checks the precondition right after the capability
probes so the common race is rejected before any IAM write or remote join.
When the add's commit CAS still fails, the error says the peers may already
be joined and that re-running the add reconverges. The add adopts only the
fields it computed (exhaustive destructure — adding a state field is a
compile error until classified); fields owned by writers that do not bump
updated_at keep their freshly loaded values.
Ordering of peer-edit deliveries now rests on the generation fence landed in
PR1 rather than on a guard that could never order two nodes: the add's
finalize fan-out carries the generation allocated in its commit. An accepted
peer join PRESERVES the applied-generation high-water marks — join fan-outs
are routine (adds and rotations both deliver SRPeerJoin to existing peers),
so wiping them would let stalled older edits land after any join; the
unilateral-removal rejoin misfence that a wipe would have patched is
pre-existing since the fence landed and needs an epoch in the fence instead.
The rotation handler now takes the lifecycle guard: the background
service-account reconciler runs its repair under a lifecycle try-acquire,
and its pending-rotation precheck is only sound if a rotation cannot start
mid-repair — an exclusion the removed process mutex used to provide as a
side effect.
update_site_replication_state_when_changed adds persist-or-skip so ack
markers and pending-clearing paths stop rewriting the object on a miss —
load-bearing, because the shared persist helper clears the whole object for
a ≤1-peer pending-free state — and save_site_replication_state is now
cfg(test): the pre-P1-15 shape can no longer be written in production code.
No on-disk format change.
Verification: cargo nextest run -p rustfs -E
'test(/admin::handlers::site_replication::/)' (181 passed); site-replication
dual/three-node e2e (13 passed); cargo clippy -p rustfs --all-targets -D
warnings; make pre-commit. Mutation checks: dropping the state-object lock
from the boundary reds the separate-node concurrency tests; flipping a
persist-or-skip miss to a persist reds
test_missed_pending_clear_must_not_rewrite_the_state_object. Reviewed by
three independent adversarial passes (correctness/concurrency,
security/compatibility, simplicity/test-coverage); their confirmed findings
are folded in.
* fix(site-replication): serialize peer-join admission around its IAM write
Review follow-up (overtrue): two joins accepted by the same node could
interleave as "A checks a stale snapshot and pauses reading its body, B
applies secret B and commits, A resumes, overwrites IAM with secret A, and
A's commit is refused as superseded" — the persisted state advertised B's
contract while IAM only accepted A's secret, failing every peer
control-plane call. The pre-P1-15 process mutex serialized same-node joins
end to end; removing it dropped that exclusion.
admit_peer_join now runs the staleness check, the IAM upsert and the state
commit under the lifecycle guard, with the authoritative pre-check taken
against a load under that guard BEFORE IAM changes anything. The closing
transaction still re-checks staleness: the guard is process-local (exactly
as far as the old mutex reached) and the state-object lock arbitrates joins
accepted by different nodes. The body is fully read before the guard so a
stalling sender cannot block add/remove/rotate/reconciler.
The IAM step is injected, and the gated-body regression test reproduces the
review's ordering: join A is held mid-IAM while a newer join B arrives; B
must wait at the guard, and both IAM order and the final persisted state end
on B. Mutation-verified: removing the lifecycle guard from admit_peer_join
turns the test red.
Verification: cargo nextest run -p rustfs -E
'test(/admin::handlers::site_replication::/)' (182 passed);
site-replication dual/three-node e2e (13 passed); cargo clippy -p rustfs
--all-targets -D warnings; make pre-commit.
* fix(site-replication): fence peer-join admission across nodes
Review follow-up (overtrue, round 2): the lifecycle guard only serializes
joins within one process. Node A could pass the staleness check for an
older T1, node B write secret B to IAM and commit a newer T2, and node A
then overwrite IAM with secret A while its own state commit is refused as
superseded — state advertising T2's contract while IAM only accepts A's
secret.
The admission (staleness check -> IAM upsert -> state commit) now also runs
under a distributed join-admission lock, a namespace-lock key with no
backing object, following the repair execution lock's pattern — including
its nesting of config-object locks (admission -> state), and delegating
crash safety to the lock subsystem's lease expiry instead of a hand-rolled
TTL. The staleness check runs against a load taken inside the lock, before
IAM changes anything, so a superseded join exits without touching IAM. The
closing transaction keeps its re-check for defence in depth and for
old-version nodes that do not take the admission lock during a rolling
upgrade (that mixed-version window keeps today's behavior and closes when
the upgrade completes).
admit_peer_join_across_nodes is the admission minus the process-local
lifecycle guard — exactly what a second node runs — and the new
separate-nodes regression test drives it directly with join A gated
mid-IAM: join B must wait at the distributed lock, and both the IAM write
order and the final persisted state end on B. Mutation-verified: removing
the admission lock turns the test red while the same-node test (which
drives the full admit_peer_join) stays green.
Verification: cargo nextest run -p rustfs -E
'test(/admin::handlers::site_replication::/)' (183 passed);
site-replication dual/three-node e2e (13 passed); cargo clippy -p rustfs
--all-targets -D warnings; make pre-commit.
Completes PR3 of the issue.
Store doubles: NoopTableCatalogStore (193 lines, a pure stub answering "nothing here") and TestTableCatalogStore (416 lines, a stateful fake with commit pauses and failure injection) move into test_support.rs, along with TestCatalogPublishPause which the latter needs. Per the issue's ruling both shapes are kept — they are different tools, not duplicates of each other. Being honest about the benefit: this does not reduce the number of TableCatalogStore implementations, it puts both in one file so a trait change is one file to edit instead of two.
row_level_conflict fold: rejects_stale_new_manifest_sequence, rejects_stale_added_entry_sequence, and rejects_historical_change_in_new_manifest were identical apart from four literals (manifest-list sequence, data-file name, manifest-entry snapshot id, failure message). They become one table-driven test with three rows, each row keeping its original values, and every assertion carries the case name.
Verification: cargo test -p rustfs --lib table_catalog 479 passed and --lib admin::handlers::table_catalog 165 passed (both down exactly 2 from the 3->1 fold; the store filter is a substring match that also covers the admin tests); clippy --lib --tests -D warnings clean; make pre-commit green.
Ref rustfs/backlog#1837 (PR3).
Stream materialized GET bodies by moving the buffered Bytes once instead of wrapping the stream in an extra bytes_stream layer.
Add an operations runbook for object I/O tuning A/B sweeps.
Co-authored-by: heihutu <heihutu@gmail.com>
* test(table-catalog): move the store-side stateful object backend into test_support
First half of the issue's PR2: the store tests' TestCatalogObjectBackend cluster (state/record/locks/pause types, the seed/fail/pause instrumented inherent impl, the TableCatalogObjectBackend trait impl, and the BlockingObjectPublication/UnserializedTestPublication commit-publication fakes — 544 lines) moves verbatim from table_catalog/tests.rs into test_support.rs, with pub(crate) visibility on the items and fields the tests reach directly. Pure move, no behavior change; the admin handler tests' TestTableCatalogObjectBackend union (its put barrier / fail-path / lock-attempt instrumentation folding into this fake) is the second half.
Verification: cargo test -p rustfs --lib table_catalog 481 passed; clippy --lib --tests -D warnings clean; make pre-commit green.
Ref rustfs/backlog#1837 (PR2, part 1).
* test(table-catalog): fold the admin object backend into the shared fake
Second half of PR2: the admin handler tests' TestTableCatalogObjectBackend (struct, inherent impl, trait impl, lock alias — 201 lines) is deleted and its instrumentation folded into the shared TestCatalogObjectBackend, which the admin tests now take through a type alias so no call site is renamed.
Two behavioral differences between the two fakes were found by the test suites rather than assumed away, and both are preserved:
- Lock observability: the admin fake implemented only acquire_write_lock, so the trait's default acquire_read_lock -> acquire_write_lock delegation made read acquisitions visible in lock_attempts. The shared fake implements both independently, so five fence/lock tests timed out until the read path also records attempts.
- Etag generation: the admin fake used content-addressed sha256 etags (its tests observe an etag and expect rewriting identical bytes to reproduce it) while the store fake uses an incrementing counter. Instead of silently picking one, the union carries a content_addressed_etags flag; the 80 admin construction sites go through TestCatalogObjectBackend::content_addressed() and the store tests keep counter semantics.
The six one-shot path-keyed injection knobs (fail/corrupt put, missing/fail read, put barrier) run before the store fake's attempt-indexed injection maps, matching each fake's original ordering.
Verification: cargo test -p rustfs --lib table_catalog 481 passed; --lib admin::handlers::table_catalog 167 passed; clippy --lib --tests -D warnings clean; make pre-commit green.
Ref rustfs/backlog#1837 (PR2, part 2).
* refactor(sse): sink managed-SSE attribution into the shared encryption-keys module
Moves the managed-SSE classifier — stored_managed_encryption_key, contains_managed_encryption_metadata, normalize_managed_metadata — and the SSEType enum from rustfs/src/storage/sse.rs into crates/utils/src/http/object_encryption_keys.rs, the module that already owns every constant they read. This is PR-B0 of rustfs/backlog#1643: crates/scanner must never depend on the rustfs binary crate, so encryption attribution has to live in a shared lower layer before the scanner can report per-scheme coverage without growing a second classifier.
SSEType moves wholesale (option a): its only impl is the dependency-free audit_label(), so the enum relocates verbatim (audit_label becomes pub) and rustfs::storage::sse re-exports it, keeping every existing path compiling. The one piece that cannot move verbatim is normalize_managed_metadata's KMS-context branch, which needs base64 and serde_json — dependencies rustfs-utils does not have and does not gain here. The shared normalizer instead takes an injected Option<fn(&str) -> Option<String>> context recoder; sse.rs passes recode_minio_kms_context, the old inline chain verbatim including the silent skip on decode failure. stored_managed_encryption_key passes no recoder because the context mapping only ever inserts the context key, which the key-id lookup never reads, so its output is identical.
Every metadata lookup stays a case-sensitive exact match (lowercase x-amz-* stored forms, TitleCase MinIO-internal names) per the backlog#1775 trap; new shared-module tests pin that, and a source-scan test in sse.rs asserts the classifier has exactly one definition so a second copy cannot silently return.
* fix(utils): satisfy encryption key test clippy
---------
Co-authored-by: cxymds <cxymds@gmail.com>
Keep S3 CopyObject's real outer owner task alive across caller cancellation so the source/destination bucket guards, same-key copy guard, storage commit, and post-commit publication hooks complete as one request-owned transaction boundary.
Co-authored-by: heihutu <heihutu@gmail.com>
The two table_catalog test files (27.5K lines combined) each maintained a parallel constructor stack for Iceberg metadata JSON and avro manifest-list/manifest bytes. Per the issue's adversarial ruling the parameterized admin variants are canonical (the store file hardcoded sequence 7 / snapshot 20); the two stacks were verified structurally identical first — schemas byte-equal, field lists and values aligned.
New #[cfg(test)] table_catalog/test_support.rs owns the seven constructors (metadata JSON, three manifest-list variants, two manifest variants, nullable_long). The admin tests import them under their old names; the store tests keep their historical signatures as thin delegates passing the fixed values explicitly — every produced byte is identical to the pre-extraction fixtures (the delegate's argument order was cross-checked against the canonical destructuring after an initial swap surfaced as five sequence-bound validation failures).
Ref rustfs/backlog#1837 (PR1).
Use BytesMut spare capacity for direct and pooled small PUT body reads while preserving exact-length validation.
Co-authored-by: heihutu <heihutu@gmail.com>
RemoteAddr, the DependencyReadiness family (DependencyReadiness, ReadinessDegradedReason, DependencyReadinessReport), and convert_ecstore_object_info (with its offset_date_time_to_timestamp helper) are consumed across app, infra, and interface layers but lived under server, so every lower-layer import was an upward app->interface or infra->interface edge the layer guard had to baseline.
They now live in a new layer-neutral rustfs/src/shared_types.rs (classified infra by the guard, making all consumer imports downward or lateral). server::readiness and server::event re-export for their own internals; the eight consumer sites (admin_usecase, bucket_usecase, object_usecase, cluster_snapshot, storage/access, storage/helper, plus the admin handler tests) import from the new home. Pure move: no type, impl, or behavior change.
The regenerated layer-dependency baseline shrinks by exactly eight lines with zero additions — the ratchet's intended direction. The two remaining readiness entries (collect/snapshot fn imports) need the collection machinery itself extracted from server and are left for the issue's PR5 scope.
Ref rustfs/backlog#1834 (PR4).
The 14 tests carried #[ignore = "requires isolated global object layer state"], and the only CI lane that runs ignored tests filters for lifecycle tests — so they executed nowhere. Under nextest, the authoritative runner, every test owns its process and the stale reason no longer applies; all 14 pass.
Ten of them assert the InternalError path taken while the global object layer is uninitialized, a premise a sibling test can destroy under the documented shared-process cargo test fallback. Those ten now start with an explicit premise guard: when a sibling already initialized the store the test skips with a message instead of asserting against a scenario it does not describe. Under nextest the guard never fires and the assertions always run.
Dual-runner evidence: nextest 79 passed; cargo test module-scoped 79 passed; the full storage-tree cargo test sweep returns to its pre-existing baseline (8 unrelated in-process failures, none introduced or worsened here). No test deleted.
Ref rustfs/backlog#1830 (PR1).
* perf(metrics): attribute PUT stage costs
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): move PUT metadata during shuffle
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(s3): reuse PUT object lock state
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): trim PUT metadata fanout clones
Build per-disk PUT metadata only for committed writer slots, move the response metadata out of the fanout vector, and preserve fresh FileInfo shuffle semantics.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(metrics): make PUT stage attribution opt-in
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): commit inline PUT shards directly
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): streamline rename staging cleanup
Use the directory-specific removal operation for rename_data staging parents. This avoids a guaranteed failed file-removal probe on Unix-like hosts and lets Windows remove the empty directory directly while preserving best-effort non-empty handling.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): cover inline PUT rename failures
Cache the detailed stage metrics gate once per PUT and exercise exact-quorum and quorum-minus-one failures after inline shard encoding.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
houseme