* chore(ecstore): drop the bucket dead_code blanket
The last blanket of the backlog#1823 burn-down, and the largest: 71 items across lifecycle, replication, metadata, quota, object lock and bucket utils. Four are deleted.
Deleted, all trivial:
- check_valid_object_name and check_valid_object_name_prefix, a pair that only calls into each other with no external caller. Worth stating plainly so nobody reads this as a validation gap: object names are validated through check_object_name_for_length_and_slash, which is live; this pair is a second, unwired entry point.
- DEFAULT_HEALTH_CHECK_RELOAD_DURATION, a lone unused constant.
- The LifecycleReplicationConfig alias, which orphaned a re-export in replication/mod.rs that goes with it.
Everything else is kept, in four groups, because the blanket here was hiding structure rather than rot:
Windows platform gating. WINDOWS_RESERVED_NAMES, the two reason constants and object_name_has_windows_incompatible_segment are called from inside the #[cfg(target_os = "windows")] block in check_object_name_for_length_and_slash (utils.rs:228-255), so they only read as dead on non-Windows hosts. As with the Linux gating in the disk root, this cannot be adjudicated locally: cargo check for both x86_64-pc-windows-msvc and x86_64-unknown-linux-gnu fails in the aws-lc-sys build script for want of a cross C toolchain. CI covers both.
Declared boundary surface. The *_boundary.rs and *_bridge.rs files carry the replication split plan's contracts, which scripts/check_architecture_migration_rules.sh pins through the EcstoreReplicationBoundaryImports section of the split-plan doc. Their unused items are declarations, not leftovers.
test-util seams. ConfigWriteLockProbe with install/wait_until_attempted follows the same pattern as the barriers in the services and set_disk roots.
MinIO-parity tier/lifecycle entry points that this port never wired: apply_lifecycle_action, get_transitioned_object_reader, recover_tier_free_versions, delete_object_from_remote_tier, abort_tier_delete_journal_entry and the replication pool's worker-management surface. These are complete, substantial machinery with no caller — the same shape as data_usage's local_snapshot feature. Removing them is a product decision, so they are made explicit here rather than deleted.
Verification, four lanes warning-free: default, --tests, --features rio-v2 --tests, --features test-util --tests. cargo nextest run -p rustfs-ecstore 4096 passed; clippy --lib --tests -D warnings clean; make pre-commit exit 0. Note that clippy is what caught the orphaned re-export above: cargo check and pre-commit both treat unused_imports as a warning.
Ref rustfs/backlog#1823 (step 2, final root).
* chore(ecstore): correct inaccurate dead_code reasons in the bucket root
Six items were labelled 'asserted by this file's tests' or as MinIO-parity
entry points while having no caller at all - free get_bucket_acl_config and
created_at only reach their own live methods (production goes through
created_at_in), BucketVersioningSys::get_in, utils::serialize_content and
ServiceType have no reference anywhere, and with_transition_queue_env_async
is an unused test fixture, not a tier entry point. Name what each one is so
the next reader does not assume coverage that is not there.
Ref rustfs/backlog#1823.
* chore(ecstore): drop the set_disk dead_code blanket
Removing the blanket exposes 39 items; exactly one is deleted. The low share is a finding, not caution: unlike the disk root, where platform gating made local adjudication impossible, here the items were checked and nearly all of them are live.
Deleted: HealEntryResult, the only item with no reference anywhere.
What the checks turned up, in the order the warnings suggest deleting them:
SetDisks::rename_data looked like the head of a dead chain feeding into_legacy_tuple and RenameDataLegacyTuple. It is not: production goes through rename_data_owned, and rename_data itself has test callers at mod.rs:5809 and 5880. The chain below it is therefore live through the tests, and inferring "this is dead, so its callee is dead" would have removed three working items.
create_bitrot_readers_until_quorum, read_multiple_files and map_cleanup_join_result all have callers inside their files' test modules, so they only look dead in the lib target.
TransitionCommitBarrier and TransitionUploadedSaveProbe, with their install/wait_until_paused/release surfaces, are installed by tests behind #[cfg(all(test, feature = "test-util"))].
ctx.rs's SetDisksCtx accessors are the split seam left by the SetDisks god-object break-up (backlog#815).
heal_object_dir's two apparent references are comments, and they document an index-alignment contract that live code maintains for it, so they stay as they are.
Worth a maintainer decision: the metadata early-stop switch has a complete percentage-rollout facet — ENV_RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT, get_metadata_early_stop_rollout_pct and should_use_metadata_early_stop — with no caller, no test and no documentation, while its sibling enable flag is live. It is kept with an allow that says so rather than removed, since a rollout knob is a product call.
One placement note for anyone adding allows near heal code: check_logging_guardrails.sh requires #[instrument(level = "trace")] to sit immediately before async fn heal_object_dir, so the allow goes above the instrument attribute. Putting it between the two drops the guard's match count and fails the check.
Verification, four lanes warning-free: default, --tests, --features rio-v2 --tests, --features test-util --tests. cargo nextest run -p rustfs-ecstore 4096 passed; clippy --lib --tests -D warnings clean; make pre-commit exit 0.
Ref rustfs/backlog#1823 (step 2).
* chore(ecstore): fix duplicated and inaccurate dead_code reasons in set_disk
format_lock_error carried the same #[allow] twice. Five items in the
locking/heal roots were labelled 'asserted by this file's tests' while
having no reference at all - heal_object_dir's only two references are
comments, as this branch's own notes point out. Say what each item
actually is instead, so the next reader does not assume test coverage
that is not there.
Ref rustfs/backlog#1823.
* chore(ecstore): correct the bounded_spare_disk_index dead_code reason
The mod.rs copy is an unused test fixture, not something this module's
tests assert; the namesake that is exercised lives in the io_primitives
test module.
Ref rustfs/backlog#1823.
* chore(ecstore): drop the disk dead_code blanket
Removing the blanket exposes 36 items in the lowest storage layer: 7 deleted, 29 kept with reasoned item-level allows. That is the smallest deletion share of this burn-down, and the reason is a verification limit rather than a judgement call.
disk/local.rs carries 141 `#[cfg(target_os = "linux")]` sites — the densest platform gating in the tree, because O_DIRECT and io_uring only exist there. The direct-I/O cluster (six ENV_RUSTFS_OBJECT_DIRECT_IO_* constants plus is_direct_io_read_enabled, is_direct_io_write_enabled, get_direct_io_read_threshold, direct_write_staging_capacity, direct_write_tail_split and DIRECT_WRITE_STAGING_BYTES) reads as dead on macOS purely because its production callers at local.rs:1766, 3114 and 4605 sit inside Linux-gated blocks. direct_write_staging_capacity even documents itself as "Platform-independent (no O_DIRECT), so it is unit-tested on any host".
Deleting those would leave every local check green — 4096 tests pass, clippy is clean, make pre-commit exits 0 — and break the Linux build in CI, because all four local lanes compile for aarch64-apple-darwin. Cross-checking locally is not available either: cargo check --target x86_64-unknown-linux-gnu fails in the aws-lc-sys build script for want of a Linux C cross-compiler. Their allows name the platform reason so the next reader on a non-Linux host does not repeat the investigation.
Deleted, all in files with no target_os gating at all (os.rs, disk_store.rs):
- HealthDiskCtxKey and HealthDiskCtxValue with its private log_success. Note that DiskHealthTracker::log_success is a different method of the same name and is live from cluster/rpc/peer_s3_client.rs and remote_disk.rs — the two have to be told apart by type, not by name.
- LocalDiskWrapper::new_with_health and check_id.
- os.rs file_exists and lock_destination_directory_for_path_access.
Kept with allows: DiskHealthTracker's set_faulty, mark_offline, waiting_count and last_success have test callers in remote_disk.rs, so they only look dead in the lib target. to_disk_error, remove_all and sync_dir_files are asserted by their own files' tests. The reclaim, mmap and path-cache field groups are written but never read back.
Placement follows the same rule as the earlier roots: per-method allows inside impl DiskHealthTracker and impl LocalDisk, since both are mostly live and a block-level allow would be a smaller version of the blanket this issue removes. Struct-level allows are used only where the warning covers that struct's own fields. The three cached_read_env! functions take their allow inside the macro invocation, before the fn line, because the macro forwards $(#[$meta:meta])* onto the generated item.
Verification, four lanes warning-free: default, --tests, --features rio-v2 --tests, --features test-util --tests. cargo nextest run -p rustfs-ecstore 4096 passed; clippy --lib --tests -D warnings clean; make pre-commit exit 0. The Linux lane is not covered locally and is left to CI.
Ref rustfs/backlog#1823 (step 2).
* chore(ecstore): correct two dead_code reasons in the disk root
check_valid_path and reject_symlink_components have no caller at all -
not even a test - so 'asserted by this file's tests' misreads them as
covered. Both are method wrappers over live free functions; say that
instead.
Ref rustfs/backlog#1823.
* test(ecstore): pin madmin-compatible ARN partition contract
Red-light evidence for backlog#1675 P1-7: madmin-go's ParseARN
hard-rejects any ARN that does not start with 'arn:minio:', while RustFS
generates and only accepts 'arn:rustfs:'. mc/madmin tooling therefore
cannot decode RustFS remote-target listings, and MinIO-era replication
configs are rejected as StaleTarget when re-registered. The new tests
pin the target contract (generate arn:minio:, parse both partitions,
reject unknown partitions) and fail against the current single-partition
gate.
* fix(ecstore): mint bucket-target ARNs in the madmin arn:minio partition
madmin-go's ParseARN hard-rejects any partition other than 'arn:minio:',
so native mc/madmin tooling could not decode RustFS remote-target
listings, and re-registering a MinIO-era replication config failed its
StaleTarget check against freshly minted arn:rustfs: targets
(backlog#1675 P1-7, route A).
- ARN Display now emits 'arn:minio:'; FromStr accepts a {minio, rustfs}
partition whitelist (the legacy partition stays readable forever for
persisted bucket-targets.json / replication configs). The whitelist is
the only structural gate — BucketTargetType::from_str never fails —
so it deliberately rejects foreign partitions such as arn:aws:.
- No data migration: every runtime match between targets, rules and
stats keys is full-string equality, so existing arn:rustfs: targets
keep matching their persisted rules; site replication already
preserves MinIO-era ARNs on reconcile (pinned by existing tests).
- Rolling upgrade note: upgrade all cluster nodes before creating new
remote targets — a not-yet-upgraded node rejects remove-remote-target
for a freshly minted arn:minio: ARN with BucketRemoteArnInvalid.
- Out of scope: notification/SQS ARNs (crates/targets) keep the
arn:rustfs:sqs: partition; they have their own compatibility story.
* test(replication): pin missing LWW timestamp header transport
Red-light tests for the replication timestamp three-header contract:
- put_object_headers_carry_replication_timestamp_headers pins that
PutObjectOptions::header() must emit the
x-{rustfs,minio}-source-replication-{tagging,retention,legalhold}-timestamp
headers when the internal timestamps are set (currently missing).
- test_put_opts_from_headers_gates_replication_timestamp_persistence_on_authorization
and test_complete_multipart_opts_persist_replication_timestamps_when_authorized
pin that an authorized replication PUT / multipart complete must persist
the inbound timestamps into the internal metadata keys while unauthorized
requests must not (currently never persisted).
- fake_s3_target journals the three timestamp headers per request
(ReplicationTimestampHeaders on RequestRecord) so sender-side e2e
assertions can observe what a real target receives; self-test included.
* fix(replication): transport and persist LWW timestamps for tag, retention, and legal hold
Active-active conflict resolution for concurrent tag/retention/legal-hold
edits needs the source's per-category modification times on both sides of
the wire; the three AdvancedPutOptions timestamp fields were dead and the
headers were neither sent nor parsed.
- Emit x-{rustfs,minio}-source-replication-{tagging,retention,legalhold}-
timestamp from PutObjectOptions::header(); names and RFC3339 values
interoperate with MinIO (minio-go constants.go, object-api-options.go),
pinned by a header_compat wire-name test.
- Default the three AdvancedPutOptions timestamps to UNIX_EPOCH and skip
epoch values in header(), so "never modified" is not sent as a
modification made now.
- Parse the headers only on authorized replication PUTs and multipart
completes, expose them as Option<OffsetDateTime> on ObjectOptions, and
persist them into the dual-prefix internal metadata keys so the
outbound pass (replication_target_boundary) reads the source's
timestamps instead of the mod_time fallback.
- Record the local tagging timestamp in the PutObjectTagging and
DeleteObjectTagging eval metadata, mirroring the object-lock handlers;
without it the sender only ever had the mod_time fallback to offer.
Receiver-side LWW comparison (keep newer stored category metadata over a
stale inbound copy) is left as a TODO at the parse site.
* fix(replication): load the stored tagging timestamp independently of remaining tags
Review: DeleteObjectTagging persists the tagging-timestamp internal key
but leaves the object tagless, and the outbound mapper only loaded the
key inside the user_tags-nonempty branch — the deletion's LWW timestamp
stayed at the epoch and the header was omitted, so the deletion could
never win conflict resolution on the replica. The stored key is now
loaded unconditionally; the mod_time fallback still applies only while
tags exist (MinIO parity), and a tagless object without the key keeps
the epoch default (no header). Deletion-path regression test added.
* fix(storage): reserve replication transport names at metadata ingest
Second review round: a client PUT of
x-amz-meta-x-rustfs-source-replication-tagging-timestamp materialized
the bare transport key as stored user metadata. The outbound
replication header builder forwards user metadata verbatim on a
server-authorized request, so the receiver would persist the
attacker-chosen value as trusted internal LWW state — and for a
tagless object nothing later overwrites it.
The ingest namespacing guard now reserves the whole
x-rustfs-source- / x-minio-source- families (the new timestamps and
their siblings: source-mtime/-etag/-version-id/-replication-request),
folding forged keys back under x-amz-meta-. Forged-ingress regression
covers both prefixes and a sibling.
* fix(replication): harden timestamp replay
* fix(app): route retention helper through facade
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
* test(site-replication): pin ILM expiry merge contract for incoming lc-config
Red-light evidence for backlog#1675 P1-1: the lc-config receiver
overwrites the whole local lifecycle config with whatever the peer
sends (and deletes it wholesale on peer delete), so an expiry-only
document erases the receiver's local tier/transition rules, and peer
transition rules get installed across sites. The new tests pin the
MinIO mergeWithCurrentLCConfig semantics plus RustFS hardening:
- incoming expiry documents merge with (never replace) local rules
- local transition sides are authoritative for same-id rules
- incoming transition fields are discarded at the trust boundary
- dropped expiry rules strip the expiry side but keep transitions;
pure-expiry rules are removed
- delete merges with the empty set instead of dropping the config
- disabled rules survive; abort-mpu-only rules stay site-local
- deterministic order (idempotent re-delivery) and expiry_updated_at
stamping for the staleness axis
All fail against the current overwrite implementation (identity
extraction of merge_incoming_lifecycle_config).
* fix(site-replication): merge incoming ILM expiry documents instead of overwriting
The lc-config receiver replaced the whole local lifecycle config with
the peer's document (and deleted it wholesale on peer delete), so an
expiry-only update erased the receiver's local tier/transition rules,
and a peer's transition rules were installed across sites
(backlog#1675 P1-1).
Receiver (apply_bucket_meta_item):
- lc-config now merges via merge_incoming_lifecycle_config, mirroring
MinIO's mergeWithCurrentLCConfig with a trust-boundary hardening:
incoming transition fields are discarded outright; the local
transition side of a same-id rule is authoritative. A peer delete
merges with the empty set — pure-expiry rules go away, transition
rules survive with their expiry side cleared, and only an empty
result deletes the config file.
- Staleness moves to the expiry axis (config.expiry_updated_at):
lifecycle_config_updated_at also moves on local transition-only
edits, which shadowed newer peer expiry updates.
- Receiver-side replicateILMExpiry gate, symmetric with the sender
hook (previously any peer could install expiry rules while the
option was off).
- Rule order is deterministic (local order, incoming-new appended), so
re-delivering the same document is byte-stable and does not rewrite
bucket metadata per broadcast.
Sender:
- Both admin choke points — the bucket-meta hook and the SRInfo bucket
entry feeding bootstrap/repair and consistency views — now emit only
the expiry subset (transition fields stripped, non-expiry rules
dropped). MinIO receivers install incoming rules verbatim, so
transition rules must never leave the site. An unparseable local
config is forwarded unfiltered rather than degraded to a delete.
Not covered here (follow-up): a two-site e2e with a real tier backend
to exercise transition-rule preservation end to end; receiver-side
validate_transition_tier for merged configs.
* fix(site-replication): close ILM merge review findings
Adversarial review of the lc-config merge surfaced four real defects,
all fixed here:
- Deletion tombstone regression: with the staleness axis moved to the
in-config expiry_updated_at, a deleted lifecycle config fell back to
UNIX_EPOCH and any delayed stale broadcast could resurrect deleted
expiry rules. The axis now falls back to the whole-config write time
(which survives deletion in bucket metadata as the deletion's lower
bound), also covering legacy configs that predate the axis field.
- MinIO zero-rule documents: MinIO's delete tombstone / transition-only
state marshals a lifecycle document with no <Rule>, which the strict
s3s deserializer rejects — the receiver now recognizes it as the 'no
expiry rules here' statement (delete semantics) instead of erroring
on every MinIO heal pass.
- Inflated expiry axis at the sender: PutBucketLifecycle stamped
expiry_updated_at unconditionally, so a transition-only edit advanced
the axis and let this site's stale expiry subset shadow and roll back
newer peer expiry edits fleet-wide. The stamp is now conditional
(expiry subset present before or after the edit, MinIO parity), the
hook item travels with the config's expiry axis (UNIX_EPOCH when the
site has none), and the SRInfo bucket entry feeds bootstrap/repair
the same axis instead of the whole-config write time.
- Del-marker parity: MinIO's CloneNonTransition never emits del-marker
or abort-mpu fields, so treating del_marker_expiration as traveling
expiry let a MinIO broadcast delete this site's del-marker-only
rules. Both fields are now site-local on every edge: stripped from
outbound subsets and inbound rules, restored from the local side on
same-id merges, and never a deletion criterion.
Receiver-side validation of merged configs (object-lock / tier
constraints, MinIO runs finalLcCfg.Validate) remains a follow-up.
* fix(site-replication): close the second ILM review round
- Missed-delete repair: a deleted expiry state now travels through
bootstrap/repair as an explicit timestamped lc-config delete item
(lifecycle_expiry_statement distinguishes deletion — whole-config
write time advanced past the created backfill — from never-configured
buckets and from transition-only configs without an expiry axis,
which say nothing). A peer that missed the live delete converges on
repair; the receiver's staleness guard protects newer peer state.
- Strict tombstone recognition: only a well-delimited zero-rule
<LifecycleConfiguration> document maps to delete semantics; truncated
or foreign payloads that fail the strict deserializer are rejected
instead of being treated as a delete that erases local expiry rules.
- Staleness fallback axis narrowed: the whole-config write time is used
only for deleted or legacy-with-expiry state. A present
transition-only config without an expiry axis compares at epoch — its
whole-config time moves on transition edits and must not shadow or
block independent peer expiry updates and same-timestamp repairs.
* fix(site-replication): validate tombstone children structurally
Second review round: a well-delimited root could still smuggle
malformed content — e.g. <LifecycleConfiguration><ExpiryUpdatedAt>
</LifecycleConfiguration> passed the no-<Rule check and was applied as
a delete. The tombstone body must now be a sequence of well-formed
simple children (matching open/close or self-closing, no nested markup,
no stray text, none named Rule); anything else surfaces InvalidRequest.
Malformed-child cases pinned in the recognition test.
* fix(site-replication): serialize lifecycle merges
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
* test(admin): pin minio-go Metrics/MetricsV2 wire contract for replication metrics
Red-light evidence for backlog#1675 P1-11: ?replication-metrics[=2]
serializes the internal snake_case BucketStats family straight onto the
wire, while minio-go's replication.Metrics/MetricsV2 expect camelCase
tags (currStats/queueStats/replicaCount/queued/...). Go's decoder is
case-insensitive but does not ignore underscores, so 'mc replicate
status' shows all zeros without any error. The rewritten snapshot tests
assert the minio-go tags (plus a synthesized queueStats node — the
aggregation path leaves queue_stats.nodes empty today) and fail against
the current pass-through serialization.
* fix(admin): serialize replication metrics in minio-go wire shapes
?replication-metrics[=2] and the admin replicationmetrics endpoint
serialized the internal snake_case BucketStats family straight onto the
wire, so 'mc replicate status' decoded all zeros without any error
(backlog#1675 P1-11). The internal structs cannot be renamed: they are
the intra-cluster peer-RPC wire format (rmp_serde to_vec_named in
node_service.rs), pinned by a new regression test.
- New admin/replication_metrics_wire.rs: Serialize-only projections onto
minio-go replication.Metrics (v1 body, currStats) and MetricsV2
(uptime/currStats/queueStats/downtimeInfo) with the exact json tags;
per-target failed becomes the TimedErrStats envelope fed from the
FailStats rolling window; the queue peak is dual-emitted as max
(MinIO server tag) and peak (minio-go tag).
- queueStats synthesizes one node from the bucket queue snapshot — the
aggregation path leaves queue_stats.nodes empty, and mc treats an
empty node list as 'no data' — and carries transfer summaries
(Large/Small/Total) derived from the per-target xfer rates.
- Both endpoints share the DTOs; source-health extension keys
(provider_available/cluster_complete/...) ride along and are ignored
by Go decoders.
- Widen the ecstore replication_stats_boundary re-exports
(BucketReplicationStat/InQueueMetric/XferStats) so the admin facade
chain can name the projected types.
* fix(replication): carry failure rolling windows through cluster aggregation
Review: both metrics endpoints aggregate first, and FailStats::merge
dropped the process-local samples (which also never cross the peer-RPC
wire — serde-skipped), so lastMinute/lastHour serialized as zero right
after a failure while totals was nonzero.
- FailStats gains serializable last_minute/last_hour window snapshots
(serde default: old nodes read zeros, new fields are ignored by old
decoders), recomputed on every add_size and re-stamped at the
per-node collection point (get_latest_replication_stats), and summed
by merge.
- The wire DTO takes the component-wise max of the live samples and the
snapshot, so both the single-node and the aggregated path report the
window.
- Regression test drives a stat through rmp round trip + merge before
serialization, as requested.
Also restore the #[allow(dead_code)] attribute to route_policy — the
new module declaration had been inserted between the attribute and its
item, which broke the -D warnings CI lanes.
* fix(replication): bin transfer summaries at 128 MiB and keep window refresh off the hot path
Second review round:
- update_xfer_rate split at 1 MiB while the minio-go transferSummary
labels (and RustFS's own worker-pool split) mean >= 128 MiB for
Large, so a 2 MiB replication reported under Large with Small stuck
at zero. The producer now bins on MIN_LARGE_OBJ_SIZE; a MetricsV2
assertion covers 2 MiB / 127 MiB / exactly 128 MiB.
- add_size no longer recomputes the rolling windows: two full
one-hour-deque scans per failure under the bucket-stats write lock
made failure bursts quadratic (30k events ~2.1s). The windows are
stamped only at the collection point (get_latest_replication_stats,
which serves both the local leg and the peer RPC); the aggregation
regression now drives that path explicitly before the RPC round trip
and merge.
* fix(replication): average transfer summaries
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix(storage): restore multipart disk compression and make the legacy decompressor resumable
Multipart uploads have bypassed disk compression since #5169 removed the session marker as a stopgap for mid-stream GET failures. The actual root cause was never the multipart layout: the legacy DecompressReader reset its payload consumption state on every poll re-entry, so a Poll::Pending in the middle of a block payload (routine under the erasure duplex) desynchronized the block framing and surfaced as LZ4 frameType errors. This rewrites the decoder as a resumable state machine, restores the multipart session compression marker, reports logical part sizes in ListParts, and makes the rebalance migration read raw stored bytes so compressed and encrypted objects survive migration verbatim.
Fixes#5957. Internal tracking: backlog#1848, backlog#1850.
* feat(storage): stage multipart compression behind RUSTFS_COMPRESSION_MULTIPART_ENABLED
Review follow-up: a rolling-upgrade window must not create new compressed multipart objects while pre-fix nodes (whose decompressor is not resumable) may still serve reads. The session marker is now additionally gated on RUSTFS_COMPRESSION_MULTIPART_ENABLED, default off, so the restored capability stays dark until the operator confirms fleet convergence. The default flips per the multipart-compression-default-off-window entry in docs/architecture/compat-cleanup-register.md once the minimum supported direct-upgrade release ships the resumable decoder.
* chore(compat): satisfy the cleanup-register guard for the multipart compression switch
The architecture guard requires every backticked identifier in a register entry to carry a RUSTFS_COMPAT_TODO source marker: keep only the entry slug in backticks, and add the marker (with its literal Remove-after condition) at the switch definition.
* chore(rio): drop a dead store in the poison guard and note the end-block branch
Review follow-up: the poison gate re-assigned an already-true flag, and the COMPRESS_TYPE_END branch reads as dead without stating that the writer never emits an end block — that absence is exactly what lets concatenated per-part streams decode as one.
* fix(s3): report empty compressed multipart part size
* fix(s3): report empty encrypted multipart part size
* fix(tier): decrypt transitioned objects instead of serving their ciphertext
A GET on a managed-SSE object that lifecycle had transitioned to a remote tier returned the ciphertext with the plaintext's Content-Length and no error: silent corruption on read-through, and worse than a failed request because nothing signals it. Restore of the same object failed server-side with IncompleteBody while POST ?restore still answered 200, so the object simply never came back and HEAD never showed an x-amz-restore marker.
Both symptoms are one cause. The transitioned read path built its fetch through new_getobjectreader, which decides nothing about encryption: it derived the range from the parts table — whose sizes are PLAINTEXT sizes — then used that range to fetch the object's STORED bytes from the tier, and handed the stream to the caller without any decrypt transform. The GET therefore served the first plaintext-length bytes of ciphertext; the restore copy-back, which validates against the stored size, came up short by exactly the encryption overhead.
The path now builds the same ReadPlan the local read path uses, so a single place decides how stored bytes map to requested bytes. ReadPlan gains a two-phase API — build_for_request to learn the storage coordinates before issuing the tier fetch, into_object_reader to wrap the returned stream — because the tier fetch has to be positioned before a stream exists. The encryption resolver reaches the path from InstanceContext, the same source the local read uses.
A restore read additionally stops synthesizing a range from the part number. A restore serves the stored representation (restore_request_active already forces the Plain branch), so a plaintext-coordinate range would be reinterpreted as a storage range and truncate the payload by its encoding overhead. An explicit caller range is already in storage coordinates on that path and is still honored, which two existing tests pin.
crates/e2e_test/src/kms/kms_ilm_sse_kms_test.rs drops its #[ignore]: the transition test now runs and asserts the plaintext round-trips byte-identically through transition, read-through and restore. The same file had its enforcement switch stuck at false from a control experiment; it is back to true, so the test again exercises what its name and module docs claim.
Fixes#6025. Refs rustfs/backlog#1582, rustfs/backlog#1637.
* test(tier): pass resolver to transitioned reader tests
Removing the blanket exposes eighteen items. Six are deleted; the rest belong to one feature that was never wired up.
crates/ecstore/src/data_usage/local_snapshot.rs and its aggregation entry point, aggregate_local_snapshots, form a complete per-disk usage-snapshot feature: read/write of snapshot files under the metadata bucket, cross-disk aggregation, and tests. Nothing calls it. It landed in #5307 on 2026-07-27 and git log -S over the whole history shows the entry point has never had a caller; the live data-usage path is load_data_usage_from_backend / store_data_usage_in_backend. Rather than delete a recent, tested feature on cleanup grounds, the module gets a header explaining the situation and its items carry individual allows, so the gap stays greppable without reintroducing a blanket. One commit flips this to deletion if that is preferred.
Deleted:
- DATA_USAGE_BLOOM_NAME together with DATA_USAGE_BLOOM_NAME_PATH. Both are a dead duplicate of the pair in crates/scanner/src/data_usage_define.rs, which has roughly fifty consumers across scanner.rs and remote_scanner.rs; the ecstore copies have none.
- increment_bucket_usage_memory and decrement_bucket_usage_memory, thin wrappers over the live record_bucket_object_write_memory and record_bucket_object_delete_memory.
- sync_memory_cache_with_backend, whose doc comment says it is "called by scanner" — nothing calls it.
- create_cache_entry_from_summary and cache_to_data_usage_info, neither with a consumer in any lane.
resolve_loaded_snapshot is kept with an allow: nine tests in the same file assert its primary/backup fallback.
Verification, four lanes warning-free: default, --tests, --features rio-v2 --tests, --features test-util --tests. rustfs-scanner still compiles clean. cargo nextest run -p rustfs-ecstore 4041 passed; clippy --lib --tests -D warnings clean; make pre-commit exit 0.
Ref rustfs/backlog#1823 (step 2).
Removing the blanket exposes twenty-five items across tier, notification and rebalance. Only eight are deleted — the lowest ratio of this burn-down so far, and the reason is that these subsystems carry heavy test coverage, so the blanket was mostly hiding test-only seams rather than dead weight.
Deleted:
- crates/ecstore/src/services/tier/warm_backend_s3sdk.rs entirely (200 lines). Its WarmBackendS3 is never constructed; the type of the same name in warm_backend_s3.rs is the live one, wrapped by the Azure backend. Two implementations of one S3 warm backend, one of them never wired.
- TierConfigMgr::begin_publish_transition and publish_candidate_inner, thin wrappers whose _with_allowed_mutation_blocks siblings carry every real caller, plus retire_driver.
- The GCS backend's MAX_MULTIPART_PUT_OBJECT_SIZE, MAX_PARTS_COUNT and MIN_PART_SIZE, and its write-only storage_class field.
- mark_started_rebalance_pools_stopped and the RStats alias.
Two deletions were withdrawn after a per-name grep, both because of an inference rather than a check:
AsyncBatchProcessor::new was deleted on the strength of grepping only BATCH_PROCESSOR_OPERATION_CUSTOM, whose two hits are its definition and its use inside new. That looked like a self-contained dead pair; new in fact has seven test callers. The warning listed both items, and only one of them was actually checked.
Deleting the two dead publish wrappers then revealed a second layer — publish_candidate_owned, remove_and_save_with, clear_and_save_with, save_tiering_config_if_current. These are not dead: publish_candidate, their caller, is #[cfg(test)], so a callee that lives in the main body has no caller in the lib build and a live one in the test build. rustc reports the roots of a dead subgraph, and the next layer down can have a different character, so each layer needs its own grep.
Kept with allows: the tier mutation-intent record helpers (asserted by store::init tests), affected_targets, tier_object_blocks_target_rebind, the rebalance snapshot and retry-wait helpers, notification_sys's tier_config_reload_worker_active and call_peer_with_timeout, and active_operation_lease_count, whose only caller sits behind #[cfg(feature = "test-util")].
Also kept, with a module note rather than removal: the ecstore-side EventNotifier. All four of its methods are unreachable and init_bucket_targets logs that it is a no-op in this build; the working stack is rustfs-notify, whose own EventNotifier drives bucket configuration. Removing it means also retiring the InstanceContext slot that holds it (backlog#939 Phase 5), which belongs in its own PR.
Worth a separate issue: MAX_MULTIPART_PUT_OBJECT_SIZE, MAX_PARTS_COUNT and MIN_PART_SIZE are declared independently in eight warm-backend files plus client/constants.rs. Only the GCS copies were dead; the other seven backends each use their own.
Verification, four lanes warning-free: default, --tests, --features rio-v2 --tests, --features test-util --tests. cargo nextest run -p rustfs-ecstore 4041 passed; clippy --lib --tests -D warnings clean; make pre-commit exit 0.
Ref rustfs/backlog#1823 (step 2).
* chore(ecstore): remove the pool-level ListObjects pagination copy
The ListObjects pagination pipeline existed in three near-copies in one file; production listing never reaches the Sets copy, which ECStore bypasses by expanding straight to per-set disks. This removes it: impl ListOperations for Sets (61 lines of pure forwarding in core/sets.rs) and the impl Sets pagination block (826 lines of inner_list_objects_v2 / list_objects_generic / inner_list_object_versions / list_path / list_merged / walk_internal in store/list_objects.rs).
Two preconditions verified before deleting rather than taken on faith: the architecture guard pins only set_disks_implements_storage_list_operations_contract, so nothing requires the Sets trait impl; and the four Sets pagination methods had no cross-file caller besides that trait impl.
The single test consumer moves to the surviving pipeline instead of being deleted: writes still go through the pool, and the listing assertion now targets the set-level implementation. It is renamed accordingly so the name still describes what it covers.
The logging guardrail's TRACE-only requirement for Sets::list_objects_v2 retires in the same diff — the wrapper it pinned no longer exists. The ECStore and SetDisks entries are untouched.
The SetDisks copy stays for now: its trait impl is guard-pinned, so replacing the duplicate pipeline behind it needs the generic helper the issue schedules for post-1.0.
Verification: cargo nextest run -p rustfs-ecstore 4020 passed; check_architecture_migration_rules.sh and check_logging_guardrails.sh pass; clippy --lib --tests -D warnings clean; make pre-commit green.
Ref rustfs/backlog#1821 (PR1).
* chore(ecstore): fold the ListObjects forwarders into the ECStore impl
store/list.rs held two thin forwarders, handle_list_objects_v2 and handle_list_object_versions, that only re-entered the inner_* implementations. The ListOperations impl now calls those directly and the file goes away.
The logging guardrail's trace_hot_spans list pinned handle_list_objects_v2 as TRACE-only; that entry is retired in the same diff, adjacent to the sets.rs entry retired by the preceding commit.
Ref rustfs/backlog#1821.
* chore(ecstore): drop the type aliases orphaned by the pagination removal
core/sets.rs declared four local type aliases — ListObjectsV2Info, ListObjectVersionsInfo, ObjectInfoOrErr and WalkOptions — used only by the pool-level pagination pipeline removed earlier in this branch. store/list_objects.rs keeps its own live copies of the same aliases.
They only surface now that #6087 removed the core module's dead_code blanket: on that older base each PR was warning-free on its own, and the combination is what exposes them. Their storage_api_contracts imports go with them.
Ref rustfs/backlog#1823, rustfs/backlog#1821.
* fix(ecstore): preserve Sets listing compatibility
Removing both blankets exposes 23 items, of which only four are deleted. The ratio is the point: close to the core data path the blankets were hiding test assertions and migration seams, not dead code.
A cfg-split function is the reason two symbols in the internode transport look dead when neither is. build_internode_data_transport_from_env has two bodies, one under #[cfg(test)] that calls build_internode_data_transport directly and one under #[cfg(not(test))] that goes through the INTERNODE_DATA_TRANSPORT static so tests do not share process-global transport state. Each half's helper is live in exactly one build, and because cargo check --tests compiles both the lib target and the test harness, both symbols appear in one warning list. Deleting either one breaks the other lane. Both are kept with allows naming their half.
Three deletion candidates were withdrawn after a per-name grep: ParallelReader::new, ErasureDecodeReader::new and SyncErasureDecodeReader::new all have test callers. The last two are exactly the shape of the dead wrapper deleted in #6084 — a thin forward to a new_with_metrics_path sibling — except that sibling is live in production (set_disk/read.rs) and the wrappers are used by tests.
Deleted:
- RemotePeerS3Client::get_addr and RemoteLocker::from_url, neither with a consumer in any lane.
- RemotePeerS3Client's node field, which new writes after using it to derive addr and nothing ever reads. Its only other writer was a test helper that built a whole Node solely to fill the field; that block goes too.
- ParallelReader::can_decode, superseded by an inlined copy. The copy's comment named the method it replaced, so deleting the method alone would have left a dangling reference; the comment now describes the check instead of pointing at a method that no longer exists.
Kept with allows: the erasure items are decode/encode invariants asserted by their own files' tests (shard_read_launch_order, decode_with_read_costs, emit_data_shards, queued_block_bytes, the engine trait facets, the ParallelReader and decode-reader constructors, encode_stream_callback_async). On the cluster side, peer_replay_state, heal_bucket_local and clone_drives are test-only, InternodeDataTransportCapabilities and tcp_http are constructed only by transport test doubles, and the InternodeDataTransport trait's name/capabilities pair is an unused capability-negotiation facet kept for the transport split (backlog#1350) — six impls provide them and no caller negotiates on them yet.
Verification, four lanes warning-free: default, --tests, --features rio-v2 --tests, --features test-util --tests. cargo nextest run -p rustfs-ecstore 4041 passed; clippy --lib --tests -D warnings clean; make pre-commit exit 0.
Ref rustfs/backlog#1823 (step 2).
heal::Error carried six variants with zero construction and zero match sites (ConfigurationError, NotFound, TaskAlreadyExists, ManagerNotRunning, EventProcessingFailed, ProgressTrackingFailed) plus IO(String), which was never constructed either — its only appearances were two or-pattern match arms that could never fire (task.rs's demotion match and the recoverability classifier). All seven are deleted and the two or-patterns lose their dead alternative.
Config(String) stays (live, four construction sites); Io(std::io::Error) stays; the retry classifier's behavior is untouched per the issue constraint — removing an arm that can never match is not a classification change.
Ref rustfs/backlog#1831 (PR3).
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
Record local read_version path resolution, path length check, xl.meta read, and metadata decode durations through the existing GET stage metrics channel. The new samples are gated by GET stage metrics so metrics-off reads avoid timer and recorder work.
Co-authored-by: heihutu <heihutu@gmail.com>
* refactor(sse): sink managed-SSE attribution into the shared encryption-keys module
Moves the managed-SSE classifier — stored_managed_encryption_key, contains_managed_encryption_metadata, normalize_managed_metadata — and the SSEType enum from rustfs/src/storage/sse.rs into crates/utils/src/http/object_encryption_keys.rs, the module that already owns every constant they read. This is PR-B0 of rustfs/backlog#1643: crates/scanner must never depend on the rustfs binary crate, so encryption attribution has to live in a shared lower layer before the scanner can report per-scheme coverage without growing a second classifier.
SSEType moves wholesale (option a): its only impl is the dependency-free audit_label(), so the enum relocates verbatim (audit_label becomes pub) and rustfs::storage::sse re-exports it, keeping every existing path compiling. The one piece that cannot move verbatim is normalize_managed_metadata's KMS-context branch, which needs base64 and serde_json — dependencies rustfs-utils does not have and does not gain here. The shared normalizer instead takes an injected Option<fn(&str) -> Option<String>> context recoder; sse.rs passes recode_minio_kms_context, the old inline chain verbatim including the silent skip on decode failure. stored_managed_encryption_key passes no recoder because the context mapping only ever inserts the context key, which the key-id lookup never reads, so its output is identical.
Every metadata lookup stays a case-sensitive exact match (lowercase x-amz-* stored forms, TitleCase MinIO-internal names) per the backlog#1775 trap; new shared-module tests pin that, and a source-scan test in sse.rs asserts the classifier has exactly one definition so a second copy cannot silently return.
* fix(utils): satisfy encryption key test clippy
---------
Co-authored-by: cxymds <cxymds@gmail.com>
* fix(kms): construct wrap_budget_reserved in the VaultKeyData deserializer
main does not compile: #6019 added VaultKeyData.wrap_budget_reserved on a base that predated #6003's hand-written Deserialize, so the visitor's struct literal never learned about the field. Each PR was green on its own base; the breakage only exists in their merge.
The field joins the other three lists the hand-written impl maintains (Field enum, match arm, struct literal, FIELDS) and defaults to 0 when absent — the value a record written before wrap accounting, or rewritten by an older build, carries; zero restarts the reservation rather than blocking a wrap.
vault_key_data_deserializer_covers_every_serialized_field turns this class of mistake into a test failure instead of a merge-order accident: it serializes a fully populated record and asserts the deserializer recognizes every emitted key (unknown-field counter stays zero) and reads every value back. Mutation-verified by dropping the new match arm.
* feat(kms): report a key as due for rotation once its wrap budget is spent
The rotation readiness verdict only knew about age; the wrap accounting landed by #6019 counted wraps and published an aggregate gauge but never fed the per-key verdict, leaving the criterion backlog#1636 asks for unimplemented.
RUSTFS_KMS_ROTATION_MAX_WRAPS adds the second, independent threshold, parsed with the same discipline as the age one: unset or unparsable leaves the verdict unreported rather than inventing a policy, and values below one million are raised to it because wraps are reserved in blocks of that size and a smaller threshold would trip on the first reservation regardless of how many wraps happened.
The wrap check runs before the age check so that a key crossing both reports 'wraps': the AES-GCM random-nonce ceiling is a cryptographic bound an operator cannot negotiate, while the age period is a policy they chose. Backends that report no count — Transit and AWS wrap externally, and pre-accounting records carry nothing — leave the wrap half silent instead of guessing, and a backend that cannot rotate is still never told to.
Refs rustfs/backlog#1636 (PR-3 acceptance criterion), rustfs/backlog#1562.
---------
Co-authored-by: houseme <housemecn@gmail.com>
Add a default-off inline-only data-read metadata early-stop gate that verifies inline plaintext before cancelling pending metadata tasks.
Keep non-inline, prepared, and request-shape-sensitive reads on full fanout, and record scheduled/completed/cancelled ReadVersion lifecycle metrics for normal fanout completion.
Co-authored-by: heihutu <heihutu@gmail.com>
Zhengchao An