test(kms): move the Vault KV2 Transit-wrapping doc guard into check_fips_wording.sh
`test_vault_kv2_sources_do_not_claim_transit_wrapping` asserted that four
`include_str!`-pinned files never describe the Vault KV2 backend as wrapping key
material through Vault's Transit engine. The invariant is a documentation-claim
invariant with no behavioral twin by construction, and the test form was weak in
both directions: it saw only four files (the same prose in a fifth file passed
silently) and it stopped compiling — rather than reporting a violation — as soon
as one of them was renamed.
Move the four literals verbatim into `scripts/check_fips_wording.sh`, which
already guards the adjacent cryptographic over-claim class (unsupported FIPS
validation wording) and is anchored to the same policy document. The guard now
greps every file under `crates/kms` for the same four case-sensitive literals and
separately reports a moved pinned source instead of failing to build.
`check_fips_wording.sh` previously ran only in `make pre-commit` / `pre-pr`, so
wire it into the Quick Checks job of both CI workflows to keep the invariant's
failure visibility at least as strong as the deleted test's.
* refactor(replication): split four oversized hot-path functions into focused helpers
Pure-move decomposition of the four oversized functions flagged by the
replication compatibility review (P1-18), unblocking migration milestone
M2 which requires resyncer moves to stay mechanical:
- resync_bucket (522 lines -> 61-line step sequence): leader lock,
target resolution, walk/collector/worker spawning, and dispatch loop
extracted into focused helpers; pure decision helpers (DTO builders,
HEAD-result classification) separated from IO orchestration.
- replicate_all (411 lines -> 113-line main body): initial target-info
seeding, read/stat option builders, skip-path notes, target HEAD
action resolution, and the multipart/single-put payload transport
extracted as private free functions.
- start_mrf_processor (306 lines -> 46-line spawn body): recovery guard,
ledger load, per-entry replay (delete/object/metadata), and retained
entry resolution extracted; retry bookkeeping semantics preserved
exactly (inner continue-paths push inside helpers, outer Missed push
stays in the loop).
- apply_iam_item (255 lines -> match dispatch skeleton): one helper per
IAM item type.
No behavior change: log texts, error paths, event emissions, and metric
counts are byte-identical; existing tests unchanged and green (238
ecstore replication/mrf/resync + 232 rustfs site-replication).
* feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets (#6172)
* feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets
Implements the MinIO active-active read-proxy protocol (P1-5 of the
replication compatibility review): when a GET/HEAD/GetObjectTagging/
PutObjectTagging/DeleteObjectTagging request fails locally with
not-found and the bucket has replication targets, the request is proxied
to the targets in rule order, mirroring bucket-replication.go
proxyGetToReplicationTarget/proxyHeadToRepTarget/proxyTaggingToRepTarget.
Protocol surface:
- Anti-loop: inbound {x-rustfs-,x-minio-}source-proxy-request is parsed
into ObjectOptions (proxy_request + proxy_header_set, matching MinIO
ProxyRequest/ProxyHeaderSet); a request carrying the marker with ANY
value is never re-proxied. Outbound client proxy calls send the marker
as "true"; replication worker convergence HEADs send it as "false" so
a peer's proxy layer cannot answer a convergence check by proxying
back to the source (which would fake Completed without a PUT).
- Target selection: new replication_proxy.rs get_proxy_targets — empty
when the marker is set, versioning is suspended, or no replication
config; otherwise filter_target_arns -> TargetClient lookup, skipping
targets with proxying disabled.
- TargetClient gains head_object_for_proxy/get_object (streaming) and
the three tagging calls. Proxy calls never send the replication-check
SSE-C exemption header; customer SSE-C keys are forwarded verbatim so
the target performs real decryption. Conditional (If-*) headers are
not forwarded (MinIO parity); Range and part_number are, with
parts_count/tag_count/storage_class/expiration passed through.
- Metrics: proxy counters now count only real client proxy traffic,
MinIO-aligned (one total per proxied request, one failed when no
target served it). The previous misattributed counters — replication
worker HEAD/PUT (#2672) and local tagging operations (#2682) — are
removed; ReplProxyMetric now maps the tagging counters instead of
dropping them.
e2e (fake_s3_target extended with tagging + header journaling): proxied
GET body + outbound header contract (marker present, no
replication-check, SSE-C passthrough), HEAD, anti-loop 404 with zero
outbound requests, GetObjectTagging, and metric mapping unit tests.
Rolling note: proxying only activates for buckets with replication
targets; requests carrying the marker keep pre-upgrade behavior.
Refs rustfs/backlog#1675 (P1-5)
* fix(replication): fail SSE-C passthrough closed on targets that drop transport headers (#6178)
SSE-C ciphertext passthrough replicates via X-Rustfs-Replication-* transport
headers. A MinIO/generic-S3 target silently discards them, storing bare
ciphertext with no decryption material — yet the PUT succeeded, so the object
reported COMPLETED with a silently unreadable replica (backlog#1675 N2).
Fail-closed design:
- SsecPassthroughCapability {Unknown, Supported, Unsupported} cached in
BucketTargetSys per target ARN with a recording timestamp. Entries reset
whenever the target is rebuilt, edited, or removed (arn_remotes_map
lifecycle) and expire after SSEC_PASSTHROUGH_CAPABILITY_TTL (10 minutes):
an expired verdict in either direction is re-earned through the audit, so
an Unsupported target recovers automatically after an upgrade (at most one
wasted PUT+HEAD audit per bad target per TTL window) and a Supported
verdict cannot outlive a backend swapped behind the same endpoint.
- Replication worker (replicate_object and replicate_all): fresh Unsupported
targets never receive the PUT — the attempt fails immediately into the
normal MRF retry channel with a "run ?replication-check to re-probe" hint.
Unknown or expired verdicts are audited: after the PUT the worker HEADs
the replica back through the replication-check channel (source version id
mapped through resolve_read_api_version_id, so null-version objects audit
correctly) and requires SSE-C evidence (the echoed customer-algorithm
header); missing evidence records Unsupported and fails the attempt.
Convergence HEADs are audited the same way, so a broken ciphertext replica
from an earlier attempt can never launder itself into COMPLETED via an
ETag match. The gate/evidence policy is pure (replication_target_boundary,
staleness folded in as an input) for the M2 worker migration.
- replication-check grows an SsecPassthrough probe phase: a probe PUT
carrying the live transport-header shape, HEAD-back for evidence, and a
machine-readable Code BucketRemoteSsecPassthroughUnsupported on failure.
The probe verdict is synced into the runtime capability cache. Unlike
VersionFidelity, a failed SsecPassthrough phase does NOT fail the target
overall — it is a capability limit, not a broken replication contract,
and a plaintext-only deployment against such a target must not turn red.
- fake_s3_target: default mode now models a RustFS target (stores the
transport headers, echoes SSE-C evidence); the new
drop_unlisted_replication_headers mode models MinIO. The journal records
whether a request carried transport headers.
Receiver-echo verification: the replication-check HEAD exemption only skips
SSE-C key validation; the response has always built sse-customer-algorithm
from stored metadata (rustfs/src/app/object_usecase.rs), so no receiver
change was needed — pinned end to end by the replication-check e2e against
a real RustFS target.
Rolling-upgrade constraint: RustFS targets older than the replication-check
HEAD exemption (#5898) answer the audit HEAD without SSE-C evidence (or fail
it outright), so SSE-C replication to such targets reports FAILED. This is
deliberate — FAILED-and-retryable beats a silently undecryptable replica —
and self-heals: once the target is upgraded, the next TTL expiry (or a
manual ?replication-check re-probe) re-audits and records Supported.
Plaintext and managed-SSE replication are unaffected. The capability cache
is per-node; each node audits independently.
Known limitations:
- The audit judges evidence from the echoed customer-algorithm header only.
A hypothetical target that preserves that one header while dropping other
transport headers (partial-drop) would pass the audit; no known target
behaves this way — observed targets drop the whole unknown-header family.
- A mixed-version target cluster can flap the verdict between audits routed
to different target nodes until the rollout completes; the TTL bounds how
long each stale verdict persists.
New e2e (backlog#1675 C1 + N2, red-first): fail-closed against a
header-dropping fake (FAILED + no second PUT via the capability cache,
journal-asserted; red run showed the old COMPLETED), replication-check
reports the SsecPassthrough phase Code while the target stays OK overall,
SSE-C heal convergence after a real target outage, and SSE-C
existing-object resync landing a REPLICA readable with the customer key.
TTL expiry in both directions is pinned at the cache and gate seams.
* refactor(replication): move resyncer pure decision logic into rustfs-replication (M2) (#6180)
* refactor(replication): move resyncer pure decision logic into rustfs-replication (M2)
Pure-move milestone M2 of the ECStore replication split (backlog#1675
P1-17): relocate the resyncer's IO-free decision helpers, with their unit
tests, into the crates they already belong to by type ownership. No
behavior change.
Moved into crates/replication:
- resync.rs: resync_status_duration
- delete.rs: resync_existing_delete_replication_info,
replicate_delete_outcome, target_delete_version_id,
delete_marker_purge_version_id, delete_marker_purge_mrf_entry
- object.rs: version_identity_drifted, is_replication_target_offline_error,
SsecPassthroughCapability, SsecPassthroughGate, ssec_passthrough_gate,
ssec_passthrough_evidence_present (param-demoted to the echoed
customer-algorithm string; ECStore keeps the HeadObjectOutput adapter)
- filemeta.rs: NULL_VERSION_ID wire literal (crate-owned copy per the
filemeta-independence contract)
ECStore rewiring (Rule #14: imports stay in *_boundary.rs):
- resync/object-decision/target boundaries re-export the moved symbols;
resyncer call sites are unchanged
- bucket_target_sys keeps only the verdict cache + TTL and re-exports the
capability enum so existing consumer paths keep compiling
Not moved (signatures carry ECStore or aws-sdk types):
verify_resync_head_result, resync_target_error_detail, the SdkError
classifiers, the replicate_all_* option/info builders, and the env-coupled
bounded_resync_max_jobs admission clamp. README milestone table updated.
* chore(replication): retire the datatypes.rs relay early
README sanctions retiring datatypes.rs ahead of M4. The module was a
pure relay (resync boundary -> datatypes -> mod.rs facade) with no
external consumer importing it directly, so the facade now re-exports
ResyncStatusType from replication_resync_boundary and the relay file is
deleted. Consumers stay behind the ECStore facade, keeping Migration
Rule #15 intact — the original retirement wording ("consumers import
through rustfs-replication directly") conflicted with that rule and is
corrected in the README.
* chore(arch): extend migration guards to the M2-moved decision contracts
The adversarial review of the M2 move found the per-symbol ratchet in
check_architecture_migration_rules.sh was not extended for the moved
symbols, leaving them free to be redefined in ECStore or imported past
their boundary without CI noticing:
- resync definition pin + boundary fences gain resync_status_duration;
- the object-decision boundary fences gain the five delete-family
helpers (delete_marker_purge_mrf_entry, delete_marker_purge_version_id,
replicate_delete_outcome, resync_existing_delete_replication_info,
target_delete_version_id);
- the target-boundary fence gains the SSE-C gate family, the offline
classifier, and version_identity_drifted;
- a new definition pin rejects ECStore redefinitions of the M2-moved
fns/enums (ssec_passthrough_evidence_present deliberately excluded:
ECStore keeps a thin HeadObjectOutput adapter under that name).
Mutation-verified: a probe fn ssec_passthrough_gate under
crates/ecstore/src/bucket/replication trips the new pin.
Also anchors the intentionally-duplicated NULL_VERSION_ID wire literal
from the filemeta side and tightens the M2 README note on
bounded_resync_max_jobs.
* test(io-metrics): assert lib.rs record helper emissions
The 29 assertion-less record_* smoke tests in io-metrics/src/lib.rs called
their helpers and checked nothing; because METRICS_ENABLED defaults to
false they did not even reach the emission bodies. Replace them with six
DebuggingRecorder tests that enable the gate, pin every metric name the
helpers own, and pin the derived values, branch selection and label
mapping (rustfs/backlog#1836).
* test(tooling): anchor assertless-census delegation tokens to name segments
Use the merged s3s single-chunk StreamingBlob support for exact-length materialized GET bodies when RUSTFS_GET_SMALL_BODY_ONCE_ENABLE is enabled.
Keep the default path unchanged and fall back to the guarded MemoryTrackedBytesStream on length mismatch.
Co-authored-by: heihutu <heihutu@gmail.com>
Second batch of the #[serial] sweep started in #6209. nextest is the
repository's authoritative runner and executes every test in its own
process, so serial_test's in-process mutex cannot serialize tests against
each other -- docs/testing/README.md documents this, and the mechanism
that actually serializes across the process boundary is a
.config/nextest.toml [test-groups] entry with max-threads = 1.
Unlike the first batch (e2e, process-isolated by construction), these are
in-crate unit tests that could genuinely share process state under the
`cargo test` fallback runner, where #[serial] IS still effective. Every
marker was therefore reviewed individually and removed only where the
test provably touches neither the process environment nor a process-global.
Removed (47, pure deletions, no test bodies touched):
crates/lifecycle/src/core.rs 35
crates/scanner/src/scanner.rs 12
The lifecycle removals are all validate_* / filter_rules_* /
has_active_rules_* / noncurrent_versions_expiration_limit_* tests that
build a local BucketLifecycleConfiguration and call a &self method
walking only that value. The scanner removals are pure duration
arithmetic (randomized_cycle_delay_for, initial_scanner_delay_for with an
explicit Some(secs), the bitrot-disabled early return of
scanner_clean_idle_max_interval) and background_heal_info_for_scan_complete
/ _for_scan_result field comparisons over locally built values.
Retained deliberately -- see the PR body for the full list and reasons:
crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs 101 (all)
crates/lifecycle/src/core.rs 44
crates/scanner/src/scanner.rs 53
bucket_lifecycle_ops.rs keeps every marker: its test module caches a
process-wide `static STALE_MULTIPART_TEST_ENV: OnceLock<(Vec<PathBuf>,
Arc<ECStore>)>`, and its own reregister_env_local_disks helper documents
in-tree that sibling #[serial] tests reset and reshape the shared
local-disk registry for each other. That sharing is real, so the markers
stay.
No test was renamed, added, or deleted; no reserved migration-gate name
substring is affected; no .config/nextest.toml entry references any of
the 47 removed tests.
rustfs-zip has one workspace consumer, and it uses only
CompressionFormat::{from_extension, extension, get_decoder} and
ArchiveLimits. Remove the tar/zip extract, zip create, and in-memory
compress helpers together with the types and dependencies that only
served them. Trimming public API is semver-major once the stable tag is
cut, so it costs least now.
Add default-off PUT stage attribution for the rename_data sync tail so strict durability probes can split queue wait, fdatasync, directory fsync, rename, per-disk wait, and quorum wait without changing commit ordering or S3-visible behavior.
Co-authored-by: heihutu <heihutu@gmail.com>
test(e2e): drop no-op serial markers from the three densest e2e suites
serial_test's #[serial] is an in-process mutex. cargo-nextest, this repo's
authoritative runner, executes every test in its own process, so the mutex is
never contended and the attribute is a documented no-op -- see the "Serial
execution & nextest profiles" section of docs/testing/README.md and the header
of .config/nextest.toml. Cross-process serialization is provided only by a
[test-groups] entry with max-threads = 1.
Remove 187 such markers (plus 3 now-unused imports) from the three
marker-densest modules of crates/e2e_test:
multipart_auth_test.rs 85
replication_extension_test.rs 68
object_lock/object_lock_test.rs 34
None of these modules is covered by any [test-groups] entry, so the markers
were carrying no isolation for any lane. Every test in all three files builds
its own server via RustFSTestEnvironment::new(), which gives a UUID temp dir
and a uniquely allocated port -- the .config/nextest.toml comment on
replication_extension_test already states this explicitly ("parallel-safe by
construction"). No test mutates process env, binds a fixed port, or touches
process-global state, so nothing here needed temp_env or a test-group instead.
Pure deletion: 190 lines removed, 0 added, no test renamed, no behaviour
changed. serial_test stays in Cargo.toml -- 338 markers across 90 other files
in the crate still use it.
Refs: backlog#1846 (T1).
* feat(heal): incremental heal status cursors and typed overlap policy (HS-06)
Incremental results: every retained result item now carries a monotonic
sequence number. The status query accepts a client cursor (sinceSeq on
the admin wire, Option<u64> internally) and returns only newer items,
plus nextSeq (the next cursor) and minSeq (the oldest retained
sequence). A cursor that fell behind the 1024-item retention window is
flagged through the existing truncated signal together with minSeq so
the client can restart from it. Sequencing survives task completion:
the completion archive stores the seq-stamped window. None keeps the
exact legacy full-snapshot behavior, so existing clients see no change.
Typed overlap handling for admin starts: RUSTFS_HEAL_OVERLAP_POLICY
(merge default | minio_error). Under minio_error, an admin start whose
path overlaps an active or queued task rejects with typed
already-running / overlapping-paths admission reasons (surfaced through
reason_label in the admin error body, sharing the existing
OperationAborted site because the s3s footprint ratchet forbids new
s3_error! sites); an exact duplicate start rejects with
already-running instead of silently merging. Scanner/autoheal/
read-repair sources never take the rejection path.
forceStart semantics now match MinIO for admin requests: an admin
forceStart first cancels the overlapping active admin task, then
admits the replacement.
Wire: the heal-control Query command grows an optional sinceSeq
(defaulted and skipped when absent, so older peers stay compatible);
the admin handler accepts the sinceSeq query parameter; the local
channel query gains the same cursor.
Tests: seq monotonicity and incremental slicing, window slide moving
minSeq with lagging-cursor flags, overlap matrix (same/containing/
contained/disjoint x policy x source), forceStart cancel-then-admit,
and the completion-archive window handoff.
Co-Authored-By: heihutu <heihutu@gmail.com>
* style: fmt after main merge
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): single-flight remote disk recovery
* test(ecstore): cover remote recovery review cases
* test(ecstore): exercise recovery through disk slot
* test(ecstore): match format reads exactly
* test(ecstore): cover recovery teardown races
* style(ecstore): format recovery race tests
* fix(ecstore): remove unused health snapshot helper
* fix(ecstore): group recovery monitor test state
* fix(protos): preserve production source in compatibility checks
Neither rustfs/src/storage/backpressure.rs nor rustfs/src/storage/lock_optimizer.rs had a production caller: their only non-self references were the pub mod lines in storage/mod.rs and a cfg(test) module, so the object transfer path never applied this backpressure and never took these lock shortcuts.
The six removed tests in concurrent_fix_test.rs duplicated tests that lived inside the deleted files; the shared primitives they shadowed keep their own coverage in rustfs-io-core.
Remove every bare `#[allow(dead_code)]` in io-core, object-capacity, targets, rio, and scanner. Each allow was stripped first and clippy was then asked which ones the compiler actually missed, so the verdicts rest on the diagnostic rather than on inspection.
23 were inert: they sat on `pub fn`s inside `pub mod`s, where `dead_code` does not apply, or on scanner integration-test helpers that the tests in the same file do call.
The remaining 3 are in rio's private `compress_index` module and the code behind them is deleted rather than annotated. `remove_index_headers` is dead and also wrong — after skipping the 4-byte chunk header it matches against `S2_INDEX_TRAILER` where `S2_INDEX_HEADER` sits, so it returns `None` for every well-formed index; rio-v2 carries the correct equivalent that is actually in use. `restore_index_headers` is its unreachable counterpart, likewise duplicated live in rio-v2. `Index::reset` is a private method with no caller.
Refs backlog#1823
* feat(common): add MRF intent channel and Mrf request source (HS-01)
Introduce the producer-facing half of the mission repair feed: a global
bounded (8192) channel carrying lightweight MrfIntent values from IO
error paths, plus the RUSTFS_HEAL_MRF_ENABLE delivery kill-switch and
config constants for queue/journal sizing. Delivery is strictly
non-blocking (try_send, drop-on-full) so it can sit on decode-failure
and partial-write paths without adding latency. HealRequestSource grows
a 'mrf' variant so admission accounting can attribute replayed intents.
Part of backlog#1865 (option a: wire HealEvent-style intents with a
durable retry ledger).
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(heal): add MRF queue, durable journal, and intent consumer (HS-01)
Consumer half of the mission repair feed: a bounded pending queue
(100k intents / 8 MiB dual ceiling, drop-newest on overflow), a durable
journal at buckets/.heal/mrf/journal.bin holding the unaccepted pending
snapshot, and a consumer task that batches intents off the global
channel, translates them into prioritized heal requests (decode
failure -> Urgent ECDecode, metadata corruption -> High Metadata,
partial write -> Normal object heal), and retries full admissions with
a 5s backoff and a 3-attempt ceiling.
Durability: every journal record carries its own CRC32 and a
format/version header, so a torn tail truncates cleanly at replay; the
journal is deleted after a successful replay and when the pending set
drains (mirroring MinIO's post-replay list.bin unlink). Losing the last
500 ms flush window is acceptable: replayed duplicates merge via the
manager dedup key and read-repair remains the safety net.
Metrics: rustfs_heal_mrf_queue_depth/_queue_bytes, _dropped_total
{reason}, _replayed_total, _journal_bytes, _journal_fsync_total.
The consumer is wired at heal runtime bootstrap right after manager
start, honoring RUSTFS_HEAL_MRF_ENABLE (default on, rollback = off).
Tests: unit tests for the dual ceiling, record roundtrip, torn-tail
truncation, and the priority mapping; integration tests against a real
4-disk ECStore proving channel intents reach the manager queue as
Urgent/mrf-attributed requests and journal replay arms intents, drops
torn tails, and removes the file.
Part of backlog#1865 (option a).
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ecstore,scanner): deliver MRF intents from error paths (HS-01)
Wire the three production delivery points, each a single non-blocking
try_send next to the existing in-memory heal paths, which stay as the
fast path:
- read.rs decode-error branch: DecodeFailure intent beside the existing
read-repair submit, so an Urgent ECDecode request survives restarts
even when the Low-priority read-repair request was dropped or lost.
- add_partial: PartialWrite intent, giving partial-write recovery a
durable Normal-priority object heal across restarts.
- scanner_folder metadata-corruption classification: MetadataCorruption
intent beside the existing High-priority scanner heal request.
All three are on error paths only: zero cost on healthy IO.
Part of backlog#1865 (option a).
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix: include mrf heal source counts
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix: keep node heal status wire compatibility
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Add an experimental fixed-count foreground PutObject admission gate for #1882 Phase 0 validation. The gate is default-off, returns SlowDown before body ingest when saturated, and keeps the admission permit with the spawned store commit owner until store PUT returns.
Co-authored-by: heihutu <heihutu@gmail.com>
* test(heal): add concurrency invariants for heal vs delete/overwrite races (HS-12)
Audit conclusion for backlog#1874: RustFS does not need a persistent
object-level healing marker (MinIO x-minio-healing) because every path
that can touch the same (bucket, object) commit surface serializes on
the same namespace write lock, and the heal lock guard spans the whole
rename commit including the HEAL_RENAME_INCOMPLETE partial path.
Lock the conclusion in with two race regression tests:
- heal_racing_version_delete_never_resurrects_the_deleted_version:
shard damage is injected on the doomed version so a Deep heal has real
reconstruction work while a versioned DELETE runs concurrently; the
deleted version must stay deleted and the survivor intact.
- heal_racing_unversioned_overwrites_preserves_the_last_commit:
unversioned overwrites (activating the post-commit tail that deletes
the replaced data dir without the ns lock) race a Deep heal in a
loop; the final current version must be exactly the last commit.
Also adds docs/operations/heal-concurrency-safety-notes-zh.md with the
full intersection matrix (17 intersections), lock-coverage argument,
and the residual-window classification (commit tail races are
fail-into-retry safe; bare prefix delete has zero production callers;
admin no_lock is an explicit operator opt-in).
Co-Authored-By: heihutu <heihutu@gmail.com>
* test: remove redundant heal etag clone
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
heal_bucket, list_bucket, get_bucket_info and delete_bucket returned Error::other("") when a peer answered success=false without an error payload, so operators saw a bare "io error " after quorum reduction. Route all five bucket RPCs through peer_failure_without_details, which names the operation and bucket while staying identical across the peers of one operation so reduce_errs keeps grouping them into a single dominant error.
* feat(scanner): emit excess alerts as S3 notification events
The excess-versions / excess-version-size / excess-folders alerts were
metrics-and-logs only; consoles and external auditors had no way to hear
them (rustfs/backlog#1868, HS-04). MinIO emits s3:ObjectManyVersions /
s3:ObjectLargeVersions / s3:PrefixManyFolders for the same conditions —
RustFS carries those as EventName::Scanner* with s3:Scanner:* wire names
that already existed unpublished.
The three alert sites now also dispatch through the standard event
pipeline (send_event via the storage_api owner facade), carrying the
actual values and thresholds in req_params and UserAgent "Scanner".
Without a cooldown a single over-threshold object would re-emit on every
~60s scan cycle, so emissions are edge-held per (kind, bucket, object)
for 24h (RUSTFS_SCANNER_ALERT_COOLDOWN_SECS, 0 = every cycle), backed by
a process-global map with a 4096-key hard cap that clears rather than
grows. Metrics and structured logs stay level-triggered every cycle;
only the notification events are held back. A restart resets the
cooldown deliberately: one re-emission per still-hot key buys back
visibility after the restarts that accompany incident response.
Tests pin the edge-hold semantics (first fires, immediate re-check held,
independent keys, cooldown expiry re-fires, zero cooldown always emits,
hard bound) in one sequential test for the process-global map, and pin
the emitted wire names against EventName's canonical string forms so a
subscribed bucket notification can never silently stop matching.
docs/operations/scanner-excess-alerts.md documents the three events,
the metric-vs-event cadence difference, and the HS-15 threshold deltas
(alert_excess_folders 65538 vs MinIO 50000 is deliberate: Proxmox
Backup Server chunk layout compatibility).
Closesrustfs/backlog#1868.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(operations): split scanner excess alerts into English and Chinese pages
The page shipped Chinese-only; keep it as scanner-excess-alerts_zh.md and
add a faithful English translation at the original path, cross-linked at
the top of both.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(heal): track erasure set progress baseline
Record erasure-set heal byte progress from per-object results and seed progress totals from complete usage-cache snapshots when available.
Keep usage-cache failures observational so heal execution continues without a baseline.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(heal): skip filtered erasure set versions
Skip erasure-set versions written after the durable heal start time, and queue lifecycle-expired versions for expiry before skipping them.
Track new-version and ILM-expired skips separately so progress can explain completed baseline work without treating these skips as retry-blocking failures.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(heal): wire abandoned data-dir cleanup check
Connect check_abandoned_parts through ECStore, pool, and set layers so heal can invoke the existing orphan data-dir reclaim path instead of returning NotImplemented.
Add dry-run support to the reclaim scan and cover dry-run plus scoped set behavior with regression tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(obs): add heal scanner trace bus
Introduce an in-process broadcast trace bus with typed heal and scanner events, lazy event construction, and bounded lagged-subscriber behavior.
Cover zero-subscriber publishing, subscription delivery, drop accounting, and lagged receivers with focused common-crate tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(obs): stream heal trace events from admin API
Wire the admin trace endpoint to the common trace bus for heal/scanner events, including kind, regex, and threshold filtering.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(obs): emit heal trace events
Publish heal task lifecycle and abandoned-parts cleanup events through the common trace bus so the admin trace stream has live heal diagnostics.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(obs): emit scanner trace events
Publish scanner folder, lifecycle action, and heal-candidate events through the common trace bus for live admin scanner diagnostics.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(heal): route data usage loader through storage api
Keep ECStore data-usage facade access behind the heal storage_api boundary so architecture migration guards can validate the heal progress path.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(heal): avoid lifecycle snapshots on ordinary heal pages
Only request lifecycle object snapshots when the heal pass has lifecycle expiry context. This keeps ordinary listing and disk-walk pages from cloning FileInfo/ObjectInfo payloads while preserving the skip path that queues expired versions.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(heal): update bug-fix mocks for lifecycle snapshots
Carry the lifecycle snapshot opt-in argument through the remaining heal bug-fix test mocks so all-targets clippy covers the updated storage trait.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rustfs): sync heal storage mock signature
Update the rustfs storage RPC test mock for the lifecycle snapshot opt-in argument and cover it with rustfs all-targets clippy.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): allocate smoke ports across nextest processes
Serialize E2E port selection with a small /tmp allocator so nextest workers do not reuse the same just-released ephemeral port before RustFS binds it.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Replace every bare `#[allow(dead_code)]` in ecstore with either a deletion or a per-item allow carrying a `reason`. Blanket allows at module, struct, and impl level silence the lint for future members too, so each is narrowed to the members that are actually dead.
Delete the dead cluster in `config/heal.rs` (`Config`, its three methods, `RUSTFS_BITROT_CYCLE_IN_MONTHS`, `parse_bitrot_config`) rather than annotate it: it has no callers and is unreachable outside the crate, and `parse_bitrot_config` would panic on its disabled path via `Duration::from_secs_f64(-1.0)`. `DEFAULT_KVS` stays, since the config registry uses it.
Correct two `reason` strings on `Checksum::new` and `PutObjReader::md5_current_hex_string`, which are methods but carried a field-only rationale.
Refs backlog#1823
Co-authored-by: houseme <housemecn@gmail.com>
Add an optional TLS passthrough listener to the Gateway API support. When gatewayApi.listeners.tls.enabled is true, the Gateway gets a TLS listener with tls.mode: Passthrough and a TLSRoute is rendered to the RustFS service so TLS terminates at the backend (end-to-end encryption).
Refs rustfs/rustfs#3862.
feat(scanner): expose prefix-level bucket usage via admin API
The scanner's per-bucket, per-set usage caches already hold a path-keyed
prefix tree, but dui() flattened it only to bucket names — consoles and
operators had no way to ask "what does this prefix hold" without an S3
listing sweep (rustfs/backlog#1872, MinIO loadPrefixUsageFromBackend
parity).
Add:
- data-usage: prefix_usage_in_cache — a shared aggregation over the
entry map (arbitrary prefix, full counters, one-level sub-prefix
breakdown with names recovered from the literal-path cache keys),
hardened like the scanner's checked flatten: cycles, dangling child
links, over-deep trees, and overflowing counters yield None rather
than unbounded recursion or wrapped totals.
- ecstore: ECStore::all_set_disks — iterate every erasure set so a
query can read each set's own cache copy; the hash-routed store path
would always land on one set.
- scanner: bucket_prefix_usage — per-set loads (5s budget each, a slow
set degrades to not-reporting instead of stalling the caller),
merged across sets with partial/compacted/truncated flags, served
from a bounded 30s cache (128 entries, hard-capped) that bucket
writes invalidate through the dirty-usage hook.
- admin: GET /rustfs/admin/v3/usage/{bucket}?prefix=&max-entries=
behind the same any-of gate as datausageinfo (DataUsageInfoAdminAction
OR ListBucketAction), rejecting unknown query parameters and
clamping max-entries to 1..=10000. Route registered in the policy
table (deferred MultipleActions, matching datausageinfo) and the
route matrix test.
Closesrustfs/backlog#1872.
Co-authored-by: heihutu <heihutu@gmail.com>
feat(madmin): add a SigV4-signed admin client for heal and scanner APIs
The madmin crate held only wire types; automation and mc-style tooling
had no way to drive the heal/scanner admin surface without hand-rolled
HTTP. Add `AdminClient`, which signs with the same rustfs-signer path
the server authenticates (UNSIGNED-PAYLOAD marker, matching RustFS peer
admin calls) and wraps:
- heal_start / heal_status / heal_stop over POST /rustfs/admin/v3/heal/
(bucket/prefix path params percent-encoded per segment; stop models
the server's two cancel branches: token-scoped task status vs
path-scoped start-success receipt);
- background_heal_status, scanner_status (freshness typed), plus
ilm_expiry_status / replacement_recovery_status passthroughs;
- a public get_json escape hatch for endpoints not wrapped yet.
Wire types follow the madmin-go model (SDK-owned mirrors pinned by
round-trip tests): HealOpts with serde defaults so partial settings
objects decode, HealScanMode accepting both the numeric and name
encodings, and status structs that type the fields operators branch on
while flattening unknown nested payloads verbatim so server additions
cannot break the client. Errors map to a closed AdminClientError enum
(InvalidEndpoint / Transport / HttpStatus with body / Decode).
Tests cover wire round-trips, path building, both stop branches, error
mapping, and — via a dependency-free raw-TCP test server — that signed
requests carry a SigV4 Authorization header, the right method/path/
query, and the expected JSON body.
Closesrustfs/backlog#1869 (first increment; single-sourcing the wire
structs server-side and an embedded-server e2e roundtrip are noted as
follow-ups there).
Co-authored-by: heihutu <heihutu@gmail.com>
feat(ecstore): pin bitrot algorithms with a startup self-test
A drifted HighwayHash implementation fails silently: every shard reads
back corrupt, heal rewrites healthy data, and cross-platform clusters
disagree about which copy is good. Mirror MinIO's bitrotSelfTest by
verifying, once at process start:
- known-answer digests for HighwayHash256S / HighwayHash256SLegacy over
a deterministic 4096-byte xorshift64* payload, plus the externally
verifiable FIPS SHA-256 "abc" vector guarding the HashAlgorithm
plumbing itself;
- an end-to-end roundtrip per streaming variant (encode -> size formula
-> bitrot_verify -> BitrotReader read-back), over full blocks and a
partial tail;
- tamper detection: one flipped byte in the final data block and one in
the leading hash must both be rejected as a hash mismatch, not by an
incidental read error.
The check costs microseconds and runs inline in
init_background_service_runtime before any shard can be written or
verified. Outcome surfaces as one structured bitrot_selftest log event,
the rustfs_bitrot_selftest_status gauge (1=passed / 0=failed / 2=skipped),
a bitrotSelftest field on the admin server-info response, and
RUSTFS_BITROT_SELFTEST_STRICT=on turns a failure into a startup error
(MinIO Fatal parity; the default only degrades the status so a bad build
cannot brick an existing fleet on upgrade).
Closesrustfs/backlog#1873 (HS-11).
Co-authored-by: heihutu <heihutu@gmail.com>
backlog#1823 step 10, batch 1 of the repo-wide item-allow sweep. 227 bare #[allow(dead_code)] remain across 83 files; this takes the 19 in utils, notify, checksums, policy, keystone and trusted-proxies, which are small enough to verify end to end.
Removing all 19 first, before writing any reason, matters: 8 of them suppress nothing. Every allow in utils, one in policy and three in notify sit on items that are publicly reachable, so dead_code never applied to them — the same shape as the swift module and kms's dek.rs. Writing a reason onto a no-op allow would dress noise up as considered judgement, so those are simply deleted.
Three items are genuinely dead and go with their allows: notify's new_target_id_set, the AWS metadata fetcher's get_metadata_token, and policy's empty `pub struct Value;`, none of which is referenced anywhere in the tree.
The remaining eight keep an allow, now saying why the item survives rather than who calls it. Two are exercised only by their own crate's tests (checksums' MD5_HEADER_NAME, policy's is_match_as_pattern_prefix). Four are fields written but never read back: keystone's verify_ssl, parsed from config after the reqwest client is already built; keystone's client handle, which keeps the Keystone client alive for the mapper's lifetime; the AWS IMDS endpoint, kept beside the client while requests build their own URLs; and notify's rules_map, whose own comment retains it for snapshot-time judgements no code performs.
checksums' Md5 needed the most care. Crc32, Sha256 and seven others each have an arm in ChecksumAlgorithm::into_impl, and Md5 has none, which reads like a missing algorithm. It is not: ChecksumAlgorithm has no Md5 variant at all. S3 carries Content-MD5 as its own header, separate from the x-amz-checksum-* family, and this impl exists so both paths share the Checksum trait. The reason records that, so the next reader does not re-derive it.
One measurement note for anyone continuing this sweep: cargo does not re-emit warnings for cached compilations, so a per-crate loop of `cargo check -p <crate>` under-reports. checksums showed zero that way while actually carrying three. Touch the sources and check the crates in one invocation, then attribute by path.
Verification: the six crates are warning-free under cargo check --tests; clippy --lib --tests -D warnings clean; cargo nextest run 1096 passed; make pre-commit exit 0.
Ref rustfs/backlog#1823 (step 10).
backlog#1823 step 10, batch 2. Eighteen of the nineteen suppress nothing and are deleted; one was real and keeps an allow that now says why.
Rotation::Never is constructed only by the rolling-appender tests at rolling.rs:456, 477 and 498, so the lib target reports it as never constructed. Its allow is restored with that reason.
Finding it corrected the method used for batch 1. Removing all nineteen and running cargo check -p rustfs-obs --tests reported zero warnings even after touching every source file, while clippy --lib --tests -D warnings caught Rotation::Never. cargo's warning output is not a reliable completeness check — it does not re-emit for cached compilations, and touching the sources did not cover the lib target here. Later batches should treat clippy -D warnings as the gate; batch 1's six crates were re-checked under clippy and are clean.
Taken with #6086, which cleared this crate's 44 module-level blankets and left six real items, obs has now had 63 dead-code suppressions examined, of which seven were suppressing anything at all. The rest sat on items that are publicly reachable, where dead_code never applied — the same shape as the swift module and kms's dek.rs.
Verification: clippy --lib --tests -D warnings clean in the default, gpu and pyroscope lanes; cargo nextest run -p rustfs-obs 324 passed; make pre-commit exit 0.
Ref rustfs/backlog#1823 (step 10).
Use MemoryTrackedBytesStream directly as an s3s ByteStream so in-memory GET bodies avoid the generic StreamingBlob::wrap adapter while preserving exact remaining length, request lifecycle tracking, and length-mismatch failure semantics.
Co-authored-by: heihutu <heihutu@gmail.com>
backlog#1823 step 9. The step 2 burn-down cleared every module-level #![allow(dead_code)] from ecstore, but nothing stops the next PR from adding one back, and the other module-level blankets (unused_variables, unused_must_use, clippy::all) were never counted at all.
Two rules land in check_architecture_migration_rules.sh:
ecstore must carry zero module-level #![allow(dead_code)]. The count is asserted at zero rather than registered, since there is nothing left to grandfather; a genuinely unused item takes an item-level allow with a reason, which is what step 2 produced roughly 350 times.
Every other module-level blanket must match scripts/ecstore-module-lint-register.txt exactly — 94 entries across 33 files, nearly all of them in the MinIO-ported client module.
The exact match is the point. A "no new entries" rule lets the register rot into an amnesty list, which is the failure mode backlog#1834 found in the layer-dependency baseline: rebuilt at 29 entries, 2 more added by a later PR, zero retired. Here, removing a blanket costs one line in the register, so it can only shrink, and adding one shows up as a register line a reviewer has to accept.
Verified by injection, since a guard that cannot fail is worse than no guard: adding a dead_code blanket, adding an unregistered clippy::all blanket, and deleting a registered blanket without updating the register each produce the expected failure, and the tree passes once reverted.
make pre-commit exit 0.
Ref rustfs/backlog#1823 (step 9).
backlog#1823 step 1, the diagnosis half. Temporarily removing set_disk/mod.rs's #![allow(unused_variables)] surfaced nine bindings. The issue asks that values computed and then dropped on write/quorum paths be diagnosed before being underscored, and that turned out to matter: only four were plain leftovers.
Two errors were bound and then left out of the log they were bound for. complete_multipart_upload's checksum failures read `if let Err(err) = ...` and then log part_id, bucket and object with no `err` anywhere in the message, so a checksum failure in production told you which part failed but not why. Both messages now carry the error.
One is a lock guard. heal's write_lock_guard holds a namespace write lock for the rest of the scope; renaming it to a bare `_` would drop it immediately and release the lock. It is now `_write_lock_guard`, with a comment saying why it must not be `_`.
One was kept alive by a corpse. `errors` in read_multiple_files is read by nothing except two commented-out debug! lines directly below it; the binding and the commented lines go together.
One is a cfg split. heal's disk_index is read only inside the #[cfg(test)] fault-injection branch, so underscoring it would break the test build; a `#[cfg(not(test))] let _ = disk_index;` covers the non-test lane instead.
The remaining four are genuine leftovers: an unused enumerate index in list_object_parts, a discarded error in a heal reader loop, an inner binding shadowing its own iterator variable, and delete_object's write_quorum.
That last one is worth a separate look: delete_object asks get_object_info_and_quorum for a write quorum and never uses it, because delete_object_version below recomputes its own as disks.len() / 2 + 1. The two are not the same number — one comes from the object's erasure configuration, the other is a plain majority of the disk array. Pre-existing behaviour, untouched here.
The blankets stay for now. Removing #![allow(unused_imports)] exposes 76 unused imports in set_disk/mod.rs, and they cannot be removed per-lane: cargo fix, working from the lib lane, produced 54 compile errors in the test lane. That needs its own pass with both lanes checked per import.
Verification: cargo check -p rustfs-ecstore --tests and --features test-util --tests both warning-free; clippy --lib --tests -D warnings clean; cargo nextest run -p rustfs-ecstore 4101 passed; make pre-commit exit 0.
Ref rustfs/backlog#1823 (step 1).
backlog#1834 PR5. Whether the scanner, heal, audit and notify modules are on gets read from infra (storage helpers, node-service RPC) and from interface (admin handlers), but the switches lived in startup_background (composition) and server (interface). Every one of those reads was an upward edge carried in the layer-dependency baseline.
The env-derived scanner/heal predicates and the audit/notify state cells now live in rustfs/src/module_switches.rs, at the bottom of the layer order, so the same reads are ordinary downward edges. startup_background and server import from there; server keeps re-exporting the getters for its own consumers.
The issue's plan was to move is/refresh_audit/notify_module_enabled as a group. Moving refresh_* wholesale would have dragged resolve_audit_module_state and resolve_notify_module_state — server-side configuration logic — down into infra, which breaks more layering than it fixes. State and resolution are split instead: module_switches owns the atomics plus is_*/set_* accessors, and server's refresh_* keeps the configuration logic and publishes through the setter.
That leaves storage/helper.rs's test module importing refresh_* from server, so two infra->interface edges stay. Those tests assert that a configuration change takes effect through refresh, which a plain setter would no longer exercise; the edges are worth more than the two baseline lines.
Baseline drops 44 -> 36 lines, deletions only:
- 4 interface/infra -> composition edges for ENV_SCANNER_ENABLED, scanner_enabled_from_env and heal_enabled_from_env
- 2 infra -> interface edges for is_audit_module_enabled and is_notify_module_enabled
- cycle|composition<->infra and cycle|composition<->interface
The two cycles were not expected to go until whole subsystems moved out; clearing composition's inbound upward edges dissolved both, leaving three of the original five.
Verification: scripts/check_layer_dependencies.sh passes, cargo check -p rustfs warning-free, make pre-commit exit 0.
backlog#1823 step 8, partial. The swift module carries 43 #[allow(dead_code)] attributes, most with a comment naming a consumer: "Used by handler", "Handler integration: GET container", "Used by handler and object.rs".
Every one of them suppresses nothing. crates/protocols/src/lib.rs declares `pub mod swift`, and swift/mod.rs declares all 22 submodules `pub mod`, so every item is publicly reachable and dead_code never applied to it. Removing all 43 leaves the warning count at zero, in both the default and --features swift lanes.
That is also why those comments survived. They assert who calls the item — a claim the compiler normally settles on its own — and the compiler had been silenced by the visibility chain.
The rest of step 8 needs a decision this PR does not make. Downgrading the 22 submodules to `pub(crate) mod` does restore detection, and it surfaces 39 real items, 16 of them the whole of sync.rs: SyncConfig, SyncStatus, SyncQueueEntry, ConflictResolution and every function and constant around them, i.e. Swift container sync is built and never wired.
But the `pub mod` chain is load-bearing. Six integration tests under crates/protocols/tests are separate crates that import the submodules directly (swift::quota, swift::slo, swift::symlink, swift::sync, swift::tempurl, swift::container), and the downgrade fails to compile them. Restoring dead-code detection for this module therefore depends on first deciding whether those tests move in-crate — which is a testing-strategy call, not a cleanup one.
Verification: cargo check -p rustfs-protocols warning-free in the default lane and with --features swift (lib and --tests); clippy --features swift --lib --tests -D warnings clean; cargo nextest run -p rustfs-protocols --features swift 441 passed; make pre-commit exit 0.
Ref rustfs/backlog#1823 (step 8).
Zhengchao An