The last item-level bare allow of backlog#1823 step 10. `SessionDiag` itself is live — `sftp/server.rs` constructs one per accepted connection and `wedge_watchdog` reads `session_id`, `peer` and `last_activity_ms` off it — so the struct-level blanket was covering exactly one field: `accepted_at`, which is written at accept time and never read back. The allow moves onto that field with a reason.
The three remaining `#![allow(dead_code)]` in this crate (`sftp/test_support.rs`, `common/dummy_storage.rs`) are module-root blankets in test-support files, which belong to steps 1-5 rather than step 10.
Refs backlog#1823
docs(operations): land the heal/scanner MinIO audit baseline with closure results
Move the comprehensive heal/scanner vs MinIO analysis (2026-08-16) into
docs/operations/ so it finally enters the tree — the docs/ root is
ignored by the gitignore whitelist, which is why the baseline the audit
issue referenced as "to be merged with a PR" never landed. Append §9
closure results: all 14 backlog sub-issues (#1865-#1878) closed with the
per-item PR map, two further misjudgment corrections (HS-17 was already
implemented; HS-14's MinIO idle semantics drifted upstream), HS-12/HS-18
audit conclusions, and the registered follow-ups.
Backlog issue: rustfs/backlog#1862
Co-authored-by: heihutu <heihutu@gmail.com>
* refactor(scanner): drop the always-None single-disk default cycle hook
single_disk_default_cycle_secs returned None for every maintenance
feature combination, so the single-disk startup path already resolved
its default cycle from the speed preset (60s at 'default'). Remove the
never-wired hook and its pin tests, keep the explicit reset, and record
the decision: no special single-disk cycle override without measured
cold-start ILM latency evidence; clean-idle backoff already stretches
idle cadence (backlog#1878 HS-16).
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(operations): add heal/scanner MinIO parity decision notes
Document the HS-14/16/18 decision batch from backlog#1878: the scanner
idle throttling semantics matrix (RUSTFS_SCANNER_IDLE_MODE x speed
preset x foreground read backoff) side by side with MinIO's current
static idle_speed switch as verified against upstream master, the
migration warnings for env names and value vocabularies, the bitrot
cycle default divergence (30d vs off), the stale-multipart / tmp / trash
three-stage cleanup comparison with the crash-residue window grading,
and the single-disk default cycle decision.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
`audit_runtime_facade_stops_empty_replay_workers` called the stop path and checked nothing, the same shape as the notify facade test in the previous commit. It now asserts the worker manager is empty afterwards and that a second call — which shutdown paths make — stays harmless.
The two heal timestamp tests bound their fields to `_`. Both timestamps come from `SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default()`, so a pre-epoch clock yields 0; binding to `_` could not tell that apart from a real reading, which is precisely what the tests said they were guarding. They now require the value to be past 2020-01-01, and `last_update` not to predate `start_time`.
`test_config_parsing_with_multiple_instances` is left as it was — see the issue comment. Asserting on it turned up something bigger than a missing assertion.
Refs backlog#1836
`SizeSummary` and `ReplTargetSizeSummary` existed in both `rustfs-data-usage` and `rustfs-scanner`, and the two copies had drifted three ways: four size fields were `usize` in one and `i64` in the other, only the scanner's carried `tier_stats`, and — the difference that matters — the scanner's `add` saturated while the data-usage copy used plain `+=`, which panics on overflow in a debug build and wraps in a release one.
The data-usage copy is now the only definition and takes the scanner's shape and semantics, since that is the side a test already pinned (`MAX + 1 == MAX`). An equivalent saturation test now guards it in its new home. The scanner re-exports both types alongside the ones it already re-exported.
`DataUsageEntry::add_sizes` and `BucketUsageInfo::add_size_summary` are removed. Both took a `SizeSummary` and had no callers anywhere — they were the duplicate fold paths, and `apply_scanner_size_summary` is now the only one.
`actions_accounting` stays in the scanner as the `ScannerSizeSummaryExt` extension trait: it needs `ObjectInfo`, which sits above `rustfs-data-usage`, and an inherent impl on a foreign type is not allowed. The three call sites are unchanged.
Refs backlog#1828
Eight tests in this crate called a recorder and asserted nothing. Five of them were worse than that: every `record_*` in `list_objects_metrics` returns early unless `get_stage_metrics_enabled()` is true, and that flag defaults to false, so those tests only ever exercised the early return — never the code their names describe.
They now run against a local `DebuggingRecorder` with the flag on, and each asserts the boundary it is named for: an empty page reports the scan count as its amplification instead of dividing by zero, a zero read quorum is recorded rather than skipped, index serving divides verification attempts by returned objects, and the `-1` whole-directory sentinel reaches the limit histogram unclamped.
`msgpack_json_fallback_counter_records_without_panicking` has no in-struct total to check, so it now asserts the emission: two direction/message pairs must land in two separate series, which a dropped label would collapse into one.
The two process-sampler tests discarded their snapshots. They now assert what cannot differ between callers — a process has one start time and one descriptor limit regardless of which entry point or which sampler observed it, and the status enum must match its numeric projection.
This clears io-metrics from the census (`scripts/find_assertless_tests.py`), taking the tree from 61 candidates to 53.
Refs backlog#1836
`is_data_usage_cache_absent` matched `FileNotFound | VolumeNotFound`, but `SetDisks::get_object_reader` runs its failures through `to_object_err`, which rewrites those to `ObjectNotFound` and `BucketNotFound` before they reach the caller. The classifier therefore never matched in production: a cache object that simply does not exist was treated as a transient failure, retried five times with backoff, and then reported as an error instead of an empty cache. Admin server-info resolves one cache per erasure set, so that is roughly 1.5s of pointless backoff per set on any cluster whose scanner has not written a cache yet.
The same rewrite is why the pre-existing `FileNotFound | VolumeNotFound` arm in the old loop never fired either, which left the legacy-key fallback beside it unreachable — it only ever returned an empty cache through the catch-all break.
The classifier now covers the rewritten variants as well as the raw pair, the test store reports absence the way `to_object_err` does, and a new test pins which variants actually arrive.
Refs backlog#1828
Avoid fixed response-layer work on the ordinary GET path by bypassing CORS request cloning when no Origin header is present and by only splitting/rebuilding compatibility responses when their target conditions match.
Add service-level regression tests for CORS, S3 error, Iceberg REST, ObjectAttributes, and bodyless-status compatibility paths.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ecstore): make the data-usage cache load actually retry
`load_data_usage_cache` wrapped its read in `while retries < 5`, but every arm of the match inside broke out of the loop, so `retries` was never incremented and the random sleep below it was unreachable: the loop always ran exactly once. The fallback arm compounded this by re-matching the *outer* error after the legacy-key read failed, which meant its second arm could not be reached either.
The read now goes through `rustfs_utils::retry::retry_with_backoff`. A key that is absent under both the prefixed and the legacy name still yields an empty cache without retrying, since retrying a definitive absence cannot turn it into a hit. A transient failure is retried with capped, jittered backoff and surfaces as an error once the attempts are exhausted, instead of being reported as an empty cache — the sole caller already maps `Err` to `usage_error = DATA_USAGE_UNAVAILABLE`, so a read failure now says "unavailable" rather than "zero usage".
`load_data_usage_cache` is generic over `ObjectIO` rather than taking `&SetDisks`, which is what makes the retry and fallback ordering testable at all; being untestable is why the inert loop survived. The call site passes `as_ref()` instead of cloning the `Arc` it immediately borrowed.
Refs backlog#1828
* fix(ecstore): route the load bound through the storage-api contracts
The generic bound named `rustfs_storage_api::ObjectIO` directly, which the architecture guard rejects: ecstore modules must reach storage-api symbols through `crates/ecstore/src/storage_api_contracts`. The bound is now the crate's own `EcstoreObjectIO` alias, which pins the same associated types in one place.
That alias is `pub(crate)`, so `load_data_usage_cache` becomes `pub(crate)` too rather than exposing a crate-private bound on a public signature. Nothing outside ecstore called it — its only caller is `diagnostics/admin_server_info.rs`, and it was never re-exported from the crate root.
---------
Co-authored-by: houseme <housemecn@gmail.com>
* fix(sse): read objects that MinIO encrypted
RustFS could not read a single MinIO-encrypted object. Two independent blockers, and backlog#1638 could only argue them statically because the fixtures the interop tests consume are generated, not checked in — so those tests had never once run. With the fixture lab working, both are now measured, fixed and covered.
The detection gate required `x-amz-server-side-encryption` to be present. MinIO never persists it: `crypto.S3.CreateMetadata` writes only the `X-Minio-Internal-*` family and the public header is synthesized onto the response by `DecryptObjectInfo`. Every MinIO object therefore fell out of the managed path and failed with "encrypted object metadata is incomplete". The scheme is now inferred from which sealed-key slot is present, which is self-consistent by construction: the slot decides both which header the unseal reads and which domain string the sealing key is derived under, so an inference that disagreed with the slot could not silently derive a wrong key. Inferring from the KMS key id would NOT be safe — MinIO writes `-S3-Kms-Key-Id` on SSE-S3 objects too, which the fixtures show and a mutation test pins.
Past the gate, the data key itself could not be unwrapped. Its wire format is `sealed_bytes || iv[16] || nonce[12]` — the randomness trails the ciphertext rather than leading it — with a per-ciphertext sealing key of `HMAC-SHA256(master, iv)` and the encryption context bound as associated data (`internal/kms/secret-key.go`). Note this is not the `{"aead":...}` JSON that backlog#1638's analysis described: current MinIO writes the raw layout and treats JSON only as a legacy encoding, normalizing it into the same byte order. Both are decoded here, in a decoder of their own — `LocalSseDekEnvelope`'s `deny_unknown_fields` is untouched, since loosening it to admit MinIO's shape would also admit malformed RustFS envelopes that backlog#1567 requires to keep failing closed.
Routing between the two decoders cannot key on metadata: RustFS's own writer fills MinIO's slots while storing a RustFS envelope in them, so neither the slot nor the header name distinguishes writers. It keys on the data key's own shape instead, recognizing the two strict RustFS JSON shapes positively and leaving only the remainder to MinIO — so neither decoder is ever handed the other's format. Three round-trip tests caught an earlier slot-based attempt doing exactly that.
Fail-closed is preserved throughout: a scheme that cannot be established still returns None, and the read plan independently classifies the object as encrypted from its markers and refuses to serve it without material, so no path degrades into returning ciphertext as plaintext.
The interop harness also gets a provider reset. The DEK provider is cached process-wide, so a case that ran earlier kept serving its master key to every later case — which silently made the wrong-key negative test unable to fail. It fails correctly now, and the whole suite is meaningful for the first time.
Refs rustfs/backlog#1638.
* fix(sse): gate the MinIO data-key trait method behind rio-v2
The method's only call site sits in the rio-v2 branch of the managed read path, so a build without that feature carried a trait method nothing could reach — a warning under default features and, with -D warnings, a hard failure of the sftp lane. The declaration now carries the same gate its implementation and its sibling decrypt_legacy_sse_dek already had.
Verified against the lane that caught it (cargo clippy -p rustfs --features sftp --all-targets -- -D warnings, clean), plus the default build and the rio-v2 interop suite (4 passed).
Refs rustfs/backlog#1638.
---------
Co-authored-by: houseme <housemecn@gmail.com>
Route strict inline rename_data dst-parent fsync through the default-off group-commit helper when enabled while preserving the namespace file-sync limited path by default.
Co-authored-by: heihutu <heihutu@gmail.com>
The Code Map entry still summarised io-core as "buffer pool, storage profiling, admission control", which predates #6201 removing eight zero-consumer modules. The crate now exposes pool, io_profile, config, backpressure, deadlock_detector, lock_optimizer, and progress, so the summary names the policy, lock, and progress helpers as well.
Refs backlog#1824
Avoid constructing the typed Accept-Ranges string on the GetObject output path. Inject the static header after CORS wrapping so the final S3 response remains unchanged while the hot path avoids one fixed per-GET allocation/conversion.
Co-authored-by: heihutu <heihutu@gmail.com>
Use Linux openat2 with RESOLVE_BENEATH and RESOLVE_NO_SYMLINKS for LocalDisk I/O path validation while keeping the existing lstat walk as the public-path and unsupported-kernel fallback. Add focused regression coverage for traversal, symlink swaps, missing leaves, recreated parents, high-cardinality prefixes, final symlink leaves, and concurrent validation.
Co-authored-by: heihutu <heihutu@gmail.com>
Byte-exactness tests stay green if the compressed range seek regresses into
decoding from byte zero: the returned bytes are still correct and only the
read amplification explodes. Assert the cost side as well.
The observation reuses rustfs_io_get_object_shard_read_observed_bytes_total,
already emitted per shard read by the erasure layer, so no production code is
instrumented. The OTLP collector learns to accumulate a second counter, keyed
by its path/role/outcome labels rather than by data-point position, which is
not stable across exports.
Two failure modes the assertions guard against:
- With RUSTFS_OBS_METER_INTERVAL=1, treating one unchanged sample as settled
measures a delta of zero, because the range read's counter has not been
exported yet. Settling now requires several consecutive equal samples.
- An upper bound alone passes vacuously on a zero delta, so a lower bound
turns "measured nothing" into a failure instead of a green run.
Refs rustfs/rustfs#5957, backlog#1848.
test(kms): move the Vault KV2 Transit-wrapping doc guard into check_fips_wording.sh
`test_vault_kv2_sources_do_not_claim_transit_wrapping` asserted that four
`include_str!`-pinned files never describe the Vault KV2 backend as wrapping key
material through Vault's Transit engine. The invariant is a documentation-claim
invariant with no behavioral twin by construction, and the test form was weak in
both directions: it saw only four files (the same prose in a fifth file passed
silently) and it stopped compiling — rather than reporting a violation — as soon
as one of them was renamed.
Move the four literals verbatim into `scripts/check_fips_wording.sh`, which
already guards the adjacent cryptographic over-claim class (unsupported FIPS
validation wording) and is anchored to the same policy document. The guard now
greps every file under `crates/kms` for the same four case-sensitive literals and
separately reports a moved pinned source instead of failing to build.
`check_fips_wording.sh` previously ran only in `make pre-commit` / `pre-pr`, so
wire it into the Quick Checks job of both CI workflows to keep the invariant's
failure visibility at least as strong as the deleted test's.
* refactor(replication): split four oversized hot-path functions into focused helpers
Pure-move decomposition of the four oversized functions flagged by the
replication compatibility review (P1-18), unblocking migration milestone
M2 which requires resyncer moves to stay mechanical:
- resync_bucket (522 lines -> 61-line step sequence): leader lock,
target resolution, walk/collector/worker spawning, and dispatch loop
extracted into focused helpers; pure decision helpers (DTO builders,
HEAD-result classification) separated from IO orchestration.
- replicate_all (411 lines -> 113-line main body): initial target-info
seeding, read/stat option builders, skip-path notes, target HEAD
action resolution, and the multipart/single-put payload transport
extracted as private free functions.
- start_mrf_processor (306 lines -> 46-line spawn body): recovery guard,
ledger load, per-entry replay (delete/object/metadata), and retained
entry resolution extracted; retry bookkeeping semantics preserved
exactly (inner continue-paths push inside helpers, outer Missed push
stays in the loop).
- apply_iam_item (255 lines -> match dispatch skeleton): one helper per
IAM item type.
No behavior change: log texts, error paths, event emissions, and metric
counts are byte-identical; existing tests unchanged and green (238
ecstore replication/mrf/resync + 232 rustfs site-replication).
* feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets (#6172)
* feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets
Implements the MinIO active-active read-proxy protocol (P1-5 of the
replication compatibility review): when a GET/HEAD/GetObjectTagging/
PutObjectTagging/DeleteObjectTagging request fails locally with
not-found and the bucket has replication targets, the request is proxied
to the targets in rule order, mirroring bucket-replication.go
proxyGetToReplicationTarget/proxyHeadToRepTarget/proxyTaggingToRepTarget.
Protocol surface:
- Anti-loop: inbound {x-rustfs-,x-minio-}source-proxy-request is parsed
into ObjectOptions (proxy_request + proxy_header_set, matching MinIO
ProxyRequest/ProxyHeaderSet); a request carrying the marker with ANY
value is never re-proxied. Outbound client proxy calls send the marker
as "true"; replication worker convergence HEADs send it as "false" so
a peer's proxy layer cannot answer a convergence check by proxying
back to the source (which would fake Completed without a PUT).
- Target selection: new replication_proxy.rs get_proxy_targets — empty
when the marker is set, versioning is suspended, or no replication
config; otherwise filter_target_arns -> TargetClient lookup, skipping
targets with proxying disabled.
- TargetClient gains head_object_for_proxy/get_object (streaming) and
the three tagging calls. Proxy calls never send the replication-check
SSE-C exemption header; customer SSE-C keys are forwarded verbatim so
the target performs real decryption. Conditional (If-*) headers are
not forwarded (MinIO parity); Range and part_number are, with
parts_count/tag_count/storage_class/expiration passed through.
- Metrics: proxy counters now count only real client proxy traffic,
MinIO-aligned (one total per proxied request, one failed when no
target served it). The previous misattributed counters — replication
worker HEAD/PUT (#2672) and local tagging operations (#2682) — are
removed; ReplProxyMetric now maps the tagging counters instead of
dropping them.
e2e (fake_s3_target extended with tagging + header journaling): proxied
GET body + outbound header contract (marker present, no
replication-check, SSE-C passthrough), HEAD, anti-loop 404 with zero
outbound requests, GetObjectTagging, and metric mapping unit tests.
Rolling note: proxying only activates for buckets with replication
targets; requests carrying the marker keep pre-upgrade behavior.
Refs rustfs/backlog#1675 (P1-5)
* fix(replication): fail SSE-C passthrough closed on targets that drop transport headers (#6178)
SSE-C ciphertext passthrough replicates via X-Rustfs-Replication-* transport
headers. A MinIO/generic-S3 target silently discards them, storing bare
ciphertext with no decryption material — yet the PUT succeeded, so the object
reported COMPLETED with a silently unreadable replica (backlog#1675 N2).
Fail-closed design:
- SsecPassthroughCapability {Unknown, Supported, Unsupported} cached in
BucketTargetSys per target ARN with a recording timestamp. Entries reset
whenever the target is rebuilt, edited, or removed (arn_remotes_map
lifecycle) and expire after SSEC_PASSTHROUGH_CAPABILITY_TTL (10 minutes):
an expired verdict in either direction is re-earned through the audit, so
an Unsupported target recovers automatically after an upgrade (at most one
wasted PUT+HEAD audit per bad target per TTL window) and a Supported
verdict cannot outlive a backend swapped behind the same endpoint.
- Replication worker (replicate_object and replicate_all): fresh Unsupported
targets never receive the PUT — the attempt fails immediately into the
normal MRF retry channel with a "run ?replication-check to re-probe" hint.
Unknown or expired verdicts are audited: after the PUT the worker HEADs
the replica back through the replication-check channel (source version id
mapped through resolve_read_api_version_id, so null-version objects audit
correctly) and requires SSE-C evidence (the echoed customer-algorithm
header); missing evidence records Unsupported and fails the attempt.
Convergence HEADs are audited the same way, so a broken ciphertext replica
from an earlier attempt can never launder itself into COMPLETED via an
ETag match. The gate/evidence policy is pure (replication_target_boundary,
staleness folded in as an input) for the M2 worker migration.
- replication-check grows an SsecPassthrough probe phase: a probe PUT
carrying the live transport-header shape, HEAD-back for evidence, and a
machine-readable Code BucketRemoteSsecPassthroughUnsupported on failure.
The probe verdict is synced into the runtime capability cache. Unlike
VersionFidelity, a failed SsecPassthrough phase does NOT fail the target
overall — it is a capability limit, not a broken replication contract,
and a plaintext-only deployment against such a target must not turn red.
- fake_s3_target: default mode now models a RustFS target (stores the
transport headers, echoes SSE-C evidence); the new
drop_unlisted_replication_headers mode models MinIO. The journal records
whether a request carried transport headers.
Receiver-echo verification: the replication-check HEAD exemption only skips
SSE-C key validation; the response has always built sse-customer-algorithm
from stored metadata (rustfs/src/app/object_usecase.rs), so no receiver
change was needed — pinned end to end by the replication-check e2e against
a real RustFS target.
Rolling-upgrade constraint: RustFS targets older than the replication-check
HEAD exemption (#5898) answer the audit HEAD without SSE-C evidence (or fail
it outright), so SSE-C replication to such targets reports FAILED. This is
deliberate — FAILED-and-retryable beats a silently undecryptable replica —
and self-heals: once the target is upgraded, the next TTL expiry (or a
manual ?replication-check re-probe) re-audits and records Supported.
Plaintext and managed-SSE replication are unaffected. The capability cache
is per-node; each node audits independently.
Known limitations:
- The audit judges evidence from the echoed customer-algorithm header only.
A hypothetical target that preserves that one header while dropping other
transport headers (partial-drop) would pass the audit; no known target
behaves this way — observed targets drop the whole unknown-header family.
- A mixed-version target cluster can flap the verdict between audits routed
to different target nodes until the rollout completes; the TTL bounds how
long each stale verdict persists.
New e2e (backlog#1675 C1 + N2, red-first): fail-closed against a
header-dropping fake (FAILED + no second PUT via the capability cache,
journal-asserted; red run showed the old COMPLETED), replication-check
reports the SsecPassthrough phase Code while the target stays OK overall,
SSE-C heal convergence after a real target outage, and SSE-C
existing-object resync landing a REPLICA readable with the customer key.
TTL expiry in both directions is pinned at the cache and gate seams.
* refactor(replication): move resyncer pure decision logic into rustfs-replication (M2) (#6180)
* refactor(replication): move resyncer pure decision logic into rustfs-replication (M2)
Pure-move milestone M2 of the ECStore replication split (backlog#1675
P1-17): relocate the resyncer's IO-free decision helpers, with their unit
tests, into the crates they already belong to by type ownership. No
behavior change.
Moved into crates/replication:
- resync.rs: resync_status_duration
- delete.rs: resync_existing_delete_replication_info,
replicate_delete_outcome, target_delete_version_id,
delete_marker_purge_version_id, delete_marker_purge_mrf_entry
- object.rs: version_identity_drifted, is_replication_target_offline_error,
SsecPassthroughCapability, SsecPassthroughGate, ssec_passthrough_gate,
ssec_passthrough_evidence_present (param-demoted to the echoed
customer-algorithm string; ECStore keeps the HeadObjectOutput adapter)
- filemeta.rs: NULL_VERSION_ID wire literal (crate-owned copy per the
filemeta-independence contract)
ECStore rewiring (Rule #14: imports stay in *_boundary.rs):
- resync/object-decision/target boundaries re-export the moved symbols;
resyncer call sites are unchanged
- bucket_target_sys keeps only the verdict cache + TTL and re-exports the
capability enum so existing consumer paths keep compiling
Not moved (signatures carry ECStore or aws-sdk types):
verify_resync_head_result, resync_target_error_detail, the SdkError
classifiers, the replicate_all_* option/info builders, and the env-coupled
bounded_resync_max_jobs admission clamp. README milestone table updated.
* chore(replication): retire the datatypes.rs relay early
README sanctions retiring datatypes.rs ahead of M4. The module was a
pure relay (resync boundary -> datatypes -> mod.rs facade) with no
external consumer importing it directly, so the facade now re-exports
ResyncStatusType from replication_resync_boundary and the relay file is
deleted. Consumers stay behind the ECStore facade, keeping Migration
Rule #15 intact — the original retirement wording ("consumers import
through rustfs-replication directly") conflicted with that rule and is
corrected in the README.
* chore(arch): extend migration guards to the M2-moved decision contracts
The adversarial review of the M2 move found the per-symbol ratchet in
check_architecture_migration_rules.sh was not extended for the moved
symbols, leaving them free to be redefined in ECStore or imported past
their boundary without CI noticing:
- resync definition pin + boundary fences gain resync_status_duration;
- the object-decision boundary fences gain the five delete-family
helpers (delete_marker_purge_mrf_entry, delete_marker_purge_version_id,
replicate_delete_outcome, resync_existing_delete_replication_info,
target_delete_version_id);
- the target-boundary fence gains the SSE-C gate family, the offline
classifier, and version_identity_drifted;
- a new definition pin rejects ECStore redefinitions of the M2-moved
fns/enums (ssec_passthrough_evidence_present deliberately excluded:
ECStore keeps a thin HeadObjectOutput adapter under that name).
Mutation-verified: a probe fn ssec_passthrough_gate under
crates/ecstore/src/bucket/replication trips the new pin.
Also anchors the intentionally-duplicated NULL_VERSION_ID wire literal
from the filemeta side and tightens the M2 README note on
bounded_resync_max_jobs.
* test(io-metrics): assert lib.rs record helper emissions
The 29 assertion-less record_* smoke tests in io-metrics/src/lib.rs called
their helpers and checked nothing; because METRICS_ENABLED defaults to
false they did not even reach the emission bodies. Replace them with six
DebuggingRecorder tests that enable the gate, pin every metric name the
helpers own, and pin the derived values, branch selection and label
mapping (rustfs/backlog#1836).
* test(tooling): anchor assertless-census delegation tokens to name segments
Use the merged s3s single-chunk StreamingBlob support for exact-length materialized GET bodies when RUSTFS_GET_SMALL_BODY_ONCE_ENABLE is enabled.
Keep the default path unchanged and fall back to the guarded MemoryTrackedBytesStream on length mismatch.
Co-authored-by: heihutu <heihutu@gmail.com>
Second batch of the #[serial] sweep started in #6209. nextest is the
repository's authoritative runner and executes every test in its own
process, so serial_test's in-process mutex cannot serialize tests against
each other -- docs/testing/README.md documents this, and the mechanism
that actually serializes across the process boundary is a
.config/nextest.toml [test-groups] entry with max-threads = 1.
Unlike the first batch (e2e, process-isolated by construction), these are
in-crate unit tests that could genuinely share process state under the
`cargo test` fallback runner, where #[serial] IS still effective. Every
marker was therefore reviewed individually and removed only where the
test provably touches neither the process environment nor a process-global.
Removed (47, pure deletions, no test bodies touched):
crates/lifecycle/src/core.rs 35
crates/scanner/src/scanner.rs 12
The lifecycle removals are all validate_* / filter_rules_* /
has_active_rules_* / noncurrent_versions_expiration_limit_* tests that
build a local BucketLifecycleConfiguration and call a &self method
walking only that value. The scanner removals are pure duration
arithmetic (randomized_cycle_delay_for, initial_scanner_delay_for with an
explicit Some(secs), the bitrot-disabled early return of
scanner_clean_idle_max_interval) and background_heal_info_for_scan_complete
/ _for_scan_result field comparisons over locally built values.
Retained deliberately -- see the PR body for the full list and reasons:
crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs 101 (all)
crates/lifecycle/src/core.rs 44
crates/scanner/src/scanner.rs 53
bucket_lifecycle_ops.rs keeps every marker: its test module caches a
process-wide `static STALE_MULTIPART_TEST_ENV: OnceLock<(Vec<PathBuf>,
Arc<ECStore>)>`, and its own reregister_env_local_disks helper documents
in-tree that sibling #[serial] tests reset and reshape the shared
local-disk registry for each other. That sharing is real, so the markers
stay.
No test was renamed, added, or deleted; no reserved migration-gate name
substring is affected; no .config/nextest.toml entry references any of
the 47 removed tests.
rustfs-zip has one workspace consumer, and it uses only
CompressionFormat::{from_extension, extension, get_decoder} and
ArchiveLimits. Remove the tar/zip extract, zip create, and in-memory
compress helpers together with the types and dependencies that only
served them. Trimming public API is semver-major once the stable tag is
cut, so it costs least now.
Add default-off PUT stage attribution for the rename_data sync tail so strict durability probes can split queue wait, fdatasync, directory fsync, rename, per-disk wait, and quorum wait without changing commit ordering or S3-visible behavior.
Co-authored-by: heihutu <heihutu@gmail.com>
test(e2e): drop no-op serial markers from the three densest e2e suites
serial_test's #[serial] is an in-process mutex. cargo-nextest, this repo's
authoritative runner, executes every test in its own process, so the mutex is
never contended and the attribute is a documented no-op -- see the "Serial
execution & nextest profiles" section of docs/testing/README.md and the header
of .config/nextest.toml. Cross-process serialization is provided only by a
[test-groups] entry with max-threads = 1.
Remove 187 such markers (plus 3 now-unused imports) from the three
marker-densest modules of crates/e2e_test:
multipart_auth_test.rs 85
replication_extension_test.rs 68
object_lock/object_lock_test.rs 34
None of these modules is covered by any [test-groups] entry, so the markers
were carrying no isolation for any lane. Every test in all three files builds
its own server via RustFSTestEnvironment::new(), which gives a UUID temp dir
and a uniquely allocated port -- the .config/nextest.toml comment on
replication_extension_test already states this explicitly ("parallel-safe by
construction"). No test mutates process env, binds a fixed port, or touches
process-global state, so nothing here needed temp_env or a test-group instead.
Pure deletion: 190 lines removed, 0 added, no test renamed, no behaviour
changed. serial_test stays in Cargo.toml -- 338 markers across 90 other files
in the crate still use it.
Refs: backlog#1846 (T1).
* feat(heal): incremental heal status cursors and typed overlap policy (HS-06)
Incremental results: every retained result item now carries a monotonic
sequence number. The status query accepts a client cursor (sinceSeq on
the admin wire, Option<u64> internally) and returns only newer items,
plus nextSeq (the next cursor) and minSeq (the oldest retained
sequence). A cursor that fell behind the 1024-item retention window is
flagged through the existing truncated signal together with minSeq so
the client can restart from it. Sequencing survives task completion:
the completion archive stores the seq-stamped window. None keeps the
exact legacy full-snapshot behavior, so existing clients see no change.
Typed overlap handling for admin starts: RUSTFS_HEAL_OVERLAP_POLICY
(merge default | minio_error). Under minio_error, an admin start whose
path overlaps an active or queued task rejects with typed
already-running / overlapping-paths admission reasons (surfaced through
reason_label in the admin error body, sharing the existing
OperationAborted site because the s3s footprint ratchet forbids new
s3_error! sites); an exact duplicate start rejects with
already-running instead of silently merging. Scanner/autoheal/
read-repair sources never take the rejection path.
forceStart semantics now match MinIO for admin requests: an admin
forceStart first cancels the overlapping active admin task, then
admits the replacement.
Wire: the heal-control Query command grows an optional sinceSeq
(defaulted and skipped when absent, so older peers stay compatible);
the admin handler accepts the sinceSeq query parameter; the local
channel query gains the same cursor.
Tests: seq monotonicity and incremental slicing, window slide moving
minSeq with lagging-cursor flags, overlap matrix (same/containing/
contained/disjoint x policy x source), forceStart cancel-then-admit,
and the completion-archive window handoff.
Co-Authored-By: heihutu <heihutu@gmail.com>
* style: fmt after main merge
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): single-flight remote disk recovery
* test(ecstore): cover remote recovery review cases
* test(ecstore): exercise recovery through disk slot
* test(ecstore): match format reads exactly
* test(ecstore): cover recovery teardown races
* style(ecstore): format recovery race tests
* fix(ecstore): remove unused health snapshot helper
* fix(ecstore): group recovery monitor test state
* fix(protos): preserve production source in compatibility checks
overtrue