* test(odm): drive the migration cases from an env-named source
The ODM e2e suite only ever migrates from the in-process fake source, so
path-style addressing, region handling, ETag shape and list pagination on
real implementations stay untested. OdmInteropEnv resolves the source from
RUSTFS_ODM_INTEROP_*, seeding into a per-run source_prefix so a shared real
bucket can host concurrent runs and every seeded key is removed afterwards.
A named provider with a missing variable is an error, never a silent
fallback to the fake source.
interop_test holds the four cases that run against either source, and the
e2e-odm-interop profile is the lane that selects them; e2e-full excludes
them, so its committed selection is unchanged. wait_until_odm_engaged
replaces the fake source's journal probe for the readiness wait, since a
real source keeps no journal.
* ci(odm): add the scheduled provider interop lane
on-demand-migration-interop.yml runs the interop cases against a pinned
MinIO container with a 5,000-object backfill - past the fake source's 4,096
version and journal caps - and the three-case minimum against AWS, R2 and
GCS when their ODM_INTEROP_* secrets exist, skipping with a summary note
when they do not. Each provider gets one JSON report merging the per-case
entries with the nextest JUnit, which stays authoritative for what ran.
Report-only and never required: it depends on third-party endpoints and on
secrets a fork does not have.
* feat(odm): merge the source listing into ListObjectsV2
Adds policy.list_through: ListObjectsV2 merges the local and source
listings into one ordered page so clients see the whole namespace during
an on-demand migration. Local entries win a key both sides hold,
CommonPrefixes are unioned under a delimiter, and the continuation token
is an opaque versioned envelope carrying both cursors.
A source listing failure or an open breaker follows policy.source_error:
propagate answers 424, not_found answers from local state and marks the
response x-rustfs-on-demand-migration-list: local_only. Source listings
are capped at 10 per second per bucket.
* test(odm): refresh the e2e-full darwin selection digest
The list-through e2e module adds seven cases to the merge lane.
* fix(odm): declare the remote client retry policy per consumer
The SDK retry policy was an inherited default: one logical call could cost
three wire requests, so the migration breaker counted logical calls on top
of a threefold amplification against a source that was already failing.
Make it an explicit RemoteS3EndpointSpec field. Replication targets declare
today's standard three attempts and keep their behaviour; the on-demand
migration source and its admin probe declare a disabled policy, so one
counted failure is exactly one source request and pull.rs owns the only
retry budget.
* fix(odm): count a stalled inline source as a source timeout
The inline tee wraps its source body in the idle guard, but the tee turns a
stalled source into an ordinary body read error, so the write-back reported
it as a local write failure. Hand commit_inline the guard so the pull is
counted under source_timeout instead.
The background pump now enforces the idle budget through the same guard
rather than a second copy of the timeout loop.
* test(odm): cover a stalled source body end to end
The fake target can now deliver a GetObject body in slices with a pause
between them, so the inline abort can be driven by a stalled source instead
of a truncated one. Two fault cases drop the workarounds they carried for
the SDK's retries: the scripted fault count and the observed source request
count now have to agree.
The operations guide records the retry and idle-timeout guarantees.
* fix(replication): send an integrity header on Object Lock replication PUTs
AWS S3, MinIO and most compatible targets reject a PutObject that carries
x-amz-object-lock-* headers unless it also carries Content-MD5 or an
x-amz-checksum-* header. Since rustfs#6895 the replication client sends
plain signed payloads with no SDK checksum, so every replicated object
with a retention period or legal hold failed against such targets.
TargetClient::put_object now decides per request through the pure
rustfs_replication::object_lock_put_integrity: a plaintext single-part
object whose source ETag is its MD5 gets Content-MD5 derived from the
ETag (no body pass, framing unchanged); a multipart-layout ETag, managed
SSE or SSE-C passthrough falls back to an SDK CRC32; a forwarded source
checksum or an unlocked PUT is left alone.
The outbound target matrix flips its two KnownFailing(rustfs#7082) cells
to Completed and every Completed cell now asserts that a locked
PutObject carried an integrity header.
Fixes rustfs#7082.
* test(e2e): keep the matrix expectation table clippy-clean under -D warnings
The CI lint runs cargo clippy --all-targets -- -D warnings. With every cell
green the single-arm match tripped match_single_binding and the unused
KnownFailing variant tripped dead_code, and the target-client tests tripped
field_reassign_with_default. Drive the expectation table from a
KNOWN_FAILING_CELLS constant (so the variant stays live and adding a red
cell is a one-line entry), build the test options as struct literals, and
refresh the e2e-repl-nightly selection digest for the renamed table test.
test(e2e): add the outbound target matrix and the replication checksum postmortem
Defense work for rustfs#7082, the regression rustfs#6895 introduced while
fixing rustfs#6853: a fix for one target class changed a client default for
every target class and nothing in tree modeled the other classes.
- docs/postmortems: timeline, root cause, why four defense layers missed
it, and the SOP for changing any outbound client default; AGENTS.md and
the adversarial compatibility lens point at it; the two env knobs from
rustfs#6895 are documented in docs/operations.
- fake_s3_target: reject_aws_chunked_uploads, require_checksum_for_object_lock
(Content-MD5 always verified), create_bucket_with_object_lock with a
GetObjectLockConfiguration handler, and a TransportSnapshot on every
journal record.
- replication_target_matrix_test: six object shapes against four target
modes with an explicit expectation table; the two rustfs#7082 cells are
pinned KnownFailing and fail with an XPASS message once the fix lands.
Wired into e2e-repl-nightly, excluded from e2e-full.
* feat(odm): enable on-demand migration by default
The module switch RUSTFS_ON_DEMAND_MIGRATION_ENABLED now defaults to true,
so the feature is reachable without an opt-in; setting it to false still
keeps the module out of the read path entirely. A bucket without an
on-demand-migration.json is never resolved by the runtime and makes no
source call, so the flip changes nothing for unconfigured buckets.
The admin plane now reads the switch through the predicate published by
module_switches.rs instead of its own duplicated env constant; the
behaviour (an environment read per call) is unchanged.
* test(e2e): wire three on-demand migration cases into e2e-smoke
The PR smoke lane gains one case per user-visible contract: a GET miss
that pulls and persists, a HEAD miss that answers from the source and
stores nothing, and the admin config/status pair that must redact the
source secret. The HEAD case did not exist outside the nightly
real-source lane, so it is added to get_basic_test.
Measured on darwin: the lane goes from 168 tests in 101.98 s to 171
tests in 101.92 s, since the three cases overlap the lane's existing
work. The darwin selection digests for e2e-smoke and e2e-full are
regenerated; the e2e-full linux digest still needs a Linux runner.
* docs(changelog): record the on-demand migration feature
* docs(odm): add the on-demand migration operations guide
Positioning against replication, site replication, tiering and the
replication read-proxy; the admin API walkthrough; the full config field
table with defaults and bounds; the semantics table with the test that
pins each row; provider presets and least-privilege source permissions;
integrity, ETag and metadata mapping; protections, error codes,
observability with PromQL, troubleshooting, known limitations and
security notes.
Adds docs/architecture/background-services-inventory.md with the
write-back pipeline, backfill job and backfill recovery loop rows, and
registers it in the architecture index.
Refs rustfs/backlog#2160, rustfs/backlog#2147.
* docs(odm): note migration sources on the replication egress guard
Refs rustfs/backlog#2160.
The guard added in #7021 fails a >5 GiB single-PutObject replication up
front instead of streaming the body to a target that must reject it. Its
message asserted a conclusion: "was not written as multipart on the
source ... re-upload it with multipart". That text is only as right as
the transport decision feeding it, and until #7047 that decision was
wrong for multipart objects carrying a full-object checksum. On 1.0.0-rc.5
a 768-part object was misrouted to the single-PUT path, and the new
default-level error line told the operator to re-upload as multipart an
object whose own ETag ended in -768.
State the evidence instead of the conclusion. The message now quotes the
ETag the decision was read from and says what was read from it (no
part-count suffix), so an operator can check the line against the
object's listing. A misroute then reads as a visible contradiction --
a suffixed ETag on a single-PUT line -- and the message says that case is
a transport-selection defect to report, not something to fix by
re-uploading. A missing or empty ETag is printed as <none> rather than
hidden.
The routing itself is already fixed by #7047; this changes only what the
guard says when it fires.
* refactor(ecstore): extract shared remote S3 client builder
Move the aws_sdk_s3 client construction out of bucket_target_sys into
bucket/remote_s3_client.rs: endpoint assembly, credential provider,
path-style selection, custom CA / skip-TLS transports and the outbound
SSRF gate now build from a neutral RemoteS3EndpointSpec so replication
targets and the upcoming on-demand migration source client share one
policy. Replication builds its client through From<&BucketTarget>; the
gate keeps its relaxed semantics (private allowed, loopback only behind
RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) verbatim. The builder also
gains optional connect/read timeouts and a User-Agent suffix
interceptor, both unset for replication.
Refs rustfs/backlog#2149
* feat(ecstore): add on-demand migration SourceClient
Add bucket/on_demand_migration/source_client.rs on top of the shared
remote S3 builder: HEAD, ranged streaming GET, ListObjectsV2 with
source-prefix mapping, GetObjectTagging and an admin probe. Every request
carries the x-rustfs-/x-minio-source-proxy-request anti-loop markers and
a RustFS-OnDemandMigration/<version> User-Agent suffix; SSE-C source
objects are rejected as unsupported. SourceError classifies SDK failures
(not found, access denied, throttled, timeout, connect, server error)
with retryability and a stable metrics label. Debug output redacts
credentials.
Refs rustfs/backlog#2149
* docs(operations): point outbound policy at shared remote S3 client builder
A 6 GiB object uploaded to the source as a 768-part multipart upload was
replicated to a generic S3 target with a single PutObject, and the target
rejected the body with EntityTooLarge. No CreateMultipartUpload was ever
issued, so the multipart replication transport never ran for the object
it exists for.
`replication_put_object_options` seeded the transport from
`object_info.is_multipart()` and then overwrote it with the second
return value of `decrypt_checksums`. Those two booleans do not mean the
same thing: the first is the object's storage shape, read from the ETag,
while the second reports whether the stored *checksum record* carries
per-part data. A full-object checksum -- what `aws s3 cp` writes by
default for a CRC algorithm -- is serialized with no MULTIPART flag even
on a multipart upload, so the record reports false and the object was
routed as a single PUT. `decrypt_checksums` documents this in
object_api/types.rs: callers that need routing must consult
`is_multipart()`. Replication did the opposite.
Route on the object's own shape, and let the checksum record only add
multipart-ness, never take it away. Objects already stored with such a
record are fixed too: the ETag was always right.
This also repairs the diagnosis of rustfs#6825, where the single-PUT
5 GiB guard fired against an object that was multipart all along and
told the operator to re-upload it as multipart.
Tests cover the three shapes the router has to separate: a multipart
object with a full-object checksum record (the regression, which fails
without this change), a multipart object with a composite record, and a
single-part object that must not be promoted onto multipart.
Replication could fail an object with nothing in the server log an
operator could act on. Every failure branch in the resyncer is quieter
than `error` on purpose — most sit on the hot path and fire once per
object per ARN — but `DEFAULT_LOG_LEVEL` is `error`, so on a stock
deployment a failed object produced no line at all. Raising those
branches to `warn` (#6840) did not close this: the default filter still
dropped them.
Report the terminal outcome instead of the branches. `replicate_object_
with_outcome` and `replicate_delete_with_outcome` now emit one `error`
per failed (object, target) once the per-target results are merged,
carrying the object key, version id, target ARN and endpoint, and the
target's own error, redacted through `sanitize_resync_error_detail` so
an echoed credential cannot reach the log. Volume is bounded by objects
that actually fail rather than by attempts inside a transfer.
Also state the single-PutObject size limit instead of discovering it at
the target. Replication picks its transport from the source object's
storage shape, not its size, so an object written with one PutObject
replicates with one PutObject however large it is — and S3 caps that at
5 GiB. Such an object could never reach a generic S3 target, and only
found out after streaming the whole body. `replication_single_put_size_
error` fails it up front with a message naming the size, the limit, and
the remedy.
Version-identity drift moves to `error` on a 10-minute per-ARN throttle.
It was `warn` deduped once per ARN per process, so the one line
explaining why a purged version is still on the target was both filtered
out by default and gone for good after it first fired.
Fixes#6825
Refs #6822
Add an authenticated scanner usage-state reset endpoint that publishes a fenced bootstrap marker for full rebuilds instead of requiring operators to delete usage metadata by hand.
Guard the reset with the scanner leader lock, storage publication epoch, and per-slot revision preconditions, and make startup resumable across stale cleanup leftovers while still rejecting newer conflicting usage state.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Lower the default scanner cache save timeout so the derived usage persistence budget stays inside the effective distributed publication lease window.
Add focused regressions for the default publication budget and bootstrap-pending observational baselines, and update operator docs with the new default.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(kms): restore persisted configuration after restart
* docs(kms): cover the reload route and startup load states
The admin contract matrix pins every dynamic KMS route for the rc and
console handoff, so the new POST /kms/reload needs a row there, and the
reload response reuses the configure snapshot shape rather than adding a
wire type. The observability runbook gains the operator procedure the
reload exists for: telling a load_failed startup apart from a server
that was never configured, and recovering without resubmitting secrets.
Every consumer now imports rustfs-heal-contracts / rustfs-scanner-contracts
directly and rg 'rustfs_common::(metrics|heal_channel|last_minute)' reports
zero hits, so the backlog#1843 re-export shims and the transitional
rustfs-common -> contracts dependency edges can go. rustfs-common no longer
recompiles on scanner/heal type changes. Doc references to the moved files
follow the new paths.
* feat(madmin): add account and two-factor wire contract
Defines the self-service account and MFA API shapes in one place so the
console and the `rc` CLI decode identical payloads instead of each
carrying its own copy of the contract.
`AccountMutability` is part of the contract on purpose: a client needs to
know whether the server will accept a password change for this identity
before offering the control, rather than discovering it from a rejected
request.
* feat(s3-types): add IAM identity audit events
Adds `iam:Identity:CredentialChanged` and `iam:Identity:AuthChallenge`
so account and authentication activity reaches the audit pipeline in its
own namespace, the way the KMS events already do. Neither is reachable
from a bucket notification config.
Two variants for the whole surface rather than one per operation:
`mask()` gives every variant its own bit in a `u64`, and the budget is
nearly spent (63 of 64 used after this). The per-operation detail lives
in `AuditEntry::api.name` and the `iamOperation` tag, which is what a
SIEM filters on anyway. Splitting these further needs `mask()` widened
first.
* feat(iam): add two-factor authentication primitives
Implements the state machine behind TOTP enrollment and verification in
the IAM domain, so the admin handlers stay HTTP plumbing and the console
and CLI drive identical logic.
* `totp`: RFC 6238 over the workspace's existing hmac/sha1, pinned to the
published Appendix B vectors. SHA-1, 6 digits, 30s: the parameters every
mainstream authenticator app implements. Verification returns the
matched time step so the caller can burn it.
* `recovery`: ten single-use codes, 100 bits each, in a Crockford base32
alphabet without I/L/O/U. Stored as domain-separated SHA-256 digests —
a password KDF would have to run once per stored code on every attempt,
turning each guess into an attacker-controlled cost, and with uniform
100-bit input there is no dictionary for it to defend against.
* `challenge`: stateless HMAC tokens. A TTL cache would be node-local, so
a cluster without session affinity would issue on one node and verify
on another; nothing here needs replicating.
* `record`: two-phase enrollment, replay high-water mark, and lockout.
Pending enrollment never gates a login, so a mis-scanned QR cannot lock
an operator out, and re-configuring keeps the old factor working until
the new one is confirmed.
* `store`: one object per identity under `config/mfa/`, a sibling of
`config/iam/` so the IAM cache loader's startup walk does not sweep it
up. Optimistic `If-Match` writes; deliberately uncached, because a cache
would need cluster-wide invalidation to keep the replay mark and the
lockout counter honest.
* `qr`: server-side rendering, so neither client needs a QR encoder.
Enrollment is refused without `RUSTFS_IAM_MASTER_KEY`. A TOTP secret is
credential-equivalent, and one written in plaintext could be lifted off a
disk — worse than no second factor, because the user believes they have
one. IAM identities tolerate a missing master key for backward
compatibility; a new feature has no such history to honour.
Also adds `IamSys::revoke_sts_sessions_for_parent`, so a credential
rotation can invalidate the sessions minted under the old secret.
* feat(admin): add self-service account endpoints and the two-factor login gate
Adds the account surface (`/v3/account/*`), the second-factor endpoints,
the administrative reset (`/v3/user/mfa`), and `PUT
/v3/set-user-secret-key`, plus the gate on `AssumeRole`.
What the gate covers, and what it deliberately does not:
* `AssumeRole` is the only interactive login RustFS has, so it is where a
second factor can be enforced. With one enrolled it requires
`TokenCode`; without an enrollment the code path is unchanged, so
existing deployments are untouched.
* A request signed directly with a long-term access key stays ungated.
Gating it would break every script and CLI the moment a human enabled
2FA on their own account, and would add no protection: whoever holds
the secret key already has full access without presenting a code. This
is the division AWS draws; making 2FA meaningful for API access needs an
`aws:MultiFactorAuthPresent` policy condition, tracked separately.
`SerialNumber`/`TokenCode` are STS's own parameters, so an SDK or script
authenticates the same way the console does.
`caller_identity` resolves who a request acts as. The console signs with
a short-lived STS session, so "the caller" is almost never the key that
signed. It reports two separate capabilities: root cannot rotate its
secret (a process-wide `OnceLock` that also derives the internode RPC
secret) but *can* enroll a second factor — conflating the two would leave
the default deployment's console login unprotectable.
The self-service routes carry no admin action. Giving them one would be
wrong in both directions: it would stop an ordinary user from changing
their own password, and let any holder of that action change someone
else's. They gate on possession of the credential plus, for the
mutations, knowledge of the current secret — a signature only proves a
credential was used, so without that a hijacked tab could rewrite the
account's credentials or strip its second factor.
`set-user-secret-key` exists because the only prior way to change a
password was to re-POST the whole user through `add-user`, which rewrote
`status` and dropped the policy field — a password reset that silently
re-enabled a disabled account.
Wrong, replayed and malformed codes are indistinguishable on the wire;
the distinction survives only in the audit trail, where no submitted
value, secret or code is ever recorded.
* test(e2e): cover the two-factor lifecycle and its regressions
Unit tests cover the state machine at its edges; only an end-to-end test
proves the pieces are wired together and that the existing
authentication paths still behave.
Asserts, against a real server: enrollment is refused without a master
key; the full enroll/activate flow works with a genuine RFC 6238 code;
`AssumeRole` refuses without a factor and accepts a valid one; a recovery
code works exactly once; a direct SigV4 admin request keeps working with
a factor enrolled; `AssumeRole` for an unenrolled identity is unchanged;
and a password rotation invalidates the old secret.
The test computes TOTP codes itself rather than calling the server's
implementation — a shared helper could agree with a bug on both sides.
This suite caught a real defect during development: enrollment was
refused for root because its *password* is immutable, which would have
left the default deployment — an administrator signing into the console
as root — unable to protect the one login the feature exists for.
* docs(operations): document the two-factor authentication model
Records what the second factor protects and what it deliberately does
not, because several of the boundaries look like gaps until the
alternative is spelled out: why direct SigV4 access stays ungated, why
root credentials cannot be rotated at runtime, why secret keys cannot be
hashed in an S3 server, and why at-rest protection is mandatory for a
TOTP secret but optional for an IAM identity.
Also states the limitations plainly, including that GHSA-m77q-r63m-pj89
is unaffected: a holder of the root secret can still forge a session
token, 2FA claim included.
Placed alongside the other authentication and KMS security documents
rather than under a new `docs/security/`, which `.gitignore` excludes.
* fix(admin): route the new account handlers through the admin s3 facade
Two of the guardrails in the CI "Quick Checks" job rejected the previous
commits, so the required check would have gone red as soon as a maintainer
approved the workflow run.
`check_architecture_migration_rules.sh` requires everything under
`rustfs/src/admin` to reach `ECStore` through a domain module rather than
the root of `storage_api`. The MFA handler and the two `AssumeRole`
signatures now use `storage_api::runtime::ECStore`, which is where the
other ten admin handlers already take it from.
`check_s3s_footprint.sh` ratchets two counters that new code may not grow:
files referencing `s3s` and error-macro invocation lines. This branch added
four files and thirty-two lines to them. The ratchet is lower-only and its
header forbids raising a baseline to get green, so the construction moves
behind the facade instead: `storage_api::s3` now re-exports the request and
body types these handlers need and gains an `error` constructor over
`S3Error::with_message`. That is the same constructor the macro expands to
and the one `handlers/mod.rs`, `rebalance_internal_error` and
`invalid_object_lock_configuration` already call, so this is the existing
practice rather than a new one, and it keeps the `s3s` dependency in the
boundary file the s3gate migration replaces.
Every error code and message is carried over unchanged. In `sts.rs` only
the call site this branch added is converted; the sixteen that predate it
are left alone, because rewriting them would put unrelated churn in a
feature PR and push the counter below the baseline it is meant to hold.
* fix(ecstore): persist unresolved decommission entries
* fix(ecstore): type decommission completion result
* fix(ecstore): allow intentional decommission listing signatures under strict clippy
The sftp/swift feature-matrix clippy gates run with -D warnings and
flag the unresolved-entry resolver (large Err payload by design, 8
context parameters) and the decommission listing driver (9 args).
Document why and align with the existing decommission_entry precedent.
The merge of rustfs#6261 lost the last 64 lines of the English
translation: merging main (to pick up rustfs#6258) resolved the
conflict on the renamed file by cutting it mid-table in section 6,
which dropped section 7 (backlog/history index), section 8 (audit
method and limitations) and section 9 (landing results) that the
Chinese counterpart still carries. Restore them verbatim from the
translation commit (0e051602f) so both language versions are complete
568-line mirrors of the full 0-9 baseline, as the PR body promised.
Co-authored-by: heihutu <heihutu@gmail.com>
* docs(operations): land the heal/scanner MinIO audit baseline with closure results
Move the comprehensive heal/scanner vs MinIO analysis (2026-08-16) into
docs/operations/ so it finally enters the tree — the docs/ root is
ignored by the gitignore whitelist, which is why the baseline the audit
issue referenced as "to be merged with a PR" never landed. Append §9
closure results: all 14 backlog sub-issues (#1865-#1878) closed with the
per-item PR map, two further misjudgment corrections (HS-17 was already
implemented; HS-14's MinIO idle semantics drifted upstream), HS-12/HS-18
audit conclusions, and the registered follow-ups.
Backlog issue: rustfs/backlog#1862
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(operations): add an English counterpart of the audit baseline
Rename the Chinese analysis to *_zh.md (matching the repo's bilingual
convention of scanner-excess-alerts.md / _zh.md) and add a full English
translation at the original path, cross-linked at the top of both files.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
docs(operations): land the heal/scanner MinIO audit baseline with closure results
Move the comprehensive heal/scanner vs MinIO analysis (2026-08-16) into
docs/operations/ so it finally enters the tree — the docs/ root is
ignored by the gitignore whitelist, which is why the baseline the audit
issue referenced as "to be merged with a PR" never landed. Append §9
closure results: all 14 backlog sub-issues (#1865-#1878) closed with the
per-item PR map, two further misjudgment corrections (HS-17 was already
implemented; HS-14's MinIO idle semantics drifted upstream), HS-12/HS-18
audit conclusions, and the registered follow-ups.
Backlog issue: rustfs/backlog#1862
Co-authored-by: heihutu <heihutu@gmail.com>
* refactor(scanner): drop the always-None single-disk default cycle hook
single_disk_default_cycle_secs returned None for every maintenance
feature combination, so the single-disk startup path already resolved
its default cycle from the speed preset (60s at 'default'). Remove the
never-wired hook and its pin tests, keep the explicit reset, and record
the decision: no special single-disk cycle override without measured
cold-start ILM latency evidence; clean-idle backoff already stretches
idle cadence (backlog#1878 HS-16).
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(operations): add heal/scanner MinIO parity decision notes
Document the HS-14/16/18 decision batch from backlog#1878: the scanner
idle throttling semantics matrix (RUSTFS_SCANNER_IDLE_MODE x speed
preset x foreground read backoff) side by side with MinIO's current
static idle_speed switch as verified against upstream master, the
migration warnings for env names and value vocabularies, the bitrot
cycle default divergence (30d vs off), the stale-multipart / tmp / trash
three-stage cleanup comparison with the crash-residue window grading,
and the single-disk default cycle decision.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
test(kms): move the Vault KV2 Transit-wrapping doc guard into check_fips_wording.sh
`test_vault_kv2_sources_do_not_claim_transit_wrapping` asserted that four
`include_str!`-pinned files never describe the Vault KV2 backend as wrapping key
material through Vault's Transit engine. The invariant is a documentation-claim
invariant with no behavioral twin by construction, and the test form was weak in
both directions: it saw only four files (the same prose in a fifth file passed
silently) and it stopped compiling — rather than reporting a violation — as soon
as one of them was renamed.
Move the four literals verbatim into `scripts/check_fips_wording.sh`, which
already guards the adjacent cryptographic over-claim class (unsupported FIPS
validation wording) and is anchored to the same policy document. The guard now
greps every file under `crates/kms` for the same four case-sensitive literals and
separately reports a moved pinned source instead of failing to build.
`check_fips_wording.sh` previously ran only in `make pre-commit` / `pre-pr`, so
wire it into the Quick Checks job of both CI workflows to keep the invariant's
failure visibility at least as strong as the deleted test's.
* test(heal): add concurrency invariants for heal vs delete/overwrite races (HS-12)
Audit conclusion for backlog#1874: RustFS does not need a persistent
object-level healing marker (MinIO x-minio-healing) because every path
that can touch the same (bucket, object) commit surface serializes on
the same namespace write lock, and the heal lock guard spans the whole
rename commit including the HEAL_RENAME_INCOMPLETE partial path.
Lock the conclusion in with two race regression tests:
- heal_racing_version_delete_never_resurrects_the_deleted_version:
shard damage is injected on the doomed version so a Deep heal has real
reconstruction work while a versioned DELETE runs concurrently; the
deleted version must stay deleted and the survivor intact.
- heal_racing_unversioned_overwrites_preserves_the_last_commit:
unversioned overwrites (activating the post-commit tail that deletes
the replaced data dir without the ns lock) race a Deep heal in a
loop; the final current version must be exactly the last commit.
Also adds docs/operations/heal-concurrency-safety-notes-zh.md with the
full intersection matrix (17 intersections), lock-coverage argument,
and the residual-window classification (commit tail races are
fail-into-retry safe; bare prefix delete has zero production callers;
admin no_lock is an explicit operator opt-in).
Co-Authored-By: heihutu <heihutu@gmail.com>
* test: remove redundant heal etag clone
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(scanner): emit excess alerts as S3 notification events
The excess-versions / excess-version-size / excess-folders alerts were
metrics-and-logs only; consoles and external auditors had no way to hear
them (rustfs/backlog#1868, HS-04). MinIO emits s3:ObjectManyVersions /
s3:ObjectLargeVersions / s3:PrefixManyFolders for the same conditions —
RustFS carries those as EventName::Scanner* with s3:Scanner:* wire names
that already existed unpublished.
The three alert sites now also dispatch through the standard event
pipeline (send_event via the storage_api owner facade), carrying the
actual values and thresholds in req_params and UserAgent "Scanner".
Without a cooldown a single over-threshold object would re-emit on every
~60s scan cycle, so emissions are edge-held per (kind, bucket, object)
for 24h (RUSTFS_SCANNER_ALERT_COOLDOWN_SECS, 0 = every cycle), backed by
a process-global map with a 4096-key hard cap that clears rather than
grows. Metrics and structured logs stay level-triggered every cycle;
only the notification events are held back. A restart resets the
cooldown deliberately: one re-emission per still-hot key buys back
visibility after the restarts that accompany incident response.
Tests pin the edge-hold semantics (first fires, immediate re-check held,
independent keys, cooldown expiry re-fires, zero cooldown always emits,
hard bound) in one sequential test for the process-global map, and pin
the emitted wire names against EventName's canonical string forms so a
subscribed bucket notification can never silently stop matching.
docs/operations/scanner-excess-alerts.md documents the three events,
the metric-vs-event cadence difference, and the HS-15 threshold deltas
(alert_excess_folders 65538 vs MinIO 50000 is deliberate: Proxmox
Backup Server chunk layout compatibility).
Closesrustfs/backlog#1868.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(operations): split scanner excess alerts into English and Chinese pages
The page shipped Chinese-only; keep it as scanner-excess-alerts_zh.md and
add a faithful English translation at the original path, cross-linked at
the top of both.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Stream materialized GET bodies by moving the buffered Bytes once instead of wrapping the stream in an extra bytes_stream layer.
Add an operations runbook for object I/O tuning A/B sweeps.
Co-authored-by: heihutu <heihutu@gmail.com>
cxymds