* test(kms): cover directory outages and missing keys on main
(cherry picked from commit 8fc1f0c41d)
* fix(kms): preserve directory availability errors on main
(cherry picked from commit c2a8e476f7)
* fix(ecstore): attest fresh multi-pool bootstrap per pool creator
A fresh deployment whose pools have their first endpoint on different nodes could never publish its initial pool.bin: each node held fresh-bootstrap proof only for the pool it formatted, combine_across_pools collapsed the deployment-wide proof to None, the elected writer never wrote a pending identity, and startup died with "no durable bootstrap identity or pool.bin replica is available" once the init retry budget ran out.
Track first-hand bootstrap authority per pool. The first pool's creator mints the pending cluster identity on the pool it created, every other creator copies that nonce-bound identity onto the pool it formatted first-hand (a scoped identity write that only ever touches pools the process holds first-hand proof for), and the elected writer publishes pool.bin once it holds first-hand proof for pool 0 and every pool replica carries the same pending identity. Fresh + None is still never promoted, corrupt or disagreeing replicas still fail closed, an initialized deployment never reopens bootstrap for an expansion pool, and an elected restart without first-hand proof still cannot reuse a complete pending set.
Startup classification no longer latches the pool-metadata write gate for the two transient outcomes a healthy bootstrap passes through (a non-elected node waiting for the elected writer, the elected writer waiting for the other creators); recover_pool_meta_transaction never clears write_blocked, so a non-elected node that started before pool.bin existed stayed write-blocked for the life of the process. Genuine recovery-required states still latch.
Refs rustfs/backlog#2375, rustfs/backlog#2338
* fix(lock): contain remote lock RPC timeout storms
A lock RPC deadline evicted the shared internode HTTP/2 channel and re-dialed it unconditionally, so one slow lock endpoint produced a cluster-wide RST_STREAM / GOAWAY too_many_resets / reconnect loop (rustfs#7363).
The remote lock client now keeps a per-peer channel history: a timeout evicts only when the peer has completed no lock RPC for two deadlines, evictions and transport-failure re-dials are rate limited per peer (RUSTFS_OBJECT_LOCK_RPC_EVICTION_COOLDOWN_MS, default 5 s), and a timed-out request is detached instead of cancelled, bounded per peer by RUSTFS_OBJECT_LOCK_RPC_DETACHED_LIMIT (default 256) and by the internode RPC timeout; a lock granted after its caller gave up is released immediately. Unlocks that fail the quick retries continue on a deferred 1/2/4/8/16 s schedule before the server lease reclaims them. New rustfs_remote_lock_* metrics cover timeouts, evictions, suppressed evictions, detached streams, late completions and late releases per peer; docs/operations/lock-rpc-storm-protection.md documents the policy.
Refs rustfs/backlog#2375, rustfs#7363
* ci: refresh nightly test selection digests
The replication nightly membership guard expected the 68-test digest from #7422 while the current listing has 71 tests (additions only: test_bucket_replication_sse_c_compressed_passthrough from #7366, matrix_mint_own_version_ids_addresses_mutations_through_the_ledger and matrix_removed_replication_config_abandons_pending_purge from #7368), and the cluster fault lane expected 50 tests while #7374 added test_cluster_root_heal_recovers_remote_shards_after_background_target_crash. Both lanes have failed before running a single test since 2026-09-07. Bind the Linux digests to the listings from scheduled run 34187469350 and the Darwin e2e-nightly digest to the matching local listing.
Refs rustfs/backlog#2375
* test(scanner): measure heal pacing and cache cost
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(error): merge equivalent api message branches
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): cleanup consumed MRF replay journals
Do not retain Accepted or Merged replay intents as startup anchors after they have been handed to the heal manager. Only refused or still-pending replay records keep the journal on disk until a successor snapshot can persist them.
This keeps successor snapshots limited to the pending queue, which lets successful replay remove both authoritative and legacy journal paths and restores the crash-boundary tests around successor flush.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit d5b8f49c9d)
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
* test(scanner): report heal release gate status
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(error): merge equivalent api message branches
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): cleanup consumed MRF replay journals
Do not retain Accepted or Merged replay intents as startup anchors after they have been handed to the heal manager. Only refused or still-pending replay records keep the journal on disk until a successor snapshot can persist them.
This keeps successor snapshots limited to the pending queue, which lets successful replay remove both authoritative and legacy journal paths and restores the crash-boundary tests around successor flush.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit d5b8f49c9d)
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
* fix(scanner): retain raw enumeration quantum
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(error): merge equivalent api message branches
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): cleanup consumed MRF replay journals
Do not retain Accepted or Merged replay intents as startup anchors after they have been handed to the heal manager. Only refused or still-pending replay records keep the journal on disk until a successor snapshot can persist them.
This keeps successor snapshots limited to the pending queue, which lets successful replay remove both authoritative and legacy journal paths and restores the crash-boundary tests around successor flush.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit d5b8f49c9d)
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
* test(heal): cover admin lock timeout progress
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(error): merge equivalent api message branches
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): cleanup consumed MRF replay journals
Do not retain Accepted or Merged replay intents as startup anchors after they have been handed to the heal manager. Only refused or still-pending replay records keep the journal on disk until a successor snapshot can persist them.
This keeps successor snapshots limited to the pending queue, which lets successful replay remove both authoritative and legacy journal paths and restores the crash-boundary tests around successor flush.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit d5b8f49c9d)
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
* test: summarize scanner heal perf artifacts
Add a quiet Scanner/Heal performance artifact summarizer that normalizes ABBA report verdicts, key regression metrics, cache-cost profile records, and provenance hashes for CI or PR handoff.
Document the summary command in the scanner benchmark runbook and cover measured, synthetic, pending, and invalid cache-cost paths with focused Python tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(error): merge equivalent api message branches
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): cleanup consumed MRF replay journals
Do not retain Accepted or Merged replay intents as startup anchors after they have been handed to the heal manager. Only refused or still-pending replay records keep the journal on disk until a successor snapshot can persist them.
This keeps successor snapshots limited to the pending queue, which lets successful replay remove both authoritative and legacy journal paths and restores the crash-boundary tests around successor flush.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit d5b8f49c9d)
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
* feat(heal): publish verified MRF repair events
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(common): add durable MRF proof matching (#7429)
Add a fail-closed MRF repair proof adapter that only consumes anchors when the durable anchor and verified proof share the same full identity, ingress lease, and bucket incarnation.
Legacy replay intents without leases cannot become dischargeable anchors, so the current retained journal behavior remains unchanged until a durable writer and producer proof source are connected.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): rearm MRF replay leases before admission (#7435)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(error): merge equivalent api message branches
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): cleanup consumed MRF replay journals
Do not retain Accepted or Merged replay intents as startup anchors after they have been handed to the heal manager. Only refused or still-pending replay records keep the journal on disk until a successor snapshot can persist them.
This keeps successor snapshots limited to the pending queue, which lets successful replay remove both authoritative and legacy journal paths and restores the crash-boundary tests around successor flush.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit d5b8f49c9d)
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix(kms): report a missing KMS key as 400 KMS.NotFoundException
A PutObject whose resolved SSE-KMS key (request header or bucket default
rule) does not exist in the KMS answered 500 InternalError with a generic
message: KmsError::KeyNotFound fell through to the default arm of the
StorageError-to-ApiError mapping. S3 reports this client mistake as 400
KMS.NotFoundException; the mapping now does the same and names the key.
s3s has no status for a custom code, so the ApiError-to-S3Error conversion
supplies it.
The legacy create-key aliases behind /minio/admin/v3/kms/key/create ignored
the key-id query parameter that mc sends, creating a key under a generated
id instead of the requested name. The alias now honors key-id (and its
keyId/key spellings) alongside the name tag, and refuses a request whose
two sources disagree.
Refs: rustfs/backlog#2330 (KMS-312, KMS-110)
* fix(error): merge equivalent api message branches
Combine the MaxVersionsExceeded and internal IO message branches so Clippy no longer flags identical if blocks while preserving the existing response messages.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): cleanup consumed MRF replay journals
Do not retain Accepted or Merged replay intents as startup anchors after they have been handed to the heal manager. Only refused or still-pending replay records keep the journal on disk until a successor snapshot can persist them.
This keeps successor snapshots limited to the pending queue, which lets successful replay remove both authoritative and legacy journal paths and restores the crash-boundary tests around successor flush.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit d5b8f49c9d)
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
* feat(heal): expose admission observability
Track heal admission outcomes and bounded lock-phase latency through the existing operations snapshot so distributed E2E gates can assert duplicate, forceStart, and displacement behavior without relying on logs.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(error): merge equivalent api message branches
Combine the MaxVersionsExceeded and internal IO message branches so Clippy no longer flags identical if blocks while preserving the existing response messages.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
* feat(observability): expose pool write-block diagnostics
* fix(error): merge equivalent api message branches
Combine the MaxVersionsExceeded and internal IO message branches so Clippy no longer flags identical if blocks while preserving the existing response messages.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): cleanup consumed MRF replay journals
Do not retain Accepted or Merged replay intents as startup anchors after they have been handed to the heal manager. Only refused or still-pending replay records keep the journal on disk until a successor snapshot can persist them.
This keeps successor snapshots limited to the pending queue, which lets successful replay remove both authoritative and legacy journal paths and restores the crash-boundary tests around successor flush.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit d5b8f49c9d)
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
* fix(s3): reject unsigned x-amz headers on presigned requests
A SigV4 presigned URL only binds the headers listed in X-Amz-SignedHeaders, but the handlers applied every x-amz-* request header regardless. The holder of a presigned PutObject URL signed with SignedHeaders=host could add x-amz-tagging, x-amz-storage-class, x-amz-website-redirect-location, ACL, metadata, Object Lock or SSE headers and have them applied (GHSA-g8w9-qw9q-fghr). Reject such requests at the S3 access boundary with 403 AccessDenied and the AWS message "There were headers present in the request which were not signed"; x-amz-cf-id stays tolerated for CloudFront. SigV2 and header-signed SigV4 requests are unchanged.
Regression tests are named after the advisory (unit tests in rustfs/src/auth.rs, e2e in crates/e2e_test/src/presigned_negative_test.rs with a signed-tagging positive control); the security smoke floor rises to 20 and the e2e selection digests are refreshed for the two new cases.
* fix(s3): apply presigned signed-header rule to custom routes and harden parsing
Move the GHSA-g8w9-qw9q-fghr check to the first statement of S3Access::check, apply it in S3Router::check_access so admin, console, STS and extension routes that never reach the access hook enforce the same rule, read X-Amz-SignedHeaders with the exact key the verifier uses and treat a duplicate as signing nothing, and log the rejection as a warn event with the repository field shape. Add presigned GET, unsigned x-amz-copy-source and unsigned Content-Type e2e cases plus a router unit test; raise the security smoke floor to 26 and refresh the selection digests.
* docs(testing): list the full GHSA-g8w9 regression set
* fix(error): merge equivalent api message branches
Combine the MaxVersionsExceeded and internal IO message branches so Clippy no longer flags identical if blocks while preserving the existing response messages.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Keep replayed MRF records crash-replayable after manager admission until a later durable successor proof can tombstone them. Queue-full and transient replay submission failures now also preserve the old journal anchor instead of allowing cleanup to erase the only recovery source.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
A multipart UploadPart queued for a foreground write permit is not read while it waits, so the client's socket write stalls for the whole wait and the client's own write timeout decides the outcome; botocore reports that as ConnectionClosedError. Lower the default queue wait from 30 s to 10 s so the part receives SlowDown before mainstream SDK timeouts, and stop forcing a 4 MiB SO_RCVBUF on the API listener (new RUSTFS_HTTP_SOCKET_RECV_BUFFER_BYTES restores a fixed size) so a queued connection no longer lets up to 8 MiB of unread body accumulate in kernel memory.
Fixes#7385.
Co-authored-by: houseme <housemecn@gmail.com>
Keep the production scoped scanner test from mutating an unselected bucket during the dirty-only cycle. That cycle intentionally reuses the clean baseline, while the later Deep and full maintenance cycles still mutate cold storage and prove a full walk refreshes it.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Pin the star history workflow to the freshly released v1.4.0 of
overtrue/repo-visuals-action, which adds the chart-layout input and a
refreshed editorial chart/contributor-wall style. The workflow keeps its
existing inputs, so chart-layout stays at the editorial default.
Add an MRF pipeline regression that replays a single authoritative journal containing same-object records that differ by kind and erasure-set scope while a stale legacy mirror is present. The test proves restart replay admits each authoritative responsibility independently and never merges the stale mirror epoch.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Introduce an object heal receipt wrapper so storage owners can report a positive repaired or verified result without changing legacy heal consumers.
Task-level object heal now records only receipts that match the requested object identity, version, pool, set, and carry a bucket incarnation. Legacy and mismatched receipts still fall back to unknown outcome accounting.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Require scanner/heal ABBA adapters to emit foreground pressure, heal lock wait p99, and heal attempt counters before a measured W10/W11 run can be accepted.
Report per-leg pressure ratios, lock p99 samples, and attempt cost per healed object so synthetic harness runs remain evidence-contract validation rather than performance proof.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Extend the segment observation fixture with durable activation prerequisites and add scanner oracles for cold segment zero-walk and distributed invalidation fallback.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
overtrue