gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-09-09 05:36:24 +00:00
Code Issues Packages Projects Releases Wiki Activity
6,665 Commits 63 Branches 132 Tags
76e025a50cec2bf3bb026df84cc276d981552c2a
Commit Graph

820 Commits

Author SHA1 Message Date
houseme 76e025a50c test(scanner): add W13 MRF evidence runner (#7547) 
Add a W13 durable MRF evidence runner that emits measured G07, G08, and P4 JSON artifacts and release descriptors through the existing scanner/heal bundle gate.

The runner now executes the ignored MRF replay evidence test with an exact full test path, validates raw artifact kinds and gate decisions, prepares Linux tmpfs-backed ENOSPC roots for G08, and documents the Linux/long-soak boundaries.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
 2026-09-09 10:08:59 +08:00
houseme b40eb193df test(scanner): roll out distributed e2e selection (#7549) 
Refresh the committed e2e-distributed selection digest after the EC8+4 Scanner/Heal evidence case joined the profile, and document the registry-aware runner entry for the distributed case.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
 2026-09-09 10:08:49 +08:00
Zhengchao An e24eae9eaa fix: address release acceptance regressions (#7541) 2026-09-09 08:33:59 +08:00
houseme 8617f2701b test(scanner): add W16 release evidence runner (#7543) 
Add a W16 Scanner/Heal evidence runner that executes the recovery-intent crash-boundary and quota-authority lanes, emits measured G04/G12 JSON artifacts, and validates the resulting single-gate release descriptors.

Wire its shell self-test into script-tests and document the release evidence entry point.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
 2026-09-09 08:26:13 +08:00
houseme b5c9229e77 chore: integrate main CI gates into release 
Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
 2026-09-09 08:02:35 +08:00
Zhengchao An a722fa80d5 fix(ci): require every selected validation lane before merge (#7529) 2026-09-09 05:16:48 +08:00
houseme 3e8b0e7c83 test(e2e): add G09 upgrade evidence runner (#7525) 
* fix(replication): close the GA blocker set from backlog#2366 (#7503)

* fix(replication): close GA blockers from backlog#2366

Implements the P1 set from the pre-GA replication audit:

- Replication rule tag filters now require every And.Tag to match, replacing
  the s3s OR semantics with a local AND matcher that fails closed on a
  malformed tag.
- A replicated group membership change no longer writes the group status, so
  a membership update carrying the default Enabled status cannot silently
  re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
  peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
  carries its own ilm-expiry override, and no longer blocks a remove that
  drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
  node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
  follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
  support boundary section, and the Replication changelog section.

* fix(site-replication): resume only a locally driven endpoint refresh

The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.

Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.

* fix(site-replication): keep an explicit disabled group status on a snapshot

Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.

The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.

* fix(site-replication): schedule the import snapshot without recording a failure

`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.

Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.

* fix(admin): stop reporting replication failures as retries

`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.

* perf(site-replication): aggregate failure windows without cloning bucket stats

`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.

It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.

* fix(site-replication): reject a conflicting ilm-expiry override on a re-run

The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.

* fix(admin): do not fail a remote-target write on a peer reload error

set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.

* fix(site-replication): undo every bucket a cut-short refresh rewrote

When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.

* fix(site-replication): keep replay running while an endpoint refresh is pending

A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.

* test(e2e): cover the AND semantics of a two-tag replication filter

The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.

No new test function, so the nightly selection digest is unchanged.

* refactor(site-replication): fold the refresh state-change error into one constructor

The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.

* fix(site-replication): report a peer whose IAM snapshot waits for a repair

An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.

* docs(operations): state the group-status and escalation convergence limits

Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.

* fix(ci): bind performance runs to selected inputs (#7512)

* test(e2e): add G09 upgrade evidence runner

Add a Linux x86_64 runner that downloads the pinned previous release, builds the current RustFS binary, runs the mixed-version and rollback upgrade compatibility lanes, and verifies the required Scanner/Heal G09 raw evidence artifacts.

Document the runner and add a shell self-test for help, dry-run, SHA validation, and non-empty artifact directory guards.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
 2026-09-09 00:05:36 +08:00
houseme 5db0e14d04 test(scanner): preflight G09 evidence disk space (#7527) 
* fix(replication): close the GA blocker set from backlog#2366 (#7503)

* fix(replication): close GA blockers from backlog#2366

Implements the P1 set from the pre-GA replication audit:

- Replication rule tag filters now require every And.Tag to match, replacing
  the s3s OR semantics with a local AND matcher that fails closed on a
  malformed tag.
- A replicated group membership change no longer writes the group status, so
  a membership update carrying the default Enabled status cannot silently
  re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
  peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
  carries its own ilm-expiry override, and no longer blocks a remove that
  drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
  node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
  follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
  support boundary section, and the Replication changelog section.

* fix(site-replication): resume only a locally driven endpoint refresh

The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.

Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.

* fix(site-replication): keep an explicit disabled group status on a snapshot

Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.

The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.

* fix(site-replication): schedule the import snapshot without recording a failure

`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.

Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.

* fix(admin): stop reporting replication failures as retries

`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.

* perf(site-replication): aggregate failure windows without cloning bucket stats

`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.

It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.

* fix(site-replication): reject a conflicting ilm-expiry override on a re-run

The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.

* fix(admin): do not fail a remote-target write on a peer reload error

set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.

* fix(site-replication): undo every bucket a cut-short refresh rewrote

When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.

* fix(site-replication): keep replay running while an endpoint refresh is pending

A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.

* test(e2e): cover the AND semantics of a two-tag replication filter

The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.

No new test function, so the nightly selection digest is unchanged.

* refactor(site-replication): fold the refresh state-change error into one constructor

The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.

* fix(site-replication): report a peer whose IAM snapshot waits for a repair

An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.

* docs(operations): state the group-status and escalation convergence limits

Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.

* fix(ci): bind performance runs to selected inputs (#7512)

* fix(targets): reject trailing batch items (#7508)

* test(scanner): preflight G09 evidence disk space

Fail the Scanner/Heal G09 upgrade evidence runner before downloading or building when the validation host does not have enough free space for a full raw evidence pass.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: cui fliter <imcusg@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
 2026-09-08 23:58:08 +08:00
houseme 857584b3c2 test(scanner): add G09 upgrade evidence runner (#7524) 
* fix(replication): close the GA blocker set from backlog#2366 (#7503)

* fix(replication): close GA blockers from backlog#2366

Implements the P1 set from the pre-GA replication audit:

- Replication rule tag filters now require every And.Tag to match, replacing
  the s3s OR semantics with a local AND matcher that fails closed on a
  malformed tag.
- A replicated group membership change no longer writes the group status, so
  a membership update carrying the default Enabled status cannot silently
  re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
  peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
  carries its own ilm-expiry override, and no longer blocks a remove that
  drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
  node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
  follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
  support boundary section, and the Replication changelog section.

* fix(site-replication): resume only a locally driven endpoint refresh

The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.

Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.

* fix(site-replication): keep an explicit disabled group status on a snapshot

Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.

The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.

* fix(site-replication): schedule the import snapshot without recording a failure

`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.

Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.

* fix(admin): stop reporting replication failures as retries

`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.

* perf(site-replication): aggregate failure windows without cloning bucket stats

`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.

It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.

* fix(site-replication): reject a conflicting ilm-expiry override on a re-run

The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.

* fix(admin): do not fail a remote-target write on a peer reload error

set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.

* fix(site-replication): undo every bucket a cut-short refresh rewrote

When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.

* fix(site-replication): keep replay running while an endpoint refresh is pending

A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.

* test(e2e): cover the AND semantics of a two-tag replication filter

The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.

No new test function, so the nightly selection digest is unchanged.

* refactor(site-replication): fold the refresh state-change error into one constructor

The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.

* fix(site-replication): report a peer whose IAM snapshot waits for a repair

An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.

* docs(operations): state the group-status and escalation convergence limits

Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.

* fix(ci): bind performance runs to selected inputs (#7512)

* test(scanner): add G09 upgrade evidence runner

Add a reusable Scanner/Heal G09 runner for Linux PR-head validation. The script downloads the pinned previous release, builds the current checkout, runs the mixed-version and rollback upgrade E2E lanes, and validates the measured raw artifacts before they can be consumed by the release bundle gate.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
 2026-09-08 23:50:02 +08:00
houseme 8e987ce0a6 test(scanner): add G09 upgrade evidence runner (#7522) 
* fix(replication): close the GA blocker set from backlog#2366 (#7503)

* fix(replication): close GA blockers from backlog#2366

Implements the P1 set from the pre-GA replication audit:

- Replication rule tag filters now require every And.Tag to match, replacing
  the s3s OR semantics with a local AND matcher that fails closed on a
  malformed tag.
- A replicated group membership change no longer writes the group status, so
  a membership update carrying the default Enabled status cannot silently
  re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
  peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
  carries its own ilm-expiry override, and no longer blocks a remove that
  drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
  node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
  follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
  support boundary section, and the Replication changelog section.

* fix(site-replication): resume only a locally driven endpoint refresh

The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.

Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.

* fix(site-replication): keep an explicit disabled group status on a snapshot

Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.

The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.

* fix(site-replication): schedule the import snapshot without recording a failure

`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.

Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.

* fix(admin): stop reporting replication failures as retries

`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.

* perf(site-replication): aggregate failure windows without cloning bucket stats

`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.

It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.

* fix(site-replication): reject a conflicting ilm-expiry override on a re-run

The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.

* fix(admin): do not fail a remote-target write on a peer reload error

set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.

* fix(site-replication): undo every bucket a cut-short refresh rewrote

When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.

* fix(site-replication): keep replay running while an endpoint refresh is pending

A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.

* test(e2e): cover the AND semantics of a two-tag replication filter

The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.

No new test function, so the nightly selection digest is unchanged.

* refactor(site-replication): fold the refresh state-change error into one constructor

The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.

* fix(site-replication): report a peer whose IAM snapshot waits for a repair

An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.

* docs(operations): state the group-status and escalation convergence limits

Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.

* fix(ci): bind performance runs to selected inputs (#7512)

* test(scanner): add G09 upgrade evidence runner

Add a reusable Linux x86_64 runner for the Scanner/Heal G09 mixed-version and rollback upgrade evidence lanes.

The helper reads the pinned previous-release asset metadata from the upgrade workflow, verifies the downloaded binary, builds the current head, runs both ignored E2E tests, and fails unless the expected G09 JSON artifacts exist.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
 2026-09-08 23:38:54 +08:00
houseme c7dec044eb test(e2e): emit scanner heal G09 upgrade evidence (#7519) 
Record measured Scanner/Heal G09 evidence artifacts from the upgrade compatibility lanes when a fresh evidence directory is provided.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
 2026-09-08 23:18:36 +08:00
houseme 14a59f7770 test(scanner): require bounded retry window evidence (#7517) 
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
 2026-09-08 23:10:02 +08:00
唐小鸭 46907c05cf fix(replication): close the GA blocker set from backlog#2366 (#7503) 
* fix(replication): close GA blockers from backlog#2366

Implements the P1 set from the pre-GA replication audit:

- Replication rule tag filters now require every And.Tag to match, replacing
  the s3s OR semantics with a local AND matcher that fails closed on a
  malformed tag.
- A replicated group membership change no longer writes the group status, so
  a membership update carrying the default Enabled status cannot silently
  re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
  peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
  carries its own ilm-expiry override, and no longer blocks a remove that
  drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
  node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
  follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
  support boundary section, and the Replication changelog section.

* fix(site-replication): resume only a locally driven endpoint refresh

The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.

Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.

* fix(site-replication): keep an explicit disabled group status on a snapshot

Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.

The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.

* fix(site-replication): schedule the import snapshot without recording a failure

`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.

Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.

* fix(admin): stop reporting replication failures as retries

`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.

* perf(site-replication): aggregate failure windows without cloning bucket stats

`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.

It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.

* fix(site-replication): reject a conflicting ilm-expiry override on a re-run

The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.

* fix(admin): do not fail a remote-target write on a peer reload error

set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.

* fix(site-replication): undo every bucket a cut-short refresh rewrote

When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.

* fix(site-replication): keep replay running while an endpoint refresh is pending

A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.

* test(e2e): cover the AND semantics of a two-tag replication filter

The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.

No new test function, so the nightly selection digest is unchanged.

* refactor(site-replication): fold the refresh state-change error into one constructor

The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.

* fix(site-replication): report a peer whose IAM snapshot waits for a repair

An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.

* docs(operations): state the group-status and escalation convergence limits

Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
 2026-09-08 14:58:41 +00:00
houseme bfa8df00e7 test(scanner): add release bundle dry-run fixture (#7515) 
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
 2026-09-08 22:53:04 +08:00
唐小鸭 ae9fe62fb1 fix(sse): resolve 1.0.0 SSE/KMS blockers and P1 findings (#7511) 
* fix(sse): resolve bucket default encryption per request

PUT and the POST-object/extract path resolved a bucket's default
encryption with a hard-coded "no explicit SSE-C" flag, so the default was
layered onto a request that already carried an SSE-C header triple and
then tripped that request's own mutual-exclusion check. Every bucket with
default encryption refused SSE-C single PUTs with 400 InvalidArgument,
while CreateMultipartUpload on the same bucket succeeded because it
resolves SSE elsewhere. Both call sites now derive the flag from the
request headers, as COPY already did.

The bucket default's KMS key id was also inherited independently of the
effective algorithm, so an explicit AES256 request against an aws:kms
default bucket produced a self-contradictory algorithm/key-id pair and
was rejected. The key id is now inherited only when the effective
algorithm is aws:kms, matching the storage-layer resolver.

Refs backlog#2368 B1, B2.

* fix(sse): refuse SSE-KMS without a running KMS service

A write requesting aws:kms on a node with no KMS service fell back to the
node-local SSE-S3 provider: the data key was wrapped with
RUSTFS_SSE_S3_MASTER_KEY while the object metadata still recorded
aws:kms and the requested KMS key id. The stored object claimed a KMS
protection it never had, under a key that was never consulted, and no
signal distinguished it from a genuine SSE-KMS object.

The managed-encryption path now asks the resolved DEK provider whether it
wraps with a node-local master key and refuses SSE-KMS in that case:
InvalidRequest when KMS was never configured, ServiceUnavailable when a
configured service is not running. The check sits after the per-key
authorization gate so an unauthorized caller still receives AccessDenied
whatever the KMS runtime state is, and asks the provider rather than a
parallel availability signal because the provider is what actually wraps
the key. A missing master key no longer answers an SSE-KMS request with
an SSE-S3-worded configuration error.

The SSE-S3 local fallback is unchanged.

Refs backlog#2368 B4.

* fix(ecstore): restore and archive tiers in stored coordinates

Multipart restore addressed the remote tier in plaintext coordinates
while the copy-back reads the stored representation. Each part received a
misaligned slice of the remote object whose length still satisfied the
range, the hash reader and the completion size check, so the restore
reported success and silently replaced the object's bytes. Encrypted and
compressed multipart objects were both affected. Restore now accumulates
stored part sizes, passes the stored length to the hash reader alongside
the plaintext length, and validates against the stored size.

The copy-back digests stored bytes, so its computed MD5 is not the
object's public ETag. Restore now preserves the object ETag on both the
single-part and multipart paths, and gives each restored part its own
recorded part ETag rather than the object-level value.

Transition also handed the tier the object's SSE headers and its
RustFS-wrapped data key as request headers. Any S3 target rejected an
SSE-C archive outright, an SSE-KMS archive asked the target to encrypt a
second time under a key id it does not own, and the wrapped DEK left the
cluster. The archive request now strips every SSE header and encryption
marker with the predicate the replication path already uses; the local
xl.meta keeps all of it, so read-through and restore are unaffected.

Objects restored by an affected release are not detected or repaired
retroactively and must be re-restored from the tier.

Refs backlog#2368 B3, B5; backlog#2369 P7.1.

* fix(rio): lock the v1 nonce layout within a segment

Decrypting a v1 segment tried three historical nonce layouts per frame,
independently for every frame. The last of them exists for streams
written before 1.0.0-alpha.91, which reused a segment's part nonce for
every block in it; because block zero's derived nonce equals that base
nonce, a frame encrypted at index zero authenticated at any position. An
attacker able to rewrite the underlying shards could replay it and have
the forged plaintext returned with 200 and an unchanged length. Shard
integrity uses a keyed-hash-free checksum, which such an attacker can
recompute, so it is not a barrier.

A segment now locks onto whichever layout decoded its first non-zero-index
frame and rejects any later frame needing a different one. That leaves one
residual shape: a stream built purely from repeats of frame zero has no
later frame to disagree. New RUSTFS_ENCRYPTION_LEGACY_NONCE_FALLBACK
(default true, so pre-alpha.91 objects keep decrypting) drops the third
layout entirely when set to false, which closes it. Turning it off refuses
pre-alpha.91 objects, so migrate them first by rewriting in place.

Refs backlog#2369 P2.

* fix(kms): reload a service that failed to start

POST /rustfs/admin/v3/kms/reload short-circuited whenever the persisted
configuration matched the in-memory one byte for byte. A node whose KMS
failed to start keeps that configuration and sits in Error, so the
documented recovery call returned "reloaded successfully" while leaving
the node down. Peers reached the same path through the reload broadcast,
so a cluster that lost Vault during a rolling restart had no working
recovery route other than the node-local start endpoint. Reload now
short-circuits only for a service that is actually running, and otherwise
reconfigures, which starts a service that is not running.

The AWS backend also advertised key-version enumeration through
kms/status, which its own documentation says it cannot do; the capability
and its golden snapshot now say false.

Refs backlog#2369 P1, P7.3.

* docs: record the SSE and KMS changes for 1.0.0

The Unreleased changelog section carried no entry for any encryption work
merged since 1.0.0-rc.5, including three items with operational impact:
the config-secret variable whose absence persists secrets in cleartext
with only a warning, the v2 frame write switch and its rolling-upgrade
constraint, and per-key authorization making a public bucket incompatible
with SSE-KMS objects. Adds those plus this batch, including the SSE-KMS
refusal as a breaking change with both routes out.

Also corrects four places where documentation contradicted the code: the
cleanup register still called encrypted range seek opt-in after its
default flipped, the Helm README claimed vault_mount_path only applies to
Transit while the template also feeds the KV2 mount, the disaster-recovery
drill listed bundle contents for backends whose export is refused with
501, and the Chinese README capability table predated most of the feature
set. Documents the SSE-S3 local master key as a first-class operational
mode with its rotation dead end, and what the v1 frame layout does and
does not authenticate.

Refs backlog#2369 P5.

* fix(kms): classify data-path KMS failures by what the caller can do

Only "key not found" and a backend outage were classified; every other
KMS failure that reached the S3 data path fell through to
500 InternalError with a generic message. A disabled or pending-deletion
key, a denied KMS grant, an encryption-context mismatch, an unsupported
algorithm, a credential or timeout failure, and a capability the
configured backend does not have all looked identical to a server fault.
SDKs therefore applied exponential backoff to configuration errors no
retry can fix, and monitoring counted every one of them against the
server's own error rate.

Unusable-key and request-side failures now answer 400, a denied grant
403, transient backend failures 503, and a missing backend capability
501. Damaged, unreadable, or unknown-format key material keeps its 500:
it is a server-side integrity fault, and existing tests pin it.

The classifier is deliberately separate from the admin lifecycle
mapping, which answers 404 for a missing key because there a key id is
the resource being addressed; on the data path it arrives inside a
request header or a bucket default. Messages either name what the caller
asked for or stay generic, with deployment-side detail left on the error
source the way the storage-IO mapping already does.

Refs backlog#2368 B6.

* fix(kms): track and renew static Vault tokens

Token authentication hard-coded "this token carries no lease", so the
renewal task never started, the remaining-TTL gauge was never published,
and nothing looked wrong. `vault token create` grants a 768-hour TTL by
default, so a cluster that had been healthy for a month turned every KMS
call into a 403 and could not recover without a restart or a
reconfigure. Production configuration validation only rejects the
literal dev-token, so an ordinary expiring token reaches a whole cluster.

The source now reads `auth/token/lookup-self` at login and adopts what
Vault reports. A token with no expiry behaves exactly as before. An
expiring renewable one is picked up by the existing renewal loop and
renewed at half TTL like every other auth method. An expiring
non-renewable one warns with its remaining lifetime and publishes the
gauge, so the fail-closed window is visible before it arrives.

The probe never fails the login: a policy that omits lookup-self, or a
Vault that is briefly unreachable, warns and falls back to exactly the
previous behaviour rather than taking down a deployment that works
today. The scripted Vault test double answers the lookup out of band so
existing scripts keep describing only the protocol under test.

Refs backlog#2369 P3.

* feat(sse): report SSE-C requests that arrive without TLS

An SSE-C request carries the customer's AES key in a request header, so
AWS S3 and MinIO both refuse one that did not arrive over TLS. RustFS
accepted them on any transport: a plaintext hop hands the key to anyone
on the path, and since the object cannot be read without that same key,
the exposure lasts as long as the object does.

Refusing outright is the correct end state but not a safe default to
adopt inside a release window, because the project's own s3-tests and
e2e lanes and most staging deployments speak plain HTTP. This release
reports instead: each such request increments
rustfs_ssec_plaintext_requests_total and logs one warning per process, so
an operator can confirm nothing would break before the default flips.
RUSTFS_SSE_C_REQUIRE_TLS=true opts into the AWS 400 now.

The verdict is per connection rather than per deployment: the layer is
built with whether this listener terminated TLS, and additionally accepts
an https protocol forwarded by a proxy the trusted-proxy configuration
already vetted. It sits beside the rate limiter, after the layer that
makes a forwarded protocol trustworthy and after the request context, so
a rejection can echo the request id.

Refs backlog#2369 P7.2.

* fix(kms): say what a node-local backend means for a cluster

The Local backend keeps key material on each node's own disk and
generates its Argon2id salt per node, so two nodes derive different keys
from the same master_key and an object encrypted on one node cannot be
decrypted on another. Behind a load balancer that surfaces as
intermittent 500s on reads that succeeded moments earlier, with nothing
tying the symptom to the cause: the only signal was a generic
"development, testing and demos only" positioning warning that says
nothing about what actually breaks.

Configuring or reconfiguring Local while the deployment is distributed
now logs a dedicated event and appends the consequence to the configure
response, so the operator who made the change sees it. The product
decision to warn rather than refuse is unchanged.

Refs backlog#2369 P7.4.

* docs: record the remaining SSE and KMS changes for 1.0.0

Adds changelog entries for the KMS data-path status classification, the
Vault static-token lease probe, the SSE-C plaintext-transport report and
its switch, and the node-local backend warning.

Documents two things the backend security guide never stated: that SSE-C
belongs on a secure transport, with the counter and switch to plan the
change around, and that the Local backend cannot be shared by a
multi-node deployment because each node derives different keys from the
same master key.

Refs backlog#2368 B6; backlog#2369 P3, P5, P7.2, P7.4.

* fix(kms): report an unreadable key store as an outage on the S3 path

A backend now distinguishes a key store it could not read from a key
that is genuinely absent, but the S3 boundary collapsed the first one
back onto 500 InternalError through the fallthrough for integrity
faults. The distinction was therefore invisible to the client: a
temporary key-directory outage looked exactly like a permanently damaged
key record, and neither the status nor the metric said the request was
worth retrying.

An unreadable key store joins the retryable class and answers 503, next
to a backend error and a credential failure. Damaged, unreadable or
unknown-format key material keeps its 500.

Refs backlog#2368 B6; builds on rustfs/rustfs#7470.
 2026-09-08 22:37:53 +08:00
overtrue 929a9f27c4 chore(release): merge main into release 
Merge main at 73957d0faf into release. Resolve the multipart test conflict with the current main fixture and remove its superseded ReadPlan test export.
 2026-09-08 22:28:29 +08:00
houseme a950f914ca test(scanner): bind release JSON artifact provenance (#7500) 
Require scanner/heal release evidence JSON artifacts to repeat their measured source revision, run identity, measurement window, gate, and field identity inside the artifact payload. This keeps a refreshed outer bundle hash from accepting stale summary or profile JSON from another run.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
 2026-09-08 21:48:24 +08:00
houseme 3fa2b334be test(scanner): require two-hour measured ABBA windows (#7493) 
Reject measured Scanner/Heal release ABBA manifests and summaries whose evidence window is shorter than the W21 two-hour release requirement.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
 2026-09-08 20:48:33 +08:00
houseme 1748814bbf test(scanner): bind profile artifacts in release evidence (#7487) 
Require Scanner/Heal release bundles to attach every required profiling artifact to P1 profile evidence with relative paths, artifact formats, non-empty files, hashes, and optional per-artifact measurement-window checks.

Document the tightened release bundle profile contract and cover missing, tampered, and mismatched-window profile artifact regressions in the existing checker self-test.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
 2026-09-08 20:32:37 +08:00
cxymds 73957d0faf fix(ecstore): preserve online writes during pool retirement (#7472) 
* fix(ecstore): reconcile identical scanner backlog replicas

* fix(ecstore): type invalid decommission requests

* chore(ecstore): tighten typed-error ratchet baseline

* fix(ecstore): fence late writes to retiring pools

* fix(ecstore): share healthy pool capacity during decommission

* fix: allow active multipart uploads to drain

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
 2026-09-08 12:30:45 +00:00
Zhengchao An 0db77be5c6 fix: attest multi-pool bootstrap per creator and contain lock RPC storms (#7473) 
* fix(ecstore): attest fresh multi-pool bootstrap per pool creator

A fresh deployment whose pools have their first endpoint on different nodes could never publish its initial pool.bin: each node held fresh-bootstrap proof only for the pool it formatted, combine_across_pools collapsed the deployment-wide proof to None, the elected writer never wrote a pending identity, and startup died with "no durable bootstrap identity or pool.bin replica is available" once the init retry budget ran out.

Track first-hand bootstrap authority per pool. The first pool's creator mints the pending cluster identity on the pool it created, every other creator copies that nonce-bound identity onto the pool it formatted first-hand (a scoped identity write that only ever touches pools the process holds first-hand proof for), and the elected writer publishes pool.bin once it holds first-hand proof for pool 0 and every pool replica carries the same pending identity. Fresh + None is still never promoted, corrupt or disagreeing replicas still fail closed, an initialized deployment never reopens bootstrap for an expansion pool, and an elected restart without first-hand proof still cannot reuse a complete pending set.

Startup classification no longer latches the pool-metadata write gate for the two transient outcomes a healthy bootstrap passes through (a non-elected node waiting for the elected writer, the elected writer waiting for the other creators); recover_pool_meta_transaction never clears write_blocked, so a non-elected node that started before pool.bin existed stayed write-blocked for the life of the process. Genuine recovery-required states still latch.

Refs rustfs/backlog#2375, rustfs/backlog#2338

* fix(lock): contain remote lock RPC timeout storms

A lock RPC deadline evicted the shared internode HTTP/2 channel and re-dialed it unconditionally, so one slow lock endpoint produced a cluster-wide RST_STREAM / GOAWAY too_many_resets / reconnect loop (rustfs#7363).

The remote lock client now keeps a per-peer channel history: a timeout evicts only when the peer has completed no lock RPC for two deadlines, evictions and transport-failure re-dials are rate limited per peer (RUSTFS_OBJECT_LOCK_RPC_EVICTION_COOLDOWN_MS, default 5 s), and a timed-out request is detached instead of cancelled, bounded per peer by RUSTFS_OBJECT_LOCK_RPC_DETACHED_LIMIT (default 256) and by the internode RPC timeout; a lock granted after its caller gave up is released immediately. Unlocks that fail the quick retries continue on a deferred 1/2/4/8/16 s schedule before the server lease reclaims them. New rustfs_remote_lock_* metrics cover timeouts, evictions, suppressed evictions, detached streams, late completions and late releases per peer; docs/operations/lock-rpc-storm-protection.md documents the policy.

Refs rustfs/backlog#2375, rustfs#7363

* ci: refresh nightly test selection digests

The replication nightly membership guard expected the 68-test digest from #7422 while the current listing has 71 tests (additions only: test_bucket_replication_sse_c_compressed_passthrough from #7366, matrix_mint_own_version_ids_addresses_mutations_through_the_ledger and matrix_removed_replication_config_abandons_pending_purge from #7368), and the cluster fault lane expected 50 tests while #7374 added test_cluster_root_heal_recovers_remote_shards_after_background_target_crash. Both lanes have failed before running a single test since 2026-09-07. Bind the Linux digests to the listings from scheduled run 34187469350 and the Darwin e2e-nightly digest to the matching local listing.

Refs rustfs/backlog#2375
 2026-09-08 17:05:14 +08:00
houseme 80321e5bb4 test(scanner): bind hard evidence bundle provenance (#7467) 
Require Scanner/Heal release bundle fields to carry source, run, window, timestamp, command, and artifact format provenance before a measured gate can pass.

Keep EC8+4 and performance gate fields tied to a single measurement window so unrelated artifacts cannot be stitched into a release approval.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
 2026-09-08 14:03:37 +08:00
houseme 6fe83f87a4 test(scanner): require hard evidence ABBA manifest (#7459) 
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
 2026-09-08 11:32:34 +08:00
houseme df6981d88e test(scanner): gate release evidence bundles (#7457) 
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
 2026-09-08 11:32:04 +08:00
houseme be5d14c985 test(scanner): bind release evidence fields (#7452) 
Require explicit scanner/heal release evidence field contracts for the scoped ACK and mixed-version rollback gates.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
 2026-09-08 10:14:57 +08:00
houseme 703086d71d test(scanner): require perf summary evidence fields (#7453) 
Fail closed when measured passing Scanner/Heal ABBA summaries omit W10/W11 foreground pressure, lock wait, or attempt-cost evidence.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
 2026-09-08 10:14:41 +08:00
houseme ee752b0b03 test(scanner): summarize failed heal perf reports (#7446) 
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
 2026-09-08 09:16:39 +08:00
houseme 944e26d432 test(scanner): structure heal release evidence lanes (#7442) 
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
 2026-09-08 08:43:43 +08:00
houseme 585b5e1c52 test(scanner): measure heal pacing and cache cost (#7433) 
* test(scanner): measure heal pacing and cache cost

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(error): merge equivalent api message branches

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): cleanup consumed MRF replay journals

Do not retain Accepted or Merged replay intents as startup anchors after they have been handed to the heal manager. Only refused or still-pending replay records keep the journal on disk until a successor snapshot can persist them.

This keeps successor snapshots limited to the pending queue, which lets successful replay remove both authoritative and legacy journal paths and restores the crash-boundary tests around successor flush.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit d5b8f49c9d)

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
 2026-09-08 07:22:10 +08:00
houseme bc468a4986 test(scanner): report heal release gate status (#7432) 
* test(scanner): report heal release gate status

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(error): merge equivalent api message branches

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): cleanup consumed MRF replay journals

Do not retain Accepted or Merged replay intents as startup anchors after they have been handed to the heal manager. Only refused or still-pending replay records keep the journal on disk until a successor snapshot can persist them.

This keeps successor snapshots limited to the pending queue, which lets successful replay remove both authoritative and legacy journal paths and restores the crash-boundary tests around successor flush.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit d5b8f49c9d)

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
 2026-09-08 07:22:04 +08:00
houseme 807f85cbd7 test: summarize scanner heal perf artifacts (#7426) 
* test: summarize scanner heal perf artifacts

Add a quiet Scanner/Heal performance artifact summarizer that normalizes ABBA report verdicts, key regression metrics, cache-cost profile records, and provenance hashes for CI or PR handoff.

Document the summary command in the scanner benchmark runbook and cover measured, synthetic, pending, and invalid cache-cost paths with focused Python tests.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(error): merge equivalent api message branches

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): cleanup consumed MRF replay journals

Do not retain Accepted or Merged replay intents as startup anchors after they have been handed to the heal manager. Only refused or still-pending replay records keep the journal on disk until a successor snapshot can persist them.

This keeps successor snapshots limited to the pending queue, which lets successful replay remove both authoritative and legacy journal paths and restores the crash-boundary tests around successor flush.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit d5b8f49c9d)

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
 2026-09-08 07:21:45 +08:00
唐小鸭 d7e88529c9 fix(kms): report a missing KMS key as 400 KMS.NotFoundException (#7423) 
* fix(kms): report a missing KMS key as 400 KMS.NotFoundException

A PutObject whose resolved SSE-KMS key (request header or bucket default
rule) does not exist in the KMS answered 500 InternalError with a generic
message: KmsError::KeyNotFound fell through to the default arm of the
StorageError-to-ApiError mapping. S3 reports this client mistake as 400
KMS.NotFoundException; the mapping now does the same and names the key.
s3s has no status for a custom code, so the ApiError-to-S3Error conversion
supplies it.

The legacy create-key aliases behind /minio/admin/v3/kms/key/create ignored
the key-id query parameter that mc sends, creating a key under a generated
id instead of the requested name. The alias now honors key-id (and its
keyId/key spellings) alongside the name tag, and refuses a request whose
two sources disagree.

Refs: rustfs/backlog#2330 (KMS-312, KMS-110)

* fix(error): merge equivalent api message branches

Combine the MaxVersionsExceeded and internal IO message branches so Clippy no longer flags identical if blocks while preserving the existing response messages.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): cleanup consumed MRF replay journals

Do not retain Accepted or Merged replay intents as startup anchors after they have been handed to the heal manager. Only refused or still-pending replay records keep the journal on disk until a successor snapshot can persist them.

This keeps successor snapshots limited to the pending queue, which lets successful replay remove both authoritative and legacy journal paths and restores the crash-boundary tests around successor flush.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit d5b8f49c9d)

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
 2026-09-08 07:21:32 +08:00
cxymds 6b05fb6b42 feat(observability): expose pool write-block diagnostics (#7417) 
* feat(observability): expose pool write-block diagnostics

* fix(error): merge equivalent api message branches

Combine the MaxVersionsExceeded and internal IO message branches so Clippy no longer flags identical if blocks while preserving the existing response messages.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): cleanup consumed MRF replay journals

Do not retain Accepted or Merged replay intents as startup anchors after they have been handed to the heal manager. Only refused or still-pending replay records keep the journal on disk until a successor snapshot can persist them.

This keeps successor snapshots limited to the pending queue, which lets successful replay remove both authoritative and legacy journal paths and restores the crash-boundary tests around successor flush.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit d5b8f49c9d)

---------

Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
 2026-09-08 07:21:19 +08:00
Zhengchao An d6bb61d420 fix(s3): reject unsigned x-amz headers on presigned requests (#7425) 
* fix(s3): reject unsigned x-amz headers on presigned requests

A SigV4 presigned URL only binds the headers listed in X-Amz-SignedHeaders, but the handlers applied every x-amz-* request header regardless. The holder of a presigned PutObject URL signed with SignedHeaders=host could add x-amz-tagging, x-amz-storage-class, x-amz-website-redirect-location, ACL, metadata, Object Lock or SSE headers and have them applied (GHSA-g8w9-qw9q-fghr). Reject such requests at the S3 access boundary with 403 AccessDenied and the AWS message "There were headers present in the request which were not signed"; x-amz-cf-id stays tolerated for CloudFront. SigV2 and header-signed SigV4 requests are unchanged.

Regression tests are named after the advisory (unit tests in rustfs/src/auth.rs, e2e in crates/e2e_test/src/presigned_negative_test.rs with a signed-tagging positive control); the security smoke floor rises to 20 and the e2e selection digests are refreshed for the two new cases.

* fix(s3): apply presigned signed-header rule to custom routes and harden parsing

Move the GHSA-g8w9-qw9q-fghr check to the first statement of S3Access::check, apply it in S3Router::check_access so admin, console, STS and extension routes that never reach the access hook enforce the same rule, read X-Amz-SignedHeaders with the exact key the verifier uses and treat a duplicate as signing nothing, and log the rejection as a warn event with the repository field shape. Add presigned GET, unsigned x-amz-copy-source and unsigned Content-Type e2e cases plus a router unit test; raise the security smoke floor to 26 and refresh the selection digests.

* docs(testing): list the full GHSA-g8w9 regression set

* fix(error): merge equivalent api message branches

Combine the MaxVersionsExceeded and internal IO message branches so Clippy no longer flags identical if blocks while preserving the existing response messages.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
 2026-09-07 22:46:02 +00:00
Zhengchao An 7c85c72fd1 docs: scope agent guidance and consolidate review workflows (#7419) 2026-09-07 23:58:05 +08:00
houseme 1a5e2b6256 test(scanner): bind segment proof generations (#7416) 
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
 2026-09-07 23:43:42 +08:00
houseme 752d4a81ab test(scanner): prove restart quantum stages (#7414) 
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
 2026-09-07 23:22:02 +08:00
cxymds 7b40b9503b feat(observability): diagnose node-local S3 write failures (#7407) 2026-09-07 22:24:50 +08:00
cxymds fd92853ac4 fix(ecstore): recover interrupted pool metadata writes (#7387) 
* fix(ecstore): recover interrupted pool metadata writes

* chore(ci): update error format ratchet baseline

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
 2026-09-07 22:17:51 +08:00
唐小鸭 899f81f3ad fix(replication): resolve drifted replicas via a target version ledger (#7368) 
* fix(replication): resolve drifted replicas via a target version ledger

A replication target that mints its own version ids (Wasabi, AWS S3)
never answers to the source uuid, so every version-addressed mutation
after the initial PUT failed forever: permanent version deletes answered
NoSuchVersion every heal cycle, and tag / retention / legal-hold updates
re-PUT the object, minting one more target version per update
(rustfs/backlog#2340).

Record the id the target assigned as a per-target ledger on the source
version (replication-target-version-<arn>, written through the existing
status writeback) and resolve every later mutation through it: version
deletes DELETE the ledger id, metadata updates go through the
metadata-only Object Lock and tagging APIs. Replicas written before the
ledger existed are located by exact key and ETag, minus the candidates
other generations of the key already claim through their own ledgers; an
ambiguous remainder is refused with a backoff instead of guessed, since a
wrong pick would destroy a live generation. A fresh write never consults
content identity. NoSuchVersion on a version-addressed DELETE counts as
purged.

The fake target gains the Wasabi shape (404 NoSuchVersion on an unknown
id, per-version Object Lock APIs) and the matrix covers the three
mutation classes plus the same-bytes generation case.

* fix(scanner): drop the unused Digest import

Same one-line change as rustfs/rustfs#7366 (main is red with it under -D warnings); carried here so the stacked PRs' merge commits compile until that fix lands.

* fix(admin): probe replication-check mutations by the assigned version id (#7373)

On a target that mints its own version ids the DeleteMarker and
VersionDelete phases of ?replication-check were skipped: they addressed
the source id, which such a target never had. The replication worker now
addresses the id the target assigned (the target-version ledger), and
the probe already holds that id from its own PUT, so run both phases
against it. VersionFidelity keeps failing with the mismatch code and the
target stays FAILED; the phases report whether ledger-addressed purges
work against this endpoint (rustfs/backlog#2340).

* fix(replication): abandon purges to targets the bucket no longer names (#7377)

* fix(admin): probe replication-check mutations by the assigned version id

On a target that mints its own version ids the DeleteMarker and
VersionDelete phases of ?replication-check were skipped: they addressed
the source id, which such a target never had. The replication worker now
addresses the id the target assigned (the target-version ledger), and
the probe already holds that id from its own PUT, so run both phases
against it. VersionFidelity keeps failing with the mismatch code and the
target stays FAILED; the phases report whether ledger-addressed purges
work against this endpoint (rustfs/backlog#2340).

* fix(replication): abandon purges to targets the bucket no longer names

A permanent version delete whose replication keeps failing stays in
xl.meta as a PENDING purge, hidden from listings, until every target
confirms it. Once the operator removes the replication configuration or
the rule naming that target nothing ever confirms it: the heal path
derived its delete decision from the configuration (the decision string
is not persisted) and skipped the version forever, so DeleteBucket
answered BucketNotEmpty for a residue the client could neither list nor
remove (rustfs/backlog#2340).

Owe a version purge to the targets its purge state names, let the heal
path through without a configuration, and have the delete worker settle
a target the configuration no longer names as abandoned: the purge is
reported complete locally through the normal writeback, the replica on
the former target is left alone, and the event
replication_purge_abandoned plus a counter are the record.

* fix(admin): send replication-check marker creation without a version id

Running the DeleteMarker / VersionDelete phases on a target that mints
its own version ids exposed two probe-shape bugs on real Wasabi:

- the DeleteMarker phase put the assigned version id on its DELETE. A
  RustFS peer reads the source-deletemarker header and creates a marker,
  but a generic S3 target executes it as a permanent delete of the probe
  version, so VersionDelete then answered NoSuchVersion. Use the same wire
  shape as live delete replication: no versionId on a marker creation.
- cleanup treated NoSuchVersion on the version the VersionDelete phase had
  already removed as a failure (RustFS/MinIO answer 204 there).

Also gate the no-configuration heal pass-through for pending purges on a
purge state that actually names targets, so a purge without a recorded
target keeps the ordinary skip (scanner unit test), and merge origin/main
(#7365 settles the pool-metadata probe test that failed in CI).

---------

Co-authored-by: houseme <housemecn@gmail.com>
 2026-09-07 11:23:32 +00:00
cxymds 37bda24e1c fix: reject unsupported pool expansion with actionable errors (#7360) 
* fix: reject unsupported pool expansion with actionable errors

Report singleton-pool and persisted-topology constraints before misleading startup retries. Preserve single-node multi-drive admission and existing parity policies, and cover format preservation plus operator recovery guidance for issue #6186.

* fix: keep pool layout errors typed

Preserve actionable pool layout diagnostics without adding generic formatted errors. Tighten the shrink-only baseline and assert that both typed payloads survive the I/O boundary.

---------

Co-authored-by: houseme <housemecn@gmail.com>
 2026-09-07 11:07:42 +00:00
GatewayJ c9c6bb7a24 fix(oidc): support workload discovery and JWKS negotiation (#7348) 
* fix(oidc): support workload discovery and JWKS negotiation

* test(oidc): cover existing Console authorization code flows

* test(oidc): cover admin validation and document workload setup
 2026-09-07 06:38:04 +00:00
Zhengchao An 4c4dcb6f5e fix(storage): queue multipart parts for foreground write permits (#7337) 
Multipart parts shared the 250 ms direct-PutObject wait on the foreground
write permit pool, so SDK-default concurrency (many parts per upload in
flight at once) was rejected wholesale with SlowDown at stock settings.
Keep the pool that bounds in-flight bodies, but let parts wait in a
bounded queue with their own timeout before body ingest, report the
queue depth in the ForegroundWrite admission snapshot, and document the
foreground write admission environment variables.
 2026-09-07 04:13:30 +00:00
cxymds 6018dd372f perf(ilm): reduce transition transaction mutations (#7320) 
* perf(ilm): reduce transition transaction mutations

* test(ilm): rename transition kill points

---------

Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
 2026-09-07 02:40:57 +00:00
cxymds a70c96d520 feat(ilm): execute legacy recovery dispositions (#7304) 
* feat(ilm): execute legacy recovery dispositions

* feat(ilm): retry retained transition recovery (#7308)

* fix(admin): use gateway errors for recovery retries

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
 2026-09-07 01:46:43 +00:00
唐小鸭 3b404e56c0 fix(replication): forward single-part object checksums as headers (#7313) 2026-09-07 01:17:24 +00:00
唐小鸭 cf1c45eb91 test(replication): pin directory-marker null version and replication (#7315) 2026-09-06 19:26:09 +00:00
唐小鸭 7993e11058 docs(s3): record object-key path-segment validation as intentional (#7314) 2026-09-06 23:27:10 +08:00
Zhengchao An 22bff27aee fix(storage): derive multipart identity from stored parts (#7305) 2026-09-06 22:04:15 +08:00
houseme c3e6d90c6f docs(heal): clarify start retries and execution budgets (#7300) 
Distinguish control requests, cumulative task execution and object retries. Explain ambiguous responses and bounded envelope replay without promising HTTP idempotency or changing runtime policy.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
 2026-09-06 21:27:37 +08:00