The legacy bucket metadata migration writes the `.bucket-incarnation`
sidecar before `.metadata.bin`. A crash or a lost namespace lease between
the two writes left the bucket with a sidecar and no metadata, and every
later load failed closed with "bucket incarnation sidecar exists without
bucket metadata", so the bucket answered 500 to every request on every
node with no repair path (rustfs/rustfs#8003).
Load the sidecar-only state as a legacy bucket so the migration runs
again. The migration and the force-create path re-read the sidecar under
the transaction lock and keep its incarnation when persisting the
metadata, so the retry never replaces an identity other nodes fenced on.
A retired incarnation is never adopted: it is residue of a deleted
bucket, and re-publishing it would let heal reclaim the new objects.
Consolidate the operator entry point for planning, storage classes and
EC:0 consequences, expansion, rebalance, decommission, heal, drive
replacement, restart recovery, and the rc CLI mapping. Every claim was
checked against the current handlers, storage-class validation, quorum
arithmetic, and the rustfs/cli admin reference; the runbook links the
normative contracts instead of restating them.
Refs rustfs/backlog#2607, rustfs/backlog#2608
fix(ci): align security workflow test with the performance suite's own concurrency group
#7984 moved rustfs-performance-test.yml to the rustfs-performance-suite group but left test_all_suites_hold_the_shared_lock_for_manual_and_chain_runs asserting the shared functional group, so Quick Checks (make script-tests) fails on main and on every PR. Expect the dedicated group for the performance suite and drop the workflow comment that still described the shared lock.
The performance suite runs on the pf-testing fleet - a completely
separate environment (own runner, own target nodes) from the
smoke-testing fleet the functional suites share. Holding
rustfs-shared-functional-tests-v2 made an in-flight performance run
block every functional dispatch (observed: kms queued pending with
zero jobs for 4+ hours behind a 4-hour performance run), and
performance benchmarks queued behind functional suites in return.
Give it its own group; nothing else changes.
* test(ecstore): bind MRF manifest CAS dirsync recovery (#7482)
Cover the committed MRF manifest path through LocalDisk conditional CAS when the metadata directory fsync fails. The fixture proves the previous manifest anchor survives rollback, an unanchored first successor is removed, and the legacy MRF journal remains readable even while global durability is relaxed.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): derive evidence runner profile from registry (#7484)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require mixed-version evidence roles (#7485)
Tighten the Scanner/Heal release bundle checker so mixed-version, scoped-ACK, and rollback fields cannot reuse a generic versions list without proving the expected evidence role.
Require version lists to use source revision identities and include the tested source revision. Also require profile evidence fields to name the core profiling artifacts before release approval.
The release gate remains blocked until measured field evidence is present.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): publish committed MRF runtime checkpoints (#7490)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind profile artifacts in release evidence (#7487)
Require Scanner/Heal release bundles to attach every required profiling artifact to P1 profile evidence with relative paths, artifact formats, non-empty files, hashes, and optional per-artifact measurement-window checks.
Document the tightened release bundle profile contract and cover missing, tampered, and mismatched-window profile artifact regressions in the existing checker self-test.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): harden measured ABBA evidence claims (#7491)
Reject measured Scanner/Heal ABBA manifests whose mixed-version evidence uses the same baseline and candidate source revision or binary hash.
Require crash fault modes and profile artifact names to match the exact supported sets, rejecting missing, duplicate, and unknown values.
Update harness fixtures and regression coverage for same-build mixed-version claims and exact-set release evidence fields.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(e2e): group scanner heal evidence payload (#7494)
Group the EC8+4 Scanner/Heal evidence writer inputs into a typed payload so the distributed e2e crate stays within the clippy argument limit without weakening the lint.
The evidence writer still validates the same S3 bodies, physical shard census, process restart PIDs, and node listings.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF replay bundle fields (#7486)
Bind Scanner/Heal release bundle evidence for MRF durable replay to replay counts, retained responsibility anchors, and successor snapshot publication evidence.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): profile EC84 evidence case runs (#7495)
Give Scanner/Heal evidence cases explicit runtime profiles so EC8+4 background restart and crash cases use their own object count, object size, and partial-progress timeout defaults instead of inheriting the legacy 4x1 case assumptions.
Expose the runtime profile in plan-only output and cover every registry case in the script self-test.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): reject empty release evidence artifacts (#7496)
Require Scanner/Heal release bundle artifact paths to resolve to non-empty files before hashing them.
Cover empty hard-gate artifacts in the existing release bundle checker self-test and keep profile artifact size checking on the shared artifact boundary.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require G14 same-window field coverage (#7489)
Tighten the Scanner/Heal release bundle checker so G14 same-window evidence must name the EC8+4, multi-set, and multi-pool fields covered in that measurement window.
Keep the release gate blocked when same-window evidence omits one of the required G14 fields, without changing production runtime behavior.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require two-hour measured ABBA windows (#7493)
Reject measured Scanner/Heal release ABBA manifests and summaries whose evidence window is shorter than the W21 two-hour release requirement.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF idle checkpoint cleanup (#7497)
Add a runtime cleanup regression test that publishes both retained replay and runtime committed checkpoints, writes scoped and legacy journals, and verifies idle cleanup removes every recovery anchor from the registered local disks.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(e2e): ignore untracked files in build identity (#7498)
Align e2e_test build provenance with Scanner/Heal evidence receipts and server binary provenance by treating only tracked source changes as dirty.
This prevents unrelated untracked worktrees or evidence directories from causing compiled test identity mismatches before real evidence cases can run.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF disk-full evidence fields (#7499)
Require the Scanner/Heal release registry and bundle checker to carry explicit G08 MRF capacity, disk-full, and replica-loss evidence fields before release approval can pass.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind release JSON artifact provenance (#7500)
Require scanner/heal release evidence JSON artifacts to repeat their measured source revision, run identity, measurement window, gate, and field identity inside the artifact payload. This keeps a refreshed outer bundle hash from accepting stale summary or profile JSON from another run.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF rollback mirror filtering (#7501)
Add a regression oracle that keeps scoped-only MRF responsibilities in the authoritative runtime snapshot while omitting them from the v1 legacy rollback mirror.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require segment activation evidence (#7504)
* test(scanner): require segment activation evidence
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): cover activation proof inputs
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF crash matrix cases (#7505)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require release bundle domain evidence (#7502)
Reject scanner/heal release bundles that omit field-level domain evidence for scoped ACK, durable intent, mixed-version, scheduler pressure, profile cost, and two-hour pressure lanes.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF retention GC evidence (#7506)
Require P4 retained-responsibility release bundle evidence to list the retained replay anchor and idle cleanup cases, prove a two-hour retention window, and record both idle cleanup and verified-proof discharge observations.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF cleanup GC soak evidence (#7507)
Require the P4 release bundle to carry measured cleanup/GC soak evidence for retained MRF replay responsibilities. The bundle now needs a two-hour cleanup window, exact cleanup case coverage, observed verified idle GC, and zero pending responsibilities or stale journals after GC.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF disk-full commit anchors (#7509)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): report port allocator bind errors (#7510)
Surface the scanned port window, attempt count, and last bind error when the e2e port allocator cannot reserve a localhost port. This keeps Scanner/Heal evidence failures actionable when the environment blocks binds before business assertions run.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(sse): resolve 1.0.0 SSE/KMS blockers and P1 findings (#7511)
* fix(sse): resolve bucket default encryption per request
PUT and the POST-object/extract path resolved a bucket's default
encryption with a hard-coded "no explicit SSE-C" flag, so the default was
layered onto a request that already carried an SSE-C header triple and
then tripped that request's own mutual-exclusion check. Every bucket with
default encryption refused SSE-C single PUTs with 400 InvalidArgument,
while CreateMultipartUpload on the same bucket succeeded because it
resolves SSE elsewhere. Both call sites now derive the flag from the
request headers, as COPY already did.
The bucket default's KMS key id was also inherited independently of the
effective algorithm, so an explicit AES256 request against an aws:kms
default bucket produced a self-contradictory algorithm/key-id pair and
was rejected. The key id is now inherited only when the effective
algorithm is aws:kms, matching the storage-layer resolver.
Refs backlog#2368 B1, B2.
* fix(sse): refuse SSE-KMS without a running KMS service
A write requesting aws:kms on a node with no KMS service fell back to the
node-local SSE-S3 provider: the data key was wrapped with
RUSTFS_SSE_S3_MASTER_KEY while the object metadata still recorded
aws:kms and the requested KMS key id. The stored object claimed a KMS
protection it never had, under a key that was never consulted, and no
signal distinguished it from a genuine SSE-KMS object.
The managed-encryption path now asks the resolved DEK provider whether it
wraps with a node-local master key and refuses SSE-KMS in that case:
InvalidRequest when KMS was never configured, ServiceUnavailable when a
configured service is not running. The check sits after the per-key
authorization gate so an unauthorized caller still receives AccessDenied
whatever the KMS runtime state is, and asks the provider rather than a
parallel availability signal because the provider is what actually wraps
the key. A missing master key no longer answers an SSE-KMS request with
an SSE-S3-worded configuration error.
The SSE-S3 local fallback is unchanged.
Refs backlog#2368 B4.
* fix(ecstore): restore and archive tiers in stored coordinates
Multipart restore addressed the remote tier in plaintext coordinates
while the copy-back reads the stored representation. Each part received a
misaligned slice of the remote object whose length still satisfied the
range, the hash reader and the completion size check, so the restore
reported success and silently replaced the object's bytes. Encrypted and
compressed multipart objects were both affected. Restore now accumulates
stored part sizes, passes the stored length to the hash reader alongside
the plaintext length, and validates against the stored size.
The copy-back digests stored bytes, so its computed MD5 is not the
object's public ETag. Restore now preserves the object ETag on both the
single-part and multipart paths, and gives each restored part its own
recorded part ETag rather than the object-level value.
Transition also handed the tier the object's SSE headers and its
RustFS-wrapped data key as request headers. Any S3 target rejected an
SSE-C archive outright, an SSE-KMS archive asked the target to encrypt a
second time under a key id it does not own, and the wrapped DEK left the
cluster. The archive request now strips every SSE header and encryption
marker with the predicate the replication path already uses; the local
xl.meta keeps all of it, so read-through and restore are unaffected.
Objects restored by an affected release are not detected or repaired
retroactively and must be re-restored from the tier.
Refs backlog#2368 B3, B5; backlog#2369 P7.1.
* fix(rio): lock the v1 nonce layout within a segment
Decrypting a v1 segment tried three historical nonce layouts per frame,
independently for every frame. The last of them exists for streams
written before 1.0.0-alpha.91, which reused a segment's part nonce for
every block in it; because block zero's derived nonce equals that base
nonce, a frame encrypted at index zero authenticated at any position. An
attacker able to rewrite the underlying shards could replay it and have
the forged plaintext returned with 200 and an unchanged length. Shard
integrity uses a keyed-hash-free checksum, which such an attacker can
recompute, so it is not a barrier.
A segment now locks onto whichever layout decoded its first non-zero-index
frame and rejects any later frame needing a different one. That leaves one
residual shape: a stream built purely from repeats of frame zero has no
later frame to disagree. New RUSTFS_ENCRYPTION_LEGACY_NONCE_FALLBACK
(default true, so pre-alpha.91 objects keep decrypting) drops the third
layout entirely when set to false, which closes it. Turning it off refuses
pre-alpha.91 objects, so migrate them first by rewriting in place.
Refs backlog#2369 P2.
* fix(kms): reload a service that failed to start
POST /rustfs/admin/v3/kms/reload short-circuited whenever the persisted
configuration matched the in-memory one byte for byte. A node whose KMS
failed to start keeps that configuration and sits in Error, so the
documented recovery call returned "reloaded successfully" while leaving
the node down. Peers reached the same path through the reload broadcast,
so a cluster that lost Vault during a rolling restart had no working
recovery route other than the node-local start endpoint. Reload now
short-circuits only for a service that is actually running, and otherwise
reconfigures, which starts a service that is not running.
The AWS backend also advertised key-version enumeration through
kms/status, which its own documentation says it cannot do; the capability
and its golden snapshot now say false.
Refs backlog#2369 P1, P7.3.
* docs: record the SSE and KMS changes for 1.0.0
The Unreleased changelog section carried no entry for any encryption work
merged since 1.0.0-rc.5, including three items with operational impact:
the config-secret variable whose absence persists secrets in cleartext
with only a warning, the v2 frame write switch and its rolling-upgrade
constraint, and per-key authorization making a public bucket incompatible
with SSE-KMS objects. Adds those plus this batch, including the SSE-KMS
refusal as a breaking change with both routes out.
Also corrects four places where documentation contradicted the code: the
cleanup register still called encrypted range seek opt-in after its
default flipped, the Helm README claimed vault_mount_path only applies to
Transit while the template also feeds the KV2 mount, the disaster-recovery
drill listed bundle contents for backends whose export is refused with
501, and the Chinese README capability table predated most of the feature
set. Documents the SSE-S3 local master key as a first-class operational
mode with its rotation dead end, and what the v1 frame layout does and
does not authenticate.
Refs backlog#2369 P5.
* fix(kms): classify data-path KMS failures by what the caller can do
Only "key not found" and a backend outage were classified; every other
KMS failure that reached the S3 data path fell through to
500 InternalError with a generic message. A disabled or pending-deletion
key, a denied KMS grant, an encryption-context mismatch, an unsupported
algorithm, a credential or timeout failure, and a capability the
configured backend does not have all looked identical to a server fault.
SDKs therefore applied exponential backoff to configuration errors no
retry can fix, and monitoring counted every one of them against the
server's own error rate.
Unusable-key and request-side failures now answer 400, a denied grant
403, transient backend failures 503, and a missing backend capability
501. Damaged, unreadable, or unknown-format key material keeps its 500:
it is a server-side integrity fault, and existing tests pin it.
The classifier is deliberately separate from the admin lifecycle
mapping, which answers 404 for a missing key because there a key id is
the resource being addressed; on the data path it arrives inside a
request header or a bucket default. Messages either name what the caller
asked for or stay generic, with deployment-side detail left on the error
source the way the storage-IO mapping already does.
Refs backlog#2368 B6.
* fix(kms): track and renew static Vault tokens
Token authentication hard-coded "this token carries no lease", so the
renewal task never started, the remaining-TTL gauge was never published,
and nothing looked wrong. `vault token create` grants a 768-hour TTL by
default, so a cluster that had been healthy for a month turned every KMS
call into a 403 and could not recover without a restart or a
reconfigure. Production configuration validation only rejects the
literal dev-token, so an ordinary expiring token reaches a whole cluster.
The source now reads `auth/token/lookup-self` at login and adopts what
Vault reports. A token with no expiry behaves exactly as before. An
expiring renewable one is picked up by the existing renewal loop and
renewed at half TTL like every other auth method. An expiring
non-renewable one warns with its remaining lifetime and publishes the
gauge, so the fail-closed window is visible before it arrives.
The probe never fails the login: a policy that omits lookup-self, or a
Vault that is briefly unreachable, warns and falls back to exactly the
previous behaviour rather than taking down a deployment that works
today. The scripted Vault test double answers the lookup out of band so
existing scripts keep describing only the protocol under test.
Refs backlog#2369 P3.
* feat(sse): report SSE-C requests that arrive without TLS
An SSE-C request carries the customer's AES key in a request header, so
AWS S3 and MinIO both refuse one that did not arrive over TLS. RustFS
accepted them on any transport: a plaintext hop hands the key to anyone
on the path, and since the object cannot be read without that same key,
the exposure lasts as long as the object does.
Refusing outright is the correct end state but not a safe default to
adopt inside a release window, because the project's own s3-tests and
e2e lanes and most staging deployments speak plain HTTP. This release
reports instead: each such request increments
rustfs_ssec_plaintext_requests_total and logs one warning per process, so
an operator can confirm nothing would break before the default flips.
RUSTFS_SSE_C_REQUIRE_TLS=true opts into the AWS 400 now.
The verdict is per connection rather than per deployment: the layer is
built with whether this listener terminated TLS, and additionally accepts
an https protocol forwarded by a proxy the trusted-proxy configuration
already vetted. It sits beside the rate limiter, after the layer that
makes a forwarded protocol trustworthy and after the request context, so
a rejection can echo the request id.
Refs backlog#2369 P7.2.
* fix(kms): say what a node-local backend means for a cluster
The Local backend keeps key material on each node's own disk and
generates its Argon2id salt per node, so two nodes derive different keys
from the same master_key and an object encrypted on one node cannot be
decrypted on another. Behind a load balancer that surfaces as
intermittent 500s on reads that succeeded moments earlier, with nothing
tying the symptom to the cause: the only signal was a generic
"development, testing and demos only" positioning warning that says
nothing about what actually breaks.
Configuring or reconfiguring Local while the deployment is distributed
now logs a dedicated event and appends the consequence to the configure
response, so the operator who made the change sees it. The product
decision to warn rather than refuse is unchanged.
Refs backlog#2369 P7.4.
* docs: record the remaining SSE and KMS changes for 1.0.0
Adds changelog entries for the KMS data-path status classification, the
Vault static-token lease probe, the SSE-C plaintext-transport report and
its switch, and the node-local backend warning.
Documents two things the backend security guide never stated: that SSE-C
belongs on a secure transport, with the counter and switch to plan the
change around, and that the Local backend cannot be shared by a
multi-node deployment because each node derives different keys from the
same master key.
Refs backlog#2368 B6; backlog#2369 P3, P5, P7.2, P7.4.
* fix(kms): report an unreadable key store as an outage on the S3 path
A backend now distinguishes a key store it could not read from a key
that is genuinely absent, but the S3 boundary collapsed the first one
back onto 500 InternalError through the fallthrough for integrity
faults. The distinction was therefore invisible to the client: a
temporary key-directory outage looked exactly like a permanently damaged
key record, and neither the status nor the metric said the request was
worth retrying.
An unreadable key store joins the retryable class and answers 503, next
to a backend error and a credential failure. Damaged, unreadable or
unknown-format key material keeps its 500.
Refs backlog#2368 B6; builds on rustfs/rustfs#7470.
* test(storage): cover MRF cleanup delete ENOSPC anchor (#7513)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require heal retry stats evidence (#7514)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add release bundle dry-run fixture (#7515)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): confirm recovery intent accept readback (#7516)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF cleanup delete ENOSPC evidence (#7518)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require bounded retry window evidence (#7517)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): emit scanner heal G09 upgrade evidence (#7519)
Record measured Scanner/Heal G09 evidence artifacts from the upgrade compatibility lanes when a fresh evidence directory is provided.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): persist MRF rollback mirror boundary (#7520)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add G09 upgrade evidence runner (#7522)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(ci): bind performance runs to selected inputs (#7512)
* test(scanner): add G09 upgrade evidence runner
Add a reusable Linux x86_64 runner for the Scanner/Heal G09 mixed-version and rollback upgrade evidence lanes.
The helper reads the pinned previous-release asset metadata from the upgrade workflow, verifies the downloaded binary, builds the current head, runs both ignored E2E tests, and fails unless the expected G09 JSON artifacts exist.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): implement V2 evidence preflights (#7523)
* feat(scanner): wire dirty usage producer identities
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add segment activation preflight proof
Keep scanner segment reuse behind a structured activation preflight so release evidence can prove the production gate remains disabled until every producer, generation, overflow, cold-oracle, and distributed invalidation check is satisfied.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): expose distributed invalidation evidence
Record an explicit distributed segment invalidation evidence summary when remote dirty usage snapshots are bound to the current activity window and the authenticated scoped ACK capability probe succeeds.
Reject peer dirty usage snapshots that contradict the peer activity pending bit so scoped ACKs fail closed instead of clearing an unadvertised remote mutation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): assemble release evidence bundles (#7526)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add G09 upgrade evidence runner (#7524)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(ci): bind performance runs to selected inputs (#7512)
* test(scanner): add G09 upgrade evidence runner
Add a reusable Scanner/Heal G09 runner for Linux PR-head validation. The script downloads the pinned previous release, builds the current checkout, runs the mixed-version and rollback upgrade E2E lanes, and validates the measured raw artifacts before they can be consumed by the release bundle gate.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): replay recovery intents while disabled (#7521)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): preflight G09 evidence disk space (#7527)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(ci): bind performance runs to selected inputs (#7512)
* fix(targets): reject trailing batch items (#7508)
* test(scanner): preflight G09 evidence disk space
Fail the Scanner/Heal G09 upgrade evidence runner before downloading or building when the validation host does not have enough free space for a full raw evidence pass.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: cui fliter <imcusg@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): respect cargo target dir in G09 runner (#7528)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(ci): bind performance runs to selected inputs (#7512)
* fix(targets): reject trailing batch items (#7508)
* fix(scanner): respect cargo target dir in G09 runner
Write the RustFS feature stamp under the resolved Cargo target directory so remote validation hosts with CARGO_TARGET_DIR set can reuse the built PR-head binary.
Extend the runner plan/self-test path to cover relative target-dir resolution.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: cui fliter <imcusg@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): add G09 upgrade evidence runner (#7525)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(ci): bind performance runs to selected inputs (#7512)
* test(e2e): add G09 upgrade evidence runner
Add a Linux x86_64 runner that downloads the pinned previous release, builds the current RustFS binary, runs the mixed-version and rollback upgrade compatibility lanes, and verifies the required Scanner/Heal G09 raw evidence artifacts.
Document the runner and add a shell self-test for help, dry-run, SHA validation, and non-empty artifact directory guards.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): emit G09 release bundle gate evidence (#7530)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(ci): bind performance runs to selected inputs (#7512)
* fix(targets): reject trailing batch items (#7508)
* test(scanner): emit G09 release bundle gate evidence
Write a bundle-ready G09 gate descriptor from the Linux upgrade evidence runner and validate the single G09 gate with the shared release-bundle rules without approving the full Scanner/Heal release.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: cui fliter <imcusg@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): reject invalid G09 runner test selection (#7531)
Validate the selected G09 evidence lane before process-substitution case expansion so --plan-only cannot turn an unknown --test value into an empty successful plan.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): keep G09 source binary path clean (#7532)
Keep checksum verification output out of the command substitution that resolves the previous-release binary for the G09 runner.
Also tolerate non-GNU sha256sum in local self-tests by falling back to shasum when GNU --check support is unavailable.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF replay proof cleanup (#7534)
Exercise the committed MRF replay checkpoint path with a real ECStore-backed bucket incarnation, retain the replay anchor until an exact verified proof arrives, then delete the proof-discharged checkpoint and prove a subsequent restart does not resurrect the replayed work.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): carry segment activation preflight evidence (#7535)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): surface cold segment reuse oracle (#7536)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require hard evidence artifact payloads (#7537)
Require Scanner/Heal release bundle JSON artifacts to carry hard-domain evidence fields for mixed-version, crash, capacity, disk-full, replica-loss, and MRF cleanup gates. This prevents a descriptor from approving a hard gate while pointing at a generic measured artifact that lacks the boundary-specific oracle fields.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): bind producer evidence to dirty generations (#7538)
Record scanner segment producer identities with the dirty usage generation they invalidated, then expose a cycle-local producer evidence snapshot for segment reuse activation preflight.
Production activation remains fail-closed until durable producer identity and restart-gap proof are available.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): persist segment invalidation proof metadata (#7539)
* test(scanner): wait for EC8+4 heal control readiness (#7540)
* test(scanner): list full P4 release evidence fields (#7542)
Align the Scanner/Heal release requirement registry with the release bundle gate for P4 so the closure checklist advertises MRF scale, replay cost, retained responsibility, and cleanup/GC soak evidence together.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add W16 release evidence runner (#7543)
Add a W16 Scanner/Heal evidence runner that executes the recovery-intent crash-boundary and quota-authority lanes, emits measured G04/G12 JSON artifacts, and validates the resulting single-gate release descriptors.
Wire its shell self-test into script-tests and document the release evidence entry point.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix: address release acceptance regressions (#7541)
* feat(scanner): admit durable segment proof replay (#7544)
* feat(scanner): admit durable segment proof evidence
Validate complete set snapshot segment invalidation proof metadata against the current dirty usage generation window and scanner process epoch before clearing durable producer and restart-gap activation blockers.
Production segment reuse remains gated by the explicit activation flag.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): replay segment proof from root snapshots
Carry segment invalidation proof metadata into root snapshot set states so a complete published baseline can replay the durable producer evidence recorded by each set cache.
Keep the field additive for older readers and leave incomplete or LKG set states unproven.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind MRF release artifacts to evidence kinds (#7545)
Require Scanner/Heal G07/G08/P4 JSON artifacts to declare the exact MRF evidence kind expected for each release-bundle field. This keeps generic measured JSON from satisfying durable replay, ENOSPC, replica-loss, scale, replay-cost, retained-responsibility, or cleanup-GC soak gates.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): enable segment reuse activation gate (#7546)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): run scoped fallback cases on dedicated stack (#7550)
Linux libtest defaults can abort these deep EC-store futures before the assertions run. Wrap the async scoped fallback cases in the repository's dedicated-stack test pattern so the default cargo test path exercises the oracle directly.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): roll out distributed e2e selection (#7549)
Refresh the committed e2e-distributed selection digest after the EC8+4 Scanner/Heal evidence case joined the profile, and document the registry-aware runner entry for the distributed case.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add W13 MRF evidence runner (#7547)
Add a W13 durable MRF evidence runner that emits measured G07, G08, and P4 JSON artifacts and release descriptors through the existing scanner/heal bundle gate.
The runner now executes the ignored MRF replay evidence test with an exact full test path, validates raw artifact kinds and gate decisions, prepares Linux tmpfs-backed ENOSPC roots for G08, and documents the Linux/long-soak boundaries.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): fix segment proof fixtures
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): reduce W13 evidence helper arguments
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* ci(test): enable local KMS for s3 compatibility tests (#7553)
Configure the s3-tests harness with a local KMS key so SSE-KMS cases run in CI without relying on an external KMS service.
Also move the anonymous POST default SSE-KMS regression onto the shared local KMS test environment.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bind segment proof to typed dirty suffix (#7564)
Track producer identity completeness per pending bucket and retain process coverage across durable acknowledgements. Admit a prior same-process segment proof only for an exact or strictly earlier fully typed generation window.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): validate scoped ACK clear receipts (#7563)
Fail closed when a signed scoped ACK response claims more cleared entries than its request contained, while preserving partial-clear semantics and zero-mutation capability probes.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bind cold scoped cache reuse (#7565)
Record an explicit cold bucket reuse proof for scoped set scans and treat fully cold sets as valid cold-zero-walk reuse when the baseline and bucket incarnations are bound.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(s3): preserve delete marker read error headers (#7558)
* chore(release): sync configurable nightly build workflow from main (#7567)
feat(nightly): make the build branch configurable (NIGHTLY_BRANCH var + dispatch input) (#7557)
* docs(obs): correct metric export interval and explain alert timing (#7571)
* fix(heal): persist scoped admin admission handoff (#7569)
* fix(heal): persist root admission handoff
Persist administrator root heal ownership before queue publication so crash restart can replay an admitted control-plane start without relying on process-local replay cache state.
Protect queued durable root owners from priority displacement; callers must use forceStart cancellation semantics to replace them.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): persist admin heal handoff types
Extend durable admin heal admission handoff beyond cluster roots while preserving legacy root recovery records. Persist the heal type in schema v2, replay typed admin requests after restart, keep non-admin requests out of the durable control-plane handoff, and allow path cancellation to clear durable-only non-root records.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): preserve disjoint durable admin owners
Limit admin forceStart durable-owner cleanup to overlapping heal requests after the admin recovery journal begins storing non-root work. Add fail-closed and displacement tests so durable bucket owners cannot be silently dropped by unrelated replacement or priority pressure.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): persist admin terminal receipts
Persist terminal receipts for durable administrator heal starts before retiring their pending intent so restart can answer completed or cancelled scoped tasks without replaying them.
Retain pending work when terminal publication fails and keep timeout owners on checkpoint-only replay semantics.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): mark committed replication and tier mutations (#7568)
* fix(scanner): mark committed replication and tier mutations
Bridge ECStore committed replication status writebacks and lifecycle tier expiration cleanup into scanner dirty-usage producer identity tracking. Only successful replication metadata updates mark Replication; validate-only, superseded, and retry paths remain unmarked.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): replay durable dirty producer records
Add a strict scanner dirty-usage producer replay schema that can restore typed producer coverage after restart without treating ordinary process-local mutations as durable evidence. Replay records validate schema, cache key format, generations, scopes, producer identities, duplicate buckets, byte limits, and entry limits before modifying dirty state.
Keep segment activation fail-closed unless the restored bucket state came from durable replay; subsequent local mutations clear the durable authority bit for that bucket. Also fix the release scanner clippy lint in scoped cold-reuse proof creation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): recover expansion buckets and resume retirement (#7574)
* fix(ecstore): recover partially created expansion bucket volumes
* fix(ecstore): resume retirement after startup safety checks
* fix(ci): align S3 KMS fixtures and default key expectations (#7573)
* fix(heal): prune retained root terminal receipts (#7576)
Bound durable root-heal terminal receipt retention to the completed-status TTL while keeping expired receipts authoritative enough to retire stale pending intents before deleting the receipt itself.
Add a budgeted background GC pass and fail closed on corrupt or non-terminal receipt records.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(storage): preserve retryable publication guard errors (#7561)
* fix(storage): preserve retryable publication guard errors
* fix(scanner): satisfy scoped cache clippy lint
Use bool::then_some for the scoped cold bucket reuse proof construction after merging the latest release branch.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(health): report storage quorum in node readiness (#7566)
* fix: preserve scanner backlog during pool retirement (#7555)
* fix: hand off native scanner backlog before pool retirement
* test: initialize optional proof in data usage fixtures
* perf: bound parallel V3 pool metadata phase writes
* test: keep native backlog fault errors typed
* test: skip unchanged pool metadata in crash barriers
---------
Co-authored-by: cxymds <cxymds@gmail.com>
* test(scanner): bind profile artifact provenance (#7577)
* fix(scanner): persist dirty producer journal (#7580)
Persist scanner dirty producer replay records from committed replication and tier mutations, hydrate journal state on restart, and clear durable records only after acknowledged dirty generations.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test: add scanner heal scheduler pressure evidence (#7584)
Add measured G10/P1/P3 scheduler-pressure release descriptor generation, ABBA scheduler/profile provenance, and focused self-tests for the release bundle gate.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test: add scanner heal G14 multiset evidence (#7583)
Add EC8+4 multi-set and multi-pool scanner/heal release evidence coverage, including e2e registry cases, oracle checks, and a G14 descriptor assembler for measured case artifacts.
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(nightly): push assets with plain git on release — the build fleet has no gh CLI (#7586)
fix(nightly): push assets with plain git — the build fleet has no gh CLI
The sm-standard-4 runners used by the nightly build have no gh binary
(functional-chain workflows run elsewhere, on the jumpbox). The assets
publish step died on 'gh: command not found' at the credential-helper
setup, and the preceding clone failure had been masked by 2>/dev/null,
misleading the step into the orphan path. Swap clone and remote setup
to plain git with the token embedded in the URL; push semantics are
unchanged.
* fix(scanner): stabilize retired backlog survivor commits (#7585)
Treat an authoritative scanner pause-backlog commit as stable when every member in that commit stores the matching stable and committed record. Retired source replicas can remain missing in the observed topology while survivor membership is already stable.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(metrics): prevent duplicate drive reporting across nodes (#7582)
Co-authored-by: houseme <housemecn@gmail.com>
* fix(ecstore): avoid Windows batch-delete cleanup sharing conflicts (#7579)
Co-authored-by: houseme <housemecn@gmail.com>
* feat(ilm): inspect legacy transition version state (#7581)
* test(scanner): add release evidence ingestion gates (#7588)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix: reuse admin error boundary for legacy tier reconciliation (#7587)
* fix(test): stabilize release CI heal fixtures (#7589)
Keep heal admission-only tests off durable root recovery so mock storage does not require local metadata disks. Align forceStart retry assertions with replacement semantics and fix the e2e metric clippy collapse.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(nightly): publish packages as assets of the rolling 'nightly' release (#7592)
Replace the assets-branch scheme with a proper GitHub Release on
rustfs/auto-testing: a single 'nightly' release whose deb/rpm assets
are replaced in place on every build. This is the standard channel —
visible on the repo's Releases page, stable download URLs, no git
history growth (release assets live outside the repository).
- New scripts/release/publish_nightly_assets.sh: resolves-or-creates
the 'nightly' release via the REST API, deletes same-name assets,
uploads rustfs-nightly-latest.{deb,rpm}, then PATCHes the release
body with the build provenance (ref@sha, run link, sizes, SHA256).
Plain curl + python3, no gh CLI (the build fleet has none — #7586).
- The workflow step shrinks to invoking the script; full flow
exercised end-to-end against the real release with probe files
(create / upload / overwrite / download round-trip / body update).
* test(scanner): add release bundle evidence descriptors (#7595)
* test(scanner): merge release evidence descriptors
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add authority and legacy release descriptors
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add scoped ack release descriptor
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require all release evidence fields
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add mrf release descriptor producer
Add a standalone W13 MRF descriptor producer for measured G07/G08/P4 raw artifacts, with fail-closed self-tests for missing responsibility cases, disk-full ENOSPC observations, synthetic artifacts, and short P4 cleanup soaks.
Also bind G12 quota path evidence to explicit case lists so W16 descriptors cannot pass without reset and settlement coverage.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): reuse mrf release descriptor producer
Route the W13 runner through the shared MRF descriptor producer so measured runs and offline re-packaging use the same fail-closed validation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(ilm): backfill proven legacy transition state (#7591)
* feat(ilm): backfill proven legacy transition state
* fix(ilm): remove redundant reconcile test clone
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): remove redundant transition clone (#7597)
Drop an unnecessary clone in the legacy transition state digest regression so release CI clippy lanes can run with warnings denied.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): seed current backlog members before committing (#7594)
* fix(scanner): seed current backlog members before committing
* test(app): run recursive force delete test on large stack
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind raw profile artifacts (#7598)
Require Scanner/Heal release profile wrappers to carry bundled raw profile artifacts plus the matching profile cost metrics.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): surface first failed object in status (#7599)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover retry budget across attempts (#7600)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): harden durable producer evidence gate (#7601)
Require Scanner/Heal G11 maintenance evidence to prove durable dirty producer journal replay, replay-state hydration, generation-bound ACK clearing, and invalid journal fail-closed cases before segment reuse can be activated.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): harden scoped ack evidence proof (#7602)
Require Scanner/Heal G03 scoped ACK evidence fields to carry their own measured provenance and concrete ACK, capability, and mixed-peer observations before release-bundle gate verification.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind status outcome raw provenance (#7603)
Require measured G05/G06/R-D raw status, compatibility, and disposition artifacts to share run identity before producing release bundle descriptors.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): retry fully recoverable batch failures (#7604)
Allow recursive bucket and cluster heal attempts whose retained batch failure is entirely retryable to use the existing bounded manager retry budget. Permanent or mixed batch failures remain terminal so heal outcome semantics stay strict.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): avoid movement debt for stale remote lease (#7605)
Classify a final remote scanner publication lease validation failure from a fresh activity snapshot instead of treating every validation error as data movement. If the peer session and movement generation still match the granted leases and publication is otherwise allowed, keep the publication rejected as an activity-baseline miss without creating pause backlog movement debt. Preserve DataMovement for expired leases, peer restarts, movement generation changes, and active publication blocks.
Tests cover namespace-only validation invalidation and the remote fence-loss cases that must still defer as DataMovement.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* chore(deps): refresh workspace dependencies (#7606)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind G14 case artifacts (#7611)
Require Scanner/Heal G14 release bundle JSON wrappers to mirror their outer evidence and carry self-contained case artifact provenance.
Copy proof-json case artifacts into the generated G14 descriptor bundle so assembled release bundles can validate case file hashes after relocation.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): retry contended decommission cancel target locks (#7612)
* test: align transition fixtures and KMS context expectations (#7610)
* test(scanner): stabilize W13 release gate evidence (#7613)
* feat(nightly): publish packages as assets of the rolling 'nightly' release (sync from release) (#7593)
feat(nightly): publish packages as assets of the rolling 'nightly' release (#7592)
Replace the assets-branch scheme with a proper GitHub Release on
rustfs/auto-testing: a single 'nightly' release whose deb/rpm assets
are replaced in place on every build. This is the standard channel —
visible on the repo's Releases page, stable download URLs, no git
history growth (release assets live outside the repository).
- New scripts/release/publish_nightly_assets.sh: resolves-or-creates
the 'nightly' release via the REST API, deletes same-name assets,
uploads rustfs-nightly-latest.{deb,rpm}, then PATCHes the release
body with the build provenance (ref@sha, run link, sizes, SHA256).
Plain curl + python3, no gh CLI (the build fleet has none — #7586).
- The workflow step shrinks to invoking the script; full flow
exercised end-to-end against the real release with probe files
(create / upload / overwrite / download round-trip / body update).
* test(scanner): stabilize W13 release gate evidence
Treat only real raw-entry windows as replayed scanner enumeration in the restart diagnostic, so final completion rounds without raw entries are not fail-closed as raw replays.
Allow the EC8:4 multi-pool heal evidence case to select an outage object key that routes to an online pool while an entire target pool is down, preserving strict behavior for single-pool cases.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bound native backlog fixture resources (#7609)
* fix(scanner): bound native backlog fixture resources
Keep native scanner backlog restart fixtures within low file descriptor limits by reducing the native-only disk layout, making fixture shutdown drain background work, and avoiding long-lived ECStore retention from background loops.
Convert ECStore-backed background refresh/recovery/monitor tasks to upgrade weak owners only while doing work so completed test stores release their disk graph before the next native fixture starts.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(tier): refresh release-merged failure fixtures
Reuse the raw legacy-transition fixture helper so delete-all expiry actually exercises Unknown transitioned history metadata after the release merge.
Rewrite restore-failure disk fixtures through per-disk xl.meta snapshots so stale destination identities can be persisted without tripping ordinary metadata update guards.
Update the SSE KMS mismatch expectation to the InvalidRequest/context_mismatch behavior now returned by the rio-v2 path.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ci): align release E2E gate and repair regression tests (#7607)
* fix(scanner): harden heal release gates (#7614)
* test(scanner): reject synthetic heal evidence inputs (#7617)
Harden the Scanner/Heal checkpoint restart and G14 EC evidence producers so release descriptors cannot be assembled from marked fixture, dry-run, or synthetic case inputs.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): collect status outcome raw evidence (#7618)
Normalize live Scanner/Heal status/outcome observations into the measured raw artifacts consumed by the G05/G06/R-D release descriptor producer.
Reject synthetic observations, incomplete required cases, and reused run/window identities before raw artifacts can enter the release bundle flow.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bind scoped ack confirmation to generation (#7619)
Require lost scoped dirty-usage ACK reconciliation to observe a clean peer activity generation that covers the requested ACK generation. Apply the same guard in the scanner aggregation path so a stale clean activity snapshot cannot discharge pending maintenance after an uncertain ACK response.
Refs rustfs/backlog#2427
Refs rustfs/backlog#2281
Refs rustfs/backlog#2240
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test: report tier reference proof setup failures without hanging (#7608)
* test(scanner): plan Linux evidence validation (#7620)
Add a Scanner/Heal Linux release-evidence planner that emits a machine-readable execution manifest for the remaining measured validation lanes.
The planner can run only lightweight preflight checks and keeps plan-only output distinct from measured release evidence.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): select G14 replacement drive by evidence (#7622)
Choose the replacement disk from the target node by the presence of complete pool metadata instead of assuming the first configured drive is the scanner metadata holder. This keeps the G14 multi-set harness aligned with multi-drive EC layouts and adds clearer diagnostics when multi-pool outage writes fail closed.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): report Linux evidence plan status (#7624)
Add a status mode for the Scanner/Heal Linux evidence planner so operators can identify missing or incomplete artifacts before final bundle assembly.
The status output stays plan-only and never reports release approval.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add G14 multi-pool background heal evidence (#7628)
test(scanner): align G14 multi-pool outage evidence
Treat the localhost multi-pool topology as whole-pool loss when the target node is down. If that topology cannot admit the outage object while the pool is offline, defer that object write until the pool rejoins and record the oracle marker instead of failing before the real crash/restart evidence runs.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): verify deferred multipool outage evidence (#7629)
Validate the G14 multi-pool oracle field that records down-window outage PUT refusal and requires the deferred post-rejoin outage object to be accepted and checked through S3.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): allow metadata snapshots at read quorum (#7627)
fix(ecstore): admit metadata snapshots at read quorum
Allow guarded bucket metadata snapshot existence checks to use read quorum so degraded erasure sets can continue Object Lock snapshot reads without weakening bucket mutation or object write quorum.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* chore(deps): refresh release dependencies (#7632)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): use pool lock quorum for set writes (#7630)
Use the pool-wide namespace lock client domain for every set in a pool so degraded EC8+4 multi-set writes are gated by node-level lock quorum instead of the narrower per-set endpoint host slice.
Keep namespace-lock domain deduplication tied to both the pool namespace and shared clients, and add regression coverage for three-locker degraded writes plus cross-pool domain separation.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): keep G14 multi-set restart graceful (#7633)
test(e2e): keep multi-set restart graceful
Include the EC8+4 multi-set restart scenario in the graceful interruption lane so the unclean-shutdown marker assertion matches the scenario semantics.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): record deferred G14 outage writes (#7637)
Include optional outage-write diagnostics in G14 evidence oracles so deferred multi-pool outage writes bind their down-window refusals and post-rejoin acceptance.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bound G14 multi-pool evidence size (#7640)
Keep the multi-pool evidence runner within the registry object budget while retaining deferred outage-write diagnostics for release-gate validation.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): find nextest junit fallback for evidence cases (#7643)
Handle cargo-nextest writing JUnit reports under the workspace target directory even when the Rust build uses CARGO_TARGET_DIR. This keeps measured Scanner/Heal evidence cases from passing the real test but failing final receipt packaging.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ilm): preserve cleanup ownership on tiered overwrites (#7639)
* fix(ilm): preserve cleanup ownership on tiered overwrites
* fix(ci): refresh E2E selection for tier overwrite regression
* test(ilm): restore noncurrent compensation tests to serial CI (#7645)
* fix(heal): preserve null-version tags and encoded object paths (#7644)
* fix(e2e): record EC8+4 drive restart set size (#7648)
* fix(e2e): record EC8+4 drive restart set size
Include the erasure set drive count in the distributed EC8+4 drive restart oracle so Scanner/Heal release evidence matches the registry contract.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(s3): tighten s3s footprint ratchet
Replace release-merge s3_error! macro calls with equivalent S3Error constructors so the s3gate migration ratchet does not grow on the PR merge tree.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): target multi-set outage heal candidate (#7653)
* test(e2e): target multi-set outage heal candidate
Require the outage write used by EC8+4 multi-set root-heal evidence to miss the same erasure index owned by the selected replacement drive. This avoids accepting a candidate from a different set and turning a valid heal into a false negative.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(e2e): satisfy G14 heal lint gates
Remove clippy-only noise from the G14 multi-set heal evidence test and align the admin route policy inventory with the registered heal catch-all route.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* update
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): reuse API error boundary for decoding failures (#7647)
* fix(heal): reuse API error boundary for decoding failures
* Update heal.rs
Signed-off-by: houseme <housemecn@gmail.com>
* test(ecstore): synchronize batch cleanup metadata reads
Acquire object read locks while observing unversioned and explicit-version batch cleanup, then release them before waiting for progress. This prevents snapshots from spanning per-disk marker removal while retaining the existing quorum, timeout, and remote delete count assertions.
Validation: cargo fmt --all --check and git diff --check passed. The focused nextest test passed 20 stress iterations each with test-util and test-util,rio-v2, with retries disabled.
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
* fix(scanner): wait for ABBA telemetry window (#7660)
Size the scanner collector wait budget from the requested sample count and interval instead of using a fixed 120 second timeout. This preserves the full duration/60+1 telemetry sample set for measured ABBA runs.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(ci): add fault-tolerance degradation suite to the functional chain
Adds a fault-tolerance suite that verifies read/write behavior under
drive and node loss against the erasure-coding contract and snapshots
health-endpoint responses at every degradation tier. Scenarios are
derived from product source (default_parity_count, erasure set sizing):
- A: single-node 4 drives (EC:2, read quorum 2): hide 1/2/3 drives
- B: multi-node 4x1 (one set of 4, EC:2): stop 1/2/3 nodes
- C: multi-node 4x4 (one set of 16, EC:4, read quorum 12): 1 node down
lands exactly on the read-quorum boundary; 2 nodes down breaks it
- C2: multi-node 4x4 with RUSTFS_STORAGE_CLASS_STANDARD=EC:8 (read
quorum 8, write quorum 9, lock majority 9): 2 nodes down puts reads
inside the reported divergence window (read quorum met while the
lock majority is broken)
By default a reads-refused-despite-met-read-quorum observation is
reported as known-divergence without failing the suite; the strict
input escalates it. Chain order becomes:
upgrade -> s3 -> kms -> tier -> storage -> heal -> pool -> security ->
replication -> fault-tolerance -> performance.
Covers the findings of the 2026-09 external degradation report
(reads at read quorum, health-endpoint readiness truthfulness,
degradedReasons capture) as automated regression probes.
* Revert "feat(ci): add fault-tolerance degradation suite to the functional chain" (#7666)
This reverts commit 6eb200022e.
* fix(ecstore): preserve multi-pool version histories (#7658)
* fix(ecstore): preserve multi-pool version histories
* fix(ecstore): satisfy clippy in rebalance test
* fix(ecstore): preserve decommission delete compatibility
* fix(heal): recover stale peer state after restart (#7665)
* fix(heal): select writable recovery intent owner (#7668)
* feat(ci): add fault-tolerance degradation suite to the functional chain (#7669)
* fix(get): reject pre-header read quorum failures (#7670)
* fix(get): reject pre-header read quorum failures
* fix(ci): unblock get pre-header quorum checks
* ci: stop defaulting suite dispatches to a pinned release (#7681)
The rustfs_version dispatch input defaulted to 1.0.0-rc.4-preview.1,
which shadowed the nightly fallback and started 404ing once that
release was deleted. Manual dispatches with no inputs now fall through
to the nightly package (same contract the pool suite already has);
passing rustfs_version or package_url still pins the build exactly as
before. Chain (repository_dispatch) runs are unaffected: the inputs
context is empty there, so they always used the nightly fallback.
* ci: manage backlog issues by signal instead of per-run filing (#7680)
* ci: manage backlog issues by signal instead of per-run filing
The suite workflows used to file one backlog issue per failed run
(dedup was by run ID, which never matched), so issues accumulated
without bound. Replace the inline filing step in every suite workflow
(s3, kms, tier, storage, heal, pool, security, replication, upgrade,
performance) with a single call to
auto-testing/scripts/issue_manager.py, which:
- dedups by signal: failing cases are searched among open issues by
label (suite category + case ID); covered cases become a coalesced
comment on the existing issue, only uncovered cases file a new one
- labels new issues with functional-test, the suite category, one
label per failing case ID (lazily created), and env for
bootstrap-class failures (no cases ran, wholesale failure, or
404/ssh/clone/dpkg signatures in the log)
- closes open issues of the suite after a fully green run, citing the
run as evidence; cancelled runs never file or close anything
The step is skipped cleanly when auto-testing (private checkout) does
not contain the manager, or when PF_TESTING_GH_TOKEN is unset.
* fix(ci): satisfy actionlint and workflow contract tests for the manager step
- heal and performance workflows have no rustfs_version dispatch input;
referencing `${{ inputs.rustfs_version }}` in the manager step failed
actionlint's expression type check. Their package source now resolves
from package_url with the nightly fallback.
- scripts/test_security_workflow.py pinned the removed inline filing
step. The wiring assertions now pin the manager step (manager path +
per-suite report argument), and the evidence/stale-file tests assert
the skip contract instead: without the private auto-testing checkout
present, the step exits 0, publishes nothing, and leaves stale
evidence untouched.
Verified locally: actionlint clean, shellcheck clean,
test_security_workflow.py 21/21.
* fix(storage): alias legacy meta bucket over internode rpc (#7685)
* fix(storage): alias legacy meta bucket over internode rpc
Retry read-only internode RPC metadata access from legacy .minio.sys to .rustfs.sys when mixed-version peers report missing metadata during rolling upgrades.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): target durable ILM receipt quorum fixture
Use the actual durable ILM receipt object path when taking target disks offline so the test exercises receipt write quorum instead of whichever set owns the source record path.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): stabilize cancelled rename test (#7684)
* test(scanner): report Linux descriptor drift ledger (#7692)
Add a read-only descriptor ledger mode to the Scanner/Heal Linux evidence planner so release operators can separate current-head measured descriptors from old-head measured artifacts and case-level inputs before final bundle assembly.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): retry recovery readback SlowDownRead (#7695)
Allow the Scanner/Heal interruption oracle to retry transient retryable GET failures after replacement recovery has converged. The readback still verifies exact object bytes and keeps a bounded timeout, so permanently unreadable objects continue to fail the case.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(pool): default the node-endpoints fallback to the 4-node fleet (#7696)
RUSTFS_POOL_NODE_ENDPOINTS has no secret/var configured, so the suite
ran with the workflow's inline 3-endpoint fallback and the explicit
--node-endpoints flag overrode the script default fixed in
rustfs/auto-testing#61 — every dispatch died at startup with 'must
provide at least 4 direct node endpoints' (run 34677538650). Add
rustfs-node4 to the fallback; explicit secret/var still wins.
* fix(scanner): require ABBA collector credentials (#7700)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): label G14 recovery read failures (#7701)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): preserve settings in token status (#7691)
* fix(heal): preserve settings in token status
* fix(heal): group status response context
* fix(heal): decode partial status settings
* ci: green runs for case failures; red only for harness breakdowns (#7698)
A failing product case used to turn the whole workflow red, so the run
conclusion carried no signal beyond 'something failed' and the report
was suppressed. New semantics across the functional suites:
- Suite steps run with continue-on-error: the outcome is still recorded
for the report and the backlog issue manager (security/tier already
carried the flag).
- Generate report always publishes the full per-case table plus a
'Product result: N passed, M failed' summary, and its exit gate is
harness health: red only when the suite never reached case level (no
case verdicts), failed wholesale (zero passes, >=3 failures), or was
cancelled/skipped. performance is unchanged (parked).
- tier's structured gate no longer fails on case failures; it keeps red
for evidence-init and missing-gate-result breakdowns.
- pool/performance keep their existing red sources (install/benchmark).
Workflow contract tests updated to the new exit semantics: the security
report matrix keys green off the suite outcome, the evidence matrix
expects green for failure outcomes with recorded case rows (except
performance), the heal staged-rerun block expects the per-step table to
always publish, and run steps are now required to carry
continue-on-error.
Verified locally: actionlint clean; test_security_workflow.py 21/21.
* test(heal): retry G14 deferred outage put (#7703)
Treat SlowDownRead as a bounded retryable deferred PUT response after the target pool rejoins, and label terminal deferred outage PUT failures with stage context.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): retry G14 outage put probes (#7704)
Treat SlowDownRead as a retryable outage PUT probe response in the G14 multi-pool runner, and label terminal candidate failures with stage context.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): remove unused G14 put helper (#7705)
Remove the superseded ServiceUnavailable-only PUT helper after the G14 outage PUT probe switched to the shared retry classifier.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(tables): authorize reserved metadata reads (#7672)
* ci: harden report steps after first full new-semantics run (#7708)
First full serial pass with the green-on-case-failure semantics
(run 34693745171 / 34695021651) exposed three report-layer defects:
security: the report step referenced LOG_FILE, which is undefined in
this workflow (set -u killed the step before writing report.md), and
the verdict greps could not match the ANSI-escaped [PASS]/[FAIL] tags
in the real suite log. Point it at the artifacts suite.log, allow any
number of color escapes before the verdict tag, and count [SKIP]
lines separately (45 passed, 6 failed, 3 skipped was reported as an
unbound-variable crash).
pool: warp is stopped early (SIGINT) at the storage threshold and
only writes its final report on a clean exit, so an empty warp.log is
the expected shape of a healthy run - require its presence, not its
size. A mid-script die() abort or a FAIL step verdict must also turn
the validator red now that the run step is continue-on-error.
tier: add the standard 'Product result: N passed, M failed' summary
line computed from the case table, matching the other suites.
The contract test fixture previously injected LOG_FILE into the
environment and wrote verdict lines without ANSI escapes, which hid
both real-world defects; the fixture now mirrors the real suite
(stdout+tee with color tags) and asserts the pass/fail/skip counters.
* ci: tolerate ANSI reset escapes in security verdicts; fix tier product counts (#7712)
The overnight run after #7708 proved the security verdict greps still
counted zero: real verdict lines are '\e[1;31m[FAIL]\e[0m STS-105 ...'
— the reset escape sits between the tag and the case id, and the
pattern only tolerated escapes before the tag. Allow escapes on both
sides; the fixture now emits the reset too, mirroring the real suite.
The tier case table is produced by rustfs_tier_report.py and its rows
lead with the topology column, so the case-ID-first table grep matched
nothing ('Product result: 0 passed, 0 failed'). Parse the PASS/FAIL
counts from the '## Case Summary' bullets the report always emits,
falling back to a topology-aware table grep.
* fix(heal): reject overlapping admin heal owners (#7723)
* fix(heal): reject invalid admin selectors before admission (#7724)
* fix(heal): pass endpoints to overlap receipt test (#7738)
Reuse the heal-control endpoint fixture in the overlap receipt regression so the test matches the updated execution helper signature and selector validation boundary.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: hehutu <hehutu@gmail.com>
* fix(heal): verify inline shard bitrot during deep scans (#7731)
* fix(heal): verify inline shard bitrot during deep scans
* test(heal): fix CI lint and topology fixtures
* fix(heal): persist and retry partial-write repairs (#7729)
* fix(heal): persist and retry partial-write repairs
* test(heal): pass topology to overlap RPC tests
Use the existing coordinator endpoint fixture for the three overlap-test
calls to the endpoint-aware heal control executor. This repairs the E0061
test-build failure inherited from the release base.
* fix(scanner): prove dirty ACK after confirmed root write (#7732)
Allow a confirmed scanner root data-usage CAS write to prove its own publication when the follow-up root readback cannot provide a proof. Keep AlreadyDurable and all stale or companion paths on the existing readback-only proof boundary.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): rebuild truncated xl.meta from healthy quorum (#7730)
* fix(heal): rebuild truncated xl.meta from healthy quorum
* fix(test): pass topology to heal overlap RPC regression
* fix(test): drive heal admission alongside partial PUT
Poll the partial PUT and its mock heal receiver together, bound their handshake, and retain the existing repair-scope assertions.
* fix(test): prepare durable MRF fixtures and Linux heal stack
* fix(test): drive tier cleanup recovery after deferred attempts
---------
Co-authored-by: Hauser <housemecn@gmail.com>
* fix(heal): prove completed historical version cleanup (#7736)
* fix(heal): prove completed historical version cleanup
* test(heal): use debug runtime stack for C06 regression
* test(ecstore): acknowledge partial PUT heal admission concurrently
* test(mrf): fix journal fault and ownership fixtures
* fix(tables): retain dropped warehouse protection (#7675)
* fix(ecstore): gate shard integrity and preserve legacy recovery (#7741)
* fix(ecstore)!: bind bitrot shards to immutable part identities
Verify part, coding-index, and block identity across write, GET, and Heal
paths. Preserve identities across metadata-only copies and repair, include
them in multipart quorum selection, and require payload proof for receipts.
Keep legacy decoding with conservative parity and target-digest validation,
and document its unsupported cases and additional verification I/O.
BREAKING CHANGE: New bound-v1 shards require compatible readers throughout
the fleet. Legacy objects without sufficient integrity evidence return an
error; binary rollback after new writes requires verified data migration.
Refs: rustfs/backlog#2497
* fix(ecstore): preserve shard framing with independent integrity
Commit immutable part-generation Merkle roots and replicated proof indexes without changing existing checksum frames. Verify reads, reconstruction and Deep Heal against metadata quorum; keep legacy reads and explicitly defer unproven legacy data repair.
Preserve multipart rollback generations, require acknowledged durable index publication, and add decoder compatibility and donor-shard regression coverage.
* fix(heal): verify protected partial-write replay
* fix(heal): retire stale delete markers after bucket recreation (#7743)
* fix(heal): prove completed historical version cleanup
* test(heal): use debug runtime stack for C06 regression
* fix(heal): retire stale delete markers after bucket recreation
* test(ecstore): fix Clippy in retired marker regressions
* fix(heal): preserve historical null version identity (#7749)
* fix(s3): preserve version identities on suspended reads (#7751)
* fix(rebalance): retry contended runtime metadata access (#7752)
* fix(rebalance): retry contended runtime metadata access
* chore(rebalance): shrink typed error ratchet baseline
* fix(heal): bind bucket recovery to its original incarnation (#7744)
* fix(heal): bind bucket recovery to its original incarnation
* fix(heal): preserve stable bucket heal task errors
* fix(s3): classify copy source part read failures (#7746)
Map PartMissingOrCorrupt to SlowDownRead only at the GetObject/CopyObject source-reader boundary so quota metadata corruption keeps its internal fail-closed response.
Add store and e2e coverage for Harbor-style multipart staging CopyObject boundaries.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(tier): classify invalid remote credentials (#7753)
* fix(tier): classify invalid remote credentials
Classify deterministic remote-tier auth rejections before the probe cleanup fallback, and preserve the admin 4xx status for invalid credential responses.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): stabilize delete-marker movement regressions
Ignore target-local bucket incarnation fence metadata when comparing data-movement delete-marker identities, keep the release null-version listing fixture valid, and fix the nextest decommission-family filter so fault-hook tests actually join the serial group.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): serialize tier cleanup owner matrix (#7765)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): run bootstrap usage rebuild promptly (#7758)
Treat non-authoritative usage floor startup as pending bootstrap rebuild work so reset-published bootstrap markers cannot sit behind clean-idle or empty pause-backlog delay.
Wire recovery wakeups into the normal scanner cycle wait and expose the pending rebuild state in scanner status.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(tier): preserve cleanup scheduling for overwrites (#7766)
Queue committed tier free-version cleanup receipts for PUT and materialized CopyObject overwrites of transitioned null versions, while keeping remote deletion behind the existing persisted free-version cleanup path.
Tighten data-movement delete-marker retry equivalence by ignoring local bucket-incarnation fencing metadata, avoid retry fallback to the source pool, and keep version-list pagination from manufacturing an empty final page.
Refresh the e2e-distributed selector hash for the current release test set.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(release): backport main fixes and stabilize tier cleanup tests (#7793)
* fix(ecstore): stop pruning at nonempty directories (#7616)
* fix(ecstore): stop pruning at nonempty directories
* test(ecstore): release pruning fixtures before temp cleanup
(cherry picked from commit 8f150d1d8e)
* fix(heal): preserve retryable batch failures during recovery (#7642)
* fix(heal): preserve retryable batch failures during recovery
* test(heal): pin prebuilt hooks binaries in ci
(cherry picked from commit 5cd58319ed)
* fix(s3): reject oversize single PUT early and map body errors to 4xx (#7635)
* fix(s3): reject oversize single PUT early and map body errors to 4xx
A single PutObject above the 5 GiB single-request ceiling was only
rejected after the client had streamed 5 GiB into s3s's read-time body
budget, and the resulting BodySizeLimitExceeded surfaced from the erasure
writer as 500 InternalError. A body whose connection hit EOF before
Content-Length bytes arrived (hyper's IncompleteBody) was also a 500.
SDKs retry 500s, so one oversize upload was resent from offset 0 five
times.
- PutObject and UploadPart reject a declared length above
MAX_SINGLE_PUT_OBJECT_SIZE with 400 EntityTooLarge before reading the
body; the constant moves to rustfs_config so the s3s limit and the
admission check share one value.
- ApiError maps BodySizeLimitExceeded to EntityTooLarge and a hyper body
EOF to IncompleteBody across both io::Error conversions.
Fixes#7596.
* test(s3): cover UploadPart admission, aws-chunked length, real s3s limit
- Poll-counting test body proves PutObject and UploadPart reject a
declared size above the ceiling with zero body polls; exact-cap and
zero-length parts pass admission.
- A STREAMING-* aws-chunked PUT whose framed Content-Length exceeds the
cap is admitted when the decoded length is within it and rejected when
the decoded length is over it.
- The display-based BodySizeLimitExceeded matcher is checked against the
real error produced by the pinned s3s Body budget.
(cherry picked from commit 50b31bc75b)
* fix(ecstore): make directory mtime fixture portable (#7623)
* fix(ecstore): make directory mtime fixture portable
* style(ecstore): format mtime fixture assertion
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
(cherry picked from commit b1cc286cac)
* fix(storage): prevent readiness after native migration failures (#7652)
* fix(storage): prevent readiness after native migration failures
* fix(storage): skip unsupported IAM records before reading
* fix(storage): use stable typed migration metadata errors
* fix(storage): include migration record in startup errors
* test(storage): cover native migration startup failures
* test(storage): use array chunks in migration fixture
---------
Co-authored-by: RJ Regenold <214054+rjregenold@users.noreply.github.com>
Co-authored-by: cxymds <cxymds@gmail.com>
(cherry picked from commit 0cbc3ffe61)
* fix(admin): expose OIDC account display fields (#7654)
Expose verified OIDC username and email claims as display-only metadata on self-account responses while preserving the virtual parent as the authorization identity.\n\nKeep rustfs-madmin public response structs unchanged by adding the optional wire fields through private handler response wrappers.
(cherry picked from commit f02bc947cd)
* fix(replication): correct peer joins and remote-state reporting (#7650)
* fix(replication): propagate verified peer deployment identities
* fix(replication): report actual remote peer state
* fix(replication): defer initial sync until all peers join
* test(replication): shut down TLS fixtures cleanly
---------
Co-authored-by: houseme <housemecn@gmail.com>
(cherry picked from commit 853ae63b6a)
* fix(s3): bound stalled UploadPart request bodies (#7659)
* fix(s3): bound stalled UploadPart request bodies
* fix(ci): preserve the S3S footprint ratchet
(cherry picked from commit 666dfd9f9f)
* fix(tables): reject reserved warehouse locations (#7671)
Co-authored-by: cxymds <cxymds@gmail.com>
(cherry picked from commit 414176c47f)
* fix(ci): bind nightly lanes to one resolved source (#7688)
(cherry picked from commit 01d8e4347f)
* fix(replication): close the pre-stable convergence gaps from backlog#2367 (#7626)
* fix(replication): total-order rule sort and honor V1 top-level Prefix
Rule matching had two defects from the pre-GA replication audit
(rustfs/backlog#2367 C-1 and C-2):
- The actionable-rule sort compared same-destination rules by priority but
answered Equal for any other pair, which is not a total order; the
standard library sort panics on such comparators once a slice exceeds the
insertion-sort threshold, so an object matching more than 20 enabled rules
across two or more targets could panic the PUT or DELETE task. Rules now
sort by priority descending with destination and id as tie-breakers, and
filter_target_arns preserves that order instead of draining a HashSet.
- A V1 rule written without a <Filter> carries its prefix at the top level;
that field was never read, so <Prefix>logs/</Prefix> matched every object.
ReplicationRuleExt::prefix now falls back to it, with a <Filter> keeping
precedence. The existing prefix fixtures were built this way and had been
asserting nothing.
* fix(admin): advertise data-usage and listen capabilities to rc
The rc client gated `rc du` and `rc watch` on a pinned contract that
matched server versions by the string prefix `1.0.0-rc.`; a server that
reports `1.0.0` no longer matches, and the dynamic `advertised` list did
not carry either name, so `rc du` against a GA server fails with an
unsupported-capability error (rustfs/backlog#2367 E-2).
Advertise `admin.data-usage` from the admin route inventory like the IAM
entries, and `listen_notification` for the bucket `?events=` extension
route the admin router dispatches. The client merges advertised entries
ahead of its pinned contract, so no version sniffing is needed.
* fix(site-replication): stop notifying the local site on remove and rotate
The pending-remove and pending-rotation notification loops skipped the
local site by endpoint only, while finalization identifies it by
deployment id or endpoint. The reconcile tick resolves the local peer from
the node's own listen address (and a handler from the request Host), so
`remove --all` dialed the site's registered endpoint, waited out the
request timeout against the lifecycle lock it was holding, and answered
`Partial: failed to notify 1 peer(s)` for a removal that had succeeded
(rustfs/backlog#2367 A-4, backlog#2195 item 3).
Both loops now iterate the peers still awaiting notification through one
helper that applies the finalization identity.
* fix(site-replication): promote and settle IAM retries without a tick of slack
Two retry-queue behaviours kept an IAM change from converging for ten to
twenty minutes after a peer came back (rustfs/backlog#2367 A-1 and A-3,
backlog#2305):
- The lightweight 30-second pass filtered its reachability probe to bucket
ops, so a backed-off IAM or bucket-metadata snapshot waited for the
600-second tick to notice the peer. It now probes every backed-off class
and still replays only bounded bucket ops; promotion is a state flip the
heavyweight tick acts on.
- Backoffs are multiples of the tick interval, so a failure stamped δ
seconds after a tick was 600 − δ old at the next tick and slipped a whole
extra interval. The heavyweight drain now evaluates backoff halfway to its
next tick.
- An IAM entry first created by a non-deletion failure (the add bootstrap's
snapshot send, the drain's own replay, an import-iam schedule) was never
stamped `deletions_recorded`, so a later recorded deletion could not
settle it and it escalated to the marker only `replicate repair` clears.
Entries created by this binary now start recorded; a row persisted by an
older binary keeps the escalation semantics.
* fix(site-replication): reload peer node caches after bucket wiring writes
Every S3 bucket-config write ends by asking the other nodes of the cluster
to reload the bucket's metadata; the site-replication writers never did.
On a multi-node site the node that ran the pairing (or applied a peer's
bucket-meta item) rewrote the bucket targets and the derived replication
rules on disk, while every other node kept serving its cached copy for up
to the 15-minute refresh. A `resync start` routed to such a node reported
every freshly wired bucket as `Config not found` and a bucket whose
operator target the pairing had replaced as `recorded remote target no
longer exists` (rustfs/backlog#2367 A-5, backlog#2195 item 2; functional
SITE-105).
Add one best-effort reload helper in the site-replication hooks and call it
after the bucket setup, versioning, peer bucket-meta apply, removed-peer
cleanup, make-with-versioning, and endpoint-refresh writes; the ensure
helpers now report whether they wrote so unchanged passes stay silent. The
resync manifest and start now read the persisted wiring instead of the
node-local cache, matching the target read the start path already did.
The new four-node e2e pairs two clusters and starts a resync through a
non-coordinator node right after pairing; it also covers an IAM user
created on a non-coordinator node converging to the peer site.
* test(e2e): cover delete-marker replication from a multi-node source
The functional suite reported delete markers created on a 3-node source
never reaching the target (rustfs/backlog#2195 item 4, REP-105). The report
was a probe defect, but the shape had no coverage: the existing
delete-marker e2e runs a single-node source. Pin it against a four-node
source replicating to a four-node peer and to a single-node target, with
the write and the delete issued through different nodes.
* ci(e2e): refresh the distributed selection for the new replication cases
Four distributed cases were added (two site-replication, two delete-marker
replication). The linux digest is derived from the last CI listing of the
lane (34 cases, matching the previous pin) plus the four new names; the
darwin digest is the local listing, which selects the same 38 cases.
(cherry picked from commit aeaba86d73)
* fix(e2e): require a verified server binary for every e2e run (#7687)
* fix(ci): share quick checks and lint workflows
* fix(ci): install actionlint from its verified release
* fix(ci): reject dependencies on required quick checks
* feat(test): verify the E2E server build and source identity
* test(e2e): register verified Darwin test membership
* test(e2e): record verified Linux receipt test membership
* test(e2e): record compiled Darwin receipt test membership
* test(e2e): record compiled Linux receipt test membership
* test(e2e): record Darwin e2e-full membership after merging main
* fix(test): route scanner/heal evidence E2E runs through the verified server binary
The evidence runners built rustfs with plain cargo and then ran e2e_test directly, which now fails without a run receipt. They build through scripts/e2e_binary.py and run the e2e_test invocations under e2e_binary.py run; the obsolete rustfs.features stamp is removed.
* docs(e2e): run server-backed e2e commands through the verified binary wrapper
* test(e2e): record Linux e2e-full membership from the branch CI listing
(cherry picked from commit 2909b1bfe1)
* fix(kms): classify KMS/SSE error contracts and SSE-S3 headers (#7697)
* fix(sse): classify bare SSE-KMS writes when no KMS is available
A `aws:kms` request without a key id, on a bucket without a default key,
returned `500 InternalError` whenever no KMS service was running: the
"no KMS key available" branch exited with an untyped storage error before
the availability classification that the keyed form already received.
Route that branch through the same split: `503 ServiceUnavailable` while
a configured KMS is stopped, `400 InvalidRequest` when KMS was never
configured, and `400 InvalidRequest` naming the missing key id when a
running KMS has no default key. `CreateMultipartUpload` shares the path.
Adds a unit test for the bare form and an e2e module that stops KMS
through the admin API, runs a master-key-only node, and runs a Local KMS
without a default key; refreshes the e2e-full selection digests.
(cherry picked from commit c3259dadc3d603a9185a5b0ad9f83dfb884e61c8)
* fix(sse): keep KMS error classes on the encrypted read path
GetObject, CopyObject and UploadPartCopy on an SSE-KMS object whose key
no longer exists answered `500 InternalError` ("KMS key not found") while
PutObject under the same key already answered `400 KMS.NotFoundException`.
The read path carries its classification through ecstore's
`EncryptionResolutionErrorKind`, which had no kind for a missing key, a
denied KMS grant or a missing backend capability, so all three folded
onto `DecryptionFailed` and the S3 layer reported an internal fault.
Add `KeyNotFound`, `AccessDenied` and `NotImplemented` kinds, map them on
both sides of the boundary, and give an envelope the configured backend
cannot unwrap a diagnosable message while keeping its `500`.
Unit tests cover the kind round trip and the reader wrapping; a new e2e
test deletes a key immediately and checks GET/Copy return 400 with
`KMS.NotFoundException` while HEAD stays 200. The e2e-full selection
digests are refreshed from the current listing (the previous digests
predated the delete-authorization tests) and the e2e `create_default_key`
helper is updated to the accepted `EncryptDecrypt` spelling.
(cherry picked from commit 2523a9814e97caea318d4ff1a51bef3a4d4445b2)
* fix(kms): classify key-management errors on the admin routes
`POST /kms/keys`, the legacy `create-key` alias and `generate-data-key`
reported every backend refusal as `500`: a blank key name (which each
backend failed on differently, the Local backend by writing a key file
with an empty stem), a name already taken, an unknown key, a disabled key
and a capability the backend lacks. `delete` and the lifecycle routes
already classified the same errors.
Refuse a blank or whitespace name in `KmsManager::create_key` before any
backend sees it, and share one `KmsError` to status mapping across
create, delete and generate-data-key (400 for validation and key state,
404 for an unknown key, 409 for a taken name, 501 for a missing
capability, 500 only for damaged material). The XML-error routes carry
the same status explicitly since s3s derives none for a custom code.
The read-only Static backend now reports create, delete and
cancel-deletion as `UnsupportedCapability`, matching its rotate and
enable/disable answers, so the admin API returns 501 for all of them.
(cherry picked from commit e33cac5493c4d9d6662e0d2980b58ba2b24a6d1b)
* fix(sse): stop SSE-S3 responses from naming the wrapping KMS key
`x-amz-server-side-encryption-aws-kms-key-id` is defined for `aws:kms`
objects only, but PutObject, CopyObject, CreateMultipartUpload and
GetObject returned it for `AES256` objects too, carrying the KMS key that
wraps the SSE-S3 data key (the service default, or the literal `default`
on a node without KMS). The write paths copied `kms_key_id` from the
encryption material unconditionally, and the single-decrypt GET
classification did the same after resolving the key for authorization.
Add `EncryptionMaterial::response_kms_key_id`, which yields the id only
for SSE-KMS, use it at the four write-response sites, and gate the GET
classification the same way. CompleteMultipartUpload and HeadObject
already omitted the header.
Unit tests pin both directions; a new e2e test covers Put/Get/Head/Copy
and CreateMultipartUpload for AES256 with an aws:kms control. The
e2e-full selection digests are refreshed from the current listing.
(cherry picked from commit 29d793a63352b0b60fd53c565e80fdbede8964bb)
* fix(s3): validate PutBucketEncryption rules before storing them
A default-encryption rule naming an unknown `SSEAlgorithm` (for example
`AES128`), a rule without `ApplyServerSideEncryptionByDefault`, an empty
rule list, or a `KMSMasterKeyID` on an `AES256` rule was stored as
written: the only algorithm check on the route decided whether to fill
in the default KMS key. `GetBucketEncryption` then advertised that
configuration while the write path encrypted header-less writes under
its `AES256` fallback, so the bucket's declared and actual schemes
disagreed. Two comments claimed the route already refused unknown
algorithms.
Validate the configuration before any of it is applied: `MalformedXML`
for a malformed rule set or unknown algorithm, `InvalidArgument` for a
key id on a non-KMS rule, and nothing stored on refusal. Correct the two
comments to describe when the AES256 fallback is still reachable.
Unit tests cover every refusal and the accepted shapes; an e2e test
checks the refusals leave the previous configuration in place. The
e2e-full selection digests are refreshed from the current listing.
(cherry picked from commit 29e4486dce41197ed93f5253cdbabc57d27a4ddb)
* test(e2e): refresh e2e-full selection for the combined KMS/SSE fixes
* test: align two unit tests with the new KMS and bucket-encryption contracts
`scheduled_deletion_carries_a_deadline_and_can_be_cancelled` still
expects the state error (`InvalidOperation`) for cancelling a key that
is not pending deletion; only the Static backend's mutations moved to
`UnsupportedCapability`. The uninitialized-store PutBucketEncryption
test now sends a well-formed AES256 rule so it reaches the store lookup
instead of the new configuration validation.
(cherry picked from commit e2e6a2535a)
* fix(site-replication): keep an operator's bucket-level target to a peer instead of taking it over (#7709)
* fix(site-replication): keep an operator's bucket-level target to a peer instead of taking it over
Site replication wired each bucket by looking for an existing replication
target "to the same peer" and rewriting the first match in place as its own
same-name target. An operator's bucket-level target that happened to point
at that site (different target bucket, operator credentials) was the first
match whenever it pre-dated the join, and the reconciler repeats the pass
every 600s, so the takeover also depended on target order afterwards. The
operator's rule then named an ARN no target backed and their bucket
replication stopped silently, while the inherited bucket-level reset id
made every site resync report the bucket as owned by another resync
(rustfs/backlog#2479, rustfs/backlog#2489).
Follow MinIO's `getRemoteARN` / `getRemoteARNForPeer` shape instead:
- Wiring updates a target in place only under the same ARN, or when it is
recognisably the site's own under an older ARN shape (same peer,
same-name target bucket, site replication service account). Anything
else gets the site target added next to it.
- The site resync manifest takes the target the derived
`site-repl-<deployment id>` rule names (same-name shape as fallback), so
an operator target to the peer neither aborts the bucket as "multiple
remote targets matched peer" nor gets resynced into.
- Peer removal prunes only targets a pruned derived rule names or the
same-name target bucket; operator targets stamped with the peer's
deployment id survive together with their rules.
Unit tests cover the three predicates. e2e
`test_site_replication_keeps_operator_bucket_target_to_peer` runs a
bucket-level replication plus `replication-reset` to the future peer, joins
the sites, and requires the operator target untouched, both paths
delivering, the site resync completing against the site target, and the
operator target and rule surviving `replicate remove --all`; without the
fix it fails at the join with the operator target gone. The repl-nightly
selection digest is refreshed for the new case.
* test(site-replication): drop a redundant clone flagged by clippy
The reconcile unit test cloned the remote peer into the state map although
the binding is not used afterwards; workspace clippy (-D warnings) rejects
that as redundant_clone.
(cherry picked from commit ecdc55fa4b)
* fix: enforce S3 permissions for recursive force deletion (#7661)
* fix: enforce S3 authorization for recursive deletion
* fix: satisfy the s3s footprint guard
* fix: restore list versions policy compatibility (#7686)
(cherry picked from commit 3fd1ce414d)
* fix(ci): repair functional defaults and chain regression checks (#7664)
* fix(ci): default functional suites to nightly packages
* test(ci): follow the fault-tolerance chain handoff
(cherry picked from commit 509a0fa90c)
* fix(ci): align security workflow tests with chain (#7679)
(cherry picked from commit d9e47d2813)
* test(ecstore): keep tier cleanup tests stable after immediate receipt queueing
Release PR #7766 made PUT/CopyObject overwrites queue the tier free-version cleanup receipt immediately, which broke two ecstore tests on release CI. In tier_overwrite_put_and_self_copy_recover_persisted_cleanup_owners the restarted store already runs expiry workers from the second iteration on, so they deleted the remote bytes before the test could assert that the commit leaves them in place; the test now fails the first remote DELETE via set_remove_failure(true) so the cleanup owner stays durable and the later restart still has to rediscover it from xl.meta (the failed remove does not bump remove_count, and the test re-enables removes before the recovery wait). In batch_transitioned_delete_post_commit_failures_roll_back_without_free_version_receipt the convergence loop now treats a transient InsufficientReadQuorum as "not yet converged", because cleanup rewrites xl.meta disk by disk and a racing read can briefly miss quorum (seen in the rio-v2 lane); any other error still panics.
* Revert "fix(ci): align security workflow tests with chain (#7679)"
This reverts commit 4544359f6d.
* Revert "fix(ci): repair functional defaults and chain regression checks (#7664)"
This reverts commit 5ba7ec0291.
* fix(ecstore): pass shard integrity to backported ingest-mode test
The stalled-reader test backported with #7659 used main's four-argument encode_with_ingest_mode, but release's signature takes an optional IntegrityBuilder, so pass None to keep the test focused on ingest-mode cleanup.
---------
Co-authored-by: Henry Guo <marshawcoco@gmail.com>
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: RJ Regenold <rregenold@teamraft.com>
Co-authored-by: RJ Regenold <214054+rjregenold@users.noreply.github.com>
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: GatewayJ <835269233@qq.com>
Co-authored-by: Jason Kossis <jkossis@gmail.com>
* fix(ci): restore release test compilation and e2e membership (#7799)
* fix(s3): return stored content language on local GET (#7755)
* fix(s3): return resolved object tagging version identities (#7757)
* fix(tables): bound warehouse index recovery (#7677)
Co-authored-by: Chris <anzhengchao@gmail.com>
* fix(upgrade): probe legacy IAM namespace before migration (#7816)
* fix(release): backport IAM migration startup fixes and recent main fixes (#7825)
* fix(auth): reject unsigned x-amz headers on header-signed SigV4 requests (release) (#7813)
* fix(s3): preserve missing version errors on object reads (#7800)
* fix(s3): preserve missing version errors on object reads
* test(s3): align missing-version assertions with read contract
* fix(heal): bind admin traversal to bucket incarnation (#7802)
* fix(heal): bind admin traversal to bucket incarnation
* fix(ecstore): reopen healthy hedged readers after peer loss
* test(ecstore): isolate tier overwrite recovery scheduling
* test(ecstore): observe tier cleanup under object read locks
* fix(ecstore): bound decommission entry tracing
* test(ecstore): drain incarnation heal fixture writes
* test(heal): match debug server stack in deep heal fixtures
* test(heal): include identities in C06 count failures
* test(heal): drain PUT tails before inspecting B920 fixtures
* test(scanner): isolate retained MRF retry slots
* fix(ecstore): separate PUT cleanup intent from persisted metadata
* fix(ecstore): capture failed object lock acquisitions (#7803)
* fix(ecstore): capture failed object lock acquisitions
* test(heal): include identities in C06 count failures
* test(heal): drain PUT tails before inspecting B920 fixtures
* fix(ecstore): reopen healthy hedged readers after peer loss
* test(ecstore): isolate tier overwrite recovery scheduling
* test(ecstore): observe tier cleanup under object read locks
* fix(ecstore): bound decommission entry tracing
* test(ecstore): drain incarnation heal fixture writes
* test(heal): match debug server stack in deep heal fixtures
* test(scanner): isolate retained MRF retry slots
* fix(ecstore): separate PUT cleanup intent from persisted metadata
* fix(heal): retain terminal outcomes across restart (#7805)
* fix(heal): retain terminal outcomes across restart
* test(heal): use octal mode for terminal recovery fixture
* fix(ecstore): reopen healthy hedged readers after peer loss
* test(ecstore): isolate tier overwrite recovery scheduling
* test(ecstore): observe tier cleanup under object read locks
* fix(ecstore): bound decommission entry tracing
* test(ecstore): drain incarnation heal fixture writes
* test(heal): match debug server stack in deep heal fixtures
* test(heal): include identities in C06 count failures
* test(heal): drain PUT tails before inspecting B920 fixtures
* test(scanner): isolate retained MRF retry slots
* fix(ecstore): separate PUT cleanup intent from persisted metadata
* fix(heal): recover replacement ownership across VM restarts (#7810)
* fix(heal): recover replacement ownership across VM restarts
* test(ecstore): isolate tier overwrite recovery scheduling
* test(ecstore): observe tier cleanup under object read locks
* fix(ecstore): bound decommission entry tracing
* test(ecstore): drain incarnation heal fixture writes
* fix(ecstore): reopen healthy hedged readers after peer loss
* test(heal): match debug server stack in deep heal fixtures
* test(heal): include identities in C06 count failures
* test(heal): drain PUT tails before inspecting B920 fixtures
* test(scanner): isolate retained MRF retry slots
* fix(ecstore): separate PUT cleanup intent from persisted metadata
* fix(tables): isolate protected S3 listings (#7676)
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix(ci): restore Linux E2E selection and split offline enrollment (#7843)
* fix(heal): assign stable identities to scanner repair requests (#7823)
* fix(heal): assign stable identities to scanner repair requests
* test(s3): isolate UploadPart inactivity timing
* fix(heal): backport pool metadata scope to release (#7846)
* fix(heal): scope replacement pool metadata to its owning set
(cherry picked from commit a72a75569c)
* fix(heal): use domain imports in pool metadata regression test
(cherry picked from commit e8c5448e21)
* test(heal): set pool metadata regression recursion limit
---------
Co-authored-by: Hiroaki KAWAI <1468181+hkwi@users.noreply.github.com>
* fix(heal): persist pool/set outcomes and recovery scope (#7839)
* fix(heal): persist pool/set outcomes and recovery scope
* test(heal): verify selector receipts and process-crash recovery
* fix(heal): scope replacement pool metadata to its owning set
(cherry picked from commit a72a75569c)
* fix(heal): use domain imports in pool metadata regression test
(cherry picked from commit e8c5448e21)
* test(heal): set pool metadata regression recursion limit
---------
Co-authored-by: Hiroaki KAWAI <1468181+hkwi@users.noreply.github.com>
* fix(ecstore): scope recursive walks to object prefixes (#7850)
* fix(heal): scope nested walks and reject admin read-repair (#7851)
* fix(heal): scope nested walks and reject admin read-repair
* fix(ci): allocate stack for heal metadata regression
* test: cover overlapped multipart staging copy (#7865)
Add Harbor-style E2E coverage for a CopyObject racing with CompleteMultipartUpload. The test allows the copy to observe an unpublished source as NoSuchKey, but rejects 5xx leakage and proves retrying the completed source remains readable.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(release): repair acceptance checks and prepare 1.0.0 (#7866)
* test(e2e): stabilize release acceptance checkpoints
* chore(release): prepare 1.0.0
* test(scanner): stabilize W13 release evidence gates (#7870)
Add the object-only completion diagnostic case and make the evidence runner resolve CARGO_TARGET_DIR before recording the rustfs binary path.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(kms): cover non-default Vault Transit paths (#7657)
Backport the Vault coverage from 4d31621f0a. Keep the release cleanup polls synchronized with object read locks.
* fix(replication): count a bodiless 405 as a replicated delete marker (#7756)
* fix(replication): accept a bodiless 405 as a replicated delete marker during resync
A resync verifies each delete marker with `HEAD ?versionId=<marker>` on the
target. S3 targets (RustFS, MinIO, AWS) answer that with 405 and no body,
and the SDK only synthesizes an error code for 404, so the error arrived
with `code() == None`. `is_retryable_delete_replication_head_error` then
treated it as an ambiguous failure: every delete marker counted as a failed
object with `target service error`, and a site resync over any bucket that
holds a delete marker reported the whole bucket as failed
(rustfs/backlog#2479, SITE-105 on rc.6 and nightly).
Use the raw HTTP status the way the 404 path already does: a 405 without a
code confirms the marker propagated. Ambiguous statuses still fail.
- Unit tests drive `verify_resync_head_result` against a scripted target
answering 405 (accepted) and 503 (still failed).
- e2e `test_site_replication_resync_replicates_delete_marker` joins two
sites, converges a live object and a delete marker, and requires the site
resync to complete with zero failed objects; the repl-nightly selection
digest is refreshed for the new case.
* fix(replication): verify marker-version purges by absence during resync
A delete-marker resync entry with a pending or failed version purge asks
the target to remove the marker, so the bodiless 405 that proves a
created marker propagated proves the purge did not happen. Accept the
405-as-success mapping only for marker creation (empty
version_purge_status); a purge counts as replicated only when the target
answers not found, and any other HEAD outcome stays failed. Unit tests
drive both purge statuses against the 405 fixture and the 404 fixture.
(cherry picked from commit 1ef3974bad)
* fix(kms): refuse key ids that leave the key prefix on the Vault backends (#7727)
Only the Local backend refused a key identifier containing a path
separator. On Vault KV2 a create with the name bad/name succeeded and
produced a nested KV2 path that the listing then reported as a
directory rather than a key, and an identifier containing .. addressed
a record outside the configured key prefix once the HTTP client
normalised the URL; Vault Transit built its transit key name and its
metadata path from the same unchecked identifier.
Lift the Local backend's containment rule into a shared segment check
(empty, /, backslash, NUL, . and ..) and apply it at the single point
where each backend turns the identifier into a path or a Transit key
name, so create, describe, encrypt, delete and the metadata writes all
refuse with InvalidKey before any request reaches Vault. The admin API
already maps that to 400. The AWS backend is untouched: it addresses
keys by ARN and alias, both of which contain /.
Refs rustfs/backlog#2474 (KMS-213 CreateNegatives on vault-kv2).
Co-authored-by: Hauser <housemecn@gmail.com>
(cherry picked from commit 2f2e775655)
* fix: resume GET body after peer short EOF (#7852)
Backport GET recovery from afc8861fec. The Connect TLS classification change requires the main-only proxy transport layer.
* fix(s3): honor presigned UploadPart checksum queries (#7748)
* fix(s3): honor presigned UploadPart checksum queries
* fix(s3): keep checksum errors within gateway boundary
---------
Co-authored-by: Hauser <housemecn@gmail.com>
(cherry picked from commit a8ace89516)
* fix(table-catalog): map transient lock failures to unavailable (#7841)
(cherry picked from commit 5c73a3d602)
* fix(ecstore): preserve ready read errors during encoding (#7798)
Backport the erasure fix from c95b65a7fc while retaining release shard integrity support. Dependency upgrades and the main-only Connect proxy and diagnostic fixes are excluded.
* ci: add the S3 Tables (Iceberg REST Catalog) functional suite workflow (#7873)
New workflow rustfs-table-test.yml following the shared suite contract:
Initialize functional evidence (FUNCTIONAL_ARTIFACTS_DIR/LOG_FILE/
REPORT_FILE/TMPDIR), checkout for the report parser, auto-testing
clone with retry, continue-on-error on the run step, report generation
through scripts/functional_case_report.py with the product gate (case
failures stay green; harness breakdowns turn red), dashboard upload
under functional-reports/table/, backlog issue manager wiring
(--suite table --category table), artifact upload (report.md, cases.md,
suite.log), before/after environment cleanup, and the chain handoff
step (repository_dispatch only, next: pool expansion).
The suite script itself lives in rustfs/auto-testing (TBL-101..107,
paired PR). The run step passes the product smoke script path
(scripts/table-catalog/pyiceberg_smoke.py) via --smoke-script and the
log via --log-file, matching the other suites.
Contract tests: the table suite is registered in the generic evidence
and workflow matrices (JOBS/DIRECT_TESTS/allowlist) plus a dedicated
TableSuiteTests wiring check. Full run: 21/21.
* test(ecstore): tolerate slow read-repair diagnostics CI (#7892)
* fix(heal): report verified normal shard repairs (#7890)
* fix(heal): retry interrupted internode shard writes
(cherry picked from commit 0bd75f47b6326edd8167ee43b8776f6276692e92)
* fix(deps): update rustls for RUSTSEC-2026-0285
(cherry picked from commit 938e1259d13da7832d85ac5c066a59ab3a28bc63)
* test(heal): wait for replacement disk and coordinator readiness
(cherry picked from commit feeaa6f053aac4b7c8e32477c0ab5ebbd3dcdff3)
* fix(heal): retain RPC transport failures across metadata probes
(cherry picked from commit f48ec9f0c6e37b1f941620c666a9357c4b892d68)
* fix(s3): resume GET bodies after erasure read quorum loss
(cherry picked from commit a38e859905)
* test(heal): keep outage objects in the replacement erasure set
(cherry picked from commit a669f50a366e35505e7745f8b680d8ecf1aa31be)
* fix(rpc): retry authenticated requests after peer epoch changes
Backport the stale-epoch recovery fix from main follow-up #7889. Keep invalid signatures and nonce replays terminal.
* test(rpc): assert retryable rejection of stale boot epochs
(cherry picked from commit 1c4b39b99)
* ci: bound concurrent heal clusters in the full E2E gate
Apply the #7889 scheduling fix while preserving release test membership and zero retries.
* fix(ecstore): hide pending purge entries from listings (#7895)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): scope pending purge listing regression (#7897)
Keep the store-level pending purge listing regression focused on the ordinary recursive path and exact-prefix shortcut. The deeper delimiter CommonPrefix case is a separate physical-walker boundary and should not leave the release test suite red.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(obs): expose scanner usage freshness metrics (#7896)
Add Prometheus scanner metrics for dirty-usage pending buckets, usage-cache save/publication timestamps, publication result codes, and deferred publication state. This gives operators a direct scrape path for confirming scanner dirty ACK behavior without relying on admin JSON.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): backport delete quorum rollback to release (#7902)
* ci: rotate the shared functional-tests concurrency group to -v2 (#7900)
* fix(s3): render null write version IDs consistently (#7904)
* fix(heal): quarantine corrupt root recovery records (#7905)
* fix(heal): quarantine corrupt root recovery records
Keep invalid root-heal records durable while isolating them from healthy recovery owners. Serialize startup replay with administrator admission and fence quarantined task IDs so independent compensation heals can proceed safely.
* fix(ci): align release workflow contract tests
* fix: satisfy root recovery clippy checks
* fix(heal): fence canonical repair receipts (#7909)
* fix(heal): fence canonical repair receipts
* fix(heal): preserve unversioned receipt identity
* fix(heal): persist replacement intent on remote survivor (#7912)
* chore(release): merge main into release (#7913)
* feat(nightly): publish packages as assets of the rolling 'nightly' release (sync from release) (#7593)
feat(nightly): publish packages as assets of the rolling 'nightly' release (#7592)
Replace the assets-branch scheme with a proper GitHub Release on
rustfs/auto-testing: a single 'nightly' release whose deb/rpm assets
are replaced in place on every build. This is the standard channel —
visible on the repo's Releases page, stable download URLs, no git
history growth (release assets live outside the repository).
- New scripts/release/publish_nightly_assets.sh: resolves-or-creates
the 'nightly' release via the REST API, deletes same-name assets,
uploads rustfs-nightly-latest.{deb,rpm}, then PATCHes the release
body with the build provenance (ref@sha, run link, sizes, SHA256).
Plain curl + python3, no gh CLI (the build fleet has none — #7586).
- The workflow step shrinks to invoking the script; full flow
exercised end-to-end against the real release with probe files
(create / upload / overwrite / download round-trip / body update).
* fix(ecstore): stop pruning at nonempty directories (#7616)
* fix(ecstore): stop pruning at nonempty directories
* test(ecstore): release pruning fixtures before temp cleanup
* fix(replication): close the pre-stable convergence gaps from backlog#2367 (#7626)
* fix(replication): total-order rule sort and honor V1 top-level Prefix
Rule matching had two defects from the pre-GA replication audit
(rustfs/backlog#2367 C-1 and C-2):
- The actionable-rule sort compared same-destination rules by priority but
answered Equal for any other pair, which is not a total order; the
standard library sort panics on such comparators once a slice exceeds the
insertion-sort threshold, so an object matching more than 20 enabled rules
across two or more targets could panic the PUT or DELETE task. Rules now
sort by priority descending with destination and id as tie-breakers, and
filter_target_arns preserves that order instead of draining a HashSet.
- A V1 rule written without a <Filter> carries its prefix at the top level;
that field was never read, so <Prefix>logs/</Prefix> matched every object.
ReplicationRuleExt::prefix now falls back to it, with a <Filter> keeping
precedence. The existing prefix fixtures were built this way and had been
asserting nothing.
* fix(admin): advertise data-usage and listen capabilities to rc
The rc client gated `rc du` and `rc watch` on a pinned contract that
matched server versions by the string prefix `1.0.0-rc.`; a server that
reports `1.0.0` no longer matches, and the dynamic `advertised` list did
not carry either name, so `rc du` against a GA server fails with an
unsupported-capability error (rustfs/backlog#2367 E-2).
Advertise `admin.data-usage` from the admin route inventory like the IAM
entries, and `listen_notification` for the bucket `?events=` extension
route the admin router dispatches. The client merges advertised entries
ahead of its pinned contract, so no version sniffing is needed.
* fix(site-replication): stop notifying the local site on remove and rotate
The pending-remove and pending-rotation notification loops skipped the
local site by endpoint only, while finalization identifies it by
deployment id or endpoint. The reconcile tick resolves the local peer from
the node's own listen address (and a handler from the request Host), so
`remove --all` dialed the site's registered endpoint, waited out the
request timeout against the lifecycle lock it was holding, and answered
`Partial: failed to notify 1 peer(s)` for a removal that had succeeded
(rustfs/backlog#2367 A-4, backlog#2195 item 3).
Both loops now iterate the peers still awaiting notification through one
helper that applies the finalization identity.
* fix(site-replication): promote and settle IAM retries without a tick of slack
Two retry-queue behaviours kept an IAM change from converging for ten to
twenty minutes after a peer came back (rustfs/backlog#2367 A-1 and A-3,
backlog#2305):
- The lightweight 30-second pass filtered its reachability probe to bucket
ops, so a backed-off IAM or bucket-metadata snapshot waited for the
600-second tick to notice the peer. It now probes every backed-off class
and still replays only bounded bucket ops; promotion is a state flip the
heavyweight tick acts on.
- Backoffs are multiples of the tick interval, so a failure stamped δ
seconds after a tick was 600 − δ old at the next tick and slipped a whole
extra interval. The heavyweight drain now evaluates backoff halfway to its
next tick.
- An IAM entry first created by a non-deletion failure (the add bootstrap's
snapshot send, the drain's own replay, an import-iam schedule) was never
stamped `deletions_recorded`, so a later recorded deletion could not
settle it and it escalated to the marker only `replicate repair` clears.
Entries created by this binary now start recorded; a row persisted by an
older binary keeps the escalation semantics.
* fix(site-replication): reload peer node caches after bucket wiring writes
Every S3 bucket-config write ends by asking the other nodes of the cluster
to reload the bucket's metadata; the site-replication writers never did.
On a multi-node site the node that ran the pairing (or applied a peer's
bucket-meta item) rewrote the bucket targets and the derived replication
rules on disk, while every other node kept serving its cached copy for up
to the 15-minute refresh. A `resync start` routed to such a node reported
every freshly wired bucket as `Config not found` and a bucket whose
operator target the pairing had replaced as `recorded remote target no
longer exists` (rustfs/backlog#2367 A-5, backlog#2195 item 2; functional
SITE-105).
Add one best-effort reload helper in the site-replication hooks and call it
after the bucket setup, versioning, peer bucket-meta apply, removed-peer
cleanup, make-with-versioning, and endpoint-refresh writes; the ensure
helpers now report whether they wrote so unchanged passes stay silent. The
resync manifest and start now read the persisted wiring instead of the
node-local cache, matching the target read the start path already did.
The new four-node e2e pairs two clusters and starts a resync through a
non-coordinator node right after pairing; it also covers an IAM user
created on a non-coordinator node converging to the peer site.
* test(e2e): cover delete-marker replication from a multi-node source
The functional suite reported delete markers created on a 3-node source
never reaching the target (rustfs/backlog#2195 item 4, REP-105). The report
was a probe defect, but the shape had no coverage: the existing
delete-marker e2e runs a single-node source. Pin it against a four-node
source replicating to a four-node peer and to a single-node target, with
the write and the delete issued through different nodes.
* ci(e2e): refresh the distributed selection for the new replication cases
Four distributed cases were added (two site-replication, two delete-marker
replication). The linux digest is derived from the last CI listing of the
lane (34 cases, matching the previous pin) plus the four new names; the
darwin digest is the local listing, which selects the same 38 cases.
* feat(console): support a configurable console URL prefix (#7634)
Co-authored-by: houseme <housemecn@gmail.com>
* chore(release): merge release into main for rc.6 (#7638)
* test(heal): cover MRF manifest CAS legacy transition (#7443)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ci): enable release branch checks and repair test imports (#7441)
* test(scanner): structure heal release evidence lanes (#7442)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): require segment producer identities (#7444)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): expose recovery intent identity (#7445)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): summarize failed heal perf reports (#7446)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(common): retain unmatched MRF repair proofs (#7447)
Add a recorded verified-repair consumer that discharges only exact retained anchors for the requested bucket while leaving unmatched proofs in the event ring. This gives the future durable successor writer a fail-closed primitive before any tombstone or GC path is enabled.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ci): route replication read plan through boundary (#7448)
* test(scanner): cover flat-bucket quantum fairness (#7449)
Add a production-entry scanner cohort regression that combines a wide flat bucket with a small bucket under a fixed object budget. The test keeps partial budgeted rounds unpublished, verifies small buckets are only marked after real execution, and confirms a later unbudgeted round can publish the complete aggregate.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): reuse same admission request id (#7451)
Keep a retried heal start with the same request id from bypassing admission deduplication when the transport replay cache is unavailable but the manager still owns the task.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require perf summary evidence fields (#7453)
Fail closed when measured passing Scanner/Heal ABBA summaries omit W10/W11 foreground pressure, lock wait, or attempt-cost evidence.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind release evidence fields (#7452)
Require explicit scanner/heal release evidence field contracts for the scoped ACK and mixed-version rollback gates.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): tag dirty usage producer identities (#7454)
Record production-facing segment invalidation producer identities when existing object-level dirty usage hooks observe PUT, CopyObject, DeleteObject/DeleteMarker, and CompleteMultipartUpload mutations. Keep the data non-authoritative and process-local so segment reuse activation still requires durable generation-window proof.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): gate release evidence bundles (#7457)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require hard evidence ABBA manifest (#7459)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(heal): add MRF committed snapshot writer (#7458)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover EC8+4 restart shard rebuild (#7461)
Retry heal-control RPCs once after transport auth rejects a stale replay-scope epoch, and add a distributed EC8+4 restart heal evidence case that rebuilds a replaced drive with exact shard/body assertions.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): echo measured release evidence in ABBA fake adapter (#7460)
Keep the synthetic ABBA test adapter aligned with the measured release evidence contract so result validation covers release_evidence drift.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require complete ABBA summary matrix (#7462)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* heal: retain MRF replay journal after accepted/merged replays
Keep startup replay journal until a durable successor snapshot exists after Accepted/Merged admission.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* heal: reclaim superseded MRF snapshots after readback
Add a committed snapshot predecessor cleanup primitive that deletes only an older slot after the successor is read back as the current committed snapshot.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* heal: require exact owner for object receipts
Require object heal receipts to match the expected bucket incarnation before they can be recorded as positive repair evidence. This prevents stale or cross-incarnation receipts from clearing the wrong heal responsibility.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* heal: latch object receipt owner before repair
Capture the expected bucket incarnation before invoking object repair so a post-repair owner change cannot rewrite the responsibility that a storage receipt is allowed to prove.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* heal: replay committed MRF checkpoints durably (#7465)
Prefer committed MRF checkpoints during startup replay, retain accepted replay responsibilities until exact verified repair proofs arrive, and reclaim committed manifests only after discharge.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind hard evidence bundle provenance (#7467)
Require Scanner/Heal release bundle fields to carry source, run, window, timestamp, command, and artifact format provenance before a measured gate can pass.
Keep EC8+4 and performance gate fields tied to a single measurement window so unrelated artifacts cannot be stitched into a release approval.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* heal: reject dry-run object repair receipts
Do not record positive storage repair receipts for dry-run object heal tasks, even if a producer accidentally returns a matching receipt.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* heal: reject cancelled object repair receipts
Do not record positive storage repair receipts once an object heal task has been cancelled, even if the receipt still matches the requested owner and object identity.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): reject failed object repair receipts
Cover the receipt consumer path where a storage repair result carries both an error and a matching positive receipt. The failure may be recorded, but the receipt must not create repaired, healthy, or absent proof.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover bucket object repair receipts (#7468)
Cover bucket and root heal sweeps recording authoritative object outcomes only when storage receipts match the latched bucket incarnation.
Verify unavailable or stale receipt ownership keeps object repair execution intact while leaving canonical outcome proof as Unknown.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add EC8+4 heal restart evidence (#7469)
Register and wire Scanner/Heal background target restart and crash evidence cases for the 3x4 EC8+4 topology.
Validate the observed data/parity geometry in scanner-heal evidence receipts so multi-drive runs cannot satisfy the gate without proving EC8+4 metadata.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* heal: verify replacement pool metadata repair (#7471)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover quorum and mixed repair receipts
Add focused oracles for transient quorum results that carry a matching receipt and for mixed grace plus repaired receipt batches. Only the repaired object may produce positive proof.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* heal: verify admin recreate pool metadata (#7474)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* heal: preserve merged result for duplicate submits
Keep same-request-id replay receipts accepted for the receipt API, but preserve the legacy submit_heal_request duplicate admission result as Merged.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): preserve EC84 restart semantics (#7478)
Keep the EC8+4 background restart lane on the graceful-stop path and assert clean-restart marker absence only for restart scenarios. This prevents the hard evidence gate from silently exercising the crash path when it claims restart coverage.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF snapshot torn successor recovery
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): write EC84 distributed restart oracle (#7481)
Bind the distributed EC8+4 restart evidence lane to a scanner/heal oracle artifact so release validation can consume the real nextest run instead of accepting only a passing test.
Require the registry to assert 8+4 erasure geometry for the three-node, four-drive case.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): harden MRF replay boundaries (#7483)
Reject journal records with unknown version-presence flags even when their CRC is valid, so rollback/future payloads cannot be accepted as known records.
Gate committed checkpoint cleanup by the writer owner captured from the replay source, preserving retained manifests from other owners inside the same sequence window.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): bind MRF manifest CAS dirsync recovery (#7482)
Cover the committed MRF manifest path through LocalDisk conditional CAS when the metadata directory fsync fails. The fixture proves the previous manifest anchor survives rollback, an unanchored first successor is removed, and the legacy MRF journal remains readable even while global durability is relaxed.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): derive evidence runner profile from registry (#7484)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require mixed-version evidence roles (#7485)
Tighten the Scanner/Heal release bundle checker so mixed-version, scoped-ACK, and rollback fields cannot reuse a generic versions list without proving the expected evidence role.
Require version lists to use source revision identities and include the tested source revision. Also require profile evidence fields to name the core profiling artifacts before release approval.
The release gate remains blocked until measured field evidence is present.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): publish committed MRF runtime checkpoints (#7490)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind profile artifacts in release evidence (#7487)
Require Scanner/Heal release bundles to attach every required profiling artifact to P1 profile evidence with relative paths, artifact formats, non-empty files, hashes, and optional per-artifact measurement-window checks.
Document the tightened release bundle profile contract and cover missing, tampered, and mismatched-window profile artifact regressions in the existing checker self-test.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): harden measured ABBA evidence claims (#7491)
Reject measured Scanner/Heal ABBA manifests whose mixed-version evidence uses the same baseline and candidate source revision or binary hash.
Require crash fault modes and profile artifact names to match the exact supported sets, rejecting missing, duplicate, and unknown values.
Update harness fixtures and regression coverage for same-build mixed-version claims and exact-set release evidence fields.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(e2e): group scanner heal evidence payload (#7494)
Group the EC8+4 Scanner/Heal evidence writer inputs into a typed payload so the distributed e2e crate stays within the clippy argument limit without weakening the lint.
The evidence writer still validates the same S3 bodies, physical shard census, process restart PIDs, and node listings.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF replay bundle fields (#7486)
Bind Scanner/Heal release bundle evidence for MRF durable replay to replay counts, retained responsibility anchors, and successor snapshot publication evidence.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): profile EC84 evidence case runs (#7495)
Give Scanner/Heal evidence cases explicit runtime profiles so EC8+4 background restart and crash cases use their own object count, object size, and partial-progress timeout defaults instead of inheriting the legacy 4x1 case assumptions.
Expose the runtime profile in plan-only output and cover every registry case in the script self-test.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): reject empty release evidence artifacts (#7496)
Require Scanner/Heal release bundle artifact paths to resolve to non-empty files before hashing them.
Cover empty hard-gate artifacts in the existing release bundle checker self-test and keep profile artifact size checking on the shared artifact boundary.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require G14 same-window field coverage (#7489)
Tighten the Scanner/Heal release bundle checker so G14 same-window evidence must name the EC8+4, multi-set, and multi-pool fields covered in that measurement window.
Keep the release gate blocked when same-window evidence omits one of the required G14 fields, without changing production runtime behavior.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require two-hour measured ABBA windows (#7493)
Reject measured Scanner/Heal release ABBA manifests and summaries whose evidence window is shorter than the W21 two-hour release requirement.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF idle checkpoint cleanup (#7497)
Add a runtime cleanup regression test that publishes both retained replay and runtime committed checkpoints, writes scoped and legacy journals, and verifies idle cleanup removes every recovery anchor from the registered local disks.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(e2e): ignore untracked files in build identity (#7498)
Align e2e_test build provenance with Scanner/Heal evidence receipts and server binary provenance by treating only tracked source changes as dirty.
This prevents unrelated untracked worktrees or evidence directories from causing compiled test identity mismatches before real evidence cases can run.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF disk-full evidence fields (#7499)
Require the Scanner/Heal release registry and bundle checker to carry explicit G08 MRF capacity, disk-full, and replica-loss evidence fields before release approval can pass.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind release JSON artifact provenance (#7500)
Require scanner/heal release evidence JSON artifacts to repeat their measured source revision, run identity, measurement window, gate, and field identity inside the artifact payload. This keeps a refreshed outer bundle hash from accepting stale summary or profile JSON from another run.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF rollback mirror filtering (#7501)
Add a regression oracle that keeps scoped-only MRF responsibilities in the authoritative runtime snapshot while omitting them from the v1 legacy rollback mirror.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require segment activation evidence (#7504)
* test(scanner): require segment activation evidence
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): cover activation proof inputs
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF crash matrix cases (#7505)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require release bundle domain evidence (#7502)
Reject scanner/heal release bundles that omit field-level domain evidence for scoped ACK, durable intent, mixed-version, scheduler pressure, profile cost, and two-hour pressure lanes.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF retention GC evidence (#7506)
Require P4 retained-responsibility release bundle evidence to list the retained replay anchor and idle cleanup cases, prove a two-hour retention window, and record both idle cleanup and verified-proof discharge observations.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF cleanup GC soak evidence (#7507)
Require the P4 release bundle to carry measured cleanup/GC soak evidence for retained MRF replay responsibilities. The bundle now needs a two-hour cleanup window, exact cleanup case coverage, observed verified idle GC, and zero pending responsibilities or stale journals after GC.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF disk-full commit anchors (#7509)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): report port allocator bind errors (#7510)
Surface the scanned port window, attempt count, and last bind error when the e2e port allocator cannot reserve a localhost port. This keeps Scanner/Heal evidence failures actionable when the environment blocks binds before business assertions run.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(sse): resolve 1.0.0 SSE/KMS blockers and P1 findings (#7511)
* fix(sse): resolve bucket default encryption per request
PUT and the POST-object/extract path resolved a bucket's default
encryption with a hard-coded "no explicit SSE-C" flag, so the default was
layered onto a request that already carried an SSE-C header triple and
then tripped that request's own mutual-exclusion check. Every bucket with
default encryption refused SSE-C single PUTs with 400 InvalidArgument,
while CreateMultipartUpload on the same bucket succeeded because it
resolves SSE elsewhere. Both call sites now derive the flag from the
request headers, as COPY already did.
The bucket default's KMS key id was also inherited independently of the
effective algorithm, so an explicit AES256 request against an aws:kms
default bucket produced a self-contradictory algorithm/key-id pair and
was rejected. The key id is now inherited only when the effective
algorithm is aws:kms, matching the storage-layer resolver.
Refs backlog#2368 B1, B2.
* fix(sse): refuse SSE-KMS without a running KMS service
A write requesting aws:kms on a node with no KMS service fell back to the
node-local SSE-S3 provider: the data key was wrapped with
RUSTFS_SSE_S3_MASTER_KEY while the object metadata still recorded
aws:kms and the requested KMS key id. The stored object claimed a KMS
protection it never had, under a key that was never consulted, and no
signal distinguished it from a genuine SSE-KMS object.
The managed-encryption path now asks the resolved DEK provider whether it
wraps with a node-local master key and refuses SSE-KMS in that case:
InvalidRequest when KMS was never configured, ServiceUnavailable when a
configured service is not running. The check sits after the per-key
authorization gate so an unauthorized caller still receives AccessDenied
whatever the KMS runtime state is, and asks the provider rather than a
parallel availability signal because the provider is what actually wraps
the key. A missing master key no longer answers an SSE-KMS request with
an SSE-S3-worded configuration error.
The SSE-S3 local fallback is unchanged.
Refs backlog#2368 B4.
* fix(ecstore): restore and archive tiers in stored coordinates
Multipart restore addressed the remote tier in plaintext coordinates
while the copy-back reads the stored representation. Each part received a
misaligned slice of the remote object whose length still satisfied the
range, the hash reader and the completion size check, so the restore
reported success and silently replaced the object's bytes. Encrypted and
compressed multipart objects were both affected. Restore now accumulates
stored part sizes, passes the stored length to the hash reader alongside
the plaintext length, and validates against the stored size.
The copy-back digests stored bytes, so its computed MD5 is not the
object's public ETag. Restore now preserves the object ETag on both the
single-part and multipart paths, and gives each restored part its own
recorded part ETag rather than the object-level value.
Transition also handed the tier the object's SSE headers and its
RustFS-wrapped data key as request headers. Any S3 target rejected an
SSE-C archive outright, an SSE-KMS archive asked the target to encrypt a
second time under a key id it does not own, and the wrapped DEK left the
cluster. The archive request now strips every SSE header and encryption
marker with the predicate the replication path already uses; the local
xl.meta keeps all of it, so read-through and restore are unaffected.
Objects restored by an affected release are not detected or repaired
retroactively and must be re-restored from the tier.
Refs backlog#2368 B3, B5; backlog#2369 P7.1.
* fix(rio): lock the v1 nonce layout within a segment
Decrypting a v1 segment tried three historical nonce layouts per frame,
independently for every frame. The last of them exists for streams
written before 1.0.0-alpha.91, which reused a segment's part nonce for
every block in it; because block zero's derived nonce equals that base
nonce, a frame encrypted at index zero authenticated at any position. An
attacker able to rewrite the underlying shards could replay it and have
the forged plaintext returned with 200 and an unchanged length. Shard
integrity uses a keyed-hash-free checksum, which such an attacker can
recompute, so it is not a barrier.
A segment now locks onto whichever layout decoded its first non-zero-index
frame and rejects any later frame needing a different one. That leaves one
residual shape: a stream built purely from repeats of frame zero has no
later frame to disagree. New RUSTFS_ENCRYPTION_LEGACY_NONCE_FALLBACK
(default true, so pre-alpha.91 objects keep decrypting) drops the third
layout entirely when set to false, which closes it. Turning it off refuses
pre-alpha.91 objects, so migrate them first by rewriting in place.
Refs backlog#2369 P2.
* fix(kms): reload a service that failed to start
POST /rustfs/admin/v3/kms/reload short-circuited whenever the persisted
configuration matched the in-memory one byte for byte. A node whose KMS
failed to start keeps that configuration and sits in Error, so the
documented recovery call returned "reloaded successfully" while leaving
the node down. Peers reached the same path through the reload broadcast,
so a cluster that lost Vault during a rolling restart had no working
recovery route other than the node-local start endpoint. Reload now
short-circuits only for a service that is actually running, and otherwise
reconfigures, which starts a service that is not running.
The AWS backend also advertised key-version enumeration through
kms/status, which its own documentation says it cannot do; the capability
and its golden snapshot now say false.
Refs backlog#2369 P1, P7.3.
* docs: record the SSE and KMS changes for 1.0.0
The Unreleased changelog section carried no entry for any encryption work
merged since 1.0.0-rc.5, including three items with operational impact:
the config-secret variable whose absence persists secrets in cleartext
with only a warning, the v2 frame write switch and its rolling-upgrade
constraint, and per-key authorization making a public bucket incompatible
with SSE-KMS objects. Adds those plus this batch, including the SSE-KMS
refusal as a breaking change with both routes out.
Also corrects four places where documentation contradicted the code: the
cleanup register still called encrypted range seek opt-in after its
default flipped, the Helm README claimed vault_mount_path only applies to
Transit while the template also feeds the KV2 mount, the disaster-recovery
drill listed bundle contents for backends whose export is refused with
501, and the Chinese README capability table predated most of the feature
set. Documents the SSE-S3 local master key as a first-class operational
mode with its rotation dead end, and what the v1 frame layout does and
does not authenticate.
Refs backlog#2369 P5.
* fix(kms): classify data-path KMS failures by what the caller can do
Only "key not found" and a backend outage were classified; every other
KMS failure that reached the S3 data path fell through to
500 InternalError with a generic message. A disabled or pending-deletion
key, a denied KMS grant, an encryption-context mismatch, an unsupported
algorithm, a credential or timeout failure, and a capability the
configured backend does not have all looked identical to a server fault.
SDKs therefore applied exponential backoff to configuration errors no
retry can fix, and monitoring counted every one of them against the
server's own error rate.
Unusable-key and request-side failures now answer 400, a denied grant
403, transient backend failures 503, and a missing backend capability
501. Damaged, unreadable, or unknown-format key material keeps its 500:
it is a server-side integrity fault, and existing tests pin it.
The classifier is deliberately separate from the admin lifecycle
mapping, which answers 404 for a missing key because there a key id is
the resource being addressed; on the data path it arrives inside a
request header or a bucket default. Messages either name what the caller
asked for or stay generic, with deployment-side detail left on the error
source the way the storage-IO mapping already does.
Refs backlog#2368 B6.
* fix(kms): track and renew static Vault tokens
Token authentication hard-coded "this token carries no lease", so the
renewal task never started, the remaining-TTL gauge was never published,
and nothing looked wrong. `vault token create` grants a 768-hour TTL by
default, so a cluster that had been healthy for a month turned every KMS
call into a 403 and could not recover without a restart or a
reconfigure. Production configuration validation only rejects the
literal dev-token, so an ordinary expiring token reaches a whole cluster.
The source now reads `auth/token/lookup-self` at login and adopts what
Vault reports. A token with no expiry behaves exactly as before. An
expiring renewable one is picked up by the existing renewal loop and
renewed at half TTL like every other auth method. An expiring
non-renewable one warns with its remaining lifetime and publishes the
gauge, so the fail-closed window is visible before it arrives.
The probe never fails the login: a policy that omits lookup-self, or a
Vault that is briefly unreachable, warns and falls back to exactly the
previous behaviour rather than taking down a deployment that works
today. The scripted Vault test double answers the lookup out of band so
existing scripts keep describing only the protocol under test.
Refs backlog#2369 P3.
* feat(sse): report SSE-C requests that arrive without TLS
An SSE-C request carries the customer's AES key in a request header, so
AWS S3 and MinIO both refuse one that did not arrive over TLS. RustFS
accepted them on any transport: a plaintext hop hands the key to anyone
on the path, and since the object cannot be read without that same key,
the exposure lasts as long as the object does.
Refusing outright is the correct end state but not a safe default to
adopt inside a release window, because the project's own s3-tests and
e2e lanes and most staging deployments speak plain HTTP. This release
reports instead: each such request increments
rustfs_ssec_plaintext_requests_total and logs one warning per process, so
an operator can confirm nothing would break before the default flips.
RUSTFS_SSE_C_REQUIRE_TLS=true opts into the AWS 400 now.
The verdict is per connection rather than per deployment: the layer is
built with whether this listener terminated TLS, and additionally accepts
an https protocol forwarded by a proxy the trusted-proxy configuration
already vetted. It sits beside the rate limiter, after the layer that
makes a forwarded protocol trustworthy and after the request context, so
a rejection can echo the request id.
Refs backlog#2369 P7.2.
* fix(kms): say what a node-local backend means for a cluster
The Local backend keeps key material on each node's own disk and
generates its Argon2id salt per node, so two nodes derive different keys
from the same master_key and an object encrypted on one node cannot be
decrypted on another. Behind a load balancer that surfaces as
intermittent 500s on reads that succeeded moments earlier, with nothing
tying the symptom to the cause: the only signal was a generic
"development, testing and demos only" positioning warning that says
nothing about what actually breaks.
Configuring or reconfiguring Local while the deployment is distributed
now logs a dedicated event and appends the consequence to the configure
response, so the operator who made the change sees it. The product
decision to warn rather than refuse is unchanged.
Refs backlog#2369 P7.4.
* docs: record the remaining SSE and KMS changes for 1.0.0
Adds changelog entries for the KMS data-path status classification, the
Vault static-token lease probe, the SSE-C plaintext-transport report and
its switch, and the node-local backend warning.
Documents two things the backend security guide never stated: that SSE-C
belongs on a secure transport, with the counter and switch to plan the
change around, and that the Local backend cannot be shared by a
multi-node deployment because each node derives different keys from the
same master key.
Refs backlog#2368 B6; backlog#2369 P3, P5, P7.2, P7.4.
* fix(kms): report an unreadable key store as an outage on the S3 path
A backend now distinguishes a key store it could not read from a key
that is genuinely absent, but the S3 boundary collapsed the first one
back onto 500 InternalError through the fallthrough for integrity
faults. The distinction was therefore invisible to the client: a
temporary key-directory outage looked exactly like a permanently damaged
key record, and neither the status nor the metric said the request was
worth retrying.
An unreadable key store joins the retryable class and answers 503, next
to a backend error and a credential failure. Damaged, unreadable or
unknown-format key material keeps its 500.
Refs backlog#2368 B6; builds on rustfs/rustfs#7470.
* test(storage): cover MRF cleanup delete ENOSPC anchor (#7513)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require heal retry stats evidence (#7514)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add release bundle dry-run fixture (#7515)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): confirm recovery intent accept readback (#7516)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF cleanup delete ENOSPC evidence (#7518)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require bounded retry window evidence (#7517)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): emit scanner heal G09 upgrade evidence (#7519)
Record measured Scanner/Heal G09 evidence artifacts from the upgrade compatibility lanes when a fresh evidence directory is provided.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): persist MRF rollback mirror boundary (#7520)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add G09 upgrade evidence runner (#7522)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(ci): bind performance runs to selected inputs (#7512)
* test(scanner): add G09 upgrade evidence runner
Add a reusable Linux x86_64 runner for the Scanner/Heal G09 mixed-version and rollback upgrade evidence lanes.
The helper reads the pinned previous-release asset metadata from the upgrade workflow, verifies the downloaded binary, builds the current head, runs both ignored E2E tests, and fails unless the expected G09 JSON artifacts exist.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): implement V2 evidence preflights (#7523)
* feat(scanner): wire dirty usage producer identities
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add segment activation preflight proof
Keep scanner segment reuse behind a structured activation preflight so release evidence can prove the production gate remains disabled until every producer, generation, overflow, cold-oracle, and distributed invalidation check is satisfied.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): expose distributed invalidation evidence
Record an explicit distributed segment invalidation evidence summary when remote dirty usage snapshots are bound to the current activity window and the authenticated scoped ACK capability probe succeeds.
Reject peer dirty usage snapshots that contradict the peer activity pending bit so scoped ACKs fail closed instead of clearing an unadvertised remote mutation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): assemble release evidence bundles (#7526)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add G09 upgrade evidence runner (#7524)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(ci): bind performance runs to selected inputs (#7512)
* test(scanner): add G09 upgrade evidence runner
Add a reusable Scanner/Heal G09 runner for Linux PR-head validation. The script downloads the pinned previous release, builds the current checkout, runs the mixed-version and rollback upgrade E2E lanes, and validates the measured raw artifacts before they can be consumed by the release bundle gate.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): replay recovery intents while disabled (#7521)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): preflight G09 evidence disk space (#7527)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics…
* fix(list): reconcile split overwrite metadata (#7910)
fix(list): fail closed on split overwrite metadata
* fix(heal): persist explicit delete-marker purges (#7908)
* fix(heal): persist explicit delete-marker purges
* fix(ecstore): remove unused object info wrapper
* test(rpc): use valid incarnation in digest gate
---------
Signed-off-by: Hauser <housemecn@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
* ci(table): pass --outcome to the backlog issue manager (#7914)
The manager step failed with 'the following arguments are required:
--outcome' in the first table run (hidden by continue-on-error): every
other suite workflow passes the test step outcome through.
* fix(list): reconcile encoded directory objects (#7915)
* fix(ci): restore release checks after marker purge changes (#7918)
* fix(rpc): recover tier reloads after peer capability revocation (#7920)
* test(ecstore): drain expiry workers before restart (#7923)
* test: report tier cleanup recovery progress on timeout (#7922)
* fix(heal): resume replacement generation after target restart (#7926)
* ci(release): use sm-standard-4 for Linux builds (#7925)
* feat(release): add workspace cargo publish tooling
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* chore(release): prepare 1.0.1 metadata
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix
* revert
---------
Signed-off-by: houseme <housemecn@gmail.com>
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Hauser <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: cui fliter <imcusg@gmail.com>
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: xiaomage <devops008@sina.com>
Co-authored-by: GatewayJ <835269233@qq.com>
Co-authored-by: hehutu <hehutu@gmail.com>
Co-authored-by: Henry Guo <marshawcoco@gmail.com>
Co-authored-by: RJ Regenold <rregenold@teamraft.com>
Co-authored-by: RJ Regenold <214054+rjregenold@users.noreply.github.com>
Co-authored-by: Jason Kossis <jkossis@gmail.com>
Co-authored-by: Hiroaki KAWAI <1468181+hkwi@users.noreply.github.com>
Co-authored-by: Hiroaki KAWAI <hiroaki.kawai@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Source cleanup treated a retryable conflict such as a lock acquisition
timeout as a permanent cleanup warning. The warning was persisted, mapped
the pool to failed_ignored, and blocked pool completion for the rest of
the run while the source replica stayed on disk. Retryable storage
failures now defer the bucket instead, and only non-retryable failures
keep a permanent warning, so completion still requires a resolved delete
or a successful not-found check.
Recording a bucket deferral no longer overwrites the pool's pending
migration-deferral marker, because that marker is the only signal keeping
a pool from completing at the free-space goal while an entry is retried.
Teach the transient matchers the rendered lock timeout text, unwrap
data-movement stage context before classifying not-found and transient
cleanup failures, and raise the bounded source cleanup deferral budget.
Keep recent writable observations during transient pool metadata inspection timeouts while preserving confirmed write blocks and fail-closed behavior. Consolidate the runtime observation cache into one ordered state machine and expose timeout metrics.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): hide ancestors of delete residue in prefix listings
Since #7342 the never-versioned fast path hides a directory whose children are all deleted-version data dirs, but only that one level: the date-style ancestors above a deleted key (`metrics/<stream>/2026/08/28/23/`) still surfaced as empty folders that nothing could remove, which is what #6898 keeps reporting on rc.6 and 1.0.0. The residue probe now walks down through metadata-less ancestors depth-first, ending at the first file met outside a UUID data dir, with a total read budget so a large residue tree surfaces and is hidden one level down instead of costing an unbounded walk. An empty delimiter listing of the ancestor then reclaims the whole committed residue tree in one pass.
Refs #6898
* fix(ecstore): sample before full reads in the residue probe and purge orphan subtrees independently
Adversarial pass 1 findings: the probe read a wide directory in full at every level of a genuine prefix once its 8-entry batch was full, so a leaf holding thousands of objects was materialised per listed prefix; the probe now descends into the sampled children first and only reads the remainder once every sample proved unlistable, and a test pins one bounded read per level and zero complete reads on a wide genuine prefix. Hiding ancestors also made purge_orphan_dir_object fail closed on a whole ancestor when committed residue shared it with residue an older build left without markers, which 1.0.0 users could still reclaim by browsing one level lower; the purge now blocks only the ancestor chain of unpurgeable data and reclaims purgeable sibling subtrees. A UUID directory holding a subdirectory surfaces again as before.
Refs #6898
* fix(ecstore): back off repeated orphan purge scans of unpurgeable trees
Adversarial pass 2: with ancestors hidden, every empty listing of a phantom prefix scanned the whole residue subtree on every disk even when nothing under it can be purged (no committed marker), and the scan no longer stops at the first blocked directory. Remember such prefixes per set for 60 s and skip the rescan. Build committed residue file paths like directory paths so the blocked filter matches under keys containing repeated slashes.
Refs #6898
This is particularly useful for integration testing of services that use
an S3 API, as it lets many different instances bind a port on different
IPs.
Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
The fault-tolerance suite was the only one without a dashboard upload
step - its reports never reached functional-reports/fault-tolerance/
without manual uploads. Add the standard upload step (continue-on-error,
same shape as the other ten suites), placed before the artifact upload.
Co-authored-by: Hauser <housemecn@gmail.com>
* fix(heal): certify metadata health and marker repair receipts
* fix(heal): match nil selectors to absent null metadata
* test(heal): satisfy clippy for nil selector proof
* test(heal): align normal-scan receipt with nil selector proof
A nil or omitted selector now matches only a nil or absent metadata
version, so a normal scan of the latest UUID version leaves
metadata_verified unset and issues no receipt.
Assert metadata health through the exact latest version id instead, and
pin the omitted-selector path to fail closed.
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Route capacity-owned unversioned multipart uploads through the durable decommission target reservation and fail closed when the reservation lease has expired.
* test(scanner): derive evidence runner profile from registry (#7484)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require mixed-version evidence roles (#7485)
Tighten the Scanner/Heal release bundle checker so mixed-version, scoped-ACK, and rollback fields cannot reuse a generic versions list without proving the expected evidence role.
Require version lists to use source revision identities and include the tested source revision. Also require profile evidence fields to name the core profiling artifacts before release approval.
The release gate remains blocked until measured field evidence is present.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): publish committed MRF runtime checkpoints (#7490)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind profile artifacts in release evidence (#7487)
Require Scanner/Heal release bundles to attach every required profiling artifact to P1 profile evidence with relative paths, artifact formats, non-empty files, hashes, and optional per-artifact measurement-window checks.
Document the tightened release bundle profile contract and cover missing, tampered, and mismatched-window profile artifact regressions in the existing checker self-test.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): harden measured ABBA evidence claims (#7491)
Reject measured Scanner/Heal ABBA manifests whose mixed-version evidence uses the same baseline and candidate source revision or binary hash.
Require crash fault modes and profile artifact names to match the exact supported sets, rejecting missing, duplicate, and unknown values.
Update harness fixtures and regression coverage for same-build mixed-version claims and exact-set release evidence fields.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(e2e): group scanner heal evidence payload (#7494)
Group the EC8+4 Scanner/Heal evidence writer inputs into a typed payload so the distributed e2e crate stays within the clippy argument limit without weakening the lint.
The evidence writer still validates the same S3 bodies, physical shard census, process restart PIDs, and node listings.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF replay bundle fields (#7486)
Bind Scanner/Heal release bundle evidence for MRF durable replay to replay counts, retained responsibility anchors, and successor snapshot publication evidence.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): profile EC84 evidence case runs (#7495)
Give Scanner/Heal evidence cases explicit runtime profiles so EC8+4 background restart and crash cases use their own object count, object size, and partial-progress timeout defaults instead of inheriting the legacy 4x1 case assumptions.
Expose the runtime profile in plan-only output and cover every registry case in the script self-test.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): reject empty release evidence artifacts (#7496)
Require Scanner/Heal release bundle artifact paths to resolve to non-empty files before hashing them.
Cover empty hard-gate artifacts in the existing release bundle checker self-test and keep profile artifact size checking on the shared artifact boundary.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require G14 same-window field coverage (#7489)
Tighten the Scanner/Heal release bundle checker so G14 same-window evidence must name the EC8+4, multi-set, and multi-pool fields covered in that measurement window.
Keep the release gate blocked when same-window evidence omits one of the required G14 fields, without changing production runtime behavior.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require two-hour measured ABBA windows (#7493)
Reject measured Scanner/Heal release ABBA manifests and summaries whose evidence window is shorter than the W21 two-hour release requirement.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF idle checkpoint cleanup (#7497)
Add a runtime cleanup regression test that publishes both retained replay and runtime committed checkpoints, writes scoped and legacy journals, and verifies idle cleanup removes every recovery anchor from the registered local disks.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(e2e): ignore untracked files in build identity (#7498)
Align e2e_test build provenance with Scanner/Heal evidence receipts and server binary provenance by treating only tracked source changes as dirty.
This prevents unrelated untracked worktrees or evidence directories from causing compiled test identity mismatches before real evidence cases can run.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF disk-full evidence fields (#7499)
Require the Scanner/Heal release registry and bundle checker to carry explicit G08 MRF capacity, disk-full, and replica-loss evidence fields before release approval can pass.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind release JSON artifact provenance (#7500)
Require scanner/heal release evidence JSON artifacts to repeat their measured source revision, run identity, measurement window, gate, and field identity inside the artifact payload. This keeps a refreshed outer bundle hash from accepting stale summary or profile JSON from another run.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF rollback mirror filtering (#7501)
Add a regression oracle that keeps scoped-only MRF responsibilities in the authoritative runtime snapshot while omitting them from the v1 legacy rollback mirror.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require segment activation evidence (#7504)
* test(scanner): require segment activation evidence
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): cover activation proof inputs
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF crash matrix cases (#7505)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require release bundle domain evidence (#7502)
Reject scanner/heal release bundles that omit field-level domain evidence for scoped ACK, durable intent, mixed-version, scheduler pressure, profile cost, and two-hour pressure lanes.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF retention GC evidence (#7506)
Require P4 retained-responsibility release bundle evidence to list the retained replay anchor and idle cleanup cases, prove a two-hour retention window, and record both idle cleanup and verified-proof discharge observations.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF cleanup GC soak evidence (#7507)
Require the P4 release bundle to carry measured cleanup/GC soak evidence for retained MRF replay responsibilities. The bundle now needs a two-hour cleanup window, exact cleanup case coverage, observed verified idle GC, and zero pending responsibilities or stale journals after GC.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF disk-full commit anchors (#7509)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): report port allocator bind errors (#7510)
Surface the scanned port window, attempt count, and last bind error when the e2e port allocator cannot reserve a localhost port. This keeps Scanner/Heal evidence failures actionable when the environment blocks binds before business assertions run.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(sse): resolve 1.0.0 SSE/KMS blockers and P1 findings (#7511)
* fix(sse): resolve bucket default encryption per request
PUT and the POST-object/extract path resolved a bucket's default
encryption with a hard-coded "no explicit SSE-C" flag, so the default was
layered onto a request that already carried an SSE-C header triple and
then tripped that request's own mutual-exclusion check. Every bucket with
default encryption refused SSE-C single PUTs with 400 InvalidArgument,
while CreateMultipartUpload on the same bucket succeeded because it
resolves SSE elsewhere. Both call sites now derive the flag from the
request headers, as COPY already did.
The bucket default's KMS key id was also inherited independently of the
effective algorithm, so an explicit AES256 request against an aws:kms
default bucket produced a self-contradictory algorithm/key-id pair and
was rejected. The key id is now inherited only when the effective
algorithm is aws:kms, matching the storage-layer resolver.
Refs backlog#2368 B1, B2.
* fix(sse): refuse SSE-KMS without a running KMS service
A write requesting aws:kms on a node with no KMS service fell back to the
node-local SSE-S3 provider: the data key was wrapped with
RUSTFS_SSE_S3_MASTER_KEY while the object metadata still recorded
aws:kms and the requested KMS key id. The stored object claimed a KMS
protection it never had, under a key that was never consulted, and no
signal distinguished it from a genuine SSE-KMS object.
The managed-encryption path now asks the resolved DEK provider whether it
wraps with a node-local master key and refuses SSE-KMS in that case:
InvalidRequest when KMS was never configured, ServiceUnavailable when a
configured service is not running. The check sits after the per-key
authorization gate so an unauthorized caller still receives AccessDenied
whatever the KMS runtime state is, and asks the provider rather than a
parallel availability signal because the provider is what actually wraps
the key. A missing master key no longer answers an SSE-KMS request with
an SSE-S3-worded configuration error.
The SSE-S3 local fallback is unchanged.
Refs backlog#2368 B4.
* fix(ecstore): restore and archive tiers in stored coordinates
Multipart restore addressed the remote tier in plaintext coordinates
while the copy-back reads the stored representation. Each part received a
misaligned slice of the remote object whose length still satisfied the
range, the hash reader and the completion size check, so the restore
reported success and silently replaced the object's bytes. Encrypted and
compressed multipart objects were both affected. Restore now accumulates
stored part sizes, passes the stored length to the hash reader alongside
the plaintext length, and validates against the stored size.
The copy-back digests stored bytes, so its computed MD5 is not the
object's public ETag. Restore now preserves the object ETag on both the
single-part and multipart paths, and gives each restored part its own
recorded part ETag rather than the object-level value.
Transition also handed the tier the object's SSE headers and its
RustFS-wrapped data key as request headers. Any S3 target rejected an
SSE-C archive outright, an SSE-KMS archive asked the target to encrypt a
second time under a key id it does not own, and the wrapped DEK left the
cluster. The archive request now strips every SSE header and encryption
marker with the predicate the replication path already uses; the local
xl.meta keeps all of it, so read-through and restore are unaffected.
Objects restored by an affected release are not detected or repaired
retroactively and must be re-restored from the tier.
Refs backlog#2368 B3, B5; backlog#2369 P7.1.
* fix(rio): lock the v1 nonce layout within a segment
Decrypting a v1 segment tried three historical nonce layouts per frame,
independently for every frame. The last of them exists for streams
written before 1.0.0-alpha.91, which reused a segment's part nonce for
every block in it; because block zero's derived nonce equals that base
nonce, a frame encrypted at index zero authenticated at any position. An
attacker able to rewrite the underlying shards could replay it and have
the forged plaintext returned with 200 and an unchanged length. Shard
integrity uses a keyed-hash-free checksum, which such an attacker can
recompute, so it is not a barrier.
A segment now locks onto whichever layout decoded its first non-zero-index
frame and rejects any later frame needing a different one. That leaves one
residual shape: a stream built purely from repeats of frame zero has no
later frame to disagree. New RUSTFS_ENCRYPTION_LEGACY_NONCE_FALLBACK
(default true, so pre-alpha.91 objects keep decrypting) drops the third
layout entirely when set to false, which closes it. Turning it off refuses
pre-alpha.91 objects, so migrate them first by rewriting in place.
Refs backlog#2369 P2.
* fix(kms): reload a service that failed to start
POST /rustfs/admin/v3/kms/reload short-circuited whenever the persisted
configuration matched the in-memory one byte for byte. A node whose KMS
failed to start keeps that configuration and sits in Error, so the
documented recovery call returned "reloaded successfully" while leaving
the node down. Peers reached the same path through the reload broadcast,
so a cluster that lost Vault during a rolling restart had no working
recovery route other than the node-local start endpoint. Reload now
short-circuits only for a service that is actually running, and otherwise
reconfigures, which starts a service that is not running.
The AWS backend also advertised key-version enumeration through
kms/status, which its own documentation says it cannot do; the capability
and its golden snapshot now say false.
Refs backlog#2369 P1, P7.3.
* docs: record the SSE and KMS changes for 1.0.0
The Unreleased changelog section carried no entry for any encryption work
merged since 1.0.0-rc.5, including three items with operational impact:
the config-secret variable whose absence persists secrets in cleartext
with only a warning, the v2 frame write switch and its rolling-upgrade
constraint, and per-key authorization making a public bucket incompatible
with SSE-KMS objects. Adds those plus this batch, including the SSE-KMS
refusal as a breaking change with both routes out.
Also corrects four places where documentation contradicted the code: the
cleanup register still called encrypted range seek opt-in after its
default flipped, the Helm README claimed vault_mount_path only applies to
Transit while the template also feeds the KV2 mount, the disaster-recovery
drill listed bundle contents for backends whose export is refused with
501, and the Chinese README capability table predated most of the feature
set. Documents the SSE-S3 local master key as a first-class operational
mode with its rotation dead end, and what the v1 frame layout does and
does not authenticate.
Refs backlog#2369 P5.
* fix(kms): classify data-path KMS failures by what the caller can do
Only "key not found" and a backend outage were classified; every other
KMS failure that reached the S3 data path fell through to
500 InternalError with a generic message. A disabled or pending-deletion
key, a denied KMS grant, an encryption-context mismatch, an unsupported
algorithm, a credential or timeout failure, and a capability the
configured backend does not have all looked identical to a server fault.
SDKs therefore applied exponential backoff to configuration errors no
retry can fix, and monitoring counted every one of them against the
server's own error rate.
Unusable-key and request-side failures now answer 400, a denied grant
403, transient backend failures 503, and a missing backend capability
501. Damaged, unreadable, or unknown-format key material keeps its 500:
it is a server-side integrity fault, and existing tests pin it.
The classifier is deliberately separate from the admin lifecycle
mapping, which answers 404 for a missing key because there a key id is
the resource being addressed; on the data path it arrives inside a
request header or a bucket default. Messages either name what the caller
asked for or stay generic, with deployment-side detail left on the error
source the way the storage-IO mapping already does.
Refs backlog#2368 B6.
* fix(kms): track and renew static Vault tokens
Token authentication hard-coded "this token carries no lease", so the
renewal task never started, the remaining-TTL gauge was never published,
and nothing looked wrong. `vault token create` grants a 768-hour TTL by
default, so a cluster that had been healthy for a month turned every KMS
call into a 403 and could not recover without a restart or a
reconfigure. Production configuration validation only rejects the
literal dev-token, so an ordinary expiring token reaches a whole cluster.
The source now reads `auth/token/lookup-self` at login and adopts what
Vault reports. A token with no expiry behaves exactly as before. An
expiring renewable one is picked up by the existing renewal loop and
renewed at half TTL like every other auth method. An expiring
non-renewable one warns with its remaining lifetime and publishes the
gauge, so the fail-closed window is visible before it arrives.
The probe never fails the login: a policy that omits lookup-self, or a
Vault that is briefly unreachable, warns and falls back to exactly the
previous behaviour rather than taking down a deployment that works
today. The scripted Vault test double answers the lookup out of band so
existing scripts keep describing only the protocol under test.
Refs backlog#2369 P3.
* feat(sse): report SSE-C requests that arrive without TLS
An SSE-C request carries the customer's AES key in a request header, so
AWS S3 and MinIO both refuse one that did not arrive over TLS. RustFS
accepted them on any transport: a plaintext hop hands the key to anyone
on the path, and since the object cannot be read without that same key,
the exposure lasts as long as the object does.
Refusing outright is the correct end state but not a safe default to
adopt inside a release window, because the project's own s3-tests and
e2e lanes and most staging deployments speak plain HTTP. This release
reports instead: each such request increments
rustfs_ssec_plaintext_requests_total and logs one warning per process, so
an operator can confirm nothing would break before the default flips.
RUSTFS_SSE_C_REQUIRE_TLS=true opts into the AWS 400 now.
The verdict is per connection rather than per deployment: the layer is
built with whether this listener terminated TLS, and additionally accepts
an https protocol forwarded by a proxy the trusted-proxy configuration
already vetted. It sits beside the rate limiter, after the layer that
makes a forwarded protocol trustworthy and after the request context, so
a rejection can echo the request id.
Refs backlog#2369 P7.2.
* fix(kms): say what a node-local backend means for a cluster
The Local backend keeps key material on each node's own disk and
generates its Argon2id salt per node, so two nodes derive different keys
from the same master_key and an object encrypted on one node cannot be
decrypted on another. Behind a load balancer that surfaces as
intermittent 500s on reads that succeeded moments earlier, with nothing
tying the symptom to the cause: the only signal was a generic
"development, testing and demos only" positioning warning that says
nothing about what actually breaks.
Configuring or reconfiguring Local while the deployment is distributed
now logs a dedicated event and appends the consequence to the configure
response, so the operator who made the change sees it. The product
decision to warn rather than refuse is unchanged.
Refs backlog#2369 P7.4.
* docs: record the remaining SSE and KMS changes for 1.0.0
Adds changelog entries for the KMS data-path status classification, the
Vault static-token lease probe, the SSE-C plaintext-transport report and
its switch, and the node-local backend warning.
Documents two things the backend security guide never stated: that SSE-C
belongs on a secure transport, with the counter and switch to plan the
change around, and that the Local backend cannot be shared by a
multi-node deployment because each node derives different keys from the
same master key.
Refs backlog#2368 B6; backlog#2369 P3, P5, P7.2, P7.4.
* fix(kms): report an unreadable key store as an outage on the S3 path
A backend now distinguishes a key store it could not read from a key
that is genuinely absent, but the S3 boundary collapsed the first one
back onto 500 InternalError through the fallthrough for integrity
faults. The distinction was therefore invisible to the client: a
temporary key-directory outage looked exactly like a permanently damaged
key record, and neither the status nor the metric said the request was
worth retrying.
An unreadable key store joins the retryable class and answers 503, next
to a backend error and a credential failure. Damaged, unreadable or
unknown-format key material keeps its 500.
Refs backlog#2368 B6; builds on rustfs/rustfs#7470.
* test(storage): cover MRF cleanup delete ENOSPC anchor (#7513)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require heal retry stats evidence (#7514)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add release bundle dry-run fixture (#7515)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): confirm recovery intent accept readback (#7516)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF cleanup delete ENOSPC evidence (#7518)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require bounded retry window evidence (#7517)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): emit scanner heal G09 upgrade evidence (#7519)
Record measured Scanner/Heal G09 evidence artifacts from the upgrade compatibility lanes when a fresh evidence directory is provided.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): persist MRF rollback mirror boundary (#7520)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add G09 upgrade evidence runner (#7522)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(ci): bind performance runs to selected inputs (#7512)
* test(scanner): add G09 upgrade evidence runner
Add a reusable Linux x86_64 runner for the Scanner/Heal G09 mixed-version and rollback upgrade evidence lanes.
The helper reads the pinned previous-release asset metadata from the upgrade workflow, verifies the downloaded binary, builds the current head, runs both ignored E2E tests, and fails unless the expected G09 JSON artifacts exist.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): implement V2 evidence preflights (#7523)
* feat(scanner): wire dirty usage producer identities
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add segment activation preflight proof
Keep scanner segment reuse behind a structured activation preflight so release evidence can prove the production gate remains disabled until every producer, generation, overflow, cold-oracle, and distributed invalidation check is satisfied.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): expose distributed invalidation evidence
Record an explicit distributed segment invalidation evidence summary when remote dirty usage snapshots are bound to the current activity window and the authenticated scoped ACK capability probe succeeds.
Reject peer dirty usage snapshots that contradict the peer activity pending bit so scoped ACKs fail closed instead of clearing an unadvertised remote mutation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): assemble release evidence bundles (#7526)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add G09 upgrade evidence runner (#7524)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(ci): bind performance runs to selected inputs (#7512)
* test(scanner): add G09 upgrade evidence runner
Add a reusable Scanner/Heal G09 runner for Linux PR-head validation. The script downloads the pinned previous release, builds the current checkout, runs the mixed-version and rollback upgrade E2E lanes, and validates the measured raw artifacts before they can be consumed by the release bundle gate.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): replay recovery intents while disabled (#7521)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): preflight G09 evidence disk space (#7527)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(ci): bind performance runs to selected inputs (#7512)
* fix(targets): reject trailing batch items (#7508)
* test(scanner): preflight G09 evidence disk space
Fail the Scanner/Heal G09 upgrade evidence runner before downloading or building when the validation host does not have enough free space for a full raw evidence pass.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: cui fliter <imcusg@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): respect cargo target dir in G09 runner (#7528)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(ci): bind performance runs to selected inputs (#7512)
* fix(targets): reject trailing batch items (#7508)
* fix(scanner): respect cargo target dir in G09 runner
Write the RustFS feature stamp under the resolved Cargo target directory so remote validation hosts with CARGO_TARGET_DIR set can reuse the built PR-head binary.
Extend the runner plan/self-test path to cover relative target-dir resolution.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: cui fliter <imcusg@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): add G09 upgrade evidence runner (#7525)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(ci): bind performance runs to selected inputs (#7512)
* test(e2e): add G09 upgrade evidence runner
Add a Linux x86_64 runner that downloads the pinned previous release, builds the current RustFS binary, runs the mixed-version and rollback upgrade compatibility lanes, and verifies the required Scanner/Heal G09 raw evidence artifacts.
Document the runner and add a shell self-test for help, dry-run, SHA validation, and non-empty artifact directory guards.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): emit G09 release bundle gate evidence (#7530)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(ci): bind performance runs to selected inputs (#7512)
* fix(targets): reject trailing batch items (#7508)
* test(scanner): emit G09 release bundle gate evidence
Write a bundle-ready G09 gate descriptor from the Linux upgrade evidence runner and validate the single G09 gate with the shared release-bundle rules without approving the full Scanner/Heal release.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: cui fliter <imcusg@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): reject invalid G09 runner test selection (#7531)
Validate the selected G09 evidence lane before process-substitution case expansion so --plan-only cannot turn an unknown --test value into an empty successful plan.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): keep G09 source binary path clean (#7532)
Keep checksum verification output out of the command substitution that resolves the previous-release binary for the G09 runner.
Also tolerate non-GNU sha256sum in local self-tests by falling back to shasum when GNU --check support is unavailable.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF replay proof cleanup (#7534)
Exercise the committed MRF replay checkpoint path with a real ECStore-backed bucket incarnation, retain the replay anchor until an exact verified proof arrives, then delete the proof-discharged checkpoint and prove a subsequent restart does not resurrect the replayed work.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): carry segment activation preflight evidence (#7535)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): surface cold segment reuse oracle (#7536)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require hard evidence artifact payloads (#7537)
Require Scanner/Heal release bundle JSON artifacts to carry hard-domain evidence fields for mixed-version, crash, capacity, disk-full, replica-loss, and MRF cleanup gates. This prevents a descriptor from approving a hard gate while pointing at a generic measured artifact that lacks the boundary-specific oracle fields.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): bind producer evidence to dirty generations (#7538)
Record scanner segment producer identities with the dirty usage generation they invalidated, then expose a cycle-local producer evidence snapshot for segment reuse activation preflight.
Production activation remains fail-closed until durable producer identity and restart-gap proof are available.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): persist segment invalidation proof metadata (#7539)
* test(scanner): wait for EC8+4 heal control readiness (#7540)
* test(scanner): list full P4 release evidence fields (#7542)
Align the Scanner/Heal release requirement registry with the release bundle gate for P4 so the closure checklist advertises MRF scale, replay cost, retained responsibility, and cleanup/GC soak evidence together.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add W16 release evidence runner (#7543)
Add a W16 Scanner/Heal evidence runner that executes the recovery-intent crash-boundary and quota-authority lanes, emits measured G04/G12 JSON artifacts, and validates the resulting single-gate release descriptors.
Wire its shell self-test into script-tests and document the release evidence entry point.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix: address release acceptance regressions (#7541)
* feat(scanner): admit durable segment proof replay (#7544)
* feat(scanner): admit durable segment proof evidence
Validate complete set snapshot segment invalidation proof metadata against the current dirty usage generation window and scanner process epoch before clearing durable producer and restart-gap activation blockers.
Production segment reuse remains gated by the explicit activation flag.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): replay segment proof from root snapshots
Carry segment invalidation proof metadata into root snapshot set states so a complete published baseline can replay the durable producer evidence recorded by each set cache.
Keep the field additive for older readers and leave incomplete or LKG set states unproven.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind MRF release artifacts to evidence kinds (#7545)
Require Scanner/Heal G07/G08/P4 JSON artifacts to declare the exact MRF evidence kind expected for each release-bundle field. This keeps generic measured JSON from satisfying durable replay, ENOSPC, replica-loss, scale, replay-cost, retained-responsibility, or cleanup-GC soak gates.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): enable segment reuse activation gate (#7546)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): run scoped fallback cases on dedicated stack (#7550)
Linux libtest defaults can abort these deep EC-store futures before the assertions run. Wrap the async scoped fallback cases in the repository's dedicated-stack test pattern so the default cargo test path exercises the oracle directly.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): roll out distributed e2e selection (#7549)
Refresh the committed e2e-distributed selection digest after the EC8+4 Scanner/Heal evidence case joined the profile, and document the registry-aware runner entry for the distributed case.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add W13 MRF evidence runner (#7547)
Add a W13 durable MRF evidence runner that emits measured G07, G08, and P4 JSON artifacts and release descriptors through the existing scanner/heal bundle gate.
The runner now executes the ignored MRF replay evidence test with an exact full test path, validates raw artifact kinds and gate decisions, prepares Linux tmpfs-backed ENOSPC roots for G08, and documents the Linux/long-soak boundaries.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): fix segment proof fixtures
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): reduce W13 evidence helper arguments
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* ci(test): enable local KMS for s3 compatibility tests (#7553)
Configure the s3-tests harness with a local KMS key so SSE-KMS cases run in CI without relying on an external KMS service.
Also move the anonymous POST default SSE-KMS regression onto the shared local KMS test environment.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bind segment proof to typed dirty suffix (#7564)
Track producer identity completeness per pending bucket and retain process coverage across durable acknowledgements. Admit a prior same-process segment proof only for an exact or strictly earlier fully typed generation window.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): validate scoped ACK clear receipts (#7563)
Fail closed when a signed scoped ACK response claims more cleared entries than its request contained, while preserving partial-clear semantics and zero-mutation capability probes.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bind cold scoped cache reuse (#7565)
Record an explicit cold bucket reuse proof for scoped set scans and treat fully cold sets as valid cold-zero-walk reuse when the baseline and bucket incarnations are bound.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(s3): preserve delete marker read error headers (#7558)
* chore(release): sync configurable nightly build workflow from main (#7567)
feat(nightly): make the build branch configurable (NIGHTLY_BRANCH var + dispatch input) (#7557)
* docs(obs): correct metric export interval and explain alert timing (#7571)
* fix(heal): persist scoped admin admission handoff (#7569)
* fix(heal): persist root admission handoff
Persist administrator root heal ownership before queue publication so crash restart can replay an admitted control-plane start without relying on process-local replay cache state.
Protect queued durable root owners from priority displacement; callers must use forceStart cancellation semantics to replace them.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): persist admin heal handoff types
Extend durable admin heal admission handoff beyond cluster roots while preserving legacy root recovery records. Persist the heal type in schema v2, replay typed admin requests after restart, keep non-admin requests out of the durable control-plane handoff, and allow path cancellation to clear durable-only non-root records.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): preserve disjoint durable admin owners
Limit admin forceStart durable-owner cleanup to overlapping heal requests after the admin recovery journal begins storing non-root work. Add fail-closed and displacement tests so durable bucket owners cannot be silently dropped by unrelated replacement or priority pressure.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): persist admin terminal receipts
Persist terminal receipts for durable administrator heal starts before retiring their pending intent so restart can answer completed or cancelled scoped tasks without replaying them.
Retain pending work when terminal publication fails and keep timeout owners on checkpoint-only replay semantics.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): mark committed replication and tier mutations (#7568)
* fix(scanner): mark committed replication and tier mutations
Bridge ECStore committed replication status writebacks and lifecycle tier expiration cleanup into scanner dirty-usage producer identity tracking. Only successful replication metadata updates mark Replication; validate-only, superseded, and retry paths remain unmarked.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): replay durable dirty producer records
Add a strict scanner dirty-usage producer replay schema that can restore typed producer coverage after restart without treating ordinary process-local mutations as durable evidence. Replay records validate schema, cache key format, generations, scopes, producer identities, duplicate buckets, byte limits, and entry limits before modifying dirty state.
Keep segment activation fail-closed unless the restored bucket state came from durable replay; subsequent local mutations clear the durable authority bit for that bucket. Also fix the release scanner clippy lint in scoped cold-reuse proof creation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): recover expansion buckets and resume retirement (#7574)
* fix(ecstore): recover partially created expansion bucket volumes
* fix(ecstore): resume retirement after startup safety checks
* fix(ci): align S3 KMS fixtures and default key expectations (#7573)
* fix(heal): prune retained root terminal receipts (#7576)
Bound durable root-heal terminal receipt retention to the completed-status TTL while keeping expired receipts authoritative enough to retire stale pending intents before deleting the receipt itself.
Add a budgeted background GC pass and fail closed on corrupt or non-terminal receipt records.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(storage): preserve retryable publication guard errors (#7561)
* fix(storage): preserve retryable publication guard errors
* fix(scanner): satisfy scoped cache clippy lint
Use bool::then_some for the scoped cold bucket reuse proof construction after merging the latest release branch.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(health): report storage quorum in node readiness (#7566)
* fix: preserve scanner backlog during pool retirement (#7555)
* fix: hand off native scanner backlog before pool retirement
* test: initialize optional proof in data usage fixtures
* perf: bound parallel V3 pool metadata phase writes
* test: keep native backlog fault errors typed
* test: skip unchanged pool metadata in crash barriers
---------
Co-authored-by: cxymds <cxymds@gmail.com>
* test(scanner): bind profile artifact provenance (#7577)
* fix(scanner): persist dirty producer journal (#7580)
Persist scanner dirty producer replay records from committed replication and tier mutations, hydrate journal state on restart, and clear durable records only after acknowledged dirty generations.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test: add scanner heal scheduler pressure evidence (#7584)
Add measured G10/P1/P3 scheduler-pressure release descriptor generation, ABBA scheduler/profile provenance, and focused self-tests for the release bundle gate.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test: add scanner heal G14 multiset evidence (#7583)
Add EC8+4 multi-set and multi-pool scanner/heal release evidence coverage, including e2e registry cases, oracle checks, and a G14 descriptor assembler for measured case artifacts.
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(nightly): push assets with plain git on release — the build fleet has no gh CLI (#7586)
fix(nightly): push assets with plain git — the build fleet has no gh CLI
The sm-standard-4 runners used by the nightly build have no gh binary
(functional-chain workflows run elsewhere, on the jumpbox). The assets
publish step died on 'gh: command not found' at the credential-helper
setup, and the preceding clone failure had been masked by 2>/dev/null,
misleading the step into the orphan path. Swap clone and remote setup
to plain git with the token embedded in the URL; push semantics are
unchanged.
* fix(scanner): stabilize retired backlog survivor commits (#7585)
Treat an authoritative scanner pause-backlog commit as stable when every member in that commit stores the matching stable and committed record. Retired source replicas can remain missing in the observed topology while survivor membership is already stable.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(metrics): prevent duplicate drive reporting across nodes (#7582)
Co-authored-by: houseme <housemecn@gmail.com>
* fix(ecstore): avoid Windows batch-delete cleanup sharing conflicts (#7579)
Co-authored-by: houseme <housemecn@gmail.com>
* feat(ilm): inspect legacy transition version state (#7581)
* test(scanner): add release evidence ingestion gates (#7588)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix: reuse admin error boundary for legacy tier reconciliation (#7587)
* fix(test): stabilize release CI heal fixtures (#7589)
Keep heal admission-only tests off durable root recovery so mock storage does not require local metadata disks. Align forceStart retry assertions with replacement semantics and fix the e2e metric clippy collapse.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(nightly): publish packages as assets of the rolling 'nightly' release (#7592)
Replace the assets-branch scheme with a proper GitHub Release on
rustfs/auto-testing: a single 'nightly' release whose deb/rpm assets
are replaced in place on every build. This is the standard channel —
visible on the repo's Releases page, stable download URLs, no git
history growth (release assets live outside the repository).
- New scripts/release/publish_nightly_assets.sh: resolves-or-creates
the 'nightly' release via the REST API, deletes same-name assets,
uploads rustfs-nightly-latest.{deb,rpm}, then PATCHes the release
body with the build provenance (ref@sha, run link, sizes, SHA256).
Plain curl + python3, no gh CLI (the build fleet has none — #7586).
- The workflow step shrinks to invoking the script; full flow
exercised end-to-end against the real release with probe files
(create / upload / overwrite / download round-trip / body update).
* test(scanner): add release bundle evidence descriptors (#7595)
* test(scanner): merge release evidence descriptors
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add authority and legacy release descriptors
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add scoped ack release descriptor
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require all release evidence fields
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add mrf release descriptor producer
Add a standalone W13 MRF descriptor producer for measured G07/G08/P4 raw artifacts, with fail-closed self-tests for missing responsibility cases, disk-full ENOSPC observations, synthetic artifacts, and short P4 cleanup soaks.
Also bind G12 quota path evidence to explicit case lists so W16 descriptors cannot pass without reset and settlement coverage.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): reuse mrf release descriptor producer
Route the W13 runner through the shared MRF descriptor producer so measured runs and offline re-packaging use the same fail-closed validation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(ilm): backfill proven legacy transition state (#7591)
* feat(ilm): backfill proven legacy transition state
* fix(ilm): remove redundant reconcile test clone
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): remove redundant transition clone (#7597)
Drop an unnecessary clone in the legacy transition state digest regression so release CI clippy lanes can run with warnings denied.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): seed current backlog members before committing (#7594)
* fix(scanner): seed current backlog members before committing
* test(app): run recursive force delete test on large stack
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind raw profile artifacts (#7598)
Require Scanner/Heal release profile wrappers to carry bundled raw profile artifacts plus the matching profile cost metrics.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): surface first failed object in status (#7599)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover retry budget across attempts (#7600)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): harden durable producer evidence gate (#7601)
Require Scanner/Heal G11 maintenance evidence to prove durable dirty producer journal replay, replay-state hydration, generation-bound ACK clearing, and invalid journal fail-closed cases before segment reuse can be activated.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): harden scoped ack evidence proof (#7602)
Require Scanner/Heal G03 scoped ACK evidence fields to carry their own measured provenance and concrete ACK, capability, and mixed-peer observations before release-bundle gate verification.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind status outcome raw provenance (#7603)
Require measured G05/G06/R-D raw status, compatibility, and disposition artifacts to share run identity before producing release bundle descriptors.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): retry fully recoverable batch failures (#7604)
Allow recursive bucket and cluster heal attempts whose retained batch failure is entirely retryable to use the existing bounded manager retry budget. Permanent or mixed batch failures remain terminal so heal outcome semantics stay strict.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): avoid movement debt for stale remote lease (#7605)
Classify a final remote scanner publication lease validation failure from a fresh activity snapshot instead of treating every validation error as data movement. If the peer session and movement generation still match the granted leases and publication is otherwise allowed, keep the publication rejected as an activity-baseline miss without creating pause backlog movement debt. Preserve DataMovement for expired leases, peer restarts, movement generation changes, and active publication blocks.
Tests cover namespace-only validation invalidation and the remote fence-loss cases that must still defer as DataMovement.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* chore(deps): refresh workspace dependencies (#7606)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind G14 case artifacts (#7611)
Require Scanner/Heal G14 release bundle JSON wrappers to mirror their outer evidence and carry self-contained case artifact provenance.
Copy proof-json case artifacts into the generated G14 descriptor bundle so assembled release bundles can validate case file hashes after relocation.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): retry contended decommission cancel target locks (#7612)
* test: align transition fixtures and KMS context expectations (#7610)
* test(scanner): stabilize W13 release gate evidence (#7613)
* feat(nightly): publish packages as assets of the rolling 'nightly' release (sync from release) (#7593)
feat(nightly): publish packages as assets of the rolling 'nightly' release (#7592)
Replace the assets-branch scheme with a proper GitHub Release on
rustfs/auto-testing: a single 'nightly' release whose deb/rpm assets
are replaced in place on every build. This is the standard channel —
visible on the repo's Releases page, stable download URLs, no git
history growth (release assets live outside the repository).
- New scripts/release/publish_nightly_assets.sh: resolves-or-creates
the 'nightly' release via the REST API, deletes same-name assets,
uploads rustfs-nightly-latest.{deb,rpm}, then PATCHes the release
body with the build provenance (ref@sha, run link, sizes, SHA256).
Plain curl + python3, no gh CLI (the build fleet has none — #7586).
- The workflow step shrinks to invoking the script; full flow
exercised end-to-end against the real release with probe files
(create / upload / overwrite / download round-trip / body update).
* test(scanner): stabilize W13 release gate evidence
Treat only real raw-entry windows as replayed scanner enumeration in the restart diagnostic, so final completion rounds without raw entries are not fail-closed as raw replays.
Allow the EC8:4 multi-pool heal evidence case to select an outage object key that routes to an online pool while an entire target pool is down, preserving strict behavior for single-pool cases.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bound native backlog fixture resources (#7609)
* fix(scanner): bound native backlog fixture resources
Keep native scanner backlog restart fixtures within low file descriptor limits by reducing the native-only disk layout, making fixture shutdown drain background work, and avoiding long-lived ECStore retention from background loops.
Convert ECStore-backed background refresh/recovery/monitor tasks to upgrade weak owners only while doing work so completed test stores release their disk graph before the next native fixture starts.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(tier): refresh release-merged failure fixtures
Reuse the raw legacy-transition fixture helper so delete-all expiry actually exercises Unknown transitioned history metadata after the release merge.
Rewrite restore-failure disk fixtures through per-disk xl.meta snapshots so stale destination identities can be persisted without tripping ordinary metadata update guards.
Update the SSE KMS mismatch expectation to the InvalidRequest/context_mismatch behavior now returned by the rio-v2 path.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ci): align release E2E gate and repair regression tests (#7607)
* fix(scanner): harden heal release gates (#7614)
* test(scanner): reject synthetic heal evidence inputs (#7617)
Harden the Scanner/Heal checkpoint restart and G14 EC evidence producers so release descriptors cannot be assembled from marked fixture, dry-run, or synthetic case inputs.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): collect status outcome raw evidence (#7618)
Normalize live Scanner/Heal status/outcome observations into the measured raw artifacts consumed by the G05/G06/R-D release descriptor producer.
Reject synthetic observations, incomplete required cases, and reused run/window identities before raw artifacts can enter the release bundle flow.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bind scoped ack confirmation to generation (#7619)
Require lost scoped dirty-usage ACK reconciliation to observe a clean peer activity generation that covers the requested ACK generation. Apply the same guard in the scanner aggregation path so a stale clean activity snapshot cannot discharge pending maintenance after an uncertain ACK response.
Refs rustfs/backlog#2427
Refs rustfs/backlog#2281
Refs rustfs/backlog#2240
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test: report tier reference proof setup failures without hanging (#7608)
* test(scanner): plan Linux evidence validation (#7620)
Add a Scanner/Heal Linux release-evidence planner that emits a machine-readable execution manifest for the remaining measured validation lanes.
The planner can run only lightweight preflight checks and keeps plan-only output distinct from measured release evidence.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): select G14 replacement drive by evidence (#7622)
Choose the replacement disk from the target node by the presence of complete pool metadata instead of assuming the first configured drive is the scanner metadata holder. This keeps the G14 multi-set harness aligned with multi-drive EC layouts and adds clearer diagnostics when multi-pool outage writes fail closed.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): report Linux evidence plan status (#7624)
Add a status mode for the Scanner/Heal Linux evidence planner so operators can identify missing or incomplete artifacts before final bundle assembly.
The status output stays plan-only and never reports release approval.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add G14 multi-pool background heal evidence (#7628)
test(scanner): align G14 multi-pool outage evidence
Treat the localhost multi-pool topology as whole-pool loss when the target node is down. If that topology cannot admit the outage object while the pool is offline, defer that object write until the pool rejoins and record the oracle marker instead of failing before the real crash/restart evidence runs.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): verify deferred multipool outage evidence (#7629)
Validate the G14 multi-pool oracle field that records down-window outage PUT refusal and requires the deferred post-rejoin outage object to be accepted and checked through S3.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): allow metadata snapshots at read quorum (#7627)
fix(ecstore): admit metadata snapshots at read quorum
Allow guarded bucket metadata snapshot existence checks to use read quorum so degraded erasure sets can continue Object Lock snapshot reads without weakening bucket mutation or object write quorum.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* chore(deps): refresh release dependencies (#7632)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): use pool lock quorum for set writes (#7630)
Use the pool-wide namespace lock client domain for every set in a pool so degraded EC8+4 multi-set writes are gated by node-level lock quorum instead of the narrower per-set endpoint host slice.
Keep namespace-lock domain deduplication tied to both the pool namespace and shared clients, and add regression coverage for three-locker degraded writes plus cross-pool domain separation.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): keep G14 multi-set restart graceful (#7633)
test(e2e): keep multi-set restart graceful
Include the EC8+4 multi-set restart scenario in the graceful interruption lane so the unclean-shutdown marker assertion matches the scenario semantics.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): record deferred G14 outage writes (#7637)
Include optional outage-write diagnostics in G14 evidence oracles so deferred multi-pool outage writes bind their down-window refusals and post-rejoin acceptance.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bound G14 multi-pool evidence size (#7640)
Keep the multi-pool evidence runner within the registry object budget while retaining deferred outage-write diagnostics for release-gate validation.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): find nextest junit fallback for evidence cases (#7643)
Handle cargo-nextest writing JUnit reports under the workspace target directory even when the Rust build uses CARGO_TARGET_DIR. This keeps measured Scanner/Heal evidence cases from passing the real test but failing final receipt packaging.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ilm): preserve cleanup ownership on tiered overwrites (#7639)
* fix(ilm): preserve cleanup ownership on tiered overwrites
* fix(ci): refresh E2E selection for tier overwrite regression
* test(ilm): restore noncurrent compensation tests to serial CI (#7645)
* fix(heal): preserve null-version tags and encoded object paths (#7644)
* fix(e2e): record EC8+4 drive restart set size (#7648)
* fix(e2e): record EC8+4 drive restart set size
Include the erasure set drive count in the distributed EC8+4 drive restart oracle so Scanner/Heal release evidence matches the registry contract.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(s3): tighten s3s footprint ratchet
Replace release-merge s3_error! macro calls with equivalent S3Error constructors so the s3gate migration ratchet does not grow on the PR merge tree.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): target multi-set outage heal candidate (#7653)
* test(e2e): target multi-set outage heal candidate
Require the outage write used by EC8+4 multi-set root-heal evidence to miss the same erasure index owned by the selected replacement drive. This avoids accepting a candidate from a different set and turning a valid heal into a false negative.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(e2e): satisfy G14 heal lint gates
Remove clippy-only noise from the G14 multi-set heal evidence test and align the admin route policy inventory with the registered heal catch-all route.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* update
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): reuse API error boundary for decoding failures (#7647)
* fix(heal): reuse API error boundary for decoding failures
* Update heal.rs
Signed-off-by: houseme <housemecn@gmail.com>
* test(ecstore): synchronize batch cleanup metadata reads
Acquire object read locks while observing unversioned and explicit-version batch cleanup, then release them before waiting for progress. This prevents snapshots from spanning per-disk marker removal while retaining the existing quorum, timeout, and remote delete count assertions.
Validation: cargo fmt --all --check and git diff --check passed. The focused nextest test passed 20 stress iterations each with test-util and test-util,rio-v2, with retries disabled.
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
* fix(scanner): wait for ABBA telemetry window (#7660)
Size the scanner collector wait budget from the requested sample count and interval instead of using a fixed 120 second timeout. This preserves the full duration/60+1 telemetry sample set for measured ABBA runs.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(ci): add fault-tolerance degradation suite to the functional chain
Adds a fault-tolerance suite that verifies read/write behavior under
drive and node loss against the erasure-coding contract and snapshots
health-endpoint responses at every degradation tier. Scenarios are
derived from product source (default_parity_count, erasure set sizing):
- A: single-node 4 drives (EC:2, read quorum 2): hide 1/2/3 drives
- B: multi-node 4x1 (one set of 4, EC:2): stop 1/2/3 nodes
- C: multi-node 4x4 (one set of 16, EC:4, read quorum 12): 1 node down
lands exactly on the read-quorum boundary; 2 nodes down breaks it
- C2: multi-node 4x4 with RUSTFS_STORAGE_CLASS_STANDARD=EC:8 (read
quorum 8, write quorum 9, lock majority 9): 2 nodes down puts reads
inside the reported divergence window (read quorum met while the
lock majority is broken)
By default a reads-refused-despite-met-read-quorum observation is
reported as known-divergence without failing the suite; the strict
input escalates it. Chain order becomes:
upgrade -> s3 -> kms -> tier -> storage -> heal -> pool -> security ->
replication -> fault-tolerance -> performance.
Covers the findings of the 2026-09 external degradation report
(reads at read quorum, health-endpoint readiness truthfulness,
degradedReasons capture) as automated regression probes.
* Revert "feat(ci): add fault-tolerance degradation suite to the functional chain" (#7666)
This reverts commit 6eb200022e.
* fix(ecstore): preserve multi-pool version histories (#7658)
* fix(ecstore): preserve multi-pool version histories
* fix(ecstore): satisfy clippy in rebalance test
* fix(ecstore): preserve decommission delete compatibility
* fix(heal): recover stale peer state after restart (#7665)
* fix(heal): select writable recovery intent owner (#7668)
* feat(ci): add fault-tolerance degradation suite to the functional chain (#7669)
* fix(get): reject pre-header read quorum failures (#7670)
* fix(get): reject pre-header read quorum failures
* fix(ci): unblock get pre-header quorum checks
* ci: stop defaulting suite dispatches to a pinned release (#7681)
The rustfs_version dispatch input defaulted to 1.0.0-rc.4-preview.1,
which shadowed the nightly fallback and started 404ing once that
release was deleted. Manual dispatches with no inputs now fall through
to the nightly package (same contract the pool suite already has);
passing rustfs_version or package_url still pins the build exactly as
before. Chain (repository_dispatch) runs are unaffected: the inputs
context is empty there, so they always used the nightly fallback.
* ci: manage backlog issues by signal instead of per-run filing (#7680)
* ci: manage backlog issues by signal instead of per-run filing
The suite workflows used to file one backlog issue per failed run
(dedup was by run ID, which never matched), so issues accumulated
without bound. Replace the inline filing step in every suite workflow
(s3, kms, tier, storage, heal, pool, security, replication, upgrade,
performance) with a single call to
auto-testing/scripts/issue_manager.py, which:
- dedups by signal: failing cases are searched among open issues by
label (suite category + case ID); covered cases become a coalesced
comment on the existing issue, only uncovered cases file a new one
- labels new issues with functional-test, the suite category, one
label per failing case ID (lazily created), and env for
bootstrap-class failures (no cases ran, wholesale failure, or
404/ssh/clone/dpkg signatures in the log)
- closes open issues of the suite after a fully green run, citing the
run as evidence; cancelled runs never file or close anything
The step is skipped cleanly when auto-testing (private checkout) does
not contain the manager, or when PF_TESTING_GH_TOKEN is unset.
* fix(ci): satisfy actionlint and workflow contract tests for the manager step
- heal and performance workflows have no rustfs_version dispatch input;
referencing `${{ inputs.rustfs_version }}` in the manager step failed
actionlint's expression type check. Their package source now resolves
from package_url with the nightly fallback.
- scripts/test_security_workflow.py pinned the removed inline filing
step. The wiring assertions now pin the manager step (manager path +
per-suite report argument), and the evidence/stale-file tests assert
the skip contract instead: without the private auto-testing checkout
present, the step exits 0, publishes nothing, and leaves stale
evidence untouched.
Verified locally: actionlint clean, shellcheck clean,
test_security_workflow.py 21/21.
* fix(storage): alias legacy meta bucket over internode rpc (#7685)
* fix(storage): alias legacy meta bucket over internode rpc
Retry read-only internode RPC metadata access from legacy .minio.sys to .rustfs.sys when mixed-version peers report missing metadata during rolling upgrades.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): target durable ILM receipt quorum fixture
Use the actual durable ILM receipt object path when taking target disks offline so the test exercises receipt write quorum instead of whichever set owns the source record path.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): stabilize cancelled rename test (#7684)
* test(scanner): report Linux descriptor drift ledger (#7692)
Add a read-only descriptor ledger mode to the Scanner/Heal Linux evidence planner so release operators can separate current-head measured descriptors from old-head measured artifacts and case-level inputs before final bundle assembly.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): retry recovery readback SlowDownRead (#7695)
Allow the Scanner/Heal interruption oracle to retry transient retryable GET failures after replacement recovery has converged. The readback still verifies exact object bytes and keeps a bounded timeout, so permanently unreadable objects continue to fail the case.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(pool): default the node-endpoints fallback to the 4-node fleet (#7696)
RUSTFS_POOL_NODE_ENDPOINTS has no secret/var configured, so the suite
ran with the workflow's inline 3-endpoint fallback and the explicit
--node-endpoints flag overrode the script default fixed in
rustfs/auto-testing#61 — every dispatch died at startup with 'must
provide at least 4 direct node endpoints' (run 34677538650). Add
rustfs-node4 to the fallback; explicit secret/var still wins.
* fix(scanner): require ABBA collector credentials (#7700)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): label G14 recovery read failures (#7701)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): preserve settings in token status (#7691)
* fix(heal): preserve settings in token status
* fix(heal): group status response context
* fix(heal): decode partial status settings
* ci: green runs for case failures; red only for harness breakdowns (#7698)
A failing product case used to turn the whole workflow red, so the run
conclusion carried no signal beyond 'something failed' and the report
was suppressed. New semantics across the functional suites:
- Suite steps run with continue-on-error: the outcome is still recorded
for the report and the backlog issue manager (security/tier already
carried the flag).
- Generate report always publishes the full per-case table plus a
'Product result: N passed, M failed' summary, and its exit gate is
harness health: red only when the suite never reached case level (no
case verdicts), failed wholesale (zero passes, >=3 failures), or was
cancelled/skipped. performance is unchanged (parked).
- tier's structured gate no longer fails on case failures; it keeps red
for evidence-init and missing-gate-result breakdowns.
- pool/performance keep their existing red sources (install/benchmark).
Workflow contract tests updated to the new exit semantics: the security
report matrix keys green off the suite outcome, the evidence matrix
expects green for failure outcomes with recorded case rows (except
performance), the heal staged-rerun block expects the per-step table to
always publish, and run steps are now required to carry
continue-on-error.
Verified locally: actionlint clean; test_security_workflow.py 21/21.
* test(heal): retry G14 deferred outage put (#7703)
Treat SlowDownRead as a bounded retryable deferred PUT response after the target pool rejoins, and label terminal deferred outage PUT failures with stage context.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): retry G14 outage put probes (#7704)
Treat SlowDownRead as a retryable outage PUT probe response in the G14 multi-pool runner, and label terminal candidate failures with stage context.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): remove unused G14 put helper (#7705)
Remove the superseded ServiceUnavailable-only PUT helper after the G14 outage PUT probe switched to the shared retry classifier.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(tables): authorize reserved metadata reads (#7672)
* ci: harden report steps after first full new-semantics run (#7708)
First full serial pass with the green-on-case-failure semantics
(run 34693745171 / 34695021651) exposed three report-layer defects:
security: the report step referenced LOG_FILE, which is undefined in
this workflow (set -u killed the step before writing report.md), and
the verdict greps could not match the ANSI-escaped [PASS]/[FAIL] tags
in the real suite log. Point it at the artifacts suite.log, allow any
number of color escapes before the verdict tag, and count [SKIP]
lines separately (45 passed, 6 failed, 3 skipped was reported as an
unbound-variable crash).
pool: warp is stopped early (SIGINT) at the storage threshold and
only writes its final report on a clean exit, so an empty warp.log is
the expected shape of a healthy run - require its presence, not its
size. A mid-script die() abort or a FAIL step verdict must also turn
the validator red now that the run step is continue-on-error.
tier: add the standard 'Product result: N passed, M failed' summary
line computed from the case table, matching the other suites.
The contract test fixture previously injected LOG_FILE into the
environment and wrote verdict lines without ANSI escapes, which hid
both real-world defects; the fixture now mirrors the real suite
(stdout+tee with color tags) and asserts the pass/fail/skip counters.
* ci: tolerate ANSI reset escapes in security verdicts; fix tier product counts (#7712)
The overnight run after #7708 proved the security verdict greps still
counted zero: real verdict lines are '\e[1;31m[FAIL]\e[0m STS-105 ...'
— the reset escape sits between the tag and the case id, and the
pattern only tolerated escapes before the tag. Allow escapes on both
sides; the fixture now emits the reset too, mirroring the real suite.
The tier case table is produced by rustfs_tier_report.py and its rows
lead with the topology column, so the case-ID-first table grep matched
nothing ('Product result: 0 passed, 0 failed'). Parse the PASS/FAIL
counts from the '## Case Summary' bullets the report always emits,
falling back to a topology-aware table grep.
* fix(heal): reject overlapping admin heal owners (#7723)
* fix(heal): reject invalid admin selectors before admission (#7724)
* fix(heal): pass endpoints to overlap receipt test (#7738)
Reuse the heal-control endpoint fixture in the overlap receipt regression so the test matches the updated execution helper signature and selector validation boundary.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: hehutu <hehutu@gmail.com>
* fix(heal): verify inline shard bitrot during deep scans (#7731)
* fix(heal): verify inline shard bitrot during deep scans
* test(heal): fix CI lint and topology fixtures
* fix(heal): persist and retry partial-write repairs (#7729)
* fix(heal): persist and retry partial-write repairs
* test(heal): pass topology to overlap RPC tests
Use the existing coordinator endpoint fixture for the three overlap-test
calls to the endpoint-aware heal control executor. This repairs the E0061
test-build failure inherited from the release base.
* fix(scanner): prove dirty ACK after confirmed root write (#7732)
Allow a confirmed scanner root data-usage CAS write to prove its own publication when the follow-up root readback cannot provide a proof. Keep AlreadyDurable and all stale or companion paths on the existing readback-only proof boundary.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): rebuild truncated xl.meta from healthy quorum (#7730)
* fix(heal): rebuild truncated xl.meta from healthy quorum
* fix(test): pass topology to heal overlap RPC regression
* fix(test): drive heal admission alongside partial PUT
Poll the partial PUT and its mock heal receiver together, bound their handshake, and retain the existing repair-scope assertions.
* fix(test): prepare durable MRF fixtures and Linux heal stack
* fix(test): drive tier cleanup recovery after deferred attempts
---------
Co-authored-by: Hauser <housemecn@gmail.com>
* fix(heal): prove completed historical version cleanup (#7736)
* fix(heal): prove completed historical version cleanup
* test(heal): use debug runtime stack for C06 regression
* test(ecstore): acknowledge partial PUT heal admission concurrently
* test(mrf): fix journal fault and ownership fixtures
* fix(tables): retain dropped warehouse protection (#7675)
* fix(ecstore): gate shard integrity and preserve legacy recovery (#7741)
* fix(ecstore)!: bind bitrot shards to immutable part identities
Verify part, coding-index, and block identity across write, GET, and Heal
paths. Preserve identities across metadata-only copies and repair, include
them in multipart quorum selection, and require payload proof for receipts.
Keep legacy decoding with conservative parity and target-digest validation,
and document its unsupported cases and additional verification I/O.
BREAKING CHANGE: New bound-v1 shards require compatible readers throughout
the fleet. Legacy objects without sufficient integrity evidence return an
error; binary rollback after new writes requires verified data migration.
Refs: rustfs/backlog#2497
* fix(ecstore): preserve shard framing with independent integrity
Commit immutable part-generation Merkle roots and replicated proof indexes without changing existing checksum frames. Verify reads, reconstruction and Deep Heal against metadata quorum; keep legacy reads and explicitly defer unproven legacy data repair.
Preserve multipart rollback generations, require acknowledged durable index publication, and add decoder compatibility and donor-shard regression coverage.
* fix(heal): verify protected partial-write replay
* fix(heal): retire stale delete markers after bucket recreation (#7743)
* fix(heal): prove completed historical version cleanup
* test(heal): use debug runtime stack for C06 regression
* fix(heal): retire stale delete markers after bucket recreation
* test(ecstore): fix Clippy in retired marker regressions
* fix(heal): preserve historical null version identity (#7749)
* fix(s3): preserve version identities on suspended reads (#7751)
* fix(rebalance): retry contended runtime metadata access (#7752)
* fix(rebalance): retry contended runtime metadata access
* chore(rebalance): shrink typed error ratchet baseline
* fix(heal): bind bucket recovery to its original incarnation (#7744)
* fix(heal): bind bucket recovery to its original incarnation
* fix(heal): preserve stable bucket heal task errors
* fix(s3): classify copy source part read failures (#7746)
Map PartMissingOrCorrupt to SlowDownRead only at the GetObject/CopyObject source-reader boundary so quota metadata corruption keeps its internal fail-closed response.
Add store and e2e coverage for Harbor-style multipart staging CopyObject boundaries.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(tier): classify invalid remote credentials (#7753)
* fix(tier): classify invalid remote credentials
Classify deterministic remote-tier auth rejections before the probe cleanup fallback, and preserve the admin 4xx status for invalid credential responses.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): stabilize delete-marker movement regressions
Ignore target-local bucket incarnation fence metadata when comparing data-movement delete-marker identities, keep the release null-version listing fixture valid, and fix the nextest decommission-family filter so fault-hook tests actually join the serial group.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): serialize tier cleanup owner matrix (#7765)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): run bootstrap usage rebuild promptly (#7758)
Treat non-authoritative usage floor startup as pending bootstrap rebuild work so reset-published bootstrap markers cannot sit behind clean-idle or empty pause-backlog delay.
Wire recovery wakeups into the normal scanner cycle wait and expose the pending rebuild state in scanner status.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(tier): preserve cleanup scheduling for overwrites (#7766)
Queue committed tier free-version cleanup receipts for PUT and materialized CopyObject overwrites of transitioned null versions, while keeping remote deletion behind the existing persisted free-version cleanup path.
Tighten data-movement delete-marker retry equivalence by ignoring local bucket-incarnation fencing metadata, avoid retry fallback to the source pool, and keep version-list pagination from manufacturing an empty final page.
Refresh the e2e-distributed selector hash for the current release test set.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(release): backport main fixes and stabilize tier cleanup tests (#7793)
* fix(ecstore): stop pruning at nonempty directories (#7616)
* fix(ecstore): stop pruning at nonempty directories
* test(ecstore): release pruning fixtures before temp cleanup
(cherry picked from commit 8f150d1d8e)
* fix(heal): preserve retryable batch failures during recovery (#7642)
* fix(heal): preserve retryable batch failures during recovery
* test(heal): pin prebuilt hooks binaries in ci
(cherry picked from commit 5cd58319ed)
* fix(s3): reject oversize single PUT early and map body errors to 4xx (#7635)
* fix(s3): reject oversize single PUT early and map body errors to 4xx
A single PutObject above the 5 GiB single-request ceiling was only
rejected after the client had streamed 5 GiB into s3s's read-time body
budget, and the resulting BodySizeLimitExceeded surfaced from the erasure
writer as 500 InternalError. A body whose connection hit EOF before
Content-Length bytes arrived (hyper's IncompleteBody) was also a 500.
SDKs retry 500s, so one oversize upload was resent from offset 0 five
times.
- PutObject and UploadPart reject a declared length above
MAX_SINGLE_PUT_OBJECT_SIZE with 400 EntityTooLarge before reading the
body; the constant moves to rustfs_config so the s3s limit and the
admission check share one value.
- ApiError maps BodySizeLimitExceeded to EntityTooLarge and a hyper body
EOF to IncompleteBody across both io::Error conversions.
Fixes#7596.
* test(s3): cover UploadPart admission, aws-chunked length, real s3s limit
- Poll-counting test body proves PutObject and UploadPart reject a
declared size above the ceiling with zero body polls; exact-cap and
zero-length parts pass admission.
- A STREAMING-* aws-chunked PUT whose framed Content-Length exceeds the
cap is admitted when the decoded length is within it and rejected when
the decoded length is over it.
- The display-based BodySizeLimitExceeded matcher is checked against the
real error produced by the pinned s3s Body budget.
(cherry picked from commit 50b31bc75b)
* fix(ecstore): make directory mtime fixture portable (#7623)
* fix(ecstore): make directory mtime fixture portable
* style(ecstore): format mtime fixture assertion
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
(cherry picked from commit b1cc286cac)
* fix(storage): prevent readiness after native migration failures (#7652)
* fix(storage): prevent readiness after native migration failures
* fix(storage): skip unsupported IAM records before reading
* fix(storage): use stable typed migration metadata errors
* fix(storage): include migration record in startup errors
* test(storage): cover native migration startup failures
* test(storage): use array chunks in migration fixture
---------
Co-authored-by: RJ Regenold <214054+rjregenold@users.noreply.github.com>
Co-authored-by: cxymds <cxymds@gmail.com>
(cherry picked from commit 0cbc3ffe61)
* fix(admin): expose OIDC account display fields (#7654)
Expose verified OIDC username and email claims as display-only metadata on self-account responses while preserving the virtual parent as the authorization identity.\n\nKeep rustfs-madmin public response structs unchanged by adding the optional wire fields through private handler response wrappers.
(cherry picked from commit f02bc947cd)
* fix(replication): correct peer joins and remote-state reporting (#7650)
* fix(replication): propagate verified peer deployment identities
* fix(replication): report actual remote peer state
* fix(replication): defer initial sync until all peers join
* test(replication): shut down TLS fixtures cleanly
---------
Co-authored-by: houseme <housemecn@gmail.com>
(cherry picked from commit 853ae63b6a)
* fix(s3): bound stalled UploadPart request bodies (#7659)
* fix(s3): bound stalled UploadPart request bodies
* fix(ci): preserve the S3S footprint ratchet
(cherry picked from commit 666dfd9f9f)
* fix(tables): reject reserved warehouse locations (#7671)
Co-authored-by: cxymds <cxymds@gmail.com>
(cherry picked from commit 414176c47f)
* fix(ci): bind nightly lanes to one resolved source (#7688)
(cherry picked from commit 01d8e4347f)
* fix(replication): close the pre-stable convergence gaps from backlog#2367 (#7626)
* fix(replication): total-order rule sort and honor V1 top-level Prefix
Rule matching had two defects from the pre-GA replication audit
(rustfs/backlog#2367 C-1 and C-2):
- The actionable-rule sort compared same-destination rules by priority but
answered Equal for any other pair, which is not a total order; the
standard library sort panics on such comparators once a slice exceeds the
insertion-sort threshold, so an object matching more than 20 enabled rules
across two or more targets could panic the PUT or DELETE task. Rules now
sort by priority descending with destination and id as tie-breakers, and
filter_target_arns preserves that order instead of draining a HashSet.
- A V1 rule written without a <Filter> carries its prefix at the top level;
that field was never read, so <Prefix>logs/</Prefix> matched every object.
ReplicationRuleExt::prefix now falls back to it, with a <Filter> keeping
precedence. The existing prefix fixtures were built this way and had been
asserting nothing.
* fix(admin): advertise data-usage and listen capabilities to rc
The rc client gated `rc du` and `rc watch` on a pinned contract that
matched server versions by the string prefix `1.0.0-rc.`; a server that
reports `1.0.0` no longer matches, and the dynamic `advertised` list did
not carry either name, so `rc du` against a GA server fails with an
unsupported-capability error (rustfs/backlog#2367 E-2).
Advertise `admin.data-usage` from the admin route inventory like the IAM
entries, and `listen_notification` for the bucket `?events=` extension
route the admin router dispatches. The client merges advertised entries
ahead of its pinned contract, so no version sniffing is needed.
* fix(site-replication): stop notifying the local site on remove and rotate
The pending-remove and pending-rotation notification loops skipped the
local site by endpoint only, while finalization identifies it by
deployment id or endpoint. The reconcile tick resolves the local peer from
the node's own listen address (and a handler from the request Host), so
`remove --all` dialed the site's registered endpoint, waited out the
request timeout against the lifecycle lock it was holding, and answered
`Partial: failed to notify 1 peer(s)` for a removal that had succeeded
(rustfs/backlog#2367 A-4, backlog#2195 item 3).
Both loops now iterate the peers still awaiting notification through one
helper that applies the finalization identity.
* fix(site-replication): promote and settle IAM retries without a tick of slack
Two retry-queue behaviours kept an IAM change from converging for ten to
twenty minutes after a peer came back (rustfs/backlog#2367 A-1 and A-3,
backlog#2305):
- The lightweight 30-second pass filtered its reachability probe to bucket
ops, so a backed-off IAM or bucket-metadata snapshot waited for the
600-second tick to notice the peer. It now probes every backed-off class
and still replays only bounded bucket ops; promotion is a state flip the
heavyweight tick acts on.
- Backoffs are multiples of the tick interval, so a failure stamped δ
seconds after a tick was 600 − δ old at the next tick and slipped a whole
extra interval. The heavyweight drain now evaluates backoff halfway to its
next tick.
- An IAM entry first created by a non-deletion failure (the add bootstrap's
snapshot send, the drain's own replay, an import-iam schedule) was never
stamped `deletions_recorded`, so a later recorded deletion could not
settle it and it escalated to the marker only `replicate repair` clears.
Entries created by this binary now start recorded; a row persisted by an
older binary keeps the escalation semantics.
* fix(site-replication): reload peer node caches after bucket wiring writes
Every S3 bucket-config write ends by asking the other nodes of the cluster
to reload the bucket's metadata; the site-replication writers never did.
On a multi-node site the node that ran the pairing (or applied a peer's
bucket-meta item) rewrote the bucket targets and the derived replication
rules on disk, while every other node kept serving its cached copy for up
to the 15-minute refresh. A `resync start` routed to such a node reported
every freshly wired bucket as `Config not found` and a bucket whose
operator target the pairing had replaced as `recorded remote target no
longer exists` (rustfs/backlog#2367 A-5, backlog#2195 item 2; functional
SITE-105).
Add one best-effort reload helper in the site-replication hooks and call it
after the bucket setup, versioning, peer bucket-meta apply, removed-peer
cleanup, make-with-versioning, and endpoint-refresh writes; the ensure
helpers now report whether they wrote so unchanged passes stay silent. The
resync manifest and start now read the persisted wiring instead of the
node-local cache, matching the target read the start path already did.
The new four-node e2e pairs two clusters and starts a resync through a
non-coordinator node right after pairing; it also covers an IAM user
created on a non-coordinator node converging to the peer site.
* test(e2e): cover delete-marker replication from a multi-node source
The functional suite reported delete markers created on a 3-node source
never reaching the target (rustfs/backlog#2195 item 4, REP-105). The report
was a probe defect, but the shape had no coverage: the existing
delete-marker e2e runs a single-node source. Pin it against a four-node
source replicating to a four-node peer and to a single-node target, with
the write and the delete issued through different nodes.
* ci(e2e): refresh the distributed selection for the new replication cases
Four distributed cases were added (two site-replication, two delete-marker
replication). The linux digest is derived from the last CI listing of the
lane (34 cases, matching the previous pin) plus the four new names; the
darwin digest is the local listing, which selects the same 38 cases.
(cherry picked from commit aeaba86d73)
* fix(e2e): require a verified server binary for every e2e run (#7687)
* fix(ci): share quick checks and lint workflows
* fix(ci): install actionlint from its verified release
* fix(ci): reject dependencies on required quick checks
* feat(test): verify the E2E server build and source identity
* test(e2e): register verified Darwin test membership
* test(e2e): record verified Linux receipt test membership
* test(e2e): record compiled Darwin receipt test membership
* test(e2e): record compiled Linux receipt test membership
* test(e2e): record Darwin e2e-full membership after merging main
* fix(test): route scanner/heal evidence E2E runs through the verified server binary
The evidence runners built rustfs with plain cargo and then ran e2e_test directly, which now fails without a run receipt. They build through scripts/e2e_binary.py and run the e2e_test invocations under e2e_binary.py run; the obsolete rustfs.features stamp is removed.
* docs(e2e): run server-backed e2e commands through the verified binary wrapper
* test(e2e): record Linux e2e-full membership from the branch CI listing
(cherry picked from commit 2909b1bfe1)
* fix(kms): classify KMS/SSE error contracts and SSE-S3 headers (#7697)
* fix(sse): classify bare SSE-KMS writes when no KMS is available
A `aws:kms` request without a key id, on a bucket without a default key,
returned `500 InternalError` whenever no KMS service was running: the
"no KMS key available" branch exited with an untyped storage error before
the availability classification that the keyed form already received.
Route that branch through the same split: `503 ServiceUnavailable` while
a configured KMS is stopped, `400 InvalidRequest` when KMS was never
configured, and `400 InvalidRequest` naming the missing key id when a
running KMS has no default key. `CreateMultipartUpload` shares the path.
Adds a unit test for the bare form and an e2e module that stops KMS
through the admin API, runs a master-key-only node, and runs a Local KMS
without a default key; refreshes the e2e-full selection digests.
(cherry picked from commit c3259dadc3d603a9185a5b0ad9f83dfb884e61c8)
* fix(sse): keep KMS error classes on the encrypted read path
GetObject, CopyObject and UploadPartCopy on an SSE-KMS object whose key
no longer exists answered `500 InternalError` ("KMS key not found") while
PutObject under the same key already answered `400 KMS.NotFoundException`.
The read path carries its classification through ecstore's
`EncryptionResolutionErrorKind`, which had no kind for a missing key, a
denied KMS grant or a missing backend capability, so all three folded
onto `DecryptionFailed` and the S3 layer reported an internal fault.
Add `KeyNotFound`, `AccessDenied` and `NotImplemented` kinds, map them on
both sides of the boundary, and give an envelope the configured backend
cannot unwrap a diagnosable message while keeping its `500`.
Unit tests cover the kind round trip and the reader wrapping; a new e2e
test deletes a key immediately and checks GET/Copy return 400 with
`KMS.NotFoundException` while HEAD stays 200. The e2e-full selection
digests are refreshed from the current listing (the previous digests
predated the delete-authorization tests) and the e2e `create_default_key`
helper is updated to the accepted `EncryptDecrypt` spelling.
(cherry picked from commit 2523a9814e97caea318d4ff1a51bef3a4d4445b2)
* fix(kms): classify key-management errors on the admin routes
`POST /kms/keys`, the legacy `create-key` alias and `generate-data-key`
reported every backend refusal as `500`: a blank key name (which each
backend failed on differently, the Local backend by writing a key file
with an empty stem), a name already taken, an unknown key, a disabled key
and a capability the backend lacks. `delete` and the lifecycle routes
already classified the same errors.
Refuse a blank or whitespace name in `KmsManager::create_key` before any
backend sees it, and share one `KmsError` to status mapping across
create, delete and generate-data-key (400 for validation and key state,
404 for an unknown key, 409 for a taken name, 501 for a missing
capability, 500 only for damaged material). The XML-error routes carry
the same status explicitly since s3s derives none for a custom code.
The read-only Static backend now reports create, delete and
cancel-deletion as `UnsupportedCapability`, matching its rotate and
enable/disable answers, so the admin API returns 501 for all of them.
(cherry picked from commit e33cac5493c4d9d6662e0d2980b58ba2b24a6d1b)
* fix(sse): stop SSE-S3 responses from naming the wrapping KMS key
`x-amz-server-side-encryption-aws-kms-key-id` is defined for `aws:kms`
objects only, but PutObject, CopyObject, CreateMultipartUpload and
GetObject returned it for `AES256` objects too, carrying the KMS key that
wraps the SSE-S3 data key (the service default, or the literal `default`
on a node without KMS). The write paths copied `kms_key_id` from the
encryption material unconditionally, and the single-decrypt GET
classification did the same after resolving the key for authorization.
Add `EncryptionMaterial::response_kms_key_id`, which yields the id only
for SSE-KMS, use it at the four write-response sites, and gate the GET
classification the same way. CompleteMultipartUpload and HeadObject
already omitted the header.
Unit tests pin both directions; a new e2e test covers Put/Get/Head/Copy
and CreateMultipartUpload for AES256 with an aws:kms control. The
e2e-full selection digests are refreshed from the current listing.
(cherry picked from commit 29d793a63352b0b60fd53c565e80fdbede8964bb)
* fix(s3): validate PutBucketEncryption rules before storing them
A default-encryption rule naming an unknown `SSEAlgorithm` (for example
`AES128`), a rule without `ApplyServerSideEncryptionByDefault`, an empty
rule list, or a `KMSMasterKeyID` on an `AES256` rule was stored as
written: the only algorithm check on the route decided whether to fill
in the default KMS key. `GetBucketEncryption` then advertised that
configuration while the write path encrypted header-less writes under
its `AES256` fallback, so the bucket's declared and actual schemes
disagreed. Two comments claimed the route already refused unknown
algorithms.
Validate the configuration before any of it is applied: `MalformedXML`
for a malformed rule set or unknown algorithm, `InvalidArgument` for a
key id on a non-KMS rule, and nothing stored on refusal. Correct the two
comments to describe when the AES256 fallback is still reachable.
Unit tests cover every refusal and the accepted shapes; an e2e test
checks the refusals leave the previous configuration in place. The
e2e-full selection digests are refreshed from the current listing.
(cherry picked from commit 29e4486dce41197ed93f5253cdbabc57d27a4ddb)
* test(e2e): refresh e2e-full selection for the combined KMS/SSE fixes
* test: align two unit tests with the new KMS and bucket-encryption contracts
`scheduled_deletion_carries_a_deadline_and_can_be_cancelled` still
expects the state error (`InvalidOperation`) for cancelling a key that
is not pending deletion; only the Static backend's mutations moved to
`UnsupportedCapability`. The uninitialized-store PutBucketEncryption
test now sends a well-formed AES256 rule so it reaches the store lookup
instead of the new configuration validation.
(cherry picked from commit e2e6a2535a)
* fix(site-replication): keep an operator's bucket-level target to a peer instead of taking it over (#7709)
* fix(site-replication): keep an operator's bucket-level target to a peer instead of taking it over
Site replication wired each bucket by looking for an existing replication
target "to the same peer" and rewriting the first match in place as its own
same-name target. An operator's bucket-level target that happened to point
at that site (different target bucket, operator credentials) was the first
match whenever it pre-dated the join, and the reconciler repeats the pass
every 600s, so the takeover also depended on target order afterwards. The
operator's rule then named an ARN no target backed and their bucket
replication stopped silently, while the inherited bucket-level reset id
made every site resync report the bucket as owned by another resync
(rustfs/backlog#2479, rustfs/backlog#2489).
Follow MinIO's `getRemoteARN` / `getRemoteARNForPeer` shape instead:
- Wiring updates a target in place only under the same ARN, or when it is
recognisably the site's own under an older ARN shape (same peer,
same-name target bucket, site replication service account). Anything
else gets the site target added next to it.
- The site resync manifest takes the target the derived
`site-repl-<deployment id>` rule names (same-name shape as fallback), so
an operator target to the peer neither aborts the bucket as "multiple
remote targets matched peer" nor gets resynced into.
- Peer removal prunes only targets a pruned derived rule names or the
same-name target bucket; operator targets stamped with the peer's
deployment id survive together with their rules.
Unit tests cover the three predicates. e2e
`test_site_replication_keeps_operator_bucket_target_to_peer` runs a
bucket-level replication plus `replication-reset` to the future peer, joins
the sites, and requires the operator target untouched, both paths
delivering, the site resync completing against the site target, and the
operator target and rule surviving `replicate remove --all`; without the
fix it fails at the join with the operator target gone. The repl-nightly
selection digest is refreshed for the new case.
* test(site-replication): drop a redundant clone flagged by clippy
The reconcile unit test cloned the remote peer into the state map although
the binding is not used afterwards; workspace clippy (-D warnings) rejects
that as redundant_clone.
(cherry picked from commit ecdc55fa4b)
* fix: enforce S3 permissions for recursive force deletion (#7661)
* fix: enforce S3 authorization for recursive deletion
* fix: satisfy the s3s footprint guard
* fix: restore list versions policy compatibility (#7686)
(cherry picked from commit 3fd1ce414d)
* fix(ci): repair functional defaults and chain regression checks (#7664)
* fix(ci): default functional suites to nightly packages
* test(ci): follow the fault-tolerance chain handoff
(cherry picked from commit 509a0fa90c)
* fix(ci): align security workflow tests with chain (#7679)
(cherry picked from commit d9e47d2813)
* test(ecstore): keep tier cleanup tests stable after immediate receipt queueing
Release PR #7766 made PUT/CopyObject overwrites queue the tier free-version cleanup receipt immediately, which broke two ecstore tests on release CI. In tier_overwrite_put_and_self_copy_recover_persisted_cleanup_owners the restarted store already runs expiry workers from the second iteration on, so they deleted the remote bytes before the test could assert that the commit leaves them in place; the test now fails the first remote DELETE via set_remove_failure(true) so the cleanup owner stays durable and the later restart still has to rediscover it from xl.meta (the failed remove does not bump remove_count, and the test re-enables removes before the recovery wait). In batch_transitioned_delete_post_commit_failures_roll_back_without_free_version_receipt the convergence loop now treats a transient InsufficientReadQuorum as "not yet converged", because cleanup rewrites xl.meta disk by disk and a racing read can briefly miss quorum (seen in the rio-v2 lane); any other error still panics.
* Revert "fix(ci): align security workflow tests with chain (#7679)"
This reverts commit 4544359f6d.
* Revert "fix(ci): repair functional defaults and chain regression checks (#7664)"
This reverts commit 5ba7ec0291.
* fix(ecstore): pass shard integrity to backported ingest-mode test
The stalled-reader test backported with #7659 used main's four-argument encode_with_ingest_mode, but release's signature takes an optional IntegrityBuilder, so pass None to keep the test focused on ingest-mode cleanup.
---------
Co-authored-by: Henry Guo <marshawcoco@gmail.com>
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: RJ Regenold <rregenold@teamraft.com>
Co-authored-by: RJ Regenold <214054+rjregenold@users.noreply.github.com>
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: GatewayJ <835269233@qq.com>
Co-authored-by: Jason Kossis <jkossis@gmail.com>
* fix(ci): restore release test compilation and e2e membership (#7799)
* fix(s3): return stored content language on local GET (#7755)
* fix(s3): return resolved object tagging version identities (#7757)
* fix(tables): bound warehouse index recovery (#7677)
Co-authored-by: Chris <anzhengchao@gmail.com>
* fix(upgrade): probe legacy IAM namespace before migration (#7816)
* fix(release): backport IAM migration startup fixes and recent main fixes (#7825)
* fix(auth): reject unsigned x-amz headers on header-signed SigV4 requests (release) (#7813)
* fix(s3): preserve missing version errors on object reads (#7800)
* fix(s3): preserve missing version errors on object reads
* test(s3): align missing-version assertions with read contract
* fix(heal): bind admin traversal to bucket incarnation (#7802)
* fix(heal): bind admin traversal to bucket incarnation
* fix(ecstore): reopen healthy hedged readers after peer loss
* test(ecstore): isolate tier overwrite recovery scheduling
* test(ecstore): observe tier cleanup under object read locks
* fix(ecstore): bound decommission entry tracing
* test(ecstore): drain incarnation heal fixture writes
* test(heal): match debug server stack in deep heal fixtures
* test(heal): include identities in C06 count failures
* test(heal): drain PUT tails before inspecting B920 fixtures
* test(scanner): isolate retained MRF retry slots
* fix(ecstore): separate PUT cleanup intent from persisted metadata
* fix(ecstore): capture failed object lock acquisitions (#7803)
* fix(ecstore): capture failed object lock acquisitions
* test(heal): include identities in C06 count failures
* test(heal): drain PUT tails before inspecting B920 fixtures
* fix(ecstore): reopen healthy hedged readers after peer loss
* test(ecstore): isolate tier overwrite recovery scheduling
* test(ecstore): observe tier cleanup under object read locks
* fix(ecstore): bound decommission entry tracing
* test(ecstore): drain incarnation heal fixture writes
* test(heal): match debug server stack in deep heal fixtures
* test(scanner): isolate retained MRF retry slots
* fix(ecstore): separate PUT cleanup intent from persisted metadata
* fix(heal): retain terminal outcomes across restart (#7805)
* fix(heal): retain terminal outcomes across restart
* test(heal): use octal mode for terminal recovery fixture
* fix(ecstore): reopen healthy hedged readers after peer loss
* test(ecstore): isolate tier overwrite recovery scheduling
* test(ecstore): observe tier cleanup under object read locks
* fix(ecstore): bound decommission entry tracing
* test(ecstore): drain incarnation heal fixture writes
* test(heal): match debug server stack in deep heal fixtures
* test(heal): include identities in C06 count failures
* test(heal): drain PUT tails before inspecting B920 fixtures
* test(scanner): isolate retained MRF retry slots
* fix(ecstore): separate PUT cleanup intent from persisted metadata
* fix(heal): recover replacement ownership across VM restarts (#7810)
* fix(heal): recover replacement ownership across VM restarts
* test(ecstore): isolate tier overwrite recovery scheduling
* test(ecstore): observe tier cleanup under object read locks
* fix(ecstore): bound decommission entry tracing
* test(ecstore): drain incarnation heal fixture writes
* fix(ecstore): reopen healthy hedged readers after peer loss
* test(heal): match debug server stack in deep heal fixtures
* test(heal): include identities in C06 count failures
* test(heal): drain PUT tails before inspecting B920 fixtures
* test(scanner): isolate retained MRF retry slots
* fix(ecstore): separate PUT cleanup intent from persisted metadata
* fix(tables): isolate protected S3 listings (#7676)
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix(ci): restore Linux E2E selection and split offline enrollment (#7843)
* fix(heal): assign stable identities to scanner repair requests (#7823)
* fix(heal): assign stable identities to scanner repair requests
* test(s3): isolate UploadPart inactivity timing
* fix(heal): backport pool metadata scope to release (#7846)
* fix(heal): scope replacement pool metadata to its owning set
(cherry picked from commit a72a75569c)
* fix(heal): use domain imports in pool metadata regression test
(cherry picked from commit e8c5448e21)
* test(heal): set pool metadata regression recursion limit
---------
Co-authored-by: Hiroaki KAWAI <1468181+hkwi@users.noreply.github.com>
* fix(heal): persist pool/set outcomes and recovery scope (#7839)
* fix(heal): persist pool/set outcomes and recovery scope
* test(heal): verify selector receipts and process-crash recovery
* fix(heal): scope replacement pool metadata to its owning set
(cherry picked from commit a72a75569c)
* fix(heal): use domain imports in pool metadata regression test
(cherry picked from commit e8c5448e21)
* test(heal): set pool metadata regression recursion limit
---------
Co-authored-by: Hiroaki KAWAI <1468181+hkwi@users.noreply.github.com>
* fix(ecstore): scope recursive walks to object prefixes (#7850)
* fix(heal): scope nested walks and reject admin read-repair (#7851)
* fix(heal): scope nested walks and reject admin read-repair
* fix(ci): allocate stack for heal metadata regression
* test: cover overlapped multipart staging copy (#7865)
Add Harbor-style E2E coverage for a CopyObject racing with CompleteMultipartUpload. The test allows the copy to observe an unpublished source as NoSuchKey, but rejects 5xx leakage and proves retrying the completed source remains readable.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(release): repair acceptance checks and prepare 1.0.0 (#7866)
* test(e2e): stabilize release acceptance checkpoints
* chore(release): prepare 1.0.0
* test(scanner): stabilize W13 release evidence gates (#7870)
Add the object-only completion diagnostic case and make the evidence runner resolve CARGO_TARGET_DIR before recording the rustfs binary path.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(kms): cover non-default Vault Transit paths (#7657)
Backport the Vault coverage from 4d31621f0a. Keep the release cleanup polls synchronized with object read locks.
* fix(replication): count a bodiless 405 as a replicated delete marker (#7756)
* fix(replication): accept a bodiless 405 as a replicated delete marker during resync
A resync verifies each delete marker with `HEAD ?versionId=<marker>` on the
target. S3 targets (RustFS, MinIO, AWS) answer that with 405 and no body,
and the SDK only synthesizes an error code for 404, so the error arrived
with `code() == None`. `is_retryable_delete_replication_head_error` then
treated it as an ambiguous failure: every delete marker counted as a failed
object with `target service error`, and a site resync over any bucket that
holds a delete marker reported the whole bucket as failed
(rustfs/backlog#2479, SITE-105 on rc.6 and nightly).
Use the raw HTTP status the way the 404 path already does: a 405 without a
code confirms the marker propagated. Ambiguous statuses still fail.
- Unit tests drive `verify_resync_head_result` against a scripted target
answering 405 (accepted) and 503 (still failed).
- e2e `test_site_replication_resync_replicates_delete_marker` joins two
sites, converges a live object and a delete marker, and requires the site
resync to complete with zero failed objects; the repl-nightly selection
digest is refreshed for the new case.
* fix(replication): verify marker-version purges by absence during resync
A delete-marker resync entry with a pending or failed version purge asks
the target to remove the marker, so the bodiless 405 that proves a
created marker propagated proves the purge did not happen. Accept the
405-as-success mapping only for marker creation (empty
version_purge_status); a purge counts as replicated only when the target
answers not found, and any other HEAD outcome stays failed. Unit tests
drive both purge statuses against the 405 fixture and the 404 fixture.
(cherry picked from commit 1ef3974bad)
* fix(kms): refuse key ids that leave the key prefix on the Vault backends (#7727)
Only the Local backend refused a key identifier containing a path
separator. On Vault KV2 a create with the name bad/name succeeded and
produced a nested KV2 path that the listing then reported as a
directory rather than a key, and an identifier containing .. addressed
a record outside the configured key prefix once the HTTP client
normalised the URL; Vault Transit built its transit key name and its
metadata path from the same unchecked identifier.
Lift the Local backend's containment rule into a shared segment check
(empty, /, backslash, NUL, . and ..) and apply it at the single point
where each backend turns the identifier into a path or a Transit key
name, so create, describe, encrypt, delete and the metadata writes all
refuse with InvalidKey before any request reaches Vault. The admin API
already maps that to 400. The AWS backend is untouched: it addresses
keys by ARN and alias, both of which contain /.
Refs rustfs/backlog#2474 (KMS-213 CreateNegatives on vault-kv2).
Co-authored-by: Hauser <housemecn@gmail.com>
(cherry picked from commit 2f2e775655)
* fix: resume GET body after peer short EOF (#7852)
Backport GET recovery from afc8861fec. The Connect TLS classification change requires the main-only proxy transport layer.
* fix(s3): honor presigned UploadPart checksum queries (#7748)
* fix(s3): honor presigned UploadPart checksum queries
* fix(s3): keep checksum errors within gateway boundary
---------
Co-authored-by: Hauser <housemecn@gmail.com>
(cherry picked from commit a8ace89516)
* fix(table-catalog): map transient lock failures to unavailable (#7841)
(cherry picked from commit 5c73a3d602)
* fix(ecstore): preserve ready read errors during encoding (#7798)
Backport the erasure fix from c95b65a7fc while retaining release shard integrity support. Dependency upgrades and the main-only Connect proxy and diagnostic fixes are excluded.
* ci: add the S3 Tables (Iceberg REST Catalog) functional suite workflow (#7873)
New workflow rustfs-table-test.yml following the shared suite contract:
Initialize functional evidence (FUNCTIONAL_ARTIFACTS_DIR/LOG_FILE/
REPORT_FILE/TMPDIR), checkout for the report parser, auto-testing
clone with retry, continue-on-error on the run step, report generation
through scripts/functional_case_report.py with the product gate (case
failures stay green; harness breakdowns turn red), dashboard upload
under functional-reports/table/, backlog issue manager wiring
(--suite table --category table), artifact upload (report.md, cases.md,
suite.log), before/after environment cleanup, and the chain handoff
step (repository_dispatch only, next: pool expansion).
The suite script itself lives in rustfs/auto-testing (TBL-101..107,
paired PR). The run step passes the product smoke script path
(scripts/table-catalog/pyiceberg_smoke.py) via --smoke-script and the
log via --log-file, matching the other suites.
Contract tests: the table suite is registered in the generic evidence
and workflow matrices (JOBS/DIRECT_TESTS/allowlist) plus a dedicated
TableSuiteTests wiring check. Full run: 21/21.
* test(ecstore): tolerate slow read-repair diagnostics CI (#7892)
* fix(heal): report verified normal shard repairs (#7890)
* fix(heal): retry interrupted internode shard writes
(cherry picked from commit 0bd75f47b6326edd8167ee43b8776f6276692e92)
* fix(deps): update rustls for RUSTSEC-2026-0285
(cherry picked from commit 938e1259d13da7832d85ac5c066a59ab3a28bc63)
* test(heal): wait for replacement disk and coordinator readiness
(cherry picked from commit feeaa6f053aac4b7c8e32477c0ab5ebbd3dcdff3)
* fix(heal): retain RPC transport failures across metadata probes
(cherry picked from commit f48ec9f0c6e37b1f941620c666a9357c4b892d68)
* fix(s3): resume GET bodies after erasure read quorum loss
(cherry picked from commit a38e859905)
* test(heal): keep outage objects in the replacement erasure set
(cherry picked from commit a669f50a366e35505e7745f8b680d8ecf1aa31be)
* fix(rpc): retry authenticated requests after peer epoch changes
Backport the stale-epoch recovery fix from main follow-up #7889. Keep invalid signatures and nonce replays terminal.
* test(rpc): assert retryable rejection of stale boot epochs
(cherry picked from commit 1c4b39b99)
* ci: bound concurrent heal clusters in the full E2E gate
Apply the #7889 scheduling fix while preserving release test membership and zero retries.
* fix(ecstore): hide pending purge entries from listings (#7895)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): scope pending purge listing regression (#7897)
Keep the store-level pending purge listing regression focused on the ordinary recursive path and exact-prefix shortcut. The deeper delimiter CommonPrefix case is a separate physical-walker boundary and should not leave the release test suite red.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(obs): expose scanner usage freshness metrics (#7896)
Add Prometheus scanner metrics for dirty-usage pending buckets, usage-cache save/publication timestamps, publication result codes, and deferred publication state. This gives operators a direct scrape path for confirming scanner dirty ACK behavior without relying on admin JSON.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): backport delete quorum rollback to release (#7902)
* ci: rotate the shared functional-tests concurrency group to -v2 (#7900)
* fix(s3): render null write version IDs consistently (#7904)
* fix(heal): quarantine corrupt root recovery records (#7905)
* fix(heal): quarantine corrupt root recovery records
Keep invalid root-heal records durable while isolating them from healthy recovery owners. Serialize startup replay with administrator admission and fence quarantined task IDs so independent compensation heals can proceed safely.
* fix(ci): align release workflow contract tests
* fix: satisfy root recovery clippy checks
* fix(heal): fence canonical repair receipts (#7909)
* fix(heal): fence canonical repair receipts
* fix(heal): preserve unversioned receipt identity
* fix(heal): persist replacement intent on remote survivor (#7912)
* chore(release): merge main into release (#7913)
* feat(nightly): publish packages as assets of the rolling 'nightly' release (sync from release) (#7593)
feat(nightly): publish packages as assets of the rolling 'nightly' release (#7592)
Replace the assets-branch scheme with a proper GitHub Release on
rustfs/auto-testing: a single 'nightly' release whose deb/rpm assets
are replaced in place on every build. This is the standard channel —
visible on the repo's Releases page, stable download URLs, no git
history growth (release assets live outside the repository).
- New scripts/release/publish_nightly_assets.sh: resolves-or-creates
the 'nightly' release via the REST API, deletes same-name assets,
uploads rustfs-nightly-latest.{deb,rpm}, then PATCHes the release
body with the build provenance (ref@sha, run link, sizes, SHA256).
Plain curl + python3, no gh CLI (the build fleet has none — #7586).
- The workflow step shrinks to invoking the script; full flow
exercised end-to-end against the real release with probe files
(create / upload / overwrite / download round-trip / body update).
* fix(ecstore): stop pruning at nonempty directories (#7616)
* fix(ecstore): stop pruning at nonempty directories
* test(ecstore): release pruning fixtures before temp cleanup
* fix(replication): close the pre-stable convergence gaps from backlog#2367 (#7626)
* fix(replication): total-order rule sort and honor V1 top-level Prefix
Rule matching had two defects from the pre-GA replication audit
(rustfs/backlog#2367 C-1 and C-2):
- The actionable-rule sort compared same-destination rules by priority but
answered Equal for any other pair, which is not a total order; the
standard library sort panics on such comparators once a slice exceeds the
insertion-sort threshold, so an object matching more than 20 enabled rules
across two or more targets could panic the PUT or DELETE task. Rules now
sort by priority descending with destination and id as tie-breakers, and
filter_target_arns preserves that order instead of draining a HashSet.
- A V1 rule written without a <Filter> carries its prefix at the top level;
that field was never read, so <Prefix>logs/</Prefix> matched every object.
ReplicationRuleExt::prefix now falls back to it, with a <Filter> keeping
precedence. The existing prefix fixtures were built this way and had been
asserting nothing.
* fix(admin): advertise data-usage and listen capabilities to rc
The rc client gated `rc du` and `rc watch` on a pinned contract that
matched server versions by the string prefix `1.0.0-rc.`; a server that
reports `1.0.0` no longer matches, and the dynamic `advertised` list did
not carry either name, so `rc du` against a GA server fails with an
unsupported-capability error (rustfs/backlog#2367 E-2).
Advertise `admin.data-usage` from the admin route inventory like the IAM
entries, and `listen_notification` for the bucket `?events=` extension
route the admin router dispatches. The client merges advertised entries
ahead of its pinned contract, so no version sniffing is needed.
* fix(site-replication): stop notifying the local site on remove and rotate
The pending-remove and pending-rotation notification loops skipped the
local site by endpoint only, while finalization identifies it by
deployment id or endpoint. The reconcile tick resolves the local peer from
the node's own listen address (and a handler from the request Host), so
`remove --all` dialed the site's registered endpoint, waited out the
request timeout against the lifecycle lock it was holding, and answered
`Partial: failed to notify 1 peer(s)` for a removal that had succeeded
(rustfs/backlog#2367 A-4, backlog#2195 item 3).
Both loops now iterate the peers still awaiting notification through one
helper that applies the finalization identity.
* fix(site-replication): promote and settle IAM retries without a tick of slack
Two retry-queue behaviours kept an IAM change from converging for ten to
twenty minutes after a peer came back (rustfs/backlog#2367 A-1 and A-3,
backlog#2305):
- The lightweight 30-second pass filtered its reachability probe to bucket
ops, so a backed-off IAM or bucket-metadata snapshot waited for the
600-second tick to notice the peer. It now probes every backed-off class
and still replays only bounded bucket ops; promotion is a state flip the
heavyweight tick acts on.
- Backoffs are multiples of the tick interval, so a failure stamped δ
seconds after a tick was 600 − δ old at the next tick and slipped a whole
extra interval. The heavyweight drain now evaluates backoff halfway to its
next tick.
- An IAM entry first created by a non-deletion failure (the add bootstrap's
snapshot send, the drain's own replay, an import-iam schedule) was never
stamped `deletions_recorded`, so a later recorded deletion could not
settle it and it escalated to the marker only `replicate repair` clears.
Entries created by this binary now start recorded; a row persisted by an
older binary keeps the escalation semantics.
* fix(site-replication): reload peer node caches after bucket wiring writes
Every S3 bucket-config write ends by asking the other nodes of the cluster
to reload the bucket's metadata; the site-replication writers never did.
On a multi-node site the node that ran the pairing (or applied a peer's
bucket-meta item) rewrote the bucket targets and the derived replication
rules on disk, while every other node kept serving its cached copy for up
to the 15-minute refresh. A `resync start` routed to such a node reported
every freshly wired bucket as `Config not found` and a bucket whose
operator target the pairing had replaced as `recorded remote target no
longer exists` (rustfs/backlog#2367 A-5, backlog#2195 item 2; functional
SITE-105).
Add one best-effort reload helper in the site-replication hooks and call it
after the bucket setup, versioning, peer bucket-meta apply, removed-peer
cleanup, make-with-versioning, and endpoint-refresh writes; the ensure
helpers now report whether they wrote so unchanged passes stay silent. The
resync manifest and start now read the persisted wiring instead of the
node-local cache, matching the target read the start path already did.
The new four-node e2e pairs two clusters and starts a resync through a
non-coordinator node right after pairing; it also covers an IAM user
created on a non-coordinator node converging to the peer site.
* test(e2e): cover delete-marker replication from a multi-node source
The functional suite reported delete markers created on a 3-node source
never reaching the target (rustfs/backlog#2195 item 4, REP-105). The report
was a probe defect, but the shape had no coverage: the existing
delete-marker e2e runs a single-node source. Pin it against a four-node
source replicating to a four-node peer and to a single-node target, with
the write and the delete issued through different nodes.
* ci(e2e): refresh the distributed selection for the new replication cases
Four distributed cases were added (two site-replication, two delete-marker
replication). The linux digest is derived from the last CI listing of the
lane (34 cases, matching the previous pin) plus the four new names; the
darwin digest is the local listing, which selects the same 38 cases.
* feat(console): support a configurable console URL prefix (#7634)
Co-authored-by: houseme <housemecn@gmail.com>
* chore(release): merge release into main for rc.6 (#7638)
* test(heal): cover MRF manifest CAS legacy transition (#7443)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ci): enable release branch checks and repair test imports (#7441)
* test(scanner): structure heal release evidence lanes (#7442)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): require segment producer identities (#7444)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): expose recovery intent identity (#7445)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): summarize failed heal perf reports (#7446)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(common): retain unmatched MRF repair proofs (#7447)
Add a recorded verified-repair consumer that discharges only exact retained anchors for the requested bucket while leaving unmatched proofs in the event ring. This gives the future durable successor writer a fail-closed primitive before any tombstone or GC path is enabled.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ci): route replication read plan through boundary (#7448)
* test(scanner): cover flat-bucket quantum fairness (#7449)
Add a production-entry scanner cohort regression that combines a wide flat bucket with a small bucket under a fixed object budget. The test keeps partial budgeted rounds unpublished, verifies small buckets are only marked after real execution, and confirms a later unbudgeted round can publish the complete aggregate.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): reuse same admission request id (#7451)
Keep a retried heal start with the same request id from bypassing admission deduplication when the transport replay cache is unavailable but the manager still owns the task.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require perf summary evidence fields (#7453)
Fail closed when measured passing Scanner/Heal ABBA summaries omit W10/W11 foreground pressure, lock wait, or attempt-cost evidence.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind release evidence fields (#7452)
Require explicit scanner/heal release evidence field contracts for the scoped ACK and mixed-version rollback gates.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): tag dirty usage producer identities (#7454)
Record production-facing segment invalidation producer identities when existing object-level dirty usage hooks observe PUT, CopyObject, DeleteObject/DeleteMarker, and CompleteMultipartUpload mutations. Keep the data non-authoritative and process-local so segment reuse activation still requires durable generation-window proof.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): gate release evidence bundles (#7457)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require hard evidence ABBA manifest (#7459)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(heal): add MRF committed snapshot writer (#7458)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover EC8+4 restart shard rebuild (#7461)
Retry heal-control RPCs once after transport auth rejects a stale replay-scope epoch, and add a distributed EC8+4 restart heal evidence case that rebuilds a replaced drive with exact shard/body assertions.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): echo measured release evidence in ABBA fake adapter (#7460)
Keep the synthetic ABBA test adapter aligned with the measured release evidence contract so result validation covers release_evidence drift.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require complete ABBA summary matrix (#7462)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* heal: retain MRF replay journal after accepted/merged replays
Keep startup replay journal until a durable successor snapshot exists after Accepted/Merged admission.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* heal: reclaim superseded MRF snapshots after readback
Add a committed snapshot predecessor cleanup primitive that deletes only an older slot after the successor is read back as the current committed snapshot.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* heal: require exact owner for object receipts
Require object heal receipts to match the expected bucket incarnation before they can be recorded as positive repair evidence. This prevents stale or cross-incarnation receipts from clearing the wrong heal responsibility.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* heal: latch object receipt owner before repair
Capture the expected bucket incarnation before invoking object repair so a post-repair owner change cannot rewrite the responsibility that a storage receipt is allowed to prove.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* heal: replay committed MRF checkpoints durably (#7465)
Prefer committed MRF checkpoints during startup replay, retain accepted replay responsibilities until exact verified repair proofs arrive, and reclaim committed manifests only after discharge.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind hard evidence bundle provenance (#7467)
Require Scanner/Heal release bundle fields to carry source, run, window, timestamp, command, and artifact format provenance before a measured gate can pass.
Keep EC8+4 and performance gate fields tied to a single measurement window so unrelated artifacts cannot be stitched into a release approval.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* heal: reject dry-run object repair receipts
Do not record positive storage repair receipts for dry-run object heal tasks, even if a producer accidentally returns a matching receipt.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* heal: reject cancelled object repair receipts
Do not record positive storage repair receipts once an object heal task has been cancelled, even if the receipt still matches the requested owner and object identity.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): reject failed object repair receipts
Cover the receipt consumer path where a storage repair result carries both an error and a matching positive receipt. The failure may be recorded, but the receipt must not create repaired, healthy, or absent proof.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover bucket object repair receipts (#7468)
Cover bucket and root heal sweeps recording authoritative object outcomes only when storage receipts match the latched bucket incarnation.
Verify unavailable or stale receipt ownership keeps object repair execution intact while leaving canonical outcome proof as Unknown.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add EC8+4 heal restart evidence (#7469)
Register and wire Scanner/Heal background target restart and crash evidence cases for the 3x4 EC8+4 topology.
Validate the observed data/parity geometry in scanner-heal evidence receipts so multi-drive runs cannot satisfy the gate without proving EC8+4 metadata.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* heal: verify replacement pool metadata repair (#7471)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover quorum and mixed repair receipts
Add focused oracles for transient quorum results that carry a matching receipt and for mixed grace plus repaired receipt batches. Only the repaired object may produce positive proof.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* heal: verify admin recreate pool metadata (#7474)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* heal: preserve merged result for duplicate submits
Keep same-request-id replay receipts accepted for the receipt API, but preserve the legacy submit_heal_request duplicate admission result as Merged.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): preserve EC84 restart semantics (#7478)
Keep the EC8+4 background restart lane on the graceful-stop path and assert clean-restart marker absence only for restart scenarios. This prevents the hard evidence gate from silently exercising the crash path when it claims restart coverage.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF snapshot torn successor recovery
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): write EC84 distributed restart oracle (#7481)
Bind the distributed EC8+4 restart evidence lane to a scanner/heal oracle artifact so release validation can consume the real nextest run instead of accepting only a passing test.
Require the registry to assert 8+4 erasure geometry for the three-node, four-drive case.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): harden MRF replay boundaries (#7483)
Reject journal records with unknown version-presence flags even when their CRC is valid, so rollback/future payloads cannot be accepted as known records.
Gate committed checkpoint cleanup by the writer owner captured from the replay source, preserving retained manifests from other owners inside the same sequence window.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): bind MRF manifest CAS dirsync recovery (#7482)
Cover the committed MRF manifest path through LocalDisk conditional CAS when the metadata directory fsync fails. The fixture proves the previous manifest anchor survives rollback, an unanchored first successor is removed, and the legacy MRF journal remains readable even while global durability is relaxed.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): derive evidence runner profile from registry (#7484)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require mixed-version evidence roles (#7485)
Tighten the Scanner/Heal release bundle checker so mixed-version, scoped-ACK, and rollback fields cannot reuse a generic versions list without proving the expected evidence role.
Require version lists to use source revision identities and include the tested source revision. Also require profile evidence fields to name the core profiling artifacts before release approval.
The release gate remains blocked until measured field evidence is present.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): publish committed MRF runtime checkpoints (#7490)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind profile artifacts in release evidence (#7487)
Require Scanner/Heal release bundles to attach every required profiling artifact to P1 profile evidence with relative paths, artifact formats, non-empty files, hashes, and optional per-artifact measurement-window checks.
Document the tightened release bundle profile contract and cover missing, tampered, and mismatched-window profile artifact regressions in the existing checker self-test.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): harden measured ABBA evidence claims (#7491)
Reject measured Scanner/Heal ABBA manifests whose mixed-version evidence uses the same baseline and candidate source revision or binary hash.
Require crash fault modes and profile artifact names to match the exact supported sets, rejecting missing, duplicate, and unknown values.
Update harness fixtures and regression coverage for same-build mixed-version claims and exact-set release evidence fields.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(e2e): group scanner heal evidence payload (#7494)
Group the EC8+4 Scanner/Heal evidence writer inputs into a typed payload so the distributed e2e crate stays within the clippy argument limit without weakening the lint.
The evidence writer still validates the same S3 bodies, physical shard census, process restart PIDs, and node listings.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF replay bundle fields (#7486)
Bind Scanner/Heal release bundle evidence for MRF durable replay to replay counts, retained responsibility anchors, and successor snapshot publication evidence.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): profile EC84 evidence case runs (#7495)
Give Scanner/Heal evidence cases explicit runtime profiles so EC8+4 background restart and crash cases use their own object count, object size, and partial-progress timeout defaults instead of inheriting the legacy 4x1 case assumptions.
Expose the runtime profile in plan-only output and cover every registry case in the script self-test.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): reject empty release evidence artifacts (#7496)
Require Scanner/Heal release bundle artifact paths to resolve to non-empty files before hashing them.
Cover empty hard-gate artifacts in the existing release bundle checker self-test and keep profile artifact size checking on the shared artifact boundary.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require G14 same-window field coverage (#7489)
Tighten the Scanner/Heal release bundle checker so G14 same-window evidence must name the EC8+4, multi-set, and multi-pool fields covered in that measurement window.
Keep the release gate blocked when same-window evidence omits one of the required G14 fields, without changing production runtime behavior.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require two-hour measured ABBA windows (#7493)
Reject measured Scanner/Heal release ABBA manifests and summaries whose evidence window is shorter than the W21 two-hour release requirement.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF idle checkpoint cleanup (#7497)
Add a runtime cleanup regression test that publishes both retained replay and runtime committed checkpoints, writes scoped and legacy journals, and verifies idle cleanup removes every recovery anchor from the registered local disks.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(e2e): ignore untracked files in build identity (#7498)
Align e2e_test build provenance with Scanner/Heal evidence receipts and server binary provenance by treating only tracked source changes as dirty.
This prevents unrelated untracked worktrees or evidence directories from causing compiled test identity mismatches before real evidence cases can run.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF disk-full evidence fields (#7499)
Require the Scanner/Heal release registry and bundle checker to carry explicit G08 MRF capacity, disk-full, and replica-loss evidence fields before release approval can pass.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind release JSON artifact provenance (#7500)
Require scanner/heal release evidence JSON artifacts to repeat their measured source revision, run identity, measurement window, gate, and field identity inside the artifact payload. This keeps a refreshed outer bundle hash from accepting stale summary or profile JSON from another run.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF rollback mirror filtering (#7501)
Add a regression oracle that keeps scoped-only MRF responsibilities in the authoritative runtime snapshot while omitting them from the v1 legacy rollback mirror.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require segment activation evidence (#7504)
* test(scanner): require segment activation evidence
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): cover activation proof inputs
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF crash matrix cases (#7505)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require release bundle domain evidence (#7502)
Reject scanner/heal release bundles that omit field-level domain evidence for scoped ACK, durable intent, mixed-version, scheduler pressure, profile cost, and two-hour pressure lanes.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF retention GC evidence (#7506)
Require P4 retained-responsibility release bundle evidence to list the retained replay anchor and idle cleanup cases, prove a two-hour retention window, and record both idle cleanup and verified-proof discharge observations.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF cleanup GC soak evidence (#7507)
Require the P4 release bundle to carry measured cleanup/GC soak evidence for retained MRF replay responsibilities. The bundle now needs a two-hour cleanup window, exact cleanup case coverage, observed verified idle GC, and zero pending responsibilities or stale journals after GC.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF disk-full commit anchors (#7509)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): report port allocator bind errors (#7510)
Surface the scanned port window, attempt count, and last bind error when the e2e port allocator cannot reserve a localhost port. This keeps Scanner/Heal evidence failures actionable when the environment blocks binds before business assertions run.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(sse): resolve 1.0.0 SSE/KMS blockers and P1 findings (#7511)
* fix(sse): resolve bucket default encryption per request
PUT and the POST-object/extract path resolved a bucket's default
encryption with a hard-coded "no explicit SSE-C" flag, so the default was
layered onto a request that already carried an SSE-C header triple and
then tripped that request's own mutual-exclusion check. Every bucket with
default encryption refused SSE-C single PUTs with 400 InvalidArgument,
while CreateMultipartUpload on the same bucket succeeded because it
resolves SSE elsewhere. Both call sites now derive the flag from the
request headers, as COPY already did.
The bucket default's KMS key id was also inherited independently of the
effective algorithm, so an explicit AES256 request against an aws:kms
default bucket produced a self-contradictory algorithm/key-id pair and
was rejected. The key id is now inherited only when the effective
algorithm is aws:kms, matching the storage-layer resolver.
Refs backlog#2368 B1, B2.
* fix(sse): refuse SSE-KMS without a running KMS service
A write requesting aws:kms on a node with no KMS service fell back to the
node-local SSE-S3 provider: the data key was wrapped with
RUSTFS_SSE_S3_MASTER_KEY while the object metadata still recorded
aws:kms and the requested KMS key id. The stored object claimed a KMS
protection it never had, under a key that was never consulted, and no
signal distinguished it from a genuine SSE-KMS object.
The managed-encryption path now asks the resolved DEK provider whether it
wraps with a node-local master key and refuses SSE-KMS in that case:
InvalidRequest when KMS was never configured, ServiceUnavailable when a
configured service is not running. The check sits after the per-key
authorization gate so an unauthorized caller still receives AccessDenied
whatever the KMS runtime state is, and asks the provider rather than a
parallel availability signal because the provider is what actually wraps
the key. A missing master key no longer answers an SSE-KMS request with
an SSE-S3-worded configuration error.
The SSE-S3 local fallback is unchanged.
Refs backlog#2368 B4.
* fix(ecstore): restore and archive tiers in stored coordinates
Multipart restore addressed the remote tier in plaintext coordinates
while the copy-back reads the stored representation. Each part received a
misaligned slice of the remote object whose length still satisfied the
range, the hash reader and the completion size check, so the restore
reported success and silently replaced the object's bytes. Encrypted and
compressed multipart objects were both affected. Restore now accumulates
stored part sizes, passes the stored length to the hash reader alongside
the plaintext length, and validates against the stored size.
The copy-back digests stored bytes, so its computed MD5 is not the
object's public ETag. Restore now preserves the object ETag on both the
single-part and multipart paths, and gives each restored part its own
recorded part ETag rather than the object-level value.
Transition also handed the tier the object's SSE headers and its
RustFS-wrapped data key as request headers. Any S3 target rejected an
SSE-C archive outright, an SSE-KMS archive asked the target to encrypt a
second time under a key id it does not own, and the wrapped DEK left the
cluster. The archive request now strips every SSE header and encryption
marker with the predicate the replication path already uses; the local
xl.meta keeps all of it, so read-through and restore are unaffected.
Objects restored by an affected release are not detected or repaired
retroactively and must be re-restored from the tier.
Refs backlog#2368 B3, B5; backlog#2369 P7.1.
* fix(rio): lock the v1 nonce layout within a segment
Decrypting a v1 segment tried three historical nonce layouts per frame,
independently for every frame. The last of them exists for streams
written before 1.0.0-alpha.91, which reused a segment's part nonce for
every block in it; because block zero's derived nonce equals that base
nonce, a frame encrypted at index zero authenticated at any position. An
attacker able to rewrite the underlying shards could replay it and have
the forged plaintext returned with 200 and an unchanged length. Shard
integrity uses a keyed-hash-free checksum, which such an attacker can
recompute, so it is not a barrier.
A segment now locks onto whichever layout decoded its first non-zero-index
frame and rejects any later frame needing a different one. That leaves one
residual shape: a stream built purely from repeats of frame zero has no
later frame to disagree. New RUSTFS_ENCRYPTION_LEGACY_NONCE_FALLBACK
(default true, so pre-alpha.91 objects keep decrypting) drops the third
layout entirely when set to false, which closes it. Turning it off refuses
pre-alpha.91 objects, so migrate them first by rewriting in place.
Refs backlog#2369 P2.
* fix(kms): reload a service that failed to start
POST /rustfs/admin/v3/kms/reload short-circuited whenever the persisted
configuration matched the in-memory one byte for byte. A node whose KMS
failed to start keeps that configuration and sits in Error, so the
documented recovery call returned "reloaded successfully" while leaving
the node down. Peers reached the same path through the reload broadcast,
so a cluster that lost Vault during a rolling restart had no working
recovery route other than the node-local start endpoint. Reload now
short-circuits only for a service that is actually running, and otherwise
reconfigures, which starts a service that is not running.
The AWS backend also advertised key-version enumeration through
kms/status, which its own documentation says it cannot do; the capability
and its golden snapshot now say false.
Refs backlog#2369 P1, P7.3.
* docs: record the SSE and KMS changes for 1.0.0
The Unreleased changelog section carried no entry for any encryption work
merged since 1.0.0-rc.5, including three items with operational impact:
the config-secret variable whose absence persists secrets in cleartext
with only a warning, the v2 frame write switch and its rolling-upgrade
constraint, and per-key authorization making a public bucket incompatible
with SSE-KMS objects. Adds those plus this batch, including the SSE-KMS
refusal as a breaking change with both routes out.
Also corrects four places where documentation contradicted the code: the
cleanup register still called encrypted range seek opt-in after its
default flipped, the Helm README claimed vault_mount_path only applies to
Transit while the template also feeds the KV2 mount, the disaster-recovery
drill listed bundle contents for backends whose export is refused with
501, and the Chinese README capability table predated most of the feature
set. Documents the SSE-S3 local master key as a first-class operational
mode with its rotation dead end, and what the v1 frame layout does and
does not authenticate.
Refs backlog#2369 P5.
* fix(kms): classify data-path KMS failures by what the caller can do
Only "key not found" and a backend outage were classified; every other
KMS failure that reached the S3 data path fell through to
500 InternalError with a generic message. A disabled or pending-deletion
key, a denied KMS grant, an encryption-context mismatch, an unsupported
algorithm, a credential or timeout failure, and a capability the
configured backend does not have all looked identical to a server fault.
SDKs therefore applied exponential backoff to configuration errors no
retry can fix, and monitoring counted every one of them against the
server's own error rate.
Unusable-key and request-side failures now answer 400, a denied grant
403, transient backend failures 503, and a missing backend capability
501. Damaged, unreadable, or unknown-format key material keeps its 500:
it is a server-side integrity fault, and existing tests pin it.
The classifier is deliberately separate from the admin lifecycle
mapping, which answers 404 for a missing key because there a key id is
the resource being addressed; on the data path it arrives inside a
request header or a bucket default. Messages either name what the caller
asked for or stay generic, with deployment-side detail left on the error
source the way the storage-IO mapping already does.
Refs backlog#2368 B6.
* fix(kms): track and renew static Vault tokens
Token authentication hard-coded "this token carries no lease", so the
renewal task never started, the remaining-TTL gauge was never published,
and nothing looked wrong. `vault token create` grants a 768-hour TTL by
default, so a cluster that had been healthy for a month turned every KMS
call into a 403 and could not recover without a restart or a
reconfigure. Production configuration validation only rejects the
literal dev-token, so an ordinary expiring token reaches a whole cluster.
The source now reads `auth/token/lookup-self` at login and adopts what
Vault reports. A token with no expiry behaves exactly as before. An
expiring renewable one is picked up by the existing renewal loop and
renewed at half TTL like every other auth method. An expiring
non-renewable one warns with its remaining lifetime and publishes the
gauge, so the fail-closed window is visible before it arrives.
The probe never fails the login: a policy that omits lookup-self, or a
Vault that is briefly unreachable, warns and falls back to exactly the
previous behaviour rather than taking down a deployment that works
today. The scripted Vault test double answers the lookup out of band so
existing scripts keep describing only the protocol under test.
Refs backlog#2369 P3.
* feat(sse): report SSE-C requests that arrive without TLS
An SSE-C request carries the customer's AES key in a request header, so
AWS S3 and MinIO both refuse one that did not arrive over TLS. RustFS
accepted them on any transport: a plaintext hop hands the key to anyone
on the path, and since the object cannot be read without that same key,
the exposure lasts as long as the object does.
Refusing outright is the correct end state but not a safe default to
adopt inside a release window, because the project's own s3-tests and
e2e lanes and most staging deployments speak plain HTTP. This release
reports instead: each such request increments
rustfs_ssec_plaintext_requests_total and logs one warning per process, so
an operator can confirm nothing would break before the default flips.
RUSTFS_SSE_C_REQUIRE_TLS=true opts into the AWS 400 now.
The verdict is per connection rather than per deployment: the layer is
built with whether this listener terminated TLS, and additionally accepts
an https protocol forwarded by a proxy the trusted-proxy configuration
already vetted. It sits beside the rate limiter, after the layer that
makes a forwarded protocol trustworthy and after the request context, so
a rejection can echo the request id.
Refs backlog#2369 P7.2.
* fix(kms): say what a node-local backend means for a cluster
The Local backend keeps key material on each node's own disk and
generates its Argon2id salt per node, so two nodes derive different keys
from the same master_key and an object encrypted on one node cannot be
decrypted on another. Behind a load balancer that surfaces as
intermittent 500s on reads that succeeded moments earlier, with nothing
tying the symptom to the cause: the only signal was a generic
"development, testing and demos only" positioning warning that says
nothing about what actually breaks.
Configuring or reconfiguring Local while the deployment is distributed
now logs a dedicated event and appends the consequence to the configure
response, so the operator who made the change sees it. The product
decision to warn rather than refuse is unchanged.
Refs backlog#2369 P7.4.
* docs: record the remaining SSE and KMS changes for 1.0.0
Adds changelog entries for the KMS data-path status classification, the
Vault static-token lease probe, the SSE-C plaintext-transport report and
its switch, and the node-local backend warning.
Documents two things the backend security guide never stated: that SSE-C
belongs on a secure transport, with the counter and switch to plan the
change around, and that the Local backend cannot be shared by a
multi-node deployment because each node derives different keys from the
same master key.
Refs backlog#2368 B6; backlog#2369 P3, P5, P7.2, P7.4.
* fix(kms): report an unreadable key store as an outage on the S3 path
A backend now distinguishes a key store it could not read from a key
that is genuinely absent, but the S3 boundary collapsed the first one
back onto 500 InternalError through the fallthrough for integrity
faults. The distinction was therefore invisible to the client: a
temporary key-directory outage looked exactly like a permanently damaged
key record, and neither the status nor the metric said the request was
worth retrying.
An unreadable key store joins the retryable class and answers 503, next
to a backend error and a credential failure. Damaged, unreadable or
unknown-format key material keeps its 500.
Refs backlog#2368 B6; builds on rustfs/rustfs#7470.
* test(storage): cover MRF cleanup delete ENOSPC anchor (#7513)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require heal retry stats evidence (#7514)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add release bundle dry-run fixture (#7515)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): confirm recovery intent accept readback (#7516)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF cleanup delete ENOSPC evidence (#7518)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require bounded retry window evidence (#7517)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): emit scanner heal G09 upgrade evidence (#7519)
Record measured Scanner/Heal G09 evidence artifacts from the upgrade compatibility lanes when a fresh evidence directory is provided.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): persist MRF rollback mirror boundary (#7520)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add G09 upgrade evidence runner (#7522)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(ci): bind performance runs to selected inputs (#7512)
* test(scanner): add G09 upgrade evidence runner
Add a reusable Linux x86_64 runner for the Scanner/Heal G09 mixed-version and rollback upgrade evidence lanes.
The helper reads the pinned previous-release asset metadata from the upgrade workflow, verifies the downloaded binary, builds the current head, runs both ignored E2E tests, and fails unless the expected G09 JSON artifacts exist.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): implement V2 evidence preflights (#7523)
* feat(scanner): wire dirty usage producer identities
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add segment activation preflight proof
Keep scanner segment reuse behind a structured activation preflight so release evidence can prove the production gate remains disabled until every producer, generation, overflow, cold-oracle, and distributed invalidation check is satisfied.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): expose distributed invalidation evidence
Record an explicit distributed segment invalidation evidence summary when remote dirty usage snapshots are bound to the current activity window and the authenticated scoped ACK capability probe succeeds.
Reject peer dirty usage snapshots that contradict the peer activity pending bit so scoped ACKs fail closed instead of clearing an unadvertised remote mutation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): assemble release evidence bundles (#7526)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add G09 upgrade evidence runner (#7524)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(ci): bind performance runs to selected inputs (#7512)
* test(scanner): add G09 upgrade evidence runner
Add a reusable Scanner/Heal G09 runner for Linux PR-head validation. The script downloads the pinned previous release, builds the current checkout, runs the mixed-version and rollback upgrade E2E lanes, and validates the measured raw artifacts before they can be consumed by the release bundle gate.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): replay recovery intents while disabled (#7521)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): preflight G09 evidence disk space (#7527)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics…
* fix(list): reconcile split overwrite metadata (#7910)
fix(list): fail closed on split overwrite metadata
* fix(heal): persist explicit delete-marker purges (#7908)
* fix(heal): persist explicit delete-marker purges
* fix(ecstore): remove unused object info wrapper
* test(rpc): use valid incarnation in digest gate
---------
Signed-off-by: Hauser <housemecn@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
* ci(table): pass --outcome to the backlog issue manager (#7914)
The manager step failed with 'the following arguments are required:
--outcome' in the first table run (hidden by continue-on-error): every
other suite workflow passes the test step outcome through.
* fix(list): reconcile encoded directory objects (#7915)
* fix(ci): restore release checks after marker purge changes (#7918)
* fix(rpc): recover tier reloads after peer capability revocation (#7920)
* test(ecstore): drain expiry workers before restart (#7923)
* test: report tier cleanup recovery progress on timeout (#7922)
* fix(heal): resume replacement generation after target restart (#7926)
* ci(release): use sm-standard-4 for Linux builds (#7925)
* chore(deps): refresh release dependencies
Update the release branch dependency set and lockfile for the latest compatible AWS Smithy and RustFS allocator patch versions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(rustfs): backport skip of redundant content-sha256 pass (#7930)
perf(rustfs): skip redundant content-sha256 pass on signed PUTs (#7928)
A SigV4 header-signed or presigned PUT whose x-amz-content-sha256 header carries a real digest is already wrapped in a hash-checking UploadStream by the s3s auth layer, so the storage-side HashReader was hashing the same payload a second time. Hand the digest to the storage layer only when it arrives in a presigned query string alone, the one case s3s does not read.
(cherry picked from commit 95c06966cc)
* chore(deps): bump qrcode-rs to 2.1.1
* fix(ci): preserve tier failure enforcement after release merge
* docs(release): record 1.0.0 stable RPM release
---------
Signed-off-by: houseme <housemecn@gmail.com>
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Hauser <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: cui fliter <imcusg@gmail.com>
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: xiaomage <devops008@sina.com>
Co-authored-by: GatewayJ <835269233@qq.com>
Co-authored-by: hehutu <hehutu@gmail.com>
Co-authored-by: Henry Guo <marshawcoco@gmail.com>
Co-authored-by: RJ Regenold <rregenold@teamraft.com>
Co-authored-by: RJ Regenold <214054+rjregenold@users.noreply.github.com>
Co-authored-by: Jason Kossis <jkossis@gmail.com>
Co-authored-by: Hiroaki KAWAI <1468181+hkwi@users.noreply.github.com>
Co-authored-by: Hiroaki KAWAI <hiroaki.kawai@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
The fault-tolerance suite (disk-outage and peer-failure scenarios,
FT-CASE verdicts) has never been dispatchable: the workflow file only
exists on release, and GitHub registers workflow_dispatch targets
exclusively from the default branch. The suite is otherwise ready - it
runs on the smoke-testing fleet and already uses the
rustfs-shared-functional-tests-v2 lock.
Adds the workflow adapted to this branch's current (pre-sync) gate
semantics - the run step's continue-on-error is dropped so the existing
contract holds - plus the evidence init aligned with the shared
four-variable pattern (FUNCTIONAL_ARTIFACTS_DIR under
rustfs-fault-tolerance-*, LOG_FILE/REPORT_FILE/EVIDENCE_DIR/TMPDIR,
scratch dir) and the standard artifact allowlist.
Contract registration: JOBS / DIRECT_TESTS entries, upload allowlist,
and fixture teaching for the FT-specific shapes (composite
package-url fallback expression, script-driven --cleanup instead of
ssh-driven cleanup). The generic legacy matrices exclude FT the same
way they exclude table/performance until the release->main sync lands.
A SigV4 header-signed or presigned PUT whose x-amz-content-sha256 header carries a real digest is already wrapped in a hash-checking UploadStream by the s3s auth layer, so the storage-side HashReader was hashing the same payload a second time. Hand the digest to the storage layer only when it arrives in a presigned query string alone, the one case s3s does not read.
ci(chain): gate the performance lane on its fleet probe (rebase + contract)
Rebased onto latest main; adds the contract-test update that the first
push of this branch missed:
- test_functional_chain: the performance lane now asserts the new
probe-gated 'if' (prepare success AND performance_ready == online);
the other nine lanes keep the plain prepare gate; prepare must carry
the two preflight steps (hard smoke-testing check, pf-testing probe)
and publish the performance_ready output
Carried over unchanged from the previous push: per-fleet preflight
(prepare hard-fails only on smoke-testing), the pf-testing probe step,
the probe-gated performance lane, and aggregate's --allow-skipped
(permitted skips shrink the evidence set; unpermitted skips still fail
aggregation). Chain contract tests: 27/27 locally.
* fix(ecstore): admit transformed small objects inline by plaintext size
* test(e2e): expect the small compressed object inline in compression roundtrip
---------
Co-authored-by: Hauser <housemecn@gmail.com>
* chore(deps): refresh RustFS tokio tar dependency
Switch the astral-tokio-tar workspace alias to the released rustfs-tokio-tar 0.7.1 crate and remove the obsolete cargo-deny git source allowance.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* chore(deps): align AWS smithy dependency family
Upgrade the AWS SDK and smithy crates together while patching aws-smithy-json 0.63.0 for the aws-smithy-types 1.7.0 DocumentObject API used by aws-config 1.12.0.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* revert(deps): use official AWS smithy compatible set
Remove the local aws-smithy-json patch and roll the AWS smithy family back to the latest official combination that compiles with aws-config 1.12.0.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* docs(architecture): retire tokio tar cleanup entry
Remove the stale cleanup-register entry after switching back to the released rustfs-tokio-tar crate, leaving no unmatched compatibility marker for the architecture guard.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* ci: restore table workflow failure contract
Align the table functional workflow with the quick-check guard by letting suite failures stay red, generating PASS reports only from complete successful evidence, and preserving the shared backlog issue fallback for harness failures.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(deps): expose rustfs tokio tar dependency name
Use the published rustfs-tokio-tar dependency key throughout the workspace while keeping source imports on the crate's tokio_tar lib target name.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* ci: register the table suite workflow on the default branch
The table suite workflow (#7873) only exists on release so far, and
GitHub registers workflow_dispatch targets exclusively from the default
branch — the suite cannot be triggered until this file lands on main
(same mechanism that keeps the fault-tolerance suite undispatchable).
This adds the identical rustfs-table-test.yml (byte-for-byte the
release version) plus the contract-test registration (JOBS /
DIRECT_TESTS / upload allowlist), and nothing else from release.
* ci: pin actions in the table workflow to full commit SHAs
The default branch enforces check_workflow_pins.sh (unpinned
third-party action refs fail CI). Use the same pinned SHAs as the other
suite workflows on main: actions/checkout@9c091bb... (#v7) and
actions/upload-artifact@b7c566a... (#v6).
Note: the release copy of this workflow still uses @v4 tags; release
does not run the pin gate. Release and main will converge on the pinned
form during the full sync.
* test(heal): keep outage objects in the replacement erasure set
* fix(rpc): retry authenticated requests after peer epoch changes
* test(rpc): assert retryable rejection of stale boot epochs
* ci: bound concurrent heal clusters in the full E2E gate
* fix(heal): retry interrupted internode shard writes
* fix(deps): update rustls for RUSTSEC-2026-0285
* fix(heal): retain RPC transport failures across metadata probes
* test(heal): wait for replacement disk and coordinator readiness
* test(connect): validate service-only diagnostic capabilities
* fix(s3): resume GET bodies after erasure read quorum loss
#7847 routed schedule-failure alerting to rustfs/backlog by switching the
alert jobs' token to secrets.BACKLOG_ISSUE_TOKEN, but check_test_wiring.py
still asserts the literal 'github-token: ${{ secrets.GITHUB_TOKEN }}', so
Quick Checks fails on every open PR. Update the wiring assertions and the
self-test fixtures to the new literal.
Both `--self-test` (52 tests) and the full `validate()` pass locally.
* chore(deps): bump ed25519-dalek in the dependencies group
Bumps the dependencies group with 1 update: [ed25519-dalek](https://github.com/dalek-cryptography/curve25519-dalek).
Updates `ed25519-dalek` from 2.2.0 to 3.0.0
- [Release notes](https://github.com/dalek-cryptography/curve25519-dalek/releases)
- [Changelog](https://github.com/dalek-cryptography/curve25519-dalek/blob/3.0.0/CHANGELOG.md)
- [Commits](https://github.com/dalek-cryptography/curve25519-dalek/compare/ed25519-2.2.0...3.0.0)
---
updated-dependencies:
- dependency-name: ed25519-dalek
dependency-version: 3.0.0
dependency-type: direct:production
update-type: version-update:semver-major
dependency-group: dependencies
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): bump jiff, aws-sdk-s3, datafusion and smallvec
Bump the dependencies group:
- jiff 0.2.35 -> 0.2.37
- aws-sdk-s3 1.146.0 -> 1.146.1
- datafusion 55.0.0 -> 55.1.0
- smallvec 1.16.0 -> 1.16.1
Update Cargo.lock for the matching transitive bumps (datafusion
subcrates, hickory, cc, crc32fast, tinyvec, toml_edit and others);
tinyvec_macros is no longer required.
* chore(deps): use s3s 0.16.0 release and bump rust-version to 1.98.1
* fix: preserve ready reader errors during erasure write failures
When erasure writers fail after the producer has already surfaced a reader error, return the reader error instead of aborting the producer and masking it with the write-side failure.
This keeps UploadPart body read timeouts classified as RequestTimeout under loaded pipeline timing.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test: use current checksum API in multipart regressions
* fix: stabilize connect dependency update tests
Classify Connect proxy failures only when an explicit proxy is configured, preserve TLS classification across request error shapes, keep short object performance windows from spending too much budget on cleanup, and update multipart checksum tests for the current HashReader API.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Hauser <housemecn@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix(auth): reject unsigned x-amz headers on header-signed SigV4 requests
A SigV4 request authenticated with an Authorization header only binds the headers named in its SignedHeaders list, but RustFS acted on every x-amz-* header that arrived, so a replayed header-signed PutObject carrying an unsigned x-amz-copy-source became a CopyObject run as the signer that could copy any object the signer can read (GHSA-xm99-m3gq-83g8). The presigned form was already closed by GHSA-g8w9-qw9q-fghr.
reject_unsigned_amz_headers_on_sigv4_request now guards S3Access::check and S3Router::check_access: every SigV4 signed-header list the request carries must cover every x-amz-* header, the Authorization header is parsed with the verifier's own s3s-sigv4 parser and compared case-insensitively, the algorithm token is pinned to AWS4-HMAC-SHA256 because the upstream header path accepts any token, and the exempt set mirrors the upstream s3s fix (x-amz-content-sha256, x-amz-decoded-content-length, x-amz-trailer, x-amz-checksum-algorithm) plus x-amz-cf-id. Adds ghsa_xm99 unit, router and e2e regressions, raises the security smoke floor to 28, and records the advisory in docs/testing/security-regressions.md and CHANGELOG.md.
* chore(deps): switch s3s and s3s-sigv4 to crates.io 0.16.0
* fix(server): enforce the SigV4 header guard ahead of s3s dispatch
s3s 0.16 verifies the claimed algorithm as the first step of its own
signature flow, so a request whose Authorization header swaps the
AWS4-HMAC-SHA256 token was answered with 501 NotImplemented before
RustFS's access layer could rule on the unsigned x-amz-copy-source
(GHSA-xm99-m3gq-83g8). Add the SigV4HeaderGuardLayer as the innermost
external stack layer, running reject_unsigned_amz_headers_on_sigv4_request
in front of s3s and serializing its rejections as the same AccessDenied
S3 error document the access layer produces.
---------
Co-authored-by: Hauser <housemecn@gmail.com>
The gateway key inventory handler added in #7785 imported s3s directly and built four errors with s3_error!, pushing the s3s footprint ratchet to 210 files and 1592 s3_error! lines on main. Route its S3 types and error construction through the crate::admin::storage_api::s3 facade like sibling admin handlers; the codes, HTTP statuses and messages are unchanged, and a new unit test pins them.
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Scheduled pipeline failures currently open [scheduled-failure] tracking
issues in rustfs/rustfs itself, adding bot noise to the code repository's
issue tracker. Route that alerting to rustfs/backlog instead.
- schedule-failure-issue gains target-repository (default rustfs/backlog);
dedupe lookup, comment appends and issue creation target that repository.
- New source-token input (default ${{ github.token }}) reads the failed-job
list from the reported run; the run still lives in and links to
rustfs/rustfs. The issue body now carries a '- Repository:' line.
- All consumer workflows pass secrets.BACKLOG_ISSUE_TOKEN; the
workflow-scoped GITHUB_TOKEN cannot open issues cross-repo. Alert job
permissions blocks are unchanged.
* fix: resume get body after short eof
Treat mid-stream short EOF errors as resumable when the S3 GET body has an attached resume context. This lets peer-kill reads reopen the same object version at the emitted offset instead of aborting the 200 response body.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(connect): surface TLS peer bootstrap failures
Map Connect registration TLS peer validation failures to the dedicated bootstrap error so wrong CA inputs fail closed with the sanitized TLS variant instead of the generic exchange error.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Only the Local backend refused a key identifier containing a path
separator. On Vault KV2 a create with the name bad/name succeeded and
produced a nested KV2 path that the listing then reported as a
directory rather than a key, and an identifier containing .. addressed
a record outside the configured key prefix once the HTTP client
normalised the URL; Vault Transit built its transit key name and its
metadata path from the same unchecked identifier.
Lift the Local backend's containment rule into a shared segment check
(empty, /, backslash, NUL, . and ..) and apply it at the single point
where each backend turns the identifier into a path or a Transit key
name, so create, describe, encrypt, delete and the metadata writes all
refuse with InvalidKey before any request reaches Vault. The admin API
already maps that to 400. The AWS backend is untouched: it addresses
keys by ARN and alias, both of which contain /.
Refs rustfs/backlog#2474 (KMS-213 CreateNegatives on vault-kv2).
Co-authored-by: Hauser <housemecn@gmail.com>
* fix(replication): accept a bodiless 405 as a replicated delete marker during resync
A resync verifies each delete marker with `HEAD ?versionId=<marker>` on the
target. S3 targets (RustFS, MinIO, AWS) answer that with 405 and no body,
and the SDK only synthesizes an error code for 404, so the error arrived
with `code() == None`. `is_retryable_delete_replication_head_error` then
treated it as an ambiguous failure: every delete marker counted as a failed
object with `target service error`, and a site resync over any bucket that
holds a delete marker reported the whole bucket as failed
(rustfs/backlog#2479, SITE-105 on rc.6 and nightly).
Use the raw HTTP status the way the 404 path already does: a 405 without a
code confirms the marker propagated. Ambiguous statuses still fail.
- Unit tests drive `verify_resync_head_result` against a scripted target
answering 405 (accepted) and 503 (still failed).
- e2e `test_site_replication_resync_replicates_delete_marker` joins two
sites, converges a live object and a delete marker, and requires the site
resync to complete with zero failed objects; the repl-nightly selection
digest is refreshed for the new case.
* fix(replication): verify marker-version purges by absence during resync
A delete-marker resync entry with a pending or failed version purge asks
the target to remove the marker, so the bodiless 405 that proves a
created marker propagated proves the purge did not happen. Accept the
405-as-success mapping only for marker creation (empty
version_purge_status); a purge counts as replicated only when the target
answers not found, and any other HEAD outcome stays failed. Unit tests
drive both purge statuses against the 405 fixture and the 404 fixture.
* fix(io-metrics): drop the rustfs-common edge by injecting the S3 telemetry observer
dc700d366 (#7775) made the io-metrics leaf crate depend on rustfs-common
to publish typed telemetry trace events from S3HttpRequestGuard, which
breaks the backlog#1834 leaf-crate rule (io-metrics may only depend on
rustfs-s3-ops).
S3HttpRequestGuard now takes an optional fn-pointer completion observer
(operation, latency, 2xx?) and knows nothing about the trace bus. The
server builds guards through rustfs::server::s3_http_request_guard, which
attaches the observer only while a telemetry subscriber exists and owns
the S3Operation -> TelemetryTraceOperation mapping.
* chore(guard): admit #7785 s3s ratchet growth (+1 file, +4 lines)
#7785 landed the gateway key inventory admin handler after the
baselines were verified, leaving check_s3s_footprint.sh red on main
and every branch cut from it (measured 210 files / 1592 lines vs
209/1588 baselines).
The handler follows the house admin convention whose Operation::call
signature is s3s-typed (S3Request/S3Result), so it cannot route
through a non-s3s seam until the s3gate admin migration replaces the
admin router (rustfs/backlog#1677 F1); no local refactor can shed the
file-level import. Record the measured growth with rationale:
files 209 -> 210, s3_error! lines 1588 -> 1592.
* fix
* fix: resolve clippy gate failures in connect and CLI surfaces
- Box the two large CommandResult performance variants and
LicenseRenewalOutcome::Installed (clippy::large_enum_variant)
- Return RelayError instead of () from RelayTransport::deliver and the
test destination (clippy::result_unit_err)
- Drop an unused mut on the protected relay file handle (unused_mut)
- Pass the digest by value to base64 encoding and drop a redundant
result_json clone (clippy::needless_borrows_for_generic_args,
clippy::redundant_clone)
* fix(connect): resolve the remaining site-replication clippy errors
- Group the endpoint credential triple into SiteReplicationCredentials
so SiteReplicationEndpoint::new takes 6 arguments
(clippy::too_many_arguments)
- Take the build-feature validator by &str (clippy::ptr_arg); the
validator closure stays because &String needs the deref coercion
* style: apply rustfmt to the site-replication credential grouping
* fix(connect): keep the CPU profile fixture off the ecstore import scan
#7811 used "rustfs_ecstore::disk::read_object" as the raw-symbol fixture
of the profile redaction test, which the architecture guard counts as a
direct rustfs_ecstore reference outside the compatibility boundaries
(its scan deliberately includes inline test modules). The accumulator
treats symbols opaquely, so use the internal rustfs::storage:: path that
keeps the fixture realistic without matching the import scanner.
* fix
* fix
* fix
* fix
---------
Co-authored-by: Hauser <housemecn@gmail.com>
* fix(connect): restore the site replication producer build
The producer shadowed its object_url helper with a local of the same
name, and its test placed the consent nonce on the request, so neither
the rustfs library nor its tests compiled on main.
* feat(server): add RUSTFS_S3_STACK with a gateway GetBucketLocation path
RUSTFS_S3_STACK=legacy (default) keeps the s3s service as the whole S3
entry. RUSTFS_S3_STACK=gateway routes GetBucketLocation through the
RustFS Gateway pipeline pinned at rustfs/gateway@90b83a20 and falls back
to the same s3s service for every other request, decided before the
body is read. Refs rustfs/backlog#1752.
GET /rustfs/admin/v3/gateway-key-inventory (admin:InspectData) lists
every stored object key the RustFS S3 gateway would refuse on every
operation, so an operator can copy those objects to a safe key before
switching stacks. The rules mirror the gateway key floor; . and ..
segments never appear because ecstore refuses them on every write.
Refs rustfs/gateway#754
Replace s3s::dto::{RestoreStatus, Timestamp} in filemeta with a
filemeta-owned RestoreStatus (same field names, OffsetDateTime expiry).
RestoreStatusOps and parse_restore_obj_status keep their signatures and
persisted rendering; the ecstore restore finalize / lifecycle restore
writers and rustfs RestoreObject now build the filemeta type.
- crates/filemeta: drop s3s; tokio "time" becomes a dev-dependency
(the metacache tests got it through s3s feature unification).
- metadata_keys: drop the s3s half of the historical-source cross-check;
PINNED, the pre-A3a fixture and per-character mutation still pin keys.
- new test re-renders the pre-A3a fixture restore value byte-for-byte.
- s3s footprint baselines 210 -> 209 files, ecstore 37 -> 36.
Refs rustfs/backlog#1735
#7769 left rustfs/src/connect/diagnostics/perf_site_replication.rs failing
to compile: a local binding named object_url shadowed the object_url()
helper in the same scope (E0618), and a test put the consent nonce on the
request instead of LocalSiteReplicationConsent (E0063/E0560).
Rename the binding to source_url and move the test nonce into the consent.
Migrate ecstore and rustfs-lifecycle consumers of the persisted xl.meta
meta_user keys (object lock, restore, replication status, storage class,
SSE read) from s3s::header / rustfs_utils header constants to
rustfs_filemeta::metadata_keys. HTTP header production is unchanged and
every lookup keeps its previous exact / case-insensitive mode.
- SSE key: persisted only as lowercase; the mixed-case spelling is
outbound replication user metadata. Keep the case-insensitive read and
pin it against the pre-A3a fixture.
- warm tier: strip promoted keys case-insensitively so the persisted
X-Amz-Replication-Status is no longer forwarded to the tier.
- fix stale HashReader::add_checksum_from_s3s call in ecstore tests.
Refs rustfs/backlog#1735
Close the remaining paths where a stored bucket sub-configuration whose
bytes cannot be parsed still read as absent (rustfs/backlog#1734, slice 2):
- Object writes resolve versioning through BucketVersioningSys::get_for_write;
RUSTFS_BUCKET_CONFIG_PARSE_MODE=strict refuses them, the default permissive
mode keeps the historical unversioned write and records it.
- Versioning, Object Lock, encryption, public access block and notification
getters and the delete-time versioning check return the typed
UnreadableBucketConfig refusal, which ApiError maps to 503 naming the
bucket, config and stored length. StorageError::clone keeps it typed.
- Object Lock checks stay on when the lock config is unreadable.
- Notification setup isolates an unreadable config to its bucket instead of
clearing its rules; GetBucketNotificationConfiguration reports it.
- Add rustfs_bucket_metadata_parse_failed_total and
rustfs_bucket_metadata_unparsable_current, an invalid-mode startup check,
and the s3gate-parse-strict compat register entry.
Move the aws-chunked trailing checksum handle behind a RustFS-owned rustfs_rio::TrailerSource (Pending/Missing/Present lookup with a documented EOF timing contract) and adapt s3s::TrailingHeaders once in the application crate. rio no longer depends on s3s; behaviour is unchanged.
Refs rustfs/backlog#1735
Add rustfs_filemeta::metadata_keys as the single source of the nine xl.meta meta_user keys, byte-identical to current literals, and migrate filemeta consumers off s3s::header and rustfs_utils header constants. Pin the literals, cross-check historical sources, and prove old xl.meta bytes still decode via a captured fixture and a per-character mutation sweep.
Refs rustfs/backlog#1735
An XML bucket sub-config whose stored bytes cannot be parsed was still folded into "absent" on two paths: update_config_with handed it to mutate, which rebuilt it from nothing and overwrote the only copy of the bytes, and the tagging/lifecycle/CORS/website/logging/accelerate/request-payment getters reported ConfigNotFound.
Add an explicit ConfigState (Absent / Valid / Unreadable) over the existing retained parse failure, refuse the read-modify-write of an unreadable target config before mutate runs (per config, other configs stay writable), and make those getters fail closed. GetBucketLifecycle now only maps ConfigNotFound to NoSuchLifecycleConfiguration. The Swift tagging rewrite recognizes the refusal by type instead of a string sentinel.
Refs rustfs/backlog#1734
* fix(site-replication): keep an operator's bucket-level target to a peer instead of taking it over
Site replication wired each bucket by looking for an existing replication
target "to the same peer" and rewriting the first match in place as its own
same-name target. An operator's bucket-level target that happened to point
at that site (different target bucket, operator credentials) was the first
match whenever it pre-dated the join, and the reconciler repeats the pass
every 600s, so the takeover also depended on target order afterwards. The
operator's rule then named an ARN no target backed and their bucket
replication stopped silently, while the inherited bucket-level reset id
made every site resync report the bucket as owned by another resync
(rustfs/backlog#2479, rustfs/backlog#2489).
Follow MinIO's `getRemoteARN` / `getRemoteARNForPeer` shape instead:
- Wiring updates a target in place only under the same ARN, or when it is
recognisably the site's own under an older ARN shape (same peer,
same-name target bucket, site replication service account). Anything
else gets the site target added next to it.
- The site resync manifest takes the target the derived
`site-repl-<deployment id>` rule names (same-name shape as fallback), so
an operator target to the peer neither aborts the bucket as "multiple
remote targets matched peer" nor gets resynced into.
- Peer removal prunes only targets a pruned derived rule names or the
same-name target bucket; operator targets stamped with the peer's
deployment id survive together with their rules.
Unit tests cover the three predicates. e2e
`test_site_replication_keeps_operator_bucket_target_to_peer` runs a
bucket-level replication plus `replication-reset` to the future peer, joins
the sites, and requires the operator target untouched, both paths
delivering, the site resync completing against the site target, and the
operator target and rule surviving `replicate remove --all`; without the
fix it fails at the join with the operator target gone. The repl-nightly
selection digest is refreshed for the new case.
* test(site-replication): drop a redundant clone flagged by clippy
The reconcile unit test cloned the remote peer into the state map although
the binding is not used afterwards; workspace clippy (-D warnings) rejects
that as redundant_clone.
* fix(sse): classify bare SSE-KMS writes when no KMS is available
A `aws:kms` request without a key id, on a bucket without a default key,
returned `500 InternalError` whenever no KMS service was running: the
"no KMS key available" branch exited with an untyped storage error before
the availability classification that the keyed form already received.
Route that branch through the same split: `503 ServiceUnavailable` while
a configured KMS is stopped, `400 InvalidRequest` when KMS was never
configured, and `400 InvalidRequest` naming the missing key id when a
running KMS has no default key. `CreateMultipartUpload` shares the path.
Adds a unit test for the bare form and an e2e module that stops KMS
through the admin API, runs a master-key-only node, and runs a Local KMS
without a default key; refreshes the e2e-full selection digests.
(cherry picked from commit c3259dadc3d603a9185a5b0ad9f83dfb884e61c8)
* fix(sse): keep KMS error classes on the encrypted read path
GetObject, CopyObject and UploadPartCopy on an SSE-KMS object whose key
no longer exists answered `500 InternalError` ("KMS key not found") while
PutObject under the same key already answered `400 KMS.NotFoundException`.
The read path carries its classification through ecstore's
`EncryptionResolutionErrorKind`, which had no kind for a missing key, a
denied KMS grant or a missing backend capability, so all three folded
onto `DecryptionFailed` and the S3 layer reported an internal fault.
Add `KeyNotFound`, `AccessDenied` and `NotImplemented` kinds, map them on
both sides of the boundary, and give an envelope the configured backend
cannot unwrap a diagnosable message while keeping its `500`.
Unit tests cover the kind round trip and the reader wrapping; a new e2e
test deletes a key immediately and checks GET/Copy return 400 with
`KMS.NotFoundException` while HEAD stays 200. The e2e-full selection
digests are refreshed from the current listing (the previous digests
predated the delete-authorization tests) and the e2e `create_default_key`
helper is updated to the accepted `EncryptDecrypt` spelling.
(cherry picked from commit 2523a9814e97caea318d4ff1a51bef3a4d4445b2)
* fix(kms): classify key-management errors on the admin routes
`POST /kms/keys`, the legacy `create-key` alias and `generate-data-key`
reported every backend refusal as `500`: a blank key name (which each
backend failed on differently, the Local backend by writing a key file
with an empty stem), a name already taken, an unknown key, a disabled key
and a capability the backend lacks. `delete` and the lifecycle routes
already classified the same errors.
Refuse a blank or whitespace name in `KmsManager::create_key` before any
backend sees it, and share one `KmsError` to status mapping across
create, delete and generate-data-key (400 for validation and key state,
404 for an unknown key, 409 for a taken name, 501 for a missing
capability, 500 only for damaged material). The XML-error routes carry
the same status explicitly since s3s derives none for a custom code.
The read-only Static backend now reports create, delete and
cancel-deletion as `UnsupportedCapability`, matching its rotate and
enable/disable answers, so the admin API returns 501 for all of them.
(cherry picked from commit e33cac5493c4d9d6662e0d2980b58ba2b24a6d1b)
* fix(sse): stop SSE-S3 responses from naming the wrapping KMS key
`x-amz-server-side-encryption-aws-kms-key-id` is defined for `aws:kms`
objects only, but PutObject, CopyObject, CreateMultipartUpload and
GetObject returned it for `AES256` objects too, carrying the KMS key that
wraps the SSE-S3 data key (the service default, or the literal `default`
on a node without KMS). The write paths copied `kms_key_id` from the
encryption material unconditionally, and the single-decrypt GET
classification did the same after resolving the key for authorization.
Add `EncryptionMaterial::response_kms_key_id`, which yields the id only
for SSE-KMS, use it at the four write-response sites, and gate the GET
classification the same way. CompleteMultipartUpload and HeadObject
already omitted the header.
Unit tests pin both directions; a new e2e test covers Put/Get/Head/Copy
and CreateMultipartUpload for AES256 with an aws:kms control. The
e2e-full selection digests are refreshed from the current listing.
(cherry picked from commit 29d793a63352b0b60fd53c565e80fdbede8964bb)
* fix(s3): validate PutBucketEncryption rules before storing them
A default-encryption rule naming an unknown `SSEAlgorithm` (for example
`AES128`), a rule without `ApplyServerSideEncryptionByDefault`, an empty
rule list, or a `KMSMasterKeyID` on an `AES256` rule was stored as
written: the only algorithm check on the route decided whether to fill
in the default KMS key. `GetBucketEncryption` then advertised that
configuration while the write path encrypted header-less writes under
its `AES256` fallback, so the bucket's declared and actual schemes
disagreed. Two comments claimed the route already refused unknown
algorithms.
Validate the configuration before any of it is applied: `MalformedXML`
for a malformed rule set or unknown algorithm, `InvalidArgument` for a
key id on a non-KMS rule, and nothing stored on refusal. Correct the two
comments to describe when the AES256 fallback is still reachable.
Unit tests cover every refusal and the accepted shapes; an e2e test
checks the refusals leave the previous configuration in place. The
e2e-full selection digests are refreshed from the current listing.
(cherry picked from commit 29e4486dce41197ed93f5253cdbabc57d27a4ddb)
* test(e2e): refresh e2e-full selection for the combined KMS/SSE fixes
* test: align two unit tests with the new KMS and bucket-encryption contracts
`scheduled_deletion_carries_a_deadline_and_can_be_cancelled` still
expects the state error (`InvalidOperation`) for cancelling a key that
is not pending deletion; only the Static backend's mutations moved to
`UnsupportedCapability`. The uninitialized-store PutBucketEncryption
test now sends a well-formed AES256 rule so it reaches the store lookup
instead of the new configuration validation.
* feat(ci): measure queue and execution time by run attempt
* fix(ci): bound timing samples and recognize the old workspace lane
* fix(ci): sample completed runs for stable timing comparisons
* fix(ci): share quick checks and lint workflows
* fix(ci): install actionlint from its verified release
* fix(ci): reject dependencies on required quick checks
* feat(test): verify the E2E server build and source identity
* test(e2e): register verified Darwin test membership
* test(e2e): record verified Linux receipt test membership
* test(e2e): record compiled Darwin receipt test membership
* test(e2e): record compiled Linux receipt test membership
* test(e2e): record Darwin e2e-full membership after merging main
* fix(test): route scanner/heal evidence E2E runs through the verified server binary
The evidence runners built rustfs with plain cargo and then ran e2e_test directly, which now fails without a run receipt. They build through scripts/e2e_binary.py and run the e2e_test invocations under e2e_binary.py run; the obsolete rustfs.features stamp is removed.
* docs(e2e): run server-backed e2e commands through the verified binary wrapper
* test(e2e): record Linux e2e-full membership from the branch CI listing
* test(kms): cover custom Vault Transit metadata locations
* test(kms): include a non-default Transit mount in live coverage
* test(ecstore): tolerate cleanup quorum transition
* test(ci): restore ten-suite chain expectations
* test(ecstore): tolerate cleanup quorum transition in both causal polls
The unversioned duplicate-delete poll in batch_transitioned_delete_uses_free_version_per_item reads exact metadata while the same asynchronous free-version cleanup removes per-disk copies, so it can hit the same transient InsufficientReadQuorum as the versioned poll. Share one helper between both polls; every other error still fails immediately.
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
Adds a fault-tolerance suite that verifies read/write behavior under
drive and node loss against the erasure-coding contract and snapshots
health-endpoint responses at every degradation tier. Scenarios derived
from product source (default_parity_count, erasure set sizing):
- A: single-node 4 drives (EC:2, read quorum 2): hide 1/2/3 drives
- B: multi-node 4x1 (one set of 4, EC:2): stop 1/2/3 nodes
- C: multi-node 4x4 (one set of 16, EC:4, read quorum 12): 1 node down
lands exactly on the read-quorum boundary; 2 nodes down breaks it
- C2: multi-node 4x4 with RUSTFS_STORAGE_CLASS_STANDARD=EC:8 (read
quorum 8, write quorum 9, lock majority 9): 2 nodes down puts reads
inside the reported divergence window (read quorum met while the
lock majority is broken)
By default a reads-refused-despite-met-read-quorum observation is
reported as known-divergence without failing the suite; the strict
input escalates it. Chain order becomes:
upgrade -> s3 -> kms -> tier -> storage -> heal -> pool -> security ->
replication -> fault-tolerance -> performance.
Expose verified OIDC username and email claims as display-only metadata on self-account responses while preserving the virtual parent as the authorization identity.\n\nKeep rustfs-madmin public response structs unchanged by adding the optional wire fields through private handler response wrappers.
* fix(s3): reject oversize single PUT early and map body errors to 4xx
A single PutObject above the 5 GiB single-request ceiling was only
rejected after the client had streamed 5 GiB into s3s's read-time body
budget, and the resulting BodySizeLimitExceeded surfaced from the erasure
writer as 500 InternalError. A body whose connection hit EOF before
Content-Length bytes arrived (hyper's IncompleteBody) was also a 500.
SDKs retry 500s, so one oversize upload was resent from offset 0 five
times.
- PutObject and UploadPart reject a declared length above
MAX_SINGLE_PUT_OBJECT_SIZE with 400 EntityTooLarge before reading the
body; the constant moves to rustfs_config so the s3s limit and the
admission check share one value.
- ApiError maps BodySizeLimitExceeded to EntityTooLarge and a hyper body
EOF to IncompleteBody across both io::Error conversions.
Fixes#7596.
* test(s3): cover UploadPart admission, aws-chunked length, real s3s limit
- Poll-counting test body proves PutObject and UploadPart reject a
declared size above the ceiling with zero body polls; exact-cap and
zero-length parts pass admission.
- A STREAMING-* aws-chunked PUT whose framed Content-Length exceeds the
cap is admitted when the decoded length is within it and rejected when
the decoded length is over it.
- The display-based BodySizeLimitExceeded matcher is checked against the
real error produced by the pinned s3s Body budget.
Embed RUSTFS_CONSOLE_BASE_PATH into console routes, browser redirects, and startup URLs so OEM builds can use their own namespace. Preserve the default for an unset or empty build variable and keep route tests prefix-aware.
* test(heal): cover MRF manifest CAS legacy transition (#7443)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ci): enable release branch checks and repair test imports (#7441)
* test(scanner): structure heal release evidence lanes (#7442)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): require segment producer identities (#7444)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): expose recovery intent identity (#7445)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): summarize failed heal perf reports (#7446)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(common): retain unmatched MRF repair proofs (#7447)
Add a recorded verified-repair consumer that discharges only exact retained anchors for the requested bucket while leaving unmatched proofs in the event ring. This gives the future durable successor writer a fail-closed primitive before any tombstone or GC path is enabled.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ci): route replication read plan through boundary (#7448)
* test(scanner): cover flat-bucket quantum fairness (#7449)
Add a production-entry scanner cohort regression that combines a wide flat bucket with a small bucket under a fixed object budget. The test keeps partial budgeted rounds unpublished, verifies small buckets are only marked after real execution, and confirms a later unbudgeted round can publish the complete aggregate.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): reuse same admission request id (#7451)
Keep a retried heal start with the same request id from bypassing admission deduplication when the transport replay cache is unavailable but the manager still owns the task.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require perf summary evidence fields (#7453)
Fail closed when measured passing Scanner/Heal ABBA summaries omit W10/W11 foreground pressure, lock wait, or attempt-cost evidence.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind release evidence fields (#7452)
Require explicit scanner/heal release evidence field contracts for the scoped ACK and mixed-version rollback gates.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): tag dirty usage producer identities (#7454)
Record production-facing segment invalidation producer identities when existing object-level dirty usage hooks observe PUT, CopyObject, DeleteObject/DeleteMarker, and CompleteMultipartUpload mutations. Keep the data non-authoritative and process-local so segment reuse activation still requires durable generation-window proof.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): gate release evidence bundles (#7457)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require hard evidence ABBA manifest (#7459)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(heal): add MRF committed snapshot writer (#7458)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover EC8+4 restart shard rebuild (#7461)
Retry heal-control RPCs once after transport auth rejects a stale replay-scope epoch, and add a distributed EC8+4 restart heal evidence case that rebuilds a replaced drive with exact shard/body assertions.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): echo measured release evidence in ABBA fake adapter (#7460)
Keep the synthetic ABBA test adapter aligned with the measured release evidence contract so result validation covers release_evidence drift.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require complete ABBA summary matrix (#7462)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* heal: retain MRF replay journal after accepted/merged replays
Keep startup replay journal until a durable successor snapshot exists after Accepted/Merged admission.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* heal: reclaim superseded MRF snapshots after readback
Add a committed snapshot predecessor cleanup primitive that deletes only an older slot after the successor is read back as the current committed snapshot.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* heal: require exact owner for object receipts
Require object heal receipts to match the expected bucket incarnation before they can be recorded as positive repair evidence. This prevents stale or cross-incarnation receipts from clearing the wrong heal responsibility.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* heal: latch object receipt owner before repair
Capture the expected bucket incarnation before invoking object repair so a post-repair owner change cannot rewrite the responsibility that a storage receipt is allowed to prove.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* heal: replay committed MRF checkpoints durably (#7465)
Prefer committed MRF checkpoints during startup replay, retain accepted replay responsibilities until exact verified repair proofs arrive, and reclaim committed manifests only after discharge.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind hard evidence bundle provenance (#7467)
Require Scanner/Heal release bundle fields to carry source, run, window, timestamp, command, and artifact format provenance before a measured gate can pass.
Keep EC8+4 and performance gate fields tied to a single measurement window so unrelated artifacts cannot be stitched into a release approval.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* heal: reject dry-run object repair receipts
Do not record positive storage repair receipts for dry-run object heal tasks, even if a producer accidentally returns a matching receipt.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* heal: reject cancelled object repair receipts
Do not record positive storage repair receipts once an object heal task has been cancelled, even if the receipt still matches the requested owner and object identity.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): reject failed object repair receipts
Cover the receipt consumer path where a storage repair result carries both an error and a matching positive receipt. The failure may be recorded, but the receipt must not create repaired, healthy, or absent proof.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover bucket object repair receipts (#7468)
Cover bucket and root heal sweeps recording authoritative object outcomes only when storage receipts match the latched bucket incarnation.
Verify unavailable or stale receipt ownership keeps object repair execution intact while leaving canonical outcome proof as Unknown.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add EC8+4 heal restart evidence (#7469)
Register and wire Scanner/Heal background target restart and crash evidence cases for the 3x4 EC8+4 topology.
Validate the observed data/parity geometry in scanner-heal evidence receipts so multi-drive runs cannot satisfy the gate without proving EC8+4 metadata.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* heal: verify replacement pool metadata repair (#7471)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover quorum and mixed repair receipts
Add focused oracles for transient quorum results that carry a matching receipt and for mixed grace plus repaired receipt batches. Only the repaired object may produce positive proof.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* heal: verify admin recreate pool metadata (#7474)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* heal: preserve merged result for duplicate submits
Keep same-request-id replay receipts accepted for the receipt API, but preserve the legacy submit_heal_request duplicate admission result as Merged.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): preserve EC84 restart semantics (#7478)
Keep the EC8+4 background restart lane on the graceful-stop path and assert clean-restart marker absence only for restart scenarios. This prevents the hard evidence gate from silently exercising the crash path when it claims restart coverage.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF snapshot torn successor recovery
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): write EC84 distributed restart oracle (#7481)
Bind the distributed EC8+4 restart evidence lane to a scanner/heal oracle artifact so release validation can consume the real nextest run instead of accepting only a passing test.
Require the registry to assert 8+4 erasure geometry for the three-node, four-drive case.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): harden MRF replay boundaries (#7483)
Reject journal records with unknown version-presence flags even when their CRC is valid, so rollback/future payloads cannot be accepted as known records.
Gate committed checkpoint cleanup by the writer owner captured from the replay source, preserving retained manifests from other owners inside the same sequence window.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): bind MRF manifest CAS dirsync recovery (#7482)
Cover the committed MRF manifest path through LocalDisk conditional CAS when the metadata directory fsync fails. The fixture proves the previous manifest anchor survives rollback, an unanchored first successor is removed, and the legacy MRF journal remains readable even while global durability is relaxed.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): derive evidence runner profile from registry (#7484)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require mixed-version evidence roles (#7485)
Tighten the Scanner/Heal release bundle checker so mixed-version, scoped-ACK, and rollback fields cannot reuse a generic versions list without proving the expected evidence role.
Require version lists to use source revision identities and include the tested source revision. Also require profile evidence fields to name the core profiling artifacts before release approval.
The release gate remains blocked until measured field evidence is present.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): publish committed MRF runtime checkpoints (#7490)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind profile artifacts in release evidence (#7487)
Require Scanner/Heal release bundles to attach every required profiling artifact to P1 profile evidence with relative paths, artifact formats, non-empty files, hashes, and optional per-artifact measurement-window checks.
Document the tightened release bundle profile contract and cover missing, tampered, and mismatched-window profile artifact regressions in the existing checker self-test.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): harden measured ABBA evidence claims (#7491)
Reject measured Scanner/Heal ABBA manifests whose mixed-version evidence uses the same baseline and candidate source revision or binary hash.
Require crash fault modes and profile artifact names to match the exact supported sets, rejecting missing, duplicate, and unknown values.
Update harness fixtures and regression coverage for same-build mixed-version claims and exact-set release evidence fields.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(e2e): group scanner heal evidence payload (#7494)
Group the EC8+4 Scanner/Heal evidence writer inputs into a typed payload so the distributed e2e crate stays within the clippy argument limit without weakening the lint.
The evidence writer still validates the same S3 bodies, physical shard census, process restart PIDs, and node listings.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF replay bundle fields (#7486)
Bind Scanner/Heal release bundle evidence for MRF durable replay to replay counts, retained responsibility anchors, and successor snapshot publication evidence.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): profile EC84 evidence case runs (#7495)
Give Scanner/Heal evidence cases explicit runtime profiles so EC8+4 background restart and crash cases use their own object count, object size, and partial-progress timeout defaults instead of inheriting the legacy 4x1 case assumptions.
Expose the runtime profile in plan-only output and cover every registry case in the script self-test.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): reject empty release evidence artifacts (#7496)
Require Scanner/Heal release bundle artifact paths to resolve to non-empty files before hashing them.
Cover empty hard-gate artifacts in the existing release bundle checker self-test and keep profile artifact size checking on the shared artifact boundary.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require G14 same-window field coverage (#7489)
Tighten the Scanner/Heal release bundle checker so G14 same-window evidence must name the EC8+4, multi-set, and multi-pool fields covered in that measurement window.
Keep the release gate blocked when same-window evidence omits one of the required G14 fields, without changing production runtime behavior.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require two-hour measured ABBA windows (#7493)
Reject measured Scanner/Heal release ABBA manifests and summaries whose evidence window is shorter than the W21 two-hour release requirement.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF idle checkpoint cleanup (#7497)
Add a runtime cleanup regression test that publishes both retained replay and runtime committed checkpoints, writes scoped and legacy journals, and verifies idle cleanup removes every recovery anchor from the registered local disks.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(e2e): ignore untracked files in build identity (#7498)
Align e2e_test build provenance with Scanner/Heal evidence receipts and server binary provenance by treating only tracked source changes as dirty.
This prevents unrelated untracked worktrees or evidence directories from causing compiled test identity mismatches before real evidence cases can run.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF disk-full evidence fields (#7499)
Require the Scanner/Heal release registry and bundle checker to carry explicit G08 MRF capacity, disk-full, and replica-loss evidence fields before release approval can pass.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind release JSON artifact provenance (#7500)
Require scanner/heal release evidence JSON artifacts to repeat their measured source revision, run identity, measurement window, gate, and field identity inside the artifact payload. This keeps a refreshed outer bundle hash from accepting stale summary or profile JSON from another run.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF rollback mirror filtering (#7501)
Add a regression oracle that keeps scoped-only MRF responsibilities in the authoritative runtime snapshot while omitting them from the v1 legacy rollback mirror.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require segment activation evidence (#7504)
* test(scanner): require segment activation evidence
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): cover activation proof inputs
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF crash matrix cases (#7505)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require release bundle domain evidence (#7502)
Reject scanner/heal release bundles that omit field-level domain evidence for scoped ACK, durable intent, mixed-version, scheduler pressure, profile cost, and two-hour pressure lanes.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF retention GC evidence (#7506)
Require P4 retained-responsibility release bundle evidence to list the retained replay anchor and idle cleanup cases, prove a two-hour retention window, and record both idle cleanup and verified-proof discharge observations.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF cleanup GC soak evidence (#7507)
Require the P4 release bundle to carry measured cleanup/GC soak evidence for retained MRF replay responsibilities. The bundle now needs a two-hour cleanup window, exact cleanup case coverage, observed verified idle GC, and zero pending responsibilities or stale journals after GC.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF disk-full commit anchors (#7509)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): report port allocator bind errors (#7510)
Surface the scanned port window, attempt count, and last bind error when the e2e port allocator cannot reserve a localhost port. This keeps Scanner/Heal evidence failures actionable when the environment blocks binds before business assertions run.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(sse): resolve 1.0.0 SSE/KMS blockers and P1 findings (#7511)
* fix(sse): resolve bucket default encryption per request
PUT and the POST-object/extract path resolved a bucket's default
encryption with a hard-coded "no explicit SSE-C" flag, so the default was
layered onto a request that already carried an SSE-C header triple and
then tripped that request's own mutual-exclusion check. Every bucket with
default encryption refused SSE-C single PUTs with 400 InvalidArgument,
while CreateMultipartUpload on the same bucket succeeded because it
resolves SSE elsewhere. Both call sites now derive the flag from the
request headers, as COPY already did.
The bucket default's KMS key id was also inherited independently of the
effective algorithm, so an explicit AES256 request against an aws:kms
default bucket produced a self-contradictory algorithm/key-id pair and
was rejected. The key id is now inherited only when the effective
algorithm is aws:kms, matching the storage-layer resolver.
Refs backlog#2368 B1, B2.
* fix(sse): refuse SSE-KMS without a running KMS service
A write requesting aws:kms on a node with no KMS service fell back to the
node-local SSE-S3 provider: the data key was wrapped with
RUSTFS_SSE_S3_MASTER_KEY while the object metadata still recorded
aws:kms and the requested KMS key id. The stored object claimed a KMS
protection it never had, under a key that was never consulted, and no
signal distinguished it from a genuine SSE-KMS object.
The managed-encryption path now asks the resolved DEK provider whether it
wraps with a node-local master key and refuses SSE-KMS in that case:
InvalidRequest when KMS was never configured, ServiceUnavailable when a
configured service is not running. The check sits after the per-key
authorization gate so an unauthorized caller still receives AccessDenied
whatever the KMS runtime state is, and asks the provider rather than a
parallel availability signal because the provider is what actually wraps
the key. A missing master key no longer answers an SSE-KMS request with
an SSE-S3-worded configuration error.
The SSE-S3 local fallback is unchanged.
Refs backlog#2368 B4.
* fix(ecstore): restore and archive tiers in stored coordinates
Multipart restore addressed the remote tier in plaintext coordinates
while the copy-back reads the stored representation. Each part received a
misaligned slice of the remote object whose length still satisfied the
range, the hash reader and the completion size check, so the restore
reported success and silently replaced the object's bytes. Encrypted and
compressed multipart objects were both affected. Restore now accumulates
stored part sizes, passes the stored length to the hash reader alongside
the plaintext length, and validates against the stored size.
The copy-back digests stored bytes, so its computed MD5 is not the
object's public ETag. Restore now preserves the object ETag on both the
single-part and multipart paths, and gives each restored part its own
recorded part ETag rather than the object-level value.
Transition also handed the tier the object's SSE headers and its
RustFS-wrapped data key as request headers. Any S3 target rejected an
SSE-C archive outright, an SSE-KMS archive asked the target to encrypt a
second time under a key id it does not own, and the wrapped DEK left the
cluster. The archive request now strips every SSE header and encryption
marker with the predicate the replication path already uses; the local
xl.meta keeps all of it, so read-through and restore are unaffected.
Objects restored by an affected release are not detected or repaired
retroactively and must be re-restored from the tier.
Refs backlog#2368 B3, B5; backlog#2369 P7.1.
* fix(rio): lock the v1 nonce layout within a segment
Decrypting a v1 segment tried three historical nonce layouts per frame,
independently for every frame. The last of them exists for streams
written before 1.0.0-alpha.91, which reused a segment's part nonce for
every block in it; because block zero's derived nonce equals that base
nonce, a frame encrypted at index zero authenticated at any position. An
attacker able to rewrite the underlying shards could replay it and have
the forged plaintext returned with 200 and an unchanged length. Shard
integrity uses a keyed-hash-free checksum, which such an attacker can
recompute, so it is not a barrier.
A segment now locks onto whichever layout decoded its first non-zero-index
frame and rejects any later frame needing a different one. That leaves one
residual shape: a stream built purely from repeats of frame zero has no
later frame to disagree. New RUSTFS_ENCRYPTION_LEGACY_NONCE_FALLBACK
(default true, so pre-alpha.91 objects keep decrypting) drops the third
layout entirely when set to false, which closes it. Turning it off refuses
pre-alpha.91 objects, so migrate them first by rewriting in place.
Refs backlog#2369 P2.
* fix(kms): reload a service that failed to start
POST /rustfs/admin/v3/kms/reload short-circuited whenever the persisted
configuration matched the in-memory one byte for byte. A node whose KMS
failed to start keeps that configuration and sits in Error, so the
documented recovery call returned "reloaded successfully" while leaving
the node down. Peers reached the same path through the reload broadcast,
so a cluster that lost Vault during a rolling restart had no working
recovery route other than the node-local start endpoint. Reload now
short-circuits only for a service that is actually running, and otherwise
reconfigures, which starts a service that is not running.
The AWS backend also advertised key-version enumeration through
kms/status, which its own documentation says it cannot do; the capability
and its golden snapshot now say false.
Refs backlog#2369 P1, P7.3.
* docs: record the SSE and KMS changes for 1.0.0
The Unreleased changelog section carried no entry for any encryption work
merged since 1.0.0-rc.5, including three items with operational impact:
the config-secret variable whose absence persists secrets in cleartext
with only a warning, the v2 frame write switch and its rolling-upgrade
constraint, and per-key authorization making a public bucket incompatible
with SSE-KMS objects. Adds those plus this batch, including the SSE-KMS
refusal as a breaking change with both routes out.
Also corrects four places where documentation contradicted the code: the
cleanup register still called encrypted range seek opt-in after its
default flipped, the Helm README claimed vault_mount_path only applies to
Transit while the template also feeds the KV2 mount, the disaster-recovery
drill listed bundle contents for backends whose export is refused with
501, and the Chinese README capability table predated most of the feature
set. Documents the SSE-S3 local master key as a first-class operational
mode with its rotation dead end, and what the v1 frame layout does and
does not authenticate.
Refs backlog#2369 P5.
* fix(kms): classify data-path KMS failures by what the caller can do
Only "key not found" and a backend outage were classified; every other
KMS failure that reached the S3 data path fell through to
500 InternalError with a generic message. A disabled or pending-deletion
key, a denied KMS grant, an encryption-context mismatch, an unsupported
algorithm, a credential or timeout failure, and a capability the
configured backend does not have all looked identical to a server fault.
SDKs therefore applied exponential backoff to configuration errors no
retry can fix, and monitoring counted every one of them against the
server's own error rate.
Unusable-key and request-side failures now answer 400, a denied grant
403, transient backend failures 503, and a missing backend capability
501. Damaged, unreadable, or unknown-format key material keeps its 500:
it is a server-side integrity fault, and existing tests pin it.
The classifier is deliberately separate from the admin lifecycle
mapping, which answers 404 for a missing key because there a key id is
the resource being addressed; on the data path it arrives inside a
request header or a bucket default. Messages either name what the caller
asked for or stay generic, with deployment-side detail left on the error
source the way the storage-IO mapping already does.
Refs backlog#2368 B6.
* fix(kms): track and renew static Vault tokens
Token authentication hard-coded "this token carries no lease", so the
renewal task never started, the remaining-TTL gauge was never published,
and nothing looked wrong. `vault token create` grants a 768-hour TTL by
default, so a cluster that had been healthy for a month turned every KMS
call into a 403 and could not recover without a restart or a
reconfigure. Production configuration validation only rejects the
literal dev-token, so an ordinary expiring token reaches a whole cluster.
The source now reads `auth/token/lookup-self` at login and adopts what
Vault reports. A token with no expiry behaves exactly as before. An
expiring renewable one is picked up by the existing renewal loop and
renewed at half TTL like every other auth method. An expiring
non-renewable one warns with its remaining lifetime and publishes the
gauge, so the fail-closed window is visible before it arrives.
The probe never fails the login: a policy that omits lookup-self, or a
Vault that is briefly unreachable, warns and falls back to exactly the
previous behaviour rather than taking down a deployment that works
today. The scripted Vault test double answers the lookup out of band so
existing scripts keep describing only the protocol under test.
Refs backlog#2369 P3.
* feat(sse): report SSE-C requests that arrive without TLS
An SSE-C request carries the customer's AES key in a request header, so
AWS S3 and MinIO both refuse one that did not arrive over TLS. RustFS
accepted them on any transport: a plaintext hop hands the key to anyone
on the path, and since the object cannot be read without that same key,
the exposure lasts as long as the object does.
Refusing outright is the correct end state but not a safe default to
adopt inside a release window, because the project's own s3-tests and
e2e lanes and most staging deployments speak plain HTTP. This release
reports instead: each such request increments
rustfs_ssec_plaintext_requests_total and logs one warning per process, so
an operator can confirm nothing would break before the default flips.
RUSTFS_SSE_C_REQUIRE_TLS=true opts into the AWS 400 now.
The verdict is per connection rather than per deployment: the layer is
built with whether this listener terminated TLS, and additionally accepts
an https protocol forwarded by a proxy the trusted-proxy configuration
already vetted. It sits beside the rate limiter, after the layer that
makes a forwarded protocol trustworthy and after the request context, so
a rejection can echo the request id.
Refs backlog#2369 P7.2.
* fix(kms): say what a node-local backend means for a cluster
The Local backend keeps key material on each node's own disk and
generates its Argon2id salt per node, so two nodes derive different keys
from the same master_key and an object encrypted on one node cannot be
decrypted on another. Behind a load balancer that surfaces as
intermittent 500s on reads that succeeded moments earlier, with nothing
tying the symptom to the cause: the only signal was a generic
"development, testing and demos only" positioning warning that says
nothing about what actually breaks.
Configuring or reconfiguring Local while the deployment is distributed
now logs a dedicated event and appends the consequence to the configure
response, so the operator who made the change sees it. The product
decision to warn rather than refuse is unchanged.
Refs backlog#2369 P7.4.
* docs: record the remaining SSE and KMS changes for 1.0.0
Adds changelog entries for the KMS data-path status classification, the
Vault static-token lease probe, the SSE-C plaintext-transport report and
its switch, and the node-local backend warning.
Documents two things the backend security guide never stated: that SSE-C
belongs on a secure transport, with the counter and switch to plan the
change around, and that the Local backend cannot be shared by a
multi-node deployment because each node derives different keys from the
same master key.
Refs backlog#2368 B6; backlog#2369 P3, P5, P7.2, P7.4.
* fix(kms): report an unreadable key store as an outage on the S3 path
A backend now distinguishes a key store it could not read from a key
that is genuinely absent, but the S3 boundary collapsed the first one
back onto 500 InternalError through the fallthrough for integrity
faults. The distinction was therefore invisible to the client: a
temporary key-directory outage looked exactly like a permanently damaged
key record, and neither the status nor the metric said the request was
worth retrying.
An unreadable key store joins the retryable class and answers 503, next
to a backend error and a credential failure. Damaged, unreadable or
unknown-format key material keeps its 500.
Refs backlog#2368 B6; builds on rustfs/rustfs#7470.
* test(storage): cover MRF cleanup delete ENOSPC anchor (#7513)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require heal retry stats evidence (#7514)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add release bundle dry-run fixture (#7515)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): confirm recovery intent accept readback (#7516)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require MRF cleanup delete ENOSPC evidence (#7518)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require bounded retry window evidence (#7517)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): emit scanner heal G09 upgrade evidence (#7519)
Record measured Scanner/Heal G09 evidence artifacts from the upgrade compatibility lanes when a fresh evidence directory is provided.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): persist MRF rollback mirror boundary (#7520)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add G09 upgrade evidence runner (#7522)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(ci): bind performance runs to selected inputs (#7512)
* test(scanner): add G09 upgrade evidence runner
Add a reusable Linux x86_64 runner for the Scanner/Heal G09 mixed-version and rollback upgrade evidence lanes.
The helper reads the pinned previous-release asset metadata from the upgrade workflow, verifies the downloaded binary, builds the current head, runs both ignored E2E tests, and fails unless the expected G09 JSON artifacts exist.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): implement V2 evidence preflights (#7523)
* feat(scanner): wire dirty usage producer identities
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add segment activation preflight proof
Keep scanner segment reuse behind a structured activation preflight so release evidence can prove the production gate remains disabled until every producer, generation, overflow, cold-oracle, and distributed invalidation check is satisfied.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): expose distributed invalidation evidence
Record an explicit distributed segment invalidation evidence summary when remote dirty usage snapshots are bound to the current activity window and the authenticated scoped ACK capability probe succeeds.
Reject peer dirty usage snapshots that contradict the peer activity pending bit so scoped ACKs fail closed instead of clearing an unadvertised remote mutation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): assemble release evidence bundles (#7526)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add G09 upgrade evidence runner (#7524)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(ci): bind performance runs to selected inputs (#7512)
* test(scanner): add G09 upgrade evidence runner
Add a reusable Scanner/Heal G09 runner for Linux PR-head validation. The script downloads the pinned previous release, builds the current checkout, runs the mixed-version and rollback upgrade E2E lanes, and validates the measured raw artifacts before they can be consumed by the release bundle gate.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): replay recovery intents while disabled (#7521)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): preflight G09 evidence disk space (#7527)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(ci): bind performance runs to selected inputs (#7512)
* fix(targets): reject trailing batch items (#7508)
* test(scanner): preflight G09 evidence disk space
Fail the Scanner/Heal G09 upgrade evidence runner before downloading or building when the validation host does not have enough free space for a full raw evidence pass.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: cui fliter <imcusg@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): respect cargo target dir in G09 runner (#7528)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(ci): bind performance runs to selected inputs (#7512)
* fix(targets): reject trailing batch items (#7508)
* fix(scanner): respect cargo target dir in G09 runner
Write the RustFS feature stamp under the resolved Cargo target directory so remote validation hosts with CARGO_TARGET_DIR set can reuse the built PR-head binary.
Extend the runner plan/self-test path to cover relative target-dir resolution.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: cui fliter <imcusg@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): add G09 upgrade evidence runner (#7525)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(ci): bind performance runs to selected inputs (#7512)
* test(e2e): add G09 upgrade evidence runner
Add a Linux x86_64 runner that downloads the pinned previous release, builds the current RustFS binary, runs the mixed-version and rollback upgrade compatibility lanes, and verifies the required Scanner/Heal G09 raw evidence artifacts.
Document the runner and add a shell self-test for help, dry-run, SHA validation, and non-empty artifact directory guards.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): emit G09 release bundle gate evidence (#7530)
* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(ci): bind performance runs to selected inputs (#7512)
* fix(targets): reject trailing batch items (#7508)
* test(scanner): emit G09 release bundle gate evidence
Write a bundle-ready G09 gate descriptor from the Linux upgrade evidence runner and validate the single G09 gate with the shared release-bundle rules without approving the full Scanner/Heal release.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: cui fliter <imcusg@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): reject invalid G09 runner test selection (#7531)
Validate the selected G09 evidence lane before process-substitution case expansion so --plan-only cannot turn an unknown --test value into an empty successful plan.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): keep G09 source binary path clean (#7532)
Keep checksum verification output out of the command substitution that resolves the previous-release binary for the G09 runner.
Also tolerate non-GNU sha256sum in local self-tests by falling back to shasum when GNU --check support is unavailable.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF replay proof cleanup (#7534)
Exercise the committed MRF replay checkpoint path with a real ECStore-backed bucket incarnation, retain the replay anchor until an exact verified proof arrives, then delete the proof-discharged checkpoint and prove a subsequent restart does not resurrect the replayed work.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): carry segment activation preflight evidence (#7535)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): surface cold segment reuse oracle (#7536)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require hard evidence artifact payloads (#7537)
Require Scanner/Heal release bundle JSON artifacts to carry hard-domain evidence fields for mixed-version, crash, capacity, disk-full, replica-loss, and MRF cleanup gates. This prevents a descriptor from approving a hard gate while pointing at a generic measured artifact that lacks the boundary-specific oracle fields.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): bind producer evidence to dirty generations (#7538)
Record scanner segment producer identities with the dirty usage generation they invalidated, then expose a cycle-local producer evidence snapshot for segment reuse activation preflight.
Production activation remains fail-closed until durable producer identity and restart-gap proof are available.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): persist segment invalidation proof metadata (#7539)
* test(scanner): wait for EC8+4 heal control readiness (#7540)
* test(scanner): list full P4 release evidence fields (#7542)
Align the Scanner/Heal release requirement registry with the release bundle gate for P4 so the closure checklist advertises MRF scale, replay cost, retained responsibility, and cleanup/GC soak evidence together.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add W16 release evidence runner (#7543)
Add a W16 Scanner/Heal evidence runner that executes the recovery-intent crash-boundary and quota-authority lanes, emits measured G04/G12 JSON artifacts, and validates the resulting single-gate release descriptors.
Wire its shell self-test into script-tests and document the release evidence entry point.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix: address release acceptance regressions (#7541)
* feat(scanner): admit durable segment proof replay (#7544)
* feat(scanner): admit durable segment proof evidence
Validate complete set snapshot segment invalidation proof metadata against the current dirty usage generation window and scanner process epoch before clearing durable producer and restart-gap activation blockers.
Production segment reuse remains gated by the explicit activation flag.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): replay segment proof from root snapshots
Carry segment invalidation proof metadata into root snapshot set states so a complete published baseline can replay the durable producer evidence recorded by each set cache.
Keep the field additive for older readers and leave incomplete or LKG set states unproven.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind MRF release artifacts to evidence kinds (#7545)
Require Scanner/Heal G07/G08/P4 JSON artifacts to declare the exact MRF evidence kind expected for each release-bundle field. This keeps generic measured JSON from satisfying durable replay, ENOSPC, replica-loss, scale, replay-cost, retained-responsibility, or cleanup-GC soak gates.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): enable segment reuse activation gate (#7546)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): run scoped fallback cases on dedicated stack (#7550)
Linux libtest defaults can abort these deep EC-store futures before the assertions run. Wrap the async scoped fallback cases in the repository's dedicated-stack test pattern so the default cargo test path exercises the oracle directly.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): roll out distributed e2e selection (#7549)
Refresh the committed e2e-distributed selection digest after the EC8+4 Scanner/Heal evidence case joined the profile, and document the registry-aware runner entry for the distributed case.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add W13 MRF evidence runner (#7547)
Add a W13 durable MRF evidence runner that emits measured G07, G08, and P4 JSON artifacts and release descriptors through the existing scanner/heal bundle gate.
The runner now executes the ignored MRF replay evidence test with an exact full test path, validates raw artifact kinds and gate decisions, prepares Linux tmpfs-backed ENOSPC roots for G08, and documents the Linux/long-soak boundaries.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): fix segment proof fixtures
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): reduce W13 evidence helper arguments
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* ci(test): enable local KMS for s3 compatibility tests (#7553)
Configure the s3-tests harness with a local KMS key so SSE-KMS cases run in CI without relying on an external KMS service.
Also move the anonymous POST default SSE-KMS regression onto the shared local KMS test environment.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bind segment proof to typed dirty suffix (#7564)
Track producer identity completeness per pending bucket and retain process coverage across durable acknowledgements. Admit a prior same-process segment proof only for an exact or strictly earlier fully typed generation window.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): validate scoped ACK clear receipts (#7563)
Fail closed when a signed scoped ACK response claims more cleared entries than its request contained, while preserving partial-clear semantics and zero-mutation capability probes.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bind cold scoped cache reuse (#7565)
Record an explicit cold bucket reuse proof for scoped set scans and treat fully cold sets as valid cold-zero-walk reuse when the baseline and bucket incarnations are bound.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(s3): preserve delete marker read error headers (#7558)
* chore(release): sync configurable nightly build workflow from main (#7567)
feat(nightly): make the build branch configurable (NIGHTLY_BRANCH var + dispatch input) (#7557)
* docs(obs): correct metric export interval and explain alert timing (#7571)
* fix(heal): persist scoped admin admission handoff (#7569)
* fix(heal): persist root admission handoff
Persist administrator root heal ownership before queue publication so crash restart can replay an admitted control-plane start without relying on process-local replay cache state.
Protect queued durable root owners from priority displacement; callers must use forceStart cancellation semantics to replace them.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): persist admin heal handoff types
Extend durable admin heal admission handoff beyond cluster roots while preserving legacy root recovery records. Persist the heal type in schema v2, replay typed admin requests after restart, keep non-admin requests out of the durable control-plane handoff, and allow path cancellation to clear durable-only non-root records.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): preserve disjoint durable admin owners
Limit admin forceStart durable-owner cleanup to overlapping heal requests after the admin recovery journal begins storing non-root work. Add fail-closed and displacement tests so durable bucket owners cannot be silently dropped by unrelated replacement or priority pressure.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): persist admin terminal receipts
Persist terminal receipts for durable administrator heal starts before retiring their pending intent so restart can answer completed or cancelled scoped tasks without replaying them.
Retain pending work when terminal publication fails and keep timeout owners on checkpoint-only replay semantics.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): mark committed replication and tier mutations (#7568)
* fix(scanner): mark committed replication and tier mutations
Bridge ECStore committed replication status writebacks and lifecycle tier expiration cleanup into scanner dirty-usage producer identity tracking. Only successful replication metadata updates mark Replication; validate-only, superseded, and retry paths remain unmarked.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): replay durable dirty producer records
Add a strict scanner dirty-usage producer replay schema that can restore typed producer coverage after restart without treating ordinary process-local mutations as durable evidence. Replay records validate schema, cache key format, generations, scopes, producer identities, duplicate buckets, byte limits, and entry limits before modifying dirty state.
Keep segment activation fail-closed unless the restored bucket state came from durable replay; subsequent local mutations clear the durable authority bit for that bucket. Also fix the release scanner clippy lint in scoped cold-reuse proof creation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): recover expansion buckets and resume retirement (#7574)
* fix(ecstore): recover partially created expansion bucket volumes
* fix(ecstore): resume retirement after startup safety checks
* fix(ci): align S3 KMS fixtures and default key expectations (#7573)
* fix(heal): prune retained root terminal receipts (#7576)
Bound durable root-heal terminal receipt retention to the completed-status TTL while keeping expired receipts authoritative enough to retire stale pending intents before deleting the receipt itself.
Add a budgeted background GC pass and fail closed on corrupt or non-terminal receipt records.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(storage): preserve retryable publication guard errors (#7561)
* fix(storage): preserve retryable publication guard errors
* fix(scanner): satisfy scoped cache clippy lint
Use bool::then_some for the scoped cold bucket reuse proof construction after merging the latest release branch.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(health): report storage quorum in node readiness (#7566)
* fix: preserve scanner backlog during pool retirement (#7555)
* fix: hand off native scanner backlog before pool retirement
* test: initialize optional proof in data usage fixtures
* perf: bound parallel V3 pool metadata phase writes
* test: keep native backlog fault errors typed
* test: skip unchanged pool metadata in crash barriers
---------
Co-authored-by: cxymds <cxymds@gmail.com>
* test(scanner): bind profile artifact provenance (#7577)
* fix(scanner): persist dirty producer journal (#7580)
Persist scanner dirty producer replay records from committed replication and tier mutations, hydrate journal state on restart, and clear durable records only after acknowledged dirty generations.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test: add scanner heal scheduler pressure evidence (#7584)
Add measured G10/P1/P3 scheduler-pressure release descriptor generation, ABBA scheduler/profile provenance, and focused self-tests for the release bundle gate.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test: add scanner heal G14 multiset evidence (#7583)
Add EC8+4 multi-set and multi-pool scanner/heal release evidence coverage, including e2e registry cases, oracle checks, and a G14 descriptor assembler for measured case artifacts.
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(nightly): push assets with plain git on release — the build fleet has no gh CLI (#7586)
fix(nightly): push assets with plain git — the build fleet has no gh CLI
The sm-standard-4 runners used by the nightly build have no gh binary
(functional-chain workflows run elsewhere, on the jumpbox). The assets
publish step died on 'gh: command not found' at the credential-helper
setup, and the preceding clone failure had been masked by 2>/dev/null,
misleading the step into the orphan path. Swap clone and remote setup
to plain git with the token embedded in the URL; push semantics are
unchanged.
* fix(scanner): stabilize retired backlog survivor commits (#7585)
Treat an authoritative scanner pause-backlog commit as stable when every member in that commit stores the matching stable and committed record. Retired source replicas can remain missing in the observed topology while survivor membership is already stable.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(metrics): prevent duplicate drive reporting across nodes (#7582)
Co-authored-by: houseme <housemecn@gmail.com>
* fix(ecstore): avoid Windows batch-delete cleanup sharing conflicts (#7579)
Co-authored-by: houseme <housemecn@gmail.com>
* feat(ilm): inspect legacy transition version state (#7581)
* test(scanner): add release evidence ingestion gates (#7588)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix: reuse admin error boundary for legacy tier reconciliation (#7587)
* fix(test): stabilize release CI heal fixtures (#7589)
Keep heal admission-only tests off durable root recovery so mock storage does not require local metadata disks. Align forceStart retry assertions with replacement semantics and fix the e2e metric clippy collapse.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(nightly): publish packages as assets of the rolling 'nightly' release (#7592)
Replace the assets-branch scheme with a proper GitHub Release on
rustfs/auto-testing: a single 'nightly' release whose deb/rpm assets
are replaced in place on every build. This is the standard channel —
visible on the repo's Releases page, stable download URLs, no git
history growth (release assets live outside the repository).
- New scripts/release/publish_nightly_assets.sh: resolves-or-creates
the 'nightly' release via the REST API, deletes same-name assets,
uploads rustfs-nightly-latest.{deb,rpm}, then PATCHes the release
body with the build provenance (ref@sha, run link, sizes, SHA256).
Plain curl + python3, no gh CLI (the build fleet has none — #7586).
- The workflow step shrinks to invoking the script; full flow
exercised end-to-end against the real release with probe files
(create / upload / overwrite / download round-trip / body update).
* test(scanner): add release bundle evidence descriptors (#7595)
* test(scanner): merge release evidence descriptors
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add authority and legacy release descriptors
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add scoped ack release descriptor
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): require all release evidence fields
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add mrf release descriptor producer
Add a standalone W13 MRF descriptor producer for measured G07/G08/P4 raw artifacts, with fail-closed self-tests for missing responsibility cases, disk-full ENOSPC observations, synthetic artifacts, and short P4 cleanup soaks.
Also bind G12 quota path evidence to explicit case lists so W16 descriptors cannot pass without reset and settlement coverage.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): reuse mrf release descriptor producer
Route the W13 runner through the shared MRF descriptor producer so measured runs and offline re-packaging use the same fail-closed validation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(ilm): backfill proven legacy transition state (#7591)
* feat(ilm): backfill proven legacy transition state
* fix(ilm): remove redundant reconcile test clone
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): remove redundant transition clone (#7597)
Drop an unnecessary clone in the legacy transition state digest regression so release CI clippy lanes can run with warnings denied.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): seed current backlog members before committing (#7594)
* fix(scanner): seed current backlog members before committing
* test(app): run recursive force delete test on large stack
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind raw profile artifacts (#7598)
Require Scanner/Heal release profile wrappers to carry bundled raw profile artifacts plus the matching profile cost metrics.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): surface first failed object in status (#7599)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover retry budget across attempts (#7600)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): harden durable producer evidence gate (#7601)
Require Scanner/Heal G11 maintenance evidence to prove durable dirty producer journal replay, replay-state hydration, generation-bound ACK clearing, and invalid journal fail-closed cases before segment reuse can be activated.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): harden scoped ack evidence proof (#7602)
Require Scanner/Heal G03 scoped ACK evidence fields to carry their own measured provenance and concrete ACK, capability, and mixed-peer observations before release-bundle gate verification.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind status outcome raw provenance (#7603)
Require measured G05/G06/R-D raw status, compatibility, and disposition artifacts to share run identity before producing release bundle descriptors.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): retry fully recoverable batch failures (#7604)
Allow recursive bucket and cluster heal attempts whose retained batch failure is entirely retryable to use the existing bounded manager retry budget. Permanent or mixed batch failures remain terminal so heal outcome semantics stay strict.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): avoid movement debt for stale remote lease (#7605)
Classify a final remote scanner publication lease validation failure from a fresh activity snapshot instead of treating every validation error as data movement. If the peer session and movement generation still match the granted leases and publication is otherwise allowed, keep the publication rejected as an activity-baseline miss without creating pause backlog movement debt. Preserve DataMovement for expired leases, peer restarts, movement generation changes, and active publication blocks.
Tests cover namespace-only validation invalidation and the remote fence-loss cases that must still defer as DataMovement.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* chore(deps): refresh workspace dependencies (#7606)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bind G14 case artifacts (#7611)
Require Scanner/Heal G14 release bundle JSON wrappers to mirror their outer evidence and carry self-contained case artifact provenance.
Copy proof-json case artifacts into the generated G14 descriptor bundle so assembled release bundles can validate case file hashes after relocation.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): retry contended decommission cancel target locks (#7612)
* test: align transition fixtures and KMS context expectations (#7610)
* test(scanner): stabilize W13 release gate evidence (#7613)
* feat(nightly): publish packages as assets of the rolling 'nightly' release (sync from release) (#7593)
feat(nightly): publish packages as assets of the rolling 'nightly' release (#7592)
Replace the assets-branch scheme with a proper GitHub Release on
rustfs/auto-testing: a single 'nightly' release whose deb/rpm assets
are replaced in place on every build. This is the standard channel —
visible on the repo's Releases page, stable download URLs, no git
history growth (release assets live outside the repository).
- New scripts/release/publish_nightly_assets.sh: resolves-or-creates
the 'nightly' release via the REST API, deletes same-name assets,
uploads rustfs-nightly-latest.{deb,rpm}, then PATCHes the release
body with the build provenance (ref@sha, run link, sizes, SHA256).
Plain curl + python3, no gh CLI (the build fleet has none — #7586).
- The workflow step shrinks to invoking the script; full flow
exercised end-to-end against the real release with probe files
(create / upload / overwrite / download round-trip / body update).
* test(scanner): stabilize W13 release gate evidence
Treat only real raw-entry windows as replayed scanner enumeration in the restart diagnostic, so final completion rounds without raw entries are not fail-closed as raw replays.
Allow the EC8:4 multi-pool heal evidence case to select an outage object key that routes to an online pool while an entire target pool is down, preserving strict behavior for single-pool cases.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bound native backlog fixture resources (#7609)
* fix(scanner): bound native backlog fixture resources
Keep native scanner backlog restart fixtures within low file descriptor limits by reducing the native-only disk layout, making fixture shutdown drain background work, and avoiding long-lived ECStore retention from background loops.
Convert ECStore-backed background refresh/recovery/monitor tasks to upgrade weak owners only while doing work so completed test stores release their disk graph before the next native fixture starts.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(tier): refresh release-merged failure fixtures
Reuse the raw legacy-transition fixture helper so delete-all expiry actually exercises Unknown transitioned history metadata after the release merge.
Rewrite restore-failure disk fixtures through per-disk xl.meta snapshots so stale destination identities can be persisted without tripping ordinary metadata update guards.
Update the SSE KMS mismatch expectation to the InvalidRequest/context_mismatch behavior now returned by the rio-v2 path.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ci): align release E2E gate and repair regression tests (#7607)
* fix(scanner): harden heal release gates (#7614)
* test(scanner): reject synthetic heal evidence inputs (#7617)
Harden the Scanner/Heal checkpoint restart and G14 EC evidence producers so release descriptors cannot be assembled from marked fixture, dry-run, or synthetic case inputs.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): collect status outcome raw evidence (#7618)
Normalize live Scanner/Heal status/outcome observations into the measured raw artifacts consumed by the G05/G06/R-D release descriptor producer.
Reject synthetic observations, incomplete required cases, and reused run/window identities before raw artifacts can enter the release bundle flow.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bind scoped ack confirmation to generation (#7619)
Require lost scoped dirty-usage ACK reconciliation to observe a clean peer activity generation that covers the requested ACK generation. Apply the same guard in the scanner aggregation path so a stale clean activity snapshot cannot discharge pending maintenance after an uncertain ACK response.
Refs rustfs/backlog#2427
Refs rustfs/backlog#2281
Refs rustfs/backlog#2240
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test: report tier reference proof setup failures without hanging (#7608)
* test(scanner): plan Linux evidence validation (#7620)
Add a Scanner/Heal Linux release-evidence planner that emits a machine-readable execution manifest for the remaining measured validation lanes.
The planner can run only lightweight preflight checks and keeps plan-only output distinct from measured release evidence.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): select G14 replacement drive by evidence (#7622)
Choose the replacement disk from the target node by the presence of complete pool metadata instead of assuming the first configured drive is the scanner metadata holder. This keeps the G14 multi-set harness aligned with multi-drive EC layouts and adds clearer diagnostics when multi-pool outage writes fail closed.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): report Linux evidence plan status (#7624)
Add a status mode for the Scanner/Heal Linux evidence planner so operators can identify missing or incomplete artifacts before final bundle assembly.
The status output stays plan-only and never reports release approval.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add G14 multi-pool background heal evidence (#7628)
test(scanner): align G14 multi-pool outage evidence
Treat the localhost multi-pool topology as whole-pool loss when the target node is down. If that topology cannot admit the outage object while the pool is offline, defer that object write until the pool rejoins and record the oracle marker instead of failing before the real crash/restart evidence runs.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): verify deferred multipool outage evidence (#7629)
Validate the G14 multi-pool oracle field that records down-window outage PUT refusal and requires the deferred post-rejoin outage object to be accepted and checked through S3.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): allow metadata snapshots at read quorum (#7627)
fix(ecstore): admit metadata snapshots at read quorum
Allow guarded bucket metadata snapshot existence checks to use read quorum so degraded erasure sets can continue Object Lock snapshot reads without weakening bucket mutation or object write quorum.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* chore(deps): refresh release dependencies (#7632)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): use pool lock quorum for set writes (#7630)
Use the pool-wide namespace lock client domain for every set in a pool so degraded EC8+4 multi-set writes are gated by node-level lock quorum instead of the narrower per-set endpoint host slice.
Keep namespace-lock domain deduplication tied to both the pool namespace and shared clients, and add regression coverage for three-locker degraded writes plus cross-pool domain separation.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): keep G14 multi-set restart graceful (#7633)
test(e2e): keep multi-set restart graceful
Include the EC8+4 multi-set restart scenario in the graceful interruption lane so the unclean-shutdown marker assertion matches the scenario semantics.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): record deferred G14 outage writes (#7637)
Include optional outage-write diagnostics in G14 evidence oracles so deferred multi-pool outage writes bind their down-window refusals and post-rejoin acceptance.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): bound G14 multi-pool evidence size (#7640)
Keep the multi-pool evidence runner within the registry object budget while retaining deferred outage-write diagnostics for release-gate validation.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: cui fliter <imcusg@gmail.com>
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
* fix(replication): total-order rule sort and honor V1 top-level Prefix
Rule matching had two defects from the pre-GA replication audit
(rustfs/backlog#2367 C-1 and C-2):
- The actionable-rule sort compared same-destination rules by priority but
answered Equal for any other pair, which is not a total order; the
standard library sort panics on such comparators once a slice exceeds the
insertion-sort threshold, so an object matching more than 20 enabled rules
across two or more targets could panic the PUT or DELETE task. Rules now
sort by priority descending with destination and id as tie-breakers, and
filter_target_arns preserves that order instead of draining a HashSet.
- A V1 rule written without a <Filter> carries its prefix at the top level;
that field was never read, so <Prefix>logs/</Prefix> matched every object.
ReplicationRuleExt::prefix now falls back to it, with a <Filter> keeping
precedence. The existing prefix fixtures were built this way and had been
asserting nothing.
* fix(admin): advertise data-usage and listen capabilities to rc
The rc client gated `rc du` and `rc watch` on a pinned contract that
matched server versions by the string prefix `1.0.0-rc.`; a server that
reports `1.0.0` no longer matches, and the dynamic `advertised` list did
not carry either name, so `rc du` against a GA server fails with an
unsupported-capability error (rustfs/backlog#2367 E-2).
Advertise `admin.data-usage` from the admin route inventory like the IAM
entries, and `listen_notification` for the bucket `?events=` extension
route the admin router dispatches. The client merges advertised entries
ahead of its pinned contract, so no version sniffing is needed.
* fix(site-replication): stop notifying the local site on remove and rotate
The pending-remove and pending-rotation notification loops skipped the
local site by endpoint only, while finalization identifies it by
deployment id or endpoint. The reconcile tick resolves the local peer from
the node's own listen address (and a handler from the request Host), so
`remove --all` dialed the site's registered endpoint, waited out the
request timeout against the lifecycle lock it was holding, and answered
`Partial: failed to notify 1 peer(s)` for a removal that had succeeded
(rustfs/backlog#2367 A-4, backlog#2195 item 3).
Both loops now iterate the peers still awaiting notification through one
helper that applies the finalization identity.
* fix(site-replication): promote and settle IAM retries without a tick of slack
Two retry-queue behaviours kept an IAM change from converging for ten to
twenty minutes after a peer came back (rustfs/backlog#2367 A-1 and A-3,
backlog#2305):
- The lightweight 30-second pass filtered its reachability probe to bucket
ops, so a backed-off IAM or bucket-metadata snapshot waited for the
600-second tick to notice the peer. It now probes every backed-off class
and still replays only bounded bucket ops; promotion is a state flip the
heavyweight tick acts on.
- Backoffs are multiples of the tick interval, so a failure stamped δ
seconds after a tick was 600 − δ old at the next tick and slipped a whole
extra interval. The heavyweight drain now evaluates backoff halfway to its
next tick.
- An IAM entry first created by a non-deletion failure (the add bootstrap's
snapshot send, the drain's own replay, an import-iam schedule) was never
stamped `deletions_recorded`, so a later recorded deletion could not
settle it and it escalated to the marker only `replicate repair` clears.
Entries created by this binary now start recorded; a row persisted by an
older binary keeps the escalation semantics.
* fix(site-replication): reload peer node caches after bucket wiring writes
Every S3 bucket-config write ends by asking the other nodes of the cluster
to reload the bucket's metadata; the site-replication writers never did.
On a multi-node site the node that ran the pairing (or applied a peer's
bucket-meta item) rewrote the bucket targets and the derived replication
rules on disk, while every other node kept serving its cached copy for up
to the 15-minute refresh. A `resync start` routed to such a node reported
every freshly wired bucket as `Config not found` and a bucket whose
operator target the pairing had replaced as `recorded remote target no
longer exists` (rustfs/backlog#2367 A-5, backlog#2195 item 2; functional
SITE-105).
Add one best-effort reload helper in the site-replication hooks and call it
after the bucket setup, versioning, peer bucket-meta apply, removed-peer
cleanup, make-with-versioning, and endpoint-refresh writes; the ensure
helpers now report whether they wrote so unchanged passes stay silent. The
resync manifest and start now read the persisted wiring instead of the
node-local cache, matching the target read the start path already did.
The new four-node e2e pairs two clusters and starts a resync through a
non-coordinator node right after pairing; it also covers an IAM user
created on a non-coordinator node converging to the peer site.
* test(e2e): cover delete-marker replication from a multi-node source
The functional suite reported delete markers created on a 3-node source
never reaching the target (rustfs/backlog#2195 item 4, REP-105). The report
was a probe defect, but the shape had no coverage: the existing
delete-marker e2e runs a single-node source. Pin it against a four-node
source replicating to a four-node peer and to a single-node target, with
the write and the delete issued through different nodes.
* ci(e2e): refresh the distributed selection for the new replication cases
Four distributed cases were added (two site-replication, two delete-marker
replication). The linux digest is derived from the last CI listing of the
lane (34 cases, matching the previous pin) plus the four new names; the
darwin digest is the local listing, which selects the same 38 cases.
feat(nightly): publish packages as assets of the rolling 'nightly' release (#7592)
Replace the assets-branch scheme with a proper GitHub Release on
rustfs/auto-testing: a single 'nightly' release whose deb/rpm assets
are replaced in place on every build. This is the standard channel —
visible on the repo's Releases page, stable download URLs, no git
history growth (release assets live outside the repository).
- New scripts/release/publish_nightly_assets.sh: resolves-or-creates
the 'nightly' release via the REST API, deletes same-name assets,
uploads rustfs-nightly-latest.{deb,rpm}, then PATCHes the release
body with the build provenance (ref@sha, run link, sizes, SHA256).
Plain curl + python3, no gh CLI (the build fleet has none — #7586).
- The workflow step shrinks to invoking the script; full flow
exercised end-to-end against the real release with probe files
(create / upload / overwrite / download round-trip / body update).
2026-09-09 21:47:50 +08:00
165 changed files with 10279 additions and 1284 deletions
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
cxymds