* fix(ecstore): stop pruning at nonempty directories (#7616)
* fix(ecstore): stop pruning at nonempty directories
* test(ecstore): release pruning fixtures before temp cleanup
(cherry picked from commit 8f150d1d8e)
* fix(heal): preserve retryable batch failures during recovery (#7642)
* fix(heal): preserve retryable batch failures during recovery
* test(heal): pin prebuilt hooks binaries in ci
(cherry picked from commit 5cd58319ed)
* fix(s3): reject oversize single PUT early and map body errors to 4xx (#7635)
* fix(s3): reject oversize single PUT early and map body errors to 4xx
A single PutObject above the 5 GiB single-request ceiling was only
rejected after the client had streamed 5 GiB into s3s's read-time body
budget, and the resulting BodySizeLimitExceeded surfaced from the erasure
writer as 500 InternalError. A body whose connection hit EOF before
Content-Length bytes arrived (hyper's IncompleteBody) was also a 500.
SDKs retry 500s, so one oversize upload was resent from offset 0 five
times.
- PutObject and UploadPart reject a declared length above
MAX_SINGLE_PUT_OBJECT_SIZE with 400 EntityTooLarge before reading the
body; the constant moves to rustfs_config so the s3s limit and the
admission check share one value.
- ApiError maps BodySizeLimitExceeded to EntityTooLarge and a hyper body
EOF to IncompleteBody across both io::Error conversions.
Fixes#7596.
* test(s3): cover UploadPart admission, aws-chunked length, real s3s limit
- Poll-counting test body proves PutObject and UploadPart reject a
declared size above the ceiling with zero body polls; exact-cap and
zero-length parts pass admission.
- A STREAMING-* aws-chunked PUT whose framed Content-Length exceeds the
cap is admitted when the decoded length is within it and rejected when
the decoded length is over it.
- The display-based BodySizeLimitExceeded matcher is checked against the
real error produced by the pinned s3s Body budget.
(cherry picked from commit 50b31bc75b)
* fix(ecstore): make directory mtime fixture portable (#7623)
* fix(ecstore): make directory mtime fixture portable
* style(ecstore): format mtime fixture assertion
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
(cherry picked from commit b1cc286cac)
* fix(storage): prevent readiness after native migration failures (#7652)
* fix(storage): prevent readiness after native migration failures
* fix(storage): skip unsupported IAM records before reading
* fix(storage): use stable typed migration metadata errors
* fix(storage): include migration record in startup errors
* test(storage): cover native migration startup failures
* test(storage): use array chunks in migration fixture
---------
Co-authored-by: RJ Regenold <214054+rjregenold@users.noreply.github.com>
Co-authored-by: cxymds <cxymds@gmail.com>
(cherry picked from commit 0cbc3ffe61)
* fix(admin): expose OIDC account display fields (#7654)
Expose verified OIDC username and email claims as display-only metadata on self-account responses while preserving the virtual parent as the authorization identity.\n\nKeep rustfs-madmin public response structs unchanged by adding the optional wire fields through private handler response wrappers.
(cherry picked from commit f02bc947cd)
* fix(replication): correct peer joins and remote-state reporting (#7650)
* fix(replication): propagate verified peer deployment identities
* fix(replication): report actual remote peer state
* fix(replication): defer initial sync until all peers join
* test(replication): shut down TLS fixtures cleanly
---------
Co-authored-by: houseme <housemecn@gmail.com>
(cherry picked from commit 853ae63b6a)
* fix(s3): bound stalled UploadPart request bodies (#7659)
* fix(s3): bound stalled UploadPart request bodies
* fix(ci): preserve the S3S footprint ratchet
(cherry picked from commit 666dfd9f9f)
* fix(tables): reject reserved warehouse locations (#7671)
Co-authored-by: cxymds <cxymds@gmail.com>
(cherry picked from commit 414176c47f)
* fix(ci): bind nightly lanes to one resolved source (#7688)
(cherry picked from commit 01d8e4347f)
* fix(replication): close the pre-stable convergence gaps from backlog#2367 (#7626)
* fix(replication): total-order rule sort and honor V1 top-level Prefix
Rule matching had two defects from the pre-GA replication audit
(rustfs/backlog#2367 C-1 and C-2):
- The actionable-rule sort compared same-destination rules by priority but
answered Equal for any other pair, which is not a total order; the
standard library sort panics on such comparators once a slice exceeds the
insertion-sort threshold, so an object matching more than 20 enabled rules
across two or more targets could panic the PUT or DELETE task. Rules now
sort by priority descending with destination and id as tie-breakers, and
filter_target_arns preserves that order instead of draining a HashSet.
- A V1 rule written without a <Filter> carries its prefix at the top level;
that field was never read, so <Prefix>logs/</Prefix> matched every object.
ReplicationRuleExt::prefix now falls back to it, with a <Filter> keeping
precedence. The existing prefix fixtures were built this way and had been
asserting nothing.
* fix(admin): advertise data-usage and listen capabilities to rc
The rc client gated `rc du` and `rc watch` on a pinned contract that
matched server versions by the string prefix `1.0.0-rc.`; a server that
reports `1.0.0` no longer matches, and the dynamic `advertised` list did
not carry either name, so `rc du` against a GA server fails with an
unsupported-capability error (rustfs/backlog#2367 E-2).
Advertise `admin.data-usage` from the admin route inventory like the IAM
entries, and `listen_notification` for the bucket `?events=` extension
route the admin router dispatches. The client merges advertised entries
ahead of its pinned contract, so no version sniffing is needed.
* fix(site-replication): stop notifying the local site on remove and rotate
The pending-remove and pending-rotation notification loops skipped the
local site by endpoint only, while finalization identifies it by
deployment id or endpoint. The reconcile tick resolves the local peer from
the node's own listen address (and a handler from the request Host), so
`remove --all` dialed the site's registered endpoint, waited out the
request timeout against the lifecycle lock it was holding, and answered
`Partial: failed to notify 1 peer(s)` for a removal that had succeeded
(rustfs/backlog#2367 A-4, backlog#2195 item 3).
Both loops now iterate the peers still awaiting notification through one
helper that applies the finalization identity.
* fix(site-replication): promote and settle IAM retries without a tick of slack
Two retry-queue behaviours kept an IAM change from converging for ten to
twenty minutes after a peer came back (rustfs/backlog#2367 A-1 and A-3,
backlog#2305):
- The lightweight 30-second pass filtered its reachability probe to bucket
ops, so a backed-off IAM or bucket-metadata snapshot waited for the
600-second tick to notice the peer. It now probes every backed-off class
and still replays only bounded bucket ops; promotion is a state flip the
heavyweight tick acts on.
- Backoffs are multiples of the tick interval, so a failure stamped δ
seconds after a tick was 600 − δ old at the next tick and slipped a whole
extra interval. The heavyweight drain now evaluates backoff halfway to its
next tick.
- An IAM entry first created by a non-deletion failure (the add bootstrap's
snapshot send, the drain's own replay, an import-iam schedule) was never
stamped `deletions_recorded`, so a later recorded deletion could not
settle it and it escalated to the marker only `replicate repair` clears.
Entries created by this binary now start recorded; a row persisted by an
older binary keeps the escalation semantics.
* fix(site-replication): reload peer node caches after bucket wiring writes
Every S3 bucket-config write ends by asking the other nodes of the cluster
to reload the bucket's metadata; the site-replication writers never did.
On a multi-node site the node that ran the pairing (or applied a peer's
bucket-meta item) rewrote the bucket targets and the derived replication
rules on disk, while every other node kept serving its cached copy for up
to the 15-minute refresh. A `resync start` routed to such a node reported
every freshly wired bucket as `Config not found` and a bucket whose
operator target the pairing had replaced as `recorded remote target no
longer exists` (rustfs/backlog#2367 A-5, backlog#2195 item 2; functional
SITE-105).
Add one best-effort reload helper in the site-replication hooks and call it
after the bucket setup, versioning, peer bucket-meta apply, removed-peer
cleanup, make-with-versioning, and endpoint-refresh writes; the ensure
helpers now report whether they wrote so unchanged passes stay silent. The
resync manifest and start now read the persisted wiring instead of the
node-local cache, matching the target read the start path already did.
The new four-node e2e pairs two clusters and starts a resync through a
non-coordinator node right after pairing; it also covers an IAM user
created on a non-coordinator node converging to the peer site.
* test(e2e): cover delete-marker replication from a multi-node source
The functional suite reported delete markers created on a 3-node source
never reaching the target (rustfs/backlog#2195 item 4, REP-105). The report
was a probe defect, but the shape had no coverage: the existing
delete-marker e2e runs a single-node source. Pin it against a four-node
source replicating to a four-node peer and to a single-node target, with
the write and the delete issued through different nodes.
* ci(e2e): refresh the distributed selection for the new replication cases
Four distributed cases were added (two site-replication, two delete-marker
replication). The linux digest is derived from the last CI listing of the
lane (34 cases, matching the previous pin) plus the four new names; the
darwin digest is the local listing, which selects the same 38 cases.
(cherry picked from commit aeaba86d73)
* fix(e2e): require a verified server binary for every e2e run (#7687)
* fix(ci): share quick checks and lint workflows
* fix(ci): install actionlint from its verified release
* fix(ci): reject dependencies on required quick checks
* feat(test): verify the E2E server build and source identity
* test(e2e): register verified Darwin test membership
* test(e2e): record verified Linux receipt test membership
* test(e2e): record compiled Darwin receipt test membership
* test(e2e): record compiled Linux receipt test membership
* test(e2e): record Darwin e2e-full membership after merging main
* fix(test): route scanner/heal evidence E2E runs through the verified server binary
The evidence runners built rustfs with plain cargo and then ran e2e_test directly, which now fails without a run receipt. They build through scripts/e2e_binary.py and run the e2e_test invocations under e2e_binary.py run; the obsolete rustfs.features stamp is removed.
* docs(e2e): run server-backed e2e commands through the verified binary wrapper
* test(e2e): record Linux e2e-full membership from the branch CI listing
(cherry picked from commit 2909b1bfe1)
* fix(kms): classify KMS/SSE error contracts and SSE-S3 headers (#7697)
* fix(sse): classify bare SSE-KMS writes when no KMS is available
A `aws:kms` request without a key id, on a bucket without a default key,
returned `500 InternalError` whenever no KMS service was running: the
"no KMS key available" branch exited with an untyped storage error before
the availability classification that the keyed form already received.
Route that branch through the same split: `503 ServiceUnavailable` while
a configured KMS is stopped, `400 InvalidRequest` when KMS was never
configured, and `400 InvalidRequest` naming the missing key id when a
running KMS has no default key. `CreateMultipartUpload` shares the path.
Adds a unit test for the bare form and an e2e module that stops KMS
through the admin API, runs a master-key-only node, and runs a Local KMS
without a default key; refreshes the e2e-full selection digests.
(cherry picked from commit c3259dadc3d603a9185a5b0ad9f83dfb884e61c8)
* fix(sse): keep KMS error classes on the encrypted read path
GetObject, CopyObject and UploadPartCopy on an SSE-KMS object whose key
no longer exists answered `500 InternalError` ("KMS key not found") while
PutObject under the same key already answered `400 KMS.NotFoundException`.
The read path carries its classification through ecstore's
`EncryptionResolutionErrorKind`, which had no kind for a missing key, a
denied KMS grant or a missing backend capability, so all three folded
onto `DecryptionFailed` and the S3 layer reported an internal fault.
Add `KeyNotFound`, `AccessDenied` and `NotImplemented` kinds, map them on
both sides of the boundary, and give an envelope the configured backend
cannot unwrap a diagnosable message while keeping its `500`.
Unit tests cover the kind round trip and the reader wrapping; a new e2e
test deletes a key immediately and checks GET/Copy return 400 with
`KMS.NotFoundException` while HEAD stays 200. The e2e-full selection
digests are refreshed from the current listing (the previous digests
predated the delete-authorization tests) and the e2e `create_default_key`
helper is updated to the accepted `EncryptDecrypt` spelling.
(cherry picked from commit 2523a9814e97caea318d4ff1a51bef3a4d4445b2)
* fix(kms): classify key-management errors on the admin routes
`POST /kms/keys`, the legacy `create-key` alias and `generate-data-key`
reported every backend refusal as `500`: a blank key name (which each
backend failed on differently, the Local backend by writing a key file
with an empty stem), a name already taken, an unknown key, a disabled key
and a capability the backend lacks. `delete` and the lifecycle routes
already classified the same errors.
Refuse a blank or whitespace name in `KmsManager::create_key` before any
backend sees it, and share one `KmsError` to status mapping across
create, delete and generate-data-key (400 for validation and key state,
404 for an unknown key, 409 for a taken name, 501 for a missing
capability, 500 only for damaged material). The XML-error routes carry
the same status explicitly since s3s derives none for a custom code.
The read-only Static backend now reports create, delete and
cancel-deletion as `UnsupportedCapability`, matching its rotate and
enable/disable answers, so the admin API returns 501 for all of them.
(cherry picked from commit e33cac5493c4d9d6662e0d2980b58ba2b24a6d1b)
* fix(sse): stop SSE-S3 responses from naming the wrapping KMS key
`x-amz-server-side-encryption-aws-kms-key-id` is defined for `aws:kms`
objects only, but PutObject, CopyObject, CreateMultipartUpload and
GetObject returned it for `AES256` objects too, carrying the KMS key that
wraps the SSE-S3 data key (the service default, or the literal `default`
on a node without KMS). The write paths copied `kms_key_id` from the
encryption material unconditionally, and the single-decrypt GET
classification did the same after resolving the key for authorization.
Add `EncryptionMaterial::response_kms_key_id`, which yields the id only
for SSE-KMS, use it at the four write-response sites, and gate the GET
classification the same way. CompleteMultipartUpload and HeadObject
already omitted the header.
Unit tests pin both directions; a new e2e test covers Put/Get/Head/Copy
and CreateMultipartUpload for AES256 with an aws:kms control. The
e2e-full selection digests are refreshed from the current listing.
(cherry picked from commit 29d793a63352b0b60fd53c565e80fdbede8964bb)
* fix(s3): validate PutBucketEncryption rules before storing them
A default-encryption rule naming an unknown `SSEAlgorithm` (for example
`AES128`), a rule without `ApplyServerSideEncryptionByDefault`, an empty
rule list, or a `KMSMasterKeyID` on an `AES256` rule was stored as
written: the only algorithm check on the route decided whether to fill
in the default KMS key. `GetBucketEncryption` then advertised that
configuration while the write path encrypted header-less writes under
its `AES256` fallback, so the bucket's declared and actual schemes
disagreed. Two comments claimed the route already refused unknown
algorithms.
Validate the configuration before any of it is applied: `MalformedXML`
for a malformed rule set or unknown algorithm, `InvalidArgument` for a
key id on a non-KMS rule, and nothing stored on refusal. Correct the two
comments to describe when the AES256 fallback is still reachable.
Unit tests cover every refusal and the accepted shapes; an e2e test
checks the refusals leave the previous configuration in place. The
e2e-full selection digests are refreshed from the current listing.
(cherry picked from commit 29e4486dce41197ed93f5253cdbabc57d27a4ddb)
* test(e2e): refresh e2e-full selection for the combined KMS/SSE fixes
* test: align two unit tests with the new KMS and bucket-encryption contracts
`scheduled_deletion_carries_a_deadline_and_can_be_cancelled` still
expects the state error (`InvalidOperation`) for cancelling a key that
is not pending deletion; only the Static backend's mutations moved to
`UnsupportedCapability`. The uninitialized-store PutBucketEncryption
test now sends a well-formed AES256 rule so it reaches the store lookup
instead of the new configuration validation.
(cherry picked from commit e2e6a2535a)
* fix(site-replication): keep an operator's bucket-level target to a peer instead of taking it over (#7709)
* fix(site-replication): keep an operator's bucket-level target to a peer instead of taking it over
Site replication wired each bucket by looking for an existing replication
target "to the same peer" and rewriting the first match in place as its own
same-name target. An operator's bucket-level target that happened to point
at that site (different target bucket, operator credentials) was the first
match whenever it pre-dated the join, and the reconciler repeats the pass
every 600s, so the takeover also depended on target order afterwards. The
operator's rule then named an ARN no target backed and their bucket
replication stopped silently, while the inherited bucket-level reset id
made every site resync report the bucket as owned by another resync
(rustfs/backlog#2479, rustfs/backlog#2489).
Follow MinIO's `getRemoteARN` / `getRemoteARNForPeer` shape instead:
- Wiring updates a target in place only under the same ARN, or when it is
recognisably the site's own under an older ARN shape (same peer,
same-name target bucket, site replication service account). Anything
else gets the site target added next to it.
- The site resync manifest takes the target the derived
`site-repl-<deployment id>` rule names (same-name shape as fallback), so
an operator target to the peer neither aborts the bucket as "multiple
remote targets matched peer" nor gets resynced into.
- Peer removal prunes only targets a pruned derived rule names or the
same-name target bucket; operator targets stamped with the peer's
deployment id survive together with their rules.
Unit tests cover the three predicates. e2e
`test_site_replication_keeps_operator_bucket_target_to_peer` runs a
bucket-level replication plus `replication-reset` to the future peer, joins
the sites, and requires the operator target untouched, both paths
delivering, the site resync completing against the site target, and the
operator target and rule surviving `replicate remove --all`; without the
fix it fails at the join with the operator target gone. The repl-nightly
selection digest is refreshed for the new case.
* test(site-replication): drop a redundant clone flagged by clippy
The reconcile unit test cloned the remote peer into the state map although
the binding is not used afterwards; workspace clippy (-D warnings) rejects
that as redundant_clone.
(cherry picked from commit ecdc55fa4b)
* fix: enforce S3 permissions for recursive force deletion (#7661)
* fix: enforce S3 authorization for recursive deletion
* fix: satisfy the s3s footprint guard
* fix: restore list versions policy compatibility (#7686)
(cherry picked from commit 3fd1ce414d)
* fix(ci): repair functional defaults and chain regression checks (#7664)
* fix(ci): default functional suites to nightly packages
* test(ci): follow the fault-tolerance chain handoff
(cherry picked from commit 509a0fa90c)
* fix(ci): align security workflow tests with chain (#7679)
(cherry picked from commit d9e47d2813)
* test(ecstore): keep tier cleanup tests stable after immediate receipt queueing
Release PR #7766 made PUT/CopyObject overwrites queue the tier free-version cleanup receipt immediately, which broke two ecstore tests on release CI. In tier_overwrite_put_and_self_copy_recover_persisted_cleanup_owners the restarted store already runs expiry workers from the second iteration on, so they deleted the remote bytes before the test could assert that the commit leaves them in place; the test now fails the first remote DELETE via set_remove_failure(true) so the cleanup owner stays durable and the later restart still has to rediscover it from xl.meta (the failed remove does not bump remove_count, and the test re-enables removes before the recovery wait). In batch_transitioned_delete_post_commit_failures_roll_back_without_free_version_receipt the convergence loop now treats a transient InsufficientReadQuorum as "not yet converged", because cleanup rewrites xl.meta disk by disk and a racing read can briefly miss quorum (seen in the rio-v2 lane); any other error still panics.
* Revert "fix(ci): align security workflow tests with chain (#7679)"
This reverts commit 4544359f6d.
* Revert "fix(ci): repair functional defaults and chain regression checks (#7664)"
This reverts commit 5ba7ec0291.
* fix(ecstore): pass shard integrity to backported ingest-mode test
The stalled-reader test backported with #7659 used main's four-argument encode_with_ingest_mode, but release's signature takes an optional IntegrityBuilder, so pass None to keep the test focused on ingest-mode cleanup.
---------
Co-authored-by: Henry Guo <marshawcoco@gmail.com>
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: RJ Regenold <rregenold@teamraft.com>
Co-authored-by: RJ Regenold <214054+rjregenold@users.noreply.github.com>
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: GatewayJ <835269233@qq.com>
Co-authored-by: Jason Kossis <jkossis@gmail.com>
* fix(ecstore)!: bind bitrot shards to immutable part identities
Verify part, coding-index, and block identity across write, GET, and Heal
paths. Preserve identities across metadata-only copies and repair, include
them in multipart quorum selection, and require payload proof for receipts.
Keep legacy decoding with conservative parity and target-digest validation,
and document its unsupported cases and additional verification I/O.
BREAKING CHANGE: New bound-v1 shards require compatible readers throughout
the fleet. Legacy objects without sufficient integrity evidence return an
error; binary rollback after new writes requires verified data migration.
Refs: rustfs/backlog#2497
* fix(ecstore): preserve shard framing with independent integrity
Commit immutable part-generation Merkle roots and replicated proof indexes without changing existing checksum frames. Verify reads, reconstruction and Deep Heal against metadata quorum; keep legacy reads and explicitly defer unproven legacy data repair.
Preserve multipart rollback generations, require acknowledged durable index publication, and add decoder compatibility and donor-shard regression coverage.
* fix(heal): verify protected partial-write replay
* fix(scanner): mark committed replication and tier mutations
Bridge ECStore committed replication status writebacks and lifecycle tier expiration cleanup into scanner dirty-usage producer identity tracking. Only successful replication metadata updates mark Replication; validate-only, superseded, and retry paths remain unmarked.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): replay durable dirty producer records
Add a strict scanner dirty-usage producer replay schema that can restore typed producer coverage after restart without treating ordinary process-local mutations as durable evidence. Replay records validate schema, cache key format, generations, scopes, producer identities, duplicate buckets, byte limits, and entry limits before modifying dirty state.
Keep segment activation fail-closed unless the restored bucket state came from durable replay; subsequent local mutations clear the durable authority bit for that bucket. Also fix the release scanner clippy lint in scoped cold-reuse proof creation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(replication): resolve drifted replicas via a target version ledger
A replication target that mints its own version ids (Wasabi, AWS S3)
never answers to the source uuid, so every version-addressed mutation
after the initial PUT failed forever: permanent version deletes answered
NoSuchVersion every heal cycle, and tag / retention / legal-hold updates
re-PUT the object, minting one more target version per update
(rustfs/backlog#2340).
Record the id the target assigned as a per-target ledger on the source
version (replication-target-version-<arn>, written through the existing
status writeback) and resolve every later mutation through it: version
deletes DELETE the ledger id, metadata updates go through the
metadata-only Object Lock and tagging APIs. Replicas written before the
ledger existed are located by exact key and ETag, minus the candidates
other generations of the key already claim through their own ledgers; an
ambiguous remainder is refused with a backoff instead of guessed, since a
wrong pick would destroy a live generation. A fresh write never consults
content identity. NoSuchVersion on a version-addressed DELETE counts as
purged.
The fake target gains the Wasabi shape (404 NoSuchVersion on an unknown
id, per-version Object Lock APIs) and the matrix covers the three
mutation classes plus the same-bytes generation case.
* fix(scanner): drop the unused Digest import
Same one-line change as rustfs/rustfs#7366 (main is red with it under -D warnings); carried here so the stacked PRs' merge commits compile until that fix lands.
* fix(admin): probe replication-check mutations by the assigned version id (#7373)
On a target that mints its own version ids the DeleteMarker and
VersionDelete phases of ?replication-check were skipped: they addressed
the source id, which such a target never had. The replication worker now
addresses the id the target assigned (the target-version ledger), and
the probe already holds that id from its own PUT, so run both phases
against it. VersionFidelity keeps failing with the mismatch code and the
target stays FAILED; the phases report whether ledger-addressed purges
work against this endpoint (rustfs/backlog#2340).
* fix(replication): abandon purges to targets the bucket no longer names (#7377)
* fix(admin): probe replication-check mutations by the assigned version id
On a target that mints its own version ids the DeleteMarker and
VersionDelete phases of ?replication-check were skipped: they addressed
the source id, which such a target never had. The replication worker now
addresses the id the target assigned (the target-version ledger), and
the probe already holds that id from its own PUT, so run both phases
against it. VersionFidelity keeps failing with the mismatch code and the
target stays FAILED; the phases report whether ledger-addressed purges
work against this endpoint (rustfs/backlog#2340).
* fix(replication): abandon purges to targets the bucket no longer names
A permanent version delete whose replication keeps failing stays in
xl.meta as a PENDING purge, hidden from listings, until every target
confirms it. Once the operator removes the replication configuration or
the rule naming that target nothing ever confirms it: the heal path
derived its delete decision from the configuration (the decision string
is not persisted) and skipped the version forever, so DeleteBucket
answered BucketNotEmpty for a residue the client could neither list nor
remove (rustfs/backlog#2340).
Owe a version purge to the targets its purge state names, let the heal
path through without a configuration, and have the delete worker settle
a target the configuration no longer names as abandoned: the purge is
reported complete locally through the normal writeback, the replica on
the former target is left alone, and the event
replication_purge_abandoned plus a counter are the record.
* fix(admin): send replication-check marker creation without a version id
Running the DeleteMarker / VersionDelete phases on a target that mints
its own version ids exposed two probe-shape bugs on real Wasabi:
- the DeleteMarker phase put the assigned version id on its DELETE. A
RustFS peer reads the source-deletemarker header and creates a marker,
but a generic S3 target executes it as a permanent delete of the probe
version, so VersionDelete then answered NoSuchVersion. Use the same wire
shape as live delete replication: no versionId on a marker creation.
- cleanup treated NoSuchVersion on the version the VersionDelete phase had
already removed as a failure (RustFS/MinIO answer 204 there).
Also gate the no-configuration heal pass-through for pending purges on a
purge state that actually names targets, so a purge without a recorded
target keeps the ordinary skip (scanner unit test), and merge origin/main
(#7365 settles the pool-metadata probe test that failed in CI).
---------
Co-authored-by: houseme <housemecn@gmail.com>
* fix(ilm): reject invalid retention counts and validate lifecycle filters
`NewerNoncurrentVersions` had no lower bound at PUT, and evaluation read a
negative count through `usize::try_from(...).unwrap_or(usize::MAX)`. An
HTTP-accepted rule therefore retained (almost) everything and silently
stopped expiring versions — the one outcome a retention rule must never
produce by accident.
Reject a negative count during validation, and stop reading one as
"retain everything" anywhere it can still arrive from older persistence
or an import: evaluation takes no action for such a rule and says so in a
diagnostic, the batch limit path yields no event, and `Evaluator::eval`
reports a typed corruption error to callers that can surface one.
A count-only noncurrent expiration is a MinIO extension, not an AWS form.
It used to be rejected as an actionless rule and was never executed. It
is now accepted and honoured with the semantics MinIO gives it: the
newest N noncurrent versions are kept and every older one is due as soon
as it became noncurrent. Zero keeps the meaning the batch limit path has
always given it — no count constraint — so a zero-count rule with no age
condition still has no action.
`LifecycleRuleFilter` is an all-`Option` DTO, so the schema constraints
were not checked anywhere: validate at most one top-level predicate, an
`And` that combines at least two, no repeated tag key, tag key/value
limits, non-negative sizes, and `ObjectSizeGreaterThan <
ObjectSizeLessThan`. An empty filter stays valid — AWS documents it as
"every object in the bucket".
Schema-shape violations are reported with a distinct `ErrorKind` so the
S3 boundary answers them with `MalformedXML`; rejected values keep the
`InvalidArgument` this path has always returned.
backlog#2201
* fix(ilm): satisfy lifecycle clippy checks
* fix(ilm): fail closed on invalid lifecycle rules
* fix: initialize optional migration source fields
---------
Co-authored-by: cxymds <cxymds@gmail.com>
* feat(ecstore): add a native azure blob odm source backend
* feat(ecstore): add a native gcs odm source backend and one backend contract
* fix(ecstore): refuse an empty azure account key at client build
* fix(ecstore): probe gcs sources with the listing permission
* fix(app): drop a redundant match guard on the sse config lookup
* fix(ecstore): drop stale rename commit duplicates from local.rs
* test(ecstore): use the sanctioned placeholder key in the gcs fixture
* feat(ecstore): add the sealed remote credential seam
Replication targets, remote tiers and on-demand migration sources will all
seal their stored secrets through one envelope rather than three
(rustfs/backlog#2168, design in docs/architecture/remote-credential-sealing-adr.md).
Adds the versioned envelope, the seal scope that binds a ciphertext to the
store, owner and field it belongs to, the sealer registration point, and the
fail-closed error type. ECStore still has no rustfs-kms dependency: the binary
installs a sealer the way it installs the event dispatch hook.
Nothing is wired to a consumer yet, so no stored format changes.
* docs(ecstore): name the event dispatch hook by module, not by symbol
The architecture guard keeps EVENT_DISPATCH_HOOK references inside the
event-notification owner module; the module doc cited the symbol only as an
example of the hook shape, so cite its file instead.
`GET /v3/tier-stats` answered from whichever process received the
request, returning that node's rolling 24-hour transition counters as
if they were cluster totals, and the `TierRequestsSuccess` and
`TierRequestsFailure` metric names had no producer at all.
The body now separates the two quantities a tier carries. Stored
inventory comes from the persisted scanner usage snapshot, which is
already cluster-wide; rolling activity is summed over every member
through a new read-only `TierDailyStats` peer RPC. Rings are merged
rather than added, so an idle node's expired hours age out, and each
node counts only its own committed transitions, so a retry is counted
once. Coverage travels with the numbers: `activity.status` names the
reporting members and the ones that could not be asked, timed out, or
answered with a ring this build refuses to merge, and per-tier
inventory is absent rather than zero when the snapshot has no
accounting. The version 1 body stays reachable at `?format=legacy`.
Tier request counters are recorded at the two seams every remote
request passes through, so a new provider is counted by construction,
with a closed operation/outcome label set that can never grow a tier
name, endpoint or object key.
Closesrustfs/backlog#2207
Co-authored-by: cxymds <cxymds@gmail.com>
* feat(odm): merge the source listing into ListObjectsV2
Adds policy.list_through: ListObjectsV2 merges the local and source
listings into one ordered page so clients see the whole namespace during
an on-demand migration. Local entries win a key both sides hold,
CommonPrefixes are unioned under a delimiter, and the continuation token
is an opaque versioned envelope carrying both cursors.
A source listing failure or an open breaker follows policy.source_error:
propagate answers 424, not_found answers from local state and marks the
response x-rustfs-on-demand-migration-list: local_only. Source listings
are capped at 10 per second per bucket.
* test(odm): refresh the e2e-full darwin selection digest
The list-through e2e module adds seven cases to the merge lane.
* fix(odm): declare the remote client retry policy per consumer
The SDK retry policy was an inherited default: one logical call could cost
three wire requests, so the migration breaker counted logical calls on top
of a threefold amplification against a source that was already failing.
Make it an explicit RemoteS3EndpointSpec field. Replication targets declare
today's standard three attempts and keep their behaviour; the on-demand
migration source and its admin probe declare a disabled policy, so one
counted failure is exactly one source request and pull.rs owns the only
retry budget.
* fix(odm): count a stalled inline source as a source timeout
The inline tee wraps its source body in the idle guard, but the tee turns a
stalled source into an ordinary body read error, so the write-back reported
it as a local write failure. Hand commit_inline the guard so the pull is
counted under source_timeout instead.
The background pump now enforces the idle budget through the same guard
rather than a second copy of the timeout loop.
* test(odm): cover a stalled source body end to end
The fake target can now deliver a GetObject body in slices with a pause
between them, so the inline abort can be driven by a stalled source instead
of a truncated one. Two fault cases drop the workarounds they carried for
the SDK's retries: the scripted fault count and the observed source request
count now have to agree.
The operations guide records the retry and idle-timeout guarantees.
* feat(ecstore): add on-demand migration backfill job core
Add the background backfill job for on-demand migration
(rustfs/backlog#2159): a durable checkpoint under
buckets/<bucket>/on-demand-migration-backfill.json saved by If-Match
compare-and-set every 1000 keys or 10 s, a 60 s owner lease renewed by
every save, a recovery pass that takes over expired leases (or jobs this
node owned before a restart) and cancels jobs whose config changed, and a
main loop over the source ListObjectsV2 pages with the skip_existing
policy, dry runs, bounded outstanding pulls and wait-on-full enqueueing.
The pull queue gains per-job completion reports so the job can count
pulled/failed keys (hashes only), and pull permits become two-tier so an
online miss is never queued behind a backfill pull.
* feat(admin): expose on-demand migration backfill job
Wire the ODM-12 backfill job (rustfs/backlog#2159) to its operators:
POST /v3/on-demand-migration/{bucket}/backfill?op=start|cancel and
GET .../backfill return the checkpoint document, GET .../status gains a
backfill summary, and the recovery loop plus the process-wide runner are
installed at startup. Backfill control reuses
Set/GetBucketOnDemandMigrationAction and is recorded in the route policy,
the registration matrix and the admin route snapshot.
Add the rustfs-madmin wire types and client methods with golden fixtures
shared by the server tests, the backfill_* metric descriptors and their
collector, and three e2e scenarios: a full backfill across list pages,
cancellation, and resuming from the persisted continuation token after a
server restart.
* feat(ecstore): add on-demand migration pull queue and write-back pipeline
Background pull queue per bucket (bounded by pull_queue_capacity, concurrency via the state's pull slot), OdmWriteBack/PullSource traits, single-part and multipart write-back with a pumped body that enforces idle timeout, cancel and content length, retry policy for retryable source errors, inline commit helper, and stats accounting (rustfs/backlog#2153).
* feat(object): implement on-demand migration write-back over internal put
OdmWriteBack impl mapping source heads onto InternalPutContext (content-header allowlist, x-amz-meta copy, tags, dual-prefix odm-* provenance, ETag policy), injected into OnDemandMigrationSys at startup; removes the dead-code gates left by ODM-06a (rustfs/backlog#2153).
* feat(ecstore): add on-demand migration bucket config model
Introduce OnDemandMigrationConfig (deny_unknown_fields, version 1) with typed validation, credential redaction, a secret-free Debug impl, and the OnceLock publish hook the runtime registers into. Exported through the api facade.
* feat(ecstore): persist on-demand migration config in bucket metadata
Store the config as a RustFS extension entry (on-demand-migration.json) with its update time in .metadata.bin, add the typed BucketMetadataSys accessor, and publish the config through the hook on every cache-install path alongside the durability sync.
* refactor(ecstore): extract shared remote S3 client builder
Move the aws_sdk_s3 client construction out of bucket_target_sys into
bucket/remote_s3_client.rs: endpoint assembly, credential provider,
path-style selection, custom CA / skip-TLS transports and the outbound
SSRF gate now build from a neutral RemoteS3EndpointSpec so replication
targets and the upcoming on-demand migration source client share one
policy. Replication builds its client through From<&BucketTarget>; the
gate keeps its relaxed semantics (private allowed, loopback only behind
RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) verbatim. The builder also
gains optional connect/read timeouts and a User-Agent suffix
interceptor, both unset for replication.
Refs rustfs/backlog#2149
* feat(ecstore): add on-demand migration SourceClient
Add bucket/on_demand_migration/source_client.rs on top of the shared
remote S3 builder: HEAD, ranged streaming GET, ListObjectsV2 with
source-prefix mapping, GetObjectTagging and an admin probe. Every request
carries the x-rustfs-/x-minio-source-proxy-request anti-loop markers and
a RustFS-OnDemandMigration/<version> User-Agent suffix; SSE-C source
objects are rejected as unsupported. SourceError classifies SDK failures
(not found, access denied, throttled, timeout, connect, server error)
with retryability and a stable metrics label. Debug output redacts
credentials.
Refs rustfs/backlog#2149
* docs(operations): point outbound policy at shared remote S3 client builder
* chore: integrate ODM-01 and ODM-02 as B1 base (fix facade merge)
* feat(ecstore): add on-demand migration runtime OnDemandMigrationSys
Per-node runtime for On-Demand Migration (rustfs/backlog#2152): turns each
bucket's persisted config into a live SourceClient guarded by a three-state
circuit breaker, a TTL negative cache, per-key singleflight, a pull
concurrency semaphore and lock-free counters with a serializable snapshot.
- sys.rs: OnceLock singleton; `apply` installs/rebuilds/removes bucket state
(config compared by value, counters preserved across rebuilds, old
cancellation token fired); `publish` is the metadata publish-hook entry
(sync removal, spawned install, generation-ordered so a slow older install
cannot overwrite a newer one); `resolve(bucket, key)` judges module switch,
bucket state, prefix filter, client availability, negative cache, breaker.
- breaker.rs: Closed/Open/HalfOpen with fixed constants (5 failures / 30 s
window / 30 s open / 1 probe); NotFound resets, AccessDenied is neutral.
- negative_cache.rs: moka sync cache keyed by local key, ttl=0 disables.
- stats.rs: requests_total{op,outcome}, pulled_bytes/objects, pull_failures,
inflight/queue gauges, log-bucket latency histogram, last_source_error;
snake_case snapshot pinned by a golden JSON test.
- Anonymous sources surface as a typed `OdmStateError::AnonymousUnsupported`
until the shared client builder gains an anonymous mode.
- rustfs: `RUSTFS_ON_DEMAND_MIGRATION_ENABLED` module switch (default false)
published to module_switches and injected into ecstore before bucket
metadata loads; hook registered at the same point.
* feat(ecstore): add on-demand migration bucket config model
Introduce OnDemandMigrationConfig (deny_unknown_fields, version 1) with typed validation, credential redaction, a secret-free Debug impl, and the OnceLock publish hook the runtime registers into. Exported through the api facade.
* feat(ecstore): persist on-demand migration config in bucket metadata
Store the config as a RustFS extension entry (on-demand-migration.json) with its update time in .metadata.bin, add the typed BucketMetadataSys accessor, and publish the config through the hook on every cache-install path alongside the durability sync.
* test(e2e): rename stall timing variable flagged by typos
* test(storage): heap-pin the RestoreObject usecase future in the generation guard test
* refactor(ecstore): extract shared remote S3 client builder
Move the aws_sdk_s3 client construction out of bucket_target_sys into
bucket/remote_s3_client.rs: endpoint assembly, credential provider,
path-style selection, custom CA / skip-TLS transports and the outbound
SSRF gate now build from a neutral RemoteS3EndpointSpec so replication
targets and the upcoming on-demand migration source client share one
policy. Replication builds its client through From<&BucketTarget>; the
gate keeps its relaxed semantics (private allowed, loopback only behind
RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) verbatim. The builder also
gains optional connect/read timeouts and a User-Agent suffix
interceptor, both unset for replication.
Refs rustfs/backlog#2149
* feat(ecstore): add on-demand migration SourceClient
Add bucket/on_demand_migration/source_client.rs on top of the shared
remote S3 builder: HEAD, ranged streaming GET, ListObjectsV2 with
source-prefix mapping, GetObjectTagging and an admin probe. Every request
carries the x-rustfs-/x-minio-source-proxy-request anti-loop markers and
a RustFS-OnDemandMigration/<version> User-Agent suffix; SSE-C source
objects are rejected as unsupported. SourceError classifies SDK failures
(not found, access denied, throttled, timeout, connect, server error)
with retryability and a stable metrics label. Debug output redacts
credentials.
Refs rustfs/backlog#2149
* docs(operations): point outbound policy at shared remote S3 client builder
* fix(ilm): enqueue committed tier free versions
* fix(ilm): stabilize causal cleanup CI coverage
* test(ilm): make expire GET race deterministic
* test(ilm): synchronize expiry with active GET
* refactor(ecstore): drop the client shim, import rustfs-s3-client directly
Completes the migration window opened by the rustfs-s3-client extraction (rustfs/backlog#1842 PR3): every consumer now imports the client crate directly and the crate::client shim is deleted.
- All in-crate crate::client:: paths (tier warm backends, tier core, lifecycle tier_sweeper, replication storage boundary, set_disk) now import rustfs_s3_client::* directly; crates/ecstore/src/client/mod.rs and the lib.rs mod client declaration are gone.
- The two server-side modules historically misfiled under client/ move to their real homes: object_api_utils.rs to crates/ecstore/src/object_api/ (it builds engine-side object readers/writers), and object_handlers_common.rs to crates/ecstore/src/bucket/lifecycle/ (it is the lifecycle noncurrent-version cleanup helper). The latter now routes its replication calls through the lifecycle replication_sink boundary (schedule_delete wrapper and the sink's ReplicationObjectBridge re-export), as the lifecycle guard requires.
- The ecstore public facade drops api::client: object_api_utils is exposed as api::object_api_utils, and the rustfs crate takes admin_handler_utils (AdminError) from rustfs-s3-client directly (new dependency).
- Guard updates: the migration guard no longer pins mod client in ecstore's lib.rs or the admin_handler_utils facade module (it pins the new api::object_api_utils facade instead), and the module-lint register follows object_api_utils.rs to its new path.
Verification: cargo check -p rustfs-ecstore --all-targets and -p rustfs; cargo fmt --all; tier/transition/lifecycle-focused nextest (626 passed) and the decommission/rebalance/heal families in a filtered run (603 passed; the full-suite parallel run only fails on this machine's known decommission/rebalance baseline flakes, which pass in filtered reruns and fail identically on pristine origin/main); layer/migration/s3s/logging/error-format/doc-path guard scripts all pass.
* docs(architecture): record the S3 client extraction and reword invariant 4 (#6669)
Closes the documentation step of rustfs/backlog#1842. ARCHITECTURE.md invariant 4 now states the serving-vs-consuming distinction the adversarial ruling asked for: ecstore must not serve HTTP/S3 wire types, while consuming remote S3 endpoints is a legitimate engine capability that lives in the extracted rustfs-s3-client crate. The violation note is updated from the pre-extraction snapshot (58 files, embedded client) to the current ratcheted state (shrink-only S3S_ECSTORE_FILES_BASELINE in scripts/check_s3s_footprint.sh, object_lock converted first), and the crate map gains s3-client. ecstore-module-split-plan.md gets the client-directory entry the plan was missing: a Current Shape row and a completed-extraction section describing the pure-move + shim + direct-import sequence and the re-homing of the two misfiled server-side modules.
The object_lock module evaluated WORM state through s3s wire DTOs (ObjectLockRetention, ObjectLockLegalHold, DefaultRetention, Date) and s3s header constants, keeping the storage engine coupled to the serving protocol (rustfs/backlog#1842, ARCHITECTURE.md invariant 4). This PR gives the module its own storage-level vocabulary and pushes the DTO conversions to the boundaries that already speak s3s.
New crates/ecstore/src/bucket/object_lock/types.rs defines RetentionMode, LegalHoldStatus, ObjectRetention, ObjectLegalHold, and DefaultRetention with no s3s dependency. objectlock.rs parses persisted metadata into these types using the rustfs-utils lowercase header constants (the same literal keys as before, pinned by the existing g-key-002 test). objectlock_sys.rs evaluates retention/legal-hold/default-retention from them; the fail-closed error messages and decision logic are unchanged line for line where possible.
Boundary conversions:
- bucket/metadata_sys.rs gains default_retention_from_object_lock_config, converting the persisted s3s configuration into the storage-level DefaultRetention; a rule without a usable GOVERNANCE/COMPLIANCE mode converts to None exactly like the evaluation code always ignored it, and days/years pass through so an invalid period still fails closed at evaluation time.
- check_object_lock_for_deletion_with_config becomes check_object_lock_for_deletion_with_default_retention (it only ever read the default retention); the lifecycle object_lock_boundary keeps the old s3s-typed signature and converts.
- The ObjectLockApi / ObjectLockStatusExt trait impls for the s3s DTOs move next to the persisted configuration owner in bucket/metadata.rs; the traits stay in object_lock/mod.rs.
- check_retention_for_modification now takes Option<RetentionMode>. The serving-layer wrappers (rustfs storage_api, set_disk options path) convert the request string with the new RetentionMode::parse_exact, which accepts only the canonical spelling — preserving the historical literal comparison where a non-canonical requested mode reads as a mode change and stays blocked.
- rustfs app-layer wrappers return the storage types; the replication-overwrite gate in object_usecase.rs uses the typed API (legal_hold.is_on(), RetentionMode::Compliance).
Ratchet: the ecstore-scoped s3s counter drops 42 -> 39 and the repo-wide file counter 211 -> 208 in scripts/check_s3s_footprint.sh.
Verification: cargo check -p rustfs-ecstore --all-targets and -p rustfs (lib+bins); cargo clippy -p rustfs-ecstore --all-targets and -p rustfs --lib --bins (clean); cargo nextest run -p rustfs-ecstore --no-fail-fast (4534/4542; the 8 failures are the same store::rebalance / store::heal machine-baseline set that fails identically on pristine origin/main, plus one fencing flake that passes in isolation); all object_lock/retention/legal-hold tests pass; guard scripts (layer deps, migration rules, s3s footprint, logging, error-format ratchet, doc paths) pass.
Chris