Review findings on rustfs#7195: the receive-side staleness gate compared a
source `updatedAt` against a stamp the local write had put on the record,
and the verdict, the write and the deletion mark were three separate steps.
- IAM writes gain explicit-stamp variants (`set_policy_at`, `policy_db_set_at`,
group and user `*_at`, `new_service_account_at`, `update_service_account_at`)
so a replicated record carries its source time; local edits are unchanged.
- `apply_iam_item` runs verdict, write and mark commit under the
site-replication state transaction (distributed state-object lock), so a
concurrent older grant and newer revoke are ordered on every node.
- A replicated service account is created with its source status in one
write (`NewServiceAccountOpts::status`), never enabled transiently.
- Deletion marks are pruned by age (30 days) instead of by count.
- Bucket-config deletes persist the source stamp (`delete_if_incarnation_at`).
Regressions run through the real receiver: delayed in-order updates for every
gated item type, concurrent grant/revoke, delete then stale re-create, disabled
service-account create, delete stamping in ecstore, and mark retention.
* feat(ecstore): add a native azure blob odm source backend
* feat(ecstore): add a native gcs odm source backend and one backend contract
* fix(ecstore): refuse an empty azure account key at client build
* fix(ecstore): probe gcs sources with the listing permission
* fix(app): drop a redundant match guard on the sse config lookup
* fix(ecstore): drop stale rename commit duplicates from local.rs
* test(ecstore): use the sanctioned placeholder key in the gcs fixture
A source-side DELETE ?versionId=<marker> replicates as a version purge, and
replicate_delete_to_target addressed it by the SOURCE marker id on every
target. A generic S3 target answers a DELETE of an unknown versionId with
204 and keeps its marker, so the purge reported success and the marker
stayed; the same event also spawned a second delayed-purge watcher that
journaled a duplicate intent. Real VMs (R6.1 in backlog#2080) failed on the
persisted-id fix alone because this path never consulted the mapping.
Resolve the target version through the recorded mapping for marker purges
(a corrupt record refuses, as the watcher does; nothing recorded keeps the
source-derived id for id-mirroring peers), and do not spawn the delayed
watcher for a version purge — that purge is the replication itself and its
failures reach the journal as a purge entry.
A delete-marker purge intent that outlived its watch window was journaled
without the version ids the targets assigned to the replicated markers.
Replay rebuilt the replication state from a blank ObjectInfo, so
`delete_marker_purge_version_id` fell back to the source marker id; a
generic S3 target that mints its own ids answers that DELETE with 204,
the entry was acknowledged and the real marker stayed on the target.
- `MrfReplicateEntry` gains `targetDeleteMarkerVersionIDs` (per-ARN map)
and `targetDeleteMarkerVersionIDsCorrupt`; both default and are skipped
when empty/false, so old journals decode to the pre-existing shape.
- `DeletedObjectReplicationInfo::to_mrf_entry` copies both from the
source replication state; `reconstructed_heal_delete_info` restores
them into the replayed state so the purge addresses the recorded id
and a fail-closed refusal stays a refusal after restart.
- MRF envelope capability bit `TargetDeleteMarkerVersionIds` (1 << 4)
fences the field like `DeleteMarkerMtime`; readers without the bit
refuse envelopes that advertise it, current readers accept old ones.
(cherry picked from commit ddacaaa185fda7a5f426138ba5b179f986b862d9)
The bucket-meta receiver judged an incoming item stale by comparing its
source `updated_at` with the `*_config_updated_at` stamp of the config on
disk, but that stamp was the receiver's local clock at apply time
(`BucketMetadata::update_config`). A source edit newer than the applied one
but delivered after the local stamp was judged stale and acknowledged with
200: two quick source edits under delivery delay lose the second, and a peer
clock ahead of ours loses every follow-up edit inside the skew.
Add explicit-timestamp write entries, expanding rather than changing the
existing ones:
- `BucketMetadata::update_config_at`; `update_config` delegates to it with
the local clock.
- `metadata_sys::update_if_incarnation_at`,
`update_under_transaction_lock_at`, `update_quota_if_incarnation_at`,
threaded through the shared write-guard path as `Option<OffsetDateTime>`
(`None` keeps local stamping for every existing caller and for deletes).
- Re-exported through the ecstore `api` facade and the rustfs admin
`storage_api::metadata_sys` facade.
`apply_bucket_meta_item` now persists policy, tags, versioning, object-lock,
sse, replication, quota and cors configs with the item's source time, so the
stored stamp equals the source `updatedAt` and staleness is judged source
time against source time. Items without `updated_at` keep the local stamp.
lc-config stays on the local stamp: its staleness axis is the in-document
`expiry_updated_at` the merge records, and the whole-config time only serves
as its deletion / legacy lower bound. Local (non-replicated) edits keep
stamping the local clock — they are the source.
(cherry picked from commit c1009c018b217ef9edc7773c8e56667ea7e77335)
The `policy`, `policy-mapping` and `group-info` receive paths applied every
incoming item unconditionally, so a delayed older grant (wide policy body,
old mapping, old group add) overwrote a newer revoke on the peer. `iam-user`
and `service-account` already compared the item's `updatedAt` with the local
record.
Route the three paths through one pure verdict helper: an item older than
the local record is acknowledged without being applied; items without a
source timestamp and items targeting an absent record keep today's behaviour
(older peers, idempotent deletes from backlog#2071). Deletes are gated the
same way so an older delete cannot remove a newer record.
The group record's own timestamp now moves on every membership and status
change instead of staying at creation, so the gate judges group items
against the last change. Add the IamSys accessors the gate reads
(`get_policy_doc`, `get_mapped_policy_record`, `get_group_info`).
(cherry picked from commit 98c32093406cb47014b7eda2fe139f01079de337)
* fix(ecstore): correct sealed-credential test helper parameter type
The helper took a HashMap that nothing imports, so the ecstore test target did not compile.
* fix(ecstore): fail closed on an unreadable bucket-targets blob
An undecodable bucket-targets.json was replaced by an empty BucketTargets,
so every replication target of that bucket disappeared, replication stopped,
and no caller saw an error. A missing secretKey alone triggers it, because
Credentials has no struct-level serde(default).
parse_all_configs now retains the failure instead: the raw bytes stay and the
typed field stays None, which BucketMetadata::bucket_targets_unreadable reads
as "exists but cannot be read" — the same distinction the fabricated marker
draws for bucket metadata as a whole. One corrupt sub-config still never fails
the metadata load, so an unreadable bucket cannot take down its neighbours or
the node.
BucketTargetSys records such buckets and answers every targets query with the
new BucketRemoteTargetsUnreadable, leaving any snapshot from an earlier
readable load in place so in-flight replication is not torn down. The
replication heal queue reports Missed rather than scheduling against an empty
target set, and the admin listing surfaces the fault instead of an empty list.
Refs: rustfs/backlog#2282
* fix(ecstore): report corrupt permissive bucket configs as invalid
Audit of the remaining parse_all_configs branches. Policy, versioning, object
lock and replication already fail closed at their accessors; encryption,
public access block and quota did not, and for those three "absent" is exactly
the state that grants something — plaintext storage, anonymous access,
unbounded capacity. They now report a stored-but-undecodable payload as
invalid rather than as ConfigNotFound, matching the guard the versioning and
object-lock accessors already use. The quota enforcement path already refused
such a payload; only the metadata read path was misreporting it.
The branches left degrading, and the concrete reason each is safe, are
recorded in the table on parse_all_configs.
Refs: rustfs/backlog#2282
SealScope::encryption_context() returned a HashMap whose key order is
non-deterministic. The FakeSealer test round-trips the context through
JSON serialization, and HashMap's random iteration order caused the
prefix comparison to intermittently fail with 'encryption context mismatch'.
Switch to BTreeMap which guarantees stable key ordering.
* feat(ecstore): add the sealed remote credential seam
Replication targets, remote tiers and on-demand migration sources will all
seal their stored secrets through one envelope rather than three
(rustfs/backlog#2168, design in docs/architecture/remote-credential-sealing-adr.md).
Adds the versioned envelope, the seal scope that binds a ciphertext to the
store, owner and field it belongs to, the sealer registration point, and the
fail-closed error type. ECStore still has no rustfs-kms dependency: the binary
installs a sealer the way it installs the event dispatch hook.
Nothing is wired to a consumer yet, so no stored format changes.
* docs(ecstore): name the event dispatch hook by module, not by symbol
The architecture guard keeps EVENT_DISPATCH_HOOK references inside the
event-notification owner module; the module doc cited the symbol only as an
example of the hook shape, so cite its file instead.
* fix(restore): reject SELECT restore and keep typed S3 errors
RestoreObject accepted `Type=SELECT` requests, but the restore path can
only write the retrieved bytes back to the source key: `put_restore_opts`
built SELECT output options and `restore_transitioned_object` then PUT
them over the source bucket/object. On an unversioned bucket that dropped
`x-amz-restore`, user metadata and tags from the live object; on a
versioned bucket it published a bogus latest version. Nothing was ever
written to `OutputLocation.S3`, yet the response still carried a
fabricated `x-amz-restore-output-path`.
Reject SELECT at the API boundary with a typed NotImplemented, before any
guard or metadata write, and fail closed in `put_restore_opts` as the
backstop for any other caller.
Every other RestoreObject failure was collapsed into a `Custom` error
code, which serializes as a generic retryable 500: a missing key or
version, a malformed version-id, an object that was never transitioned,
an illegal `Days`, and authorization or storage failures all looked the
same to a client. Map them to their S3 identities instead — NoSuchKey,
NoSuchVersion, InvalidArgument, InvalidObjectState, InvalidRequest,
MalformedXML — by preserving `StorageError` through `post_restore_opts`
and letting `ApiError` do the mapping. The intentional 409
RestoreAlreadyInProgress and 503 SlowDown behaviour is unchanged, and
request validation now runs before any lock is taken.
backlog#1341, backlog#2205
* test(restore): give the typed-error regression the ecstore test stack
`execute_restore_object_maps_failures_to_typed_s3_errors` builds a real
ECStore fixture, and under nextest each test runs in a spawned thread with
libtest's 2 MiB stack. On Linux CI that overflowed: the test aborted with
SIGABRT / "fatal runtime error: stack overflow" while every other test in
the run passed.
Add it to the `ecstore-base-stack` filter in both the default and ci
profiles, alongside the other `package(rustfs)` tests that drive the same
store fixture. 4 MiB matches what the deeper multipart and access
roundtrips already use.
The batch `NewerNoncurrentVersions` expiry path took a lifecycle event
argument and ignored it: after `delete_objects` committed it only evicted
the cache and scheduled replication deletes, so a successful noncurrent
version expiry was invisible to notification subscribers while the
equivalent current-version path emitted a lifecycle expiration event.
Emit that event from the batch path too, reusing the existing lifecycle
audit sink and event contract. Only entries that actually mutated
something are announced, and cache eviction and replication scheduling
keep their existing order and admission — the event is derived from the
committed result and a send failure never rolls back a delete.
"No error" is not enough to prove a mutation: the disk layer skips an
absent version and reports success, so a batch entry for a version that
was already gone came back indistinguishable from a committed delete.
The delete plan already resolves whether the source exists, so carry that
`source_missing` result on `DeletedObject` and let the lifecycle path
stay silent for versions it did not remove.
backlog#2202
唐小鸭