Review findings on rustfs#7195: the receive-side staleness gate compared a
source `updatedAt` against a stamp the local write had put on the record,
and the verdict, the write and the deletion mark were three separate steps.
- IAM writes gain explicit-stamp variants (`set_policy_at`, `policy_db_set_at`,
group and user `*_at`, `new_service_account_at`, `update_service_account_at`)
so a replicated record carries its source time; local edits are unchanged.
- `apply_iam_item` runs verdict, write and mark commit under the
site-replication state transaction (distributed state-object lock), so a
concurrent older grant and newer revoke are ordered on every node.
- A replicated service account is created with its source status in one
write (`NewServiceAccountOpts::status`), never enabled transiently.
- Deletion marks are pruned by age (30 days) instead of by count.
- Bucket-config deletes persist the source stamp (`delete_if_incarnation_at`).
Regressions run through the real receiver: delayed in-order updates for every
gated item type, concurrent grant/revoke, delete then stale re-create, disabled
service-account create, delete stamping in ecstore, and mark retention.
* fix(ecstore): restore odm source contract tests
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ci): initialize replication evidence in chain test
Run the replication workflow's evidence initialization before the chain handoff self-test executes the suite step. This keeps the test model aligned with the workflow-provided LOG_FILE and TMPDIR values.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(odm): distinguish missing GCS buckets from object misses (#7221)
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
test(e2e): prove bucket config survives rc.5 upgrade and rollback
Add two upgrade-compatibility scenarios pinned to the on-demand-migration
series' on-disk surfaces: BucketMetadata's 44 -> 46 msgpack keys, the
fail-closed bucket-config reads of rustfs#7172, the encryption-gated PUT
path of rustfs#7183, and the default-on migration module of rustfs#7089.
The upgrade case writes versioning, SSE-S3 default encryption, a validated
replication target plus rule, lifecycle, tags, quota, a public access block,
a bucket policy and an object lock configuration with the pinned previous
release, then asserts each one reads back unchanged on the current build,
that list-remote-targets still reports the target, that writes to the
encrypted and plain buckets keep their encryption posture, that every
pre-upgrade object including a multipart one is byte-identical, and that an
unconfigured bucket reports no migration and still answers NoSuchKey.
The rollback case is the reverse: the current build writes the 46-key blob
and the previous release must decode it by skipping the two unknown keys.
* fix(ci): preserve reported functional suite failures
* fix(ci): isolate functional evidence and preserve every result
* fix(ci): exclude sensitive scratch files from suite artifacts
Two fixes for the functional test chain:
1. Report upload fails with 'jq: Argument list too long' when the base64
report is passed through '--arg content' (pool reports exceed the OS
argv limit; last night's pool run lost its Step Results report this
way). Write the base64 payload to a temp file and load it in jq via
--rawfile instead. Applied uniformly to all nine suite workflows
that share this upload step.
2. The security workflow cloned rustfs/auto-testing into the workspace
and then ran actions/checkout at the workspace root for the OIDC
live gate script, which wiped the auto-testing clone and killed the
suite with 'chmod: cannot access auto-testing/rustfs-security-test.sh'.
Check out the repository into the rustfs-repo/ subdirectory instead
and point RUSTFS_SECURITY_OIDC_LIVE_SCRIPT there.
Co-authored-by: rustfs-ci <ci@rustfs.com>
* feat(ecstore): add a native azure blob odm source backend
* feat(ecstore): add a native gcs odm source backend and one backend contract
* fix(ecstore): refuse an empty azure account key at client build
* fix(ecstore): probe gcs sources with the listing permission
* fix(app): drop a redundant match guard on the sse config lookup
* fix(ecstore): drop stale rename commit duplicates from local.rs
* test(ecstore): use the sanctioned placeholder key in the gcs fixture
A source-side DELETE ?versionId=<marker> replicates as a version purge, and
replicate_delete_to_target addressed it by the SOURCE marker id on every
target. A generic S3 target answers a DELETE of an unknown versionId with
204 and keeps its marker, so the purge reported success and the marker
stayed; the same event also spawned a second delayed-purge watcher that
journaled a duplicate intent. Real VMs (R6.1 in backlog#2080) failed on the
persisted-id fix alone because this path never consulted the mapping.
Resolve the target version through the recorded mapping for marker purges
(a corrupt record refuses, as the watcher does; nothing recorded keeps the
source-derived id for id-mirroring peers), and do not spawn the delayed
watcher for a version purge — that purge is the replication itself and its
failures reach the journal as a purge entry.
Service-account items carry their claims in a HashMap, and serde_json is
built with preserve_order, so two serializations of the same plan could
differ in key order. The repair preflight token then went stale between
dry-run and execute (412 on the real VMs once snapshots carried service
accounts) and a retry snapshot resend could never look stable. Serialize
through a key-sorted JSON value for the task id and the fingerprint.
The service-account receive path used only the live record's timestamp; a
deleted account left nothing to compare against, so a stale create from a
snapshot or a delayed delivery could recreate it. Consult the recorded
deletion mark when the record is absent, as the user path does. The site
replicator account is managed by join/rotate and stays exempt.
A delete-marker purge intent that outlived its watch window was journaled
without the version ids the targets assigned to the replicated markers.
Replay rebuilt the replication state from a blank ObjectInfo, so
`delete_marker_purge_version_id` fell back to the source marker id; a
generic S3 target that mints its own ids answers that DELETE with 204,
the entry was acknowledged and the real marker stayed on the target.
- `MrfReplicateEntry` gains `targetDeleteMarkerVersionIDs` (per-ARN map)
and `targetDeleteMarkerVersionIDsCorrupt`; both default and are skipped
when empty/false, so old journals decode to the pre-existing shape.
- `DeletedObjectReplicationInfo::to_mrf_entry` copies both from the
source replication state; `reconstructed_heal_delete_info` restores
them into the replayed state so the purge addresses the recorded id
and a fail-closed refusal stays a refusal after restart.
- MRF envelope capability bit `TargetDeleteMarkerVersionIds` (1 << 4)
fences the field like `DeleteMarkerMtime`; readers without the bit
refuse envelopes that advertise it, current readers accept old ones.
(cherry picked from commit ddacaaa185fda7a5f426138ba5b179f986b862d9)
The bucket-meta receiver judged an incoming item stale by comparing its
source `updated_at` with the `*_config_updated_at` stamp of the config on
disk, but that stamp was the receiver's local clock at apply time
(`BucketMetadata::update_config`). A source edit newer than the applied one
but delivered after the local stamp was judged stale and acknowledged with
200: two quick source edits under delivery delay lose the second, and a peer
clock ahead of ours loses every follow-up edit inside the skew.
Add explicit-timestamp write entries, expanding rather than changing the
existing ones:
- `BucketMetadata::update_config_at`; `update_config` delegates to it with
the local clock.
- `metadata_sys::update_if_incarnation_at`,
`update_under_transaction_lock_at`, `update_quota_if_incarnation_at`,
threaded through the shared write-guard path as `Option<OffsetDateTime>`
(`None` keeps local stamping for every existing caller and for deletes).
- Re-exported through the ecstore `api` facade and the rustfs admin
`storage_api::metadata_sys` facade.
`apply_bucket_meta_item` now persists policy, tags, versioning, object-lock,
sse, replication, quota and cors configs with the item's source time, so the
stored stamp equals the source `updatedAt` and staleness is judged source
time against source time. Items without `updated_at` keep the local stamp.
lc-config stays on the local stamp: its staleness axis is the in-document
`expiry_updated_at` the merge records, and the whole-config time only serves
as its deletion / legacy lower bound. Local (non-replicated) edits keep
stamping the local clock — they are the source.
(cherry picked from commit c1009c018b217ef9edc7773c8e56667ea7e77335)
The staleness gate judged an incoming IAM item against the local record's
timestamp, but a full revoke deletes the record: `policy_db_set(.., "")`
removes the mapping, `delete_policy` the document, a user delete the
identity, and the IAM cache keeps only a per-entity watermark, no per-key
deletion time. With nothing left to compare against, a delayed older grant
was still applied after the revoke (real-VM case R6.3a of backlog#2080:
detach on A, revoke reaches B, an older mapping grant lands on B with 200
and re-grants access).
Keep a bounded, persisted map of deleted entity -> source `updatedAt` of the
newest deletion committed here in the site-replication state, written
through the state transaction in two places: the local IAM change hook
records the mark before broadcasting a deletion-shaped item, and the peer
item handler records it after applying (or idempotently no-op'ing) one. The
`policy`, `policy-mapping`, `group-info` and `iam-user` receive paths feed
that mark into the shared verdict when the record is absent, so a grant
older than the recorded deletion is acknowledged without being applied.
Group member removals are marked per member and a group delete marks the
group itself, so a stale re-add of a removed member is judged against the
newest of those marks.
Marks need a source timestamp: items without `updatedAt` (older peers) and
an unreadable state object fall back to today's behaviour and apply. The
map holds at most 1024 entries, evicting the oldest, and is cleared when
this site leaves the cluster.
(cherry picked from commit 0595c600d6091f583858176232d87ef2eed2bf8f)
The `policy`, `policy-mapping` and `group-info` receive paths applied every
incoming item unconditionally, so a delayed older grant (wide policy body,
old mapping, old group add) overwrote a newer revoke on the peer. `iam-user`
and `service-account` already compared the item's `updatedAt` with the local
record.
Route the three paths through one pure verdict helper: an item older than
the local record is acknowledged without being applied; items without a
source timestamp and items targeting an absent record keep today's behaviour
(older peers, idempotent deletes from backlog#2071). Deletes are gated the
same way so an older delete cannot remove a newer record.
The group record's own timestamp now moves on every membership and status
change instead of staying at creation, so the gate judges group items
against the last change. Add the IamSys accessors the gate reads
(`get_policy_doc`, `get_mapped_policy_record`, `get_group_info`).
(cherry picked from commit 98c32093406cb47014b7eda2fe139f01079de337)
The generic JSON broadcast (make/delete bucket, bucket-meta hook, bucket
ops) returned at the first failing peer, so peers later in deployment-id
order never received the request and got no retry event; a transport
construction failure recorded nothing at all.
Attempt every remote peer like the IAM change hook does: a success settles
the peer/path retry event, a failure (transport construction included)
enqueues one under the request path, and the first error is returned after
all peers were attempted.
(cherry picked from commit ce8f73bfd74bac61c383434780f27d17ca16d75e)
With the generic broadcast now attempting every peer and returning the
first failure, stopping after the make step on that error skipped
configure-replication for the peers whose make had just succeeded, and no
retry event covered the gap. Run both steps and combine the results.
(cherry picked from commit cf5c0476dc4265b6214bf34c47a9ae4c61bf8e3e)
唐小鸭