* feat(ecstore): add a native azure blob odm source backend
* feat(ecstore): add a native gcs odm source backend and one backend contract
* fix(ecstore): refuse an empty azure account key at client build
* fix(ecstore): probe gcs sources with the listing permission
* fix(app): drop a redundant match guard on the sse config lookup
* fix(ecstore): drop stale rename commit duplicates from local.rs
* test(ecstore): use the sanctioned placeholder key in the gcs fixture
A source-side DELETE ?versionId=<marker> replicates as a version purge, and
replicate_delete_to_target addressed it by the SOURCE marker id on every
target. A generic S3 target answers a DELETE of an unknown versionId with
204 and keeps its marker, so the purge reported success and the marker
stayed; the same event also spawned a second delayed-purge watcher that
journaled a duplicate intent. Real VMs (R6.1 in backlog#2080) failed on the
persisted-id fix alone because this path never consulted the mapping.
Resolve the target version through the recorded mapping for marker purges
(a corrupt record refuses, as the watcher does; nothing recorded keeps the
source-derived id for id-mirroring peers), and do not spawn the delayed
watcher for a version purge — that purge is the replication itself and its
failures reach the journal as a purge entry.
Service-account items carry their claims in a HashMap, and serde_json is
built with preserve_order, so two serializations of the same plan could
differ in key order. The repair preflight token then went stale between
dry-run and execute (412 on the real VMs once snapshots carried service
accounts) and a retry snapshot resend could never look stable. Serialize
through a key-sorted JSON value for the task id and the fingerprint.
The service-account receive path used only the live record's timestamp; a
deleted account left nothing to compare against, so a stale create from a
snapshot or a delayed delivery could recreate it. Consult the recorded
deletion mark when the record is absent, as the user path does. The site
replicator account is managed by join/rotate and stays exempt.
A delete-marker purge intent that outlived its watch window was journaled
without the version ids the targets assigned to the replicated markers.
Replay rebuilt the replication state from a blank ObjectInfo, so
`delete_marker_purge_version_id` fell back to the source marker id; a
generic S3 target that mints its own ids answers that DELETE with 204,
the entry was acknowledged and the real marker stayed on the target.
- `MrfReplicateEntry` gains `targetDeleteMarkerVersionIDs` (per-ARN map)
and `targetDeleteMarkerVersionIDsCorrupt`; both default and are skipped
when empty/false, so old journals decode to the pre-existing shape.
- `DeletedObjectReplicationInfo::to_mrf_entry` copies both from the
source replication state; `reconstructed_heal_delete_info` restores
them into the replayed state so the purge addresses the recorded id
and a fail-closed refusal stays a refusal after restart.
- MRF envelope capability bit `TargetDeleteMarkerVersionIds` (1 << 4)
fences the field like `DeleteMarkerMtime`; readers without the bit
refuse envelopes that advertise it, current readers accept old ones.
(cherry picked from commit ddacaaa185fda7a5f426138ba5b179f986b862d9)
The bucket-meta receiver judged an incoming item stale by comparing its
source `updated_at` with the `*_config_updated_at` stamp of the config on
disk, but that stamp was the receiver's local clock at apply time
(`BucketMetadata::update_config`). A source edit newer than the applied one
but delivered after the local stamp was judged stale and acknowledged with
200: two quick source edits under delivery delay lose the second, and a peer
clock ahead of ours loses every follow-up edit inside the skew.
Add explicit-timestamp write entries, expanding rather than changing the
existing ones:
- `BucketMetadata::update_config_at`; `update_config` delegates to it with
the local clock.
- `metadata_sys::update_if_incarnation_at`,
`update_under_transaction_lock_at`, `update_quota_if_incarnation_at`,
threaded through the shared write-guard path as `Option<OffsetDateTime>`
(`None` keeps local stamping for every existing caller and for deletes).
- Re-exported through the ecstore `api` facade and the rustfs admin
`storage_api::metadata_sys` facade.
`apply_bucket_meta_item` now persists policy, tags, versioning, object-lock,
sse, replication, quota and cors configs with the item's source time, so the
stored stamp equals the source `updatedAt` and staleness is judged source
time against source time. Items without `updated_at` keep the local stamp.
lc-config stays on the local stamp: its staleness axis is the in-document
`expiry_updated_at` the merge records, and the whole-config time only serves
as its deletion / legacy lower bound. Local (non-replicated) edits keep
stamping the local clock — they are the source.
(cherry picked from commit c1009c018b217ef9edc7773c8e56667ea7e77335)
The staleness gate judged an incoming IAM item against the local record's
timestamp, but a full revoke deletes the record: `policy_db_set(.., "")`
removes the mapping, `delete_policy` the document, a user delete the
identity, and the IAM cache keeps only a per-entity watermark, no per-key
deletion time. With nothing left to compare against, a delayed older grant
was still applied after the revoke (real-VM case R6.3a of backlog#2080:
detach on A, revoke reaches B, an older mapping grant lands on B with 200
and re-grants access).
Keep a bounded, persisted map of deleted entity -> source `updatedAt` of the
newest deletion committed here in the site-replication state, written
through the state transaction in two places: the local IAM change hook
records the mark before broadcasting a deletion-shaped item, and the peer
item handler records it after applying (or idempotently no-op'ing) one. The
`policy`, `policy-mapping`, `group-info` and `iam-user` receive paths feed
that mark into the shared verdict when the record is absent, so a grant
older than the recorded deletion is acknowledged without being applied.
Group member removals are marked per member and a group delete marks the
group itself, so a stale re-add of a removed member is judged against the
newest of those marks.
Marks need a source timestamp: items without `updatedAt` (older peers) and
an unreadable state object fall back to today's behaviour and apply. The
map holds at most 1024 entries, evicting the oldest, and is cleared when
this site leaves the cluster.
(cherry picked from commit 0595c600d6091f583858176232d87ef2eed2bf8f)
The `policy`, `policy-mapping` and `group-info` receive paths applied every
incoming item unconditionally, so a delayed older grant (wide policy body,
old mapping, old group add) overwrote a newer revoke on the peer. `iam-user`
and `service-account` already compared the item's `updatedAt` with the local
record.
Route the three paths through one pure verdict helper: an item older than
the local record is acknowledged without being applied; items without a
source timestamp and items targeting an absent record keep today's behaviour
(older peers, idempotent deletes from backlog#2071). Deletes are gated the
same way so an older delete cannot remove a newer record.
The group record's own timestamp now moves on every membership and status
change instead of staying at creation, so the gate judges group items
against the last change. Add the IamSys accessors the gate reads
(`get_policy_doc`, `get_mapped_policy_record`, `get_group_info`).
(cherry picked from commit 98c32093406cb47014b7eda2fe139f01079de337)
The generic JSON broadcast (make/delete bucket, bucket-meta hook, bucket
ops) returned at the first failing peer, so peers later in deployment-id
order never received the request and got no retry event; a transport
construction failure recorded nothing at all.
Attempt every remote peer like the IAM change hook does: a success settles
the peer/path retry event, a failure (transport construction included)
enqueues one under the request path, and the first error is returned after
all peers were attempted.
(cherry picked from commit ce8f73bfd74bac61c383434780f27d17ca16d75e)
With the generic broadcast now attempting every peer and returning the
first failure, stopping after the make step on that error skipped
configure-replication for the peers whose make had just succeeded, and no
retry event covered the gap. Run both steps and combine the results.
(cherry picked from commit cf5c0476dc4265b6214bf34c47a9ae4c61bf8e3e)
The IAM snapshot used by the retry drain resend, repair and site-add
bootstrap was built from `list_users`, which strips secret keys and skips
service accounts. The plan builder dropped every user for lack of a
secret, so a user disable, secret rotation or service-account change
committed while a peer was unreachable never reached it — while the
collapsed retry entry was settled and repair reported success.
Read the credentials separately at plan time (`build_sr_iam_credentials`,
used only on peer-delivery paths) so `SRInfo`, which is served to admin
callers, stays secret-free. Users travel with secret, status and the user
record's own update time; service accounts (except the replicator's) travel
as the create item the live hook emits, after their parents. The receiver
applies a disabled status after creating a new service account, and the
retry snapshot tombstones removed service accounts like the other kinds.
`encode_service_account_replication_policy` moves into the infra layer so
the snapshot builder can share it with the live hook.
(cherry picked from commit bd8cd497c965ae331fafa20362763224cec3b5b2)
chore(deps): refresh SDKs and pin clock skew regression coverage
Refresh compatible dependencies for Scanner/Heal V2 batch 1 and verify
the production S3 retry/signing path with a deterministic clock.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
The workflow_dispatch inputs already accept arbitrary release tags, but
the run failed late and unclearly when a tag had no .deb asset, and the
from_version default pointed at 1.0.0-rc.4-preview.1, whose release
ships no .deb at all - so scheduled runs died on a 404 while installing
the old package.
- Add a fail-fast preflight that resolves each requested tag via the
GitHub release API and verifies the rustfs_<tag>_amd64.deb asset
exists before the suite starts, with an actionable error message
otherwise (e.g. 1.0.0-rc.4 ships only zip/sbom assets).
- Change the from_version default to 1.0.0-rc.3, the newest release
that actually ships a .deb asset.
- Reword the from_version/to_version descriptions so manual triggers
state the .deb-asset requirement and the nightly fallback.
- Pass PF_TESTING_GH_TOKEN as GH_TOKEN to the suite step for the gh api
release lookups, matching the other functional workflows.
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
* test(ecstore): require core invariant tests in existing CI lane
* test(ci): require a fresh core JUnit report
* test(ecstore): match sealed context fixture map type
The object write path read the bucket default encryption configuration
with `.ok()`, which made "this bucket has no default encryption" and "the
encryption configuration cannot be read" the same value. A bucket whose
encryption blob is damaged therefore stored plaintext objects the
operator had mandated be encrypted, with nothing returned to the client
and nothing in the object to tell those writes apart afterwards.
PUT, COPY and the snowball extract path now share one resolver: an
absent configuration still writes plaintext exactly as before, and every
other outcome refuses the write, carrying the accessor's typed error so
a damaged blob surfaces as a deterministic InternalError while a
transient metadata read failure surfaces as the retryable
ServiceUnavailable. A missing bucket and a cold metadata cache both
still resolve to "no configuration", so neither becomes a refusal. This
matches `prepare_sse_configuration` in `storage::sse`, the resolver the
multipart writer has always used, which fails closed on this lookup.
* fix(ecstore): correct sealed-credential test helper parameter type
The helper took a HashMap that nothing imports, so the ecstore test target did not compile.
* fix(ecstore): fail closed on an unreadable bucket-targets blob
An undecodable bucket-targets.json was replaced by an empty BucketTargets,
so every replication target of that bucket disappeared, replication stopped,
and no caller saw an error. A missing secretKey alone triggers it, because
Credentials has no struct-level serde(default).
parse_all_configs now retains the failure instead: the raw bytes stay and the
typed field stays None, which BucketMetadata::bucket_targets_unreadable reads
as "exists but cannot be read" — the same distinction the fabricated marker
draws for bucket metadata as a whole. One corrupt sub-config still never fails
the metadata load, so an unreadable bucket cannot take down its neighbours or
the node.
BucketTargetSys records such buckets and answers every targets query with the
new BucketRemoteTargetsUnreadable, leaving any snapshot from an earlier
readable load in place so in-flight replication is not torn down. The
replication heal queue reports Missed rather than scheduling against an empty
target set, and the admin listing surfaces the fault instead of an empty list.
Refs: rustfs/backlog#2282
* fix(ecstore): report corrupt permissive bucket configs as invalid
Audit of the remaining parse_all_configs branches. Policy, versioning, object
lock and replication already fail closed at their accessors; encryption,
public access block and quota did not, and for those three "absent" is exactly
the state that grants something — plaintext storage, anonymous access,
unbounded capacity. They now report a stored-but-undecodable payload as
invalid rather than as ConfigNotFound, matching the guard the versioning and
object-lock accessors already use. The quota enforcement path already refused
such a payload; only the metadata read path was misreporting it.
The branches left degrading, and the concrete reason each is safe, are
recorded in the table on parse_all_configs.
Refs: rustfs/backlog#2282
Zhengchao An