* feat(ecstore): add a native azure blob odm source backend
* feat(ecstore): add a native gcs odm source backend and one backend contract
* fix(ecstore): refuse an empty azure account key at client build
* fix(ecstore): probe gcs sources with the listing permission
* fix(app): drop a redundant match guard on the sse config lookup
* fix(ecstore): drop stale rename commit duplicates from local.rs
* test(ecstore): use the sanctioned placeholder key in the gcs fixture
* fix(ecstore): correct sealed-credential test helper parameter type
The helper took a HashMap that nothing imports, so the ecstore test target did not compile.
* fix(ecstore): fail closed on an unreadable bucket-targets blob
An undecodable bucket-targets.json was replaced by an empty BucketTargets,
so every replication target of that bucket disappeared, replication stopped,
and no caller saw an error. A missing secretKey alone triggers it, because
Credentials has no struct-level serde(default).
parse_all_configs now retains the failure instead: the raw bytes stay and the
typed field stays None, which BucketMetadata::bucket_targets_unreadable reads
as "exists but cannot be read" — the same distinction the fabricated marker
draws for bucket metadata as a whole. One corrupt sub-config still never fails
the metadata load, so an unreadable bucket cannot take down its neighbours or
the node.
BucketTargetSys records such buckets and answers every targets query with the
new BucketRemoteTargetsUnreadable, leaving any snapshot from an earlier
readable load in place so in-flight replication is not torn down. The
replication heal queue reports Missed rather than scheduling against an empty
target set, and the admin listing surfaces the fault instead of an empty list.
Refs: rustfs/backlog#2282
* fix(ecstore): report corrupt permissive bucket configs as invalid
Audit of the remaining parse_all_configs branches. Policy, versioning, object
lock and replication already fail closed at their accessors; encryption,
public access block and quota did not, and for those three "absent" is exactly
the state that grants something — plaintext storage, anonymous access,
unbounded capacity. They now report a stored-but-undecodable payload as
invalid rather than as ConfigNotFound, matching the guard the versioning and
object-lock accessors already use. The quota enforcement path already refused
such a payload; only the metadata read path was misreporting it.
The branches left degrading, and the concrete reason each is safe, are
recorded in the table on parse_all_configs.
Refs: rustfs/backlog#2282
SealScope::encryption_context() returned a HashMap whose key order is
non-deterministic. The FakeSealer test round-trips the context through
JSON serialization, and HashMap's random iteration order caused the
prefix comparison to intermittently fail with 'encryption context mismatch'.
Switch to BTreeMap which guarantees stable key ordering.
* feat(ecstore): add the sealed remote credential seam
Replication targets, remote tiers and on-demand migration sources will all
seal their stored secrets through one envelope rather than three
(rustfs/backlog#2168, design in docs/architecture/remote-credential-sealing-adr.md).
Adds the versioned envelope, the seal scope that binds a ciphertext to the
store, owner and field it belongs to, the sealer registration point, and the
fail-closed error type. ECStore still has no rustfs-kms dependency: the binary
installs a sealer the way it installs the event dispatch hook.
Nothing is wired to a consumer yet, so no stored format changes.
* docs(ecstore): name the event dispatch hook by module, not by symbol
The architecture guard keeps EVENT_DISPATCH_HOOK references inside the
event-notification owner module; the module doc cited the symbol only as an
example of the hook shape, so cite its file instead.
* fix(restore): reject SELECT restore and keep typed S3 errors
RestoreObject accepted `Type=SELECT` requests, but the restore path can
only write the retrieved bytes back to the source key: `put_restore_opts`
built SELECT output options and `restore_transitioned_object` then PUT
them over the source bucket/object. On an unversioned bucket that dropped
`x-amz-restore`, user metadata and tags from the live object; on a
versioned bucket it published a bogus latest version. Nothing was ever
written to `OutputLocation.S3`, yet the response still carried a
fabricated `x-amz-restore-output-path`.
Reject SELECT at the API boundary with a typed NotImplemented, before any
guard or metadata write, and fail closed in `put_restore_opts` as the
backstop for any other caller.
Every other RestoreObject failure was collapsed into a `Custom` error
code, which serializes as a generic retryable 500: a missing key or
version, a malformed version-id, an object that was never transitioned,
an illegal `Days`, and authorization or storage failures all looked the
same to a client. Map them to their S3 identities instead — NoSuchKey,
NoSuchVersion, InvalidArgument, InvalidObjectState, InvalidRequest,
MalformedXML — by preserving `StorageError` through `post_restore_opts`
and letting `ApiError` do the mapping. The intentional 409
RestoreAlreadyInProgress and 503 SlowDown behaviour is unchanged, and
request validation now runs before any lock is taken.
backlog#1341, backlog#2205
* test(restore): give the typed-error regression the ecstore test stack
`execute_restore_object_maps_failures_to_typed_s3_errors` builds a real
ECStore fixture, and under nextest each test runs in a spawned thread with
libtest's 2 MiB stack. On Linux CI that overflowed: the test aborted with
SIGABRT / "fatal runtime error: stack overflow" while every other test in
the run passed.
Add it to the `ecstore-base-stack` filter in both the default and ci
profiles, alongside the other `package(rustfs)` tests that drive the same
store fixture. 4 MiB matches what the deeper multipart and access
roundtrips already use.
The batch `NewerNoncurrentVersions` expiry path took a lifecycle event
argument and ignored it: after `delete_objects` committed it only evicted
the cache and scheduled replication deletes, so a successful noncurrent
version expiry was invisible to notification subscribers while the
equivalent current-version path emitted a lifecycle expiration event.
Emit that event from the batch path too, reusing the existing lifecycle
audit sink and event contract. Only entries that actually mutated
something are announced, and cache eviction and replication scheduling
keep their existing order and admission — the event is derived from the
committed result and a send failure never rolls back a delete.
"No error" is not enough to prove a mutation: the disk layer skips an
absent version and reports success, so a batch entry for a version that
was already gone came back indistinguishable from a committed delete.
The delete plan already resolves whether the source exists, so carry that
`source_missing` result on `DeletedObject` and let the lifecycle path
stay silent for versions it did not remove.
backlog#2202
* test(odm): drive the migration cases from an env-named source
The ODM e2e suite only ever migrates from the in-process fake source, so
path-style addressing, region handling, ETag shape and list pagination on
real implementations stay untested. OdmInteropEnv resolves the source from
RUSTFS_ODM_INTEROP_*, seeding into a per-run source_prefix so a shared real
bucket can host concurrent runs and every seeded key is removed afterwards.
A named provider with a missing variable is an error, never a silent
fallback to the fake source.
interop_test holds the four cases that run against either source, and the
e2e-odm-interop profile is the lane that selects them; e2e-full excludes
them, so its committed selection is unchanged. wait_until_odm_engaged
replaces the fake source's journal probe for the readiness wait, since a
real source keeps no journal.
* ci(odm): add the scheduled provider interop lane
on-demand-migration-interop.yml runs the interop cases against a pinned
MinIO container with a 5,000-object backfill - past the fake source's 4,096
version and journal caps - and the three-case minimum against AWS, R2 and
GCS when their ODM_INTEROP_* secrets exist, skipping with a summary note
when they do not. Each provider gets one JSON report merging the per-case
entries with the nextest JUnit, which stays authoritative for what ran.
Report-only and never required: it depends on third-party endpoints and on
secrets a fork does not have.
houseme