gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-09-06 12:09:12 +00:00
Code Issues Packages Projects Releases Wiki Activity

test(odm): add the scheduled provider interop lane (#7114)

Browse Source 
* test(odm): drive the migration cases from an env-named source

The ODM e2e suite only ever migrates from the in-process fake source, so
path-style addressing, region handling, ETag shape and list pagination on
real implementations stay untested. OdmInteropEnv resolves the source from
RUSTFS_ODM_INTEROP_*, seeding into a per-run source_prefix so a shared real
bucket can host concurrent runs and every seeded key is removed afterwards.
A named provider with a missing variable is an error, never a silent
fallback to the fake source.

interop_test holds the four cases that run against either source, and the
e2e-odm-interop profile is the lane that selects them; e2e-full excludes
them, so its committed selection is unchanged. wait_until_odm_engaged
replaces the fake source's journal probe for the readiness wait, since a
real source keeps no journal.

* ci(odm): add the scheduled provider interop lane

on-demand-migration-interop.yml runs the interop cases against a pinned
MinIO container with a 5,000-object backfill - past the fake source's 4,096
version and journal caps - and the three-case minimum against AWS, R2 and
GCS when their ODM_INTEROP_* secrets exist, skipping with a summary note
when they do not. Each provider gets one JSON report merging the per-case
entries with the nextest JUnit, which stays authoritative for what ran.

Report-only and never required: it depends on third-party endpoints and on
secrets a fork does not have.
This commit is contained in:
Zhengchao An
2026-09-04 10:05:40 +08:00
committed by GitHub
parent 9863f4848d
commit 9d10d69a6d
9 changed files with 1138 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/operations/on-demand-migration.md
+10 -8
View File
@@ -125,17 +125,19 @@ Validation also rejects two shapes outright: a source whose endpoint and bucket
## Provider presets and source permissions
| Provider | Endpoint | Addressing | `region` | Notes |
|---|---|---|---|---|
| `aws` | Optional; derived as `https://s3.<region>.amazonaws.com` | Virtual-host | Real region required (`auto` rejected) | The derived form only accepts `[A-Za-z0-9-]` in `region` |
| `s3` | Required | Path-style | Real region | Generic S3-compatible endpoint (Wasabi, Backblaze B2 S3 API, Ceph RGW, …) |
| `minio` | Required | Path-style | `auto` allowed | |
| `rustfs` | Required | Path-style | `auto` allowed | A RustFS source answers the migration request locally thanks to the anti-loop marker |
| `r2` | `https://<account-id>.r2.cloudflarestorage.com` | Virtual-host | `auto` allowed (signed as `us-east-1`) | |
| `gcs` | `https://storage.googleapis.com` | Virtual-host | Real region required | Uses the GCS XML interoperability API with an HMAC key pair, not a service-account JSON key |
| Provider | Endpoint | Addressing | `region` | Notes | Interop evidence |
|---|---|---|---|---|---|
| `aws` | Optional; derived as `https://s3.<region>.amazonaws.com` | Virtual-host | Real region required (`auto` rejected) | The derived form only accepts `[A-Za-z0-9-]` in `region` | `cloud-source (aws)`, only while `ODM_INTEROP_AWS_*` are configured; no difference recorded yet |
| `s3` | Required | Path-style | Real region | Generic S3-compatible endpoint (Wasabi, Backblaze B2 S3 API, Ceph RGW, …) | No lane of its own; the preset is the same code path the `minio` job exercises |
| `minio` | Required | Path-style | `auto` allowed | | `minio-source`, nightly: read-through, HEAD passthrough, merged list pagination and a 5,000-object backfill; no difference recorded yet |
| `rustfs` | Required | Path-style | `auto` allowed | A RustFS source answers the migration request locally thanks to the anti-loop marker | `real_source_test.rs` in the `e2e-nightly` lane |
| `r2` | `https://<account-id>.r2.cloudflarestorage.com` | Virtual-host | `auto` allowed (signed as `us-east-1`) | | `cloud-source (r2)`, only while `ODM_INTEROP_R2_*` are configured; no difference recorded yet |
| `gcs` | `https://storage.googleapis.com` | Virtual-host | Real region required | Uses the GCS XML interoperability API with an HMAC key pair, not a service-account JSON key | `cloud-source (gcs)`, only while `ODM_INTEROP_GCS_HMAC_*` are configured; no difference recorded yet |
Azure Blob has no preset; a native provider is deferred (rustfs/backlog#2166).
The "Interop evidence" column names the job in `.github/workflows/on-demand-migration-interop.yml` (rustfs/backlog#2167) that last exercised the preset against a real implementation, and is where a provider difference belongs once the lane finds one. That lane is report-only and scheduled: it runs `crates/e2e_test/src/on_demand_migration/interop_test.rs` — the same case bodies as the merge-gate suite, with the source injected through `RUSTFS_ODM_INTEROP_*` — against a pinned MinIO container, and against each cloud provider whose repository secrets are configured. A provider without secrets is skipped with a note in the run summary rather than failing, so "no difference recorded yet" means exactly that and not "verified clean"; see [ci-gates.md](../testing/ci-gates.md) for the row.
The credentials only ever need read access to the source bucket:
- `s3:ListBucket` on the bucket — used by the admin probe, by the backfill listing, and by every merged listing under `policy.list_through`.