feat(admin): add on-demand migration bucket admin API (#7076)

* feat(ecstore): add on-demand migration bucket config model

Introduce OnDemandMigrationConfig (deny_unknown_fields, version 1) with typed validation, credential redaction, a secret-free Debug impl, and the OnceLock publish hook the runtime registers into. Exported through the api facade.

* feat(ecstore): persist on-demand migration config in bucket metadata

Store the config as a RustFS extension entry (on-demand-migration.json) with its update time in .metadata.bin, add the typed BucketMetadataSys accessor, and publish the config through the hook on every cache-install path alongside the durability sync.

* refactor(ecstore): extract shared remote S3 client builder

Move the aws_sdk_s3 client construction out of bucket_target_sys into
bucket/remote_s3_client.rs: endpoint assembly, credential provider,
path-style selection, custom CA / skip-TLS transports and the outbound
SSRF gate now build from a neutral RemoteS3EndpointSpec so replication
targets and the upcoming on-demand migration source client share one
policy. Replication builds its client through From<&BucketTarget>; the
gate keeps its relaxed semantics (private allowed, loopback only behind
RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) verbatim. The builder also
gains optional connect/read timeouts and a User-Agent suffix
interceptor, both unset for replication.

Refs rustfs/backlog#2149

* feat(ecstore): add on-demand migration SourceClient

Add bucket/on_demand_migration/source_client.rs on top of the shared
remote S3 builder: HEAD, ranged streaming GET, ListObjectsV2 with
source-prefix mapping, GetObjectTagging and an admin probe. Every request
carries the x-rustfs-/x-minio-source-proxy-request anti-loop markers and
a RustFS-OnDemandMigration/<version> User-Agent suffix; SSE-C source
objects are rejected as unsupported. SourceError classifies SDK failures
(not found, access denied, throttled, timeout, connect, server error)
with retryability and a stable metrics label. Debug output redacts
credentials.

Refs rustfs/backlog#2149

* docs(operations): point outbound policy at shared remote S3 client builder

* chore: integrate ODM-01 and ODM-02 as B1 base (fix facade merge)

* feat(admin): add on-demand migration bucket admin API

Add the management plane for On-Demand Migration (ODM-07,
rustfs/backlog#2154): PUT/GET/DELETE /v3/on-demand-migration/{bucket},
PUT ?dry-run=true, and a GET .../status skeleton.

- PUT authorizes SetBucketOnDemandMigration, checks the bucket, the
  RUSTFS_ON_DEMAND_MIGRATION_ENABLED switch and the license, validates the
  ODM-01 config against local endpoints and replication targets, probes the
  source with SourceClient::probe(), then persists through the incarnation
  gate and asks peers to reload. Responses carry the redacted config and a
  probe summary; probe failures name only the error class.
- GET answers 404 NoSuchConfiguration when unset; DELETE is idempotent (204).
- New AdminAction variants admin:SetBucketOnDemandMigration and
  admin:GetBucketOnDemandMigration, route policy matrix rows, registration
  and MinIO alias coverage, and a doc row for the extra handler gates.
- rustfs-madmin gains on_demand_migration wire types and client methods;
  golden fixtures under crates/madmin/fixtures/on_demand_migration/ are
  asserted byte-for-byte by both the handler and the client tests.

Anonymous sources still map to a 400 naming source.credentials until the
runtime slice adds the credential-less path.

* refactor(admin): route on-demand migration handler errors through the s3 facade
This commit is contained in:
Zhengchao An
2026-09-03 01:58:49 +08:00
committed by GitHub
parent a23d4b05a3
commit a5bde8b0af
17 changed files with 2106 additions and 142 deletions
@@ -25,3 +25,14 @@ Router-level credential checks (`S3Router::check_access`) are bypassed only for:
- console assets (`/favicon.ico`, `/rustfs/console...`), only while the console is enabled.
Every other admin route requires credentials at the router and a precise `AdminAction` or `S3Action` check in the handler (metrics routes, for example, authorize `GetMetricsAction`). The MinIO alias contract is specified in [minio-rustfs-router-compatibility.md](minio-rustfs-router-compatibility.md).
## Gated Bucket Feature Routes
Some bucket-scoped routes add gates after the `AdminAction` check. The gates are enforced in the handler, so they are invisible to the route matrix and listed here instead.
| Route | Actions | Extra gates after authorization |
|---|---|---|
| `PUT`/`DELETE /rustfs/admin/v3/on-demand-migration/{bucket}` (`?dry-run=true` validates and probes without saving) | `SetBucketOnDemandMigrationAction` (`admin:SetBucketOnDemandMigration`) | bucket must exist (`NoSuchBucket`); `PUT` also requires the `RUSTFS_ON_DEMAND_MIGRATION_ENABLED` module switch (`OnDemandMigrationDisabled`, 400) and the server license (`license_check()`, same mapping as object zip downloads); the source must answer HEAD + a one-key list (`OnDemandMigrationSourceUnreachable`, 400). Handler: `rustfs/src/admin/handlers/on_demand_migration.rs` |
| `GET /rustfs/admin/v3/on-demand-migration/{bucket}` and `GET .../{bucket}/status` | `GetBucketOnDemandMigrationAction` (`admin:GetBucketOnDemandMigration`) | bucket must exist; reads work while the module switch is off so operators can inspect a disabled deployment; `GET` answers `NoSuchConfiguration` (404) when nothing is configured |
Responses on these routes carry the redacted configuration (`secret_key` and `session_token` replaced by `REDACTED`); the wire shape is pinned by the fixtures under `crates/madmin/fixtures/on_demand_migration/`, shared by the server handler tests and the `rustfs-madmin` client tests.