feat(admin): add on-demand migration bucket admin API (#7076)

* feat(ecstore): add on-demand migration bucket config model

Introduce OnDemandMigrationConfig (deny_unknown_fields, version 1) with typed validation, credential redaction, a secret-free Debug impl, and the OnceLock publish hook the runtime registers into. Exported through the api facade.

* feat(ecstore): persist on-demand migration config in bucket metadata

Store the config as a RustFS extension entry (on-demand-migration.json) with its update time in .metadata.bin, add the typed BucketMetadataSys accessor, and publish the config through the hook on every cache-install path alongside the durability sync.

* refactor(ecstore): extract shared remote S3 client builder

Move the aws_sdk_s3 client construction out of bucket_target_sys into
bucket/remote_s3_client.rs: endpoint assembly, credential provider,
path-style selection, custom CA / skip-TLS transports and the outbound
SSRF gate now build from a neutral RemoteS3EndpointSpec so replication
targets and the upcoming on-demand migration source client share one
policy. Replication builds its client through From<&BucketTarget>; the
gate keeps its relaxed semantics (private allowed, loopback only behind
RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) verbatim. The builder also
gains optional connect/read timeouts and a User-Agent suffix
interceptor, both unset for replication.

Refs rustfs/backlog#2149

* feat(ecstore): add on-demand migration SourceClient

Add bucket/on_demand_migration/source_client.rs on top of the shared
remote S3 builder: HEAD, ranged streaming GET, ListObjectsV2 with
source-prefix mapping, GetObjectTagging and an admin probe. Every request
carries the x-rustfs-/x-minio-source-proxy-request anti-loop markers and
a RustFS-OnDemandMigration/<version> User-Agent suffix; SSE-C source
objects are rejected as unsupported. SourceError classifies SDK failures
(not found, access denied, throttled, timeout, connect, server error)
with retryability and a stable metrics label. Debug output redacts
credentials.

Refs rustfs/backlog#2149

* docs(operations): point outbound policy at shared remote S3 client builder

* chore: integrate ODM-01 and ODM-02 as B1 base (fix facade merge)

* feat(admin): add on-demand migration bucket admin API

Add the management plane for On-Demand Migration (ODM-07,
rustfs/backlog#2154): PUT/GET/DELETE /v3/on-demand-migration/{bucket},
PUT ?dry-run=true, and a GET .../status skeleton.

- PUT authorizes SetBucketOnDemandMigration, checks the bucket, the
  RUSTFS_ON_DEMAND_MIGRATION_ENABLED switch and the license, validates the
  ODM-01 config against local endpoints and replication targets, probes the
  source with SourceClient::probe(), then persists through the incarnation
  gate and asks peers to reload. Responses carry the redacted config and a
  probe summary; probe failures name only the error class.
- GET answers 404 NoSuchConfiguration when unset; DELETE is idempotent (204).
- New AdminAction variants admin:SetBucketOnDemandMigration and
  admin:GetBucketOnDemandMigration, route policy matrix rows, registration
  and MinIO alias coverage, and a doc row for the extra handler gates.
- rustfs-madmin gains on_demand_migration wire types and client methods;
  golden fixtures under crates/madmin/fixtures/on_demand_migration/ are
  asserted byte-for-byte by both the handler and the client tests.

Anonymous sources still map to a 400 naming source.credentials until the
runtime slice adds the credential-less path.

* refactor(admin): route on-demand migration handler errors through the s3 facade
This commit is contained in:
Zhengchao An
2026-09-03 01:58:49 +08:00
committed by GitHub
parent a23d4b05a3
commit a5bde8b0af
17 changed files with 2106 additions and 142 deletions
+28
View File
@@ -465,6 +465,12 @@ pub enum AdminAction {
SetBucketTargetAction,
#[strum(serialize = "admin:GetBucketTarget")]
GetBucketTargetAction,
/// Configure, validate or clear a bucket's on-demand migration source.
#[strum(serialize = "admin:SetBucketOnDemandMigration")]
SetBucketOnDemandMigrationAction,
/// Read a bucket's on-demand migration configuration and status.
#[strum(serialize = "admin:GetBucketOnDemandMigration")]
GetBucketOnDemandMigrationAction,
#[strum(serialize = "admin:GetMetrics")]
GetMetricsAction,
#[strum(serialize = "admin:ReplicationDiff")]
@@ -623,6 +629,8 @@ impl AdminAction {
| AdminAction::SetBucketQuotaAdminAction
| AdminAction::SetBucketTargetAction
| AdminAction::GetBucketTargetAction
| AdminAction::SetBucketOnDemandMigrationAction
| AdminAction::GetBucketOnDemandMigrationAction
| AdminAction::GetMetricsAction
| AdminAction::ReplicationDiff
| AdminAction::GetReplicationMetricsAction
@@ -835,6 +843,26 @@ mod tests {
assert!(AdminAction::GetMetricsAction.is_valid());
}
#[test]
fn test_bucket_on_demand_migration_admin_actions_are_valid() {
let set_action = AdminAction::try_from("admin:SetBucketOnDemandMigration").expect("parse set action");
let get_action = AdminAction::try_from("admin:GetBucketOnDemandMigration").expect("parse get action");
assert_eq!(set_action, AdminAction::SetBucketOnDemandMigrationAction);
assert_eq!(get_action, AdminAction::GetBucketOnDemandMigrationAction);
assert!(set_action.is_valid());
assert!(get_action.is_valid());
assert_eq!(<&str>::from(set_action), "admin:SetBucketOnDemandMigration");
assert_eq!(<&str>::from(get_action), "admin:GetBucketOnDemandMigration");
// `admin:*` must cover the new actions without a per-action listing,
// while a read-only grant must not confer the write action.
let all_admin = Action::AdminAction(AdminAction::AllAdminActions);
assert!(all_admin.is_match(&Action::AdminAction(set_action)));
assert!(all_admin.is_match(&Action::AdminAction(get_action)));
assert!(!Action::AdminAction(get_action).is_match(&Action::AdminAction(set_action)));
}
#[test]
fn test_table_catalog_admin_action_is_valid() {
let get_action = AdminAction::try_from("admin:GetTableCatalog").expect("Should parse GetTableCatalog action");