* feat(ecstore): add on-demand migration backfill job core
Add the background backfill job for on-demand migration
(rustfs/backlog#2159): a durable checkpoint under
buckets/<bucket>/on-demand-migration-backfill.json saved by If-Match
compare-and-set every 1000 keys or 10 s, a 60 s owner lease renewed by
every save, a recovery pass that takes over expired leases (or jobs this
node owned before a restart) and cancels jobs whose config changed, and a
main loop over the source ListObjectsV2 pages with the skip_existing
policy, dry runs, bounded outstanding pulls and wait-on-full enqueueing.
The pull queue gains per-job completion reports so the job can count
pulled/failed keys (hashes only), and pull permits become two-tier so an
online miss is never queued behind a backfill pull.
* feat(admin): expose on-demand migration backfill job
Wire the ODM-12 backfill job (rustfs/backlog#2159) to its operators:
POST /v3/on-demand-migration/{bucket}/backfill?op=start|cancel and
GET .../backfill return the checkpoint document, GET .../status gains a
backfill summary, and the recovery loop plus the process-wide runner are
installed at startup. Backfill control reuses
Set/GetBucketOnDemandMigrationAction and is recorded in the route policy,
the registration matrix and the admin route snapshot.
Add the rustfs-madmin wire types and client methods with golden fixtures
shared by the server tests, the backfill_* metric descriptors and their
collector, and three e2e scenarios: a full backfill across list pages,
cancellation, and resuming from the persisted continuation token after a
server restart.
* feat(ecstore): add on-demand migration bucket config model
Introduce OnDemandMigrationConfig (deny_unknown_fields, version 1) with typed validation, credential redaction, a secret-free Debug impl, and the OnceLock publish hook the runtime registers into. Exported through the api facade.
* feat(ecstore): persist on-demand migration config in bucket metadata
Store the config as a RustFS extension entry (on-demand-migration.json) with its update time in .metadata.bin, add the typed BucketMetadataSys accessor, and publish the config through the hook on every cache-install path alongside the durability sync.
* refactor(ecstore): extract shared remote S3 client builder
Move the aws_sdk_s3 client construction out of bucket_target_sys into
bucket/remote_s3_client.rs: endpoint assembly, credential provider,
path-style selection, custom CA / skip-TLS transports and the outbound
SSRF gate now build from a neutral RemoteS3EndpointSpec so replication
targets and the upcoming on-demand migration source client share one
policy. Replication builds its client through From<&BucketTarget>; the
gate keeps its relaxed semantics (private allowed, loopback only behind
RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) verbatim. The builder also
gains optional connect/read timeouts and a User-Agent suffix
interceptor, both unset for replication.
Refs rustfs/backlog#2149
* feat(ecstore): add on-demand migration SourceClient
Add bucket/on_demand_migration/source_client.rs on top of the shared
remote S3 builder: HEAD, ranged streaming GET, ListObjectsV2 with
source-prefix mapping, GetObjectTagging and an admin probe. Every request
carries the x-rustfs-/x-minio-source-proxy-request anti-loop markers and
a RustFS-OnDemandMigration/<version> User-Agent suffix; SSE-C source
objects are rejected as unsupported. SourceError classifies SDK failures
(not found, access denied, throttled, timeout, connect, server error)
with retryability and a stable metrics label. Debug output redacts
credentials.
Refs rustfs/backlog#2149
* docs(operations): point outbound policy at shared remote S3 client builder
* chore: integrate ODM-01 and ODM-02 as B1 base (fix facade merge)
* feat(admin): add on-demand migration bucket admin API
Add the management plane for On-Demand Migration (ODM-07,
rustfs/backlog#2154): PUT/GET/DELETE /v3/on-demand-migration/{bucket},
PUT ?dry-run=true, and a GET .../status skeleton.
- PUT authorizes SetBucketOnDemandMigration, checks the bucket, the
RUSTFS_ON_DEMAND_MIGRATION_ENABLED switch and the license, validates the
ODM-01 config against local endpoints and replication targets, probes the
source with SourceClient::probe(), then persists through the incarnation
gate and asks peers to reload. Responses carry the redacted config and a
probe summary; probe failures name only the error class.
- GET answers 404 NoSuchConfiguration when unset; DELETE is idempotent (204).
- New AdminAction variants admin:SetBucketOnDemandMigration and
admin:GetBucketOnDemandMigration, route policy matrix rows, registration
and MinIO alias coverage, and a doc row for the extra handler gates.
- rustfs-madmin gains on_demand_migration wire types and client methods;
golden fixtures under crates/madmin/fixtures/on_demand_migration/ are
asserted byte-for-byte by both the handler and the client tests.
Anonymous sources still map to a 400 naming source.credentials until the
runtime slice adds the credential-less path.
* refactor(admin): route on-demand migration handler errors through the s3 facade
feat(madmin): add a SigV4-signed admin client for heal and scanner APIs
The madmin crate held only wire types; automation and mc-style tooling
had no way to drive the heal/scanner admin surface without hand-rolled
HTTP. Add `AdminClient`, which signs with the same rustfs-signer path
the server authenticates (UNSIGNED-PAYLOAD marker, matching RustFS peer
admin calls) and wraps:
- heal_start / heal_status / heal_stop over POST /rustfs/admin/v3/heal/
(bucket/prefix path params percent-encoded per segment; stop models
the server's two cancel branches: token-scoped task status vs
path-scoped start-success receipt);
- background_heal_status, scanner_status (freshness typed), plus
ilm_expiry_status / replacement_recovery_status passthroughs;
- a public get_json escape hatch for endpoints not wrapped yet.
Wire types follow the madmin-go model (SDK-owned mirrors pinned by
round-trip tests): HealOpts with serde defaults so partial settings
objects decode, HealScanMode accepting both the numeric and name
encodings, and status structs that type the fields operators branch on
while flattening unknown nested payloads verbatim so server additions
cannot break the client. Errors map to a closed AdminClientError enum
(InvalidEndpoint / Transport / HttpStatus with body / Decode).
Tests cover wire round-trips, path building, both stop branches, error
mapping, and — via a dependency-free raw-TCP test server — that signed
requests carry a SigV4 Authorization header, the right method/path/
query, and the expected JSON body.
Closesrustfs/backlog#1869 (first increment; single-sourcing the wire
structs server-side and an embedded-server e2e roundtrip are noted as
follow-ups there).
Co-authored-by: heihutu <heihutu@gmail.com>
houseme