mirror of
https://github.com/rustfs/rustfs.git
synced 2026-09-06 03:59:14 +00:00
Zhengchao An
74be040c62
* feat(ecstore): add on-demand migration backfill job core Add the background backfill job for on-demand migration (rustfs/backlog#2159): a durable checkpoint under buckets/<bucket>/on-demand-migration-backfill.json saved by If-Match compare-and-set every 1000 keys or 10 s, a 60 s owner lease renewed by every save, a recovery pass that takes over expired leases (or jobs this node owned before a restart) and cancels jobs whose config changed, and a main loop over the source ListObjectsV2 pages with the skip_existing policy, dry runs, bounded outstanding pulls and wait-on-full enqueueing. The pull queue gains per-job completion reports so the job can count pulled/failed keys (hashes only), and pull permits become two-tier so an online miss is never queued behind a backfill pull. * feat(admin): expose on-demand migration backfill job Wire the ODM-12 backfill job (rustfs/backlog#2159) to its operators: POST /v3/on-demand-migration/{bucket}/backfill?op=start|cancel and GET .../backfill return the checkpoint document, GET .../status gains a backfill summary, and the recovery loop plus the process-wide runner are installed at startup. Backfill control reuses Set/GetBucketOnDemandMigrationAction and is recorded in the route policy, the registration matrix and the admin route snapshot. Add the rustfs-madmin wire types and client methods with golden fixtures shared by the server tests, the backfill_* metric descriptors and their collector, and three e2e scenarios: a full backfill across list pages, cancellation, and resuming from the persisted continuation token after a server restart.
41 lines
5.3 KiB
Markdown
41 lines
5.3 KiB
Markdown
# Admin Route Action Snapshot
|
|
|
|
**Use this when:** you add, move, or re-authorize an admin route and need to know where the route → handler → `AdminAction` contract is enforced.
|
|
**Source of truth:** `rustfs/src/admin/route_policy.rs` (the `AdminRouteSpec` matrix, checked by `validate_admin_route_policy_specs`), `rustfs/src/admin/route_registration_test.rs` (registration coverage), `rustfs/src/admin/router.rs` (dispatch and credential checks), `rustfs/src/admin/handlers/*.rs` (handler-level authorization calls).
|
|
|
|
This page is a pointer, not a route table. The machine-checked matrix in `route_policy.rs` lists every admin route with its `AdminAction` and `RouteRiskLevel`; routes that are registered but answered by policy instead of a handler are declared there too through `DeferredRoutePolicyReason`. The `AdminRouteSpec` type lives in `crates/security-governance/src/admin_matrix.rs`.
|
|
|
|
## Prefix And Alias Contract
|
|
|
|
| Prefix | Behavior | Rule |
|
|
|---|---|---|
|
|
| `/rustfs/admin` | Canonical admin prefix used by `make_admin_route` (`rustfs/src/admin/mod.rs`) | The only registered admin prefix |
|
|
| `/minio/admin` | Compatibility alias accepted by `S3Router::is_match`; `canonicalize_admin_path` rewrites it to `/rustfs/admin` immediately before route lookup (`rustfs/src/admin/router.rs`) | Never register routes twice; preserve canonicalization |
|
|
| `/iceberg/v1` | Table catalog prefix registered by `register_table_catalog_route` (`rustfs/src/admin/handlers/table_catalog/routes.rs`) and accepted by `is_admin_path` | Stays outside `/rustfs/admin`; table actions are authorized per handler |
|
|
| `/health`, `/health/ready` | Public health endpoints, registered only when `ENV_HEALTH_ENDPOINT_ENABLE` allows | Preserve the unauthenticated bypass |
|
|
| `/profile/cpu`, `/profile/memory` | Registered by the health handler but guarded by profile authorization | Never couple to health-endpoint enablement |
|
|
|
|
## Public Exceptions
|
|
|
|
Router-level credential checks (`S3Router::check_access`) are bypassed only for:
|
|
|
|
- health routes, when they are registered;
|
|
- OIDC bootstrap paths matched by `is_oidc_path` (`providers`, `authorize/{provider_id}`, `callback/{provider_id}`, `logout`); the bypass is path-based, so it applies to any method on those paths;
|
|
- unsigned STS web-identity form posts to `/` with `application/x-www-form-urlencoded`, which the STS handler validates itself;
|
|
- console assets (`/favicon.ico`, `/rustfs/console...`), only while the console is enabled.
|
|
|
|
Every other admin route requires credentials at the router and a precise `AdminAction` or `S3Action` check in the handler (metrics routes, for example, authorize `GetMetricsAction`). The MinIO alias contract is specified in [minio-rustfs-router-compatibility.md](minio-rustfs-router-compatibility.md).
|
|
|
|
## Gated Bucket Feature Routes
|
|
|
|
Some bucket-scoped routes add gates after the `AdminAction` check. The gates are enforced in the handler, so they are invisible to the route matrix and listed here instead.
|
|
|
|
| Route | Actions | Extra gates after authorization |
|
|
|---|---|---|
|
|
| `PUT`/`DELETE /rustfs/admin/v3/on-demand-migration/{bucket}` (`?dry-run=true` validates and probes without saving) | `SetBucketOnDemandMigrationAction` (`admin:SetBucketOnDemandMigration`) | bucket must exist (`NoSuchBucket`); `PUT` also requires the `RUSTFS_ON_DEMAND_MIGRATION_ENABLED` module switch (`OnDemandMigrationDisabled`, 400) and the server license (`license_check()`, same mapping as object zip downloads); the source must answer HEAD + a one-key list (`OnDemandMigrationSourceUnreachable`, 400). Handler: `rustfs/src/admin/handlers/on_demand_migration.rs` |
|
|
| `GET /rustfs/admin/v3/on-demand-migration/{bucket}` and `GET .../{bucket}/status` | `GetBucketOnDemandMigrationAction` (`admin:GetBucketOnDemandMigration`) | bucket must exist; reads work while the module switch is off so operators can inspect a disabled deployment; `GET` answers `NoSuchConfiguration` (404) when nothing is configured; `status` carries a `backfill` summary of the latest backfill job when one was recorded |
|
|
| `POST /rustfs/admin/v3/on-demand-migration/{bucket}/backfill?op=start\|cancel` (body `{prefix?, skip_existing?, dry_run?}` for `start`) | `SetBucketOnDemandMigrationAction` (`admin:SetBucketOnDemandMigration`) | bucket must exist; `op` must be `start` or `cancel` (`InvalidArgument`, 400); `start` also requires the module switch (`OnDemandMigrationDisabled`, 400), the server license, a saved config (`NoSuchConfiguration`, 404) and an enabled bucket state on this node (`OnDemandMigrationDisabled`, 400); a job holding its lease answers `OnDemandMigrationBackfillRunning` (409); `cancel` answers `NoSuchBackfillJob` (404) when the bucket never had a job. Handler: `rustfs/src/admin/handlers/on_demand_migration.rs` |
|
|
| `GET /rustfs/admin/v3/on-demand-migration/{bucket}/backfill` | `GetBucketOnDemandMigrationAction` (`admin:GetBucketOnDemandMigration`) | bucket must exist; returns the backfill checkpoint document (`failed_keys` are key hashes, no credentials); `NoSuchBackfillJob` (404) when the bucket never had a job |
|
|
|
|
Responses on the config routes carry the redacted configuration (`secret_key` and `session_token` replaced by `REDACTED`); the wire shapes of every route in this group, including the backfill checkpoint and the `status` summary, are pinned by the fixtures under `crates/madmin/fixtures/on_demand_migration/`, shared by the server handler tests and the `rustfs-madmin` client tests.
|