gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-09-05 19:55:37 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(odm): add the on-demand migration operations guide (#7086)

Browse Source 
* docs(odm): add the on-demand migration operations guide

Positioning against replication, site replication, tiering and the
replication read-proxy; the admin API walkthrough; the full config field
table with defaults and bounds; the semantics table with the test that
pins each row; provider presets and least-privilege source permissions;
integrity, ETag and metadata mapping; protections, error codes,
observability with PromQL, troubleshooting, known limitations and
security notes.

Adds docs/architecture/background-services-inventory.md with the
write-back pipeline, backfill job and backfill recovery loop rows, and
registers it in the architecture index.

Refs rustfs/backlog#2160, rustfs/backlog#2147.

* docs(odm): note migration sources on the replication egress guard

Refs rustfs/backlog#2160.
This commit is contained in:
Zhengchao An
2026-09-03 08:48:31 +08:00
committed by GitHub
parent 759a395fec
commit df4fdef1d8
4 changed files with 346 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture/README.md
+1
View File
@@ -41,6 +41,7 @@ Required headings and strings in these files are asserted by `scripts/check_arch
| [runtime-capability-contracts.md](runtime-capability-contracts.md) | changing the read-only observability or topology snapshot contracts in `rustfs-storage-api`, their providers, or the `storage_classes` payload of `GET /rustfs/admin/v4/runtime/capabilities` |
| [workload-admission-contracts.md](workload-admission-contracts.md) | adding a workload class or snapshot provider, or consuming admission state from a background job |
| [background-controller-contract.md](background-controller-contract.md) | adding a status snapshot or reconcile surface for a background service, or being tempted to fold several services into a generic controller |
| [background-services-inventory.md](background-services-inventory.md) | you need one audited background service's desired source, current-status inputs, status surface, and declared side effects |
| [config-model-boundary-adr.md](config-model-boundary-adr.md) | touching the server-config model (`Config`, `KV`, `KVS`) or its persistence, or asking which crate owns which part of server configuration |
| [admin-route-action-snapshot.md](admin-route-action-snapshot.md) | adding, moving, or re-authorizing an admin route and needing to know where the route → handler → `AdminAction` contract is enforced |
| [kms-bulk-rekey-contract.md](kms-bulk-rekey-contract.md) | changing the bulk envelope re-wrap sweep, its admin endpoints, the re-wrap primitive, or which objects a rekey may touch |
docs/architecture/background-services-inventory.md
+20
View File
@@ -0,0 +1,20 @@
# Background Services Inventory
**Use this when:** you add, move, or audit a background worker and need its declared desired source, current-status inputs, status surface, and side effects in one row — the per-service data the vocabulary in [background-controller-contract.md](background-controller-contract.md) is defined over.
**Source of truth:** the service implementations cited per row. The contract page owns the vocabulary (desired / current / status / reconcile / side effects) and the state model; this page owns the per-service rows.
Scope: services get a row here when they are audited. This inventory currently covers the on-demand migration workers (rustfs/backlog#2147); the scanner, heal, lifecycle, replication, notification, capacity, and config-reload services are described in the "Coupling Notes" of [background-controller-contract.md](background-controller-contract.md) and gain rows as they are audited. A missing row means "not yet inventoried", never "no side effects".
Rules a row must respect, inherited from the contract: status collection is side-effect-free, a missing surface is reported as `unknown` rather than guessed, and side effects are declared before any controller touches the service.
## On-Demand Migration
Operator-facing behaviour, configuration, and troubleshooting for these services live in [../operations/on-demand-migration.md](../operations/on-demand-migration.md).
| Service | Desired source | Current-status inputs | Status surface | Side effects |
|---|---|---|---|---|
| Write-back pull pipeline (`crates/ecstore/src/bucket/on_demand_migration/pull.rs`; the local write is delegated to the app layer in `rustfs/src/app/object/on_demand_migration_put.rs`) | The bucket's `on-demand-migration.json` (`enabled`, `policy.max_concurrent_pulls`, `pull_queue_capacity`, `multipart_part_size_bytes`, `bandwidth_limit_bytes_per_sec`) together with the process switch `RUSTFS_ON_DEMAND_MIGRATION_ENABLED` | Per-bucket runtime state in `crates/ecstore/src/bucket/on_demand_migration/sys.rs`: whether a state is installed, whether its source client built, its cancellation token, queue depth, in-flight pull permits | `GET /rustfs/admin/v3/on-demand-migration/{bucket}/status` (`inflight_pulls`, `queue_depth`, `counters.pulled_*`, `counters.pull_failures_total`) and the `rustfs_on_demand_migration_*` series | Source GET/HEAD/GetObjectTagging traffic; local object writes through the internal put path, hence quota consumption, bucket default SSE, versioning, Object Lock defaults, `ObjectCreated` notifications, and outbound replication scheduling |
| Backfill job (module under `crates/ecstore/src/bucket/on_demand_migration/`, rustfs/backlog#2159 — not yet in the tree) | An admin `start` request plus the bucket config; invalidated when the config's `updated_at` changes or the config is deleted | The persisted checkpoint under the bucket's metadata prefix, its `state` field, and the owner lease | The backfill section of the bucket status endpoint and the `rustfs_on_demand_migration_backfill_*` series | Source `ListObjectsV2` paging; queue admission into the write-back pipeline (and therefore all of its side effects); checkpoint writes |
| Backfill recovery loop (registered from `rustfs/src/startup_background.rs`, rustfs/backlog#2159 — not yet in the tree) | The set of persisted checkpoints in `state = running`; runs on every node | Checkpoint owner lease expiry | Takeover is reported through the same backfill status; a takeover emits a warn-level lease event | Claims the lease and resumes the backfill job, inheriting its side effects. Scanning checkpoints is read-only |
The pull pipeline has no separate loop of its own: a bucket's queue dispatcher starts lazily on the first background pull and is cancelled when the bucket's state is rebuilt or removed, and each inline pull commits in a task that outlives its request so a client disconnect cannot truncate the stored object. Neither the switch nor the config is re-read by the workers: the bucket-metadata publish hook rebuilds the state, which is the only desired-state path.
docs/operations/on-demand-migration.md
+322
View File
@@ -0,0 +1,322 @@
# On-Demand Migration
**Use this when:** you are moving an existing S3-compatible bucket into RustFS without a stop-the-world copy, or you are debugging a bucket that serves reads from an external source (424 `SourceUnavailable`, an open circuit breaker, missing pulled objects, a source 403).
**Source of truth:** `crates/ecstore/src/bucket/on_demand_migration/` (`config.rs` for the wire model and its bounds, `sys.rs` for the per-node runtime, `source_client.rs` for the outbound client, `pull.rs` for the write-back pipeline, `breaker.rs` and `negative_cache.rs` for the protections), `rustfs/src/app/object/get.rs` and `head.rs` for the read paths, `rustfs/src/app/object/on_demand_migration_put.rs` for the local write, `rustfs/src/admin/handlers/on_demand_migration.rs` for the admin API, and `crates/obs/src/metrics/schema/on_demand_migration.rs` for the metric contract.
On-Demand Migration (ODM) attaches an external S3-compatible **source bucket** to a local RustFS bucket. When a client GETs a key that does not exist locally, RustFS fetches it from the source, streams it to the client, and stores it locally in the same pass; every later read is served locally. It is a pull-style, lazy migration path — the RustFS equivalent of Cloudflare R2 Sippy, Tigris shadow buckets, and Alibaba Cloud OSS / Tencent COS mirror-back-to-origin.
The module is off by default. Set `RUSTFS_ON_DEMAND_MIGRATION_ENABLED=true` on every node before configuring a bucket (`rustfs/src/module_switches.rs`); with the switch off, the runtime never intervenes on a read and the admin `PUT` route refuses with `OnDemandMigrationDisabled`. Reads of the configuration and of the status endpoint keep working while the switch is off, so a disabled deployment can still be inspected.
## Positioning
| Capability | Direction | What it moves | Where the authoritative copy is | When to use it instead |
|---|---|---|---|---|
| **On-demand migration** (this page) | Inbound pull from a foreign S3 bucket | The objects clients actually read, plus an optional background backfill for the rest | The source until an object is pulled; local afterwards | You are migrating away from another provider and want zero-downtime cutover |
| Bucket replication (`docs/operations/replication-check.md`) | Outbound push to a target | Every new write, continuously | Local; the target is a copy | You want an ongoing copy of local writes elsewhere |
| Site replication | Bidirectional, cluster-wide | Buckets, IAM, policies, objects | Every peer is authoritative | You want two RustFS deployments to converge |
| Tiering / ILM (`docs/operations/tier-ilm-debugging.md`) | Outbound push to a warm backend, transparent read-back | Cold local objects, by lifecycle rule | Local metadata, remote data | You want to reduce local capacity for cold data you already own |
| Replication read-proxy | Inbound proxy to a replication target | Nothing — it only proxies the response | The target | An active-active pair has a replication lag window; this runs *before* ODM on a miss and never stores anything |
The order on a GET miss is fixed: local read → replication read-proxy → on-demand migration. ODM is consulted last because it is the expensive, authoritative-external answer.
## Enabling a source
Admin requests are SigV4-signed like any S3 request; the examples use [`awscurl`](https://github.com/okigan/awscurl) with the deployment's admin credentials. The routes and their authorization are pinned in `docs/architecture/admin-route-action-snapshot.md`.
| Route | Action | Notes |
|---|---|---|
| `PUT /rustfs/admin/v3/on-demand-migration/{bucket}` | `admin:SetBucketOnDemandMigration` | Requires the module switch and the server license; validates, probes the source, then saves |
| `PUT /rustfs/admin/v3/on-demand-migration/{bucket}?dry-run=true` | `admin:SetBucketOnDemandMigration` | Same validation and probe, saves nothing (`updated_at` is `null`) |
| `GET /rustfs/admin/v3/on-demand-migration/{bucket}` | `admin:GetBucketOnDemandMigration` | Returns the redacted config; `NoSuchConfiguration` (404) when unset |
| `GET /rustfs/admin/v3/on-demand-migration/{bucket}/status` | `admin:GetBucketOnDemandMigration` | Switch state plus this node's runtime snapshot |
| `DELETE /rustfs/admin/v3/on-demand-migration/{bucket}` | `admin:SetBucketOnDemandMigration` | Idempotent, answers 204; already-pulled objects stay in place |
Validate first, with `dry-run`, so a bad endpoint or a wrong key never reaches bucket metadata:
```bash
cat > /tmp/odm.json <<'JSON'
{
"source": {
"provider": "minio",
"endpoint": "https://source.example.com:9000",
"region": "us-east-1",
"bucket": "legacy-photos",
"credentials": { "access_key": "AKIA…", "secret_key": "REDACTED" }
},
"filter": { "source_prefix": "photos/" },
"policy": { "inline_max_bytes": 16777216, "max_concurrent_pulls": 8 }
}
JSON
awscurl --service s3 --region us-east-1 \
--access_key "$RUSTFS_ACCESS_KEY" --secret_key "$RUSTFS_SECRET_KEY" \
--request PUT --header 'Content-Type: application/json' \
--data "$(cat /tmp/odm.json)" \
"http://<host>:9000/rustfs/admin/v3/on-demand-migration/photos?dry-run=true"
```
The response echoes the **redacted** config (`secret_key` and `session_token` become `REDACTED`) and a probe summary: `reachable` (the source answered `HeadBucket`), `listable` (a one-key `ListObjectsV2` succeeded) and `sample_key` (the first key that listing returned, absent for an empty source). Repeat the call without `?dry-run=true` to save it; the handler then fans a metadata reload out to the peers so the whole cluster picks the source up without waiting for the refresh loop.
```bash
# Read the saved configuration back
awscurl --service s3 --region us-east-1 \
--access_key "$RUSTFS_ACCESS_KEY" --secret_key "$RUSTFS_SECRET_KEY" \
"http://<host>:9000/rustfs/admin/v3/on-demand-migration/photos"
# Per-node runtime status (breaker, counters, queue, last source error)
awscurl --service s3 --region us-east-1 \
--access_key "$RUSTFS_ACCESS_KEY" --secret_key "$RUSTFS_SECRET_KEY" \
"http://<host>:9000/rustfs/admin/v3/on-demand-migration/photos/status"
# Stop consulting the source; pulled objects stay
awscurl --service s3 --region us-east-1 \
--access_key "$RUSTFS_ACCESS_KEY" --secret_key "$RUSTFS_SECRET_KEY" \
--request DELETE \
"http://<host>:9000/rustfs/admin/v3/on-demand-migration/photos"
```
Setting `"enabled": false` in the config has the same read-path effect as deleting it (the runtime tears the bucket state down) while keeping the source, credentials and policy on record for a later re-enable.
The status endpoint reports **the node that answered the request**. Counters, queue depth and breaker state are per-node runtime state, so in a distributed deployment query every node; the saved configuration and `updated_at` are cluster-wide.
### Backfill (ships with ODM-12)
Read-through only migrates what clients touch. The background backfill job walks the source listing and pulls the remainder, with a persisted checkpoint (`.rustfs.sys/buckets/<bucket>/on-demand-migration-backfill.json`), a single-owner lease, resume after restart, and `POST .../{bucket}/backfill?op=start|cancel` plus `GET .../{bucket}/backfill` admin routes. That slice (rustfs/backlog#2159) is not part of the build this page was written against: the shape above is the agreed design, and the exact request/response bodies must be re-checked against `docs/architecture/admin-route-action-snapshot.md` once it lands.
## Configuration reference
The persisted blob is `on-demand-migration.json` in the bucket's metadata. Unknown fields are rejected rather than dropped, so a config written by a newer build fails loudly on an older one. Every default and bound below comes from `crates/ecstore/src/bucket/on_demand_migration/config.rs`.
| Field | Type | Default | Bounds / rules |
|---|---|---|---|
| `version` | integer | `1` | Must be `1` |
| `enabled` | bool | `true` | `false` keeps the config but stops all source traffic |
| `source.provider` | `s3` \| `aws` \| `minio` \| `rustfs` \| `r2` \| `gcs` | — (required) | Drives endpoint and addressing defaults |
| `source.endpoint` | string \| null | — | `http(s)://host[:port]`, no path, query, fragment or userinfo. Required for every provider except `aws`, where it is derived from `region` |
| `source.region` | string | — (required) | Non-empty. `auto` is accepted only for `r2`, `minio`, `rustfs` and is signed as `us-east-1` |
| `source.bucket` | string | — (required) | Non-empty, no `/` and no whitespace |
| `source.path_style` | `auto` \| `path` \| `virtual` | `auto` | `auto` resolves to path-style for IP-literal or `localhost` endpoints and for `s3`/`minio`/`rustfs`; virtual-host for `aws`/`gcs`/`r2` |
| `source.credentials` | object \| null | `null` | `null` means anonymous, which the client builder does not support yet: the admin `PUT` refuses it with `InvalidArgument`, and a config that reached the metadata another way resolves as unavailable. `access_key` and `secret_key` must be non-empty; `session_token` is optional but must be non-empty when present |
| `source.tls.skip_verify` | bool | `false` | Disables certificate verification for the source connection |
| `source.tls.ca_cert_pem` | string \| null | `null` | Must contain `-----BEGIN CERTIFICATE-----` |
| `filter.prefix` | string \| null | `null` | Null or non-empty. Only local keys with this prefix consult the source |
| `filter.source_prefix` | string \| null | `null` | Null or non-empty. Prepended to the local key to form the source key |
| `policy.head` | `proxy` \| `local_only` | `proxy` | `local_only` answers a HEAD miss with 404 and no source traffic |
| `policy.range_get` | `serve_and_backfill` \| `serve_only` | `serve_and_backfill` | Whether a Range GET also queues a background pull of the whole object |
| `policy.source_error` | `propagate` \| `not_found` | `propagate` | `propagate` answers 424 `SourceUnavailable`; `not_found` degrades to 404 |
| `policy.respect_local_delete_marker` | bool | `true` | A local delete marker is the final answer; only a versioned bucket can produce one |
| `policy.preserve_etag` | bool | `true` | Keeps the source ETag on the stored object unless the bucket encrypts by default |
| `policy.copy_tags` | bool | `false` | Copies source object tags; needs `s3:GetObjectTagging` and costs one extra source call per inline pull |
| `policy.emit_events` | bool | `true` | Whether a write-back emits `ObjectCreated` notifications |
| `policy.negative_cache_ttl_secs` | integer | `30` | `0..=3600`; `0` disables the negative cache |
| `policy.inline_max_bytes` | integer | `16777216` (16 MiB) | `0..=268435456` (256 MiB). At or below this size a GET miss is teed inline; above it the response streams through and a background pull stores the object |
| `policy.multipart_part_size_bytes` | integer | `67108864` (64 MiB) | `5242880..=5368709120` (5 MiB…5 GiB). Objects needing more than 10 000 parts require a larger part size |
| `policy.max_concurrent_pulls` | integer | `8` | `1..=256`; shared by the inline and background paths |
| `policy.pull_queue_capacity` | integer | `1024` | `1..=65536`; a full queue drops the *background* job, never the client response |
| `policy.source_timeout.connect_ms` | integer | `5000` | `100..=600000` |
| `policy.source_timeout.first_byte_ms` | integer | `15000` | `100..=600000`; also the window a follower waits for the singleflight leader before streaming through |
| `policy.source_timeout.idle_ms` | integer | `30000` | `100..=600000`; enforced by the background pump only (see Known limitations) |
| `policy.bandwidth_limit_bytes_per_sec` | integer \| null | `null` | When set, at least `65536` |
Values that are **not** configurable: the breaker opens after 5 consecutive counted failures inside a 30 s window, stays open for 30 s and then admits one probe (`breaker.rs`); the negative cache holds at most 100 000 keys per bucket with LRU eviction (`negative_cache.rs`); a background pull retries a retryable source failure at most 3 times with 1 s / 4 s / 16 s base delays plus up to 25 % jitter (`pull.rs`).
Validation also rejects two shapes outright: a source whose endpoint and bucket name **this** bucket on this deployment (`SelfReference`), and a source that matches one of the bucket's own replication targets (`ReplicationLoop`) — that pairing would amplify a write-back into a loop.
## Provider presets and source permissions
| Provider | Endpoint | Addressing | `region` | Notes |
|---|---|---|---|---|
| `aws` | Optional; derived as `https://s3.<region>.amazonaws.com` | Virtual-host | Real region required (`auto` rejected) | The derived form only accepts `[A-Za-z0-9-]` in `region` |
| `s3` | Required | Path-style | Real region | Generic S3-compatible endpoint (Wasabi, Backblaze B2 S3 API, Ceph RGW, …) |
| `minio` | Required | Path-style | `auto` allowed | |
| `rustfs` | Required | Path-style | `auto` allowed | A RustFS source answers the migration request locally thanks to the anti-loop marker |
| `r2` | `https://<account-id>.r2.cloudflarestorage.com` | Virtual-host | `auto` allowed (signed as `us-east-1`) | |
| `gcs` | `https://storage.googleapis.com` | Virtual-host | Real region required | Uses the GCS XML interoperability API with an HMAC key pair, not a service-account JSON key |
Azure Blob has no preset; a native provider is deferred (rustfs/backlog#2166).
The credentials only ever need read access to the source bucket:
- `s3:ListBucket` on the bucket — used by the admin probe and by the backfill listing.
- `s3:GetObject` on `<bucket>/*` — used by every HEAD and GET against the source.
- `s3:GetObjectTagging` on `<bucket>/*` — only when `policy.copy_tags` is `true`.
No write, delete, ACL or versioning permission is required or used. Scope the policy to the prefix in `filter.source_prefix` when the source bucket holds unrelated data.
## Request semantics
Behaviour a client can observe. The "Test" column names the case that pins it: `*_test.rs` files live under `crates/e2e_test/src/on_demand_migration/`, and the unit tests live next to the code in `rustfs/src/app/object/get.rs`, `head.rs` and `shared.rs`.
| Situation | Behaviour | Test |
|---|---|---|
| GET miss, object at or below `inline_max_bytes` | One source GET, teed: the client streams while the same bytes are written locally. Later reads are local and carry no source marker | `get_basic_test.rs::get_miss_pulls_inline_and_serves_locally_afterwards`, `get.rs::odm_get_inline_streams_to_client_and_commits_the_same_bytes` |
| GET miss, object above `inline_max_bytes` | Streamed straight through (nothing stored inline) and a background pull of the whole object is queued | `get_basic_test.rs::get_large_object_streams_through_and_backfills_in_background`, `get.rs::odm_get_large_object_streams_through_and_queues_a_background_pull` |
| Range GET miss | The requested range is passed through as 206; with `range_get = serve_and_backfill` a whole-object background pull is queued, with `serve_only` nothing is queued | `get_basic_test.rs::get_range_streams_206_and_backfills_the_whole_object`, `get.rs::odm_get_range_streams_206_and_queues_per_policy` |
| Client disconnects mid-stream on an inline pull | The write-back keeps draining the source and still stores the whole object | `get.rs::odm_get_inline_client_disconnect_still_stores_the_whole_object` |
| Concurrent GET misses of one key | Singleflight: one leader tees, followers wait up to `first_byte_ms` and then re-read locally; on leader failure or timeout they stream through without queueing | `concurrency_test.rs::test_odm_concurrent_misses_on_one_key_coalesce`, `get.rs::odm_get_follower_rereads_local_after_the_leader_commits` |
| HEAD miss | Proxied to the source, nothing is written locally; `local_only` answers 404 without source traffic | `head.rs::odm_head_source_hit_returns_output_and_writes_nothing_back`, `head.rs::odm_head_local_only_policy_is_404_without_source_traffic`, `real_source_test.rs::test_odm_rustfs_source_serves_pull_head_range_and_prefixes_real_single_node` |
| LIST | Local only. A key that exists on the source but was never pulled is not listed, even while a GET of it succeeds | `interaction_test.rs::test_odm_write_back_respects_the_bucket_quota` |
| PUT / DELETE | Never touch the source. A local PUT shadows the source key for good | `interaction_test.rs::test_odm_delete_marker_shadows_the_source_but_a_plain_delete_does_not` |
| Versioned bucket, local delete marker | With `respect_local_delete_marker` (default) the marker is the final answer and the source is not consulted | `interaction_test.rs::test_odm_delete_marker_shadows_the_source_but_a_plain_delete_does_not`, `head.rs::odm_head_verdict_respects_local_delete_marker_by_policy` |
| Unversioned bucket, object deleted locally | The key becomes an ordinary miss and is pulled from the source again | `interaction_test.rs::test_odm_delete_marker_shadows_the_source_but_a_plain_delete_does_not` |
| GET with `versionId`, or a `partNumber` read | Never consults the source; the local 404 stands | `get_basic_test.rs::get_with_version_id_does_not_consult_the_source`, `get.rs::odm_get_gate_rejects_part_reads_and_version_reads` |
| Conditional headers (`If-Match`, `If-None-Match`, `If-Modified-Since`, `If-Unmodified-Since`) | Evaluated locally against the source HEAD with the usual S3 semantics; never forwarded to the source | `get.rs::odm_get_conditional_headers_are_answered_from_the_source_head`, `shared.rs::odm_source_preconditions_follow_local_semantics` |
| Source object is SSE-C encrypted | 424 `SourceUnavailable` with class `unsupported`, whatever `source_error` says | `fault_test.rs::test_odm_ssec_source_object_is_unsupported`, `head.rs::odm_head_unsupported_source_object_is_424_regardless_of_policy` |
| Source answers 404 | 404 to the client and the key is negative-cached for `negative_cache_ttl_secs` | `get_basic_test.rs::get_source_not_found_is_404_and_negative_cached`, `fault_test.rs::test_odm_source_not_found_is_negative_cached_for_the_ttl` |
| Source answers 403 | 424 (or 404 under `not_found`); the breaker is **not** opened — a credential problem is not a transient one | `fault_test.rs::test_odm_source_access_denied_propagates_without_opening_the_breaker` |
| Repeated source 5xx / timeouts | The breaker opens, GETs answer per `source_error` and HEADs answer 404; a probe closes it again after the open window | `fault_test.rs::test_odm_repeated_source_errors_open_the_breaker_and_recover`, `head.rs::odm_head_source_errors_follow_policy_and_open_the_breaker` |
| Bucket default SSE | The pulled object is stored encrypted and reads back in plaintext; the ETag override is dropped, and the source ETag is kept in metadata | `interaction_test.rs::test_odm_pulled_object_uses_bucket_default_encryption`, `on_demand_migration_put.rs::write_back_under_bucket_default_sse_stores_ciphertext_and_records_source_etag` |
| Object Lock bucket | The pulled object inherits the bucket's default retention | `interaction_test.rs::test_odm_pulled_object_inherits_object_lock_retention` |
| Bucket quota exceeded | The client is still served from the source; the write-back is rejected, nothing is stored, and the failure counts under `local_write` | `interaction_test.rs::test_odm_write_back_respects_the_bucket_quota`, `on_demand_migration_put.rs::write_back_reports_a_full_bucket_quota` |
| Notifications | A write-back emits `ObjectCreated` with principal `rustfs-on-demand-migration`, unless `emit_events` is `false` | `interaction_test.rs::test_odm_pull_emits_object_created_events_unless_disabled` |
| Replication configured on the local bucket | A pulled object replicates like any other write; configuring one of the bucket's replication targets as the source is rejected | `interaction_test.rs::test_odm_pulled_object_replicates_and_target_as_source_is_rejected`, `on_demand_migration_put.rs::write_back_schedules_replication_and_names_the_migration_principal` |
| Source is itself a RustFS/MinIO deployment with its own source | The anti-loop marker stops the chain at the first hop; mutual configurations still terminate | `real_source_test.rs::test_odm_chained_sources_stop_at_the_loop_guard_real_single_node` |
| Config disabled or deleted | Source traffic stops immediately; already-pulled objects stay readable | `get_basic_test.rs::get_after_disable_does_not_consult_the_source`, `interaction_test.rs::test_odm_disable_keeps_pulled_objects_and_stops_source_traffic` |
Every response that was answered from the source — GET or HEAD — carries `x-rustfs-on-demand-migration: source`. A local hit carries no such header, which makes it the cheapest way to tell a migrated read from a served one.
Source-backed responses report only what the source can vouch for: its ETag, its `Last-Modified`, `Content-*`, `Cache-Control`, `Expires`, `Accept-Ranges` and `x-amz-meta-*`. There is no version id, no SSE header, no storage class and no checksum on a source-backed response, because the object is not local yet.
## Integrity and ETag
- The write-back is given the size the source advertised, so a truncated or over-long source body fails the local write instead of committing a partial object; the background path additionally enforces the advertised length in its pump, ahead of the write.
- When the source ETag is a bare 32-hex-digit value (a single-part, unencrypted object), it is used as the expected plaintext MD5 of the single-part write-back. A mismatch is a `BadDigest`, classified as `etag_mismatch`, and nothing is stored.
- A multipart source ETag (`<md5>-<n>`) cannot be verified this way and is not used as a digest; the object is still stored, and the multipart write-back re-chunks it by `multipart_part_size_bytes`, so the *local* ETag will differ from the source's unless `preserve_etag` keeps it for display.
- `preserve_etag` (default on) stores the source ETag as the object's ETag. It is dropped when the bucket encrypts by default — the SSE write path owns the ETag there, exactly like replication receive.
- The source ETag is always recorded in internal metadata regardless of the policy, so an audit or a later comparison can still see it.
- `Last-Modified` of a pulled object is the local write time, not the source's. The source timestamp is preserved in metadata.
## Metadata mapping
Copied to the local object:
| Source | Stored as |
|---|---|
| `Content-Type`, `Content-Encoding`, `Content-Disposition`, `Content-Language`, `Cache-Control`, `Expires` | The same standard headers (empty values are skipped) |
| `x-amz-meta-*` | The same user metadata |
| Object tags | Local tags, only when `policy.copy_tags` is set |
Deliberately **not** copied: the storage class (the local bucket's class applies), the source `Last-Modified` (the local write time applies), ACLs, and any source SSE header (the local bucket's encryption applies).
Five provenance keys are written on every pulled object under both internal prefixes (`x-rustfs-internal-` and `x-minio-internal-`), so they never appear in client-visible metadata: `odm-source` (`<provider>:<source bucket>`), `odm-source-etag`, `odm-source-last-modified`, `odm-source-version-id` (empty for an unversioned source) and `odm-pulled-at`. They are the audit trail that tells a migrated object from a client write.
## Protection mechanisms
| Mechanism | What it protects | Behaviour |
|---|---|---|
| Circuit breaker | A struggling source | 5 consecutive transport failures (throttle, timeout, connect, 5xx) inside 30 s open it for 30 s, then one probe decides. A source 404 is a healthy answer and resets the streak; 403 and unsupported-object errors are neutral |
| Negative cache | Repeated misses for keys the source does not have | A source 404 is remembered per key for `negative_cache_ttl_secs` (100 000 keys per bucket, LRU) |
| Singleflight | A thundering herd on one cold key | One leader pull per key; followers wait `first_byte_ms` for it, then re-read locally or stream through |
| Concurrency limit | Local write amplification | `max_concurrent_pulls` permits shared by inline and background pulls |
| Bounded queue | Unbounded memory on a burst | `pull_queue_capacity` waiting jobs; overflow is counted as `queue_full` and never fails a client response |
| Bandwidth limit | Source and network saturation | `bandwidth_limit_bytes_per_sec` (minimum 64 KiB/s) on the source client |
| Retry budget | Transient source blips | Background pulls retry a retryable failure up to 3 times (1 s / 4 s / 16 s plus jitter). Inline pulls never retry: the bytes are already on their way to the client |
| Anti-loop marker | Migration chains between RustFS/MinIO deployments | Every source request carries `x-rustfs-source-proxy-request` and `x-minio-source-proxy-request`; a request carrying it is always answered locally |
| Outbound endpoint policy | SSRF | See [outbound-connection-policy.md](outbound-connection-policy.md) |
## Background services
The background workers that belong to this feature are inventoried, with their desired sources, status inputs and declared side effects, in `docs/architecture/background-services-inventory.md`.
- **Write-back commit task** — spawned per inline pull, outlives the request so a client disconnect cannot truncate the stored object.
- **Pull queue dispatcher** — one per configured bucket, started lazily on the first background pull; drains the bounded queue under the concurrency limit and is cancelled when the bucket's config changes or disappears.
- **Backfill job and its recovery loop** — ships with rustfs/backlog#2159 (see [Backfill](#backfill-ships-with-odm-12)).
## Error codes
| Code | HTTP | When |
|---|---|---|
| `SourceUnavailable` | 424 | A source failure under `source_error = propagate`. The message carries only the failure class: `timeout`, `connect`, `server_error`, `throttled`, `access_denied`, `other`, `breaker_open` (an open breaker on a GET), `client_build` or `unsupported` (a bucket whose source client could not be built, including a credential-less source) |
| `SourceUnavailable` | 424 | An SSE-C source object, always — this one does not follow `source_error` |
| `NoSuchKey` | 404 | The source answered 404, the key is negative-cached, a HEAD hit an open breaker, or `source_error = not_found` hides a source failure |
| `OnDemandMigrationDisabled` | 400 | Admin `PUT` while `RUSTFS_ON_DEMAND_MIGRATION_ENABLED` is off |
| `OnDemandMigrationSourceUnreachable` | 400 | The admin probe's `HeadBucket` or one-key listing failed; the message names the class only |
| `NoSuchConfiguration` | 404 | Admin `GET` on a bucket with no source configured |
| `InvalidArgument` | 400 | Config validation failed (bad endpoint, empty region, out-of-range policy value, self-reference, replication loop, unknown JSON field) |
| `AccessDenied` | 403 | The admin action is not authorized, or the license check denies the entitlement |
| `NoSuchBucket` | 404 | The local bucket does not exist |
Source error messages never cross the boundary verbatim: an SDK message can embed the signed request, the endpoint host and the key, so only the stable class label is returned and logged.
## Observability
All series are bucket-scoped under `rustfs_on_demand_migration_*` and appear only for buckets with a configured source; when a bucket's config disappears its series are retired on the next collection cycle.
| Metric | Type | Labels |
|---|---|---|
| `requests_total` | counter | `bucket`, `op` (`get`, `head`), `outcome` (`source_hit`, `source_miss`, `source_error`, `breaker_open`, `negative_cached`, `filtered`, `unsupported`) |
| `pulled_bytes_total` | counter | `bucket` |
| `pulled_objects_total` | counter | `bucket`, `path` (`inline`, `background`, `backfill`) |
| `pull_failures_total` | counter | `bucket`, `reason` (`source_not_found`, `source_access_denied`, `source_throttled`, `source_timeout`, `source_connect`, `source_server_error`, `source_unsupported`, `source_other`, `etag_mismatch`, `local_write`, `canceled`, `queue_full`) |
| `inflight_pulls` | gauge | `bucket` |
| `queue_depth` | gauge | `bucket` |
| `source_latency_seconds_distribution` / `_sum` / `_count` | counter | `bucket`, plus `le` on the distribution |
| `breaker_state` | gauge | `bucket` — `0` closed, `1` half-open, `2` open |
```promql
# Share of GETs that entered ODM and were answered by the source
sum by (bucket) (rate(rustfs_on_demand_migration_requests_total{op="get",outcome="source_hit"}[5m]))
/ sum by (bucket) (rate(rustfs_on_demand_migration_requests_total{op="get"}[5m]))
# Migration throughput
sum by (bucket) (rate(rustfs_on_demand_migration_pulled_bytes_total[5m]))
# Alert: a source breaker is open
max by (bucket) (rustfs_on_demand_migration_breaker_state) >= 2
# Source latency p99
histogram_quantile(0.99, sum by (bucket, le) (rate(rustfs_on_demand_migration_source_latency_seconds_distribution[5m])))
# Pull failures by reason
sum by (bucket, reason) (rate(rustfs_on_demand_migration_pull_failures_total[5m]))
```
`GET .../{bucket}/status` returns the same counters plus the configuration view: `configured`, `enabled` (the config's own switch), `module_enabled` (the process switch), `provider`, `endpoint_host`, `breaker.state`, `counters` (`requests_total`, `pulled_bytes_total`, `pulled_objects_total`, `pull_failures_total`, `source_latency`), `last_source_error` (`class` and `at`), `inflight_pulls`, `queue_depth`, `served_by_source_ratio` and `updated_at`. Two fields are always `null` in this build and are not a bug: `breaker.opened_at` (the runtime holds a monotonic instant, not a wall clock) and `served_by_source_ratio` (there is no per-bucket GET total to divide by, and a fabricated `0` would be worse than an honest `null`). When a bucket has no live state on the node — the module is off, or nothing has been read yet — the runtime fields are `null` while `provider` and `endpoint_host` still reflect the saved config.
## Troubleshooting
**Admin `PUT` returns `OnDemandMigrationDisabled`.** `RUSTFS_ON_DEMAND_MIGRATION_ENABLED` is not `true` on the node that handled the request. Set it on every node and restart; the switch is read at startup.
**Admin `PUT` returns `OnDemandMigrationSourceUnreachable`.** The probe (`HeadBucket` plus a one-key list) failed. The message names the class only, so match it against the source's own access log: `access_denied` is credentials or a missing `s3:ListBucket`; `not_found` is a wrong bucket name or the wrong addressing style; `connect` is DNS, TLS or routing; `timeout` means the source did not answer inside `connect_ms`/`first_byte_ms`.
**Reads return 424 `SourceUnavailable`.** Read the class in the message and `last_source_error` in the status. `access_denied` means the key exists but `s3:GetObject` is missing (or the credentials were rotated). `server_error`/`timeout`/`connect` are transient and will open the breaker after five in a row. `unsupported` means the source object is SSE-C. If you would rather have clients see a plain 404 while you fix the source, set `policy.source_error = not_found` — but note that a sync client will then read it as "deleted".
**Everything 404s and `breaker_state` is `2`.** The breaker is open; it re-probes after 30 s. Fix the source first — while it is open, no source traffic leaves the node, which is the point.
**A key that exists on the source 404s immediately, without source traffic.** Either the negative cache still holds a previous 404 (wait out `negative_cache_ttl_secs`, default 30 s), or `filter.prefix` does not admit the key, or a local delete marker shadows it on a versioned bucket, or the request carried a `versionId`.
**The endpoint is rejected as an SSRF risk.** Loopback source endpoints are refused unless `RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET=true`; private addresses are always allowed. See [outbound-connection-policy.md](outbound-connection-policy.md).
**Self-signed TLS on the source.** Put the issuing certificate in `source.tls.ca_cert_pem` (PEM text, in the JSON body). `source.tls.skip_verify` exists for a lab and disables verification entirely — do not use it against a source reached over an untrusted network.
**`NoSuchBucket` from the source, or a 301/307 redirect.** The addressing style is wrong: a virtual-host request against a path-style-only server looks like a missing bucket, and a path-style request against AWS gets redirected. Set `source.path_style` explicitly instead of relying on `auto`.
**Large objects are served but never stored.** Check `pull_failures_total`: `queue_full` means bursts exceed `pull_queue_capacity` (raise it, or raise `max_concurrent_pulls`); `local_write` covers both a genuine local write failure and a bucket quota rejection; `etag_mismatch` means the source body did not match the ETag the source advertised. The client response is served in all three cases — only the local copy is missing.
**The first read of a key is slow, later ones are fast.** Expected: the first read pays a source HEAD plus a source GET. Watch `source_latency_seconds_*` for the source's contribution and lower `inline_max_bytes` if teeing large objects is hurting first-byte latency.
**Nothing at all happens on a miss.** Confirm in order: the module switch (`module_enabled` in the status), `enabled` in the config, `filter.prefix`, and that the request is a plain GET/HEAD (no `versionId`, no `partNumber`, no `x-rustfs-source-proxy-request` marker from a peer).
## Known limitations
- **Source updates do not propagate.** Once an object is pulled, the local copy is authoritative; a later change on the source is never noticed. Plan the cutover so the source stops taking writes.
- **Unversioned buckets re-pull deleted keys.** An unversioned bucket keeps nothing after a delete, so the key looks like an ordinary miss and is migrated again. Only a versioned bucket can shadow the source with a delete marker (`respect_local_delete_marker`).
- **SSE-C source objects are not supported.** They are rejected with 424 `unsupported`; migrate them by another route.
- **Anonymous (credential-less) sources are not supported yet.** `source.credentials: null` parses and passes structural validation, but the client builder has no anonymous mode, so the admin `PUT` refuses it and the runtime would treat such a bucket as unavailable. A public source still needs a key pair.
- **Azure Blob is not a supported source** (rustfs/backlog#2166). GCS is supported only through its XML interoperability API with HMAC keys.
- **LIST does not merge the source** (rustfs/backlog#2164). Only local objects are listed, so a client that lists before reading will not see un-migrated keys.
- **Write-through is undecided** (rustfs/backlog#2165). PUT and DELETE never reach the source in this version.
- **The source client inherits the SDK's default retry policy.** The standard smithy strategy retries up to 3 times, so one logical source call can be up to three wire requests; every breaker count therefore sits on top of a threefold request amplification against an already-struggling source.
- **`pull_failures_total` counts abandoned pulls, not attempts.** A pull that failed twice and then succeeded contributes nothing; attempt-level failure needs a new counter.
- **A quota rejection is reported as `local_write`.** The failure-reason label set has no `quota` value, so a full bucket is indistinguishable from another local write failure in the metrics; the log line and the bucket's usage tell them apart.
- **`policy.source_timeout.idle_ms` applies to background pulls only.** The inline tee path does not enforce it, so an inline pull that stalls mid-body is bounded only by the SDK read timeout.
- **The breaker's 30 s open window is a compile-time constant** with no environment override, which is why breaker-related tests have to wait it out.
- **`breaker.opened_at` and `served_by_source_ratio` are always `null`** in the status response (see Observability).
## Security notes
Source credentials are stored **unencrypted** in the bucket's metadata, in the same trust boundary as `bucket-targets.json` and the tier configuration; anyone who can read the drives can read them. Encryption at rest for bucket-level remote credentials is tracked in rustfs/backlog#2168. Until then, treat the source key as a long-lived secret: scope it to read-only on one bucket (or one prefix), and rotate it on the source after the migration completes, then clear the config.
Credentials never leave the server in readable form: every admin response returns the redacted config (`secret_key` and `session_token` become `REDACTED`), no log line carries the configuration, and probe or source failures report only a stable class label rather than the SDK message, which can embed the signed request. Configuration endpoints are also refused entirely when the endpoint string tries to embed userinfo.
Outbound safety rests on three guards: the shared endpoint policy that refuses loopback and metadata-service addresses, the validator that refuses a source naming this bucket on this deployment or one of its own replication targets, and the anti-loop request markers that stop a migration chain between RustFS or MinIO deployments at the first hop. Every source request additionally carries a `RustFS-OnDemandMigration/<version>` `User-Agent` suffix so the source operator can attribute the traffic.
docs/operations/outbound-connection-policy.md
+3
View File
@@ -19,12 +19,15 @@ RustFS validates every operator-configured outbound destination to close a serve
| OIDC discovery, JWKS, and token requests | Full policy | A blocked provider logs `OIDC provider discovery blocked by outbound policy` naming the origin to allowlist (`crates/iam/src/oidc.rs`) |
| Object Lambda targets | Full policy | `rustfs/src/admin/router.rs` `outbound_policy` |
| Bucket replication targets | Literal check, relaxed | Private addresses are always allowed; loopback only with `RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET=true` (`crates/ecstore/src/bucket/remote_s3_client.rs` `validate_remote_endpoint`, shared with on-demand migration sources) |
| On-demand migration sources | Literal check, relaxed | Same guard and same escape hatch as replication targets (see below) |
| Site replication peers | Literal check | `rustfs/src/site_replication/mod.rs` |
| Tiering warm backends (S3, MinIO, RustFS, Azure, GCS, Aliyun, Tencent, Huawei, R2) | Literal check | `crates/ecstore/src/services/tier/warm_backend.rs` `validate_endpoint`; the RustFS provider adds a debug-only, env-gated loopback exception for e2e tests |
| Keystone `auth_url` | Literal check | `crates/keystone/src/config.rs` |
The allowlist affects only the "Full policy" rows. A literal-check subsystem rejects a hostname that is itself a restricted IP literal, does not re-check what a hostname resolves to, and cannot be widened by `RUSTFS_OUTBOUND_ALLOW_ORIGINS`.
On-demand migration sources share the replication egress guard exactly: the source endpoint an operator saves through `PUT /rustfs/admin/v3/on-demand-migration/{bucket}` goes through the same `validate_remote_endpoint` in `crates/ecstore/src/bucket/remote_s3_client.rs`, so private addresses are accepted (a source on the same private network is the normal migration case), loopback is refused unless `RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET=true`, and metadata, link-local and unspecified addresses stay blocked. The variable is named for replication because it is one switch over one shared guard; setting it for a single-host migration also relaxes loopback for replication targets, so prefer a real hostname or private address for the source and keep the switch for lab and e2e hosts. `RUSTFS_OUTBOUND_ALLOW_ORIGINS` does not apply here. Two migration-specific rules sit on top of the endpoint guard and are enforced by the config validator, not by the egress layer: a source that names this bucket on this deployment is rejected as a self-reference, and a source matching one of the bucket's own replication targets is rejected as a replication loop. See [on-demand-migration.md](on-demand-migration.md).
## Symptoms
- Bucket event rules and webhook configuration look correct and uploads succeed, but no POST reaches the receiver.