# On-Demand Migration

**Use this when:** you are moving an existing S3-compatible bucket into RustFS without a stop-the-world copy, or you are debugging a bucket that serves reads from an external source (424 `SourceUnavailable`, an open circuit breaker, missing pulled objects, a source 403).
**Source of truth:** `rustfs/src/on_demand_migration/` (`config.rs` for the wire model and its bounds, `sys.rs` for the per-node runtime, `source_client.rs` for the outbound client, `pull.rs` for the write-back pipeline, `breaker.rs` and `negative_cache.rs` for the protections), `rustfs/src/app/object/get.rs` and `head.rs` for the read paths, `rustfs/src/app/object/on_demand_migration_put.rs` for the local write, `rustfs/src/admin/handlers/on_demand_migration.rs` for the admin API, and `crates/obs/src/metrics/schema/on_demand_migration.rs` for the metric contract.

On-Demand Migration (ODM) attaches an external S3-compatible **source bucket** to a local RustFS bucket. When a client GETs a key that does not exist locally, RustFS fetches it from the source, streams it to the client, and stores it locally in the same pass; every later read is served locally. It is a pull-style, lazy migration path — the RustFS equivalent of Cloudflare R2 Sippy, Tigris shadow buckets, and Alibaba Cloud OSS / Tencent COS mirror-back-to-origin.

The module is on by default (rustfs/backlog#2163); set `RUSTFS_ON_DEMAND_MIGRATION_ENABLED=false` on every node to turn it off (`rustfs/src/module_switches.rs`). With the switch off, the runtime never intervenes on a read and the admin `PUT` route refuses with `OnDemandMigrationDisabled`. Reads of the configuration and of the status endpoint keep working while the switch is off, so a disabled deployment can still be inspected. The switch only decides whether the module may act at all: a bucket with no `on-demand-migration.json` is never resolved by the runtime and makes no source call, so turning the module on changes nothing for buckets you have not configured.

## Upgrade and rollback compatibility

ODM configuration is stored in two additional keys in the existing bucket metadata map. The metadata format version remains `1` for MinIO compatibility. RustFS `1.0.0-rc.5` only re-encodes its 44 known keys: a bucket configuration write through an rc.5 node discards the ODM configuration and timestamp, even if another node originally wrote them. Restarting a newer binary cannot recover the discarded values. This also means ODM is not supported during a rolling upgrade that still allows rc.5 nodes to write bucket metadata.

Upgrade every node before enabling ODM. Before any rollback to rc.5, stop new migration work, retain a secure copy of the original full configuration and credentials, and disable ODM on every bucket and node. The redacted configuration GET and metadata export are not credential backups. Objects still present only at the source cannot be read through RustFS while ODM is disabled or rc.5 is running; finish migration first, redirect those reads to the source, or plan a maintenance window. After all nodes return to a compatible release, reapply and validate the saved configuration; already stored local objects remain local. Turning the global module switch off alone does not make an old metadata writer preserve these keys.

The ignored `upgrade_compatibility_test::rc5_rollback_requires_restoring_odm_configuration` test pins release commit `40a2470feb567201165a5b809b7598bb4b1f68f5`, restarts against the same data directory, writes bucket tags through rc.5, and verifies configuration recovery after returning to the current binary. Set `RUSTFS_UPGRADE_SOURCE_BINARY` to that release's executable and run `cargo test -p e2e_test rc5_rollback_requires_restoring_odm_configuration -- --ignored --test-threads=1`. The test records a known old-writer limitation; it does not certify mixed-version ODM operation.

## Optional Google dependencies

The default and `full` server builds include the `gcs` Cargo feature to preserve native GCS migration and existing GCS tier support. For a server without Google SDK dependencies, build with `cargo build -p rustfs --no-default-features --features ftps,webdav`. Add `gcs` to that feature list to restore native GCS support. The ECStore library has no default Google dependency; library users that need GCS tiers must enable its `gcs` feature.

Both builds can read, redact and preserve GCS configuration. A build without `gcs` rejects native ODM client construction with `OnDemandMigrationBackendNotCompiled` (HTTP 501); persisted native sources report an unavailable client. GCS tier initialization returns `XRustFSAdminTierTypeUnsupported` (HTTP 501). Do not deploy that build to a cluster with GCS tiers containing transitioned objects: the configuration remains intact, but reading their remote data requires a GCS-capable binary. The `gcs` provider using HMAC credentials and the S3 interoperability API remains available in every build; only `gcs_native` and native GCS tier clients need the feature.

## List continuation token rollout

The two rollout switches have different defaults. Unset or invalid boolean values use the stated default. Both are node environment variables, not bucket settings:

- `RUSTFS_ON_DEMAND_MIGRATION_LIST_V2_TOKENS` defaults to `false` and allows a v1 listing to first issue a v2 token after an empty truncated merged page. Existing v2 tokens keep their budget even on reader-only nodes. Consuming an object/common prefix or reaching a new EOF resets the budget to v1 without changing the chain's framing.
- `RUSTFS_ON_DEMAND_MIGRATION_LIST_FRAMED_TOKENS` defaults to `true`, preserving the framed output of the #7187 / `e1608fbd9` generation. It allows a bare/new merged listing to first issue a NUL-prefixed JSON envelope inside the existing base64 encoding. Existing framed chains stay framed even with this switch off, including a reset to v1 and local continuation after list-through is disabled. With framing issuance off, new bare v1 output keeps its historical bytes; ordinary local listings remain unchanged. This switch does not enable the v2 budget.

When list-through or the global module is disabled, existing merged continuations retain their last emitted key and original bare/framed format, filter already returned objects and common prefixes, and make no source requests. If the local-only response remains truncated, its continuation can resume both sides after re-enabling list-through.

Choose the upgrade path from the binaries currently serving LIST requests, including every load-balancer route. This build reads complete historical bare envelopes and framed v1/v2 envelopes with the same strict version/count validation. There is no single writer format understood by both bare-only and framed-only readers. A bare-v1-only binary rejects bare v2 with `400 InvalidArgument`; a bare-only reader mistakes framed input for a local marker, while a framed-only reader mistakes bare input for one. These framing mismatches can restart a merged scan and lose its budget without returning an error. For example, with local keys `b,d` and source keys `a,c`, a new node issuing bare after `a` followed by an `e1608fbd9` reader can return `a` again. That old reader then emits framing, so the symptom need not be an infinite loop.

- **Upgrading from the framed-only #7187 / `e1608fbd9` generation:** before deploying the first new node, set `RUSTFS_ON_DEMAND_MIGRATION_LIST_FRAMED_TOKENS=true` in the new nodes' deployment environment, or leave it unset to use this build's `true` default. Remove any previous explicit `false` override. Existing framed-only nodes ignore this new variable and already emit framing. Keep framing enabled while any of those nodes serves continuations. While list-through remains active, new and existing framed v1/v2 chains then retain their cursors and any active budget in both directions. Only after all serving nodes are dual readers may you choose `false`; existing framed chains still stay framed, while newly started bare chains must stay on dual readers.
- **Upgrading from older, pre-#7187 bare-only binaries:** explicitly set `RUSTFS_ON_DEMAND_MIGRATION_LIST_FRAMED_TOKENS=false` on every new node **before its first start**. Keep it false until all serving readers accept both formats; this preserves bare v1 bytes and active bare cursors during that rollout. Keep `RUSTFS_ON_DEMAND_MIGRATION_LIST_V2_TOKENS=false` until every reader also supports v2. After reader convergence, you may enable framing and then the v2 budget; enabling framing also frames the next nonzero merged continuation of an existing bare chain, so do not do this while bare-only readers remain.

Restart nodes after changing their environment. Do not directly mix bare-only and framed-only binaries on the same continuation routes. Existing bare tokens must be routed to dual readers while any framed-only nodes remain. The v2 issuance switch is independent: keep it off until all readers support v2, but turning it off never removes an existing v2 budget.

Partial JSON-shaped object keys remain local markers. To retain already issued cursors, a bare JSON object with the ODM tag and every historical writer field (`v`, `local`, `local_done`, `source`, `source_done`, `last_key`) is treated as an envelope, then strictly validated. A valid object key can be identical to that complete envelope: the two byte strings are indistinguishable, so legacy compatibility necessarily gives the envelope interpretation precedence. Framing identifies new merged tokens unambiguously, but dual-format readers do not eliminate this old full-envelope key collision. There is no signature, session store, or automatic format negotiation.

An active v2 budget rejects the sixteenth consecutive merged page that consumes no new object/common prefix and reaches no new end-of-list state. The first fifteen empty pages can be resumed; with the existing two-fetch-per-side limit, that interval costs at most 32 fetches per side, including the failing request. A key, common prefix, or a newly exhausted side on the sixteenth request succeeds and resets the budget. A side that was already exhausted does not reset it again. A zero-sized request does not spend an existing budget. This is a resource bound, not proof of a cursor cycle: an unusually long but valid empty source-page chain also reaches the limit. Tokens are unsigned base64-encoded JSON, optionally framed, so this budget applies to clients that continue with the returned token unchanged; replaying or editing a token can reset it, and it is not a malicious-client defense or a global request quota. The two-fetch-per-side request limit and existing source rate limiter still apply. A source failure follows `policy.source_error`: `propagate` returns `424 SourceUnavailable` with `invalid_pagination`; `not_found` returns the fetched local listing with `x-rustfs-on-demand-migration-list: local_only`. A blocking local-side failure returns `InternalError`, without silently discarding local entries.

With budget issuance off, a new v1 chain retains the existing limitation: an empty source cursor cycle spanning requests can continue indefinitely. Default rollout does not fix that chain until the v2 switch is enabled. Framing alone does not impose the budget.

For rollback, turn v2 issuance off, but choose framing for the readers being restored. When returning to the framed-only generation, keep framing `true` and route any outstanding bare tokens only to dual readers. Before restoring bare-only readers, set framing `false` on the remaining dual readers and deal with all outstanding framed tokens; v1-only readers also cannot resume v2 tokens. Neither switch rewrites existing framed or v2 chains, and tokens have no expiration that proves they have drained. Retain compatible readers for those continuations or have clients explicitly restart their listings before restoring incompatible binaries. Restarting a listing is a new scan and can repeat entries. Switching issuance off alone does not make outstanding tokens safe for older readers.

## Positioning

| Capability | Direction | What it moves | Where the authoritative copy is | When to use it instead |
|---|---|---|---|---|
| **On-demand migration** (this page) | Inbound pull from a foreign S3 bucket | The objects clients actually read, plus an optional background backfill for the rest | The source until an object is pulled; local afterwards | You are migrating away from another provider and want zero-downtime cutover |
| Bucket replication (`docs/operations/replication-check.md`) | Outbound push to a target | Every new write, continuously | Local; the target is a copy | You want an ongoing copy of local writes elsewhere |
| Site replication | Bidirectional, cluster-wide | Buckets, IAM, policies, objects | Every peer is authoritative | You want two RustFS deployments to converge |
| Tiering / ILM (`docs/operations/tier-ilm-debugging.md`) | Outbound push to a warm backend, transparent read-back | Cold local objects, by lifecycle rule | Local metadata, remote data | You want to reduce local capacity for cold data you already own |
| Replication read-proxy | Inbound proxy to a replication target | Nothing — it only proxies the response | The target | An active-active pair has a replication lag window; this runs *before* ODM on a miss and never stores anything |

The order on a GET miss is fixed: local read → replication read-proxy → on-demand migration. ODM is consulted last because it is the expensive, authoritative-external answer.

## Enabling a source

Admin requests are SigV4-signed like any S3 request; the examples use [`awscurl`](https://github.com/okigan/awscurl) with the deployment's admin credentials. The routes and their authorization are pinned in `docs/architecture/admin-route-action-snapshot.md`.

| Route | Action | Notes |
|---|---|---|
| `PUT /rustfs/admin/v3/on-demand-migration/{bucket}` | `admin:SetBucketOnDemandMigration` | Requires the module switch and the server license; validates, probes the source, then saves |
| `PUT /rustfs/admin/v3/on-demand-migration/{bucket}?dry-run=true` | `admin:SetBucketOnDemandMigration` | Same validation and probe, saves nothing (`updated_at` is `null`) |
| `GET /rustfs/admin/v3/on-demand-migration/{bucket}` | `admin:GetBucketOnDemandMigration` | Returns the redacted config; `NoSuchConfiguration` (404) when unset |
| `GET /rustfs/admin/v3/on-demand-migration/{bucket}/status` | `admin:GetBucketOnDemandMigration` | Switch state plus this node's runtime snapshot |
| `DELETE /rustfs/admin/v3/on-demand-migration/{bucket}` | `admin:SetBucketOnDemandMigration` | Idempotent, answers 204; already-pulled objects stay in place |

Validate first, with `dry-run`, so a bad endpoint or a wrong key never reaches bucket metadata:

```bash
cat > /tmp/odm.json <<'JSON'
{
  "source": {
    "provider": "minio",
    "endpoint": "https://source.example.com:9000",
    "region": "us-east-1",
    "bucket": "legacy-photos",
    "credentials": { "access_key": "AKIA…", "secret_key": "REDACTED" }
  },
  "filter": { "source_prefix": "photos/" },
  "policy": { "inline_max_bytes": 16777216, "max_concurrent_pulls": 8 }
}
JSON

awscurl --service s3 --region us-east-1 \
  --access_key "$RUSTFS_ACCESS_KEY" --secret_key "$RUSTFS_SECRET_KEY" \
  --request PUT --header 'Content-Type: application/json' \
  --data "$(cat /tmp/odm.json)" \
  "http://<host>:9000/rustfs/admin/v3/on-demand-migration/photos?dry-run=true"
```

The response echoes the **redacted** config (`secret_key` and `session_token` become `REDACTED`) and a probe summary: `reachable` (the source answered `HeadBucket`), `listable` (a one-key `ListObjectsV2` succeeded) and `sample_key` (the first key that listing returned, absent for an empty source). Repeat the call without `?dry-run=true` to save it; the handler then fans a metadata reload out to the peers so the whole cluster picks the source up without waiting for the refresh loop.

```bash
# Read the saved configuration back
awscurl --service s3 --region us-east-1 \
  --access_key "$RUSTFS_ACCESS_KEY" --secret_key "$RUSTFS_SECRET_KEY" \
  "http://<host>:9000/rustfs/admin/v3/on-demand-migration/photos"

# Per-node runtime status (breaker, counters, queue, last source error)
awscurl --service s3 --region us-east-1 \
  --access_key "$RUSTFS_ACCESS_KEY" --secret_key "$RUSTFS_SECRET_KEY" \
  "http://<host>:9000/rustfs/admin/v3/on-demand-migration/photos/status"

# Stop consulting the source; pulled objects stay
awscurl --service s3 --region us-east-1 \
  --access_key "$RUSTFS_ACCESS_KEY" --secret_key "$RUSTFS_SECRET_KEY" \
  --request DELETE \
  "http://<host>:9000/rustfs/admin/v3/on-demand-migration/photos"
```

Setting `"enabled": false` in the config has the same read-path effect as deleting it (the runtime tears the bucket state down) while keeping the source, credentials and policy on record for a later re-enable.

The status endpoint reports **the node that answered the request**. Counters, queue depth and breaker state are per-node runtime state, so in a distributed deployment query every node; the saved configuration and `updated_at` are cluster-wide.

### Backfill

Backfill waits for the result of every pull, including a pull already queued by an online request. A failed or cancelled shared pull is counted as a failure, never as successful migration. The persisted continuation cursor stays at the first failed page; a takeover replays from there and skips objects already present locally. `completed_with_failures` is not a cutover-ready state.

Read-through only migrates what clients touch. The background backfill job walks the source listing and pulls the remainder, with a persisted checkpoint (`.rustfs.sys/buckets/<bucket>/on-demand-migration-backfill.json`), a single-owner lease, resume after restart, and `POST .../{bucket}/backfill?op=start|cancel` plus `GET .../{bucket}/backfill` admin routes. See `docs/architecture/admin-route-action-snapshot.md` for the route contract.

## Configuration reference

The persisted blob is `on-demand-migration.json` in the bucket's metadata. Unknown fields are rejected rather than dropped, so a config written by a newer build fails loudly on an older one. Every default and bound below comes from `rustfs/src/on_demand_migration/config.rs`.

| Field | Type | Default | Bounds / rules |
|---|---|---|---|
| `version` | integer | `1` | Must be `1` |
| `enabled` | bool | `true` | `false` keeps the config but stops all source traffic |
| `source.provider` | `s3` \| `aws` \| `minio` \| `rustfs` \| `r2` \| `gcs` \| `azure` \| `gcs_native` | — (required) | Drives endpoint and addressing defaults, and which backend the client builds: every value but `azure` and `gcs_native` speaks S3 |
| `source.endpoint` | string \| null | — | `http(s)://host[:port]`, no path, query, fragment or userinfo. Required except for `aws` (derived from `region`), `azure` (derived as `https://<account>.blob.core.windows.net`) and `gcs_native` (`https://storage.googleapis.com`). Set it explicitly to point at Azurite or fake-gcs-server, subject to the same outbound policy as any other source endpoint |
| `source.region` | string | — (required) | Non-empty. `auto` is accepted for `r2`, `minio`, `rustfs` and for the native providers, and is signed as `us-east-1`. `azure` and `gcs_native` never sign with a region, so `auto` is the honest value there |
| `source.bucket` | string | — (required) | Non-empty, no `/` and no whitespace. For `azure` this is the container name, for `gcs_native` the bucket name; the provider block never repeats it |
| `source.path_style` | `auto` \| `path` \| `virtual` | `auto` | `auto` resolves to path-style for IP-literal or `localhost` endpoints and for `s3`/`minio`/`rustfs`; virtual-host for `aws`/`gcs`/`r2` |
| `source.credentials` | object \| null | `null` | Read only by the S3 providers; `azure` and `gcs_native` must leave it `null` and carry their credentials in their own block. `null` means anonymous, which the client builder does not support yet: the admin `PUT` refuses it with `InvalidArgument`, and a config that reached the metadata another way resolves as unavailable. `access_key` and `secret_key` must be non-empty; `session_token` is optional but must be non-empty when present |
| `source.tls.skip_verify` | bool | `false` | Disables certificate verification for the source connection |
| `source.tls.ca_cert_pem` | string \| null | `null` | Must contain `-----BEGIN CERTIFICATE-----` |
| `source.azure` | object \| null | `null` | Required for `provider = "azure"` and rejected for every other provider |
| `source.azure.account` | string | — (required) | Storage account name; `[A-Za-z0-9-]` only, because it becomes the first label of the derived host |
| `source.azure.account_key` | string \| null | `null` | Base64 storage-account key, signed per request with Shared Key. Mutually exclusive with `sas_token`; exactly one of the two is required |
| `source.azure.sas_token` | string \| null | `null` | SAS query string without the leading `?` and without whitespace, appended to every request URL |
| `source.gcs` | object \| null | `null` | Required for `provider = "gcs_native"` and rejected for every other provider |
| `source.gcs.service_account_json` | string | — (required) | Service-account key JSON; must parse and carry `type: service_account`, `client_email` and `private_key`. Tokens are minted read-only (`devstorage.read_only`) |
| `filter.prefix` | string \| null | `null` | Null or non-empty. Only local keys with this prefix consult the source |
| `filter.source_prefix` | string \| null | `null` | Null or non-empty. Prepended to the local key to form the source key |
| `policy.head` | `proxy` \| `local_only` | `proxy` | `local_only` answers a HEAD miss with 404 and no source traffic |
| `policy.range_get` | `serve_and_backfill` \| `serve_only` | `serve_and_backfill` | Whether a Range GET also queues a background pull of the whole object |
| `policy.source_error` | `propagate` \| `not_found` | `propagate` | `propagate` answers 424 `SourceUnavailable`; `not_found` degrades to 404, and for a merged listing to local state only |
| `policy.list_through` | bool | `false` | Merges the source listing into `ListObjectsV2` so clients see the whole namespace during the migration. Off by default: it puts the source in the path of every listing |
| `policy.respect_local_delete_marker` | bool | `true` | A local delete marker is the final answer; only a versioned bucket can produce one |
| `policy.preserve_etag` | bool | `true` | Keeps the source ETag on the stored object unless the bucket encrypts by default |
| `policy.copy_tags` | bool | `false` | Copies source object tags; needs `s3:GetObjectTagging` and costs one extra source call per inline pull. `azure` reads blob tags instead; `gcs_native` has no tags and always finds none |
| `policy.emit_events` | bool | `true` | Whether a write-back emits `ObjectCreated` notifications |
| `policy.negative_cache_ttl_secs` | integer | `30` | `0..=3600`; `0` disables the negative cache |
| `policy.inline_max_bytes` | integer | `16777216` (16 MiB) | `0..=268435456` (256 MiB). At or below this size a GET miss is teed inline; above it the response streams through and a background pull stores the object |
| `policy.multipart_part_size_bytes` | integer | `67108864` (64 MiB) | `5242880..=5368709120` (5 MiB…5 GiB). Objects needing more than 10 000 parts require a larger part size |
| `policy.max_concurrent_pulls` | integer | `8` | `1..=256`; shared by the inline and background paths |
| `policy.pull_queue_capacity` | integer | `1024` | `1..=65536`; a full queue drops the *background* job, never the client response |
| `policy.source_timeout.connect_ms` | integer | `5000` | `100..=600000` |
| `policy.source_timeout.first_byte_ms` | integer | `15000` | `100..=600000`; also the window a follower waits for the singleflight leader before streaming through |
| `policy.source_timeout.idle_ms` | integer | `30000` | `100..=600000`; enforced per body chunk on both the background pump and the inline tee |
| `policy.bandwidth_limit_bytes_per_sec` | integer \| null | `null` | When set, at least `65536` |

Values that are **not** configurable: the breaker opens after 5 consecutive counted failures inside a 30 s window, stays open for 30 s and then admits one probe (`breaker.rs`); the negative cache holds at most 100 000 keys per bucket with LRU eviction (`negative_cache.rs`); a background pull retries a retryable source failure at most 3 times with 1 s / 4 s / 16 s base delays plus up to 25 % jitter (`pull.rs`). The SDK's own retry policy is disabled on the source client. Each SDK operation makes one wire request; an ambiguous HEAD 404 additionally probes the bucket, within the same configured first-byte budget.

Validation also rejects two shapes outright: a source whose endpoint and bucket name **this** bucket on this deployment (`SelfReference`), and a source that matches one of the bucket's own replication targets (`ReplicationLoop`) — that pairing would amplify a write-back into a loop.

## Provider presets and source permissions

| Provider | Endpoint | Addressing | `region` | Notes | Interop evidence |
|---|---|---|---|---|---|
| `aws` | Optional; derived as `https://s3.<region>.amazonaws.com` | Virtual-host | Real region required (`auto` rejected) | The derived form only accepts `[A-Za-z0-9-]` in `region` | `cloud-source (aws)`, only while `ODM_INTEROP_AWS_*` are configured; no difference recorded yet |
| `s3` | Required | Path-style | Real region | Generic S3-compatible endpoint (Wasabi, Backblaze B2 S3 API, Ceph RGW, …) | No lane of its own; the preset is the same code path the `minio` job exercises |
| `minio` | Required | Path-style | `auto` allowed | | `minio-source`, nightly: read-through, HEAD passthrough, merged list pagination and a 5,000-object backfill; no difference recorded yet |
| `rustfs` | Required | Path-style | `auto` allowed | A RustFS source answers the migration request locally thanks to the anti-loop marker | `real_source_test.rs` in the `e2e-nightly` lane |
| `r2` | `https://<account-id>.r2.cloudflarestorage.com` | Virtual-host | `auto` allowed (signed as `us-east-1`) | | `cloud-source (r2)`, only while `ODM_INTEROP_R2_*` are configured; no difference recorded yet |
| `gcs` | `https://storage.googleapis.com` | Virtual-host | Real region required | Uses the GCS XML interoperability API with an HMAC key pair, not a service-account JSON key | `cloud-source (gcs)`, only while `ODM_INTEROP_GCS_HMAC_*` are configured; no difference recorded yet |
| `azure` | Optional; derived as `https://<account>.blob.core.windows.net` | Native Blob REST, not S3 | Unused; write `auto` | Needs `source.azure`; the container is `source.bucket`. Reads need `Read` on the blob and `List` on the container, plus `Tags` when `policy.copy_tags` is on | None yet: no interop job covers Azure |
| `gcs_native` | Optional; derived as `https://storage.googleapis.com` | Native GCS API, not S3 | Unused; write `auto` | Needs `source.gcs`. Reads use the XML API for objects and `objects.list` for listings, both with an OAuth token minted from the service-account key; the key needs `storage.objects.get` and `storage.objects.list` | None yet: no interop job covers native GCS |

Every backend answers the same trait contract, pinned by `backend_contract.rs` in `rustfs/src/on_demand_migration/`, and the three differences that contract allows are the ones documented here.

`azure` differs in two of them. Its ETag is a concurrency token rather than a digest of the bytes, so it is stored as `odm-source-etag` provenance and never used as the expected MD5 of a pulled object — the write-back integrity check falls back to the local digest. And its listing paginates only with an opaque marker: there is no "start after this key" form, so a caller that asks for one gets `Unsupported` instead of a listing that silently starts over.

`gcs_native` differs in the other two. Its listing also has no exclusive "start after" form (`startOffset` is inclusive), so it refuses one the same way. And GCS has no object tagging at all: `policy.copy_tags` finds no tags rather than failing the pull, because GCS custom metadata is already carried by the head mapping. Its ETag is normally usable: the `x-goog-hash` MD5 is converted to hex and checked against the pulled bytes, except on a composite object, which has no MD5 and whose ETag is then treated as opaque.

The "Interop evidence" column names the job in `.github/workflows/on-demand-migration-interop.yml` (rustfs/backlog#2167) that last exercised the preset against a real implementation, and is where a provider difference belongs once the lane finds one. That lane is report-only and scheduled: it runs `crates/e2e_test/src/on_demand_migration/interop_test.rs` — the same case bodies as the merge-gate suite, with the source injected through `RUSTFS_ODM_INTEROP_*` — against a pinned MinIO container, and against each cloud provider whose repository secrets are configured. A provider without secrets is skipped with a note in the run summary rather than failing, so "no difference recorded yet" means exactly that and not "verified clean"; see [ci-gates.md](../testing/ci-gates.md) for the row.

The credentials only ever need read access to the source bucket:

- `s3:ListBucket` on the bucket — used by the admin probe, by the backfill listing, and by every merged listing under `policy.list_through`.
- `s3:GetObject` on `<bucket>/*` — used by every HEAD and GET against the source.
- `s3:GetObjectTagging` on `<bucket>/*` — only when `policy.copy_tags` is `true`.

No write, delete, ACL or versioning permission is required or used. Scope the policy to the prefix in `filter.source_prefix` when the source bucket holds unrelated data.

## Request semantics

Behaviour a client can observe. The "Test" column names the case that pins it: `*_test.rs` files live under `crates/e2e_test/src/on_demand_migration/`, and the unit tests live next to the code in `rustfs/src/app/object/get.rs`, `head.rs` and `shared.rs`.

ODM merged continuation tokens use bare or NUL-prefixed JSON inside the existing base64 encoding. The default writer preserves framed output; an explicit `RUSTFS_ON_DEMAND_MIGRATION_LIST_FRAMED_TOKENS=false` keeps historical bare output during older-reader rollouts. Compatible readers accept both formats and retain existing budgets. See [List continuation token rollout](#list-continuation-token-rollout) for the independent issuance switches, rolling-upgrade requirements, and the unavoidable ambiguity between a complete historical envelope and an identically named local key.

Source `HEAD` responses with status 404 require a successful bucket probe before being negative-cached. The source credential therefore needs permission for `HeadBucket` (S3 `ListBucket`); a prefix-restricted ListBucket policy can deny that probe, in which case the response is a source failure rather than a cached miss. A missing/inaccessible source bucket or a missing source version is not proof that the requested key is absent. Native GCS verifies the bucket after either HEAD or GET returns 404 and preserves a failed probe as a source error. Azure accepts explicit `BlobNotFound` only on an unversioned object read with status 404; an ambiguous HEAD may make one container probe, while an ambiguous GET remains a source error. Native probes add at most one request and retain the existing per-request timeouts, rather than a single deadline for the pair. Conditional GET validators are checked against the actual source GET metadata as well as the advisory HEAD; a missing required validator fails with 424. Source LIST entries without a key or a non-negative size fail the page rather than fabricating an empty object.

Write-back currently requires namespace locking enabled and exactly one pool with one erasure set. Other topologies fail write-back explicitly as `unsupported`: source reads remain available, but backfill cannot complete successfully or certify cutover. This restriction avoids relying on a set-local condition across distinct pool or lock domains; it does not restrict ordinary S3 writes. Full cross-pool migration requires a globally fenced commit protocol.

On the supported topology, write-back uses a create-only check under the local storage commit lock for both single-part PUT and multipart completion. A client write that commits while ODM is reading the source is preserved. With `respect_local_delete_marker=true`, a concurrent versioned deletion is preserved too. An explicit `respect_local_delete_marker=false` still permits revival; an unversioned deletion has no tombstone and therefore cannot be distinguished from a key that has never existed locally.

| Situation | Behaviour | Test |
|---|---|---|
| GET miss, object at or below `inline_max_bytes` | One source GET, teed: the client streams while the same bytes are written locally. Later reads are local and carry no source marker | `get_basic_test.rs::get_miss_pulls_inline_and_serves_locally_afterwards`, `get.rs::odm_get_inline_streams_to_client_and_commits_the_same_bytes` |
| GET miss, object above `inline_max_bytes` | Streamed straight through (nothing stored inline) and a background pull of the whole object is queued | `get_basic_test.rs::get_large_object_streams_through_and_backfills_in_background`, `get.rs::odm_get_large_object_streams_through_and_queues_a_background_pull` |
| Range GET miss | The requested range is passed through as 206; with `range_get = serve_and_backfill` a whole-object background pull is queued, with `serve_only` nothing is queued | `get_basic_test.rs::get_range_streams_206_and_backfills_the_whole_object`, `get.rs::odm_get_range_streams_206_and_queues_per_policy` |
| Client disconnects mid-stream on an inline pull | The write-back keeps draining the source and still stores the whole object | `get.rs::odm_get_inline_client_disconnect_still_stores_the_whole_object` |
| Concurrent GET misses of one key | Singleflight: one leader tees, followers wait up to `first_byte_ms` and then re-read locally; on leader failure or timeout they stream through without queueing | `concurrency_test.rs::test_odm_concurrent_misses_on_one_key_coalesce`, `get.rs::odm_get_follower_rereads_local_after_the_leader_commits` |
| HEAD miss | Proxied to the source, nothing is written locally; `local_only` answers 404 without source traffic | `head.rs::odm_head_source_hit_returns_output_and_writes_nothing_back`, `head.rs::odm_head_local_only_policy_is_404_without_source_traffic`, `real_source_test.rs::test_odm_rustfs_source_serves_pull_head_range_and_prefixes_real_single_node` |
| LIST, `policy.list_through = false` (default) | Local only. A key that exists on the source but was never pulled is not listed, even while a GET of it succeeds | `interaction_test.rs::test_odm_write_back_respects_the_bucket_quota` |
| `ListObjectsV2` with `policy.list_through = true` | The local and source listings are merged into one ordered page: byte-wise key order, local wins a key both sides hold, `CommonPrefixes` unioned and deduplicated under a delimiter. A source entry reports the source's own ETag, size and last-modified, storage class `STANDARD` and this bucket's owner. The continuation token is opaque and carries both cursors | `list_through_test.rs::list_through_merges_the_whole_namespace_across_full_pagination`, `list_through_test.rs::list_through_unions_common_prefixes_under_a_delimiter`, `list_through.rs::merged_pagination_equals_the_deduplicated_union` |
| `ListObjects` (v1) and `ListObjectVersions`, whatever `list_through` says | Local only. v1 has no opaque continuation token that could carry two cursors, and a version listing has no meaning for a source whose versions were never pulled | `list_through_test.rs::a_merged_token_keeps_paginating_after_list_through_is_turned_off` |
| Merged listing, source listing fails or the breaker is open | `propagate` answers 424 `SourceUnavailable`; `not_found` answers from local state alone and marks the response `x-rustfs-on-demand-migration-list: local_only` | `list_through_test.rs::list_through_propagates_a_source_listing_failure`, `list_through_test.rs::list_through_degrades_to_local_only_under_the_not_found_policy` |
| Merged listing, tampered continuation token | 400 `InvalidArgument`; a token issued while `list_through` was on keeps paginating the local side after it is turned off | `list_through_test.rs::list_through_rejects_a_tampered_continuation_token`, `list_through.rs::token_round_trips_and_rejects_tampering` |
| Merged listing, versioned bucket with a local delete marker | The shadowed key is dropped from the page, matching `respect_local_delete_marker` on GET. The local listing never returns delete markers, so such a bucket costs one extra local metadata read per source-only key in the page | `list_through_test.rs::a_local_delete_marker_hides_the_source_key_from_a_merged_listing` |
| PUT / DELETE | Never touch the source. A local PUT shadows the source key for good | `interaction_test.rs::test_odm_delete_marker_shadows_the_source_but_a_plain_delete_does_not` |
| Versioned bucket, local delete marker | With `respect_local_delete_marker` (default) the marker is the final answer and the source is not consulted | `interaction_test.rs::test_odm_delete_marker_shadows_the_source_but_a_plain_delete_does_not`, `head.rs::odm_head_verdict_respects_local_delete_marker_by_policy` |
| Unversioned bucket, object deleted locally | The key becomes an ordinary miss and is pulled from the source again | `interaction_test.rs::test_odm_delete_marker_shadows_the_source_but_a_plain_delete_does_not` |
| GET with `versionId`, or a `partNumber` read | Never consults the source; the local 404 stands | `get_basic_test.rs::get_with_version_id_does_not_consult_the_source`, `get.rs::odm_get_gate_rejects_part_reads_and_version_reads` |
| Conditional headers (`If-Match`, `If-None-Match`, `If-Modified-Since`, `If-Unmodified-Since`) | Evaluated locally against the source HEAD with the usual S3 semantics; never forwarded to the source | `get.rs::odm_get_conditional_headers_are_answered_from_the_source_head`, `shared.rs::odm_source_preconditions_follow_local_semantics` |
| Source object is SSE-C encrypted | 424 `SourceUnavailable` with class `unsupported`, whatever `source_error` says | `fault_test.rs::test_odm_ssec_source_object_is_unsupported`, `head.rs::odm_head_unsupported_source_object_is_424_regardless_of_policy` |
| Source answers 404 | 404 to the client and the key is negative-cached for `negative_cache_ttl_secs` | `get_basic_test.rs::get_source_not_found_is_404_and_negative_cached`, `fault_test.rs::test_odm_source_not_found_is_negative_cached_for_the_ttl` |
| Source answers 403 | 424 (or 404 under `not_found`); the breaker is **not** opened — a credential problem is not a transient one | `fault_test.rs::test_odm_source_access_denied_propagates_without_opening_the_breaker` |
| Repeated source 5xx / timeouts | The breaker opens, GETs answer per `source_error` and HEADs answer 404; a probe closes it again after the open window | `fault_test.rs::test_odm_repeated_source_errors_open_the_breaker_and_recover`, `head.rs::odm_head_source_errors_follow_policy_and_open_the_breaker` |
| Source stalls mid-body on an inline pull | `source_timeout.idle_ms` ends both tee ends: the client's read fails instead of receiving a silently truncated 200, the pull counts under `source_timeout`, and no object or multipart upload is left behind | `fault_test.rs::test_odm_inline_pull_aborts_when_the_source_body_stalls`, `pull.rs::commit_inline_reports_a_stalled_source_as_a_source_timeout` |
| Bucket default SSE | The pulled object is stored encrypted and reads back in plaintext; the ETag override is dropped, and the source ETag is kept in metadata | `interaction_test.rs::test_odm_pulled_object_uses_bucket_default_encryption`, `on_demand_migration_put.rs::write_back_under_bucket_default_sse_stores_ciphertext_and_records_source_etag` |
| Object Lock bucket | The pulled object inherits the bucket's default retention | `interaction_test.rs::test_odm_pulled_object_inherits_object_lock_retention` |
| Bucket quota exceeded | The client is still served from the source; the write-back is rejected, nothing is stored, and the failure counts under `quota` | `interaction_test.rs::test_odm_write_back_respects_the_bucket_quota`, `on_demand_migration_put.rs::write_back_reports_a_full_bucket_quota` |
| Notifications | A write-back emits `ObjectCreated` with principal `rustfs-on-demand-migration`, unless `emit_events` is `false` | `interaction_test.rs::test_odm_pull_emits_object_created_events_unless_disabled` |
| Replication configured on the local bucket | A pulled object replicates like any other write; configuring one of the bucket's replication targets as the source is rejected | `interaction_test.rs::test_odm_pulled_object_replicates_and_target_as_source_is_rejected`, `on_demand_migration_put.rs::write_back_schedules_replication_and_names_the_migration_principal` |
| Source is itself a RustFS/MinIO deployment with its own source | The anti-loop marker stops the chain at the first hop; mutual configurations still terminate | `real_source_test.rs::test_odm_chained_sources_stop_at_the_loop_guard_real_single_node` |
| Config disabled or deleted | Source traffic stops immediately; already-pulled objects stay readable | `get_basic_test.rs::get_after_disable_does_not_consult_the_source`, `interaction_test.rs::test_odm_disable_keeps_pulled_objects_and_stops_source_traffic` |

Every response that was answered from the source — GET or HEAD — carries `x-rustfs-on-demand-migration: source`. A local hit carries no such header, which makes it the cheapest way to tell a migrated read from a served one.

Source-backed responses report only what the source can vouch for: its ETag, its `Last-Modified`, `Content-*`, `Cache-Control`, `Expires`, `Accept-Ranges` and `x-amz-meta-*`. There is no version id, no SSE header, no storage class and no checksum on a source-backed response, because the object is not local yet.

## Integrity and ETag

- The write-back is given the size the source advertised, so a truncated or over-long source body fails the local write instead of committing a partial object; the background path additionally enforces the advertised length in its pump, ahead of the write.
- When the source ETag is a bare 32-hex-digit value (a single-part, unencrypted object), it is used as the expected plaintext MD5 of the single-part write-back. A mismatch is a `BadDigest`, classified as `etag_mismatch`, and nothing is stored.
- A multipart source ETag (`<md5>-<n>`) cannot be verified this way and is not used as a digest; the object is still stored, and the multipart write-back re-chunks it by `multipart_part_size_bytes`, so the *local* ETag will differ from the source's unless `preserve_etag` keeps it for display.
- `preserve_etag` (default on) stores the source ETag as the object's ETag. It is dropped when the bucket encrypts by default — the SSE write path owns the ETag there, exactly like replication receive.
- The source ETag is always recorded in internal metadata regardless of the policy, so an audit or a later comparison can still see it.
- `Last-Modified` of a pulled object is the local write time, not the source's. The source timestamp is preserved in metadata.

A preserved source ETag does not describe the local part layout. Part reads and object attributes use the stored parts, and replication uses multipart transport when their logical boundaries are available. Legacy compressed or encrypted objects without a multipart ETag can lack those boundaries; they retain their existing streaming PUT path and its 5 GiB limit. Their replication target may therefore store a different part layout.

## Metadata mapping

Copied to the local object:

| Source | Stored as |
|---|---|
| `Content-Type`, `Content-Encoding`, `Content-Disposition`, `Content-Language`, `Cache-Control`, `Expires` | The same standard headers (empty values are skipped) |
| `x-amz-meta-*` | The same user metadata |
| Object tags | Local tags, only when `policy.copy_tags` is set |

Deliberately **not** copied: the storage class (the local bucket's class applies), the source `Last-Modified` (the local write time applies), ACLs, and any source SSE header (the local bucket's encryption applies).

Five provenance keys are written on every pulled object under both internal prefixes (`x-rustfs-internal-` and `x-minio-internal-`), so they never appear in client-visible metadata: `odm-source` (`<provider>:<source bucket>`), `odm-source-etag`, `odm-source-last-modified`, `odm-source-version-id` (empty for an unversioned source) and `odm-pulled-at`. They are the audit trail that tells a migrated object from a client write.

## Protection mechanisms

| Mechanism | What it protects | Behaviour |
|---|---|---|
| Circuit breaker | A struggling source | 5 consecutive transport failures (throttle, timeout, connect, 5xx) inside 30 s open it for 30 s, then one probe decides. A source 404 is a healthy answer and resets the streak; 403 and unsupported-object errors are neutral |
| Negative cache | Repeated misses for keys the source does not have | A source 404 is remembered per key for `negative_cache_ttl_secs` (100 000 keys per bucket, LRU) |
| Singleflight | A thundering herd on one cold key | One leader pull per key; followers wait `first_byte_ms` for it, then re-read locally or stream through |
| Concurrency limit | Local write amplification | `max_concurrent_pulls` permits shared by inline and background pulls |
| Bounded queue | Unbounded memory on a burst | `pull_queue_capacity` waiting jobs; overflow is counted as `queue_full` and never fails a client response |
| Bandwidth limit | Source and network saturation | `bandwidth_limit_bytes_per_sec` (minimum 64 KiB/s) on the source client |
| Retry budget | Transient source blips | Background pulls retry a retryable failure up to 3 times (1 s / 4 s / 16 s plus jitter). Inline pulls never retry: the bytes are already on their way to the client. The SDK retry policy is disabled (`RemoteS3RetryPolicy::Disabled`); HEAD 404 also requires one bucket probe. Replication targets keep their separately declared three SDK attempts |
| Idle timeout | A source that answers and then goes quiet mid-body | `source_timeout.idle_ms` per body chunk on both paths. The budget measures the source read, upstream of the inline tee, so a slow client is never mistaken for an idle source; when it fires the client stream ends in an error and the write-back is discarded |
| Anti-loop marker | Migration chains between RustFS/MinIO deployments | Every source request carries `x-rustfs-source-proxy-request` and `x-minio-source-proxy-request`; a request carrying it is always answered locally |
| Outbound endpoint policy | SSRF | See [outbound-connection-policy.md](outbound-connection-policy.md) |

## Background services

The background workers that belong to this feature are inventoried, with their desired sources, status inputs and declared side effects, in `docs/architecture/background-services-inventory.md`.

- **Write-back commit task** — spawned per inline pull, outlives the request so a client disconnect cannot truncate the stored object.
- **Pull queue dispatcher** — one per configured bucket, started lazily on the first background pull; drains the bounded queue under the concurrency limit and is cancelled when the bucket's config changes or disappears.
- **Backfill job and its recovery loop** — ships with rustfs/backlog#2159 (see [Backfill](#backfill-ships-with-odm-12)).

## Error codes

| Code | HTTP | When |
|---|---|---|
| `SourceUnavailable` | 424 | A source failure under `source_error = propagate`. The message carries only the failure class: `timeout`, `connect`, `server_error`, `throttled`, `access_denied`, `other`, `breaker_open` (an open breaker on a GET), `client_build` or `unsupported` (a bucket whose source client could not be built, including a credential-less source) |
| `SourceUnavailable` | 424 | An SSE-C source object, always — this one does not follow `source_error` |
| `NoSuchKey` | 404 | The source answered 404, the key is negative-cached, a HEAD hit an open breaker, or `source_error = not_found` hides a source failure |
| `OnDemandMigrationDisabled` | 400 | Admin `PUT` while `RUSTFS_ON_DEMAND_MIGRATION_ENABLED` is off |
| `OnDemandMigrationSourceUnreachable` | 400 | The admin probe's `HeadBucket` or one-key listing failed; the message names the class only |
| `NoSuchConfiguration` | 404 | Admin `GET` on a bucket with no source configured |
| `InvalidArgument` | 400 | Config validation failed (bad endpoint, empty region, out-of-range policy value, self-reference, replication loop, unknown JSON field) |
| `AccessDenied` | 403 | The admin action is not authorized, or the license check denies the entitlement |
| `NoSuchBucket` | 404 | The local bucket does not exist |

Source error messages never cross the boundary verbatim: an SDK message can embed the signed request, the endpoint host and the key, so only the stable class label is returned and logged.

## Observability

All series are bucket-scoped under `rustfs_on_demand_migration_*` and appear only for buckets with a configured source; when a bucket's config disappears its series are retired on the next collection cycle.

| Metric | Type | Labels |
|---|---|---|
| `requests_total` | counter | `bucket`, `op` (`get`, `head`), `outcome` (`source_hit`, `source_miss`, `source_error`, `breaker_open`, `negative_cached`, `filtered`, `unsupported`) |
| `pulled_bytes_total` | counter | `bucket` |
| `pulled_objects_total` | counter | `bucket`, `path` (`inline`, `background`, `backfill`) |
| `pull_failures_total` | counter | `bucket`, `reason` (`source_not_found`, `source_access_denied`, `source_throttled`, `source_timeout`, `source_connect`, `source_server_error`, `source_unsupported`, `source_other`, `etag_mismatch`, `local_write`, `quota`, `canceled`, `queue_full`) |
| `inflight_pulls` | gauge | `bucket` |
| `queue_depth` | gauge | `bucket` |
| `source_latency_seconds_distribution` / `_sum` / `_count` | counter | `bucket`, plus `le` on the distribution |
| `breaker_state` | gauge | `bucket` — `0` closed, `1` half-open, `2` open |

```promql
# Share of GETs that entered ODM and were answered by the source
sum by (bucket) (rate(rustfs_on_demand_migration_requests_total{op="get",outcome="source_hit"}[5m]))
  / sum by (bucket) (rate(rustfs_on_demand_migration_requests_total{op="get"}[5m]))

# Migration throughput
sum by (bucket) (rate(rustfs_on_demand_migration_pulled_bytes_total[5m]))

# Alert: a source breaker is open
max by (bucket) (rustfs_on_demand_migration_breaker_state) >= 2

# Source latency p99
histogram_quantile(0.99, sum by (bucket, le) (rate(rustfs_on_demand_migration_source_latency_seconds_distribution[5m])))

# Pull failures by reason
sum by (bucket, reason) (rate(rustfs_on_demand_migration_pull_failures_total[5m]))
```

`GET .../{bucket}/status` returns the same counters plus the configuration view: `configured`, `enabled` (the config's own switch), `module_enabled` (the process switch), `provider`, `endpoint_host`, `breaker.state`, `counters` (`requests_total`, `pulled_bytes_total`, `pulled_objects_total`, `pull_failures_total`, `source_latency`), `last_source_error` (`class` and `at`), `inflight_pulls`, `queue_depth`, `served_by_source_ratio` and `updated_at`. Two fields are always `null` in this build and are not a bug: `breaker.opened_at` (the runtime holds a monotonic instant, not a wall clock) and `served_by_source_ratio` (there is no per-bucket GET total to divide by, and a fabricated `0` would be worse than an honest `null`). When a bucket has no live state on the node — the module is off, or nothing has been read yet — the runtime fields are `null` while `provider` and `endpoint_host` still reflect the saved config.

## Troubleshooting

**Admin `PUT` returns `OnDemandMigrationDisabled`.** `RUSTFS_ON_DEMAND_MIGRATION_ENABLED` is not `true` on the node that handled the request. Set it on every node and restart; the switch is read at startup.

**Admin `PUT` returns `OnDemandMigrationSourceUnreachable`.** The probe (`HeadBucket` plus a one-key list) failed. The message names the class only, so match it against the source's own access log: `access_denied` is credentials or a missing `s3:ListBucket`; `not_found` is a wrong bucket name or the wrong addressing style; `connect` is DNS, TLS or routing; `timeout` means the source did not answer inside `connect_ms`/`first_byte_ms`.

**Reads return 424 `SourceUnavailable`.** Read the class in the message and `last_source_error` in the status. `access_denied` means the key exists but `s3:GetObject` is missing (or the credentials were rotated). `server_error`/`timeout`/`connect` are transient and will open the breaker after five in a row. `unsupported` means the source object is SSE-C. If you would rather have clients see a plain 404 while you fix the source, set `policy.source_error = not_found` — but note that a sync client will then read it as "deleted".

**Everything 404s and `breaker_state` is `2`.** The breaker is open; it re-probes after 30 s. Fix the source first — while it is open, no source traffic leaves the node, which is the point.

**A key that exists on the source 404s immediately, without source traffic.** Either the negative cache still holds a previous 404 (wait out `negative_cache_ttl_secs`, default 30 s), or `filter.prefix` does not admit the key, or a local delete marker shadows it on a versioned bucket, or the request carried a `versionId`.

**The endpoint is rejected as an SSRF risk.** Loopback source endpoints are refused unless `RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET=true`; private addresses are always allowed. See [outbound-connection-policy.md](outbound-connection-policy.md).

**Self-signed TLS on the source.** Put the issuing certificate in `source.tls.ca_cert_pem` (PEM text, in the JSON body). `source.tls.skip_verify` exists for a lab and disables verification entirely — do not use it against a source reached over an untrusted network.

**`NoSuchBucket` from the source, or a 301/307 redirect.** The addressing style is wrong: a virtual-host request against a path-style-only server looks like a missing bucket, and a path-style request against AWS gets redirected. Set `source.path_style` explicitly instead of relying on `auto`.

**Large objects are served but never stored.** Check `pull_failures_total`: `queue_full` means bursts exceed `pull_queue_capacity` (raise it, or raise `max_concurrent_pulls`); `quota` means the bucket quota rejected the write-back and `local_write` a genuine local write failure; `etag_mismatch` means the source body did not match the ETag the source advertised. The client response is served in all these cases — only the local copy is missing. `source_timeout` is different: the source stopped sending mid-body, so the client's own read fails too and nothing is stored.

**The first read of a key is slow, later ones are fast.** Expected: the first read pays a source HEAD plus a source GET. Watch `source_latency_seconds_*` for the source's contribution and lower `inline_max_bytes` if teeing large objects is hurting first-byte latency.

**Nothing at all happens on a miss.** Confirm in order: the module switch (`module_enabled` in the status), `enabled` in the config, `filter.prefix`, and that the request is a plain GET/HEAD (no `versionId`, no `partNumber`, no `x-rustfs-source-proxy-request` marker from a peer).

## Known limitations

- **Source updates do not propagate.** Once an object is pulled, the local copy is authoritative; a later change on the source is never noticed. Plan the cutover so the source stops taking writes.
- **Unversioned buckets re-pull deleted keys.** An unversioned bucket keeps nothing after a delete, so the key looks like an ordinary miss and is migrated again. Only a versioned bucket can shadow the source with a delete marker (`respect_local_delete_marker`).
- **SSE-C source objects are not supported.** They are rejected with 424 `unsupported`; migrate them by another route.
- **Native Azure/GCS keys containing a standalone `.` or `..` path segment are unsupported.** The URL transport would remove that segment and address a different object. These keys fail before any source request; ordinary dotted names, repeated slashes and literal percent escapes keep their identity.
- **Anonymous S3 sources are not supported yet.** `source.credentials: null` parses and passes structural validation, but the S3 client builder has no anonymous mode, so the admin `PUT` refuses it and the runtime would treat such a bucket as unavailable. A public S3 source still needs a key pair; native Azure/GCS credentials belong in their provider blocks.
- **LIST merges the source only when asked, and only for v2.** With the default `policy.list_through = false` a client that lists before reading will not see un-migrated keys. Turning it on merges `ListObjectsV2` alone; `ListObjects` (v1) and `ListObjectVersions` stay local.
- **A merged listing costs up to two local listings and two source listings per page** (one per side, plus a refill when the previous page consumed most of what that side had buffered). Walking N merged keys at `max-keys=K` therefore costs ceil(N/K) requests and between ceil(N/K) and 2*ceil(N/K) source listings. Source listings are capped at 10 per second per bucket (a compile-time constant); a listing that cannot get a slot inside one second is treated like a source failure and follows `policy.source_error`.
- **A degraded merged page loses the source keys in its window.** Under `source_error = not_found` the page is answered locally and the source cursor is left where it was, so the keys the source would have contributed between the previous page's last key and this one are not shown again once pagination moves on. The `x-rustfs-on-demand-migration-list: local_only` header marks every page this happened on.
- **Write-through is undecided** (rustfs/backlog#2165). PUT and DELETE never reach the source in this version.
- **`pull_failures_total` counts abandoned pulls, not attempts.** A pull that failed twice and then succeeded contributes nothing; attempt-level failure needs a new counter.
- **The breaker's 30 s open window is a compile-time constant** with no environment override, which is why breaker-related tests have to wait it out.
- **`breaker.opened_at` and `served_by_source_ratio` are always `null`** in the status response (see Observability).

## Security notes

Source credentials are stored **unencrypted** in the bucket's metadata, in the same trust boundary as `bucket-targets.json` and the tier configuration; anyone who can read the drives can read them. Encryption at rest for bucket-level remote credentials is tracked in rustfs/backlog#2168. Until then, treat the source key as a long-lived secret: scope it to read-only on one bucket (or one prefix), and rotate it on the source after the migration completes, then clear the config.

Credentials never leave the server in readable form: every admin response returns the redacted config (`secret_key` and `session_token` become `REDACTED`), no log line carries the configuration, and probe or source failures report only a stable class label rather than the SDK message, which can embed the signed request. Configuration endpoints are also refused entirely when the endpoint string tries to embed userinfo.

Outbound safety rests on three guards: the shared endpoint policy that refuses loopback and metadata-service addresses, the validator that refuses a source naming this bucket on this deployment or one of its own replication targets, and the anti-loop request markers that stop a migration chain between RustFS or MinIO deployments at the first hop. Every source request additionally carries a `RustFS-OnDemandMigration/<version>` `User-Agent` suffix so the source operator can attribute the traffic.
