feat(odm): enable on-demand migration by default (#7089)

* feat(odm): enable on-demand migration by default

The module switch RUSTFS_ON_DEMAND_MIGRATION_ENABLED now defaults to true,
so the feature is reachable without an opt-in; setting it to false still
keeps the module out of the read path entirely. A bucket without an
on-demand-migration.json is never resolved by the runtime and makes no
source call, so the flip changes nothing for unconfigured buckets.

The admin plane now reads the switch through the predicate published by
module_switches.rs instead of its own duplicated env constant; the
behaviour (an environment read per call) is unchanged.

* test(e2e): wire three on-demand migration cases into e2e-smoke

The PR smoke lane gains one case per user-visible contract: a GET miss
that pulls and persists, a HEAD miss that answers from the source and
stores nothing, and the admin config/status pair that must redact the
source secret. The HEAD case did not exist outside the nightly
real-source lane, so it is added to get_basic_test.

Measured on darwin: the lane goes from 168 tests in 101.98 s to 171
tests in 101.92 s, since the three cases overlap the lane's existing
work. The darwin selection digests for e2e-smoke and e2e-full are
regenerated; the e2e-full linux digest still needs a Linux runner.

* docs(changelog): record the on-demand migration feature

.config/e2e-full-selection.txt

@@ -1,2 +1,2 @@
-sha256-darwin=efd7357af1e998134df7d01e0a10d7186898e47abbe86ae6ecb6aef1f8c58357
+sha256-darwin=a45665d370be1d49301599aff8fb3e1ffe349f47fd2e6734f844a850a15d08a6
 sha256-linux=86e69337ad1440252a2ee20a12063c989ed12442d3b1ddf9e9233acf0f2ec089

.config/e2e-smoke-selection.txt

@@ -1 +1 @@
-sha256=db9bd8cdcb0abe43461aa6b36499b17cabd4098e5b34e300b1a0f0d0f34d9884
+sha256=56a9de1c0f000954f1c9ec691e8f24cc2ee387e73cf8b0e684c2baf70317d910

.config/nextest.toml

@@ -394,6 +394,19 @@ test-group = 'ecstore-serial-flaky'
 # rustfs/rustfs#5169 disabled them) have PR-lane signal, not just merge-gate.
 # Single-node servers on random ports with isolated temp dirs — meets the
 # admission criteria unchanged.
+#
+# On-demand migration GA (backlog#2163 ODM-16): three named cases join the
+# lane, one per user-visible contract of the feature — a GET miss that pulls
+# the object and persists it locally, a HEAD miss that answers from the source
+# and stores nothing, and the admin config/status pair that must redact the
+# source secret. Each spawns one single-node rustfs server plus the in-process
+# fake S3 source (`fake_s3_target`, already in the first clause), so they meet
+# the admission criteria unchanged; measured at 15.8 s / 15.8 s / 15.9 s, which
+# is entirely the shared server startup and overlaps the lane's other tests.
+# The rest of `on_demand_migration::{get_basic,interaction,backfill,
+# harness_self}_test` stays in e2e-full and the fault / concurrency /
+# real-source modules stay in e2e-nightly; this is an allowlist, not a module
+# clause, so a new ODM test never lands here silently.
 [profile.e2e-smoke]
 default-filter = """
     package(e2e_test) & (
@@ -401,6 +414,7 @@ default-filter = """
         | test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
         | test(/^reliant::lifecycle::/)
         | test(/^reliant::tiering::/)
+        | test(/^on_demand_migration::(get_basic_test::(get_miss_pulls_inline_and_serves_locally_afterwards|head_miss_answers_from_the_source_without_persisting)|interaction_test::test_odm_admin_config_is_redacted_and_status_counts_match_the_source)$/)
     )
 """
 fail-fast = false

CHANGELOG.md

@@ -12,6 +12,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Per-pool erasure parity**: Erasure parity (STANDARD and reduced-redundancy) is now resolved independently for every pool instead of reusing the first pool's value. A heterogeneous topology — for example a 4-drive pool plus a 2-drive pool created during expansion — previously inherited the first pool's parity and could resolve to zero data shards in the smaller pool, panicking Reed-Solomon construction on write. Automatic parity now resolves per pool (for example `2+2` in the 4-drive pool and `1+1` in the 2-drive pool). Fixes #4801.
 
 ### Added
+- **On-Demand Migration**: Lazy, pull-style migration of an existing S3-compatible bucket into RustFS. A local bucket is attached to an external source bucket; a GET for a key that does not exist locally fetches it from the source, streams it to the client, and stores it locally in the same pass, so every later read is served locally. The module is on by default; set `RUSTFS_ON_DEMAND_MIGRATION_ENABLED=false` on every node to turn it off. A bucket with no source configured behaves exactly as before — the runtime never intervenes on its reads and makes no outbound call. Operator guide at `docs/operations/on-demand-migration.md`.
+  - Per-bucket configuration persisted as `on-demand-migration.json` in the bucket metadata: source provider (`s3`, `aws`, `minio`, `rustfs`, `r2`, `gcs`), endpoint, region, addressing style, credentials and TLS material, an optional key-prefix filter and source-prefix rewrite, and a policy block covering the inline size threshold, multipart part size, concurrency, queue capacity, timeouts, bandwidth limit and negative-cache TTL
+  - Admin routes under `/rustfs/admin/v3/on-demand-migration/{bucket}`: `PUT` (with `?dry-run=true` to validate and probe the source without saving), `GET`, `DELETE`, `GET .../status`, plus `POST .../backfill?op=start|cancel` and `GET .../backfill` for the background full-backfill job with its resumable checkpoint. Authorized by the new `admin:GetBucketOnDemandMigration` and `admin:SetBucketOnDemandMigration` actions; every response redacts `secret_key` and `session_token`
+  - Read paths: an object at or below `policy.inline_max_bytes` (16 MiB by default) is teed to the client and to the local store in a single source read; a larger object or a Range read streams through and a background pull stores the whole object. A HEAD miss is proxied to the source and stores nothing (`policy.head = local_only` disables it). Every source-backed response carries `x-rustfs-on-demand-migration: source`
+  - Protections: a per-source circuit breaker, a per-key negative cache, singleflight per key, a concurrency limit and a bounded pull queue shared by the inline and background paths, an optional bandwidth limit, an anti-loop request marker, and the shared outbound-endpoint (SSRF) policy
+  - Metrics under `rustfs_on_demand_migration_*` (`requests_total`, `pulled_bytes_total`, `pulled_objects_total`, `pull_failures_total`, `inflight_pulls`, `queue_depth`, `source_latency_seconds_*`, `breaker_state`), mirrored per node by the admin status route
+  - Limitations: listings show only local objects (the source is not merged into `ListObjectsV2`); PUT and DELETE never reach the source; a source object updated after it was pulled is not re-fetched; SSE-C source objects are unsupported and answer 424; `Last-Modified` on a pulled object is the local write time, with the source timestamp kept in metadata
 - **NATS JetStream Publish Path**: Opt-in at-least-once delivery for the NATS notify and audit targets. A NATS Core publish flushes to the connection without awaiting a broker acknowledgement, so an event can be lost across a broker restart or a reconnect after the send queue has already cleared it. A queued event now clears only after the JetStream `PublishAck`, so bucket notifications survive those interruptions. Off by default and byte-identical to the NATS Core path when disabled.
   - Three configuration keys per target: `JETSTREAM_ENABLE`, `JETSTREAM_STREAM_NAME`, and `JETSTREAM_ACK_TIMEOUT_SECS`, under the `RUSTFS_NOTIFY_NATS_` and `RUSTFS_AUDIT_NATS_` prefixes
   - Durable store-and-forward with a stable dedup id sent as the `Nats-Msg-Id` header, so a replay after a crash is collapsed by the server duplicate window

crates/e2e_test/src/on_demand_migration/get_basic_test.rs

@@ -12,9 +12,10 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-//! Basic GET read-through scenarios (rustfs/backlog#2156): inline pull and
+//! Basic read-through scenarios (rustfs/backlog#2156): inline pull and
 //! local persistence, large-object passthrough with background backfill,
-//! Range passthrough, source 404, `versionId` reads, and a disabled bucket.
+//! Range passthrough, source 404, `versionId` reads, a disabled bucket, and
+//! the HEAD passthrough that stores nothing (rustfs/backlog#2155).
 //! Every source-side expectation is asserted on the fake source's journal.
 
 use super::common::{BoxError, OdmSourceSpec, OdmTestEnv, SeedObject};
@@ -263,3 +264,35 @@ async fn get_after_disable_does_not_consult_the_source() -> TestResult {
     env.assert_local_absent(bucket, key).await;
     Ok(())
 }
+
+/// A HEAD miss is answered from the source but must not store anything: the
+/// key stays absent locally, so a second HEAD consults the source again. This
+/// is the smoke-lane guard for the HEAD passthrough (rustfs/backlog#2155).
+#[tokio::test]
+async fn head_miss_answers_from_the_source_without_persisting() -> TestResult {
+    let bucket = "odm-head-passthrough";
+    let env = configured_env(bucket, |_| {}).await?;
+    let key = "head/report.bin";
+    let body = payload(32 * 1024);
+    env.seed_source(SOURCE_BUCKET, &[SeedObject::new(key, body.clone())]);
+
+    let head = env.raw_object_request(http::Method::HEAD, bucket, key, &[]).await?;
+    assert_eq!(head.status, 200, "{}", String::from_utf8_lossy(&head.body));
+    assert_eq!(head.header(ODM_RESPONSE_HEADER), Some("source"), "a source answer is marked");
+    assert_eq!(head.header("content-length"), Some(body.len().to_string().as_str()));
+    assert!(head.body.is_empty(), "a HEAD answer carries no body");
+    assert_eq!(env.source.count_requests(Operation::HeadObject, key), 1);
+    assert_eq!(env.source.count_requests(Operation::GetObject, key), 0, "a HEAD must never pull the body");
+    env.assert_local_absent(bucket, key).await;
+
+    let again = env.raw_object_request(http::Method::HEAD, bucket, key, &[]).await?;
+    assert_eq!(again.status, 200, "{}", String::from_utf8_lossy(&again.body));
+    assert_eq!(
+        env.source.count_requests(Operation::HeadObject, key),
+        2,
+        "nothing was written back, so the second HEAD consults the source again"
+    );
+    assert_eq!(env.source.count_requests(Operation::GetObject, key), 0);
+    env.assert_local_absent(bucket, key).await;
+    Ok(())
+}

crates/ecstore/src/bucket/on_demand_migration/sys.rs

@@ -32,7 +32,7 @@
 //! spawns `apply` for installs, and a per-call generation number makes sure
 //! a slower, older install can never overwrite a newer one.
 //!
-//! The module switch (`RUSTFS_ON_DEMAND_MIGRATION_ENABLED`, default off) is
+//! The module switch (`RUSTFS_ON_DEMAND_MIGRATION_ENABLED`, default on) is
 //! injected by the `rustfs` binary through [`OnDemandMigrationSys::set_module_enabled`]
 //! before bucket metadata loads; this crate never reads the environment.
 //! The same startup step injects the [`OdmWriteBack`] the pull pipeline

docs/operations/on-demand-migration.md

@@ -5,7 +5,7 @@
 
 On-Demand Migration (ODM) attaches an external S3-compatible **source bucket** to a local RustFS bucket. When a client GETs a key that does not exist locally, RustFS fetches it from the source, streams it to the client, and stores it locally in the same pass; every later read is served locally. It is a pull-style, lazy migration path — the RustFS equivalent of Cloudflare R2 Sippy, Tigris shadow buckets, and Alibaba Cloud OSS / Tencent COS mirror-back-to-origin.
 
-The module is off by default. Set `RUSTFS_ON_DEMAND_MIGRATION_ENABLED=true` on every node before configuring a bucket (`rustfs/src/module_switches.rs`); with the switch off, the runtime never intervenes on a read and the admin `PUT` route refuses with `OnDemandMigrationDisabled`. Reads of the configuration and of the status endpoint keep working while the switch is off, so a disabled deployment can still be inspected.
+The module is on by default (rustfs/backlog#2163); set `RUSTFS_ON_DEMAND_MIGRATION_ENABLED=false` on every node to turn it off (`rustfs/src/module_switches.rs`). With the switch off, the runtime never intervenes on a read and the admin `PUT` route refuses with `OnDemandMigrationDisabled`. Reads of the configuration and of the status endpoint keep working while the switch is off, so a disabled deployment can still be inspected. The switch only decides whether the module may act at all: a bucket with no `on-demand-migration.json` is never resolved by the runtime and makes no source call, so turning the module on changes nothing for buckets you have not configured.
 
 ## Positioning
 

rustfs/src/admin/handlers/on_demand_migration.rs

@@ -54,6 +54,7 @@ use crate::admin::storage_api::s3::{Body, S3Error, S3ErrorCode, S3Request, S3Res
 use crate::admin::utils::{extract_query_params, read_compatible_admin_body};
 use crate::error::ApiError;
 use crate::license::license_check;
+use crate::module_switches::{ENV_ON_DEMAND_MIGRATION_ENABLED, on_demand_migration_enabled_from_env};
 use crate::server::ADMIN_PREFIX;
 use hyper::{Method, StatusCode};
 use matchit::Params;
@@ -93,13 +94,12 @@ pub(crate) const ERR_CODE_BACKFILL_RUNNING: &str = "OnDemandMigrationBackfillRun
 /// Error code (404) returned when the bucket never had a backfill job.
 pub(crate) const ERR_CODE_NO_SUCH_BACKFILL_JOB: &str = "NoSuchBackfillJob";
 
-/// The published switch is `RUSTFS_ON_DEMAND_MIGRATION_ENABLED`, owned by
-/// ODM-05 in `module_switches.rs`. This is the only read of it in the admin
-/// plane so the orchestrator can swap the call for the published predicate.
-const ENV_ON_DEMAND_MIGRATION_ENABLED: &str = "RUSTFS_ON_DEMAND_MIGRATION_ENABLED";
-
+/// The switch is `RUSTFS_ON_DEMAND_MIGRATION_ENABLED`, owned by ODM-05 in
+/// `module_switches.rs`. The admin plane resolves it from the environment on
+/// every call rather than from the published cell, so an admin request answers
+/// the switch the process was started with even before startup published it.
 fn module_enabled() -> bool {
-    rustfs_utils::get_env_bool(ENV_ON_DEMAND_MIGRATION_ENABLED, false)
+    on_demand_migration_enabled_from_env()
 }
 
 /// What the source answered during `PUT` validation.
@@ -1278,8 +1278,8 @@ mod tests {
     }
 
     #[test]
-    fn module_switch_defaults_off_and_reads_the_env() {
-        temp_env::with_var(ENV_ON_DEMAND_MIGRATION_ENABLED, None::<&str>, || assert!(!module_enabled()));
+    fn module_switch_defaults_on_and_reads_the_env() {
+        temp_env::with_var(ENV_ON_DEMAND_MIGRATION_ENABLED, None::<&str>, || assert!(module_enabled()));
         temp_env::with_var(ENV_ON_DEMAND_MIGRATION_ENABLED, Some("true"), || assert!(module_enabled()));
         temp_env::with_var(ENV_ON_DEMAND_MIGRATION_ENABLED, Some("false"), || assert!(!module_enabled()));
     }

rustfs/src/module_switches.rs

@@ -35,10 +35,11 @@ pub(crate) const ENV_HEAL_ENABLED: &str = "RUSTFS_HEAL_ENABLED";
 pub(crate) const ENV_HEAL_ENABLED_DEPRECATED: &str = "RUSTFS_ENABLE_HEAL";
 pub(crate) const ENV_BITROT_SELFTEST_ENABLE: &str = "RUSTFS_BITROT_SELFTEST_ENABLE";
 pub(crate) const ENV_BITROT_SELFTEST_STRICT: &str = "RUSTFS_BITROT_SELFTEST_STRICT";
-/// On-demand migration module switch (rustfs/backlog#2152). Off until GA
-/// (rustfs/backlog#2163) so every intermediate PR ships dark.
+/// On-demand migration module switch (rustfs/backlog#2152). On since GA
+/// (rustfs/backlog#2163); set it to `false` to keep the module out of the
+/// read path entirely.
 pub(crate) const ENV_ON_DEMAND_MIGRATION_ENABLED: &str = "RUSTFS_ON_DEMAND_MIGRATION_ENABLED";
-pub(crate) const DEFAULT_ON_DEMAND_MIGRATION_ENABLED: bool = false;
+pub(crate) const DEFAULT_ON_DEMAND_MIGRATION_ENABLED: bool = true;
 
 static AUDIT_MODULE_ENABLED: AtomicBool = AtomicBool::new(rustfs_config::DEFAULT_AUDIT_ENABLE);
 static NOTIFY_MODULE_ENABLED: AtomicBool = AtomicBool::new(rustfs_config::DEFAULT_NOTIFY_ENABLE);
@@ -86,7 +87,7 @@ pub(crate) fn set_notify_module_enabled(enabled: bool) {
     NOTIFY_MODULE_ENABLED.store(enabled, Ordering::Relaxed);
 }
 
-/// Whether the on-demand migration module is enabled, defaulting to off.
+/// Whether the on-demand migration module is enabled, defaulting to on.
 /// Read once at startup by `startup_bucket_metadata` and published below.
 pub(crate) fn on_demand_migration_enabled_from_env() -> bool {
     rustfs_utils::get_env_bool(ENV_ON_DEMAND_MIGRATION_ENABLED, DEFAULT_ON_DEMAND_MIGRATION_ENABLED)
@@ -110,15 +111,18 @@ mod tests {
     use super::*;
 
     #[test]
-    fn on_demand_migration_switch_defaults_off_and_follows_env() {
+    fn on_demand_migration_switch_defaults_on_and_follows_env() {
         temp_env::with_var(ENV_ON_DEMAND_MIGRATION_ENABLED, None::<&str>, || {
-            assert!(!on_demand_migration_enabled_from_env());
+            assert!(on_demand_migration_enabled_from_env());
+        });
+        temp_env::with_var(ENV_ON_DEMAND_MIGRATION_ENABLED, Some("false"), || {
+            assert!(!on_demand_migration_enabled_from_env(), "the off switch still works");
         });
         temp_env::with_var(ENV_ON_DEMAND_MIGRATION_ENABLED, Some("true"), || {
             assert!(on_demand_migration_enabled_from_env());
         });
         temp_env::with_var(ENV_ON_DEMAND_MIGRATION_ENABLED, Some("not-a-bool"), || {
-            assert!(!on_demand_migration_enabled_from_env(), "unparsable values keep the default");
+            assert!(on_demand_migration_enabled_from_env(), "unparsable values keep the default");
         });
     }