feat(get): harden codec streaming rollout (#3981)

* feat(get): consolidate GET performance optimization

Consolidated implementation of all GET performance optimizations into
a single, well-organized commit replacing the previous patch-on-patch
approach.

## Changes

### Configuration (set_disk/mod.rs)
- Consolidated all GET optimization flags into a single organized section
- Enabled by default: codec streaming, metadata early-stop, page cache reclaim
- Added codec streaming multipart flag (default: disabled)
- Added version-aware early-stop flag (default: disabled)
- Added adaptive duplex buffer sizing based on object size
- All flags use OnceLock caching with rollout percentage support

### Metadata Early-Stop (set_disk/read.rs)
- Delete marker early-stop when quorum agrees
- Version-aware early-stop for versioned GET requests
- MetadataQuorumAccumulator enhanced with:
  - delete_marker_votes tracking
  - requested_version_id and matching_version_votes tracking
  - version_early_stop_decision() method
- 6 new tests for version early-stop scenarios

### Codec Streaming (erasure/coding/decode_reader.rs)
- DualInFlight (2-stripe lookahead) enabled by default

### Decode Pipeline (erasure/coding/decode.rs)
- Stripe prefetch count configuration
- Bitrot-decode overlap configuration

### Disk Layer (disk/local.rs)
- O_DIRECT read configuration constants (preparation)

### Metrics (io-metrics/lib.rs)
- BytesPool acquisition/return metrics
- Metadata phase duration with early-stop label
- Total duration with reader_path label

### Diagnostics (diagnostics/)
- Early-stop reason constants
- Pool tier/outcome label constants

### Observability (.docker/observability/)
- 3 Grafana dashboards for GET optimization monitoring
- Prometheus alert rules (6 alerts: 3 critical, 3 warning)
- Updated README.md and README_ZH.md with usage docs

### Config (config/src/constants/runtime.rs)
- Page cache reclaim read enabled by default

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| RUSTFS_GET_CODEC_STREAMING_ENABLE | true | Codec streaming base flag |
| RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT | 100 | Codec streaming rollout % |
| RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE | false | Multipart codec streaming |
| RUSTFS_GET_METADATA_EARLY_STOP_ENABLE | true | Early-stop base flag |
| RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT | 100 | Early-stop rollout % |
| RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE | false | Version-aware early-stop |
| RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE | true | Page cache reclaim |
| RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE | false | O_DIRECT (preparation) |
| RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT | 1 | Stripe prefetch |
| RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE | false | Bitrot-decode overlap |
| RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT | 2 | DualInFlight stripes |

## Rollback

All optimizations can be disabled via environment variables:
RUSTFS_GET_CODEC_STREAMING_ENABLE=false
RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=false
RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=false

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(get): add stress test scripts for GET optimization validation

- quick-validate-get-optimization.sh: Quick 5-minute validation
- stress-test-get-optimization.sh: Full 30+ minute stress test
- README-stress-test.md: Usage documentation

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ecstore): align file cache reclaim defaults

* chore(deps): update redis and erasure codec

* test(ecstore): align decode fill policy default

* fix(get): wire codec streaming rollout gate

* perf(get): skip metrics-off codec timers

* test(get): capture codec streaming diagnostics

* test(get): add multipart fallback probe

* test(get): add encrypted fallback probe

* test(get): add compressed fallback probe

* test(get): add degraded read fallback probe

* test(get): cover remote fallback probe

* test(get): report warp request p99

* test(get): capture OTLP metric deltas

* perf(get): align codec streaming inflight default

* perf(get): reuse codec reader output buffers

* test(get): count codec reader fill starts

* perf(get): reuse codec reader fill worker

* perf(get): lazy init rustfs codec reconstruct

* test(get): cover rustfs codec source faults

* docs(get): record rustfs codec fallback scope

* feat(get): add multipart codec reader opt-in

* test(get): add multipart codec smoke option

* test(get): cover multipart codec degraded fallback

* perf(get): bound multipart codec eager setup

* test(get): satisfy codec hardening PR gate

---------

Co-authored-by: heihutu <heihutu@gmail.com>

rustfs/src/app/lifecycle_transition_api_test.rs

@@ -61,6 +61,11 @@ use uuid::Uuid;
 static GLOBAL_ENV: OnceLock<(Vec<PathBuf>, Arc<ECStore>)> = OnceLock::new();
 static INIT: Once = Once::new();
 const TRANSITION_WAIT_TIMEOUT: Duration = Duration::from_secs(15);
+const ENV_GET_CODEC_STREAMING_ENABLE: &str = "RUSTFS_GET_CODEC_STREAMING_ENABLE";
+const ENV_GET_CODEC_STREAMING_ROLLOUT: &str = "RUSTFS_GET_CODEC_STREAMING_ROLLOUT";
+const ENV_GET_CODEC_STREAMING_BODY_COMPAT_CONFIRMED: &str = "RUSTFS_GET_CODEC_STREAMING_BODY_COMPAT_CONFIRMED";
+const ENV_GET_CODEC_STREAMING_HEADER_COMPAT_CONFIRMED: &str = "RUSTFS_GET_CODEC_STREAMING_HEADER_COMPAT_CONFIRMED";
+const ENV_GET_CODEC_STREAMING_MIN_SIZE: &str = "RUSTFS_GET_CODEC_STREAMING_MIN_SIZE";
 
 fn init_tracing() {
     INIT.call_once(|| {});
@@ -405,6 +410,31 @@ where
     }
 }
 
+async fn with_get_codec_streaming_remote_probe_env<F, Fut>(test_fn: F)
+where
+    F: FnOnce() -> Fut,
+    Fut: std::future::Future<Output = ()>,
+{
+    let metrics_was_enabled = rustfs_io_metrics::get_stage_metrics_enabled();
+    rustfs_io_metrics::set_get_stage_metrics_enabled(true);
+    let result = std::panic::AssertUnwindSafe(temp_env::async_with_vars(
+        [
+            (ENV_GET_CODEC_STREAMING_ENABLE, Some("true")),
+            (ENV_GET_CODEC_STREAMING_ROLLOUT, Some("benchmark")),
+            (ENV_GET_CODEC_STREAMING_BODY_COMPAT_CONFIRMED, Some("true")),
+            (ENV_GET_CODEC_STREAMING_HEADER_COMPAT_CONFIRMED, Some("true")),
+            (ENV_GET_CODEC_STREAMING_MIN_SIZE, Some("1")),
+        ],
+        test_fn(),
+    ))
+    .catch_unwind()
+    .await;
+    rustfs_io_metrics::set_get_stage_metrics_enabled(metrics_was_enabled);
+    if let Err(err) = result {
+        std::panic::resume_unwind(err);
+    }
+}
+
 async fn wait_for_remote_absence(backend: &MockWarmBackend, object: &str, timeout: Duration) -> bool {
     let deadline = tokio::time::Instant::now() + timeout;
 
@@ -911,6 +941,66 @@ async fn complete_multipart_upload_transitions_immediately_via_usecase() {
     assert!(backend.objects.lock().await.contains_key(&info.transitioned_object.name));
 }
 
+#[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+#[serial]
+#[ignore = "requires isolated global object layer state"]
+async fn get_transitioned_object_uses_remote_codec_fallback_path() {
+    with_get_codec_streaming_remote_probe_env(|| async {
+        let (_disk_paths, ecstore) = setup_test_env().await;
+
+        let tier_name = format!("COLDTIER{}", &Uuid::new_v4().simple().to_string()[..8]).to_uppercase();
+        let backend = register_mock_tier(&tier_name).await;
+
+        let bucket = format!("test-api-get-remote-{}", &Uuid::new_v4().simple().to_string()[..8]);
+        let object = "test/remote-codec-fallback.txt";
+        let payload: Vec<u8> = (0..(1024 * 1024))
+            .map(|index| u8::try_from(index % 251).expect("payload byte fits in u8"))
+            .collect();
+
+        create_test_bucket(&ecstore, bucket.as_str()).await;
+        set_bucket_lifecycle_transition_with_tier(bucket.as_str(), &tier_name)
+            .await
+            .expect("Failed to set lifecycle configuration");
+
+        let uploaded = upload_test_object(&ecstore, bucket.as_str(), object, &payload).await;
+        let transition_opts = ObjectOptions {
+            transition: lifecycle::lifecycle_contract::TransitionOptions {
+                status: lifecycle::lifecycle_contract::TRANSITION_PENDING.to_string(),
+                tier: tier_name.clone(),
+                etag: uploaded.etag.clone().unwrap_or_default(),
+                ..Default::default()
+            },
+            version_id: uploaded.version_id.map(|version| version.to_string()),
+            versioned: true,
+            mod_time: uploaded.mod_time,
+            ..Default::default()
+        };
+        ecstore
+            .transition_object(bucket.as_str(), object, &transition_opts)
+            .await
+            .expect("Failed to transition object directly");
+
+        let transitioned = wait_for_transition(&ecstore, bucket.as_str(), object, TRANSITION_WAIT_TIMEOUT)
+            .await
+            .expect("object should transition before remote fallback GET");
+
+        assert_eq!(transitioned.transitioned_object.status, "complete");
+        assert_eq!(transitioned.transitioned_object.tier, tier_name);
+        assert!(!transitioned.transitioned_object.name.is_empty());
+        assert!(
+            backend
+                .objects
+                .lock()
+                .await
+                .contains_key(&transitioned.transitioned_object.name)
+        );
+
+        let actual = read_object_bytes(&ecstore, bucket.as_str(), object).await;
+        assert_eq!(actual, payload);
+    })
+    .await;
+}
+
 #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
 #[serial]
 #[ignore = "requires isolated global object layer state"]

rustfs/src/app/storage_api.rs

@@ -382,7 +382,9 @@ pub(crate) mod bucket {
 
             pub(crate) const TRANSITION_COMPLETE: &str =
                 crate::storage::storage_api::ecstore_bucket::lifecycle::lifecycle::TRANSITION_COMPLETE;
-
+            #[cfg(test)]
+            pub(crate) const TRANSITION_PENDING: &str =
+                crate::storage::storage_api::ecstore_bucket::lifecycle::lifecycle::TRANSITION_PENDING;
             pub(crate) fn expected_expiry_time(mod_time: time::OffsetDateTime, days: i32) -> time::OffsetDateTime {
                 crate::storage::storage_api::ecstore_bucket::lifecycle::lifecycle::expected_expiry_time(mod_time, days)
             }