fix(admin): serialize replication metrics in minio-go wire shapes (#6127)

* test(admin): pin minio-go Metrics/MetricsV2 wire contract for replication metrics

Red-light evidence for backlog#1675 P1-11: ?replication-metrics[=2]
serializes the internal snake_case BucketStats family straight onto the
wire, while minio-go's replication.Metrics/MetricsV2 expect camelCase
tags (currStats/queueStats/replicaCount/queued/...). Go's decoder is
case-insensitive but does not ignore underscores, so 'mc replicate
status' shows all zeros without any error. The rewritten snapshot tests
assert the minio-go tags (plus a synthesized queueStats node — the
aggregation path leaves queue_stats.nodes empty today) and fail against
the current pass-through serialization.

* fix(admin): serialize replication metrics in minio-go wire shapes

?replication-metrics[=2] and the admin replicationmetrics endpoint
serialized the internal snake_case BucketStats family straight onto the
wire, so 'mc replicate status' decoded all zeros without any error
(backlog#1675 P1-11). The internal structs cannot be renamed: they are
the intra-cluster peer-RPC wire format (rmp_serde to_vec_named in
node_service.rs), pinned by a new regression test.

- New admin/replication_metrics_wire.rs: Serialize-only projections onto
  minio-go replication.Metrics (v1 body, currStats) and MetricsV2
  (uptime/currStats/queueStats/downtimeInfo) with the exact json tags;
  per-target failed becomes the TimedErrStats envelope fed from the
  FailStats rolling window; the queue peak is dual-emitted as max
  (MinIO server tag) and peak (minio-go tag).
- queueStats synthesizes one node from the bucket queue snapshot — the
  aggregation path leaves queue_stats.nodes empty, and mc treats an
  empty node list as 'no data' — and carries transfer summaries
  (Large/Small/Total) derived from the per-target xfer rates.
- Both endpoints share the DTOs; source-health extension keys
  (provider_available/cluster_complete/...) ride along and are ignored
  by Go decoders.
- Widen the ecstore replication_stats_boundary re-exports
  (BucketReplicationStat/InQueueMetric/XferStats) so the admin facade
  chain can name the projected types.

* fix(replication): carry failure rolling windows through cluster aggregation

Review: both metrics endpoints aggregate first, and FailStats::merge
dropped the process-local samples (which also never cross the peer-RPC
wire — serde-skipped), so lastMinute/lastHour serialized as zero right
after a failure while totals was nonzero.

- FailStats gains serializable last_minute/last_hour window snapshots
  (serde default: old nodes read zeros, new fields are ignored by old
  decoders), recomputed on every add_size and re-stamped at the
  per-node collection point (get_latest_replication_stats), and summed
  by merge.
- The wire DTO takes the component-wise max of the live samples and the
  snapshot, so both the single-node and the aggregated path report the
  window.
- Regression test drives a stat through rmp round trip + merge before
  serialization, as requested.

Also restore the #[allow(dead_code)] attribute to route_policy — the
new module declaration had been inserted between the attribute and its
item, which broke the -D warnings CI lanes.

* fix(replication): bin transfer summaries at 128 MiB and keep window refresh off the hot path

Second review round:

- update_xfer_rate split at 1 MiB while the minio-go transferSummary
  labels (and RustFS's own worker-pool split) mean >= 128 MiB for
  Large, so a 2 MiB replication reported under Large with Small stuck
  at zero. The producer now bins on MIN_LARGE_OBJ_SIZE; a MetricsV2
  assertion covers 2 MiB / 127 MiB / exactly 128 MiB.
- add_size no longer recomputes the rolling windows: two full
  one-hour-deque scans per failure under the bucket-stats write lock
  made failure bursts quadratic (30k events ~2.1s). The windows are
  stamped only at the collection point (get_latest_replication_stats,
  which serves both the local leg and the peer RPC); the aggregation
  regression now drives that path explicitly before the RPC round trip
  and merge.

* fix(replication): average transfer summaries

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>

crates/ecstore/src/api/mod.rs

@@ -184,17 +184,18 @@ pub mod bucket {
             mrf_backlog_observability_snapshot,
         };
         pub use crate::bucket::replication::{
-            BucketReplicationResyncStatus, BucketReplicationStats, BucketStats, DeleteReplicationConfigSnapshot,
-            DeletedObjectReplicationInfo, DurableMrfBacklog, DynReplicationPool, MrfOpKind, MrfReplicateEntry,
-            MustReplicateOptions, ObjectOpts, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION, REMOTE_TARGET_UNSUPPORTED_FIELDS,
-            REMOTE_TARGET_WRITABLE_FIELDS, REPLICATE_INCOMING_DELETE, REPLICATION_CAPABILITY_CONTRACT_VERSION,
-            REPLICATION_READ_ONLY_HISTORICAL_FIELDS, REPLICATION_WRITABLE_FIELDS, ReplicateDecision, ReplicateObjectInfo,
-            ReplicationBatchAdmission, ReplicationConfig, ReplicationConfigStructureError, ReplicationConfigurationExt,
-            ReplicationDeleteScheduleInput, ReplicationDeleteStateSource, ReplicationHealQueueResult, ReplicationObjectBridge,
-            ReplicationObjectIO, ReplicationOperation, ReplicationPoolTrait, ReplicationPriority, ReplicationQueueAdmission,
-            ReplicationScannerBridge, ReplicationState, ReplicationStats, ReplicationStatusType, ReplicationStorage,
-            ReplicationTargetValidationError, ReplicationType, ResyncOpts, ResyncStatusType, RuntimeReplicationTargetBacklog,
-            TargetReplicationResyncStatus, VersionPurgeStatusType, commit_force_delete_intent, complete_force_delete_intent,
+            BucketReplicationResyncStatus, BucketReplicationStat, BucketReplicationStats, BucketStats,
+            DeleteReplicationConfigSnapshot, DeletedObjectReplicationInfo, DurableMrfBacklog, DynReplicationPool, InQueueMetric,
+            MrfOpKind, MrfReplicateEntry, MustReplicateOptions, ObjectOpts, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION,
+            REMOTE_TARGET_UNSUPPORTED_FIELDS, REMOTE_TARGET_WRITABLE_FIELDS, REPLICATE_INCOMING_DELETE,
+            REPLICATION_CAPABILITY_CONTRACT_VERSION, REPLICATION_READ_ONLY_HISTORICAL_FIELDS, REPLICATION_WRITABLE_FIELDS,
+            ReplicateDecision, ReplicateObjectInfo, ReplicationBatchAdmission, ReplicationConfig,
+            ReplicationConfigStructureError, ReplicationConfigurationExt, ReplicationDeleteScheduleInput,
+            ReplicationDeleteStateSource, ReplicationHealQueueResult, ReplicationObjectBridge, ReplicationObjectIO,
+            ReplicationOperation, ReplicationPoolTrait, ReplicationPriority, ReplicationQueueAdmission, ReplicationScannerBridge,
+            ReplicationState, ReplicationStats, ReplicationStatusType, ReplicationStorage, ReplicationTargetValidationError,
+            ReplicationType, ResyncOpts, ResyncStatusType, RuntimeReplicationTargetBacklog, TargetReplicationResyncStatus,
+            VersionPurgeStatusType, XferStats, commit_force_delete_intent, complete_force_delete_intent,
             delete_replication_state_from_config, delete_replication_version_id, get_global_replication_pool,
             get_global_replication_stats, init_background_replication, invalid_replication_config_status_field,
             persist_force_delete_intent, read_durable_mrf_backlog, replication_state_to_filemeta, replication_status_to_filemeta,

crates/ecstore/src/bucket/replication/mod.rs

@@ -81,6 +81,6 @@ pub use replication_queue_boundary::{
 pub use replication_resync_boundary::{BucketReplicationResyncStatus, ResyncOpts, TargetReplicationResyncStatus};
 pub use replication_scanner_bridge::ReplicationScannerBridge;
 pub use replication_state::{ReplicationStats, RuntimeReplicationTargetBacklog};
-pub use replication_stats_boundary::{BucketReplicationStats, BucketStats};
+pub use replication_stats_boundary::{BucketReplicationStat, BucketReplicationStats, BucketStats, InQueueMetric, XferStats};
 pub use replication_storage_boundary::{ReplicationObjectIO, ReplicationStorage};
 pub(crate) use replication_target_config_bridge::ReplicationTargetConfigBridge;

crates/ecstore/src/bucket/replication/replication_state.rs

@@ -704,6 +704,12 @@ impl ReplicationStats {
         } else {
             BucketReplicationStats::new()
         };
+        // Stamp the serializable failure windows from the live samples: the
+        // samples themselves do not cross the peer-RPC wire, so this snapshot
+        // is what cluster aggregation and the metrics endpoints see.
+        for stat in replication_stats.stats.values_mut() {
+            stat.fail_stats.refresh_windows();
+        }
         let uptime = if cache.contains_key(bucket) {
             SystemTime::now()
                 .duration_since(SystemTime::UNIX_EPOCH)

crates/ecstore/src/bucket/replication/replication_stats_boundary.rs

@@ -15,7 +15,9 @@
 #[cfg(test)]
 pub(crate) use rustfs_replication::FailStats;
 pub(crate) use rustfs_replication::{
-    ActiveWorkerStat, BucketReplicationStat, InQueueMetric, ProxyMetric, ProxyStatsCache, QueueCache, ReplicationMetricScope,
-    SRMetricsSummary, XferStats,
+    ActiveWorkerStat, ProxyMetric, ProxyStatsCache, QueueCache, ReplicationMetricScope, SRMetricsSummary,
 };
-pub use rustfs_replication::{BucketReplicationStats, BucketStats};
+// Public so the admin wire DTOs (rustfs/src/admin/replication_metrics_wire.rs)
+// can project the internal stats onto the minio-go response shapes through
+// the storage_api facade chain.
+pub use rustfs_replication::{BucketReplicationStat, BucketReplicationStats, BucketStats, InQueueMetric, XferStats};

crates/replication/src/stats.rs

@@ -520,6 +520,14 @@ struct FailureSample {
 pub struct FailStats {
     pub count: i64,
     pub size: i64,
+    /// Rolling-window snapshots refreshed at collection time
+    /// ([`Self::refresh_windows`]). The raw samples (`recent`) are process
+    /// local (serde-skipped), so these fields are what survives the peer-RPC
+    /// wire and [`Self::merge`]-based cluster aggregation.
+    #[serde(default)]
+    pub last_minute: FailedMetric,
+    #[serde(default)]
+    pub last_hour: FailedMetric,
     #[serde(skip)]
     recent: VecDeque<FailureSample>,
 }
@@ -537,6 +545,17 @@ impl FailStats {
         self.prune(observed_at);
     }
 
+    /// Recompute the serializable rolling-window snapshots from the local
+    /// samples. Called at the collection point (per-node stats snapshot),
+    /// never on the failure hot path — the two deque scans are O(window) and
+    /// `add_size` runs under the bucket-stats write lock. Only meaningful on
+    /// the live per-node struct: a deserialized or merged struct has no
+    /// samples, and refreshing it would wipe the aggregated windows.
+    pub fn refresh_windows(&mut self) {
+        self.last_minute = self.recent_since(Duration::from_secs(60));
+        self.last_hour = self.recent_since(Duration::from_secs(3600));
+    }
+
     fn prune(&mut self, observed_at: Instant) {
         while self
             .recent
@@ -565,6 +584,16 @@ impl FailStats {
         Self {
             count: self.count.saturating_add(other.count),
             size: self.size.saturating_add(other.size),
+            // The window snapshots sum across nodes; the raw samples do not
+            // travel and stay empty on aggregated structs.
+            last_minute: FailedMetric {
+                count: self.last_minute.count.saturating_add(other.last_minute.count),
+                size: self.last_minute.size.saturating_add(other.last_minute.size),
+            },
+            last_hour: FailedMetric {
+                count: self.last_hour.count.saturating_add(other.last_hour.count),
+                size: self.last_hour.size.saturating_add(other.last_hour.size),
+            },
             recent: VecDeque::new(),
         }
     }
@@ -636,7 +665,9 @@ impl BucketReplicationStat {
     }
 
     pub fn update_xfer_rate(&mut self, size: i64, duration: Duration) {
-        if size > 1024 * 1024 {
+        // Same boundary as the worker-pool split and minio-go's
+        // Large/Small transfer-summary labels: >= 128 MiB is "large".
+        if size >= crate::runtime::MIN_LARGE_OBJ_SIZE {
             self.xfer_rate_lrg.add_size(size, duration);
         } else {
             self.xfer_rate_sml.add_size(size, duration);