mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-17 10:17:55 +00:00
fix(admin): serialize replication metrics in minio-go wire shapes (#6127)
* test(admin): pin minio-go Metrics/MetricsV2 wire contract for replication metrics Red-light evidence for backlog#1675 P1-11: ?replication-metrics[=2] serializes the internal snake_case BucketStats family straight onto the wire, while minio-go's replication.Metrics/MetricsV2 expect camelCase tags (currStats/queueStats/replicaCount/queued/...). Go's decoder is case-insensitive but does not ignore underscores, so 'mc replicate status' shows all zeros without any error. The rewritten snapshot tests assert the minio-go tags (plus a synthesized queueStats node — the aggregation path leaves queue_stats.nodes empty today) and fail against the current pass-through serialization. * fix(admin): serialize replication metrics in minio-go wire shapes ?replication-metrics[=2] and the admin replicationmetrics endpoint serialized the internal snake_case BucketStats family straight onto the wire, so 'mc replicate status' decoded all zeros without any error (backlog#1675 P1-11). The internal structs cannot be renamed: they are the intra-cluster peer-RPC wire format (rmp_serde to_vec_named in node_service.rs), pinned by a new regression test. - New admin/replication_metrics_wire.rs: Serialize-only projections onto minio-go replication.Metrics (v1 body, currStats) and MetricsV2 (uptime/currStats/queueStats/downtimeInfo) with the exact json tags; per-target failed becomes the TimedErrStats envelope fed from the FailStats rolling window; the queue peak is dual-emitted as max (MinIO server tag) and peak (minio-go tag). - queueStats synthesizes one node from the bucket queue snapshot — the aggregation path leaves queue_stats.nodes empty, and mc treats an empty node list as 'no data' — and carries transfer summaries (Large/Small/Total) derived from the per-target xfer rates. - Both endpoints share the DTOs; source-health extension keys (provider_available/cluster_complete/...) ride along and are ignored by Go decoders. - Widen the ecstore replication_stats_boundary re-exports (BucketReplicationStat/InQueueMetric/XferStats) so the admin facade chain can name the projected types. * fix(replication): carry failure rolling windows through cluster aggregation Review: both metrics endpoints aggregate first, and FailStats::merge dropped the process-local samples (which also never cross the peer-RPC wire — serde-skipped), so lastMinute/lastHour serialized as zero right after a failure while totals was nonzero. - FailStats gains serializable last_minute/last_hour window snapshots (serde default: old nodes read zeros, new fields are ignored by old decoders), recomputed on every add_size and re-stamped at the per-node collection point (get_latest_replication_stats), and summed by merge. - The wire DTO takes the component-wise max of the live samples and the snapshot, so both the single-node and the aggregated path report the window. - Regression test drives a stat through rmp round trip + merge before serialization, as requested. Also restore the #[allow(dead_code)] attribute to route_policy — the new module declaration had been inserted between the attribute and its item, which broke the -D warnings CI lanes. * fix(replication): bin transfer summaries at 128 MiB and keep window refresh off the hot path Second review round: - update_xfer_rate split at 1 MiB while the minio-go transferSummary labels (and RustFS's own worker-pool split) mean >= 128 MiB for Large, so a 2 MiB replication reported under Large with Small stuck at zero. The producer now bins on MIN_LARGE_OBJ_SIZE; a MetricsV2 assertion covers 2 MiB / 127 MiB / exactly 128 MiB. - add_size no longer recomputes the rolling windows: two full one-hour-deque scans per failure under the bucket-stats write lock made failure bursts quadratic (30k events ~2.1s). The windows are stamped only at the collection point (get_latest_replication_stats, which serves both the local leg and the peer RPC); the aggregation regression now drives that path explicitly before the RPC round trip and merge. * fix(replication): average transfer summaries --------- Co-authored-by: overtrue <anzhengchao@gmail.com>
This commit is contained in:
@@ -520,6 +520,14 @@ struct FailureSample {
|
||||
pub struct FailStats {
|
||||
pub count: i64,
|
||||
pub size: i64,
|
||||
/// Rolling-window snapshots refreshed at collection time
|
||||
/// ([`Self::refresh_windows`]). The raw samples (`recent`) are process
|
||||
/// local (serde-skipped), so these fields are what survives the peer-RPC
|
||||
/// wire and [`Self::merge`]-based cluster aggregation.
|
||||
#[serde(default)]
|
||||
pub last_minute: FailedMetric,
|
||||
#[serde(default)]
|
||||
pub last_hour: FailedMetric,
|
||||
#[serde(skip)]
|
||||
recent: VecDeque<FailureSample>,
|
||||
}
|
||||
@@ -537,6 +545,17 @@ impl FailStats {
|
||||
self.prune(observed_at);
|
||||
}
|
||||
|
||||
/// Recompute the serializable rolling-window snapshots from the local
|
||||
/// samples. Called at the collection point (per-node stats snapshot),
|
||||
/// never on the failure hot path — the two deque scans are O(window) and
|
||||
/// `add_size` runs under the bucket-stats write lock. Only meaningful on
|
||||
/// the live per-node struct: a deserialized or merged struct has no
|
||||
/// samples, and refreshing it would wipe the aggregated windows.
|
||||
pub fn refresh_windows(&mut self) {
|
||||
self.last_minute = self.recent_since(Duration::from_secs(60));
|
||||
self.last_hour = self.recent_since(Duration::from_secs(3600));
|
||||
}
|
||||
|
||||
fn prune(&mut self, observed_at: Instant) {
|
||||
while self
|
||||
.recent
|
||||
@@ -565,6 +584,16 @@ impl FailStats {
|
||||
Self {
|
||||
count: self.count.saturating_add(other.count),
|
||||
size: self.size.saturating_add(other.size),
|
||||
// The window snapshots sum across nodes; the raw samples do not
|
||||
// travel and stay empty on aggregated structs.
|
||||
last_minute: FailedMetric {
|
||||
count: self.last_minute.count.saturating_add(other.last_minute.count),
|
||||
size: self.last_minute.size.saturating_add(other.last_minute.size),
|
||||
},
|
||||
last_hour: FailedMetric {
|
||||
count: self.last_hour.count.saturating_add(other.last_hour.count),
|
||||
size: self.last_hour.size.saturating_add(other.last_hour.size),
|
||||
},
|
||||
recent: VecDeque::new(),
|
||||
}
|
||||
}
|
||||
@@ -636,7 +665,9 @@ impl BucketReplicationStat {
|
||||
}
|
||||
|
||||
pub fn update_xfer_rate(&mut self, size: i64, duration: Duration) {
|
||||
if size > 1024 * 1024 {
|
||||
// Same boundary as the worker-pool split and minio-go's
|
||||
// Large/Small transfer-summary labels: >= 128 MiB is "large".
|
||||
if size >= crate::runtime::MIN_LARGE_OBJ_SIZE {
|
||||
self.xfer_rate_lrg.add_size(size, duration);
|
||||
} else {
|
||||
self.xfer_rate_sml.add_size(size, duration);
|
||||
|
||||
Reference in New Issue
Block a user