test(internode): extend fallback and transition coverage (#5232)

Add decode-error metrics for internode msgpack/json compatibility paths, extend the mixed fallback e2e assertions for transitioned multipart partNumber reads, and cover manual transition async status polling plus inactive-owner status behavior.

Co-authored-by: heihutu <heihutu@gmail.com>

docs/operations/internode-grpc-benchmark-runbook.md

@@ -50,6 +50,7 @@ gitignored — attach the paired directories to the PR / issue.
 | `rustfs_system_network_internode_operation_large_payloads_total` | large unary RPCs sharing the channel (P1 target) |
 | `rustfs_system_network_internode_dial_avg_time_nanos`, `..._dial_errors_total` | connect cost (P3 prewarm) |
 | `rustfs_system_network_internode_msgpack_json_fallback_total{direction,message}` | must be **0** before enabling both msgpack-only gates (P2) |
+| `rustfs_system_network_internode_msgpack_json_decode_error_total{direction,message,codec}` | must be **0** before enabling both msgpack-only gates (P2) |
 | `rustfs_cluster_servers_offline_total` | offline detection correctness (P3 bypass) |
 | lock p99 (lock metrics) | P1 head-of-line-blocking win |
 
@@ -86,10 +87,7 @@ column, everything else at defaults. Roll a restart between runs.
   `>4 MiB` multi-version `xl.meta` no longer fails `out_of_range`.
 - **P1** — mixed workload (large `ReadAll` + high-frequency `Refresh`). Acceptance gate from
   the design doc: **lock p99 down ≥ 20%** with `RUSTFS_INTERNODE_CHANNEL_ISOLATION=true`.
-- **P2** — observe `msgpack_json_fallback_total` across a release window; it must stay **0**
-  before flipping both `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true` and
-  `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=true` (see the msgpack convergence
-  runbook). Codec allocation via a `dhat`/`heaptrack` micro-run.
+- **P2** — observe `msgpack_json_fallback_total` and `msgpack_json_decode_error_total` across a release window; both must stay **0** before flipping both `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true` and `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=true` (see the msgpack convergence runbook). Codec allocation via a `dhat`/`heaptrack` micro-run.
 - **P3** — cold-start: first cross-node op latency should drop ~one connect RTT with prewarm.
   Failover: `run_four_node_cluster_failover_bench.sh`, kill a node with
   `RUSTFS_INTERNODE_OFFLINE_BYPASS=true`; expect faster failover and a correct

docs/operations/internode-msgpack-json-convergence-runbook.md

@@ -20,6 +20,7 @@ otherwise a rolling upgrade with mixed node versions could read an emptied field
 
 ```
 rustfs_system_network_internode_msgpack_json_fallback_total{direction, message}
+rustfs_system_network_internode_msgpack_json_decode_error_total{direction, message, codec}
 ```
 
 Incremented whenever a decode falls back to the JSON field because the msgpack payload was
@@ -29,6 +30,7 @@ absent.
 - `direction="response"` — a client decoding a peer's response (`cluster/rpc/remote_disk.rs`),
   including the list-level `ReadMultiple` / `BatchReadVersion` fallbacks.
 - `message` — the value name, e.g. `FileInfo`, `RawFileInfo`, `ReadMultipleResp`.
+- `codec` — the failed codec for decode errors: `msgpack` for corrupt non-empty `_bin`, or `json` for corrupt legacy fallback JSON.
 
 ## Stage 0 — Observe (current stage)
 
@@ -47,6 +49,16 @@ Every series must be `0`. A non-zero value means some peer is still emitting an
 `_bin` (an old node, or a message whose sender does not fill `_bin`) — investigate the
 `{direction, message}` label before proceeding.
 
+Decode errors must also stay at zero across the observation window:
+
+```promql
+sum by (direction, message, codec) (
+  increase(rustfs_system_network_internode_msgpack_json_decode_error_total[30d])
+)
+```
+
+A non-zero `codec="msgpack"` series means a peer sent corrupt or incompatible `_bin` bytes; it must fail closed and block convergence. A non-zero `codec="json"` series means the legacy fallback field was corrupt or semantically incompatible; it also blocks convergence and rollback confidence.
+
 Standing alert (keep enabled through all stages):
 
 ```yaml
@@ -57,6 +69,13 @@ Standing alert (keep enabled through all stages):
   annotations:
     summary: "Internode RPC fell back to JSON decode ({{ $labels.direction }}/{{ $labels.message }})"
     description: "A peer sent an empty msgpack _bin payload. Do NOT advance msgpack-only convergence while this fires."
+- alert: InternodeMsgpackJsonDecodeError
+  expr: sum by (direction, message, codec) (increase(rustfs_system_network_internode_msgpack_json_decode_error_total[15m])) > 0
+  for: 5m
+  labels: { severity: warning }
+  annotations:
+    summary: "Internode RPC msgpack/JSON decode failed ({{ $labels.direction }}/{{ $labels.message }}/{{ $labels.codec }})"
+    description: "A peer sent an undecodable msgpack or JSON compatibility payload. Do NOT advance msgpack-only convergence while this fires."
 ```
 
 ## Field → peer-decoder audit