test(internode): pin msgpack rollout gates (#5261)

Extend the internode gRPC benchmark driver with P2 request-only, canary, after, and rollback phases. The request-only phase proves that requesting msgpack-only without fleet confirmation keeps compatibility JSON fields, while canary and rollback materialize the operator states needed for mixed-version convergence without claiming real fleet soak evidence.

Verification:

- scripts/test_internode_grpc_ab_bench.sh

- bash -n scripts/run_internode_grpc_ab_bench.sh scripts/test_internode_grpc_ab_bench.sh

- git diff --check

Co-authored-by: heihutu <heihutu@gmail.com>

docs/operations/internode-grpc-benchmark-runbook.md

@@ -10,7 +10,7 @@ optimization stage (grpc-optimization P0–P3). Every stage is env-gated, so "be
 
 ## One-click driver
 
-`scripts/run_internode_grpc_ab_bench.sh --stage <p0|p1|p2|p3> --phase <before|after> [-- <bench args>]`
+`scripts/run_internode_grpc_ab_bench.sh --stage <p0|p1|p2|p3> --phase <before|after|request-only|canary|rollback> [-- <bench args>]`
 wraps the env matrix below: it writes the stage/phase **server** env to
 `<out-dir>/server-env.sh`, then runs the right underlying bench into
 `target/bench/internode-transport/<stage>-<phase>/`.
@@ -21,11 +21,13 @@ scripts/run_internode_grpc_ab_bench.sh --stage p1 --phase before -- --access-key
 scripts/run_internode_grpc_ab_bench.sh --stage p1 --phase after  -- --access-key AK --secret-key SK --metrics-url http://node1:9000/metrics
 # P3 failover A/B (docker four-node):
 scripts/run_internode_grpc_ab_bench.sh --stage p3 --phase after
+# P2 rollout gates (env preview for request-only rehearsal, canary, and rollback):
+scripts/run_internode_grpc_ab_bench.sh --stage p2 --phase request-only --dry-run
+scripts/run_internode_grpc_ab_bench.sh --stage p2 --phase canary --dry-run
+scripts/run_internode_grpc_ab_bench.sh --stage p2 --phase rollback --dry-run
 ```
 
-`RUSTFS_INTERNODE_*` are **server** env: for the load-driven stages (p0/p1/p2) source the emitted
-`server-env.sh` on every node and restart rustfs *before* the run — the driver cannot mutate an
-already-running server. Use `--dry-run` to preview the env and command.
+`RUSTFS_INTERNODE_*` are **server** env: for the load-driven stages (p0/p1/p2) source the emitted `server-env.sh` on every node and restart rustfs *before* the run — the driver cannot mutate an already-running server. The P2 `canary` phase is the exception: source it only on the selected canary node after the release-window counters and fleet-support checks pass, and keep the rest of the fleet on `before` or `request-only` while observing fallback/decode-error counters. Use `--dry-run` to preview the env and command.
 
 ## Harness
 
@@ -115,6 +117,7 @@ Artifact layout per stage (attach both halves + the diff):
 target/bench/internode-transport/
   p0-before/  p0-after/     # server-env.sh + object-bench summaries + metric deltas
   p1-before/  p1-after/     # + lock p99 delta (the ≥20% gate)
+  p2-before/  p2-request-only/  p2-canary/  p2-after/  p2-rollback/  # env gate artifacts + fallback/decode-error observations
   p3-before/  p3-after/     # cold-start + sustained-offline experiment + offline gauge trace
 ```
 

docs/operations/internode-msgpack-json-convergence-runbook.md

@@ -135,10 +135,21 @@ The first flag only requests msgpack-only. The second flag is the explicit proof
 Only enable it **after** Stage 0 has read zero for a full window across the fleet:
 
 1. Ship with the flag **off** (no behavior change).
-2. Enable it on one node with both `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true` and `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=true`, then restart and watch the fallback counter for a soak period. If it stays zero, enable fleet-wide.
-3. **Rollback:** set either `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=false` or `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=false` (or unset either flag) and restart. No
+2. Rehearse the `request-only` gate with `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true` and `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=false`. This must keep writing JSON compatibility fields and is safe with unsupported/old peers; any fallback or decode-error increment still blocks convergence.
+3. Enable it on one canary node with both `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true` and `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=true`, then restart that node only and watch the fallback and decode-error counters for a soak period while real internode traffic is present. If they stay zero, enable fleet-wide.
+4. **Rollback:** set either `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=false` or `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=false` (or unset either flag) and restart. No
    wire-format was broken in this stage, so rollback is immediate and safe.
 
+The benchmark helper pins these operator states as dry-run phases:
+
+```bash
+scripts/run_internode_grpc_ab_bench.sh --stage p2 --phase before --dry-run
+scripts/run_internode_grpc_ab_bench.sh --stage p2 --phase request-only --dry-run
+scripts/run_internode_grpc_ab_bench.sh --stage p2 --phase canary --dry-run
+scripts/run_internode_grpc_ab_bench.sh --stage p2 --phase after --dry-run
+scripts/run_internode_grpc_ab_bench.sh --stage p2 --phase rollback --dry-run
+```
+
 ## Stage 2 — Remove the proto JSON fields (next release, N+1)
 
 Only after Stage 1 has been stable with the flag on for a full window and the counter is