rustfs/docs/operations/internode-msgpack-json-convergence-runbook.md

Internode msgpack/JSON Convergence Runbook

Operational runbook for retiring the redundant JSON compatibility fields on internode gRPC metadata RPCs (grpc-optimization P2-1). This is a cross-version change: it proceeds strictly by observation-gated stages, never in one step.

Background

Internode RPCs dual-encode each metadata value as both:

• a msgpack binary field (*_bin, e.g. file_info_bin), and
• a JSON compatibility string (e.g. file_info).

Decoders prefer the _bin payload and fall back to the JSON string only when _bin is empty (decode_msgpack_or_json). The dual-write costs bandwidth and CPU. Before the JSON fields can be dropped, the fallback branch must be proven unused in production — otherwise a rolling upgrade with mixed node versions could read an emptied field.

The observation metric (already shipped)

rustfs_system_network_internode_msgpack_json_fallback_total{direction, message}
rustfs_system_network_internode_msgpack_json_decode_error_total{direction, message, codec}

Incremented whenever a decode falls back to the JSON field because the msgpack payload was absent.

• direction="request" — a server decoding a peer's request (node_service/disk.rs).
• direction="response" — a client decoding a peer's response (cluster/rpc/remote_disk.rs), including the list-level ReadMultiple / BatchReadVersion fallbacks.
• message — the value name, e.g. FileInfo, RawFileInfo, ReadMultipleResp.
• codec — the failed codec for decode errors: msgpack for corrupt non-empty _bin, or json for corrupt legacy fallback JSON.

Stage 0 — Observe (current stage)

Ship the current release (which contains the counter) and let it run for at least one full release window across the whole fleet. The counter must stay at zero.

Confirm zero across the observation window (adjust [30d] to the window length):

sum by (direction, message) (
  increase(rustfs_system_network_internode_msgpack_json_fallback_total[30d])
)

Every series must be 0. A non-zero value means some peer is still emitting an empty _bin (an old node, or a message whose sender does not fill _bin) — investigate the {direction, message} label before proceeding.

Decode errors must also stay at zero across the observation window:

sum by (direction, message, codec) (
  increase(rustfs_system_network_internode_msgpack_json_decode_error_total[30d])
)

A non-zero codec="msgpack" series means a peer sent corrupt or incompatible _bin bytes; it must fail closed and block convergence. A non-zero codec="json" series means the legacy fallback field was corrupt or semantically incompatible; it also blocks convergence and rollback confidence.

Standing alert (keep enabled through all stages):

- alert: InternodeMsgpackJsonFallback
  expr: sum by (direction, message) (increase(rustfs_system_network_internode_msgpack_json_fallback_total[15m])) > 0
  for: 5m
  labels: { severity: warning }
  annotations:
    summary: "Internode RPC fell back to JSON decode ({{ $labels.direction }}/{{ $labels.message }})"
    description: "A peer sent an empty msgpack _bin payload. Do NOT advance msgpack-only convergence while this fires."
- alert: InternodeMsgpackJsonDecodeError
  expr: sum by (direction, message, codec) (increase(rustfs_system_network_internode_msgpack_json_decode_error_total[15m])) > 0
  for: 5m
  labels: { severity: warning }
  annotations:
    summary: "Internode RPC msgpack/JSON decode failed ({{ $labels.direction }}/{{ $labels.message }}/{{ $labels.codec }})"
    description: "A peer sent an undecodable msgpack or JSON compatibility payload. Do NOT advance msgpack-only convergence while this fires."

Field → peer-decoder audit

The send-side change (Stage 1) may only empty a JSON field whose peer decodes _bin first. The following mapping is verified against the current code.

Convergence-ready (peer decodes _bin first)

Direction	Message / field	Peer decoder
request	WriteMetadata.file_info	FileInfo (node_service/disk.rs)
request	UpdateMetadata.file_info	FileInfo
request	UpdateMetadata.opts	UpdateMetadataOpts
request	RenameData.file_info	FileInfo
request	ReadMultiple.read_multiple_req	ReadMultipleReq
request	BatchReadVersion.batch_read_version_req	BatchReadVersionReq
request	Read*.opts	ReadOptions
response	ReadVersion.file_info	FileInfo (cluster/rpc/remote_disk.rs)
response	ReadXL.raw_file_info	RawFileInfo
response	RenameData.rename_data_resp	RenameDataResp
response	ReadMultiple resp list	per-item + list fallback
response	BatchReadVersion resp list	per-item + list fallback

_bin support added THIS release — converge after their own window

The DeleteVersion/DeleteVersions protos had no _bin fields. They gained additive *_bin fields plus bin-first server decoders in this release, and the client now dual-writes them. They are kept out of the msgpack-only set (always dual-write) until their own fallback counter has read zero across a window with the new decoders fully deployed.

Direction	Message / field	Status
request	DeleteVersion.file_info (FileInfo)	_bin added; dual-write; converge after window
request	DeleteVersion.opts (DeleteOptions)	_bin added; dual-write; converge after window
request	DeleteVersions.versions (FileInfoVersions)	_bin added; dual-write; converge after window
request	DeleteVersions.opts (DeleteOptions)	_bin added; dual-write; converge after window

Any *_bin proto field not in the tables above must be mapped to a confirmed _bin-first peer decoder before it is added to the convergence set.

Still JSON-only (no _bin field)

Direction	Message / field	Note
response	DeleteVersion.raw_file_info	proto has no _bin; needs an additive proto field before it can converge.

Stage 1 — Stop writing JSON (env-gated, after Stage 0 reads zero)

The send-side lever is implemented and requires two default-off env flags:

• RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true
• RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=true

The first flag only requests msgpack-only. The second flag is the explicit proof gate that the fleet has passed the release-window fallback, capability, and rollback checks. If only RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true is set, RustFS keeps dual-writing JSON compatibility fields so old JSON-only peers remain compatible. When both flags are enabled, the convergence-ready fields above send only _bin and leave the JSON string empty; the _bin payload is always sent and decoders keep the JSON read fallback unchanged. The delete fields are excluded (dual-write) per the section above.

Only enable it after Stage 0 has read zero for a full window across the fleet:

1. Ship with the flag off (no behavior change).
2. Rehearse the request-only gate with RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true and RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=false. This must keep writing JSON compatibility fields and is safe with unsupported/old peers; any fallback or decode-error increment still blocks convergence.
3. Enable it on one canary node with both RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true and RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=true, then restart that node only and watch the fallback and decode-error counters for a soak period while real internode traffic is present. If they stay zero, enable fleet-wide.
4. Rollback: set either RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=false or RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=false (or unset either flag) and restart. No wire-format was broken in this stage, so rollback is immediate and safe.

The benchmark helper pins these operator states as dry-run phases:

scripts/run_internode_grpc_ab_bench.sh --stage p2 --phase before --dry-run
scripts/run_internode_grpc_ab_bench.sh --stage p2 --phase request-only --dry-run
scripts/run_internode_grpc_ab_bench.sh --stage p2 --phase canary --dry-run
scripts/run_internode_grpc_ab_bench.sh --stage p2 --phase after --dry-run
scripts/run_internode_grpc_ab_bench.sh --stage p2 --phase rollback --dry-run

Stage 2 — Remove the proto JSON fields (next release, N+1)

Only after Stage 1 has been stable with the flag on for a full window and the counter is still zero.

1. Mark the retired text fields reserved in crates/protos/src/node.proto (never reuse the field numbers) and delete the JSON read-fallback branches; codec becomes msgpack-only.
2. This is a hard wire-format change — it requires the mixed-version upgrade rehearsal (four-node scripts) to pass, and it cannot be rolled back by env alone.

Rollback matrix

Stage	Wire-format broken?	Rollback
0 Observe	no	n/a (metric only)
1 msgpack-only send	no	unset either msgpack-only env flag + restart
2 remove fields	yes	redeploy prior release; field numbers stay reserved

Related

• Codec + counter implementation: commit feat(internode): P2 msgpack/JSON codec observability + encode buffer presizing.
• Decoders: decode_msgpack_or_json in crates/ecstore/src/cluster/rpc/remote_disk.rs (client) and rustfs/src/storage/rpc/node_service/disk.rs (server).