rustfs/crates/ecstore/docs/internode-transport/internode-transport-adapter-rfc.md

RFC: Internode Transport Adapter Boundary

Status: draft Last updated: 2026-05-22 Scope: OSS internode data-plane adapter analysis, benchmark baseline, and transport boundary

Summary

The current distributed internode paths use TCP-based HTTP/gRPC transports:

• tonic gRPC NodeService for most control, metadata, lock, health, and peer operations.
• HTTP streaming routes under /rustfs/rpc/ for remote disk file streams.

This document frames the existing work as an OSS InternodeDataTransport adapter boundary. The adapter keeps RustFS data-plane logic separate from the concrete transport backend while preserving the current TCP/HTTP behavior as the default implementation.

Current implementation status:

• InternodeDataTransport exists in crates/ecstore/src/rpc/internode_data_transport.rs.
• The default and only production backend is tcp-http; tcp is accepted as an alias.
• RUSTFS_INTERNODE_DATA_TRANSPORT selects the backend. Blank or unset values use tcp-http; invalid values fail closed.
• RemoteDisk::read_file_stream, RemoteDisk::create_file, RemoteDisk::append_file, and RemoteDisk::walk_dir delegate to the transport.
• NodeService gRPC remains the internode control plane and continues to carry metadata/control operations.

Related design notes in this directory:

• transport-capabilities.md
• transport-buffer-lifecycle.md
• transport-buffer-contract.md
• transport-fallback-and-selection.md
• transport-metrics-and-baseline.md

Open-source Scope

The OSS scope is:

• define a clear InternodeDataTransport adapter boundary;
• keep tcp-http as the default backend;
• keep existing TCP/HTTP behavior unchanged;
• keep internode data-plane behavior observable through metrics and baseline tooling;
• document buffer ownership, fallback, and capability expectations for maintainable transport code;
• avoid adding dependencies or backend implementations.

The OSS scope is not:

• adding another transport backend;
• replacing the gRPC control plane;
• adding benchmark plans for another transport;
• adding runtime plugin loading;
• changing object correctness semantics.

Goals

• Document the current internode control plane and data plane.
• Identify the existing transfer paths covered by the InternodeDataTransport adapter and the paths that remain on gRPC.
• Define the minimum benchmark baseline required before transport changes.
• Sketch a pluggable transport boundary that preserves the current TCP/HTTP behavior as the default backend.
• Document backend-neutral capability, fallback, buffer ownership, and observability expectations.

Non-Goals

• Implement another transport backend.
• Replace tonic gRPC for control-plane RPCs.
• Redesign erasure coding, quorum handling, disk health tracking, or object correctness semantics.
• Change default development, CI, or ordinary RustFS deployment requirements.

Current Internode Architecture

Server-side entry points

The main HTTP server builds a hybrid service per connection:

• rustfs/src/server/http.rs wires a NodeServiceServer for gRPC.
• rustfs/src/storage/rpc/InternodeRpcService intercepts HTTP paths under /rustfs/rpc/.
• Other HTTP/S3 traffic continues through the normal S3 service.

Compression logic already treats /rustfs/rpc/ and /rustfs/peer/ as internode RPC paths and skips normal response compression for them.

gRPC channel management

crates/protos/src/lib.rs creates internode gRPC channels with tonic Endpoint:

• connect timeout
• TCP keepalive
• HTTP/2 keepalive interval and timeout
• request timeout
• optional TLS configuration
• global channel caching and failed-connection eviction

This confirms the current gRPC transport is TCP/HTTP2-based.

NodeService layout

crates/protos/src/node.proto defines one NodeService that mixes several classes of RPCs:

• meta service: bucket and metadata operations
• disk service: local/remote disk operations
• lock service: distributed lock operations
• peer rest service: node health, metrics, IAM/policy reload, rebalance, profiling, events, and admin-style operations

The service layout is practical today, but it is too broad to become the transport adapter surface. A pluggable data transport should target only disk data streams and keep this gRPC service as the control plane.

Control Plane vs Data Plane

Control plane

These paths carry coordination, metadata, health, and administrative state. They should remain on gRPC/TCP:

Area	Client/server code	Examples	Notes
Bucket peer ops	crates/ecstore/src/rpc/peer_s3_client.rs, rustfs/src/storage/rpc/node_service/bucket.rs	MakeBucket, ListBucket, DeleteBucket, GetBucketInfo, HealBucket	Small metadata/control payloads.
Locking	crates/ecstore/src/rpc/remote_locker.rs, rustfs/src/storage/rpc/node_service/lock.rs	Lock, UnLock, Refresh, batch lock/unlock	Latency-sensitive but not bulk data; correctness and timeout semantics matter more than transport bandwidth.
Peer/admin state	crates/ecstore/src/rpc/peer_rest_client.rs, rustfs/src/storage/rpc/node_service/health.rs, node_service/metrics.rs, node_service/event.rs	LocalStorageInfo, ServerInfo, GetMetrics, GetLiveEvents, reload APIs, rebalance APIs	Operational control plane.
Disk metadata/control	crates/ecstore/src/rpc/remote_disk.rs, rustfs/src/storage/rpc/node_service/disk.rs	DiskInfo, ReadXL, ReadVersion, ReadMetadata, WriteMetadata, RenameFile, RenamePart, Delete*, VerifyFile, CheckParts	Usually metadata, integrity checks, or namespace mutations.
Connection health	RemoteDisk, RemotePeerS3Client, PeerRestClient	TCP connectivity probes and fault/recovery state	Must remain available even if an optional data backend is unavailable.

Data plane candidates

These paths move object shard bytes or stream potentially large disk data and are the only reasonable first candidates for a pluggable transport.

Priority	Path	Current client	Current server	Current transport	Why it matters
P0	read_file_stream	RemoteDisk::read_file_stream	handle_read_file in http_service.rs	HTTP GET /rustfs/rpc/read_file_stream with a streaming response body	Main remote disk read stream used by bitrot readers and erasure reads.
P0	put_file_stream	RemoteDisk::create_file and RemoteDisk::append_file	handle_put_file in http_service.rs	HTTP PUT /rustfs/rpc/put_file_stream with a streaming request body	Main remote disk write stream used by bitrot writers and erasure writes.
P1	walk_dir	RemoteDisk::walk_dir	handle_walk_dir in http_service.rs	HTTP GET /rustfs/rpc/walk_dir with a streamed metadata listing	Can be high-volume during scans/healing, but it is metadata-oriented rather than object byte data.
P1	ReadAll / WriteAll	RemoteDisk::read_all / write_all	gRPC unary disk handlers	gRPC unary bytes payload	Moves bytes today, but should be measured before treating it as a high-throughput data path.
P2	proto WriteStream / ReadAt	currently not used	currently returns unimplemented	gRPC streaming definitions exist but are not implemented	Declared proto shape, not a current production path.

P1 Data Path Inventory

Classification:

• Covered by InternodeDataTransport: RemoteDisk opens the transfer through the transport abstraction.
• Still direct TCP/HTTP/gRPC: bytes move over a fixed internode protocol outside the transport abstraction.
• Metadata/control-plane only: payloads are expected to be small metadata, namespace, lock, health, or admin messages.
• Not relevant: declared or test-only paths that are not current production data paths.

Covered by InternodeDataTransport

Path	Owner references	Server references	Classification	Notes
Remote shard read stream	crates/ecstore/src/rpc/remote_disk.rs::RemoteDisk::read_file_stream; crates/ecstore/src/rpc/internode_data_transport.rs::InternodeDataTransport::open_read; crates/ecstore/src/bitrot.rs::create_bitrot_reader	rustfs/src/storage/rpc/http_service.rs::handle_read_file	Covered by InternodeDataTransport	Object GET, repair reads, and erasure decode use this path for remote shard bytes.
Remote shard write stream	RemoteDisk::create_file; RemoteDisk::append_file; InternodeDataTransport::open_write; crates/ecstore/src/bitrot.rs::create_bitrot_writer	rustfs/src/storage/rpc/http_service.rs::handle_put_file	Covered by InternodeDataTransport	Object PUT and multipart part upload use this path for remote shard bytes.
Remote namespace walk stream	RemoteDisk::walk_dir; InternodeDataTransport::open_walk_dir; crates/ecstore/src/cache_value/metacache_set.rs walk producers	rustfs/src/storage/rpc/http_service.rs::handle_walk_dir	Covered by InternodeDataTransport	High-volume listing/scanner/heal metadata stream. It is not object byte data, but it is a large internode stream.
Remote mmap-copy read fallback	RemoteDisk::read_file_mmap_copy	same as remote shard read stream	Covered by InternodeDataTransport through read_file_stream	The remote path buffers the stream into Bytes; true zero-copy is not guaranteed for remote disks.

Still Direct TCP/HTTP/gRPC

Path	Owner references	Server references	Classification	Notes
ReadAll	RemoteDisk::read_all; crates/ecstore/src/store_init.rs; heal resume metadata readers	rustfs/src/storage/rpc/node_service/disk.rs::handle_read_all	Still direct gRPC	Unary bytes response. Currently used mostly for metadata/config files; measure before moving.
WriteAll	RemoteDisk::write_all; crates/ecstore/src/store_init.rs; heal resume metadata writers	rustfs/src/storage/rpc/node_service/disk.rs::handle_write_all	Still direct gRPC	Unary bytes request. Currently used mostly for metadata/config/checkpoint writes.
ReadMultiple	RemoteDisk::read_multiple; crates/ecstore/src/set_disk/read.rs::read_multiple_files	rustfs/src/storage/rpc/node_service/disk.rs::handle_read_multiple	Still direct gRPC	Returns multiple small file payloads, usually metadata/listing support. Could become large with many entries.
ReadParts	RemoteDisk::read_parts; crates/ecstore/src/set_disk/read.rs::read_parts; multipart list/complete paths	rustfs/src/storage/rpc/node_service/disk.rs::handle_read_parts	Still direct gRPC	Encoded ObjectPartInfo metadata, not object data.
RenamePart	RemoteDisk::rename_part; crates/ecstore/src/set_disk/write.rs::rename_part	rustfs/src/storage/rpc/node_service/disk.rs::handle_rename_part	Still direct gRPC	Carries part metadata while committing multipart data already written through stream writers.
ListDir	RemoteDisk::list_dir; multipart/lifecycle metadata listing callers	rustfs/src/storage/rpc/node_service/disk.rs::handle_list_dir	Still direct gRPC	Directory name listing, metadata/control-plane unless measured otherwise.
Legacy gRPC WalkDir	rustfs/src/storage/rpc/node_service.rs::NodeService::walk_dir	same file	Still direct gRPC	Server implementation remains, but current RemoteDisk::walk_dir uses HTTP through the transport. Keep until callers are audited or compatibility policy is set.

Metadata/control-plane only

Area	Owner references	Classification	Notes
Disk metadata and namespace mutations	RemoteDisk::{read_metadata,write_metadata,update_metadata,read_version,read_xl,rename_data,rename_file,delete*,verify_file,check_parts,disk_info}	Metadata/control-plane only	These remain on gRPC by design.
Peer/bucket/admin operations	crates/ecstore/src/rpc/{peer_s3_client.rs,peer_rest_client.rs,remote_locker.rs} and matching rustfs/src/storage/rpc/node_service/* handlers	Metadata/control-plane only	Not candidates for a data-plane backend without separate measurements.
Store init and format operations	crates/ecstore/src/store_init.rs	Metadata/control-plane only	Uses ReadAll/WriteAll for small format/config objects.
Heal orchestration	crates/heal/src/heal/storage.rs and crates/ecstore/src/set_disk.rs::heal_object	Metadata/control-plane plus covered data reads	Heal object data reads go through get_object_reader and then covered shard streams; resume/checkpoint metadata uses direct gRPC disk metadata calls.

Not Relevant Current Paths

Path	Owner references	Classification	Notes
Proto Write	crates/protos/src/node.proto; rustfs/src/storage/rpc/node_service/disk.rs::handle_write	Not relevant	Handler is unimplemented.
Proto WriteStream	crates/protos/src/node.proto; rustfs/src/storage/rpc/node_service.rs::write_stream	Not relevant	Returns unimplemented.
Proto ReadAt	crates/protos/src/node.proto; rustfs/src/storage/rpc/node_service.rs::read_at	Not relevant	Returns unimplemented.
E2E reliant gRPC helpers	crates/e2e_test/src/reliant/*	Not relevant	Test harnesses, not production internode data-path callers.

Current Limitations

Risk	Limitation
Medium	ReadAll and WriteAll still carry unary bytes over gRPC. They appear metadata-oriented today, but there is no size threshold or routing policy.
Medium	ReadMultiple can aggregate many metadata files into one gRPC response.
Low	Legacy gRPC WalkDir remains implemented while RemoteDisk::walk_dir uses HTTP through the transport.
Medium	Remote read_file_mmap_copy is a buffered read over the transport, not a remote zero-copy contract.
Medium	Server-side TCP HTTP route handling is outside the client-side trait.

Current Object Write Path

For object PUTs in distributed erasure mode, the relevant flow is:

1. Upper storage layers prepare object data and erasure metadata.
2. SetDisks selects local and remote disks.
3. create_bitrot_writer calls disk.create_file(...) for each shard writer.
4. For a remote disk, RemoteDisk::create_file delegates to InternodeDataTransport::open_write.
5. HttpWriter sends an HTTP PUT to /rustfs/rpc/put_file_stream.
6. The remote node's handle_put_file opens the local file writer and copies incoming body chunks into it.
7. Erasure::encode writes shards through MultiWriter to all selected writers while enforcing write quorum.

This is the primary write data-plane candidate.

Current Object Read Path

For object GETs and repair reads in distributed erasure mode, the relevant flow is:

1. SetDisks prepares shard readers for the selected disks.
2. create_bitrot_reader uses local mmap-copy reads only when disk.is_local().
3. For a remote disk, it calls disk.read_file_stream(...).
4. RemoteDisk::read_file_stream delegates to InternodeDataTransport::open_read.
5. HttpReader sends an HTTP GET to /rustfs/rpc/read_file_stream.
6. The remote node's handle_read_file opens the local disk stream and returns it as an HTTP streaming body.
7. The erasure decoder reads from the shard streams and reconstructs the object.

This is the primary read data-plane candidate.

Existing Metrics and Benchmark Surface

RustFS already has coarse internode metrics in crates/io-metrics/src/internode_metrics.rs:

• sent bytes
• received bytes
• outgoing requests
• incoming requests
• errors
• dial errors
• average dial time

These metrics are useful as a starting point. For backend comparisons, the relevant route-level and operation-level dimensions are:

• read_file_stream
• put_file_stream
• walk_dir
• gRPC ReadAll / WriteAll
• gRPC control-plane request volume

Existing benchmark assets:

• scripts/run_object_batch_bench.sh
• scripts/run_object_batch_bench_enhanced.sh
• scripts/run_object_batch_bench_abc.sh
• scripts/run_four_node_cluster_failover_bench.sh
• scripts/run_internode_transport_baseline.sh (scenario matrix wrapper for local vs distributed TCP baseline artifacts)
• Criterion benches under crates/ecstore/benches/

These mostly cover S3/object workload or erasure coding performance. They do not yet isolate internode transport cost.

Required TCP Baseline

Before changing internode data transport behavior or comparing a non-default backend, collect a baseline for the current TCP/HTTP/gRPC implementation.

Topology

Minimum:

• 1-node local erasure deployment, to measure local disk and erasure overhead.
• 4-node distributed erasure deployment, to measure internode overhead.

Preferred:

• Same host count and disk layout for every run.
• Dedicated network interface or isolated VLAN.
• Fixed CPU governor and no unrelated background load.
• Recorded kernel version, NIC model, MTU, RustFS commit, Rust toolchain, and benchmark tool versions.

Workloads

Workload	Sizes	Concurrency	Main signal
S3 PUT	4 KiB, 1 MiB, 16 MiB, 128 MiB, 1 GiB	1, 16, 64, 128	End-to-end write throughput and tail latency.
S3 GET	4 KiB, 1 MiB, 16 MiB, 128 MiB, 1 GiB	1, 16, 64, 128	End-to-end read throughput and tail latency.
Remote disk stream read	shard-sized ranges from read_file_stream	1, 16, 64	Isolated internode read path.
Remote disk stream write	shard-sized writes through put_file_stream	1, 16, 64	Isolated internode write path.
Healing / repair	missing disk or missing shard scenario	controlled	Rebuild throughput and read/write amplification.
Scanner walk	large bucket/object namespace	controlled	Metadata streaming pressure, not the primary object-byte transport path.

Measurements

Collect:

• throughput in bytes/s and objects/s
• p50, p95, p99, and max latency
• CPU utilization per process and per core
• memory RSS and allocation pressure where available
• rustfs_system_network_internode_* metrics
• TCP retransmits, socket errors, and NIC throughput
• disk throughput and utilization
• failure/retry/fallback counts

The baseline should produce a machine-readable artifact, for example target/bench/internode-transport/<timestamp>/summary.csv, plus the exact commands and configuration used.

Baseline runner entry point

Use scripts/run_internode_transport_baseline.sh to execute a reproducible S3 PUT/GET matrix against local and distributed scenarios and export:

• summary.csv (throughput/latency summary per workload and object size)
• internode_metric_deltas.csv (operation-level internode metric deltas when --metrics-url is provided)

See transport-metrics-and-baseline.md for current metric names, labels, operation values, baseline inputs, and baseline artifact fields.

Transport Abstraction Proposal

Design principle

Keep NodeService as the control plane. Introduce a separate data transport only below RemoteDisk, where remote disk byte streams are opened today.

The first implementation should be a no-behavior-change TCP/HTTP backend that wraps the current HttpReader, HttpWriter, and /rustfs/rpc/* handlers. Non-default backend work should not proceed until the default wrapper is measured and adapter gaps are documented.

Candidate boundary

The current boundary is remote disk stream transfer:

#[async_trait::async_trait]
pub trait InternodeDataTransport: Send + Sync + std::fmt::Debug {
    async fn open_read(&self, request: ReadStreamRequest) -> Result<FileReader>;
    async fn open_write(&self, request: WriteStreamRequest) -> Result<FileWriter>;
    async fn open_walk_dir(&self, request: WalkDirStreamRequest) -> Result<FileReader>;
    fn name(&self) -> &'static str;
    fn capabilities(&self) -> InternodeDataTransportCapabilities;
}

Initial request fields should mirror the current HTTP query parameters:

• peer endpoint
• disk reference
• volume
• path
• offset
• length
• append/create mode
• expected size
• optional stall timeout for long-running listing streams

The initial TCP backend can keep the current signed HTTP URLs internally.

Integration point

RemoteDisk delegates only these methods to the data transport:

• read_file_stream
• read_file_mmap_copy as the current wrapper over read_file_stream
• append_file
• create_file
• walk_dir

All other RemoteDisk methods continue using the current gRPC client in this adapter scope.

Capability model

Avoid hard-coding transport-specific assumptions into the generic interface. The current conservative capability fields are:

• streaming read
• streaming write
• streaming walk-dir
• ordered delivery
• maximum transfer size
• fallback support

The TCP/HTTP backend should report only capabilities that it actually provides.

TCP Fallback Requirements

TCP/HTTP/gRPC must remain the default and required backend.

Fallback rules:

• If no explicit data transport is configured, use the current TCP/HTTP implementation.
• The current accepted values for RUSTFS_INTERNODE_DATA_TRANSPORT are tcp-http and the tcp alias. Empty and unset values use tcp-http.
• Invalid configured values fail closed with an error that includes the env var name and invalid value.
• Unsupported configured backends fail closed during transport construction.
• Runtime fallback must preserve object correctness and quorum semantics.
• Fallback events must be logged and counted in metrics.

Do not add fallback settings until there is an implementation PR that uses them.

Baseline Validation Commands

Dry-run command:

scripts/run_internode_transport_baseline.sh \
  --access-key minioadmin \
  --secret-key minioadmin \
  --scenarios local=http://127.0.0.1:9000,distributed=http://127.0.0.1:9001 \
  --sizes 4KiB,1MiB \
  --concurrencies 1 \
  --duration 10s \
  --dry-run

Real TCP baseline command with metrics:

RUSTFS_INTERNODE_DATA_TRANSPORT=tcp-http \
scripts/run_internode_transport_baseline.sh \
  --access-key "$RUSTFS_ACCESS_KEY" \
  --secret-key "$RUSTFS_SECRET_KEY" \
  --scenarios local=http://127.0.0.1:9000,distributed=http://127.0.0.1:9001 \
  --metrics-url http://127.0.0.1:9000/metrics \
  --out-dir target/bench/internode-transport/manual-run

Expected artifacts:

• run_manifest.txt
• summary.csv
• internode_metric_deltas.csv when --metrics-url is provided

The baseline validates the default TCP/HTTP path only. It must not be used to claim support or performance for any other transport.

Adapter Constraints

The current adapter boundary has these constraints:

• tcp-http is the default and only OSS backend.
• Backend selection is explicit and fail-closed.
• The gRPC control plane remains responsible for metadata, health, locks, and coordination.
• Transport errors must preserve existing disk health, quorum, timeout, and integrity semantics.
• Metrics must identify the selected backend and operation without high-cardinality labels.

The current RemoteDisk::walk_dir stream is routed through the adapter. Metadata RPCs, locks, admin RPCs, bucket coordination, and the legacy gRPC WalkDir handler remain outside the current data-plane boundary.

Out of Scope

This RFC does not add a plugin system, split the adapter into a separate crate, add accepted backend values, or implement a new transport backend.