docs(internode): keep transport adapter OSS scoped (#3067)

docs(internode): keep transport adapter oss scoped

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
This commit is contained in:
Henry Guo
2026-05-23 13:00:36 +08:00
committed by GitHub
parent 2786a4734a
commit 53b608c089
6 changed files with 96 additions and 204 deletions
@@ -50,20 +50,15 @@ The OSS scope is:
tooling; tooling;
- document buffer ownership, fallback, and capability expectations for - document buffer ownership, fallback, and capability expectations for
maintainable transport code; maintainable transport code;
- avoid hardware-specific dependencies or backend implementations. - avoid adding dependencies or backend implementations.
The OSS scope is not: The OSS scope is not:
- RDMA support; - adding another transport backend;
- DPU support; - replacing the gRPC control plane;
- DOCA support; - adding benchmark plans for another transport;
- BlueField support; - adding runtime plugin loading;
- RoCE/InfiniBand support; - changing object correctness semantics.
- hardware benchmark planning;
- hardware-specific backend implementation.
Hardware-specific transport experiments are outside the scope of this OSS
document.
## Goals ## Goals
@@ -78,12 +73,11 @@ document.
## Non-Goals ## Non-Goals
- Implement RDMA, RoCE, InfiniBand, DPU, DOCA, DPDK, SPDK, or SmartNIC support. - Implement another transport backend.
- Replace `tonic` gRPC for control-plane RPCs. - Replace `tonic` gRPC for control-plane RPCs.
- Redesign erasure coding, quorum handling, disk health tracking, or object - Redesign erasure coding, quorum handling, disk health tracking, or object
correctness semantics. correctness semantics.
- Require specialized hardware for default development, CI, or ordinary RustFS - Change default development, CI, or ordinary RustFS deployment requirements.
deployments.
## Current Internode Architecture ## Current Internode Architecture
@@ -154,7 +148,7 @@ are the only reasonable first candidates for a pluggable transport.
| P0 | `put_file_stream` | `RemoteDisk::create_file` and `RemoteDisk::append_file` | `handle_put_file` in `http_service.rs` | HTTP `PUT /rustfs/rpc/put_file_stream` with a streaming request body | Main remote disk write stream used by bitrot writers and erasure writes. | | P0 | `put_file_stream` | `RemoteDisk::create_file` and `RemoteDisk::append_file` | `handle_put_file` in `http_service.rs` | HTTP `PUT /rustfs/rpc/put_file_stream` with a streaming request body | Main remote disk write stream used by bitrot writers and erasure writes. |
| P1 | `walk_dir` | `RemoteDisk::walk_dir` | `handle_walk_dir` in `http_service.rs` | HTTP `GET /rustfs/rpc/walk_dir` with a streamed metadata listing | Can be high-volume during scans/healing, but it is metadata-oriented rather than object byte data. | | P1 | `walk_dir` | `RemoteDisk::walk_dir` | `handle_walk_dir` in `http_service.rs` | HTTP `GET /rustfs/rpc/walk_dir` with a streamed metadata listing | Can be high-volume during scans/healing, but it is metadata-oriented rather than object byte data. |
| P1 | `ReadAll` / `WriteAll` | `RemoteDisk::read_all` / `write_all` | gRPC unary disk handlers | gRPC unary `bytes` payload | Moves bytes today, but should be measured before treating it as a high-throughput data path. | | P1 | `ReadAll` / `WriteAll` | `RemoteDisk::read_all` / `write_all` | gRPC unary disk handlers | gRPC unary `bytes` payload | Moves bytes today, but should be measured before treating it as a high-throughput data path. |
| P2 | proto `WriteStream` / `ReadAt` | currently not used | currently returns unimplemented | gRPC streaming definitions exist but are not implemented | Possible future API shape, not a current production path. | | P2 | proto `WriteStream` / `ReadAt` | currently not used | currently returns unimplemented | gRPC streaming definitions exist but are not implemented | Declared proto shape, not a current production path. |
## P1 Data Path Inventory ## P1 Data Path Inventory
@@ -425,19 +419,9 @@ Fallback rules:
`tcp-http` and the `tcp` alias. Empty and unset values use `tcp-http`. `tcp-http` and the `tcp` alias. Empty and unset values use `tcp-http`.
- Invalid configured values fail closed with an error that includes the env var - Invalid configured values fail closed with an error that includes the env var
name and invalid value. name and invalid value.
- If a future non-default backend fails initialization, either fail fast with a - Unsupported configured backends fail closed during transport construction.
clear error or fall back to TCP only when the configured policy allows
fallback.
- Runtime fallback must preserve object correctness and quorum semantics. - Runtime fallback must preserve object correctness and quorum semantics.
- Fallback events must be logged and counted in metrics. - Fallback events must be logged and counted in metrics.
- CI and local development must not require specialized transport hardware.
Suggested future configuration shape:
```text
RUSTFS_INTERNODE_DATA_TRANSPORT=tcp-http
RUSTFS_INTERNODE_DATA_TRANSPORT_FALLBACK=tcp
```
Do not add fallback settings until there is an implementation PR that uses them. Do not add fallback settings until there is an implementation PR that uses them.
@@ -475,33 +459,25 @@ Expected artifacts:
- `internode_metric_deltas.csv` when `--metrics-url` is provided - `internode_metric_deltas.csv` when `--metrics-url` is provided
The baseline validates the default TCP/HTTP path only. It must not be used to The baseline validates the default TCP/HTTP path only. It must not be used to
claim support or performance for any non-default transport backend. claim support or performance for any other transport.
## Non-default Backend Boundary ## Adapter Constraints
A future non-default backend must be explicitly enabled and must not replace The current adapter boundary has these constraints:
`tcp-http` silently. It should be designed as an optional data-plane backend,
not as a replacement for the gRPC control plane.
A future non-default backend would need an explicit design for: - `tcp-http` is the default and only OSS backend.
- Backend selection is explicit and fail-closed.
- peer capability discovery over the existing gRPC control plane; - The gRPC control plane remains responsible for metadata, health, locks, and
- connection management and health mapping into existing disk fault handling; coordination.
- backend-specific buffer lifecycle and any staging or registration cache; - Transport errors must preserve existing disk health, quorum, timeout, and
- buffer ownership, alignment, and lifetime rules; integrity semantics.
- stable buffer behavior for erasure shards; - Metrics must identify the selected backend and operation without high-cardinality
- authentication and authorization for out-of-band data transfers; labels.
- encryption or an equivalent documented security boundary;
- timeout, cancellation, retry, and fallback behavior;
- metrics for transfer latency, bytes, queue depth, retries, fallback, and
errors.
`walk_dir`, metadata RPCs, locks, admin RPCs, and bucket coordination remain `walk_dir`, metadata RPCs, locks, admin RPCs, and bucket coordination remain
outside the current data-plane boundary. outside the current data-plane boundary.
## Out of Scope ## Out of Scope
Hardware-specific transport experiments are outside the scope of this OSS This RFC does not add a plugin system, split the adapter into a separate crate,
document. This RFC does not add a plugin system, split the adapter into a add accepted backend values, or implement a new transport backend.
separate crate, add accepted backend values, or implement a new transport
backend.
@@ -9,16 +9,11 @@ does not implement a new backend and does not change production behavior.
The open-source RustFS path keeps `tcp-http` as the default internode data The open-source RustFS path keeps `tcp-http` as the default internode data
transport. This document defines adapter contracts only: transport. This document defines adapter contracts only:
- no production RDMA, DPU, DOCA, BlueField, DPDK, SPDK, or hardware - no additional production backend is introduced;
acceleration backend is introduced; - no dependency is added;
- no hardware SDK, `libibverbs`, `rdma-core`, or vendor dependency is added;
- no new accepted production backend value is added; - no new accepted production backend value is added;
- future external or separately maintained backends may implement the same - RustFS core data-plane logic remains independent of the concrete transport
adapter boundary without changing RustFS core data-plane logic. implementation.
Examples of possible future external backends include DOCA/BlueField,
RDMA/RoCE, or other DPU/NIC implementations. These are examples only and are
not implemented or scheduled by this design.
## Current Adapter Surface ## Current Adapter Surface
@@ -30,28 +25,24 @@ The current data-plane surface is byte-stream based:
| Remote write stream | `InternodeDataTransport::open_write(...) -> FileWriter` | Callers pass borrowed `&[u8]` slices into boxed `AsyncWrite`; the backend owns any async body staging. | | Remote write stream | `InternodeDataTransport::open_write(...) -> FileWriter` | Callers pass borrowed `&[u8]` slices into boxed `AsyncWrite`; the backend owns any async body staging. |
| Walk-dir stream | `InternodeDataTransport::open_walk_dir(...) -> FileReader` | Same boxed stream model as read, with a small serialized request body. | | Walk-dir stream | `InternodeDataTransport::open_walk_dir(...) -> FileReader` | Same boxed stream model as read, with a small serialized request body. |
This API is correct for the current TCP/HTTP backend. A future non-default This API is correct for the current TCP/HTTP backend. The adapter contract
backend may have stricter memory ownership, buffer lifetime, or completion describes current ownership boundaries without assuming implementation details
requirements, so the adapter contract needs to describe those boundaries outside `TcpHttpInternodeDataTransport`.
without assuming a specific implementation.
## Buffer Ownership Model ## Buffer Ownership Model
| Buffer role | Allocator | Lifetime owner | Backend-specific state | TCP/HTTP behavior | | Buffer role | Allocator | Lifetime owner | Transport state | TCP/HTTP behavior |
| --- | --- | --- | --- | --- | | --- | --- | --- | --- | --- |
| Send buffer | Caller or RustFS-owned pool | Caller until the backend accepts the buffer; backend until completion if an owned-buffer API is used | Optional backend-managed buffer, staging buffer, or registration handle | Copy into the existing `AsyncWrite` path when the backend cannot use the buffer directly. | | Send buffer | Caller or RustFS-owned pool | Caller until the writer copies or accepts bytes for the HTTP body | Existing HTTP body staging | Copy into the existing `AsyncWrite` path when the writer cannot use the borrowed slice directly. |
| Receive buffer | Caller-provided storage or backend-owned receive pool | Backend while filling; caller after completion if ownership is returned | Optional backend-owned receive buffer or backend-specific registration | Copy from `AsyncRead` into caller storage as today. | | Receive buffer | Caller-provided storage | Reader while filling; caller after `poll_read` returns | Existing HTTP response body chunks | Copy from `AsyncRead` into caller storage as today. |
| Control metadata | RustFS caller | Caller/request object | Not buffer-managed by the data-plane backend | Serialize into HTTP/gRPC/control-plane messages. | | Control metadata | RustFS caller | Caller/request object | Not buffer-managed by the data-plane backend | Serialize into HTTP/gRPC/control-plane messages. |
| Fallback staging | TCP/HTTP backend | TCP/HTTP backend | No backend-specific registration | Existing `HttpReader`/`HttpWriter` buffering semantics. | | Fallback staging | TCP/HTTP backend | TCP/HTTP backend | Existing `HttpReader`/`HttpWriter` buffers | Existing buffering semantics. |
Backends with stricter memory requirements must not let callers mutate or reuse The current writer must not retain borrowed caller slices beyond the write call.
a buffer while an async transfer is still in flight. A backend that cannot use a When bytes must outlive the call, they are copied into owned HTTP body chunks.
caller buffer directly must either reject the transfer before payload movement
or copy through a clearly documented backend-owned staging buffer.
Zero-copy is not guaranteed by this contract. Backends must document whether This contract does not claim zero-copy behavior. The current TCP/HTTP path
their path is zero-copy, copy-reduced, or staging-buffer based, and must documents where copies occur.
document where copies occur.
## Compatibility Contract ## Compatibility Contract
@@ -66,77 +57,35 @@ pub trait InternodeDataTransport {
} }
``` ```
Future extensions for backend-managed buffers should be additive and ## Current Adapter Contract
capability-gated. A possible shape is:
```rust
pub struct TransferBuffer {
pub bytes: bytes::Bytes,
pub backend_state: Option<TransportBufferState>,
}
pub struct CompletedTransfer {
pub bytes: bytes::Bytes,
pub transfer_len: usize,
}
#[async_trait::async_trait]
pub trait BackendBufferInternodeTransport: InternodeDataTransport {
async fn write_backend_buffer(&self, request: WriteStreamRequest, buffer: TransferBuffer) -> Result<CompletedTransfer>;
async fn read_backend_buffer(&self, request: ReadStreamRequest, buffer: TransferBuffer) -> Result<CompletedTransfer>;
}
```
The concrete type of `TransportBufferState` should stay backend-private. The
generic contract only needs to state whether the buffer is usable by the
backend, who owns it during transfer, and when ownership returns.
This PR does not add this extension. It documents the boundary that a future
external backend may need.
## Required Contract for Stricter Backends
| Area | Required contract | | Area | Required contract |
| --- | --- | | --- | --- |
| Ownership | Define when caller-owned bytes become backend-owned and when ownership returns. | | Ownership | Define when caller-owned bytes are copied or accepted by the transport. |
| Completion | Signal completion before RustFS can reuse or mutate backend-managed memory. | | Completion | Return from stream operations only when bytes are accepted or an error is reported. |
| Staging | Declare whether the backend copies through backend-owned staging buffers when direct use is unavailable. | | Staging | Keep staging behavior inside the TCP/HTTP implementation. |
| Size limits | Expose any RustFS-visible `max_transfer_size`. | | Size limits | Report any RustFS-visible `max_transfer_size`; TCP/HTTP currently reports none. |
| Ordering | Either provide ordered delivery or include a reassembly layer before exposing stream semantics. | | Ordering | Preserve ordered byte-stream semantics. |
| Copy accounting | Document every known copy boundary and avoid claiming zero-copy unless the path proves it. | | Copy accounting | Document known copy boundaries and avoid unmeasured zero-copy claims. |
## Optional Optimizations
| Area | Optional behavior |
| --- | --- |
| Buffer pooling | RustFS may keep reusable pools for send and receive buffers. |
| Backend state cache | A backend may cache backend-specific handles for long-lived buffers. |
| Scatter/gather | A backend may accept multiple shard slices without repacking when its completion model can report per-slice ownership safely. |
| Backend-owned receive | A backend may return owned receive chunks instead of filling caller-provided buffers. |
These are optional optimizations, not requirements for the OSS TCP/HTTP path.
## Current API Limitations ## Current API Limitations
| Current API | Limitation for stricter backends | | Current API | Current limitation |
| --- | --- | | --- | --- |
| `FileReader = Box<dyn AsyncRead + Send + Sync + Unpin>` | `AsyncRead` exposes temporary caller `ReadBuf` storage, not a stable backend-managed receive buffer or explicit completion token. | | `FileReader = Box<dyn AsyncRead + Send + Sync + Unpin>` | `AsyncRead` exposes temporary caller `ReadBuf` storage. |
| `FileWriter = Box<dyn AsyncWrite + Send + Sync + Unpin>` | `AsyncWrite::poll_write` receives borrowed `&[u8]` that cannot outlive the poll, so async direct transfer requires copying or a different ownership API. | | `FileWriter = Box<dyn AsyncWrite + Send + Sync + Unpin>` | `AsyncWrite::poll_write` receives borrowed `&[u8]` that cannot outlive the poll. |
| `HttpWriter` | The async HTTP body must own `Bytes`, so borrowed write buffers are copied into `BytesMut` or `Bytes`. | | `HttpWriter` | The async HTTP body must own `Bytes`, so borrowed write buffers are copied into `BytesMut` or `Bytes`. |
| `write_body_chunks_to_writer` | Server-side HTTP body chunks are copied into `BytesMut` before local disk write. | | `write_body_chunks_to_writer` | Server-side HTTP body chunks are copied into `BytesMut` before local disk write. |
| Erasure encode output | Encoded shards are represented as `Vec<Bytes>` and written through `AsyncWrite`, not a completion-aware backend-buffer API. | | Erasure encode output | Encoded shards are represented as `Vec<Bytes>` and written through `AsyncWrite`. |
| Erasure decode input | Shard reads allocate `Vec<u8>` buffers before decode; no backend-owned receive pool is visible at the transport boundary. | | Erasure decode input | Shard reads allocate `Vec<u8>` buffers before decode. |
These limitations do not block the current `tcp-http` backend. They describe These limitations do not block the current `tcp-http` backend.
where a future external backend would need staging or an additive API.
## External Backend Crate Compatibility ## Adapter Stability
`InternodeDataTransport` should remain implementable by future backends without `InternodeDataTransport` should keep RustFS core data-plane logic separate from
modifying RustFS core data-plane logic. In the short term, the trait and the concrete transport implementation. The trait and `tcp-http` backend remain
`tcp-http` backend may remain inside `ecstore`. inside `ecstore`.
A future external or separately maintained backend could live in a separate This PR does not perform a crate split, add runtime loading, introduce a plugin
crate if the trait, request/response types, capability report, and error model system, add a backend value, or implement a new transport backend.
are public and stable enough. This PR does not perform a crate split, add
runtime loading, or introduce a plugin system.
@@ -14,17 +14,14 @@ The OSS scope is:
- keep `tcp-http` as the default backend; - keep `tcp-http` as the default backend;
- keep existing TCP/HTTP behavior unchanged; - keep existing TCP/HTTP behavior unchanged;
- document copy hotspots and ownership gaps for maintainable transport code; - document copy hotspots and ownership gaps for maintainable transport code;
- avoid hardware-specific dependencies or backend implementations. - avoid adding dependencies or backend implementations.
The OSS scope is not: The OSS scope is not:
- RDMA support; - adding another transport backend;
- DPU support; - replacing the current TCP/HTTP path;
- DOCA support; - adding benchmark plans for another transport;
- BlueField support; - changing object correctness semantics.
- RoCE/InfiniBand support;
- hardware benchmark planning;
- hardware-specific backend implementation.
## Scope ## Scope
@@ -1,8 +1,7 @@
# Internode Transport Capabilities # Internode Transport Capabilities
Status: design note for backend-neutral capability reporting. This document Status: design note for backend-neutral capability reporting. This document
does not add a backend and does not require specialized hardware or transport does not add a backend or transport crate.
crates.
## Open-source Scope ## Open-source Scope
@@ -12,24 +11,20 @@ The OSS scope is:
- keep `tcp-http` as the default backend; - keep `tcp-http` as the default backend;
- keep existing TCP/HTTP behavior unchanged; - keep existing TCP/HTTP behavior unchanged;
- document the capability fields needed for maintainable transport code; - document the capability fields needed for maintainable transport code;
- avoid hardware-specific dependencies or backend implementations. - avoid adding dependencies or backend implementations.
The OSS scope is not: The OSS scope is not:
- RDMA support; - adding another transport backend;
- DPU support; - replacing the current TCP/HTTP path;
- DOCA support; - adding benchmark plans for another transport;
- BlueField support; - changing object correctness semantics.
- RoCE/InfiniBand support;
- hardware benchmark planning;
- hardware-specific backend implementation.
## Purpose ## Purpose
`InternodeDataTransportCapabilities` describes what a backend can honestly do `InternodeDataTransportCapabilities` describes what a backend can honestly do
for RustFS internode data-plane transfers. The fields are intentionally neutral: for RustFS internode data-plane transfers. The fields describe observable
they can describe the current TCP/HTTP backend and a future non-default backend adapter behavior without naming a specific transport implementation.
without naming a specific transport implementation.
The capability report is descriptive. It does not select a backend, negotiate The capability report is descriptive. It does not select a backend, negotiate
with peers, or weaken object correctness semantics. with peers, or weaken object correctness semantics.
@@ -41,8 +36,6 @@ with peers, or weaken object correctness semantics.
| `streaming_read` | The backend can open a remote disk reader for `read_file_stream`. | | `streaming_read` | The backend can open a remote disk reader for `read_file_stream`. |
| `streaming_write` | The backend can open a remote disk writer for `create_file` or `append_file`. | | `streaming_write` | The backend can open a remote disk writer for `create_file` or `append_file`. |
| `streaming_walk_dir` | The backend can stream `walk_dir` responses. | | `streaming_walk_dir` | The backend can stream `walk_dir` responses. |
| `zero_copy_candidate` | The backend has an API shape that could avoid an extra user-space payload copy. This is not a promise that every transfer is zero-copy. |
| `registered_memory_required` | The backend requires pinned, registered, staged, or otherwise backend-managed buffers before payload transfer. |
| `ordered_delivery` | Bytes for each opened transfer are delivered in order. | | `ordered_delivery` | Bytes for each opened transfer are delivered in order. |
| `max_transfer_size` | Optional RustFS-level cap for a single transfer. `None` means no additional cap beyond the backend/protocol/runtime limits. | | `max_transfer_size` | Optional RustFS-level cap for a single transfer. `None` means no additional cap beyond the backend/protocol/runtime limits. |
| `fallback_supported` | The backend can participate in the behavior-preserving TCP fallback path. | | `fallback_supported` | The backend can participate in the behavior-preserving TCP fallback path. |
@@ -56,21 +49,17 @@ The default TCP/HTTP backend reports only capabilities it actually provides:
| `streaming_read` | `true` | `HttpReader` streams `/rustfs/rpc/read_file_stream` responses. | | `streaming_read` | `true` | `HttpReader` streams `/rustfs/rpc/read_file_stream` responses. |
| `streaming_write` | `true` | `HttpWriter` streams `/rustfs/rpc/put_file_stream` request bodies. | | `streaming_write` | `true` | `HttpWriter` streams `/rustfs/rpc/put_file_stream` request bodies. |
| `streaming_walk_dir` | `true` | `HttpReader` streams `/rustfs/rpc/walk_dir` responses. | | `streaming_walk_dir` | `true` | `HttpReader` streams `/rustfs/rpc/walk_dir` responses. |
| `zero_copy_candidate` | `false` | The current path exposes `AsyncRead`/`AsyncWrite` and HTTP body chunks; it copies through normal user-space buffers. |
| `registered_memory_required` | `false` | TCP/HTTP does not require RustFS-managed pinned, registered, or backend-owned buffers. |
| `ordered_delivery` | `true` | Each HTTP request body or response body is consumed as an ordered byte stream. | | `ordered_delivery` | `true` | Each HTTP request body or response body is consumed as an ordered byte stream. |
| `max_transfer_size` | `None` | RustFS does not impose an extra per-transfer cap at the capability layer. | | `max_transfer_size` | `None` | RustFS does not impose an extra per-transfer cap at the capability layer. |
| `fallback_supported` | `true` | TCP/HTTP is the behavior-preserving default and fallback path. | | `fallback_supported` | `true` | TCP/HTTP is the behavior-preserving default and fallback path. |
## Non-default Backend Fit ## Capability Compatibility
A future non-default backend can be described without changing the meaning of Any new capability field should describe observable RustFS behavior without
the existing TCP report: assuming a specific transport implementation:
| Capability shape | Interpretation | | Capability shape | Interpretation |
| --- | --- | | --- | --- |
| `zero_copy_candidate=true`, `registered_memory_required=true` | The backend can only use its lower-copy path with buffers that satisfy backend-specific ownership or registration rules. |
| `zero_copy_candidate=true`, `registered_memory_required=false` | The backend may expose owned chunks or another lower-copy path without requiring caller-managed registration. |
| `max_transfer_size=Some(n)` | The backend has a RustFS-visible transfer size ceiling and callers must split larger transfers or use fallback behavior. | | `max_transfer_size=Some(n)` | The backend has a RustFS-visible transfer size ceiling and callers must split larger transfers or use fallback behavior. |
| `ordered_delivery=false` | The backend cannot be used behind the current stream API without an ordering or reassembly layer. | | `ordered_delivery=false` | The backend cannot be used behind the current stream API without an ordering or reassembly layer. |
@@ -10,17 +10,11 @@ does not change production behavior.
The open-source RustFS path keeps `tcp-http` as the default internode data The open-source RustFS path keeps `tcp-http` as the default internode data
transport. This document defines adapter contracts only: transport. This document defines adapter contracts only:
- no production RDMA, DPU, DOCA, BlueField, DPDK, SPDK, or hardware - no additional production backend is introduced;
acceleration backend is introduced; - no dependency is added;
- no hardware SDK, `libibverbs`, `rdma-core`, or vendor dependency is added;
- no new accepted production backend value is added; - no new accepted production backend value is added;
- future external or separately maintained backends may implement the same - RustFS core data-plane logic remains independent of the concrete transport
adapter boundary without changing RustFS core data-plane logic. implementation.
Examples of possible future external backends include DOCA/BlueField,
RDMA/RoCE, or other DPU/NIC implementations. These are examples only and are
not implemented or scheduled by this design. Hardware-specific backend plans
are out of scope for this document.
## Static Backend Selection ## Static Backend Selection
@@ -34,7 +28,7 @@ Static config is the first selection model. Existing accepted values remain:
| any unsupported value | Fail closed with a diagnostic naming `RUSTFS_INTERNODE_DATA_TRANSPORT` and the invalid value. | | any unsupported value | Fail closed with a diagnostic naming `RUSTFS_INTERNODE_DATA_TRANSPORT` and the invalid value. |
Unknown backend values must fail closed. Unsupported backend values must fail Unknown backend values must fail closed. Unsupported backend values must fail
closed. A future external backend must be explicitly enabled and must not closed. Any additional backend value must be explicitly added and must not
silently replace `tcp-http`. silently replace `tcp-http`.
Backend selection must expose an observable backend identity for metrics, logs, Backend selection must expose an observable backend identity for metrics, logs,
@@ -50,7 +44,7 @@ payload.
| --- | --- | --- | --- | | --- | --- | --- | --- |
| Unsupported configured backend | Fail closed during transport construction. | Fall back only when a separately configured policy explicitly allows unsupported-backend fallback. | Error includes config key and invalid value; fallback event is counted when fallback is enabled. | | Unsupported configured backend | Fail closed during transport construction. | Fall back only when a separately configured policy explicitly allows unsupported-backend fallback. | Error includes config key and invalid value; fallback event is counted when fallback is enabled. |
| Peer does not support selected backend | Fail before payload transfer. | Use TCP/HTTP only when both local policy and peer policy allow it. | Count peer mismatch and selected fallback backend. | | Peer does not support selected backend | Fail before payload transfer. | Use TCP/HTTP only when both local policy and peer policy allow it. | Count peer mismatch and selected fallback backend. |
| Capability mismatch | Fail before payload transfer. | Use TCP/HTTP only if TCP satisfies the operation and policy allows fallback. | Record missing capability names or a low-cardinality reason. | | Capability mismatch | Fail before payload transfer. | Use TCP/HTTP only if it satisfies the operation and policy allows fallback. | Record missing capability names or a low-cardinality reason. |
| Connection setup failure | Fail the operation. | Retry on TCP/HTTP only when fallback is allowed and no payload bytes have transferred. | Count setup failure, retry, fallback backend, and fallback result. | | Connection setup failure | Fail the operation. | Retry on TCP/HTTP only when fallback is allowed and no payload bytes have transferred. | Count setup failure, retry, fallback backend, and fallback result. |
| Partial transfer failure | Fail the operation and let existing object/quorum logic decide retry behavior. | Do not silently resume on another backend unless the transfer protocol can prove byte range, checksum, and idempotency boundaries. | Count partial failure with bytes completed. | | Partial transfer failure | Fail the operation and let existing object/quorum logic decide retry behavior. | Do not silently resume on another backend unless the transfer protocol can prove byte range, checksum, and idempotency boundaries. | Count partial failure with bytes completed. |
| Max transfer size exceeded | Fail before payload transfer or split at a higher layer. | Use TCP/HTTP if policy allows and TCP has no RustFS-level cap. | Record rejected size and selected backend. | | Max transfer size exceeded | Fail before payload transfer or split at a higher layer. | Use TCP/HTTP if policy allows and TCP has no RustFS-level cap. | Record rejected size and selected backend. |
@@ -61,15 +55,16 @@ uses them. A backend must define failure behavior before production use.
## Dynamic Negotiation Boundary ## Dynamic Negotiation Boundary
Dynamic negotiation, if added, belongs on the existing gRPC control plane. Data Dynamic negotiation is not implemented by this PR. If it is added later, it
transfer must start only after both peers agree on: belongs on the existing gRPC control plane. Data transfer must start only after
both peers agree on:
| Negotiated item | Required property | | Negotiated item | Required property |
| --- | --- | | --- | --- |
| Backend name | Both peers know the backend and have it enabled. | | Backend name | Both peers know the backend and have it enabled. |
| Capability set | Required capabilities match the operation. | | Capability set | Required capabilities match the operation. |
| Max transfer size | The selected operation fits or is split before transfer starts. | | Max transfer size | The selected operation fits or is split before transfer starts. |
| Buffer rules | Both peers agree whether backend-managed or staged buffers are required. | | Buffer rules | Both peers agree on staging and ownership rules. |
| Completion semantics | Both peers agree when a transfer is considered complete and when buffers may be reused. | | Completion semantics | Both peers agree when a transfer is considered complete and when buffers may be reused. |
| Security mode | Authentication and encryption requirements are satisfied before any out-of-band transfer. | | Security mode | Authentication and encryption requirements are satisfied before any out-of-band transfer. |
| Fallback policy | Both peers agree whether TCP/HTTP fallback is allowed for this operation. | | Fallback policy | Both peers agree whether TCP/HTTP fallback is allowed for this operation. |
@@ -95,12 +90,10 @@ health, quorum, timeout, and integrity semantics.
- Backend selection must preserve peer authentication. - Backend selection must preserve peer authentication.
- Fallback must not weaken encryption or authorization. - Fallback must not weaken encryption or authorization.
- Out-of-band data-plane transfers must still bind to the intended disk,
volume, path, request authority, and operation.
- Partial transfers must not bypass bitrot verification or erasure quorum - Partial transfers must not bypass bitrot verification or erasure quorum
handling. handling.
- Any future external backend must document whether it relies on the same - Any adapter implementation must preserve the same request authority, disk,
security boundary as TCP/HTTP or requires a separate deployment boundary. volume, path, and operation binding as the current TCP/HTTP path.
## Metrics and Observability Requirements ## Metrics and Observability Requirements
@@ -117,9 +110,9 @@ Metrics and logs must use low-cardinality labels or metadata:
- capability mismatch count; - capability mismatch count;
- fallback decision count. - fallback decision count.
Backends must not add high-cardinality labels such as object names, full paths, Adapter implementations must not add high-cardinality labels such as object
full URLs, peer-specific dynamic strings, memory addresses, or backend-specific names, full paths, full URLs, peer-specific dynamic strings, memory addresses,
buffer keys. or buffer identifiers.
## TCP/HTTP Compatibility ## TCP/HTTP Compatibility
@@ -128,17 +121,15 @@ uses ordinary byte streams, does not require backend-specific buffer
registration, and remains suitable as the fallback path when an explicit registration, and remains suitable as the fallback path when an explicit
fallback policy exists. fallback policy exists.
A future non-default backend must not change the correctness semantics of Any adapter implementation must not change the correctness semantics of
object writes, object reads, healing, bitrot verification, erasure quorum, object writes, object reads, healing, bitrot verification, erasure quorum,
timeouts, or disk health handling. timeouts, or disk health handling.
## External Backend Crate Compatibility ## Adapter Stability
`InternodeDataTransport` should remain implementable by future backends without `InternodeDataTransport` should keep RustFS core data-plane logic separate from
modifying RustFS core data-plane logic. In the short term, the trait and the concrete transport implementation. The trait and `tcp-http` backend remain
`tcp-http` backend may remain inside `ecstore`. inside `ecstore`.
A future external or separately maintained backend could live in a separate This PR does not perform a crate split, add runtime loading, introduce a plugin
crate if the trait, request/response types, capability report, and error model system, add a backend value, or implement a new transport backend.
are public and stable enough. This PR does not perform a crate split, add
runtime loading, or introduce a plugin system.
@@ -47,10 +47,6 @@ pub struct InternodeDataTransportCapabilities {
pub streaming_write: bool, pub streaming_write: bool,
/// Backend can stream walk-dir responses. /// Backend can stream walk-dir responses.
pub streaming_walk_dir: bool, pub streaming_walk_dir: bool,
/// Backend has an API shape that could avoid an extra user-space copy.
pub zero_copy_candidate: bool,
/// Backend cannot transfer payloads unless buffers are registered or pinned.
pub registered_memory_required: bool,
/// Backend preserves in-order delivery for each opened transfer. /// Backend preserves in-order delivery for each opened transfer.
pub ordered_delivery: bool, pub ordered_delivery: bool,
/// Largest payload the backend accepts for one transfer, or no RustFS-level cap. /// Largest payload the backend accepts for one transfer, or no RustFS-level cap.
@@ -65,8 +61,6 @@ impl InternodeDataTransportCapabilities {
streaming_read: true, streaming_read: true,
streaming_write: true, streaming_write: true,
streaming_walk_dir: true, streaming_walk_dir: true,
zero_copy_candidate: false,
registered_memory_required: false,
ordered_delivery: true, ordered_delivery: true,
max_transfer_size: None, max_transfer_size: None,
fallback_supported: true, fallback_supported: true,
@@ -241,8 +235,6 @@ mod tests {
streaming_read: true, streaming_read: true,
streaming_write: true, streaming_write: true,
streaming_walk_dir: true, streaming_walk_dir: true,
zero_copy_candidate: false,
registered_memory_required: false,
ordered_delivery: true, ordered_delivery: true,
max_transfer_size: None, max_transfer_size: None,
fallback_supported: true, fallback_supported: true,
@@ -251,11 +243,9 @@ mod tests {
} }
#[test] #[test]
fn tcp_http_capabilities_do_not_advertise_stricter_backend_features() { fn tcp_http_capabilities_are_conservative() {
let capabilities = TcpHttpInternodeDataTransport.capabilities(); let capabilities = TcpHttpInternodeDataTransport.capabilities();
assert!(!capabilities.zero_copy_candidate);
assert!(!capabilities.registered_memory_required);
assert!(capabilities.ordered_delivery); assert!(capabilities.ordered_delivery);
assert_eq!(capabilities.max_transfer_size, None); assert_eq!(capabilities.max_transfer_size, None);
assert!(capabilities.fallback_supported); assert!(capabilities.fallback_supported);