refactor(ecstore): add mmap copy disk read name (#4061)

This commit is contained in:
Zhengchao An
2026-06-29 21:18:41 +08:00
committed by GitHub
parent 9567e7b3be
commit a7dd9603e9
7 changed files with 53 additions and 30 deletions
@@ -171,7 +171,7 @@ Classification:
| Remote shard read stream | `crates/ecstore/src/rpc/remote_disk.rs::RemoteDisk::read_file_stream`; `crates/ecstore/src/rpc/internode_data_transport.rs::InternodeDataTransport::open_read`; `crates/ecstore/src/bitrot.rs::create_bitrot_reader` | `rustfs/src/storage/rpc/http_service.rs::handle_read_file` | Covered by `InternodeDataTransport` | Object GET, repair reads, and erasure decode use this path for remote shard bytes. |
| Remote shard write stream | `RemoteDisk::create_file`; `RemoteDisk::append_file`; `InternodeDataTransport::open_write`; `crates/ecstore/src/bitrot.rs::create_bitrot_writer` | `rustfs/src/storage/rpc/http_service.rs::handle_put_file` | Covered by `InternodeDataTransport` | Object PUT and multipart part upload use this path for remote shard bytes. |
| Remote namespace walk stream | `RemoteDisk::walk_dir`; `InternodeDataTransport::open_walk_dir`; `crates/ecstore/src/cache_value/metacache_set.rs` walk producers | `rustfs/src/storage/rpc/http_service.rs::handle_walk_dir` | Covered by `InternodeDataTransport` | High-volume listing/scanner/heal metadata stream. It is not object byte data, but it is a large internode stream. |
| Remote zero-copy read fallback | `RemoteDisk::read_file_zero_copy` | same as remote shard read stream | Covered by `InternodeDataTransport` through `read_file_stream` | The remote path buffers the stream into `Bytes`; true zero-copy is not guaranteed for remote disks. |
| Remote mmap-copy read fallback | `RemoteDisk::read_file_mmap_copy` | same as remote shard read stream | Covered by `InternodeDataTransport` through `read_file_stream` | The remote path buffers the stream into `Bytes`; true zero-copy is not guaranteed for remote disks. |
### Still Direct TCP/HTTP/gRPC
@@ -210,7 +210,7 @@ Classification:
| Medium | `ReadAll` and `WriteAll` still carry unary `bytes` over gRPC. They appear metadata-oriented today, but there is no size threshold or routing policy. |
| Medium | `ReadMultiple` can aggregate many metadata files into one gRPC response. |
| Low | Legacy gRPC `WalkDir` remains implemented while `RemoteDisk::walk_dir` uses HTTP through the transport. |
| Medium | Remote `read_file_zero_copy` is a buffered read over the transport, not a remote zero-copy contract. |
| Medium | Remote `read_file_mmap_copy` is a buffered read over the transport, not a remote zero-copy contract. |
| Medium | Server-side TCP HTTP route handling is outside the client-side trait. |
## Current Object Write Path
@@ -235,7 +235,7 @@ This is the primary write data-plane candidate.
For object GETs and repair reads in distributed erasure mode, the relevant flow is:
1. `SetDisks` prepares shard readers for the selected disks.
2. `create_bitrot_reader` uses local zero-copy only when `disk.is_local()`.
2. `create_bitrot_reader` uses local mmap-copy reads only when `disk.is_local()`.
3. For a remote disk, it calls `disk.read_file_stream(...)`.
4. `RemoteDisk::read_file_stream` delegates to
`InternodeDataTransport::open_read`.
@@ -385,7 +385,7 @@ The initial TCP backend can keep the current signed HTTP URLs internally.
`RemoteDisk` delegates only these methods to the data transport:
- `read_file_stream`
- `read_file_zero_copy` as the current wrapper over `read_file_stream`
- `read_file_mmap_copy` as the current wrapper over `read_file_stream`
- `append_file`
- `create_file`
- `walk_dir`
@@ -54,7 +54,7 @@ through `ParallelReader` in `crates/ecstore/src/erasure_coding/decode.rs` and
| Bitrot verification | `BitrotReader::read` | caller `&mut [u8]`, `hash_buf: Vec<u8>` | No additional payload copy | The bitrot reader reads hash bytes into `hash_buf` and payload bytes directly into the supplied output slice. Hash calculation reads the slice. |
| Erasure shard read | `ParallelReader::read` | `Vec<u8>` per shard | Yes | Each shard read allocates `vec![0u8; shard_size]`; data is filled there before decode/reconstruction. |
| Object response write | `write_data_blocks` | slices of shard `Vec<u8>` | No extra staging copy | Decoded data block slices are written to the target writer with `write_all`; the target may copy internally. |
| Remote zero-copy helper | `RemoteDisk::read_file_zero_copy` | `Vec<u8>` then `Bytes` | Yes | The remote implementation reads the full stream into a `Vec` and converts it into `Bytes`. It is a convenience fallback, not network zero-copy. |
| Remote mmap-copy helper | `RemoteDisk::read_file_mmap_copy` | `Vec<u8>` then `Bytes` | Yes | The remote implementation reads the full stream into a `Vec` and converts it into `Bytes`. It is a convenience fallback, not network zero-copy. |
## Write Stream
@@ -90,7 +90,7 @@ through `ParallelReader` in `crates/ecstore/src/erasure_coding/decode.rs` and
| 3 | `ParallelReader::read` shard buffers | High on read path | Each shard read allocates and fills a `Vec<u8>` before decode can proceed. This is also where degraded reads wait on quorum. |
| 4 | `ReaderStream::with_capacity` plus `StreamReader` | Medium on read path | Server file reads create `Bytes` chunks, then client `AsyncRead` copies those chunks into the caller's `ReadBuf`. |
| 5 | `Erasure::encode` block and shard materialization | Medium on write path | Source data is first read into a block `Vec<u8>`, then encoded into per-shard `Bytes`. This is necessary for the current erasure API. |
| 6 | `RemoteDisk::read_file_zero_copy` | Medium when used | Remote zero-copy reads buffer the whole stream into memory. The name does not mean zero-copy over the network. |
| 6 | `RemoteDisk::read_file_mmap_copy` | Medium when used | Remote mmap-copy reads buffer the whole stream into memory. The name does not mean zero-copy over the network. |
| 7 | URL/query/header/JSON serialization | Low | Metadata copies are small and not on the large payload hot path. |
## Adapter Ownership Gaps
@@ -117,6 +117,6 @@ through `ParallelReader` in `crates/ecstore/src/erasure_coding/decode.rs` and
7. The HTTP auth and URL construction boundary is part of the current TCP/HTTP
backend. A non-HTTP backend would need equivalent peer authentication and
disk addressing without assuming URL query parameters.
8. Local disk zero-copy exists only for local reads via `read_file_zero_copy`.
8. Local disk mmap-copy reads exist only for local reads via `read_file_mmap_copy`.
Remote disks deliberately fall back to network streaming and full-buffer
collection for the zero-copy helper.
collection for the legacy zero-copy helper.