mirror of
https://github.com/rustfs/rustfs.git
synced 2026-10-04 12:31:36 +00:00
Dae-Cheol Noh
1f04a12abf
* fix(ftps): bound upload memory with multipart streaming * fix(ftps): keep multipart upload within driver module
72 lines
3.3 KiB
Markdown
72 lines
3.3 KiB
Markdown
# FTPS upload memory regression
|
|
|
|
The old `FtpsDriver::put` copied the entire input into a `Vec` before calling
|
|
`PutObject`. Wrapping that completed allocation in `StreamingBlob` did not make
|
|
input consumption streaming. It also sent large files through the 5 GiB
|
|
single-PUT path.
|
|
|
|
## Automated regression
|
|
|
|
```sh
|
|
cargo test -p rustfs-protocols --no-default-features --features ftps --lib
|
|
```
|
|
|
|
`ftps_upload_streams_before_eof_and_preserves_part_bytes` uses a reader that
|
|
refuses to supply the next part until the previous part reaches the backend.
|
|
Restoring the driver from commit `9c30cc88513c5e1b5443b6ee86bf4bc7965e79e9`
|
|
while retaining the new test harness makes this test fail with
|
|
`read ahead of uploaded part`. The multipart implementation passes. Tests also
|
|
check exact bytes, empty files, part boundaries, part numbering, errors,
|
|
permissions, cancellation cleanup, and buffer capacity.
|
|
|
|
## Manual memory probe
|
|
|
|
The ignored test `ftps_large_upload_memory_probe` creates concurrent synthetic
|
|
readers. Each reader yields between reads so all uploads are active together.
|
|
It calls the real FTPS storage driver with a scripted storage backend that
|
|
returns successful S3 responses and discards the payload. It neither stores
|
|
files nor opens network connections. No large fixture is required.
|
|
|
|
Build the test executable first; measuring Cargo would include compiler memory:
|
|
|
|
```sh
|
|
export CARGO_PROFILE_DEV_DEBUG=0 CARGO_PROFILE_TEST_DEBUG=0 CARGO_INCREMENTAL=0
|
|
cargo test -p rustfs-protocols --no-default-features --features ftps --lib --no-run
|
|
```
|
|
|
|
Use the executable path printed by that command as `test_binary`, then on macOS:
|
|
|
|
```sh
|
|
/usr/bin/time -l "$test_binary" --ignored --exact \
|
|
ftps::driver::upload::tests::ftps_large_upload_memory_probe --nocapture
|
|
|
|
RUSTFS_FTPS_TEST_BYTES=10737418240 RUSTFS_FTPS_TEST_CONCURRENCY=10 \
|
|
/usr/bin/time -l "$test_binary" --ignored --exact \
|
|
ftps::driver::upload::tests::ftps_large_upload_memory_probe --nocapture
|
|
```
|
|
|
|
On Linux use `/usr/bin/time -v` (its maximum RSS is reported in KiB). The default
|
|
probe uses 64 MiB per upload and 10 uploads. These environment variables are
|
|
**test-only**, not server configuration.
|
|
|
|
Observed on macOS arm64 with Rust 1.98.1, debug information disabled, one run per
|
|
case (maximum resident set size reported in bytes):
|
|
|
|
| Driver | Bytes per upload | Concurrent uploads | Maximum RSS |
|
|
| --- | ---: | ---: | ---: |
|
|
| Original driver at `9c30cc8`, same probe | 67,108,864 | 10 | 710,033,408 (677.1 MiB) |
|
|
| Bounded multipart driver | 67,108,864 | 10 | 179,781,632 (171.5 MiB) |
|
|
| Bounded multipart driver | 10,737,418,240 | 10 | 187,465,728 (178.8 MiB) |
|
|
|
|
For the original-driver comparison, only `ftps/driver.rs` was replaced with its
|
|
base-commit version in the test checkout; the identical probe and dummy backend
|
|
were retained, the executable rebuilt, and the 64 MiB case rerun. The original
|
|
driver was not subjected to the 100 GiB aggregate case.
|
|
|
|
These measurements isolate protocol-driver buffering, not total server RSS,
|
|
persisted-data correctness, TLS behavior, or storage throughput. The large case
|
|
supplies 100 GiB of generated bytes without retaining them. A real server also
|
|
uses memory for TLS, storage, caches, and allocator overhead. Memory still scales
|
|
with active upload count. See [FTPS operations](../operations/ftps.md) for limits
|
|
and cleanup behavior.
|