mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-10 15:16:56 +00:00
houseme
7172e151de
Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: houseme <housemecn@gmail.com>
66 lines
2.3 KiB
Markdown
66 lines
2.3 KiB
Markdown
# Concurrent Download Tool (tests)
|
|
|
|
This tool downloads multiple URLs concurrently and saves files to a target directory.
|
|
|
|
Saved filename format:
|
|
|
|
`<nanoseconds>_<index>_<original_filename>`
|
|
|
|
All downloaded files are written into one output directory.
|
|
|
|
## Environment variables
|
|
|
|
- `DOWNLOAD_URLS` (required): comma-separated URLs.
|
|
- `DOWNLOAD_OUTPUT_DIR` (optional): output directory, default `target/tmp/concurrent_downloads`.
|
|
- `DOWNLOAD_CONCURRENCY` (optional): max concurrent downloads, default `8`.
|
|
- `DOWNLOAD_REPEAT` (optional): repeat count per URL, default `1`.
|
|
- `DOWNLOAD_MAX_RETRIES` (optional): retry count per task after first failure, default `0`.
|
|
- `DOWNLOAD_RETRY_BACKOFF_MS` (optional): fixed backoff between retries, default `200`.
|
|
|
|
## Statistics output
|
|
|
|
After run, the tool prints:
|
|
|
|
- total tasks
|
|
- succeeded
|
|
- failed
|
|
- total bytes
|
|
- elapsed ms
|
|
- throughput bps
|
|
- total attempts
|
|
- retried tasks
|
|
- retry attempts
|
|
- latency p50 ms
|
|
- latency p95 ms
|
|
- failure details (`[index] url => error`) when failures exist
|
|
|
|
If any task fails, the test returns error after printing the summary.
|
|
|
|
Retry is triggered only for recoverable cases:
|
|
|
|
- network/request timeout/connect errors
|
|
- HTTP `429`
|
|
- HTTP `5xx`
|
|
|
|
## Compile check
|
|
|
|
```bash
|
|
cargo test -p rustfs --test concurrent_download_tool --no-run
|
|
```
|
|
|
|
## Manual run example
|
|
|
|
The commands below are for manual execution only.
|
|
They are not part of automated test runs.
|
|
|
|
```bash
|
|
DOWNLOAD_URLS="http://127.0.0.1:9001/demo/google-cloud-aiplugin-1.46.1-253.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=HAXVOTZK9MLBJT8KWI4E%2F20260329%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260329T105159Z&X-Amz-Expires=86400&X-Amz-Security-Token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzUxMiJ9.eyJwYXJlbnQiOiJydXN0ZnNhZG1pbiIsImV4cCI6MTc3NDgyMDgyMX0.tYhQoPRcg0Ysx4KVw9ez7ZpYxsqGgqomtsP_iaeTsKzoii8EVNt74BZm2wbUjXW-FbGXc1pqEYX6wZ5Ncpk9Iw&X-Amz-Signature=15f47b19832f53b34f9e0fe1862d53d71660bbf8f1a512669bb2d041ac8d0697&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject" \
|
|
DOWNLOAD_OUTPUT_DIR="/Users/zhi/Documents/code/rust/rustfs/rustfs/target/tmp/concurrent_downloads" \
|
|
DOWNLOAD_CONCURRENCY="40" \
|
|
DOWNLOAD_REPEAT="40" \
|
|
DOWNLOAD_MAX_RETRIES="2" \
|
|
DOWNLOAD_RETRY_BACKOFF_MS="300" \
|
|
cargo test -p rustfs --test concurrent_download_tool -- --ignored --nocapture
|
|
```
|
|
|