Previously, these sometimes depended on whether the peer had
acknowledged an operation, which might lead users to incorrectly
believe that attempting to operate on a stream after stopping or
resetting it is sane.
So far those haven't been consumable by applications.
This makes the struct public, and adds an accessor
for `quinn_proto::Connection` and `quinn::Connection`.
Reserving capacity can provide more capacity than we asked for.
However we are not allowed to write more than MTU size. Therefore
the maximum allowed capacity is tracked separately.
The outgoing cmsgs can't be uninit because it breaks `CMSG_NXTHDR`.
The function will try to access the next cmsg in the buffer at
https://github.com/rust-lang/libc/blob/ae65df55fe2d0e1348fed575bfc8c0c3e4744c65/src/unix/linux_like/linux/mod.rs#L2585-L2589
This will yield an undefined result, and can let `CMSG_NEXT` return `None`.
This prevents more than 1 CMSG to be encoded. 0-initializing the
messages makes sure a length of 0 is read, and fixes the issue.
In addition this change improves an assert in the cmsg encoder, which
makes it easier to determine the required CMSG space.
This change improves the bulk benchmark:
- The total amount of requests the benchmark performs can be configured
- The amount of concurrent requests on a given connection can be
configured
- The amount of data to transfer per stream can be configured
E.g.
```
bulk --streams 300 --max_streams 100 --stream_size 10
```
will create 300 streams - with a maximum of 100 active streams at a time,
and send 10MB of data on each stream.
The "short" cli flags are taken form h2load where applicable.
The benchmark now also shows metrics which indicate the performance
of individual streams, which allows to judge fairness. Example output:
```
Overall stats:
Sent 3145728000 bytes on 300 streams in 12.11s (247.81 MiB/s)
Stream metrics:
│ Throughput │ Duration
──────┼───────────────┼──────────
AVG │ 149.17 MiB/s │ 396.00ms
P0 │ 0.83 MiB/s │ 39.00ms
P10 │ 51.19 MiB/s │ 40.00ms
P50 │ 129.75 MiB/s │ 77.00ms
P90 │ 247.00 MiB/s │ 195.00ms
P100 │ 254.75 MiB/s │ 12.11s
```
When a listener is bound to multiple network interfaces (e.g. `::0`),
it is not obvious which IP the peer used to send a packet. We however
might need this information to send packets back to the peer with the
same source address.
This problem is described in #508.
This change makes the destination IP address which was used to send
the initial packet available in the `Conneting` and `Connection` types.
The information is far available only on Linux due to missing test on
other platforms.
This extends the "cargo bench" benchmark to test transmission of data
on multiple streams in parallel.
Ideally we would want all transmission to take a similar amount of time
(bandwidth shared fairly), and the overall throughput to stay the same
or increase (but never degrade) when multiple streams are in use.
The benchmark can not show the fairness aspect very well, since it only
prints an overall throughput. But it shows that performance with
multiple streams stays similar.
Output:
```
running 4 tests
test large_data_10_streams ... bench: 180,761,260 ns/iter (+/- 12,090,679) = 58 MB/s
test large_data_1_stream ... bench: 18,459,500 ns/iter (+/- 3,804,219) = 56 MB/s
test small_data_100_streams ... bench: 4,600,890 ns/iter (+/- 373,842)
test small_data_1_stream ... bench: 71,019 ns/iter (+/- 34,298)
```
I also increased the size of the LARGE_DATA payload from 128kB to
1MB, since 128kB didn't show quinn peak performance.
The current version of send multiplexing tries to send as much data from
a single stream as possible before switching over to the next stream.
The only situation where data from another stream than the original one
is sent is if there are no peer flow control credits available anymore,
or if the application didn't feed new data.
If none of this situations occur, other streams will be starved.
This is typically not what people expect, and can lead to high latencies
on streams which are queued behind earlier streams.
This change makes the send behavior "fairer" by enqueuing stream which
just have sent data **after** all other pending streams instead of in
front of them. This is simply achieved by switching to a `VecDeque` and
using the the right insert/remove orders.
Stats (obtained via #946), when running
```
./bulk --streams 300 --max_streams 10 --stream_size 10
```
```
Stream metrics:
AVG : Throughput: 172.38 MiB/s, Duration: 395.00ms
P0 : Throughput: 0.83 MiB/s, Duration: 39.00ms
P10 : Throughput: 49.19 MiB/s, Duration: 40.00ms
P50 : Throughput: 232.50 MiB/s, Duration: 42.00ms
P90 : Throughput: 245.87 MiB/s, Duration: 203.00ms
P100: Throughput: 251.25 MiB/s, Duration: 12.08s
```
```
Stream metrics:
AVG : Throughput: 24.20 MiB/s, Duration: 413.00ms
P0 : Throughput: 21.42 MiB/s, Duration: 279.00ms
P10 : Throughput: 23.25 MiB/s, Duration: 403.00ms
P50 : Throughput: 24.34 MiB/s, Duration: 410.00ms
P90 : Throughput: 24.81 MiB/s, Duration: 430.00ms
P100: Throughput: 35.78 MiB/s, Duration: 466.00ms
```
With this change throughput on streams is a lot more consistent than
before.
This commit changes all code-path which update `MAX_DATA`,
`MAX_STREAM_DATA` and `STOP_SENDING` to use a new `ShouldTransmit` type
to indicate whether an outgoing frame needs to be enqueued. The type is
labeled with `#[must_use]` - which makes it harder to miss a place which
requires sending a frame.
Dirkjan Ochtman