When reading CIDs from packets the current code makes some short-lived
allocations due to the use of `copy_to_bytes`, which allocate for 1.5% of
CPU samples. This change avoids this.
**Before:**
```
Sent 1073741824 bytes on 1 streams in 1.72s (594.74 MiB/s)
```
**After:**
```
Sent 1073741824 bytes on 1 streams in 1.70s (602.78 MiB/s)
```
This seems slightly more efficient than extending a `Vec`.
Would probably be even more efficient if there would be an intersection
method on `SentMap` insert of returning packet numbers one by one.
Peak perf before:
```
Sent 1073741824 bytes on 1 streams in 1.77s (577.02 MiB/s)
```
After:
```
Sent 1073741824 bytes on 1 streams in 1.75s (584.25 MiB/s)
```
This change increases the amount of `CRYPTO` data sent in packets during handshakes.
Instead of reserving a fixed amount of 17 bytes for frame overhead,
we reserve the exactly required amount of bytes to encode frame type
and offset, and only 2 bytes for the frame length. This leads to an
additional 12-13 bytes of payload data being used.
The current implementation continously resizes the datagram buffer if
GSO is enabled and further datagrams are appended. Some benchmarking
and profiling showed that this doesn't have too much of an impact with
the glibc allocator, the strategy proved rather inefficient with pooling
memory allocators like jemalloc and mimalloc. For those, the cost of
calling `realloc` is rather high.
I benchmarked a bunch of strategies to determine the most efficient way forward:
1. Continously resize output buffer (Current approach)
2. Reserve maximum buffer size upfront
3. Reserve space for a single datagram to minimize over-allocation for
tiny transmits. If this is not enough, reallocate once for maximum size
Based on the results of those, I am proposing to go for approach 2) and
simply allocate the maximum buffer size upfront, which yields maximum
efficiency for mimalloc + jemalloc.
## Benchmark results
### Glibc:
Baseline:
> Sent 1073741824 bytes on 1 streams in 1.79s (572.62 MiB/s)
Allocate for 1 MTU, then for `max_datagrams`:
> Sent 1073741824 bytes on 1 streams in 1.78s (576.31 MiB/s)
Allocate for `max_datagrams` upfront:
> Sent 1073741824 bytes on 1 streams in 1.79s (572.28 MiB/s)
### Mimalloc:
Baseline:
> Sent 1073741824 bytes on 1 streams in 1.84s (557.34 MiB/s)
Allocate for 1 MTU, then for `max_datagrams`:
> Sent 1073741824 bytes on 1 streams in 1.74s (587.76 MiB/s)
Allocate for `max_datagrams` upfront:
> Sent 1073741824 bytes on 1 streams in 1.71s (600.06 MiB/s)
### Jemalloc:
Baseline:
> Sent 1073741824 bytes on 1 streams in 1.86s (551.75 MiB/s)
Allocate for 1 MTU, then for `max_datagrams`:
> Sent 1073741824 bytes on 1 streams in 1.73s (592.50 MiB/s)
Allocate for `max_datagrams` upfront:
> Sent 1073741824 bytes on 1 streams in 1.72s (596.29 MiB/s)
This change introduces a second `RangeSet` type which is based on a
linear array/vector instead of a tree. For the purpose of tracking
ACK ranges we usually should not have a big number of disjoint ranges,
since those would only occur with severe fragmentation. Therefore the
main benefit of a tree which is able to find ranges in the middle is
rarely used.
The array-based RangeSet provides 2 benefits:
- There exists an inline representation, which avoids the need of heap
allocating ACK ranges for `SentFrames` for small ranges.
- Iterating over ranges should usually be faster since there is only
a single cache-friendly contiguous range.
Performance differences:
With the `BTreeMap` based `RangeSet`:
```
Sent 1073741824 bytes on 1 streams in 1.90s (538.89 MiB/s)
```
With the `TinyVec` based `RangeSet`:
```
Sent 1073741824 bytes on 1 streams in 1.78s (574.73 MiB/s)
```
A crypto session cannot return 1-RTT keys before handshaking is finished.
Currently, rustls contains unwrap() calls to deal with this, but hoisting
out the unwrap to the `Connection` level here seems more sensible.
We can adjust the call into rustls once we update rustls for other reasons.
This change adds additional config arguments to the echo test.
We are then using those arguments to run a higher number of
streams while utilizing low flow control windows to test flow
control updates.
This pollutes the log a lot and doesn't help debugging.
All cases where a packet is not authenticated should already be
logged in different places, so just remove this.
Matthias Einwag