mirror of
https://github.com/openziti/ziti.git
synced 2026-10-26 05:30:03 +00:00
Paul Lorenz
1e85b668bf
Control channel send back-pressure was showing up as a p99 at the send timeout,
with no way to tell which handler was holding the receive goroutine or what it
was waiting on. A dump taken after the fact records the goroutine unwinding the
diagnostic rather than whatever it was blocked on, so the snapshot has to happen
while the handler is still in the handler. That costs a timer per message, which
is why this is opt-in rather than always on.
- wraps every control channel receive handler on both sides, timing it and
snapshotting all goroutines while a slow one is still in the handler
- installs nothing when disabled. A nil detector's Wrap returns the handler it
was given, so a process that has not asked for this pays no timer, no clock
read and no branch per message, rather than paying a check
- deduplicates dumps by a normalized signature so a process stuck in one place
writes one file and counts the repeats, rather than filling the disk with the
same picture
- names the subject of each dump, since the goroutine being diagnosed cannot be
picked out of the dump and would otherwise be filtered out as a singleton
- makes the thresholds and the dump tracking configurable, because what to dump
on varies by what is being chased: 500ms finds a wedged handler, lock
contention wants tens of milliseconds, and a rare event wants more distinct
dumps kept and less time between them
- builds one detector per process rather than per channel. Per channel would
quietly turn the dump interval and the budget of distinct dumps into per
connection limits, and those limits are what bound the disk a dump can cost
- refuses settings that would produce nothing, such as a zero threshold or no
dumps allowed, but only when enabled, so a config left over from an
investigation does not stop a process starting once it is switched off
- reports the router's own control channel state
It lives in common/diagnostics, shared by the controller and router rather than
duplicated per side, and is documented commented-out in the sample configs.
(cherry picked from commit d562afe1d1)