mirror of
https://github.com/openziti/ziti.git
synced 2026-09-11 13:29:03 +00:00
Paul Lorenz
d562afe1d1
Control channel send back-pressure was showing up as a p99 at the send timeout, with no way to tell which handler was holding the receive goroutine or what it was waiting on. A dump taken after the fact records the goroutine unwinding the diagnostic rather than whatever it was blocked on, so the snapshot has to happen while the handler is still in the handler. That costs a timer per message, which is why this is opt-in rather than always on. - wraps every control channel receive handler on both sides, timing it and snapshotting all goroutines while a slow one is still in the handler - installs nothing when disabled. A nil detector's Wrap returns the handler it was given, so a process that has not asked for this pays no timer, no clock read and no branch per message, rather than paying a check - deduplicates dumps by a normalized signature so a process stuck in one place writes one file and counts the repeats, rather than filling the disk with the same picture - names the subject of each dump, since the goroutine being diagnosed cannot be picked out of the dump and would otherwise be filtered out as a singleton - makes the thresholds and the dump tracking configurable, because what to dump on varies by what is being chased: 500ms finds a wedged handler, lock contention wants tens of milliseconds, and a rare event wants more distinct dumps kept and less time between them - builds one detector per process rather than per channel. Per channel would quietly turn the dump interval and the budget of distinct dumps into per connection limits, and those limits are what bound the disk a dump can cost - refuses settings that would produce nothing, such as a zero threshold or no dumps allowed, but only when enabled, so a config left over from an investigation does not stop a process starting once it is switched off - reports the router's own control channel state It lives in common/diagnostics, shared by the controller and router rather than duplicated per side, and is documented commented-out in the sample configs.