mirror of
https://github.com/openziti/ziti.git
synced 2026-10-25 21:20:03 +00:00
Paul Lorenz
28e5d0188f
A fleet reconnecting at once ran every connect setup concurrently, each building links and validating terminators, with nothing bounding how many were in flight. The work itself is sub-second; the failure is in the aggregate, and nothing reported it, so a controller burying itself under reconnects looked from the outside like one that had simply gone quiet. - splits the connect into the decision and the setup: the decision stays synchronous, so the bind handler can still fail a refused connect and at most one connection per router is current at a time - runs the setup on a goroutine bounded by a semaphore, defaulting to 200 at once, configurable as routerConnectConcurrency - never refuses for capacity: a refusal fails the bind and strands the router's channel group, and the expensive part of connecting is already paid by the time this is reached, so the setup waits instead - re-acquires the per-router lock and re-checks currency inside the setup, so it is serialized against a concurrent disconnect and skipped entirely if the connection was superseded while queued - gives up after a configurable timeout and closes the control channel rather than skipping the setup, since skipping leaves a router that is connected, believes it is, and has no links built - reports the queue depth as a gauge and the wait and the work as separate timers, since the two have opposite remedies and the timers record only on completion, so the gauge is the one signal visible during a wedge