A fleet reconnecting at once ran every connect setup concurrently, each
building links and validating terminators, with nothing bounding how many were
in flight. The work itself is sub-second; the failure is in the aggregate, and
nothing reported it, so a controller burying itself under reconnects looked
from the outside like one that had simply gone quiet.
- splits the connect into the decision and the setup: the decision stays synchronous, so the bind handler can still fail a refused connect and at most one connection per router is current at a time
- runs the setup on a goroutine bounded by a semaphore, defaulting to 200 at once, configurable as routerConnectConcurrency
- never refuses for capacity: a refusal fails the bind and strands the router's channel group, and the expensive part of connecting is already paid by the time this is reached, so the setup waits instead
- re-acquires the per-router lock and re-checks currency inside the setup, so it is serialized against a concurrent disconnect and skipped entirely if the connection was superseded while queued
- gives up after a configurable timeout and closes the control channel rather than skipping the setup, since skipping leaves a router that is connected, believes it is, and has no links built
- reports the queue depth as a gauge and the wait and the work as separate timers, since the two have opposite remedies and the timers record only on completion, so the gauge is the one signal visible during a wedge
Control channel send back-pressure was showing up as a p99 at the send timeout,
with no way to tell which handler was holding the receive goroutine or what it
was waiting on. A dump taken after the fact records the goroutine unwinding the
diagnostic rather than whatever it was blocked on, so the snapshot has to happen
while the handler is still in the handler. That costs a timer per message, which
is why this is opt-in rather than always on.
- wraps every control channel receive handler on both sides, timing it and
snapshotting all goroutines while a slow one is still in the handler
- installs nothing when disabled. A nil detector's Wrap returns the handler it
was given, so a process that has not asked for this pays no timer, no clock
read and no branch per message, rather than paying a check
- deduplicates dumps by a normalized signature so a process stuck in one place
writes one file and counts the repeats, rather than filling the disk with the
same picture
- names the subject of each dump, since the goroutine being diagnosed cannot be
picked out of the dump and would otherwise be filtered out as a singleton
- makes the thresholds and the dump tracking configurable, because what to dump
on varies by what is being chased: 500ms finds a wedged handler, lock
contention wants tens of milliseconds, and a rare event wants more distinct
dumps kept and less time between them
- builds one detector per process rather than per channel. Per channel would
quietly turn the dump interval and the budget of distinct dumps into per
connection limits, and those limits are what bound the disk a dump can cost
- refuses settings that would produce nothing, such as a zero threshold or no
dumps allowed, but only when enabled, so a config left over from an
investigation does not stop a process starting once it is switched off
- reports the router's own control channel state
It lives in common/diagnostics, shared by the controller and router rather than
duplicated per side, and is documented commented-out in the sample configs.
(cherry picked from commit d562afe1d1)
The controller decided which of two racing connections for a router was current
by comparing router instances, but loaded one per connect by evicting the router
cache and reading back through it. Two connects could both evict, and whichever
read second was handed the instance the first had just published. A shared
instance makes the two connections indistinguishable: the connect path cannot
reject the second into an occupied slot, and when either channel dies the
disconnect path finds itself current and tears down the registration the other is
still using. The surviving channel is never re-bound, so the router stays
connected at the transport layer while absent from the model, unable to recover.
Connect and disconnect were also unserialized, so a stale or superseded
disconnect could interleave with a live connection and take its links with it.
- serializes a router's connect and disconnect with a per-router striped lock
- keeps at most one connection per router: a connect into an occupied slot is
rejected via an error from ConnectRouter, so the bind fails and NewChannel
closes it without starting rx or registering it, and the occupant is displaced;
the router redials into the freed slot
- displaces an occupant by closing its channel and also invoking the teardown
directly, since a channel that is already closed never fires its close handler
again; without this a dead but still registered connection holds the slot
forever and every redial is rejected against a slot nothing can free
- refuses a connect whose control channel is already closed rather than
registering it, so a connection no disconnect could ever remove is never
published
- gives every connection its own router instance via RouterManager.NewCtrlChanRouter,
read through readUncached so the cache neither supplies nor receives it, which
is what makes comparing instances meaningful
- moves recording the channel and connect time out of the accept path, so a
caller cannot attach the wrong channel or forget to attach one
- serializes link publication with that teardown on the same per-router stripe.
Validating currency and then publishing without it is a check-then-act: a
report can find the connection current and, by the time it reaches the link
manager, the teardown has already snapshotted and cleared the router's links,
so the link is recreated after everything that would have removed it. It is
then absent from the router's own index while still in the link table with a
disconnected source, and a reconnect reporting the same iteration can adopt
that stale source instead of rebuilding the link
- guards the entire DisconnectRouter teardown by connection currency, all or
nothing, and clears the connected flag and link index only when the
registration was actually given up, with the flag cleared under the same shard
lock as the map removal so the two cannot be observed disagreeing; the connected flag decides whether the
controller accepts a router's link reports, so clearing it for the wrong
connection silences a router that is up and reporting
- reduces MarkConnected to publishing the connection; the takeover-close moves
into ConnectRouter's reject path
- makes the per-router unlock idempotent so callers can defer it as a leak-safety
net and still unlock early before closing a channel outside the lock
- stops the replaced RouterSender in routerTxMap.Add so a takeover does not leak
the old sender's goroutine when the broker's asynchronous RouterDisconnected
loses the race to the redial's RouterConnected
- discards pending peer state changes for a router whose channel has closed,
since sending on one fails immediately and the failed send is retried as soon
as the event loop turns, spinning the loop and flooding the log
- queues the peer-state send-done event on every path, so a missing channel can
no longer leave sendInProgress set and stall that router's updates permanently
- resolves a router's version from its connected instance when validating link
conn info, since the version arrives in the hello and so is absent from an
instance loaded from the database
- normalizes both endpoints to the connected instance in shortestPath, which is
keyed and compared by pointer and so treated an endpoint held as any other
instance of the same router as absent from the graph, reporting a router as
unroutable from itself. That worked before only because the connect path
published its instance into the router cache, so a cache read and the connected
map returned the same object; nothing stated the requirement
- configures test logging once per package in TestMain, so a test no longer
writes global logger state while a previous test's shutdown logging reads it
- adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop
- translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup
- adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path
- re-evaluates dialers on link group and listener changes via RescanForDialOpportunities
For #3743.
- removes the costTags option from router link listeners, which was parsed
from config, advertised to the controller, and stored on the router model
but never used for path selection or any other behavior
- drops GetLinkCostTags from the xlink.Listener interface and its transport
implementation
- reserves the corresponding ctrl_pb.Listener.costTags and
RouterLinks.RouterLink.linkCostTags protobuf fields and regenerates ctrl.pb.go
- documents the removal in the changelog
- keys router capability bits off the sdk-golang RouterCapability enum
- adds a generic capabilities.Mask[T ~int] bitmask, centralizing capability
set/check behind one value-to-bit translation
- adds capabilities.RouterCapability and ControllerCapability types, with
RouterCapabilityMask/ControllerCapabilityMask aliases, so masks and checks are
typed per namespace
- supports control-plane-only router capabilities as negative values that index
down from the top of the mask, collision-free with the SDK's upward-numbered
bits and invisible to the SDK and edge-api
- references the sdk-golang RouterCapability enum as the source of truth for
shared router capability bits
- holds the router's advertised capability mask as an instance on the router env
rather than a global, so in-process test routers do not share state
- advertises PostureChecks and BindSuccess as capability bits on both channels
while still sending the legacy boolean edge headers for backwards compatibility
- routes GetCapabilities/IsCapable and the controller's Router.Capabilities field
through the typed mask
- adds a provenance test that verifies, via go/packages, that every positive
router capability is SDK-sourced, every negative is control-plane-only, and no
two resolve to the same bit
- update sdk to v2.0.0-pre2
- gives ListenerCtrlChannel Min: 0 constraints per underlay type plus
MinTotalUnderlays: 1 in its configs, so the controller accepts the
high/low-priority grouped underlays the router dials while still closing the
channel only when its last underlay is lost
- restores the multi-underlay behavior the v4 listener-side SetMinTotal(1)
provided, which the channel/v5 migration dropped
- works around channel/v5 not yet treating MinTotalUnderlays alone as a
multi-underlay signal
- switches the xlink transport and router ctrl listeners to NewClassicListenerWithAcceptor, passing the MultiListener as a HelloAcceptor
- replaces the controller ctrl channel's NewClassicListener/UnderlayDispatcher wiring with NewClassicListenerWithAcceptor and a TypeRoutingAcceptor, adapting the mesh acceptor via AsHelloAcceptor
- removes the multiListenerAcceptor wrapper now that MultiListener implements HelloAcceptor directly
- removes the xgress_edge Acceptor.Run Create-loop, handing underlays to the MultiListener through the acceptor-based listener
- moves the controller ctrl connect handler into ListenerConfig.ConnectionHandlers
- updates ctrlchan channel tests to the new constructor
- decomposes the ctrlchan, xlink and edge-listener channels onto the v5 Senders, MessageSourceProvider and UnderlayEventListener interfaces, replacing the v4 UnderlayHandler god-interface
- replaces the hand-rolled dial/grouping/backoff machinery with channel.BackoffDialPolicy and declarative Constraints; ctrl keeps survive-to-zero (Min: 0) with MinStableDuration: 0 for prompt reconnect, while xlink and edge default underlays keep Min: 1 so loss closes the channel
- builds grouped channels via channel.NewChannel(*Config) and moves handler retrieval to GetSenders()
- records the channel via InitChannel from each bind handler, before underlay events fire, so handlers registered during bind do not dereference a nil channel
- generates a group secret for ungrouped inbound ctrl underlays on the router accept path, matching the controller, since NewChannel requires one
- preserves link-id-as-channel-id (the link dial policy wraps the cloned link-id identity dialer) and adds a test asserting dialed underlays present the link id
- registers the latency handler explicitly, as it is no longer a self-describing receiver in v5
- channel.MultiChannel -> channel.Channel
- channel.MultiChannelConfig -> channel.Config
- channel.NewMultiChannel(...) -> channel.NewChannel(...)
channel/v5 unifies Channel and MultiChannel into a single Channel abstraction. This is the
mechanical token rename; the Config field changes and handler retrieval that the unification
requires land in the following commit. The two channel.go files that are fully rewritten for
v5 (common/ctrlchan, router/xlink_transport) are excluded here and rewritten in that commit.
Does not build on its own.
- moves the channel dependency to channel/v5 v5.0.10 and sdk-golang to v1.9.0 in the root and zititest modules
- mechanically rewrites every channel/v4 import path to channel/v5
This is the import-path-only step; the API-level changes the switch requires land in the following commit. This commit does not build on its own.
- invokes bind handlers via h.BindChannel(binding) instead of binding.Bind(h), which channel v5 removes from the Binding interface
- removes WithPriority from edge dial and state message sends; priority was already a no-op on grouped channels and channel v5 removes the priority API
Adds config to controller for heartbeat config
Add config to routers for heartbeat config
Adds configurable heartbeats to controller <-> controller connections
Paul Lorenz