routerConnectChurnLimit predates the per-router connect lock. It was added
alongside the ability for a new control channel to take over from an established
one, as the guard on how often that may happen, and it was also the only thing
keeping two connections for one router out of the connected map.
That second job is gone: at most one connection per router is now enforced under
the per-router lock, where the decision is atomic. The check in the accept path
runs against the connected map with no lock held, so it can only refuse a
connection early that would be refused there anyway.
Its first job remains, and is now the only thing doing it. ConnectRouter always
displaces an occupant it does not recognise, so without the limit a spurious
first-connection hello would tear down a healthy control channel and make the
router redial. Nothing said so, and the field carried no godoc at all.
- documents on the option what it protects, that it is churn policy rather than
the uniqueness guarantee, and that zero always allows takeover
- extracts the decision so it can be tested without standing up a network, and
tests it: protected when just established, protected part way through the
window, displaceable once past it, and never protected at zero
The struct's field alignment shifts because a comment ends gofmt's alignment
group; that part of the diff is whitespace only.
Behaviour is unchanged. Worth noting for readers of the option: past the window,
the established connection is now displaced and the connect refused, so the
router redials into the freed slot, where previously the arriving connection took
over directly. Same end state, one extra round trip, and nothing unvetted is
registered on the way.
The controller decided which of two racing connections for a router was current
by comparing router instances, but loaded one per connect by evicting the router
cache and reading back through it. Two connects could both evict, and whichever
read second was handed the instance the first had just published. A shared
instance makes the two connections indistinguishable: the connect path cannot
reject the second into an occupied slot, and when either channel dies the
disconnect path finds itself current and tears down the registration the other is
still using. The surviving channel is never re-bound, so the router stays
connected at the transport layer while absent from the model, unable to recover.
Connect and disconnect were also unserialized, so a stale or superseded
disconnect could interleave with a live connection and take its links with it.
- serializes a router's connect and disconnect with a per-router striped lock
- keeps at most one connection per router: a connect into an occupied slot is
rejected via an error from ConnectRouter, so the bind fails and NewChannel
closes it without starting rx or registering it, and the occupant is displaced;
the router redials into the freed slot
- displaces an occupant by closing its channel and also invoking the teardown
directly, since a channel that is already closed never fires its close handler
again; without this a dead but still registered connection holds the slot
forever and every redial is rejected against a slot nothing can free
- refuses a connect whose control channel is already closed rather than
registering it, so a connection no disconnect could ever remove is never
published
- gives every connection its own router instance via RouterManager.NewCtrlChanRouter,
read through readUncached so the cache neither supplies nor receives it, which
is what makes comparing instances meaningful
- moves recording the channel and connect time out of the accept path, so a
caller cannot attach the wrong channel or forget to attach one
- serializes link publication with that teardown on the same per-router stripe.
Validating currency and then publishing without it is a check-then-act: a
report can find the connection current and, by the time it reaches the link
manager, the teardown has already snapshotted and cleared the router's links,
so the link is recreated after everything that would have removed it. It is
then absent from the router's own index while still in the link table with a
disconnected source, and a reconnect reporting the same iteration can adopt
that stale source instead of rebuilding the link
- guards the entire DisconnectRouter teardown by connection currency, all or
nothing, and clears the connected flag and link index only when the
registration was actually given up, with the flag cleared under the same shard
lock as the map removal so the two cannot be observed disagreeing; the connected flag decides whether the
controller accepts a router's link reports, so clearing it for the wrong
connection silences a router that is up and reporting
- reduces MarkConnected to publishing the connection; the takeover-close moves
into ConnectRouter's reject path
- makes the per-router unlock idempotent so callers can defer it as a leak-safety
net and still unlock early before closing a channel outside the lock
- stops the replaced RouterSender in routerTxMap.Add so a takeover does not leak
the old sender's goroutine when the broker's asynchronous RouterDisconnected
loses the race to the redial's RouterConnected
- discards pending peer state changes for a router whose channel has closed,
since sending on one fails immediately and the failed send is retried as soon
as the event loop turns, spinning the loop and flooding the log
- queues the peer-state send-done event on every path, so a missing channel can
no longer leave sendInProgress set and stall that router's updates permanently
- resolves a router's version from its connected instance when validating link
conn info, since the version arrives in the hello and so is absent from an
instance loaded from the database
- normalizes both endpoints to the connected instance in shortestPath, which is
keyed and compared by pointer and so treated an endpoint held as any other
instance of the same router as absent from the graph, reporting a router as
unroutable from itself. That worked before only because the connect path
published its instance into the router cache, so a cache read and the connected
map returned the same object; nothing stated the requirement
- configures test logging once per package in TestMain, so a test no longer
writes global logger state while a previous test's shutdown logging reads it
- adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop
- translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup
- adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path
- re-evaluates dialers on link group and listener changes via RescanForDialOpportunities
For #3743.
- rejects a remove or update request whose terminator is owned by a different
router, on the fabric control channel handlers
- drops ids the requesting router does not own from batch removals, keeping
absent ids so a delete racing a not-yet-applied create is still ordered after
it
- adds unit tests for the ownership filter and for the single-terminator check
- adds an end-to-end test that drives the fabric control channel from an
enrolled router against a second router's terminator, covering single remove,
batch remove, and re-weight, plus a control that a router can still remove its
own
- removes the costTags option from router link listeners, which was parsed
from config, advertised to the controller, and stored on the router model
but never used for path selection or any other behavior
- drops GetLinkCostTags from the xlink.Listener interface and its transport
implementation
- reserves the corresponding ctrl_pb.Listener.costTags and
RouterLinks.RouterLink.linkCostTags protobuf fields and regenerates ctrl.pb.go
- documents the removal in the changelog
Forward ports the GHSA-cc5m-7mhm-xh9f fix, released in 2.0.2, to main.
- runs router certificate and fingerprint validation for router control-channel
underlay types, which was previously skipped for any connection carrying a
channel type header; only types dispatched to a separate self-validating
acceptor (the raft mesh) are skipped now
- binds the enrolled-fingerprint check to the verified leaf, so a peer cannot
pass by presenting its own leaf followed by a target router's public
certificate
- applies the already-connected / churn guard only when establishing a new
channel, so additional underlays of a grouped control channel are not rejected
while the router is already connected
- extracts the first-underlay header construction so the grouped-connection
scoping is unit-testable
- adds negative-path tests for untrusted and self-signed leaves, and for
separately-validated channel types being skipped
- adds command.WasLeaderless to classify cluster-has-no-leader dispatch errors as retriable
- replies busy instead of dropping or hard-failing terminator creates when the cluster is briefly leaderless, so the router backs off and requeues promptly rather than waiting for its multi-minute recovery scan
- removes the racy up-front leaderless pre-check in the sdk create handler in favor of classifying the actual dispatch result
- applies the same retriable classification to the ert tunnel create and batch remove terminator handlers
- keys router capability bits off the sdk-golang RouterCapability enum
- adds a generic capabilities.Mask[T ~int] bitmask, centralizing capability
set/check behind one value-to-bit translation
- adds capabilities.RouterCapability and ControllerCapability types, with
RouterCapabilityMask/ControllerCapabilityMask aliases, so masks and checks are
typed per namespace
- supports control-plane-only router capabilities as negative values that index
down from the top of the mask, collision-free with the SDK's upward-numbered
bits and invisible to the SDK and edge-api
- references the sdk-golang RouterCapability enum as the source of truth for
shared router capability bits
- holds the router's advertised capability mask as an instance on the router env
rather than a global, so in-process test routers do not share state
- advertises PostureChecks and BindSuccess as capability bits on both channels
while still sending the legacy boolean edge headers for backwards compatibility
- routes GetCapabilities/IsCapable and the controller's Router.Capabilities field
through the typed mask
- adds a provenance test that verifies, via go/packages, that every positive
router capability is SDK-sourced, every negative is control-plane-only, and no
two resolve to the same bit
- update sdk to v2.0.0-pre2
- adds a common/servermetrics package that owns the metrics MetricsMessage wire
format and the reporting/usage subsystem (message builder, usage registry,
interval and usage counters), wrapping the openziti/metrics Registry for
metric collection
- moves the controllers metrics reporter into the router package and removes it
from the shared metrics package, breaking a common -> router/env import cycle
- repoints controller and router consumers to common/servermetrics; base metric
collection stays on openziti/metrics
- keeps the proto field numbers and the metrics content-type identical so the
encoding is byte-compatible across the move, and uses a distinct proto package
name so ziti's and the library's messages coexist without a global proto
registry clash
- adds a round-trip test asserting wire compatibility with the library's
MetricsMessage
- leaves openziti/metrics unchanged, so sdk-golang and the shared xgress data
plane are unaffected
- gives ListenerCtrlChannel Min: 0 constraints per underlay type plus
MinTotalUnderlays: 1 in its configs, so the controller accepts the
high/low-priority grouped underlays the router dials while still closing the
channel only when its last underlay is lost
- restores the multi-underlay behavior the v4 listener-side SetMinTotal(1)
provided, which the channel/v5 migration dropped
- works around channel/v5 not yet treating MinTotalUnderlays alone as a
multi-underlay signal
- switches the xlink transport and router ctrl listeners to NewClassicListenerWithAcceptor, passing the MultiListener as a HelloAcceptor
- replaces the controller ctrl channel's NewClassicListener/UnderlayDispatcher wiring with NewClassicListenerWithAcceptor and a TypeRoutingAcceptor, adapting the mesh acceptor via AsHelloAcceptor
- removes the multiListenerAcceptor wrapper now that MultiListener implements HelloAcceptor directly
- removes the xgress_edge Acceptor.Run Create-loop, handing underlays to the MultiListener through the acceptor-based listener
- moves the controller ctrl connect handler into ListenerConfig.ConnectionHandlers
- updates ctrlchan channel tests to the new constructor
- decomposes the ctrlchan, xlink and edge-listener channels onto the v5 Senders, MessageSourceProvider and UnderlayEventListener interfaces, replacing the v4 UnderlayHandler god-interface
- replaces the hand-rolled dial/grouping/backoff machinery with channel.BackoffDialPolicy and declarative Constraints; ctrl keeps survive-to-zero (Min: 0) with MinStableDuration: 0 for prompt reconnect, while xlink and edge default underlays keep Min: 1 so loss closes the channel
- builds grouped channels via channel.NewChannel(*Config) and moves handler retrieval to GetSenders()
- records the channel via InitChannel from each bind handler, before underlay events fire, so handlers registered during bind do not dereference a nil channel
- generates a group secret for ungrouped inbound ctrl underlays on the router accept path, matching the controller, since NewChannel requires one
- preserves link-id-as-channel-id (the link dial policy wraps the cloned link-id identity dialer) and adds a test asserting dialed underlays present the link id
- registers the latency handler explicitly, as it is no longer a self-describing receiver in v5
- channel.MultiChannel -> channel.Channel
- channel.MultiChannelConfig -> channel.Config
- channel.NewMultiChannel(...) -> channel.NewChannel(...)
channel/v5 unifies Channel and MultiChannel into a single Channel abstraction. This is the
mechanical token rename; the Config field changes and handler retrieval that the unification
requires land in the following commit. The two channel.go files that are fully rewritten for
v5 (common/ctrlchan, router/xlink_transport) are excluded here and rewritten in that commit.
Does not build on its own.
- channel.TypedReceiveHandler -> channel.ContentTypeReceiver
- binding.AddTypedReceiveHandler(h) -> channel.AddReceiveHandlers(binding, h)
channel/v5 repurposes TypedReceiveHandler for the senders-typed handler and replaces the
self-describing pattern with ContentTypeReceiver plus the AddReceiveHandlers free function
(openziti/channel#262). Mechanical conversion; does not build on its own.
- moves the channel dependency to channel/v5 v5.0.10 and sdk-golang to v1.9.0 in the root and zititest modules
- mechanically rewrites every channel/v4 import path to channel/v5
This is the import-path-only step; the API-level changes the switch requires land in the following commit. This commit does not build on its own.
- invokes bind handlers via h.BindChannel(binding) instead of binding.Bind(h), which channel v5 removes from the Binding interface
- removes WithPriority from edge dial and state message sends; priority was already a no-op on grouped channels and channel v5 removes the priority API
- changes lastResponse from plain int64 to atomic.Int64 in both router
and peer heartbeat callbacks, fixing a data race between the heartbeat
response handler and the heartbeat check ticker
- fixes peer heartbeat logger channelType from "router" to "peer"
- adds CreateCircuitV3 message type and handler for routers that have
already authorized dials locally via RDM, bypassing service session
tokens in favor of identity ID, service ID, and pre-assigned circuit ID
- renames CreateCircuitRequest/Response to CreateCircuitV2Request/V2Response
for clarity now that V3 exists
- adds CircuitManager.Reserve to atomically claim circuit IDs before routing,
preventing collisions on pre-assigned IDs
- extends CreateCircuitParams with GetCircuitId so V3 can supply a
pre-assigned circuit ID (falls back to UUID generation when empty)
- fixes IsDialableByIdentity which was incorrectly calling IsBindableByIdentity
- extracts V2 handler into its own file create_circuit_v2.go
- adds CreateCircuitV3RequestType/ResponseType (20222/20223) to edge_ctrl protobuf
- registers V3 handler in controller server
- removes legacy v1 terminator code path; all terminators now use v2 flow
- refactors edgeTerminator.close() to decouple SDK notification from control plane notification
- adds pending SDK close notification queue with retry when channel is busy
- adds post-create inspect mechanism that verifies SDK still holds the bind after terminator creation
- queues second post-create inspect when establishment takes >30s to catch SDK timeout races
- detects and discards stale reordered binds on the same connection by comparing connIds
- re-establishes replacement terminators when a delete/create race is detected
- eliminates IsEntityPresent pre-filter in removeTerminatorsHandler to prevent raft ordering races
- fixes ValidateTerminators to query identities from the correct manager with the correct filter field
- adds postCreate flag to ValidateTerminatorsV2Request so routers skip redundant SDK inspect
- returns retry-later (nil result) from router validation when inspect is temporarily unavailable
- blocks SyncAllSubscribers until completion and guards RouterDataModel replacement with in-progress flag
- fixes InheritLocalData to enable service access tracking for all subscribed identities
- adds Services.Has check in GetServiceAccessPolicies to prevent false policy grants
- validates policy-to-identity associations in ValidateServicePolicies
- adds `ziti agent tunnel dump-sdk` command for SDK context inspection via IPC agent
- adds `ziti fabric inspect sdk` command to query SDK context through routers
- fixes --expected-per-host CLI flag binding in validate terminators command
- changes bind-access-lost retry hint from NotRetriable to RetryStartOver
- moves trace route response and xgress close handling off channel handler goroutine
- fixes listTerminators test helper to URL-encode filter parameter
- improves sdk-hosting-test validation resilience with login and query retries
- adds terminator_create_flow.md documenting the full lifecycle across SDK, router, and controller
- adds detailed logging for data model sync, service access tracking, and subscriber change detection
* Allow routers to request current cluster membership information. Fixes#3503
* Get cluster membership information from raft directly, rather than trying to cache it in the DB. Fixes#3501
* Set a router data model timeline when initializing a new HA setup, rather than letting it stay blank. Fixes#3500
* Reduce router data model full state updates. Fixes#3504
* When routers change (connect/disconnect), we re-check the mesh. We are passing in the routers to the channel and they can build up, which is unnecessary. We only need to signal once for all the routers that change since the last time we checked, that the mesh needs to be checked.
* We pass link changes through the run method, which is unnecessary since we pass it to a new goroutine. This inefficiency should be fixed.
* We're evaluating all faulted links for rerouting, even if they were still pending. Only reroute connected links.
* We're using time.After in each for loop, which can accumulate timers. Use a single Ticker instead.
Paul Lorenz