Commit Graph

165 Commits

Author SHA1 Message Date
Paul Lorenz 48e4d4a224 Manage router link configuration via the controller. Fixes #4004
- adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop
- translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup
- adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path
- re-evaluates dialers on link group and listener changes via RescanForDialOpportunities

For #3743.
2026-08-13 15:37:24 -04:00
Paul Lorenz 34133bd7d6 Merge pull request #4162 from openziti/fix/leaderless-terminator-retry
Signal retry on leaderless terminator operations
2026-08-13 11:06:12 -04:00
Paul Lorenz 6031f11eaa Merge pull request #4235 from openziti/fix/terminator-ops-source-router-scoping
Scope terminator operations to the requesting router
2026-08-12 10:03:26 -04:00
Paul Lorenz b353d2d9b6 Scope terminator operations to the requesting router. Fixes #4234
- rejects a remove or update request whose terminator is owned by a different
  router, on the fabric control channel handlers
- drops ids the requesting router does not own from batch removals, keeping
  absent ids so a delete racing a not-yet-applied create is still ordered after
  it
- adds unit tests for the ownership filter and for the single-terminator check
- adds an end-to-end test that drives the fabric control channel from an
  enrolled router against a second router's terminator, covering single remove,
  batch remove, and re-weight, plus a control that a router can still remove its
  own
2026-08-07 14:27:58 -04:00
Paul Lorenz a003f18792 Remove unused link costTags support
- removes the costTags option from router link listeners, which was parsed
  from config, advertised to the controller, and stored on the router model
  but never used for path selection or any other behavior
- drops GetLinkCostTags from the xlink.Listener interface and its transport
  implementation
- reserves the corresponding ctrl_pb.Listener.costTags and
  RouterLinks.RouterLink.linkCostTags protobuf fields and regenerates ctrl.pb.go
- documents the removal in the changelog
2026-08-03 18:03:26 -04:00
Paul Lorenz f3a27b32a8 Validate router certificates on typed control channel connections
Forward ports the GHSA-cc5m-7mhm-xh9f fix, released in 2.0.2, to main.

- runs router certificate and fingerprint validation for router control-channel
  underlay types, which was previously skipped for any connection carrying a
  channel type header; only types dispatched to a separate self-validating
  acceptor (the raft mesh) are skipped now
- binds the enrolled-fingerprint check to the verified leaf, so a peer cannot
  pass by presenting its own leaf followed by a target router's public
  certificate
- applies the already-connected / churn guard only when establishing a new
  channel, so additional underlays of a grouped control channel are not rejected
  while the router is already connected
- extracts the first-underlay header construction so the grouped-connection
  scoping is unit-testable
- adds negative-path tests for untrusted and self-signed leaves, and for
  separately-validated channel types being skipped
2026-07-27 17:40:35 -04:00
Paul Lorenz d838e209ac Signal retry on leaderless terminator operations. Fixes #4160
- adds command.WasLeaderless to classify cluster-has-no-leader dispatch errors as retriable
- replies busy instead of dropping or hard-failing terminator creates when the cluster is briefly leaderless, so the router backs off and requeues promptly rather than waiting for its multi-minute recovery scan
- removes the racy up-front leaderless pre-check in the sdk create handler in favor of classifying the actual dispatch result
- applies the same retriable classification to the ert tunnel create and batch remove terminator handlers
2026-07-24 11:43:33 -04:00
Andrew Martinez 45b5046f52 fixes openziti/ziti#4071 unify router capabilities into one shared namespace (#4073)
- keys router capability bits off the sdk-golang RouterCapability enum
- adds a generic capabilities.Mask[T ~int] bitmask, centralizing capability
  set/check behind one value-to-bit translation
- adds capabilities.RouterCapability and ControllerCapability types, with
  RouterCapabilityMask/ControllerCapabilityMask aliases, so masks and checks are
  typed per namespace
- supports control-plane-only router capabilities as negative values that index
  down from the top of the mask, collision-free with the SDK's upward-numbered
  bits and invisible to the SDK and edge-api
- references the sdk-golang RouterCapability enum as the source of truth for
  shared router capability bits
- holds the router's advertised capability mask as an instance on the router env
  rather than a global, so in-process test routers do not share state
- advertises PostureChecks and BindSuccess as capability bits on both channels
  while still sending the legacy boolean edge headers for backwards compatibility
- routes GetCapabilities/IsCapable and the controller's Router.Capabilities field
  through the typed mask
- adds a provenance test that verifies, via go/packages, that every positive
  router capability is SDK-sourced, every negative is control-plane-only, and no
  two resolve to the same bit
- update sdk to v2.0.0-pre2
2026-07-02 17:02:34 -04:00
Paul Lorenz 187aa11f24 Own the metrics wire format in ziti. Fixes #4036
- adds a common/servermetrics package that owns the metrics MetricsMessage wire
  format and the reporting/usage subsystem (message builder, usage registry,
  interval and usage counters), wrapping the openziti/metrics Registry for
  metric collection
- moves the controllers metrics reporter into the router package and removes it
  from the shared metrics package, breaking a common -> router/env import cycle
- repoints controller and router consumers to common/servermetrics; base metric
  collection stays on openziti/metrics
- keeps the proto field numbers and the metrics content-type identical so the
  encoding is byte-compatible across the move, and uses a distinct proto package
  name so ziti's and the library's messages coexist without a global proto
  registry clash
- adds a round-trip test asserting wire compatibility with the library's
  MetricsMessage
- leaves openziti/metrics unchanged, so sdk-golang and the shared xgress data
  plane are unaffected
2026-06-29 22:37:25 -04:00
Paul Lorenz 2dc4075446 Make listener ctrl channels multi-underlay-capable. For #3983
- gives ListenerCtrlChannel Min: 0 constraints per underlay type plus
  MinTotalUnderlays: 1 in its configs, so the controller accepts the
  high/low-priority grouped underlays the router dials while still closing the
  channel only when its last underlay is lost
- restores the multi-underlay behavior the v4 listener-side SetMinTotal(1)
  provided, which the channel/v5 migration dropped
- works around channel/v5 not yet treating MinTotalUnderlays alone as a
  multi-underlay signal
2026-06-22 14:53:47 -04:00
Paul Lorenz 1c877e2501 Migrate to channel/v5 deferred-ack accept API. For #3983
- switches the xlink transport and router ctrl listeners to NewClassicListenerWithAcceptor, passing the MultiListener as a HelloAcceptor
- replaces the controller ctrl channel's NewClassicListener/UnderlayDispatcher wiring with NewClassicListenerWithAcceptor and a TypeRoutingAcceptor, adapting the mesh acceptor via AsHelloAcceptor
- removes the multiListenerAcceptor wrapper now that MultiListener implements HelloAcceptor directly
- removes the xgress_edge Acceptor.Run Create-loop, handing underlays to the MultiListener through the acceptor-based listener
- moves the controller ctrl connect handler into ListenerConfig.ConnectionHandlers
- updates ctrlchan channel tests to the new constructor
2026-06-18 12:51:02 -04:00
Paul Lorenz d481224c9b Decompose multi-underlay channels onto the channel/v5 API. Fixes #3983
- decomposes the ctrlchan, xlink and edge-listener channels onto the v5 Senders, MessageSourceProvider and UnderlayEventListener interfaces, replacing the v4 UnderlayHandler god-interface
- replaces the hand-rolled dial/grouping/backoff machinery with channel.BackoffDialPolicy and declarative Constraints; ctrl keeps survive-to-zero (Min: 0) with MinStableDuration: 0 for prompt reconnect, while xlink and edge default underlays keep Min: 1 so loss closes the channel
- builds grouped channels via channel.NewChannel(*Config) and moves handler retrieval to GetSenders()
- records the channel via InitChannel from each bind handler, before underlay events fire, so handlers registered during bind do not dereference a nil channel
- generates a group secret for ungrouped inbound ctrl underlays on the router accept path, matching the controller, since NewChannel requires one
- preserves link-id-as-channel-id (the link dial policy wraps the cloned link-id identity dialer) and adds a test asserting dialed underlays present the link id
- registers the latency handler explicitly, as it is no longer a self-describing receiver in v5
2026-06-18 12:51:02 -04:00
Paul Lorenz ee8ad78e3b Rename MultiChannel to the unified Channel for channel/v5. For #3983
- channel.MultiChannel -> channel.Channel
- channel.MultiChannelConfig -> channel.Config
- channel.NewMultiChannel(...) -> channel.NewChannel(...)

channel/v5 unifies Channel and MultiChannel into a single Channel abstraction. This is the
mechanical token rename; the Config field changes and handler retrieval that the unification
requires land in the following commit. The two channel.go files that are fully rewritten for
v5 (common/ctrlchan, router/xlink_transport) are excluded here and rewritten in that commit.
Does not build on its own.
2026-06-18 12:51:02 -04:00
Paul Lorenz ae806045b5 Convert self-describing receive handlers to channel/v5. For #3983
- channel.TypedReceiveHandler -> channel.ContentTypeReceiver
- binding.AddTypedReceiveHandler(h) -> channel.AddReceiveHandlers(binding, h)

channel/v5 repurposes TypedReceiveHandler for the senders-typed handler and replaces the
self-describing pattern with ContentTypeReceiver plus the AddReceiveHandlers free function
(openziti/channel#262). Mechanical conversion; does not build on its own.
2026-06-18 12:51:02 -04:00
Paul Lorenz 1c122af490 Rewrite channel/v4 imports to channel/v5. For #3983
- moves the channel dependency to channel/v5 v5.0.10 and sdk-golang to v1.9.0 in the root and zititest modules
- mechanically rewrites every channel/v4 import path to channel/v5

This is the import-path-only step; the API-level changes the switch requires land in the following commit. This commit does not build on its own.
2026-06-18 12:51:02 -04:00
Paul Lorenz b5eaeefdb3 Prep for channel v5: bind handler invocation, send priorities. Fixes #3942
- invokes bind handlers via h.BindChannel(binding) instead of binding.Bind(h), which channel v5 removes from the Binding interface
- removes WithPriority from edge dial and state message sends; priority was already a no-op on grouped channels and channel v5 removes the priority API
2026-06-05 15:24:45 -04:00
Paul Lorenz 588cb2350c Fix data race on heartbeat lastResponse and peer logger label
- changes lastResponse from plain int64 to atomic.Int64 in both router
  and peer heartbeat callbacks, fixing a data race between the heartbeat
  response handler and the heartbeat check ticker
- fixes peer heartbeat logger channelType from "router" to "peer"
2026-04-03 17:42:18 -04:00
Paul Lorenz 7b4ae12c05 Add CreateCircuitV3 for RDM-authorized circuit creation. Fixes #3721
- adds CreateCircuitV3 message type and handler for routers that have
  already authorized dials locally via RDM, bypassing service session
  tokens in favor of identity ID, service ID, and pre-assigned circuit ID
- renames CreateCircuitRequest/Response to CreateCircuitV2Request/V2Response
  for clarity now that V3 exists
- adds CircuitManager.Reserve to atomically claim circuit IDs before routing,
  preventing collisions on pre-assigned IDs
- extends CreateCircuitParams with GetCircuitId so V3 can supply a
  pre-assigned circuit ID (falls back to UUID generation when empty)
- fixes IsDialableByIdentity which was incorrectly calling IsBindableByIdentity
- extracts V2 handler into its own file create_circuit_v2.go
- adds CreateCircuitV3RequestType/ResponseType (20222/20223) to edge_ctrl protobuf
- registers V3 handler in controller server
2026-03-27 14:41:28 -04:00
Paul Lorenz e1638173cd Fixes for SDK terminator management. Add support for ziti sdk inspection. Fixes #3609
- removes legacy v1 terminator code path; all terminators now use v2 flow
- refactors edgeTerminator.close() to decouple SDK notification from control plane notification
- adds pending SDK close notification queue with retry when channel is busy
- adds post-create inspect mechanism that verifies SDK still holds the bind after terminator creation
- queues second post-create inspect when establishment takes >30s to catch SDK timeout races
- detects and discards stale reordered binds on the same connection by comparing connIds
- re-establishes replacement terminators when a delete/create race is detected
- eliminates IsEntityPresent pre-filter in removeTerminatorsHandler to prevent raft ordering races
- fixes ValidateTerminators to query identities from the correct manager with the correct filter field
- adds postCreate flag to ValidateTerminatorsV2Request so routers skip redundant SDK inspect
- returns retry-later (nil result) from router validation when inspect is temporarily unavailable
- blocks SyncAllSubscribers until completion and guards RouterDataModel replacement with in-progress flag
- fixes InheritLocalData to enable service access tracking for all subscribed identities
- adds Services.Has check in GetServiceAccessPolicies to prevent false policy grants
- validates policy-to-identity associations in ValidateServicePolicies
- adds `ziti agent tunnel dump-sdk` command for SDK context inspection via IPC agent
- adds `ziti fabric inspect sdk` command to query SDK context through routers
- fixes --expected-per-host CLI flag binding in validate terminators command
- changes bind-access-lost retry hint from NotRetriable to RetryStartOver
- moves trace route response and xgress close handling off channel handler goroutine
- fixes listTerminators test helper to URL-encode filter parameter
- improves sdk-hosting-test validation resilience with login and query retries
- adds terminator_create_flow.md documenting the full lifecycle across SDK, router, and controller
- adds detailed logging for data model sync, service access tracking, and subscriber change detection
2026-03-11 13:10:28 -04:00
Paul Lorenz 6b869cea9d Add support for ctrlChanListener on router to the model. Fixes #3635 2026-03-07 00:00:51 -05:00
Paul Lorenz 8acf90cdf6 Multi-underlay control channel doesn't correctly handle lack of group secret on non-grouped underlays. Fixes #3624 2026-03-02 13:52:11 -05:00
Paul Lorenz ab51fa8217 Fix the control channel header conflicts with channel headers. Use single strategy for sharing capabilities with bit mask. 2026-02-24 15:50:05 -05:00
Paul Lorenz 90112219a3 Support multi-underlay control channels. Fixes #3550 2026-02-11 14:20:29 -05:00
Paul Lorenz da9ef76d19 Optimize imports 2026-01-28 15:01:35 -05:00
Paul Lorenz 2ffc6e1151 Update ziti version to 2.0 2026-01-28 12:34:10 -05:00
Paul Lorenz 08359a9e2b Remove legacy link management code from the controller. Fixes #3512 2026-01-22 15:26:42 -05:00
Paul Lorenz c18a594b56 Clustering coordination fixes
* Allow routers to request current cluster membership information. Fixes #3503
* Get cluster membership information from raft directly, rather than trying to cache it in the DB. Fixes #3501
* Set a router data model timeline when initializing a new HA setup, rather than letting it stay blank. Fixes #3500
* Reduce router data model full state updates. Fixes #3504
2026-01-17 02:03:31 -05:00
Paul Lorenz 3cfc8b5cf5 Controller should clear links not in full link sync from router. Fixes #3492 2026-01-16 14:42:39 -05:00
Paul Lorenz a4b4e1dc16 Clean up some excessive logging 2025-10-29 15:30:31 -04:00
Paul Lorenz 8bedd3af1d Clean up connect events tests and remove global XG registry. Fixes #3345 2025-10-29 14:46:52 -04:00
Paul Lorenz 653063767e Add alert events. Fixes #3264 2025-10-15 14:21:07 -04:00
Paul Lorenz e5fb69935e Support multi-underlay links. Fixes #3134 2025-07-11 20:56:11 -04:00
Paul Lorenz cc7eefaf24 Add support for tracking network interfaces on routers and identities. Fixes #3082. Add network discover to router. Fixes #3083 2025-06-17 09:32:51 -04:00
Paul Lorenz 981df8269d Support xgress flow control from the SDK. Fixes #2986 2025-04-30 10:27:43 -04:00
Paul Lorenz e647d67325 Update to channel/v4 2025-04-02 15:28:59 -04:00
Paul Lorenz 8dbf6e5d85 Update errorz.MultipleErrors users to errors.Join 2025-03-22 10:14:02 -04:00
Paul Lorenz ec57c80ff7 ER/T Hosting HA chaos test and fixes (#2806)
* Add ERT hosting chaos test. Also add ert terminator validation utility. Fixes #2288

* Rework ER/T terminator management based on SDK terminator management code

* Update deps

* Make sdk/ert-terminators into a constant
2025-02-25 17:02:21 -05:00
Paul Lorenz 32eddd61ca HA SDK terminators test. Fixes #2217. Fixes #2533 2024-11-12 18:45:08 -05:00
Paul Lorenz cccf0c06af Update to channel/v3. Fixes #2390 2024-09-09 12:23:25 -04:00
Paul Lorenz c3b43133d1 Merge fabric and controller model code. Fixes #2205 2024-07-09 16:11:01 -04:00
Paul Lorenz 8d11af3d97 Optimize network run loop. Fixes #1897
* When routers change (connect/disconnect), we re-check the mesh. We are passing in the routers to the channel and they can build up, which is unnecessary. We only need to signal once for all the routers that change since the last time we checked, that the mesh needs to be checked.
* We pass link changes through the run method, which is unnecessary since we pass it to a new goroutine. This inefficiency should be fixed.
* We're evaluating all faulted links for rerouting, even if they were still pending. Only reroute connected links.
* We're using time.After in each for loop, which can accumulate timers. Use a single Ticker instead.
2024-04-04 18:46:50 -04:00
Paul Lorenz fab81e4cd4 Add terminator chaos testing and fix issues found. Fixes #1794 Fixes #1369 2024-03-12 10:31:36 -04:00
Paul Lorenz 8423a8dd6a Fix link management race conditions found by chaos testing. Fixes #1709 2024-01-29 17:23:51 -05:00
Paul Lorenz c03c3c2851 Additional fixes and tweaks 2024-01-19 17:35:25 -05:00
Paul Lorenz 70b32603ad Start on chaos testing 2024-01-19 17:34:15 -05:00
Paul Lorenz c4abcae949 Link management fixes. Fixes #1692 Fixes #1693 2024-01-19 17:33:21 -05:00
Paul Lorenz df2ab68ad5 Make sure link accepts are not single threaded. Fixes #1669 2023-12-29 10:18:50 -05:00
Paul Lorenz 9ddd1b941d Add service max idle time. Fixes #1496. Add decomission router support. Fixes #1402 2023-12-08 12:48:56 -05:00
Paul Lorenz b270d18faf Add additional logging. Remove unnecessary router lookups. May address #1460 2023-10-31 14:57:06 -04:00
Paul Lorenz 3523b34b2c Fix panic when router version info isn't set by ensuring it is set and making sure we get the right version. Fixes #1423 2023-10-11 13:55:50 -04:00