- adds RemoveTerminatorsV2, an asynchronous control-channel removal, so a router
no longer holds a request slot open waiting for a synchronous reply; gated on a
new capability so routers fall back to the synchronous path against controllers
that do not advertise it
- paces establish and remove retries with a per-terminator exponential backoff
after a controller rejection, so a rejected operation is not re-sent on the next
event-loop tick, which is what let a rate-limited removal sustain itself
- routes v2 removes to the leader and skips the ordered delete for a terminator the
router confirms it created that no longer exists, so a storm of no-op retries
cannot consume the command rate limiter
- gates that skip on the leader having applied everything committed when it took
leadership, since a leader reports itself elected before those applies finish and
would otherwise read a committed create as absent and drop its delete
- guards the leader term and every write to the caught-up flag with a dedicated
lock, so a barrier completing alongside a leadership change cannot record a result
for a term that has already ended
- scopes v2 removals to the requesting router, so a router can only remove
terminators it owns on the asynchronous path as well as the synchronous one
- tracks in-flight removal batches on the router, resolving or requeueing them when
the response arrives; lost responses self-heal via rate-limiter expiry and the
retry scan
- waits for an outstanding operation before sending a terminator delete, and checks
that before dequeuing, so a delete neither races a create the controller has
committed but not applied nor falls out of the queue until the retry scan finds it
minutes later
- matches a create response to the attempt that produced it, by control channel and
reply sequence, since attempts overlap once one overruns the establishment timeout
and every attempt carries the same terminator id; only the outstanding attempt's
response may confirm the create or report the terminator idle
- records the attempt as the sender assigns its sequence, so a response cannot be
processed before the request it answers is known, and withdraws an attempt whose
send never queued
- reports a removal that succeeds past the establishment timeout as congestion
rather than success, matching how the establish path classifies its own latency,
since both resolve into the same limiter
- extracts the transient-error test both remove handlers share, so the v2 path
treats a leaderless cluster as busy like v1 does instead of as a permanent
failure, and the two cannot drift again
- applies the same delete sequencing and response correlation to the router-hosted
(ER/T) registry, which cleared the same in-flight marker unguarded and dropped a
deferred delete from its queue rather than leaving it there
- bounds the v2 response send with a timeout, so a router that stops draining its
send queue cannot accumulate blocked responder goroutines on the controller
- rejects a remove or update request whose terminator is owned by a different
router, on the fabric control channel handlers
- drops ids the requesting router does not own from batch removals, keeping
absent ids so a delete racing a not-yet-applied create is still ordered after
it
- adds unit tests for the ownership filter and for the single-terminator check
- adds an end-to-end test that drives the fabric control channel from an
enrolled router against a second router's terminator, covering single remove,
batch remove, and re-weight, plus a control that a router can still remove its
own
- adds command.WasLeaderless to classify cluster-has-no-leader dispatch errors as retriable
- replies busy instead of dropping or hard-failing terminator creates when the cluster is briefly leaderless, so the router backs off and requeues promptly rather than waiting for its multi-minute recovery scan
- removes the racy up-front leaderless pre-check in the sdk create handler in favor of classifying the actual dispatch result
- applies the same retriable classification to the ert tunnel create and batch remove terminator handlers
- moves the channel dependency to channel/v5 v5.0.10 and sdk-golang to v1.9.0 in the root and zititest modules
- mechanically rewrites every channel/v4 import path to channel/v5
This is the import-path-only step; the API-level changes the switch requires land in the following commit. This commit does not build on its own.
- removes legacy v1 terminator code path; all terminators now use v2 flow
- refactors edgeTerminator.close() to decouple SDK notification from control plane notification
- adds pending SDK close notification queue with retry when channel is busy
- adds post-create inspect mechanism that verifies SDK still holds the bind after terminator creation
- queues second post-create inspect when establishment takes >30s to catch SDK timeout races
- detects and discards stale reordered binds on the same connection by comparing connIds
- re-establishes replacement terminators when a delete/create race is detected
- eliminates IsEntityPresent pre-filter in removeTerminatorsHandler to prevent raft ordering races
- fixes ValidateTerminators to query identities from the correct manager with the correct filter field
- adds postCreate flag to ValidateTerminatorsV2Request so routers skip redundant SDK inspect
- returns retry-later (nil result) from router validation when inspect is temporarily unavailable
- blocks SyncAllSubscribers until completion and guards RouterDataModel replacement with in-progress flag
- fixes InheritLocalData to enable service access tracking for all subscribed identities
- adds Services.Has check in GetServiceAccessPolicies to prevent false policy grants
- validates policy-to-identity associations in ValidateServicePolicies
- adds `ziti agent tunnel dump-sdk` command for SDK context inspection via IPC agent
- adds `ziti fabric inspect sdk` command to query SDK context through routers
- fixes --expected-per-host CLI flag binding in validate terminators command
- changes bind-access-lost retry hint from NotRetriable to RetryStartOver
- moves trace route response and xgress close handling off channel handler goroutine
- fixes listTerminators test helper to URL-encode filter parameter
- improves sdk-hosting-test validation resilience with login and query retries
- adds terminator_create_flow.md documenting the full lifecycle across SDK, router, and controller
- adds detailed logging for data model sync, service access tracking, and subscriber change detection
Paul Lorenz