- adds RemoveTerminatorsV2, an asynchronous control-channel removal, so a router
no longer holds a request slot open waiting for a synchronous reply; gated on a
new capability so routers fall back to the synchronous path against controllers
that do not advertise it
- paces establish and remove retries with a per-terminator exponential backoff
after a controller rejection, so a rejected operation is not re-sent on the next
event-loop tick, which is what let a rate-limited removal sustain itself
- routes v2 removes to the leader and skips the ordered delete for a terminator the
router confirms it created that no longer exists, so a storm of no-op retries
cannot consume the command rate limiter
- gates that skip on the leader having applied everything committed when it took
leadership, since a leader reports itself elected before those applies finish and
would otherwise read a committed create as absent and drop its delete
- guards the leader term and every write to the caught-up flag with a dedicated
lock, so a barrier completing alongside a leadership change cannot record a result
for a term that has already ended
- scopes v2 removals to the requesting router, so a router can only remove
terminators it owns on the asynchronous path as well as the synchronous one
- tracks in-flight removal batches on the router, resolving or requeueing them when
the response arrives; lost responses self-heal via rate-limiter expiry and the
retry scan
- waits for an outstanding operation before sending a terminator delete, and checks
that before dequeuing, so a delete neither races a create the controller has
committed but not applied nor falls out of the queue until the retry scan finds it
minutes later
- matches a create response to the attempt that produced it, by control channel and
reply sequence, since attempts overlap once one overruns the establishment timeout
and every attempt carries the same terminator id; only the outstanding attempt's
response may confirm the create or report the terminator idle
- records the attempt as the sender assigns its sequence, so a response cannot be
processed before the request it answers is known, and withdraws an attempt whose
send never queued
- reports a removal that succeeds past the establishment timeout as congestion
rather than success, matching how the establish path classifies its own latency,
since both resolve into the same limiter
- extracts the transient-error test both remove handlers share, so the v2 path
treats a leaderless cluster as busy like v1 does instead of as a permanent
failure, and the two cannot drift again
- applies the same delete sequencing and response correlation to the router-hosted
(ER/T) registry, which cleared the same in-flight marker unguarded and dropped a
deferred delete from its queue rather than leaving it there
- bounds the v2 response send with a timeout, so a router that stops draining its
send queue cannot accumulate blocked responder goroutines on the controller
- rejects a remove or update request whose terminator is owned by a different
router, on the fabric control channel handlers
- drops ids the requesting router does not own from batch removals, keeping
absent ids so a delete racing a not-yet-applied create is still ordered after
it
- adds unit tests for the ownership filter and for the single-terminator check
- adds an end-to-end test that drives the fabric control channel from an
enrolled router against a second router's terminator, covering single remove,
batch remove, and re-weight, plus a control that a router can still remove its
own
Paul Lorenz