- adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId,
ListenerBinding}) used by all staleness and GC checks, replacing the
pre-rendered key string each side used to pass around
- adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links`
CLI for operator-driven, two-sided staleness verification with optional --gc,
which acts only when both endpoints agree
- implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed):
the router walks its xlinks after each Apply that mutates listeners, dialers,
or gcMode and closes one-sided-stale entries under the configured mode
- sweeps a removal under orphaned whatever the removed config said, including
preserve, since a removal leaves no listeners and no dialers and every
remaining link is one the router can never re-establish
- reports staleness as a three-state verdict (unknown/notStale/stale) so a side
that cannot judge a link abstains instead of guessing; a destination absent
from the registry's listener snapshot means "listeners unknown", not "peer has
none", which keeps a briefly unhealthy peer from losing its live links
- rebuilds the link surface only when a listener or dialer definition actually
moves, so a gcMode-only change no longer rebinds every listen socket
- gates CheckStaleLinks on a new RouterStaleLinkCheck capability and explains
each unqueryable endpoint, including one left unasked by the filter, so a
partial verdict says which of the three causes applies
- detects a peer re-advertising the address a link was dialed to from the
dialing side, not just from the peer's own listener side
- pins every verdict to the link iteration it was computed against, faults that
iteration rather than the current one, and takes the conditional removal's
result as the decision, so a link re-dialed while the sweep was running is
neither closed nor reported as collected
- resolves which side a report speaks for from the sending router against the
controller's link map rather than from the side the report names, so one
router cannot fill both sides and satisfy the both-agree rule --gc depends on
- bounds the query fan-out with a semaphore, matching the other validation
sweeps, since each query carries a minute-long reply timeout
- fails the CLI when the result stream ends before every link is reported, and
drains results already buffered before declaring any missing
- rejects an unrecognized match mode rather than resolving it to changed, the
broadest removal criterion
- consolidates the controller-managed router configuration changelog into one
section, replacing the status that said routers do not yet consume configs at
runtime, and documents the per-config-type opt-in
- restores the Current Beta Features section, dropped when the 2.0 changelog was
archived, and lists controller-managed router configuration in it
- adds unparseable to the codespell ignore list, a spelling variant the
dictionary flags as a typo
For #3743.
BuildRouterLinks runs once per router connect and scanned the whole link table to
find the connecting router's links. The table is proportional to the square of the
router count in a full mesh, so a wave in which every router reconnects walks the
cube of it in link visits. That never shows in steady state; it arrives during a
mass reconnect, when the connect path is already the bottleneck.
Indexing every link under both endpoint router ids fixes the cost, and introduces
the problem that the table and the index are two structures written in sequence.
Three things follow from that, and the destination repair below turned out to
depend on the connect path's ordering as well.
Performance:
- indexes every link under both endpoint router ids, maintained in the add and
remove paths all link mutations funnel through, so a connect costs its own
router's link count rather than the whole table
- prunes a full refresh against the reporting router's own links. That report is
what every reconnect sends, and it walked the whole table to find the router's
stale links, so the cost the connect path shed was still being paid one message
later
- indexes link ids rather than link pointers, so a per-router query resolves each
id against the table and hands back whatever it currently holds. A reconnect
replaces a link's Router object and a higher iteration replaces the Link object,
but neither replaces the id, which is exactly why the scan existed
- holds each router's link ids in a LockedSet rather than a sharded map, which
cost a map and a mutex per shard for an index that is usually empty and is held
for as long as the router has existed: 2.9KB per router id against 150 bytes
- drops a router's index when the router is deleted, guarded on the store still
having no router under that id and checked inside the index map's removal
callback. A router id can come back, since fabric router ids are the enrollment
certificate's common name, and nothing orders the cleanup against that: store
event handlers run after bolt has released the writer lock, so one can execute
after the id was recreated, connected and reported links. Indexing a link takes
the same shard as the guarded read, so an entry for a live router cannot appear
without its create having committed first, which the read then sees, and an entry
arriving later blocks and lands in a freshly created index
- keeps an index when the store cannot be read, since that is not evidence the
router is gone, and dropping one otherwise could discard a link another goroutine
had already fetched the index to record
Keeping the two structures agreeing:
- serializes Add and Remove on the link's id. Without it a removal running between
an add's two writes takes the table entry, finds nothing yet in the index, and
lets the add index a link the table no longer holds, which nothing cleans up.
Only RouterReportedLink held that lock before; the four other removal paths held
nothing
- serializes the connect-time pairing on that id as well, which was the last writer
left outside it. A removal reading no destination could complete while the
pairing was still deciding, leaving the link in the destination's link set after
the table had dropped it, where path computation still routed over it
- unindexes an id only when the table no longer holds a link under it, so a
replacement that has taken the table slot is not unindexed by the older link's
removal. A stale id is the safe direction to err in, since a query filters it,
where an id wrongly dropped leaves a live link no per-router query finds
- has per-router queries resolve ids against the table, so an id it no longer
holds is not handed out as a live link
Pairing a link with its destination:
- re-resolves the destination once the link is in the table, and on every later
report, so a link that reached a bad state heals. A report resolves the
destination before the link is in the table and a router's connect repairs links
already in it, so a router connecting between those two moments was seen by
neither, leaving a link with no adjacency that every operator-facing view still
called healthy
- makes that re-resolve unconditional rather than only for a link holding no
destination. The resolve happens under the source router's connect stripe, not
the destination's, so the destination can be replaced between the resolve and the
report landing, and a link left on a connection that is no longer registered
carries no adjacency just as one holding none does
- registers a connecting router before building its links, which is what makes
that re-resolve close the window rather than narrow it
- adds Link.PointDestAt so the two repair paths cannot both index the link on the
destination, since that index is a slice that does not deduplicate
Also here:
- logs the link count on both sides of the reconnect exchange, and why a report was
discarded. A router announces its links once per reconnect and is not asked
again, so a discarded report left it with no links for a router that believed it
had announced them, with nothing recorded to compare. The discard reports the
state it decided on, read once, since a reconnect landing between the decision
and the log would otherwise have it name a state that discards nothing
- gives test routers version info, which a connected router always has because the
accept path refuses a hello without it, and which the router sync path
dereferences with no nil check
The destination repair is backported from the gossip link-state branch, where the
connect path registers before building links for unrelated reasons. That ordering
is load-bearing here and is now explicit at the call site.
- adds a fablab model (zititest/models/upgrade-test) that stands up a multi-region HA controller cluster, routers, and loop4 sim traffic clients (SDK, ERT, ZET, tunneler) and drives repeatable in-place upgrade/downgrade iterations, re-bootstrapping in place each iteration
- exercises HA cluster disruption and recovery (snapshot restore, node rejoin) between iterations and validates steady-state traffic and expected terminators after each disruption
- honors the raft restartSelf setting on migration-snapshot restore, so a restored controller restarts itself instead of exiting, and promotes RestartController to a package-level function
- re-sends loop4 run-scenario requests to sims that reconnect mid-scenario, tracking each scenario's expected client set explicitly and ignoring results from clients outside it, so post-disruption client reconnection no longer stalls validation
- closes leaked loop4 sim-control connections on error and on supersession
- rotates component logs and pins the log-pipe binary to the local build, and drops router debug logging, to keep long iterating runs disk-bounded
- excludes the doc/unsettled/ scratch area for in-progress design docs from version control
The controller decided which of two racing connections for a router was current
by comparing router instances, but loaded one per connect by evicting the router
cache and reading back through it. Two connects could both evict, and whichever
read second was handed the instance the first had just published. A shared
instance makes the two connections indistinguishable: the connect path cannot
reject the second into an occupied slot, and when either channel dies the
disconnect path finds itself current and tears down the registration the other is
still using. The surviving channel is never re-bound, so the router stays
connected at the transport layer while absent from the model, unable to recover.
Connect and disconnect were also unserialized, so a stale or superseded
disconnect could interleave with a live connection and take its links with it.
- serializes a router's connect and disconnect with a per-router striped lock
- keeps at most one connection per router: a connect into an occupied slot is
rejected via an error from ConnectRouter, so the bind fails and NewChannel
closes it without starting rx or registering it, and the occupant is displaced;
the router redials into the freed slot
- displaces an occupant by closing its channel and also invoking the teardown
directly, since a channel that is already closed never fires its close handler
again; without this a dead but still registered connection holds the slot
forever and every redial is rejected against a slot nothing can free
- refuses a connect whose control channel is already closed rather than
registering it, so a connection no disconnect could ever remove is never
published
- gives every connection its own router instance via RouterManager.NewCtrlChanRouter,
read through readUncached so the cache neither supplies nor receives it, which
is what makes comparing instances meaningful
- moves recording the channel and connect time out of the accept path, so a
caller cannot attach the wrong channel or forget to attach one
- serializes link publication with that teardown on the same per-router stripe.
Validating currency and then publishing without it is a check-then-act: a
report can find the connection current and, by the time it reaches the link
manager, the teardown has already snapshotted and cleared the router's links,
so the link is recreated after everything that would have removed it. It is
then absent from the router's own index while still in the link table with a
disconnected source, and a reconnect reporting the same iteration can adopt
that stale source instead of rebuilding the link
- guards the entire DisconnectRouter teardown by connection currency, all or
nothing, and clears the connected flag and link index only when the
registration was actually given up, with the flag cleared under the same shard
lock as the map removal so the two cannot be observed disagreeing; the connected flag decides whether the
controller accepts a router's link reports, so clearing it for the wrong
connection silences a router that is up and reporting
- reduces MarkConnected to publishing the connection; the takeover-close moves
into ConnectRouter's reject path
- makes the per-router unlock idempotent so callers can defer it as a leak-safety
net and still unlock early before closing a channel outside the lock
- stops the replaced RouterSender in routerTxMap.Add so a takeover does not leak
the old sender's goroutine when the broker's asynchronous RouterDisconnected
loses the race to the redial's RouterConnected
- discards pending peer state changes for a router whose channel has closed,
since sending on one fails immediately and the failed send is retried as soon
as the event loop turns, spinning the loop and flooding the log
- queues the peer-state send-done event on every path, so a missing channel can
no longer leave sendInProgress set and stall that router's updates permanently
- resolves a router's version from its connected instance when validating link
conn info, since the version arrives in the hello and so is absent from an
instance loaded from the database
- normalizes both endpoints to the connected instance in shortestPath, which is
keyed and compared by pointer and so treated an endpoint held as any other
instance of the same router as absent from the graph, reporting a router as
unroutable from itself. That worked before only because the connect path
published its instance into the router cache, so a cache read and the connected
map returned the same object; nothing stated the requirement
- configures test logging once per package in TestMain, so a test no longer
writes global logger state while a previous test's shutdown logging reads it
- adds a leadership-gained handler that establishes a cluster id when a
bootstrapped cluster has none, so clusters migrated on older builds (which
came up with an empty, non-durable cluster id) self-heal without operator
action
- makes db.InitClusterId set-once and idempotent: it keeps an existing cluster
id and returns the effective value instead of erroring on a different one, so
redundant or racing backfills cannot fail the critical InitClusterIdCmd or
diverge memory from disk
- passes the current timeline id through the backfill so it is left unchanged
- publishes the effective cluster id in memory after persisting it
- adds a clusterId field to SyncSnapshotCommand and writes it into the database
after the migration snapshot restore, so a controller bootstrapped by
migrating a database ends up with a durable cluster id instead of an empty one
- the snapshot restore replaces the whole FSM database with the migration
source, which carries no cluster id, so without this the id written during
bootstrap was silently wiped and the node came up with an empty, non-durable
cluster id, defeating the mesh cluster-id validation
- persists the raft index after the cluster id in RestoreSnapshot so the index
remains the completion gate: a failure before it halts (SyncSnapshotCommand is
a critical command) and replays/retries on restart rather than skipping the
command with a blank cluster id
- fails RaftRestoreFromBoltDb when the cluster id is blank after bootstrap
- regenerates cmd.pb.go for the new field
- adds a CriticalCommand marker interface for commands that establish base
state, and marks SyncSnapshotCommand (a full snapshot restore) as critical
- makes BoltDbFsm.Apply halt when a critical command fails to apply, rather
than logging the error and persisting the advanced raft index; the failed
apply's in-tx index update is rolled back and left unpersisted, so raft
replays and retries the command on restart instead of the node running
caught-up-on-index but empty-on-data
- attaches the command type to all apply log lines so a failure is
self-contained
- reorders DisconnectRouter so Router.MarkDisconnected runs before the
RerouteLink cascade, so shortestPath and connected-map reads see the
disconnecting router as gone and never compute a replacement path
through the router being removed
- snapshots the router's links before MarkDisconnected, which clears the
router's link set, so the link-removal/reroute loop still runs
- adds a common/servermetrics package that owns the metrics MetricsMessage wire
format and the reporting/usage subsystem (message builder, usage registry,
interval and usage counters), wrapping the openziti/metrics Registry for
metric collection
- moves the controllers metrics reporter into the router package and removes it
from the shared metrics package, breaking a common -> router/env import cycle
- repoints controller and router consumers to common/servermetrics; base metric
collection stays on openziti/metrics
- keeps the proto field numbers and the metrics content-type identical so the
encoding is byte-compatible across the move, and uses a distinct proto package
name so ziti's and the library's messages coexist without a global proto
registry clash
- adds a round-trip test asserting wire compatibility with the library's
MetricsMessage
- leaves openziti/metrics unchanged, so sdk-golang and the shared xgress data
plane are unaffected
- moves the channel dependency to channel/v5 v5.0.10 and sdk-golang to v1.9.0 in the root and zititest modules
- mechanically rewrites every channel/v4 import path to channel/v5
This is the import-path-only step; the API-level changes the switch requires land in the following commit. This commit does not build on its own.
- adds CreateCircuitV3 message type and handler for routers that have
already authorized dials locally via RDM, bypassing service session
tokens in favor of identity ID, service ID, and pre-assigned circuit ID
- renames CreateCircuitRequest/Response to CreateCircuitV2Request/V2Response
for clarity now that V3 exists
- adds CircuitManager.Reserve to atomically claim circuit IDs before routing,
preventing collisions on pre-assigned IDs
- extends CreateCircuitParams with GetCircuitId so V3 can supply a
pre-assigned circuit ID (falls back to UUID generation when empty)
- fixes IsDialableByIdentity which was incorrectly calling IsBindableByIdentity
- extracts V2 handler into its own file create_circuit_v2.go
- adds CreateCircuitV3RequestType/ResponseType (20222/20223) to edge_ctrl protobuf
- registers V3 handler in controller server
- Adds unrouteRemovedPathNodes to send Unroute to old path nodes not
present in the new path after a reroute
- Calls it from rerouteCircuit and smartReroute after successfully
routing the new path
- Uses GetConnectedRouter to get the current router object rather than
stale path pointers
- Fixes orphaned forwarder entries on transit routers replaced during
reroute, which caused circuit validation failures
* When routers change (connect/disconnect), we re-check the mesh. We are passing in the routers to the channel and they can build up, which is unnecessary. We only need to signal once for all the routers that change since the last time we checked, that the mesh needs to be checked.
* We pass link changes through the run method, which is unnecessary since we pass it to a new goroutine. This inefficiency should be fixed.
* We're evaluating all faulted links for rerouting, even if they were still pending. Only reroute connected links.
* We're using time.After in each for loop, which can accumulate timers. Use a single Ticker instead.
Paul Lorenz