A fleet reconnecting at once ran every connect setup concurrently, each
building links and validating terminators, with nothing bounding how many were
in flight. The work itself is sub-second; the failure is in the aggregate, and
nothing reported it, so a controller burying itself under reconnects looked
from the outside like one that had simply gone quiet.
- splits the connect into the decision and the setup: the decision stays synchronous, so the bind handler can still fail a refused connect and at most one connection per router is current at a time
- runs the setup on a goroutine bounded by a semaphore, defaulting to 200 at once, configurable as routerConnectConcurrency
- never refuses for capacity: a refusal fails the bind and strands the router's channel group, and the expensive part of connecting is already paid by the time this is reached, so the setup waits instead
- re-acquires the per-router lock and re-checks currency inside the setup, so it is serialized against a concurrent disconnect and skipped entirely if the connection was superseded while queued
- gives up after a configurable timeout and closes the control channel rather than skipping the setup, since skipping leaves a router that is connected, believes it is, and has no links built
- reports the queue depth as a gauge and the wait and the work as separate timers, since the two have opposite remedies and the timers record only on completion, so the gauge is the one signal visible during a wedge
The scan walks the link table and removes from it in a second pass, and on a
large table the gap between them is wide enough for a dialer to reconnect and
re-announce its links. Acting on the first pass's answer removes a link the
controller has just been told about, and a router announces its set only on
reconnect, so it does not say so again until it next connects. A soak run lost
53 links this way.
- re-tests each candidate immediately before removing it, and skips one whose dialer returned mid-scan
- takes a keep predicate, so a caller can spare links whose dialer does not re-announce them
- records whether a retired link's dialer is connected under a different object, which distinguishes a router that is genuinely away from a link still pointing at a connection that has been replaced; the second is a stale pointer rather than a dead dialer and wants a different fix
Controllers compute peer state independently and nothing orders their messages
against each other, so a router could be told a destination was removed and
then told by another controller, from state computed before the removal, that
it was healthy again. The destination came back and the router went on dialing
a router that was gone.
- carries the controller store index a peer state was computed against, and zero where the reporter has none to offer
- remembers a removal against its index, so a report computed against an older one cannot recreate the destination
- loses a tie: a router being deleted stays in a controller's connected map until its control channel teardown runs, so the same store state can still produce a healthy report
- suppresses only existence, never health, since health is each controller's own view of what it is connected to and a controller lagging in replication reports a low index alongside a perfectly good health report
- names the reporting controller on each update, so the registry can union health across controllers while ordering existence by index
- forgets a removal after a few minutes, which outlives any report already in flight
A link a router reported without re-announcing is known only from that report,
under an id the router will never mention again, so its disconnect is the only
removal the link will ever get. A router that flags its reconnect announcement
a full refresh is the opposite case: it states authoritatively which links it
still has and the controller prunes the rest against that.
- retires a disconnecting router's links only where that router does not prune its own
- leaves the links of a router that does prune them, since the announcement is a bounded best-effort send and an outage outliving its retries would otherwise leave this controller holding no links for a router that is up
- records on the router whether it announces a full refresh, taken from what it said on connect
A fleet that churns routers accumulated links to routers that were not there.
The cleanup dropped the deleted router's index but left the links themselves in
the link table and in the surviving endpoint's set, where nothing would remove
them: a deleted router's id never returns, since a recreated router is assigned
a new one, so nothing later could clear them. Each one still counted, still
validated and still offered itself for routing.
- retires every link with the deleted router at either end, dialed or accepted, since the index is keyed on both
- retires the way a fault does rather than merely deleting, so circuits using the link are rerouted and the removal reaches the event stream
- retires before the index is dropped, since retirement reads that index to find what to retire
- retires by id without checking whether the id has come back, which needs a router created with a chosen id and a delete and recreate inside the commit-handler window; guarding it is not worth the machinery
Peer state is only recomputed when a router connects or disconnects, and a
deleted router does neither again, so a removal a router misses is never
repeated and it goes on dialing the dead destination. A link report naming the
destination is the evidence that it was missed, and reports repeat, so the
correction can repeat with them.
- re-asserts a removal to the router that named it, queued against that router alone rather than fanned out to every connected one
- computes the state at send time, so a destination that turns out to exist after all is reported as it is rather than as removed
- tells every router that names the destination, whatever it advertised, since the statement is true regardless of who hears it
- answers only from the leader, since this is an unsolicited assertion that tears links down, and a controller lagging on a create would otherwise tell a router to drop a destination that is fine
A nil destination is ordinary: the link is recorded and repaired when that
router connects. It also covers a router that was deleted, and that link can
never be repaired, since a recreated router is assigned a new id. Left alone it
is a link to an entity that does not exist: it counts, it validates, it offers
itself for routing, and it cannot be rendered by the API, which fails the whole
listing rather than one row.
- refuses a report whose destination is neither connected here nor present in the store, which is what stops the dialers re-announcing it back into existence on every reconnect
- answers the lookup from the router cache, falling through to the store on a miss
- treats a missing lookup as existence, so a report is never refused on the strength of one
- gives the two tests that reported a link to an unpersisted destination a destination that exists
Control channel send back-pressure was showing up as a p99 at the send timeout,
with no way to tell which handler was holding the receive goroutine or what it
was waiting on. A dump taken after the fact records the goroutine unwinding the
diagnostic rather than whatever it was blocked on, so the snapshot has to happen
while the handler is still in the handler. That costs a timer per message, which
is why this is opt-in rather than always on.
- wraps every control channel receive handler on both sides, timing it and
snapshotting all goroutines while a slow one is still in the handler
- installs nothing when disabled. A nil detector's Wrap returns the handler it
was given, so a process that has not asked for this pays no timer, no clock
read and no branch per message, rather than paying a check
- deduplicates dumps by a normalized signature so a process stuck in one place
writes one file and counts the repeats, rather than filling the disk with the
same picture
- names the subject of each dump, since the goroutine being diagnosed cannot be
picked out of the dump and would otherwise be filtered out as a singleton
- makes the thresholds and the dump tracking configurable, because what to dump
on varies by what is being chased: 500ms finds a wedged handler, lock
contention wants tens of milliseconds, and a rare event wants more distinct
dumps kept and less time between them
- builds one detector per process rather than per channel. Per channel would
quietly turn the dump interval and the budget of distinct dumps into per
connection limits, and those limits are what bound the disk a dump can cost
- refuses settings that would produce nothing, such as a zero threshold or no
dumps allowed, but only when enabled, so a config left over from an
investigation does not stop a process starting once it is switched off
- reports the router's own control channel state
It lives in common/diagnostics, shared by the controller and router rather than
duplicated per side, and is documented commented-out in the sample configs.
(cherry picked from commit d562afe1d1)
Metric filtering ran against the raw name, but several metrics are renamed by a
mapper before they are dispatched, so a filter written against the name an
operator actually sees did not match. A raw-name match also passes the gate and
then selects no field, so the event is built and discarded.
Mappers were registrable at runtime through an exported method, which nothing
used and which invited registration after dispatch had started, when the mapper
set is read without synchronization.
- filters on the mapped name, which is the name the metric is emitted under
- unexports mapper registration so the set is fixed before dispatch begins, and
drops it from the dispatcher interface and its mock
- guards the id split against a name with too few segments to carry one, which
sliced past the start of the name and panicked the mapper goroutine
- moves the mappers into their own file, since they were embedded in the
dispatcher and are the part most likely to be extended
(cherry picked from commit a0631b46c9)
A controller or router that is starved shows up as slow handlers, timeouts and
back-pressure, all of which look like faults in whatever happened to be running
at the time. Nothing in the metrics said whether the host itself had capacity, so
telling a slow component from a slow machine meant going to the host.
- reports host CPU and memory usage through the existing metrics registry, on
both the controller and the router
- carries them on the metrics message so they arrive with everything else rather
than needing a separate collection path
- is off unless enabled, since the collection reads the OS on an interval
Unrelated to gossip; it was committed as part of that work and is separated here.
(cherry picked from commit bdab4b4aba)
A router's id is assigned when it enrolls, except on one path: the fabric router
create API takes the id from the request, and `ziti fabric create router <cert>`
fills it from the certificate's common name. That dates from when a router had a
single identifier rather than a separate id and name, so the certificate named the
router. It is the only way a router gets an id that was chosen rather than
assigned, and so the only way a deleted router's id can come back.
- deprecates the CLI command, which now prints a notice on every use
- documents the API mapper that consumes the supplied id as deprecated. Only the
fabric router create path does; the transit router path already ignores it and
lets the id be generated
- logs a warning on the controller whenever a router is created this way. The CLI
notice is only seen by CLI users, and the point of deprecating rather than
removing is to find out whether anyone is driving the API directly
- adds it to the deprecations list, with the history, since an id taken from a
certificate reads as deliberate design without it
Per-router state cleanup is written against the assumption this establishes, that
a deleted router id does not return.
(cherry picked from commit 6c968b1ed0)
- sends no signer fingerprints in the router data model when a process check has none configured,
the single stored fingerprint was wrapped into a one-element list holding an empty string
- treats an empty configured signer fingerprint as no constraint when evaluating a process check,
routers require a reported signer to match any entry in the list
- reports the configured fingerprints in the signer failure message, it reported the hashes
- adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId,
ListenerBinding}) used by all staleness and GC checks, replacing the
pre-rendered key string each side used to pass around
- adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links`
CLI for operator-driven, two-sided staleness verification with optional --gc,
which acts only when both endpoints agree
- implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed):
the router walks its xlinks after each Apply that mutates listeners, dialers,
or gcMode and closes one-sided-stale entries under the configured mode
- sweeps a removal under orphaned whatever the removed config said, including
preserve, since a removal leaves no listeners and no dialers and every
remaining link is one the router can never re-establish
- reports staleness as a three-state verdict (unknown/notStale/stale) so a side
that cannot judge a link abstains instead of guessing; a destination absent
from the registry's listener snapshot means "listeners unknown", not "peer has
none", which keeps a briefly unhealthy peer from losing its live links
- rebuilds the link surface only when a listener or dialer definition actually
moves, so a gcMode-only change no longer rebinds every listen socket
- gates CheckStaleLinks on a new RouterStaleLinkCheck capability and explains
each unqueryable endpoint, including one left unasked by the filter, so a
partial verdict says which of the three causes applies
- detects a peer re-advertising the address a link was dialed to from the
dialing side, not just from the peer's own listener side
- pins every verdict to the link iteration it was computed against, faults that
iteration rather than the current one, and takes the conditional removal's
result as the decision, so a link re-dialed while the sweep was running is
neither closed nor reported as collected
- resolves which side a report speaks for from the sending router against the
controller's link map rather than from the side the report names, so one
router cannot fill both sides and satisfy the both-agree rule --gc depends on
- bounds the query fan-out with a semaphore, matching the other validation
sweeps, since each query carries a minute-long reply timeout
- fails the CLI when the result stream ends before every link is reported, and
drains results already buffered before declaring any missing
- rejects an unrecognized match mode rather than resolving it to changed, the
broadest removal criterion
- consolidates the controller-managed router configuration changelog into one
section, replacing the status that said routers do not yet consume configs at
runtime, and documents the per-config-type opt-in
- restores the Current Beta Features section, dropped when the 2.0 changelog was
archived, and lists controller-managed router configuration in it
- adds unparseable to the codespell ignore list, a spelling variant the
dictionary flags as a typo
For #3743.
BuildRouterLinks runs once per router connect and scanned the whole link table to
find the connecting router's links. The table is proportional to the square of the
router count in a full mesh, so a wave in which every router reconnects walks the
cube of it in link visits. That never shows in steady state; it arrives during a
mass reconnect, when the connect path is already the bottleneck.
Indexing every link under both endpoint router ids fixes the cost, and introduces
the problem that the table and the index are two structures written in sequence.
Three things follow from that, and the destination repair below turned out to
depend on the connect path's ordering as well.
Performance:
- indexes every link under both endpoint router ids, maintained in the add and
remove paths all link mutations funnel through, so a connect costs its own
router's link count rather than the whole table
- prunes a full refresh against the reporting router's own links. That report is
what every reconnect sends, and it walked the whole table to find the router's
stale links, so the cost the connect path shed was still being paid one message
later
- indexes link ids rather than link pointers, so a per-router query resolves each
id against the table and hands back whatever it currently holds. A reconnect
replaces a link's Router object and a higher iteration replaces the Link object,
but neither replaces the id, which is exactly why the scan existed
- holds each router's link ids in a LockedSet rather than a sharded map, which
cost a map and a mutex per shard for an index that is usually empty and is held
for as long as the router has existed: 2.9KB per router id against 150 bytes
- drops a router's index when the router is deleted, guarded on the store still
having no router under that id and checked inside the index map's removal
callback. A router id can come back, since fabric router ids are the enrollment
certificate's common name, and nothing orders the cleanup against that: store
event handlers run after bolt has released the writer lock, so one can execute
after the id was recreated, connected and reported links. Indexing a link takes
the same shard as the guarded read, so an entry for a live router cannot appear
without its create having committed first, which the read then sees, and an entry
arriving later blocks and lands in a freshly created index
- keeps an index when the store cannot be read, since that is not evidence the
router is gone, and dropping one otherwise could discard a link another goroutine
had already fetched the index to record
Keeping the two structures agreeing:
- serializes Add and Remove on the link's id. Without it a removal running between
an add's two writes takes the table entry, finds nothing yet in the index, and
lets the add index a link the table no longer holds, which nothing cleans up.
Only RouterReportedLink held that lock before; the four other removal paths held
nothing
- serializes the connect-time pairing on that id as well, which was the last writer
left outside it. A removal reading no destination could complete while the
pairing was still deciding, leaving the link in the destination's link set after
the table had dropped it, where path computation still routed over it
- unindexes an id only when the table no longer holds a link under it, so a
replacement that has taken the table slot is not unindexed by the older link's
removal. A stale id is the safe direction to err in, since a query filters it,
where an id wrongly dropped leaves a live link no per-router query finds
- has per-router queries resolve ids against the table, so an id it no longer
holds is not handed out as a live link
Pairing a link with its destination:
- re-resolves the destination once the link is in the table, and on every later
report, so a link that reached a bad state heals. A report resolves the
destination before the link is in the table and a router's connect repairs links
already in it, so a router connecting between those two moments was seen by
neither, leaving a link with no adjacency that every operator-facing view still
called healthy
- makes that re-resolve unconditional rather than only for a link holding no
destination. The resolve happens under the source router's connect stripe, not
the destination's, so the destination can be replaced between the resolve and the
report landing, and a link left on a connection that is no longer registered
carries no adjacency just as one holding none does
- registers a connecting router before building its links, which is what makes
that re-resolve close the window rather than narrow it
- adds Link.PointDestAt so the two repair paths cannot both index the link on the
destination, since that index is a slice that does not deduplicate
Also here:
- logs the link count on both sides of the reconnect exchange, and why a report was
discarded. A router announces its links once per reconnect and is not asked
again, so a discarded report left it with no links for a router that believed it
had announced them, with nothing recorded to compare. The discard reports the
state it decided on, read once, since a reconnect landing between the decision
and the log would otherwise have it name a state that discards nothing
- gives test routers version info, which a connected router always has because the
accept path refuses a hello without it, and which the router sync path
dereferences with no nil check
The destination repair is backported from the gossip link-state branch, where the
connect path registers before building links for unrelated reasons. That ordering
is load-bearing here and is now explicit at the call site.
- resolves zt-session tokens eagerly when the request context is created, so every request carrying one marks activity and gets session lifetime headers, regardless of endpoint
- marks last activity in SecurityCtx.resolveZtSession once the session is loaded
- adds SecurityTokenCtx.HasZtSessionHeader, a header-only check that does not parse bearer tokens
- emits only session lifetime headers from the API wrappers and drops the unused session-error header branch, so a stale token on an anonymous endpoint stays silent as it did on 1.6.x
- tests that requests to authenticated and anonymous endpoints with a zt-session mark activity, carry lifetime headers, and are reflected by current-api-session
* fixesopenziti/ziti#4321 compare process hashes and fingerprints case-insensitively
- normalizes configured hashes and signer fingerprints to lowercase unseparated hex in the
PROCESS and PROCESS_MULTI stores, on load as well as on save, so values stored before this
change read back normalized without a migration
- replaces the PROCESS store's lowercase-only pass, PROCESS_MULTI stored its values verbatim
- adds store tests for both types covering configured values and values written straight to bolt
- updates the process multi patch round-trip test, the management API now returns normalized
values rather than the submitted strings
- removes the trailing newline from the signer fingerprint literal in the process and
process-multi OIDC posture tests
Registry.Meter is reference counted: every call takes a reference, including calls
that find the meter already present, and only Dispose releases one. The enforcers
resolved a meter inside Run, so each run took a reference it never shed. Disposing
such a meter decrements the count instead of stopping it, so it leaves the registry
and keeps sampling.
- resolves each enforcer's meter once, where the enforcer is built, and holds it
- holds the timers alongside, though Timer is not reference counted, so a reader
is not left working out why one of two adjacent lookups was cached
- supplies the metrics in the session enforcer test, which builds the enforcer
directly to get a negative session timeout that NewSessionEnforcer refuses
Per enforcement cycle rather than per event, so the growth was one reference per
run. Unbounded over a controller's lifetime, but slow.
grep -rnE '\.(Meter|Histogram)\([^)]*\)\.(Mark|Update)' finds this shape; there are
no others left outside tests.
- normalizes reported MAC addresses in Instance.Apply to lowercase unseparated hex,
the form MAC posture check values are persisted in
- stores a new PostureResponse_Macs rather than mutating the incoming protobuf message
- compares the normalized addresses when deciding whether posture data changed, so a
re-reported separated address no longer reads as an update
- moves the normalizer to a new common/posture package, shared by controller/db and
router/posture, and hoists its regexp out of the per-call path
- adds an oidc-auth-test fablab model that exercises OIDC authentication end to end, with an event-forwarder component, an oidc-test-client, a ziti-prox-c component, and OIDC event/gossip/traffic validations
- moves zitirest out of the public API into separate test-only shims: an integration-test shim (tests/restclient.go) and a fablab shim (zititest/zitirest), and repoints upgrade-test at the fablab shim
- adds region-isolation network partitioning and packet capture on all interfaces to the chaos toolkit
- queues oidc-test-client results reporting on a single sender goroutine with a bounded write deadline, so a stalled results circuit can't block the traffic loops
- counts results events dropped when that queue fills and reports them to the collector as errors, so a reporting outage fails validation instead of looking like a clean run
- bounds the event forwarder's event and keepalive writes, and checks for shutdown on every send attempt, so an unreadable destination can't park the forward loop holding its lock
- logs read errors on collector connections instead of ending collection for that client silently
- bounds the oidc-auth-test debug server's header and body reads
- matches ziti-prox-c process matching to the version-agnostic binary name, so a version change still finds the running process
- points upgrade-test's per-service terminator count at the shared validations helper
- adds a fablab design summary doc
- withholds an enrollment's token and jwt from a non-admin caller when the owning
identity is an admin, in the identity detail mapper and the enrollment mapper
- scopes the identity enrollments subresource so a non-admin does not see
enrollments of admin identities, matching the top level /enrollments list
- threads the request context into MapIdentityToRestModel and
MapEnrollmentToRestModel so the mappers can see the caller's permissions
- adds an integration test over the identity detail, identity list, and identity
enrollments routes for every enrollment method
- adds the advisory to the 2.1.0 changelog
- removes a stray assignment that overwrote isCertExtendable with a literal true on every OIDC certificate authentication
- reports isCertExtendable as false over OIDC for cert authenticators not issued by the network, matching legacy cert auth
- adds coverage asserting the z_ice claim and current-api-session both report a 3rd party CA certificate as not extendable
* fixesopenziti/ziti#4094 accept first-party certs issued by a separate edge signing CA
- adds FirstPartyX509CertValidation and ThirdPartyX509CertValidation usages and an
intermediates field to the router data model public keys, deprecating
ClientX509CertValidation
- publishes config CA bundle roots as first-party anchors with their intermediates and
Ca store entries as third-party anchors; controller certs carry JWT validation only,
since a controller identity is never a CA and anchors no client cert chains
- builds router first-party and client cert trust pools from the published usages,
falling back to the deprecated usage against older controllers
- propagates the full signing cert chain between controllers via a new mesh
SigningCertChainHeader and persists whole chains in controller records
- removes the orphaned InstantStrategy.AddPublicKey, dead since public key sync moved
to controller list data
- gives each command dispatcher its own decoder registry so multiple in-process
controllers no longer decode into the last-started controller's managers
- adds a three-controller in-process HA test harness with a split signing PKI,
cluster formation and first-party cert integration tests
- trusts the edge signing CA when verifying router control channel certs
- adds a variadic additionalRoots parameter to VerifyLeafCertChain, applied to a
clone of the caller's pool so an identity's live tls.Configs are unaffected
- passes the edge enrollment signing CA bundle as additional roots when admitting
a router control channel connection, so a deployment whose signing CA sits
outside the controller's own trust bundle no longer has every router refused
- leaves the fingerprint check bound to the verified leaf, so the wider anchor set
changes which chains verify, not which routers are admitted
- covers the split-root case and the caller-pool guarantee in common/cert tests
- adds a package-level buildFlags string the linker sets at build time, parsed
into a capped list of [A-Z0-9_] names with blanks, duplicates, and malformed
tokens dropped
- returns those names in the new buildFlags field on /version, separate from
capabilities, and prints them under ziti version -v
- bumps edge-api to v0.36.0 for the buildFlags field
- adds a fablab model (zititest/models/upgrade-test) that stands up a multi-region HA controller cluster, routers, and loop4 sim traffic clients (SDK, ERT, ZET, tunneler) and drives repeatable in-place upgrade/downgrade iterations, re-bootstrapping in place each iteration
- exercises HA cluster disruption and recovery (snapshot restore, node rejoin) between iterations and validates steady-state traffic and expected terminators after each disruption
- honors the raft restartSelf setting on migration-snapshot restore, so a restored controller restarts itself instead of exiting, and promotes RestartController to a package-level function
- re-sends loop4 run-scenario requests to sims that reconnect mid-scenario, tracking each scenario's expected client set explicitly and ignoring results from clients outside it, so post-disruption client reconnection no longer stalls validation
- closes leaked loop4 sim-control connections on error and on supersession
- rotates component logs and pins the log-pipe binary to the local build, and drops router debug logging, to keep long iterating runs disk-bounded
- excludes the doc/unsettled/ scratch area for in-progress design docs from version control
- requires an API session token on CreateCircuitV3 requests and validates it,
covering signature, audience, token type, and revocation by token id,
identity, and api session
- takes the dialing identity from the validated token claims rather than the
router-supplied identity id, and rejects a request whose asserted identity
does not match the token subject
- adds the api session id to the log context, matching the V1 and V2 paths
- adds tests for a missing token, an invalid token, and a token belonging to a
different identity than the one asserted
- notes the advisory in the 2.1.0 release notes
- enforces issuer and audience claims inside TokenIssuerExtJwt.VerifyToken, which
previously verified only the token signature and resolved the signing key by kid
- closes the ziti-token-issuer-id header enrollment path accepting a validly-signed
token minted for a different audience or issuer that shares signing keys
- mirrors the runtime ext-jwt authentication and by-inspection enrollment paths so
all paths validate claims consistently
- adds a controller/model unit test covering foreign-audience, foreign-issuer, and
missing-audience rejection
- adds HardenedJwksResolver, a jwks.Resolver that fetches an external jwt signer's
jwksEndpoint with an http/https-only scheme check, a total timeout and a redirect cap
- adds JwksFetchPolicy, gating a fetch on both the URL hostname and the address being
connected to, applied to the first request and to every redirect hop
- hostname gate: deniedHostnames blocks, allowedHostnames is exclusive when set; entries
are an exact hostname or a '*.suffix' wildcard that matches subdomains at any depth but
never the suffix itself, normalized to lower case punycode without a trailing dot
- address gate: built-in blocked (metadata, link-local, link-local multicast,
unspecified), then deniedIPs, then allowedIPs, then blockPrivateAddresses,
first-match-wins with deny over allow
- keeps the gates independent, so neither can authorize what the other refuses; the
address check runs in the dialer against the resolved address, so a hostname that
resolves to a blocked address is refused
- adds the [edge.externalJwtSigners.jwksFetch] config section with compatible defaults:
empty hostname lists, blockPrivateAddresses false, timeout 5s, maxRedirects 5
- takes IP lists as a flat address or a CIDR block, hostname lists as names only, and
rejects an entry belonging to the other list at startup
- rejects a jwksEndpoint the policy refuses when an external jwt signer is created or
updated, and logs an existing signer whose endpoint the configuration now refuses
when the token issuer cache loads
- documents both gates, their deny-wins precedence, the accepted entry forms and the
wildcard matching rules in etc/ctrl.with.edge.yml and CHANGELOG.md
- Creates a router sender on demand when an untracked router subscribes to the
data model, so non-edge (transit) routers receive their controller-managed
configuration without going through the edge connect/sync flow
- Marks on-demand senders as supporting the router data model so they receive
live model updates, not just the initial sync
- Restricts the legacy api-session/session broadcasts to edge routers, since
routers register those handlers universally and would otherwise load session
state they never use. A router counts as edge when an edge router record
exists for its id, which is how both the connect path and the subscribe path
classify it; senders carry the result as an atomic edge flag and the fanout
walks them via RangeEdge. Classification is fixed for the life of a
connection, so a record added or removed under a live connection takes effect
on reconnect. Transit routers also never receive the initial session state,
which only follows a hello
- Adds a race-safe routerTxMap.GetOrCreate keyed on the control channel; the
subscribe path always routes through it, replacing a sender bound to a stale
channel instead of enqueueing on a dead one, and resolves edge-router
membership up front so a replaced edge sender is not demoted to transit
- Adopts an existing sender for the connecting channel in RouterConnected instead
of rejecting it as a duplicate connect; a subscribe that creates the sender first
would otherwise leave the router with no server hello, so it never sends a client
hello and never synchronizes while still being reported online
- Removes routerTxMap.Add, leaving GetOrCreate as the only way a sender is installed
- Classifies a connecting router as edge by id rather than by fingerprint, matching
the disconnect and subscribe paths, and logs an unexpected store error instead of
discarding it
- Removes the unused RouterSender.EdgeRouter field and drops the edge router
argument from the RouterConnected handler
- Generalizes router connect/disconnect logging that previously assumed edge routers
- Updates the strategy flow comment and RouterSender godoc, which described senders
as edge-only and referenced the removed EdgeRouter field
- Documents that api-session-added events and the legacy session model (API and
service sessions) are deprecated for removal in OpenZiti 3.0
- Adds unit tests for GetOrCreate (create/reuse/stale-replace/concurrent), the
edge-only fanout filter, and RouterConnected adopting a subscribe-created sender
routerConnectChurnLimit predates the per-router connect lock. It was added
alongside the ability for a new control channel to take over from an established
one, as the guard on how often that may happen, and it was also the only thing
keeping two connections for one router out of the connected map.
That second job is gone: at most one connection per router is now enforced under
the per-router lock, where the decision is atomic. The check in the accept path
runs against the connected map with no lock held, so it can only refuse a
connection early that would be refused there anyway.
Its first job remains, and is now the only thing doing it. ConnectRouter always
displaces an occupant it does not recognise, so without the limit a spurious
first-connection hello would tear down a healthy control channel and make the
router redial. Nothing said so, and the field carried no godoc at all.
- documents on the option what it protects, that it is churn policy rather than
the uniqueness guarantee, and that zero always allows takeover
- extracts the decision so it can be tested without standing up a network, and
tests it: protected when just established, protected part way through the
window, displaceable once past it, and never protected at zero
The struct's field alignment shifts because a comment ends gofmt's alignment
group; that part of the diff is whitespace only.
Behaviour is unchanged. Worth noting for readers of the option: past the window,
the established connection is now displaced and the connect refused, so the
router redials into the freed slot, where previously the arriving connection took
over directly. Same end state, one extra round trip, and nothing unvetted is
registered on the way.
The controller decided which of two racing connections for a router was current
by comparing router instances, but loaded one per connect by evicting the router
cache and reading back through it. Two connects could both evict, and whichever
read second was handed the instance the first had just published. A shared
instance makes the two connections indistinguishable: the connect path cannot
reject the second into an occupied slot, and when either channel dies the
disconnect path finds itself current and tears down the registration the other is
still using. The surviving channel is never re-bound, so the router stays
connected at the transport layer while absent from the model, unable to recover.
Connect and disconnect were also unserialized, so a stale or superseded
disconnect could interleave with a live connection and take its links with it.
- serializes a router's connect and disconnect with a per-router striped lock
- keeps at most one connection per router: a connect into an occupied slot is
rejected via an error from ConnectRouter, so the bind fails and NewChannel
closes it without starting rx or registering it, and the occupant is displaced;
the router redials into the freed slot
- displaces an occupant by closing its channel and also invoking the teardown
directly, since a channel that is already closed never fires its close handler
again; without this a dead but still registered connection holds the slot
forever and every redial is rejected against a slot nothing can free
- refuses a connect whose control channel is already closed rather than
registering it, so a connection no disconnect could ever remove is never
published
- gives every connection its own router instance via RouterManager.NewCtrlChanRouter,
read through readUncached so the cache neither supplies nor receives it, which
is what makes comparing instances meaningful
- moves recording the channel and connect time out of the accept path, so a
caller cannot attach the wrong channel or forget to attach one
- serializes link publication with that teardown on the same per-router stripe.
Validating currency and then publishing without it is a check-then-act: a
report can find the connection current and, by the time it reaches the link
manager, the teardown has already snapshotted and cleared the router's links,
so the link is recreated after everything that would have removed it. It is
then absent from the router's own index while still in the link table with a
disconnected source, and a reconnect reporting the same iteration can adopt
that stale source instead of rebuilding the link
- guards the entire DisconnectRouter teardown by connection currency, all or
nothing, and clears the connected flag and link index only when the
registration was actually given up, with the flag cleared under the same shard
lock as the map removal so the two cannot be observed disagreeing; the connected flag decides whether the
controller accepts a router's link reports, so clearing it for the wrong
connection silences a router that is up and reporting
- reduces MarkConnected to publishing the connection; the takeover-close moves
into ConnectRouter's reject path
- makes the per-router unlock idempotent so callers can defer it as a leak-safety
net and still unlock early before closing a channel outside the lock
- stops the replaced RouterSender in routerTxMap.Add so a takeover does not leak
the old sender's goroutine when the broker's asynchronous RouterDisconnected
loses the race to the redial's RouterConnected
- discards pending peer state changes for a router whose channel has closed,
since sending on one fails immediately and the failed send is retried as soon
as the event loop turns, spinning the loop and flooding the log
- queues the peer-state send-done event on every path, so a missing channel can
no longer leave sendInProgress set and stall that router's updates permanently
- resolves a router's version from its connected instance when validating link
conn info, since the version arrives in the hello and so is absent from an
instance loaded from the database
- normalizes both endpoints to the connected instance in shortestPath, which is
keyed and compared by pointer and so treated an endpoint held as any other
instance of the same router as absent from the graph, reporting a router as
unroutable from itself. That worked before only because the connect path
published its instance into the router cache, so a cache read and the connected
map returned the same object; nothing stated the requirement
- configures test logging once per package in TestMain, so a test no longer
writes global logger state while a previous test's shutdown logging reads it
- swaps testing.TB for require.TestingT in model.NewTestContext, so this
build-included file no longer imports testing
- documents why the narrower interface is used, so it isn't reverted
- leaves hashicorp/raft as the only remaining importer of testing in the
ziti binary's dependency graph
- swaps testing.TB for require.TestingT in db.NewTestContext, so this
build-included file no longer imports testing
- documents why the narrower interface is used, so it isn't reverted
- swaps testing.TB for require.TestingT in NewTestContext and NextTest,
so this build-included file no longer imports testing
- removes the write-only t field from BaseTestContext
- documents why the narrower interface is used, so it isn't reverted
- adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop
- translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup
- adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path
- re-evaluates dialers on link group and listener changes via RescanForDialOpportunities
For #3743.
Paul Lorenz