- adds RemoveTerminatorsV2, an asynchronous control-channel removal, so a router
no longer holds a request slot open waiting for a synchronous reply; gated on a
new capability so routers fall back to the synchronous path against controllers
that do not advertise it
- paces establish and remove retries with a per-terminator exponential backoff
after a controller rejection, so a rejected operation is not re-sent on the next
event-loop tick, which is what let a rate-limited removal sustain itself
- routes v2 removes to the leader and skips the ordered delete for a terminator the
router confirms it created that no longer exists, so a storm of no-op retries
cannot consume the command rate limiter
- gates that skip on the leader having applied everything committed when it took
leadership, since a leader reports itself elected before those applies finish and
would otherwise read a committed create as absent and drop its delete
- guards the leader term and every write to the caught-up flag with a dedicated
lock, so a barrier completing alongside a leadership change cannot record a result
for a term that has already ended
- scopes v2 removals to the requesting router, so a router can only remove
terminators it owns on the asynchronous path as well as the synchronous one
- tracks in-flight removal batches on the router, resolving or requeueing them when
the response arrives; lost responses self-heal via rate-limiter expiry and the
retry scan
- waits for an outstanding operation before sending a terminator delete, and checks
that before dequeuing, so a delete neither races a create the controller has
committed but not applied nor falls out of the queue until the retry scan finds it
minutes later
- matches a create response to the attempt that produced it, by control channel and
reply sequence, since attempts overlap once one overruns the establishment timeout
and every attempt carries the same terminator id; only the outstanding attempt's
response may confirm the create or report the terminator idle
- records the attempt as the sender assigns its sequence, so a response cannot be
processed before the request it answers is known, and withdraws an attempt whose
send never queued
- reports a removal that succeeds past the establishment timeout as congestion
rather than success, matching how the establish path classifies its own latency,
since both resolve into the same limiter
- extracts the transient-error test both remove handlers share, so the v2 path
treats a leaderless cluster as busy like v1 does instead of as a permanent
failure, and the two cannot drift again
- applies the same delete sequencing and response correlation to the router-hosted
(ER/T) registry, which cleared the same in-flight marker unguarded and dropped a
deferred delete from its queue rather than leaving it there
- bounds the v2 response send with a timeout, so a router that stops draining its
send queue cannot accumulate blocked responder goroutines on the controller
- applies the router.link.v1 heartbeat settings (sendInterval, checkInterval,
closeUnresponsiveTimeout) and payload/ack sender queue sizes at runtime,
replacing the hardcoded 10s/1s link heartbeat intervals and the ignored fields;
new links take them at bind time, established links are re-tuned in place
through the channel HeartbeatControl, and queue sizes reach new links only
- publishes heartbeat settings as an immutable generation in one atomic swap, so
no reader pairs one generation's interval with another's timeout, and has each
link enforce the timeout belonging to its own intervals
- pushes a generation and its intervals as one step under the link's lock and
ignores a generation at or below the one applied, so an overtaken update cannot
roll a link back; reconciles every channel of a link, including a split link
whose channels bind on either side of a config change, to the newest generation
- adopts the first registered settings explicitly, since a router with no managed
link config publishes its defaults at generation 0
- re-arms a link's liveness deadline when its generation changes, so age
accumulated under the previous intervals cannot close a healthy link when a
timeout is shortened
- honors a closeUnresponsiveTimeout below 30s, which previously sat behind the
unhealthy threshold; treats a zero timeout as closing nothing, and reports a
closing link unhealthy
- scales the unhealthy threshold with the intervals, to max(30s, 2 x (send +
check)): reporting unhealthy poisons the latency metric the controller routes
on, so a fixed 30s priced healthy links with a long send interval out of
routing every cycle
- validates heartbeat timing on the controller and the router: durations must be
positive, their sum representable, and the timeout at least 10s and greater
than sendInterval + checkInterval; a timeout under two heartbeat cycles is
accepted with a warning
- adds a per-config-type semantic validator registry to ConfigManager, run after
schema validation, and moves the router.link config definition to
common/config/routerlink so the controller applies the same rules
- treats a link section in router YAML, of any content, as local config that
takes precedence over the controller's for the whole router.link type, since
the loader defaults every field and so now records presence separately; warns
at startup when that section has no listeners or dialers
- translates local heartbeat timing leniently, since a local failure stops the
router from starting: keeps an explicit zero, raises a timeout below the floor
to it, and falls back to the defaults for other invalid timing, each with a
warning
- stamps UpdateLinkListeners with the config generation it came from and has the
controller drop superseded sets, which a multi-underlay control channel can
deliver out of order and which would otherwise send peers to withdrawn
addresses
- logs every successful link config apply, including those that dispatch no work,
and pins that a queue-size-only change triggers no GC pass
- deprecates split links, documents the feature in the changelog and the router
config design doc, and brings both up to date
Builds on the channel/v5 HeartbeatControl from openziti/channel#265.
- the debug field bound the Code method left behind by the AppCode rename, rendering a function
value and failing to marshal under the JSON formatter; it now logs AppCode
- sends no signer fingerprints in the router data model when a process check has none configured,
the single stored fingerprint was wrapped into a one-element list holding an empty string
- treats an empty configured signer fingerprint as no constraint when evaluating a process check,
routers require a reported signer to match any entry in the list
- reports the configured fingerprints in the signer failure message, it reported the hashes
- adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId,
ListenerBinding}) used by all staleness and GC checks, replacing the
pre-rendered key string each side used to pass around
- adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links`
CLI for operator-driven, two-sided staleness verification with optional --gc,
which acts only when both endpoints agree
- implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed):
the router walks its xlinks after each Apply that mutates listeners, dialers,
or gcMode and closes one-sided-stale entries under the configured mode
- sweeps a removal under orphaned whatever the removed config said, including
preserve, since a removal leaves no listeners and no dialers and every
remaining link is one the router can never re-establish
- reports staleness as a three-state verdict (unknown/notStale/stale) so a side
that cannot judge a link abstains instead of guessing; a destination absent
from the registry's listener snapshot means "listeners unknown", not "peer has
none", which keeps a briefly unhealthy peer from losing its live links
- rebuilds the link surface only when a listener or dialer definition actually
moves, so a gcMode-only change no longer rebinds every listen socket
- gates CheckStaleLinks on a new RouterStaleLinkCheck capability and explains
each unqueryable endpoint, including one left unasked by the filter, so a
partial verdict says which of the three causes applies
- detects a peer re-advertising the address a link was dialed to from the
dialing side, not just from the peer's own listener side
- pins every verdict to the link iteration it was computed against, faults that
iteration rather than the current one, and takes the conditional removal's
result as the decision, so a link re-dialed while the sweep was running is
neither closed nor reported as collected
- resolves which side a report speaks for from the sending router against the
controller's link map rather than from the side the report names, so one
router cannot fill both sides and satisfy the both-agree rule --gc depends on
- bounds the query fan-out with a semaphore, matching the other validation
sweeps, since each query carries a minute-long reply timeout
- fails the CLI when the result stream ends before every link is reported, and
drains results already buffered before declaring any missing
- rejects an unrecognized match mode rather than resolving it to changed, the
broadest removal criterion
- consolidates the controller-managed router configuration changelog into one
section, replacing the status that said routers do not yet consume configs at
runtime, and documents the per-config-type opt-in
- restores the Current Beta Features section, dropped when the 2.0 changelog was
archived, and lists controller-managed router configuration in it
- adds unparseable to the codespell ignore list, a spelling variant the
dictionary flags as a typo
For #3743.
BuildRouterLinks runs once per router connect and scanned the whole link table to
find the connecting router's links. The table is proportional to the square of the
router count in a full mesh, so a wave in which every router reconnects walks the
cube of it in link visits. That never shows in steady state; it arrives during a
mass reconnect, when the connect path is already the bottleneck.
Indexing every link under both endpoint router ids fixes the cost, and introduces
the problem that the table and the index are two structures written in sequence.
Three things follow from that, and the destination repair below turned out to
depend on the connect path's ordering as well.
Performance:
- indexes every link under both endpoint router ids, maintained in the add and
remove paths all link mutations funnel through, so a connect costs its own
router's link count rather than the whole table
- prunes a full refresh against the reporting router's own links. That report is
what every reconnect sends, and it walked the whole table to find the router's
stale links, so the cost the connect path shed was still being paid one message
later
- indexes link ids rather than link pointers, so a per-router query resolves each
id against the table and hands back whatever it currently holds. A reconnect
replaces a link's Router object and a higher iteration replaces the Link object,
but neither replaces the id, which is exactly why the scan existed
- holds each router's link ids in a LockedSet rather than a sharded map, which
cost a map and a mutex per shard for an index that is usually empty and is held
for as long as the router has existed: 2.9KB per router id against 150 bytes
- drops a router's index when the router is deleted, guarded on the store still
having no router under that id and checked inside the index map's removal
callback. A router id can come back, since fabric router ids are the enrollment
certificate's common name, and nothing orders the cleanup against that: store
event handlers run after bolt has released the writer lock, so one can execute
after the id was recreated, connected and reported links. Indexing a link takes
the same shard as the guarded read, so an entry for a live router cannot appear
without its create having committed first, which the read then sees, and an entry
arriving later blocks and lands in a freshly created index
- keeps an index when the store cannot be read, since that is not evidence the
router is gone, and dropping one otherwise could discard a link another goroutine
had already fetched the index to record
Keeping the two structures agreeing:
- serializes Add and Remove on the link's id. Without it a removal running between
an add's two writes takes the table entry, finds nothing yet in the index, and
lets the add index a link the table no longer holds, which nothing cleans up.
Only RouterReportedLink held that lock before; the four other removal paths held
nothing
- serializes the connect-time pairing on that id as well, which was the last writer
left outside it. A removal reading no destination could complete while the
pairing was still deciding, leaving the link in the destination's link set after
the table had dropped it, where path computation still routed over it
- unindexes an id only when the table no longer holds a link under it, so a
replacement that has taken the table slot is not unindexed by the older link's
removal. A stale id is the safe direction to err in, since a query filters it,
where an id wrongly dropped leaves a live link no per-router query finds
- has per-router queries resolve ids against the table, so an id it no longer
holds is not handed out as a live link
Pairing a link with its destination:
- re-resolves the destination once the link is in the table, and on every later
report, so a link that reached a bad state heals. A report resolves the
destination before the link is in the table and a router's connect repairs links
already in it, so a router connecting between those two moments was seen by
neither, leaving a link with no adjacency that every operator-facing view still
called healthy
- makes that re-resolve unconditional rather than only for a link holding no
destination. The resolve happens under the source router's connect stripe, not
the destination's, so the destination can be replaced between the resolve and the
report landing, and a link left on a connection that is no longer registered
carries no adjacency just as one holding none does
- registers a connecting router before building its links, which is what makes
that re-resolve close the window rather than narrow it
- adds Link.PointDestAt so the two repair paths cannot both index the link on the
destination, since that index is a slice that does not deduplicate
Also here:
- logs the link count on both sides of the reconnect exchange, and why a report was
discarded. A router announces its links once per reconnect and is not asked
again, so a discarded report left it with no links for a router that believed it
had announced them, with nothing recorded to compare. The discard reports the
state it decided on, read once, since a reconnect landing between the decision
and the log would otherwise have it name a state that discards nothing
- gives test routers version info, which a connected router always has because the
accept path refuses a hello without it, and which the router sync path
dereferences with no nil check
The destination repair is backported from the gossip link-state branch, where the
connect path registers before building links for unrelated reasons. That ordering
is load-bearing here and is now explicit at the call site.
- moves the delete meter mark out of the batch-delete error branch so successful batches are counted
- counts only the fallback deletes that succeed when the batch delete fails
- asserts the meter count in the session enforcer test
- resolves zt-session tokens eagerly when the request context is created, so every request carrying one marks activity and gets session lifetime headers, regardless of endpoint
- marks last activity in SecurityCtx.resolveZtSession once the session is loaded
- adds SecurityTokenCtx.HasZtSessionHeader, a header-only check that does not parse bearer tokens
- emits only session lifetime headers from the API wrappers and drops the unused session-error header branch, so a stale token on an anonymous endpoint stays silent as it did on 1.6.x
- tests that requests to authenticated and anonymous endpoints with a zt-session mark activity, carry lifetime headers, and are reflected by current-api-session
* fixesopenziti/ziti#4321 compare process hashes and fingerprints case-insensitively
- normalizes configured hashes and signer fingerprints to lowercase unseparated hex in the
PROCESS and PROCESS_MULTI stores, on load as well as on save, so values stored before this
change read back normalized without a migration
- replaces the PROCESS store's lowercase-only pass, PROCESS_MULTI stored its values verbatim
- adds store tests for both types covering configured values and values written straight to bolt
- updates the process multi patch round-trip test, the management API now returns normalized
values rather than the submitted strings
- removes the trailing newline from the signer fingerprint literal in the process and
process-multi OIDC posture tests
Registry.Meter is reference counted: every call takes a reference, including calls
that find the meter already present, and only Dispose releases one. The enforcers
resolved a meter inside Run, so each run took a reference it never shed. Disposing
such a meter decrements the count instead of stopping it, so it leaves the registry
and keeps sampling.
- resolves each enforcer's meter once, where the enforcer is built, and holds it
- holds the timers alongside, though Timer is not reference counted, so a reader
is not left working out why one of two adjacent lookups was cached
- supplies the metrics in the session enforcer test, which builds the enforcer
directly to get a negative session timeout that NewSessionEnforcer refuses
Per enforcement cycle rather than per event, so the growth was one reference per
run. Unbounded over a controller's lifetime, but slow.
grep -rnE '\.(Meter|Histogram)\([^)]*\)\.(Mark|Update)' finds this shape; there are
no others left outside tests.
- normalizes reported MAC addresses in Instance.Apply to lowercase unseparated hex,
the form MAC posture check values are persisted in
- stores a new PostureResponse_Macs rather than mutating the incoming protobuf message
- compares the normalized addresses when deciding whether posture data changed, so a
re-reported separated address no longer reads as an update
- moves the normalizer to a new common/posture package, shared by controller/db and
router/posture, and hoists its regexp out of the per-call path
- adds an oidc-auth-test fablab model that exercises OIDC authentication end to end, with an event-forwarder component, an oidc-test-client, a ziti-prox-c component, and OIDC event/gossip/traffic validations
- moves zitirest out of the public API into separate test-only shims: an integration-test shim (tests/restclient.go) and a fablab shim (zititest/zitirest), and repoints upgrade-test at the fablab shim
- adds region-isolation network partitioning and packet capture on all interfaces to the chaos toolkit
- queues oidc-test-client results reporting on a single sender goroutine with a bounded write deadline, so a stalled results circuit can't block the traffic loops
- counts results events dropped when that queue fills and reports them to the collector as errors, so a reporting outage fails validation instead of looking like a clean run
- bounds the event forwarder's event and keepalive writes, and checks for shutdown on every send attempt, so an unreadable destination can't park the forward loop holding its lock
- logs read errors on collector connections instead of ending collection for that client silently
- bounds the oidc-auth-test debug server's header and body reads
- matches ziti-prox-c process matching to the version-agnostic binary name, so a version change still finds the running process
- points upgrade-test's per-service terminator count at the shared validations helper
- adds a fablab design summary doc
- withholds an enrollment's token and jwt from a non-admin caller when the owning
identity is an admin, in the identity detail mapper and the enrollment mapper
- scopes the identity enrollments subresource so a non-admin does not see
enrollments of admin identities, matching the top level /enrollments list
- threads the request context into MapIdentityToRestModel and
MapEnrollmentToRestModel so the mappers can see the caller's permissions
- adds an integration test over the identity detail, identity list, and identity
enrollments routes for every enrollment method
- adds the advisory to the 2.1.0 changelog
- removes a stray assignment that overwrote isCertExtendable with a literal true on every OIDC certificate authentication
- reports isCertExtendable as false over OIDC for cert authenticators not issued by the network, matching legacy cert auth
- adds coverage asserting the z_ice claim and current-api-session both report a 3rd party CA certificate as not extendable
* fixesopenziti/ziti#4094 accept first-party certs issued by a separate edge signing CA
- adds FirstPartyX509CertValidation and ThirdPartyX509CertValidation usages and an
intermediates field to the router data model public keys, deprecating
ClientX509CertValidation
- publishes config CA bundle roots as first-party anchors with their intermediates and
Ca store entries as third-party anchors; controller certs carry JWT validation only,
since a controller identity is never a CA and anchors no client cert chains
- builds router first-party and client cert trust pools from the published usages,
falling back to the deprecated usage against older controllers
- propagates the full signing cert chain between controllers via a new mesh
SigningCertChainHeader and persists whole chains in controller records
- removes the orphaned InstantStrategy.AddPublicKey, dead since public key sync moved
to controller list data
- gives each command dispatcher its own decoder registry so multiple in-process
controllers no longer decode into the last-started controller's managers
- adds a three-controller in-process HA test harness with a split signing PKI,
cluster formation and first-party cert integration tests
- trusts the edge signing CA when verifying router control channel certs
- adds a variadic additionalRoots parameter to VerifyLeafCertChain, applied to a
clone of the caller's pool so an identity's live tls.Configs are unaffected
- passes the edge enrollment signing CA bundle as additional roots when admitting
a router control channel connection, so a deployment whose signing CA sits
outside the controller's own trust bundle no longer has every router refused
- leaves the fingerprint check bound to the verified leaf, so the wider anchor set
changes which chains verify, not which routers are admitted
- covers the split-root case and the caller-pool guarantee in common/cert tests
- adds a package-level buildFlags string the linker sets at build time, parsed
into a capped list of [A-Z0-9_] names with blanks, duplicates, and malformed
tokens dropped
- returns those names in the new buildFlags field on /version, separate from
capabilities, and prints them under ziti version -v
- bumps edge-api to v0.36.0 for the buildFlags field
- adds a fablab model (zititest/models/upgrade-test) that stands up a multi-region HA controller cluster, routers, and loop4 sim traffic clients (SDK, ERT, ZET, tunneler) and drives repeatable in-place upgrade/downgrade iterations, re-bootstrapping in place each iteration
- exercises HA cluster disruption and recovery (snapshot restore, node rejoin) between iterations and validates steady-state traffic and expected terminators after each disruption
- honors the raft restartSelf setting on migration-snapshot restore, so a restored controller restarts itself instead of exiting, and promotes RestartController to a package-level function
- re-sends loop4 run-scenario requests to sims that reconnect mid-scenario, tracking each scenario's expected client set explicitly and ignoring results from clients outside it, so post-disruption client reconnection no longer stalls validation
- closes leaked loop4 sim-control connections on error and on supersession
- rotates component logs and pins the log-pipe binary to the local build, and drops router debug logging, to keep long iterating runs disk-bounded
- excludes the doc/unsettled/ scratch area for in-progress design docs from version control
- requires an API session token on CreateCircuitV3 requests and validates it,
covering signature, audience, token type, and revocation by token id,
identity, and api session
- takes the dialing identity from the validated token claims rather than the
router-supplied identity id, and rejects a request whose asserted identity
does not match the token subject
- adds the api session id to the log context, matching the V1 and V2 paths
- adds tests for a missing token, an invalid token, and a token belonging to a
different identity than the one asserted
- notes the advisory in the 2.1.0 release notes
- enforces issuer and audience claims inside TokenIssuerExtJwt.VerifyToken, which
previously verified only the token signature and resolved the signing key by kid
- closes the ziti-token-issuer-id header enrollment path accepting a validly-signed
token minted for a different audience or issuer that shares signing keys
- mirrors the runtime ext-jwt authentication and by-inspection enrollment paths so
all paths validate claims consistently
- adds a controller/model unit test covering foreign-audience, foreign-issuer, and
missing-audience rejection
- adds HardenedJwksResolver, a jwks.Resolver that fetches an external jwt signer's
jwksEndpoint with an http/https-only scheme check, a total timeout and a redirect cap
- adds JwksFetchPolicy, gating a fetch on both the URL hostname and the address being
connected to, applied to the first request and to every redirect hop
- hostname gate: deniedHostnames blocks, allowedHostnames is exclusive when set; entries
are an exact hostname or a '*.suffix' wildcard that matches subdomains at any depth but
never the suffix itself, normalized to lower case punycode without a trailing dot
- address gate: built-in blocked (metadata, link-local, link-local multicast,
unspecified), then deniedIPs, then allowedIPs, then blockPrivateAddresses,
first-match-wins with deny over allow
- keeps the gates independent, so neither can authorize what the other refuses; the
address check runs in the dialer against the resolved address, so a hostname that
resolves to a blocked address is refused
- adds the [edge.externalJwtSigners.jwksFetch] config section with compatible defaults:
empty hostname lists, blockPrivateAddresses false, timeout 5s, maxRedirects 5
- takes IP lists as a flat address or a CIDR block, hostname lists as names only, and
rejects an entry belonging to the other list at startup
- rejects a jwksEndpoint the policy refuses when an external jwt signer is created or
updated, and logs an existing signer whose endpoint the configuration now refuses
when the token issuer cache loads
- documents both gates, their deny-wins precedence, the accepted entry forms and the
wildcard matching rules in etc/ctrl.with.edge.yml and CHANGELOG.md
- Creates a router sender on demand when an untracked router subscribes to the
data model, so non-edge (transit) routers receive their controller-managed
configuration without going through the edge connect/sync flow
- Marks on-demand senders as supporting the router data model so they receive
live model updates, not just the initial sync
- Restricts the legacy api-session/session broadcasts to edge routers, since
routers register those handlers universally and would otherwise load session
state they never use. A router counts as edge when an edge router record
exists for its id, which is how both the connect path and the subscribe path
classify it; senders carry the result as an atomic edge flag and the fanout
walks them via RangeEdge. Classification is fixed for the life of a
connection, so a record added or removed under a live connection takes effect
on reconnect. Transit routers also never receive the initial session state,
which only follows a hello
- Adds a race-safe routerTxMap.GetOrCreate keyed on the control channel; the
subscribe path always routes through it, replacing a sender bound to a stale
channel instead of enqueueing on a dead one, and resolves edge-router
membership up front so a replaced edge sender is not demoted to transit
- Adopts an existing sender for the connecting channel in RouterConnected instead
of rejecting it as a duplicate connect; a subscribe that creates the sender first
would otherwise leave the router with no server hello, so it never sends a client
hello and never synchronizes while still being reported online
- Removes routerTxMap.Add, leaving GetOrCreate as the only way a sender is installed
- Classifies a connecting router as edge by id rather than by fingerprint, matching
the disconnect and subscribe paths, and logs an unexpected store error instead of
discarding it
- Removes the unused RouterSender.EdgeRouter field and drops the edge router
argument from the RouterConnected handler
- Generalizes router connect/disconnect logging that previously assumed edge routers
- Updates the strategy flow comment and RouterSender godoc, which described senders
as edge-only and referenced the removed EdgeRouter field
- Documents that api-session-added events and the legacy session model (API and
service sessions) are deprecated for removal in OpenZiti 3.0
- Adds unit tests for GetOrCreate (create/reuse/stale-replace/concurrent), the
edge-only fanout filter, and RouterConnected adopting a subscribe-created sender
routerConnectChurnLimit predates the per-router connect lock. It was added
alongside the ability for a new control channel to take over from an established
one, as the guard on how often that may happen, and it was also the only thing
keeping two connections for one router out of the connected map.
That second job is gone: at most one connection per router is now enforced under
the per-router lock, where the decision is atomic. The check in the accept path
runs against the connected map with no lock held, so it can only refuse a
connection early that would be refused there anyway.
Its first job remains, and is now the only thing doing it. ConnectRouter always
displaces an occupant it does not recognise, so without the limit a spurious
first-connection hello would tear down a healthy control channel and make the
router redial. Nothing said so, and the field carried no godoc at all.
- documents on the option what it protects, that it is churn policy rather than
the uniqueness guarantee, and that zero always allows takeover
- extracts the decision so it can be tested without standing up a network, and
tests it: protected when just established, protected part way through the
window, displaceable once past it, and never protected at zero
The struct's field alignment shifts because a comment ends gofmt's alignment
group; that part of the diff is whitespace only.
Behaviour is unchanged. Worth noting for readers of the option: past the window,
the established connection is now displaced and the connect refused, so the
router redials into the freed slot, where previously the arriving connection took
over directly. Same end state, one extra round trip, and nothing unvetted is
registered on the way.
The controller decided which of two racing connections for a router was current
by comparing router instances, but loaded one per connect by evicting the router
cache and reading back through it. Two connects could both evict, and whichever
read second was handed the instance the first had just published. A shared
instance makes the two connections indistinguishable: the connect path cannot
reject the second into an occupied slot, and when either channel dies the
disconnect path finds itself current and tears down the registration the other is
still using. The surviving channel is never re-bound, so the router stays
connected at the transport layer while absent from the model, unable to recover.
Connect and disconnect were also unserialized, so a stale or superseded
disconnect could interleave with a live connection and take its links with it.
- serializes a router's connect and disconnect with a per-router striped lock
- keeps at most one connection per router: a connect into an occupied slot is
rejected via an error from ConnectRouter, so the bind fails and NewChannel
closes it without starting rx or registering it, and the occupant is displaced;
the router redials into the freed slot
- displaces an occupant by closing its channel and also invoking the teardown
directly, since a channel that is already closed never fires its close handler
again; without this a dead but still registered connection holds the slot
forever and every redial is rejected against a slot nothing can free
- refuses a connect whose control channel is already closed rather than
registering it, so a connection no disconnect could ever remove is never
published
- gives every connection its own router instance via RouterManager.NewCtrlChanRouter,
read through readUncached so the cache neither supplies nor receives it, which
is what makes comparing instances meaningful
- moves recording the channel and connect time out of the accept path, so a
caller cannot attach the wrong channel or forget to attach one
- serializes link publication with that teardown on the same per-router stripe.
Validating currency and then publishing without it is a check-then-act: a
report can find the connection current and, by the time it reaches the link
manager, the teardown has already snapshotted and cleared the router's links,
so the link is recreated after everything that would have removed it. It is
then absent from the router's own index while still in the link table with a
disconnected source, and a reconnect reporting the same iteration can adopt
that stale source instead of rebuilding the link
- guards the entire DisconnectRouter teardown by connection currency, all or
nothing, and clears the connected flag and link index only when the
registration was actually given up, with the flag cleared under the same shard
lock as the map removal so the two cannot be observed disagreeing; the connected flag decides whether the
controller accepts a router's link reports, so clearing it for the wrong
connection silences a router that is up and reporting
- reduces MarkConnected to publishing the connection; the takeover-close moves
into ConnectRouter's reject path
- makes the per-router unlock idempotent so callers can defer it as a leak-safety
net and still unlock early before closing a channel outside the lock
- stops the replaced RouterSender in routerTxMap.Add so a takeover does not leak
the old sender's goroutine when the broker's asynchronous RouterDisconnected
loses the race to the redial's RouterConnected
- discards pending peer state changes for a router whose channel has closed,
since sending on one fails immediately and the failed send is retried as soon
as the event loop turns, spinning the loop and flooding the log
- queues the peer-state send-done event on every path, so a missing channel can
no longer leave sendInProgress set and stall that router's updates permanently
- resolves a router's version from its connected instance when validating link
conn info, since the version arrives in the hello and so is absent from an
instance loaded from the database
- normalizes both endpoints to the connected instance in shortestPath, which is
keyed and compared by pointer and so treated an endpoint held as any other
instance of the same router as absent from the graph, reporting a router as
unroutable from itself. That worked before only because the connect path
published its instance into the router cache, so a cache read and the connected
map returned the same object; nothing stated the requirement
- configures test logging once per package in TestMain, so a test no longer
writes global logger state while a previous test's shutdown logging reads it
- swaps testing.TB for require.TestingT in model.NewTestContext, so this
build-included file no longer imports testing
- documents why the narrower interface is used, so it isn't reverted
- leaves hashicorp/raft as the only remaining importer of testing in the
ziti binary's dependency graph
- swaps testing.TB for require.TestingT in db.NewTestContext, so this
build-included file no longer imports testing
- documents why the narrower interface is used, so it isn't reverted
- swaps testing.TB for require.TestingT in NewTestContext and NextTest,
so this build-included file no longer imports testing
- removes the write-only t field from BaseTestContext
- documents why the narrower interface is used, so it isn't reverted
- adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop
- translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup
- adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path
- re-evaluates dialers on link group and listener changes via RescanForDialOpportunities
For #3743.
- passes appEnv into the legacy v1 handler's request context, so token
validation no longer makes a nil-interface call and panics the controller
- adds a regression test driving the v1 handler over the control channel with
JWT-prefixed, opaque, and empty tokens, asserting an error reply comes back
and the controller keeps serving
- rejects a remove or update request whose terminator is owned by a different
router, on the fabric control channel handlers
- drops ids the requesting router does not own from batch removals, keeping
absent ids so a delete racing a not-yet-applied create is still ordered after
it
- adds unit tests for the ownership filter and for the single-terminator check
- adds an end-to-end test that drives the fabric control channel from an
enrolled router against a second router's terminator, covering single remove,
batch remove, and re-weight, plus a control that a router can still remove its
own
* fixesopenziti/ziti#4118 disambiguate overlapping ext-jwt-signer kids by issuer
- binds external JWT tokens to signers by exact issuer claim rather than by key ID, so signers drawing from a shared signing-key pool resolve deterministically
- binds controller-issued tokens by key ID first, preserving controller token resolution and preventing an external signer configured with a controller's issuer from capturing controller access tokens
- removes the external key-ID fallback so a token whose issuer matches no configured signer is not bound to an unrelated signer that happens to share its kid
- adds GetControllerIssuerByKid to the TokenIssuerCache interface and implementation
- skips disabled external signers in GetIssuerByKid so a disabled signer sharing a kid cannot poison resolution for an enabled one
- adds an integration test with two HTTPS JWKS providers sharing a key and kid, covering the enabled-collision and disabled-poison cases
- documents that controller issuers are keyed by controller id and that a controller issuer's key ID is the fingerprint of its TLS certificate
- documents that an external kid match is ambiguous because signers can share a signing-key pool, and that a definitive binding requires resolving by issuer claim
- clarifies the overlapping-kid test comment covering why issuer-claim binding is required when a disabled signer shares a kid
Paul Lorenz