- resolves zt-session tokens eagerly when the request context is created, so every request carrying one marks activity and gets session lifetime headers, regardless of endpoint
- marks last activity in SecurityCtx.resolveZtSession once the session is loaded
- adds SecurityTokenCtx.HasZtSessionHeader, a header-only check that does not parse bearer tokens
- emits only session lifetime headers from the API wrappers and drops the unused session-error header branch, so a stale token on an anonymous endpoint stays silent as it did on 1.6.x
- tests that requests to authenticated and anonymous endpoints with a zt-session mark activity, carry lifetime headers, and are reflected by current-api-session
- normalizes reported MAC addresses in Instance.Apply to lowercase unseparated hex,
the form MAC posture check values are persisted in
- stores a new PostureResponse_Macs rather than mutating the incoming protobuf message
- compares the normalized addresses when deciding whether posture data changed, so a
re-reported separated address no longer reads as an update
- moves the normalizer to a new common/posture package, shared by controller/db and
router/posture, and hoists its regexp out of the per-call path
- adds an oidc-auth-test fablab model that exercises OIDC authentication end to end, with an event-forwarder component, an oidc-test-client, a ziti-prox-c component, and OIDC event/gossip/traffic validations
- moves zitirest out of the public API into separate test-only shims: an integration-test shim (tests/restclient.go) and a fablab shim (zititest/zitirest), and repoints upgrade-test at the fablab shim
- adds region-isolation network partitioning and packet capture on all interfaces to the chaos toolkit
- queues oidc-test-client results reporting on a single sender goroutine with a bounded write deadline, so a stalled results circuit can't block the traffic loops
- counts results events dropped when that queue fills and reports them to the collector as errors, so a reporting outage fails validation instead of looking like a clean run
- bounds the event forwarder's event and keepalive writes, and checks for shutdown on every send attempt, so an unreadable destination can't park the forward loop holding its lock
- logs read errors on collector connections instead of ending collection for that client silently
- bounds the oidc-auth-test debug server's header and body reads
- matches ziti-prox-c process matching to the version-agnostic binary name, so a version change still finds the running process
- points upgrade-test's per-service terminator count at the shared validations helper
- adds a fablab design summary doc
- withholds an enrollment's token and jwt from a non-admin caller when the owning
identity is an admin, in the identity detail mapper and the enrollment mapper
- scopes the identity enrollments subresource so a non-admin does not see
enrollments of admin identities, matching the top level /enrollments list
- threads the request context into MapIdentityToRestModel and
MapEnrollmentToRestModel so the mappers can see the caller's permissions
- adds an integration test over the identity detail, identity list, and identity
enrollments routes for every enrollment method
- adds the advisory to the 2.1.0 changelog
- removes a stray assignment that overwrote isCertExtendable with a literal true on every OIDC certificate authentication
- reports isCertExtendable as false over OIDC for cert authenticators not issued by the network, matching legacy cert auth
- adds coverage asserting the z_ice claim and current-api-session both report a 3rd party CA certificate as not extendable
* fixesopenziti/ziti#4094 accept first-party certs issued by a separate edge signing CA
- adds FirstPartyX509CertValidation and ThirdPartyX509CertValidation usages and an
intermediates field to the router data model public keys, deprecating
ClientX509CertValidation
- publishes config CA bundle roots as first-party anchors with their intermediates and
Ca store entries as third-party anchors; controller certs carry JWT validation only,
since a controller identity is never a CA and anchors no client cert chains
- builds router first-party and client cert trust pools from the published usages,
falling back to the deprecated usage against older controllers
- propagates the full signing cert chain between controllers via a new mesh
SigningCertChainHeader and persists whole chains in controller records
- removes the orphaned InstantStrategy.AddPublicKey, dead since public key sync moved
to controller list data
- gives each command dispatcher its own decoder registry so multiple in-process
controllers no longer decode into the last-started controller's managers
- adds a three-controller in-process HA test harness with a split signing PKI,
cluster formation and first-party cert integration tests
- trusts the edge signing CA when verifying router control channel certs
- adds a variadic additionalRoots parameter to VerifyLeafCertChain, applied to a
clone of the caller's pool so an identity's live tls.Configs are unaffected
- passes the edge enrollment signing CA bundle as additional roots when admitting
a router control channel connection, so a deployment whose signing CA sits
outside the controller's own trust bundle no longer has every router refused
- leaves the fingerprint check bound to the verified leaf, so the wider anchor set
changes which chains verify, not which routers are admitted
- covers the split-root case and the caller-pool guarantee in common/cert tests
- adds a package-level buildFlags string the linker sets at build time, parsed
into a capped list of [A-Z0-9_] names with blanks, duplicates, and malformed
tokens dropped
- returns those names in the new buildFlags field on /version, separate from
capabilities, and prints them under ziti version -v
- bumps edge-api to v0.36.0 for the buildFlags field
- adds a fablab model (zititest/models/upgrade-test) that stands up a multi-region HA controller cluster, routers, and loop4 sim traffic clients (SDK, ERT, ZET, tunneler) and drives repeatable in-place upgrade/downgrade iterations, re-bootstrapping in place each iteration
- exercises HA cluster disruption and recovery (snapshot restore, node rejoin) between iterations and validates steady-state traffic and expected terminators after each disruption
- honors the raft restartSelf setting on migration-snapshot restore, so a restored controller restarts itself instead of exiting, and promotes RestartController to a package-level function
- re-sends loop4 run-scenario requests to sims that reconnect mid-scenario, tracking each scenario's expected client set explicitly and ignoring results from clients outside it, so post-disruption client reconnection no longer stalls validation
- closes leaked loop4 sim-control connections on error and on supersession
- rotates component logs and pins the log-pipe binary to the local build, and drops router debug logging, to keep long iterating runs disk-bounded
- excludes the doc/unsettled/ scratch area for in-progress design docs from version control
- requires an API session token on CreateCircuitV3 requests and validates it,
covering signature, audience, token type, and revocation by token id,
identity, and api session
- takes the dialing identity from the validated token claims rather than the
router-supplied identity id, and rejects a request whose asserted identity
does not match the token subject
- adds the api session id to the log context, matching the V1 and V2 paths
- adds tests for a missing token, an invalid token, and a token belonging to a
different identity than the one asserted
- notes the advisory in the 2.1.0 release notes
- enforces issuer and audience claims inside TokenIssuerExtJwt.VerifyToken, which
previously verified only the token signature and resolved the signing key by kid
- closes the ziti-token-issuer-id header enrollment path accepting a validly-signed
token minted for a different audience or issuer that shares signing keys
- mirrors the runtime ext-jwt authentication and by-inspection enrollment paths so
all paths validate claims consistently
- adds a controller/model unit test covering foreign-audience, foreign-issuer, and
missing-audience rejection
- adds HardenedJwksResolver, a jwks.Resolver that fetches an external jwt signer's
jwksEndpoint with an http/https-only scheme check, a total timeout and a redirect cap
- adds JwksFetchPolicy, gating a fetch on both the URL hostname and the address being
connected to, applied to the first request and to every redirect hop
- hostname gate: deniedHostnames blocks, allowedHostnames is exclusive when set; entries
are an exact hostname or a '*.suffix' wildcard that matches subdomains at any depth but
never the suffix itself, normalized to lower case punycode without a trailing dot
- address gate: built-in blocked (metadata, link-local, link-local multicast,
unspecified), then deniedIPs, then allowedIPs, then blockPrivateAddresses,
first-match-wins with deny over allow
- keeps the gates independent, so neither can authorize what the other refuses; the
address check runs in the dialer against the resolved address, so a hostname that
resolves to a blocked address is refused
- adds the [edge.externalJwtSigners.jwksFetch] config section with compatible defaults:
empty hostname lists, blockPrivateAddresses false, timeout 5s, maxRedirects 5
- takes IP lists as a flat address or a CIDR block, hostname lists as names only, and
rejects an entry belonging to the other list at startup
- rejects a jwksEndpoint the policy refuses when an external jwt signer is created or
updated, and logs an existing signer whose endpoint the configuration now refuses
when the token issuer cache loads
- documents both gates, their deny-wins precedence, the accepted entry forms and the
wildcard matching rules in etc/ctrl.with.edge.yml and CHANGELOG.md
- Creates a router sender on demand when an untracked router subscribes to the
data model, so non-edge (transit) routers receive their controller-managed
configuration without going through the edge connect/sync flow
- Marks on-demand senders as supporting the router data model so they receive
live model updates, not just the initial sync
- Restricts the legacy api-session/session broadcasts to edge routers, since
routers register those handlers universally and would otherwise load session
state they never use. A router counts as edge when an edge router record
exists for its id, which is how both the connect path and the subscribe path
classify it; senders carry the result as an atomic edge flag and the fanout
walks them via RangeEdge. Classification is fixed for the life of a
connection, so a record added or removed under a live connection takes effect
on reconnect. Transit routers also never receive the initial session state,
which only follows a hello
- Adds a race-safe routerTxMap.GetOrCreate keyed on the control channel; the
subscribe path always routes through it, replacing a sender bound to a stale
channel instead of enqueueing on a dead one, and resolves edge-router
membership up front so a replaced edge sender is not demoted to transit
- Adopts an existing sender for the connecting channel in RouterConnected instead
of rejecting it as a duplicate connect; a subscribe that creates the sender first
would otherwise leave the router with no server hello, so it never sends a client
hello and never synchronizes while still being reported online
- Removes routerTxMap.Add, leaving GetOrCreate as the only way a sender is installed
- Classifies a connecting router as edge by id rather than by fingerprint, matching
the disconnect and subscribe paths, and logs an unexpected store error instead of
discarding it
- Removes the unused RouterSender.EdgeRouter field and drops the edge router
argument from the RouterConnected handler
- Generalizes router connect/disconnect logging that previously assumed edge routers
- Updates the strategy flow comment and RouterSender godoc, which described senders
as edge-only and referenced the removed EdgeRouter field
- Documents that api-session-added events and the legacy session model (API and
service sessions) are deprecated for removal in OpenZiti 3.0
- Adds unit tests for GetOrCreate (create/reuse/stale-replace/concurrent), the
edge-only fanout filter, and RouterConnected adopting a subscribe-created sender
routerConnectChurnLimit predates the per-router connect lock. It was added
alongside the ability for a new control channel to take over from an established
one, as the guard on how often that may happen, and it was also the only thing
keeping two connections for one router out of the connected map.
That second job is gone: at most one connection per router is now enforced under
the per-router lock, where the decision is atomic. The check in the accept path
runs against the connected map with no lock held, so it can only refuse a
connection early that would be refused there anyway.
Its first job remains, and is now the only thing doing it. ConnectRouter always
displaces an occupant it does not recognise, so without the limit a spurious
first-connection hello would tear down a healthy control channel and make the
router redial. Nothing said so, and the field carried no godoc at all.
- documents on the option what it protects, that it is churn policy rather than
the uniqueness guarantee, and that zero always allows takeover
- extracts the decision so it can be tested without standing up a network, and
tests it: protected when just established, protected part way through the
window, displaceable once past it, and never protected at zero
The struct's field alignment shifts because a comment ends gofmt's alignment
group; that part of the diff is whitespace only.
Behaviour is unchanged. Worth noting for readers of the option: past the window,
the established connection is now displaced and the connect refused, so the
router redials into the freed slot, where previously the arriving connection took
over directly. Same end state, one extra round trip, and nothing unvetted is
registered on the way.
The controller decided which of two racing connections for a router was current
by comparing router instances, but loaded one per connect by evicting the router
cache and reading back through it. Two connects could both evict, and whichever
read second was handed the instance the first had just published. A shared
instance makes the two connections indistinguishable: the connect path cannot
reject the second into an occupied slot, and when either channel dies the
disconnect path finds itself current and tears down the registration the other is
still using. The surviving channel is never re-bound, so the router stays
connected at the transport layer while absent from the model, unable to recover.
Connect and disconnect were also unserialized, so a stale or superseded
disconnect could interleave with a live connection and take its links with it.
- serializes a router's connect and disconnect with a per-router striped lock
- keeps at most one connection per router: a connect into an occupied slot is
rejected via an error from ConnectRouter, so the bind fails and NewChannel
closes it without starting rx or registering it, and the occupant is displaced;
the router redials into the freed slot
- displaces an occupant by closing its channel and also invoking the teardown
directly, since a channel that is already closed never fires its close handler
again; without this a dead but still registered connection holds the slot
forever and every redial is rejected against a slot nothing can free
- refuses a connect whose control channel is already closed rather than
registering it, so a connection no disconnect could ever remove is never
published
- gives every connection its own router instance via RouterManager.NewCtrlChanRouter,
read through readUncached so the cache neither supplies nor receives it, which
is what makes comparing instances meaningful
- moves recording the channel and connect time out of the accept path, so a
caller cannot attach the wrong channel or forget to attach one
- serializes link publication with that teardown on the same per-router stripe.
Validating currency and then publishing without it is a check-then-act: a
report can find the connection current and, by the time it reaches the link
manager, the teardown has already snapshotted and cleared the router's links,
so the link is recreated after everything that would have removed it. It is
then absent from the router's own index while still in the link table with a
disconnected source, and a reconnect reporting the same iteration can adopt
that stale source instead of rebuilding the link
- guards the entire DisconnectRouter teardown by connection currency, all or
nothing, and clears the connected flag and link index only when the
registration was actually given up, with the flag cleared under the same shard
lock as the map removal so the two cannot be observed disagreeing; the connected flag decides whether the
controller accepts a router's link reports, so clearing it for the wrong
connection silences a router that is up and reporting
- reduces MarkConnected to publishing the connection; the takeover-close moves
into ConnectRouter's reject path
- makes the per-router unlock idempotent so callers can defer it as a leak-safety
net and still unlock early before closing a channel outside the lock
- stops the replaced RouterSender in routerTxMap.Add so a takeover does not leak
the old sender's goroutine when the broker's asynchronous RouterDisconnected
loses the race to the redial's RouterConnected
- discards pending peer state changes for a router whose channel has closed,
since sending on one fails immediately and the failed send is retried as soon
as the event loop turns, spinning the loop and flooding the log
- queues the peer-state send-done event on every path, so a missing channel can
no longer leave sendInProgress set and stall that router's updates permanently
- resolves a router's version from its connected instance when validating link
conn info, since the version arrives in the hello and so is absent from an
instance loaded from the database
- normalizes both endpoints to the connected instance in shortestPath, which is
keyed and compared by pointer and so treated an endpoint held as any other
instance of the same router as absent from the graph, reporting a router as
unroutable from itself. That worked before only because the connect path
published its instance into the router cache, so a cache read and the connected
map returned the same object; nothing stated the requirement
- configures test logging once per package in TestMain, so a test no longer
writes global logger state while a previous test's shutdown logging reads it
- swaps testing.TB for require.TestingT in model.NewTestContext, so this
build-included file no longer imports testing
- documents why the narrower interface is used, so it isn't reverted
- leaves hashicorp/raft as the only remaining importer of testing in the
ziti binary's dependency graph
- swaps testing.TB for require.TestingT in db.NewTestContext, so this
build-included file no longer imports testing
- documents why the narrower interface is used, so it isn't reverted
- swaps testing.TB for require.TestingT in NewTestContext and NextTest,
so this build-included file no longer imports testing
- removes the write-only t field from BaseTestContext
- documents why the narrower interface is used, so it isn't reverted
- adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop
- translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup
- adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path
- re-evaluates dialers on link group and listener changes via RescanForDialOpportunities
For #3743.
- passes appEnv into the legacy v1 handler's request context, so token
validation no longer makes a nil-interface call and panics the controller
- adds a regression test driving the v1 handler over the control channel with
JWT-prefixed, opaque, and empty tokens, asserting an error reply comes back
and the controller keeps serving
- rejects a remove or update request whose terminator is owned by a different
router, on the fabric control channel handlers
- drops ids the requesting router does not own from batch removals, keeping
absent ids so a delete racing a not-yet-applied create is still ordered after
it
- adds unit tests for the ownership filter and for the single-terminator check
- adds an end-to-end test that drives the fabric control channel from an
enrolled router against a second router's terminator, covering single remove,
batch remove, and re-weight, plus a control that a router can still remove its
own
* fixesopenziti/ziti#4118 disambiguate overlapping ext-jwt-signer kids by issuer
- binds external JWT tokens to signers by exact issuer claim rather than by key ID, so signers drawing from a shared signing-key pool resolve deterministically
- binds controller-issued tokens by key ID first, preserving controller token resolution and preventing an external signer configured with a controller's issuer from capturing controller access tokens
- removes the external key-ID fallback so a token whose issuer matches no configured signer is not bound to an unrelated signer that happens to share its kid
- adds GetControllerIssuerByKid to the TokenIssuerCache interface and implementation
- skips disabled external signers in GetIssuerByKid so a disabled signer sharing a kid cannot poison resolution for an enabled one
- adds an integration test with two HTTPS JWKS providers sharing a key and kid, covering the enabled-collision and disabled-poison cases
- documents that controller issuers are keyed by controller id and that a controller issuer's key ID is the fingerprint of its TLS certificate
- documents that an external kid match is ambiguous because signers can share a signing-key pool, and that a definitive binding requires resolving by issuer claim
- clarifies the overlapping-kid test comment covering why issuer-claim binding is required when a disabled signer shares a kid
* fixesopenziti/ziti#4063 enroll with empty roles when ext-jwt attribute claim is absent
- treats an absent enrollment attribute claim as no role attributes rather than
rejecting the token; a failed jsonPointer.Get means the claim is not present,
since pointer syntax is already validated at jsonpointer.New
- matches the existing unset-selector and empty-claim cases, which already
enroll with an empty attribute set; a present-but-wrong-type claim still errors
- adds unit coverage for resolveStringSliceClaimProperty
- moves the two ext-jwt enrollment integration subtests that asserted the old
failure behavior to success cases asserting an empty role-attribute set
* fixesopenziti/ziti#4063 treat a null attribute claim as no role attributes
- treats an attribute claim sent as JSON null as no role attributes, logging a
warning, matching the behavior of an absent claim
- logs at debug when the attribute claim selector does not resolve
- prints the offending value in the wrong-type error rather than the always-nil
result of the failed array assertion
- corrects the godoc and inline comment: a pointer can fail to resolve through
traversal as well as absence, and the malformed-pointer error is unreachable
- adds unit coverage for null and nested null claims, and asserts the wrong-type
error names the value
- adds enrollment tests for a null attribute claim enrolling with no role
attributes, and for a null name claim rejecting enrollment
- preserves tags when creating external JWT signers, transit routers and authenticators, all of
which previously discarded them while update and patch honored them
- switches the external JWT signer update and patch mappers to TagsOrDefault, matching the
pattern used by the other entity mappers
- adds a unit test for TagsOrDefault covering nil, nil sub-tags, empty and populated inputs
- tests that create persists tags for external JWT signers, transit routers and authenticators
- tests that patching an external JWT signer leaves existing tags untouched, and that patching
tags replaces them
- adds a changelog entry
- removes the costTags option from router link listeners, which was parsed
from config, advertised to the controller, and stored on the router model
but never used for path selection or any other behavior
- drops GetLinkCostTags from the xlink.Listener interface and its transport
implementation
- reserves the corresponding ctrl_pb.Listener.costTags and
RouterLinks.RouterLink.linkCostTags protobuf fields and regenerates ctrl.pb.go
- documents the removal in the changelog
- reports edge router policy denials with an access-denied error naming the missing policy,
replacing the session error reused on the sessionless ER/T and create-circuit-v3 paths
- adds EdgeRouterManager.GetEdgeRouterAccess, which reports which of the two required policy
links (identity-to-edge-router, service-to-edge-router) is absent, and removes the boolean
IsAccessToEdgeRouterAllowed it replaces
- logs the controller's rejection on the router at warn level, since it is recoverable and
retried by the periodic scan; the router previously discarded the error code and message
- delays a new terminator's first create attempt by a fixed 2s so config applied in quick
succession settles before the router asks, avoiding a 2-3 minute wait for the retry scan; the
delay is a deliberate stopgap until edge router policy visibility lands in the router data model
- propagates the controller's error code and retry hint to SDK clients on the dial paths, which
dropped the code and left every refusal classified as unknown
- adds the retry hint header to controller error replies, grouped with the other error-reply
headers rather than the create-circuit-v3 request headers
- notes that the sync strategy headers alias the edge namespace's 1013-1015 ids and stay
disjoint only by message content type
- tests the per-policy denial reporting, the error code carried with and without a retry hint,
the controller-to-SDK error code mapping at both dial relay sites, and the terminator settle
gate
- waits for terminator establishment in the tunneler dataflow tests instead of a fixed sleep, so
they no longer race the settle delay
- restores the tproxy multiple-lanIf and multiple-resolver changelog entries with keep markers,
which regeneration drops because their commits reference pull requests rather than issues
- releases the per-identity lock before acquiring the cmap shard lock when the scan
loop reaps an entry, establishing a single shard-lock-then-value-lock order
- extracts the entry removal into removeIfEmpty, which decides based on the value
currently in the map, so an identity that reconnects after the scan decided to
remove it is left alone
- takes the per-identity lock in the removal check, fixing an unsynchronized read of
the router map
- documents the lock ordering invariant on identityConnections
- adds tests covering concurrent scanning and connect/disconnect handling, entry
reaping, and reconnection between the scan's decision and the removal
- verifies loadFromBolt accepts a legacy opaque service session token for its owning api session
- verifies loadFromBolt rejects an opaque service session token presented under a different api session with an InvalidSessionError
Forward ports the GHSA-cc5m-7mhm-xh9f fix, released in 2.0.2, to main.
- runs router certificate and fingerprint validation for router control-channel
underlay types, which was previously skipped for any connection carrying a
channel type header; only types dispatched to a separate self-validating
acceptor (the raft mesh) are skipped now
- binds the enrolled-fingerprint check to the verified leaf, so a peer cannot
pass by presenting its own leaf followed by a target router's public
certificate
- applies the already-connected / churn guard only when establishing a new
channel, so additional underlays of a grouped control channel are not rejected
while the router is already connected
- extracts the first-underlay header construction so the grouped-connection
scoping is unit-testable
- adds negative-path tests for untrusted and self-signed leaves, and for
separately-validated channel types being skipped
Forward ports the GHSA-mrpr-756c-xm47 fix, released in 2.0.2, to main.
- adds a shared cert.VerifyLeafCertChain helper that verifies the presented leaf
(certs[0], whose private key the TLS handshake proved) against the node's full
trusted-CA pool, treating certs[1:] only as candidate intermediates
- uses it for controller cluster mesh peer connections and router link
connections, which previously accepted a connection when any presented
certificate chained to the trusted CA while taking peer identity from the leaf
- matches the pinned metrics scrape certificate against the presented leaf only,
comparing full DER rather than just the signature, and rejects a leaf outside
its validity window
- adds negative-path tests covering rogue leaves paired with CA-chained filler
certificates
- reuses the body already buffered by CreateRequestContext instead of reading the request body a second time in legacyGenericEnrollPemHandler
- removes the duplicate io.ReadAll allocation for legacy enrollment content types (text/plain, application/pkcs7, application/x-pem-file)
- adds command.WasLeaderless to classify cluster-has-no-leader dispatch errors as retriable
- replies busy instead of dropping or hard-failing terminator creates when the cluster is briefly leaderless, so the router backs off and requeues promptly rather than waiting for its multi-minute recovery scan
- removes the racy up-front leaderless pre-check in the sdk create handler in favor of classifying the actual dispatch result
- applies the same retriable classification to the ert tunnel create and batch remove terminator handlers
Paul Lorenz