- sends no signer fingerprints in the router data model when a process check has none configured,
the single stored fingerprint was wrapped into a one-element list holding an empty string
- treats an empty configured signer fingerprint as no constraint when evaluating a process check,
routers require a reported signer to match any entry in the list
- reports the configured fingerprints in the signer failure message, it reported the hashes
* fixesopenziti/ziti#4094 accept first-party certs issued by a separate edge signing CA
- adds FirstPartyX509CertValidation and ThirdPartyX509CertValidation usages and an
intermediates field to the router data model public keys, deprecating
ClientX509CertValidation
- publishes config CA bundle roots as first-party anchors with their intermediates and
Ca store entries as third-party anchors; controller certs carry JWT validation only,
since a controller identity is never a CA and anchors no client cert chains
- builds router first-party and client cert trust pools from the published usages,
falling back to the deprecated usage against older controllers
- propagates the full signing cert chain between controllers via a new mesh
SigningCertChainHeader and persists whole chains in controller records
- removes the orphaned InstantStrategy.AddPublicKey, dead since public key sync moved
to controller list data
- gives each command dispatcher its own decoder registry so multiple in-process
controllers no longer decode into the last-started controller's managers
- adds a three-controller in-process HA test harness with a split signing PKI,
cluster formation and first-party cert integration tests
- trusts the edge signing CA when verifying router control channel certs
- adds a variadic additionalRoots parameter to VerifyLeafCertChain, applied to a
clone of the caller's pool so an identity's live tls.Configs are unaffected
- passes the edge enrollment signing CA bundle as additional roots when admitting
a router control channel connection, so a deployment whose signing CA sits
outside the controller's own trust bundle no longer has every router refused
- leaves the fingerprint check bound to the verified leaf, so the wider anchor set
changes which chains verify, not which routers are admitted
- covers the split-root case and the caller-pool guarantee in common/cert tests
- Creates a router sender on demand when an untracked router subscribes to the
data model, so non-edge (transit) routers receive their controller-managed
configuration without going through the edge connect/sync flow
- Marks on-demand senders as supporting the router data model so they receive
live model updates, not just the initial sync
- Restricts the legacy api-session/session broadcasts to edge routers, since
routers register those handlers universally and would otherwise load session
state they never use. A router counts as edge when an edge router record
exists for its id, which is how both the connect path and the subscribe path
classify it; senders carry the result as an atomic edge flag and the fanout
walks them via RangeEdge. Classification is fixed for the life of a
connection, so a record added or removed under a live connection takes effect
on reconnect. Transit routers also never receive the initial session state,
which only follows a hello
- Adds a race-safe routerTxMap.GetOrCreate keyed on the control channel; the
subscribe path always routes through it, replacing a sender bound to a stale
channel instead of enqueueing on a dead one, and resolves edge-router
membership up front so a replaced edge sender is not demoted to transit
- Adopts an existing sender for the connecting channel in RouterConnected instead
of rejecting it as a duplicate connect; a subscribe that creates the sender first
would otherwise leave the router with no server hello, so it never sends a client
hello and never synchronizes while still being reported online
- Removes routerTxMap.Add, leaving GetOrCreate as the only way a sender is installed
- Classifies a connecting router as edge by id rather than by fingerprint, matching
the disconnect and subscribe paths, and logs an unexpected store error instead of
discarding it
- Removes the unused RouterSender.EdgeRouter field and drops the edge router
argument from the RouterConnected handler
- Generalizes router connect/disconnect logging that previously assumed edge routers
- Updates the strategy flow comment and RouterSender godoc, which described senders
as edge-only and referenced the removed EdgeRouter field
- Documents that api-session-added events and the legacy session model (API and
service sessions) are deprecated for removal in OpenZiti 3.0
- Adds unit tests for GetOrCreate (create/reuse/stale-replace/concurrent), the
edge-only fanout filter, and RouterConnected adopting a subscribe-created sender
The controller decided which of two racing connections for a router was current
by comparing router instances, but loaded one per connect by evicting the router
cache and reading back through it. Two connects could both evict, and whichever
read second was handed the instance the first had just published. A shared
instance makes the two connections indistinguishable: the connect path cannot
reject the second into an occupied slot, and when either channel dies the
disconnect path finds itself current and tears down the registration the other is
still using. The surviving channel is never re-bound, so the router stays
connected at the transport layer while absent from the model, unable to recover.
Connect and disconnect were also unserialized, so a stale or superseded
disconnect could interleave with a live connection and take its links with it.
- serializes a router's connect and disconnect with a per-router striped lock
- keeps at most one connection per router: a connect into an occupied slot is
rejected via an error from ConnectRouter, so the bind fails and NewChannel
closes it without starting rx or registering it, and the occupant is displaced;
the router redials into the freed slot
- displaces an occupant by closing its channel and also invoking the teardown
directly, since a channel that is already closed never fires its close handler
again; without this a dead but still registered connection holds the slot
forever and every redial is rejected against a slot nothing can free
- refuses a connect whose control channel is already closed rather than
registering it, so a connection no disconnect could ever remove is never
published
- gives every connection its own router instance via RouterManager.NewCtrlChanRouter,
read through readUncached so the cache neither supplies nor receives it, which
is what makes comparing instances meaningful
- moves recording the channel and connect time out of the accept path, so a
caller cannot attach the wrong channel or forget to attach one
- serializes link publication with that teardown on the same per-router stripe.
Validating currency and then publishing without it is a check-then-act: a
report can find the connection current and, by the time it reaches the link
manager, the teardown has already snapshotted and cleared the router's links,
so the link is recreated after everything that would have removed it. It is
then absent from the router's own index while still in the link table with a
disconnected source, and a reconnect reporting the same iteration can adopt
that stale source instead of rebuilding the link
- guards the entire DisconnectRouter teardown by connection currency, all or
nothing, and clears the connected flag and link index only when the
registration was actually given up, with the flag cleared under the same shard
lock as the map removal so the two cannot be observed disagreeing; the connected flag decides whether the
controller accepts a router's link reports, so clearing it for the wrong
connection silences a router that is up and reporting
- reduces MarkConnected to publishing the connection; the takeover-close moves
into ConnectRouter's reject path
- makes the per-router unlock idempotent so callers can defer it as a leak-safety
net and still unlock early before closing a channel outside the lock
- stops the replaced RouterSender in routerTxMap.Add so a takeover does not leak
the old sender's goroutine when the broker's asynchronous RouterDisconnected
loses the race to the redial's RouterConnected
- discards pending peer state changes for a router whose channel has closed,
since sending on one fails immediately and the failed send is retried as soon
as the event loop turns, spinning the loop and flooding the log
- queues the peer-state send-done event on every path, so a missing channel can
no longer leave sendInProgress set and stall that router's updates permanently
- resolves a router's version from its connected instance when validating link
conn info, since the version arrives in the hello and so is absent from an
instance loaded from the database
- normalizes both endpoints to the connected instance in shortestPath, which is
keyed and compared by pointer and so treated an endpoint held as any other
instance of the same router as absent from the graph, reporting a router as
unroutable from itself. That worked before only because the connect path
published its instance into the router cache, so a cache read and the connected
map returned the same object; nothing stated the requirement
- configures test logging once per package in TestMain, so a test no longer
writes global logger state while a previous test's shutdown logging reads it
* fixesopenziti/ziti#3990 push service and posture state to subscribed SDKs
- pushes indexed atomic ServiceChangeSet envelopes to subscribed SDK connections: a full snapshot on subscribe, incremental service changes per RDM scan pass, posture check definition changes as their own entries, and identity-resolved config bodies, all serialized so envelopes hit the wire in index order
- pushes per-connection PostureStateChange state (monotonic seq, resync on request) for posture pass/fail, including flips caused by definition edits that mutate no posture data
- registers pending RDM identity subscriptions for identities not yet synced to the router and sends an authoritative full sync plus full posture state when the identity arrives; an active push subscription pins the connection's RDM listener
- advertises service subscriptions and router data model support on the control-channel capability bitmask; the controller persists each router's capabilities mask and version on the EdgeRouter entity via raft and renders them on the edge APIs, so SDKs can select capable routers before connecting
- submits posture per router and corrects MFA posture semantics: pushed expiry is the earliest of timeout and pending wake/unlock grace deadlines, wake/unlock re-pass satisfies the re-prompt, api session tokens whose amr attests TOTP seed the MFA baseline from auth_time only (never iat), and token exchange carries the subject token's auth_time
- sends structured denials on dial and bind refusals: posture failures carry the failing check ids, no-policy denials are access denied, unknown services are invalid service, and session token failures are invalid session; the denial's cause no longer rides the wire as an unserializable error
- hard-closes accepted SDK connections on edge listener shutdown so clients observe a router going away immediately
- adds integration coverage: subscription snapshots and change delivery, poll and push reconciliation as capable routers come and go, posture state and definition-change push, router views over the public SDK API, typed dial errors, MFA baseline seeding, and OIDC token-exchange auth_time preservation
- removed RDM capability from SDK, router/controller only
- channel.TypedReceiveHandler -> channel.ContentTypeReceiver
- binding.AddTypedReceiveHandler(h) -> channel.AddReceiveHandlers(binding, h)
channel/v5 repurposes TypedReceiveHandler for the senders-typed handler and replaces the
self-describing pattern with ContentTypeReceiver plus the AddReceiveHandlers free function
(openziti/channel#262). Mechanical conversion; does not build on its own.
- moves the channel dependency to channel/v5 v5.0.10 and sdk-golang to v1.9.0 in the root and zititest modules
- mechanically rewrites every channel/v4 import path to channel/v5
This is the import-path-only step; the API-level changes the switch requires land in the following commit. This commit does not build on its own.
- adds a `DataState.Router` event variant (id, name, fingerprint,
configs) to the RDM protobuf and the matching ConfigType.Target field
flows
- loads routers into `RouterDataModelSender` at startup and registers
entity-change listeners so router create/update/delete and config
reassignment emit RDM events
- filters `Config` events per-router at `RouterSender`: each router
sees the full `Router` set but only its own router-target Configs
- receiver-side: parses and stores `Router` entities; GCs router-target
Configs locally when they drop off the router's `Configs` list, so
no synthetic remove events are needed on the wire
- extends the `router-data-model-test` fablab model with router-config
distribution scenarios (assignment, reassignment, controller
restart, RDM cache miss)
- updates `validate-router-data-model` to recognize the new event
shape
- refreshes the design doc to describe per-router filtering and the
change-notification flow
- collapses the edge service boltz child store into the unified service store:
EdgeService becomes a type alias for Service, and RoleAttributes, Configs, and
EncryptionRequired become top-level Service fields
- adds an IsFabricOnly discriminator separating pure-fabric ("management")
services from edge services
- adds a forward-only, irreversible migration that moves edge service data out of
the edge child bucket, classifies pre-existing services (fabric -> IsFabricOnly
with encryptionRequired=false; edge -> IsFabricOnly=false), deletes the old edge
bucket and its index, and rebuilds the role-attributes index via CheckIntegrity
- guards every edge-facing path on the unified store against fabric-only services
so they stay invisible to the edge API: by-id Read/ReadByName/Delete, list/query
and the detail lister, ReadForIdentity, identity service-config overrides,
service policy / SERP @id and #all/role denormalization, the association-list
routes (terminators, policies, configs), and the router data-model sync
- rejects edge-surface updates of fabric-only services, and preserves edge fields
on fabric-surface updates of edge services
- marks the fabric-only guards TEMPORARY(fabric-edge-collapse): they exist only to
keep seldom-used management services hidden, and are removed when the fabric/edge
distinction is erased
- adds in-CI migration tests (forward plus a round-trip asserting FK sub-bucket
relocation and forward/reverse refcount preservation), durable fabric-only
behavior tests, and store-level policy-exclusion tests
- adds one-time authentic-migration gate tooling under zititest/migration-test
(create-model.sh, query, verify) and its operator runbook
The router's CheckConnections reaper enforced only JWT expiry, so a revoked OIDC
api-session, or a disabled/deleted identity, kept its live circuits and hosted
terminators until the access token expired. The router now enforces the
RouterDataModel revocations directly, tightening access-loss propagation to the
reaper interval.
- adds a Type field to DataState_Revocation and the raft Revocation command
proto, mirroring rest_model.RevocationTypeEnum (API_SESSION/IDENTITY/JTI) so the
management API, OIDC producers, sync, and router enforcement share one
vocabulary; the common.RevocationType* constants are compile-time bound to the
enum to prevent drift
- adds an IssuedBefore cutoff so an identity revocation invalidates only sessions
issued before it; a session re-authenticated after the cutoff survives the
still-lingering revocation. Persists IssuedBefore on the db and model Revocation
and carries it (plus the Type) through the single and batched raft marshalling
- adds RouterDataModel.IsApiSessionRevoked and IsIdentityRevoked and enforces both
in CheckConnections, closing a revoked session's connections
- revokes a deleted or disabled identity's live OIDC sessions via an
IdentityRevocationConstraint in the db package, run as a store pre-commit
constraint so the revocation is written in the same transaction as the identity
change and cannot be skipped (self-contained OIDC JWTs aren't otherwise
reachable). NewIdentityManager installs it with the revocation type and lifetime
- has the OIDC end-session (TerminateSessionFromRequest) revoke the specific
api-session named by the z_asid claim, with an identity-scoped fallback; sets
IssuedBefore on the identity fallback and the management revocation API; adds
RevocationManager.CreateOrReplace, routed through by both the OIDC paths and the
management revocation API, so a repeat logout/termination/revocation refreshes
the cutoff rather than colliding on the reused id. Expiry derives from the
longest configured token duration via a shared common.MaxTokenDuration helper
- adds tests/revocation_enforcement_oidc_test.go covering api-session revocation
(and a fresh session staying unaffected), identity disable and delete, and the
identity cutoff (a post-cutoff session surviving the lingering revocation)
- fixes RevokeToken double-write: adds return after JWTID-keyed revocation
save, preventing fallthrough write with raw JWT string as unreachable key
- fixes TerminateSession key mismatch: stores revocation by identityId alone,
matching the Subject-based lookup in ValidateAccessToken
- fixes RevocationDelete sync action: passes DataState_Delete instead of
DataState_Create so routers evict the entry from their data model
- adds RevocationManager.DeleteExpired: batch-deletes expired revocations in
batches of 500 until none remain
- adds RevocationEnforcer: periodic policy runner (every 1 minute) that calls
DeleteExpired and records metrics
before test fixes
- removes legacy v1 terminator code path; all terminators now use v2 flow
- refactors edgeTerminator.close() to decouple SDK notification from control plane notification
- adds pending SDK close notification queue with retry when channel is busy
- adds post-create inspect mechanism that verifies SDK still holds the bind after terminator creation
- queues second post-create inspect when establishment takes >30s to catch SDK timeout races
- detects and discards stale reordered binds on the same connection by comparing connIds
- re-establishes replacement terminators when a delete/create race is detected
- eliminates IsEntityPresent pre-filter in removeTerminatorsHandler to prevent raft ordering races
- fixes ValidateTerminators to query identities from the correct manager with the correct filter field
- adds postCreate flag to ValidateTerminatorsV2Request so routers skip redundant SDK inspect
- returns retry-later (nil result) from router validation when inspect is temporarily unavailable
- blocks SyncAllSubscribers until completion and guards RouterDataModel replacement with in-progress flag
- fixes InheritLocalData to enable service access tracking for all subscribed identities
- adds Services.Has check in GetServiceAccessPolicies to prevent false policy grants
- validates policy-to-identity associations in ValidateServicePolicies
- adds `ziti agent tunnel dump-sdk` command for SDK context inspection via IPC agent
- adds `ziti fabric inspect sdk` command to query SDK context through routers
- fixes --expected-per-host CLI flag binding in validate terminators command
- changes bind-access-lost retry hint from NotRetriable to RetryStartOver
- moves trace route response and xgress close handling off channel handler goroutine
- fixes listTerminators test helper to URL-encode filter parameter
- improves sdk-hosting-test validation resilience with login and query retries
- adds terminator_create_flow.md documenting the full lifecycle across SDK, router, and controller
- adds detailed logging for data model sync, service access tracking, and subscriber change detection
* Allow routers to request current cluster membership information. Fixes#3503
* Get cluster membership information from raft directly, rather than trying to cache it in the DB. Fixes#3501
* Set a router data model timeline when initializing a new HA setup, rather than letting it stay blank. Fixes#3500
* Reduce router data model full state updates. Fixes#3504
* fix#3241 fix unsafe nil fingerprint dereference
- panic when a router connects and disconnects quickly or fails to
authenticate
- add safe access
- add an additional safe deref during connect
Andrew Martinez