- adds 'ziti ops cluster recover <controller-config>', an offline CLI
that opens a stopped controller's data directory, forces the raft
configuration to a single local node via raft.RecoverCluster, and
aligns the FSM-tracked member list in ctrl-ha.db so stale peers don't
leak through IsPeerMember or CtrlAddresses on restart
- removes the previous in-process recovery path: the cluster.recover
config flag and the corresponding RaftConfig.Recover field are gone,
along with the os.Exit branch in Controller.Init that consumed them
- adds BoltDbFsm.OverwriteServers and GetCachedServers so offline
tooling can update and inspect the FSM-side server list without a
live raft instance; OverwriteServers runs before raft.RecoverCluster
so the snapshot it produces captures the corrected configuration
- updates Broker.AcceptClusterEvent to call DeleteRemovedPeers on every
ClusterLeadershipGained, making the controllers entity table
self-healing for any membership change a non-leader missed (offline
recovery, or a 'cluster remove' applied while another node was leader)
- wires the new subcommand into both V1 and V2 CLI roots so it's
reachable regardless of ZITI_CLI_LAYOUT
- switches filesystem path joins in the raft package and the recover
command from path.Join to filepath.Join
- tests bootstrap a two-node configuration, run recoverDataDir, then
verify the post-recovery snapshot, the FSM-cached server list, and
Fsm.GetCurrentState (after starting a real raft instance) all report
the survivor only
- adds the missing self.startIndex assignment in BoltDbFsm.Init so
GetStartRaftIndex() returns the persisted raft index instead of 0,
which was leaving the RDM's RaftIndexProvider seeded at 0 on every
restart and reporting a stale index until the next command applied
- adds a regression test that opens an FSM, persists a raft index,
reopens it, and asserts GetStartIndex reflects the persisted value
- notes the fix in CHANGELOG.md under the ziti/v2 issue list
* fixes#3809 support CSR submission during OIDC authentication
- accepts an optional CSR during OIDC login (all auth methods) and
signs it into a session-bound certificate with a SPIFFE ID derived
from the identity and API session
- returns the signed certificate PEM as a top-level "session_cert"
field in the token endpoint JSON response (CodeExchange, RefreshToken,
TokenExchange)
- adds cert-binding verification on RefreshToken and TokenExchange:
if z_cfs is present the peer cert fingerprint must match (strict),
otherwise falls back to SPIFFE ID verification
- supports cert rotation via csr_pem form parameter on refresh and
token exchange; replaces the session cert fingerprint while
preserving the authenticating cert fingerprint
- adds AuthCertFingerprints (z_acfs) claim to track permanent auth
cert fingerprints separately from rotatable session cert fingerprints
- only the leaf certificate fingerprint is added to z_cfs and z_acfs,
intermediates are never included
- invalid CSR returns 400 Bad Request in OIDC error format
- propagates updated CustomClaims (including CertFingerprints) from
access token to renewed refresh token so rotated fingerprints are
enforced on subsequent refreshes
- adds CertGenerated field to ApiSessionEvent
- advertises OIDC_AUTH_WITH_CSR controller capability when OIDC is
enabled
- adds unit tests for verifyCertBinding (fingerprint, SPIFFE ID,
edge cases) and CsrPem field parsing
- adds integration tests for initial CSR auth (updb, cert, ext-jwt),
cert-binding on refresh/exchange, CSR rotation with cert auth,
SPIFFE fallback and z_cfs transition, and CSR property forging
resistance
* address pr concerns
* add z_cfs len tests on junk chain certs
* strip csr subject info, replace w/ santized values
* fix csr rotation rejection/paths during token exchange/refresh
- gates OIDC `z_cfs` claim on cert-based primary auth so non-cert sessions don't bind TLS-presented certs
- reorders router api session validation: fingerprint match first, then SPIFFE-on-first-party as fallback
- accepts first-party SPIFFE-Identity match as an acceptance signal (previously fell through to fingerprint check)
- adds `ziti agent inspect <value>...` CLI that sends an InspectRequest directly to a ziti process over its agent IPC channel and pretty-prints JSON values in the response
- accepts app id 0 on controller and router agent channels so a single inspect command works against any process type
- adds Controller.agentOpInspect, backed by a new InspectionsManager.InspectLocal that runs inspect processing on the local controller only, without fanning out to routers or peer controllers
- extracts the router inspect handler into a new router/inspect package and stores a single shared instance on Router, reused by both the control channel and the agent IPC channel
- adds RouterEnv.GetInspectHandler and RouterEnv.GetXgressListeners so the control channel bind pulls the shared handler from env
- removes the now-redundant InspectRouterEnv interface from handler_ctrl/bind.go
- adds tunnel HandleAgentAsyncOp with support for stackdump and sdk inspect keys
* fixes#3734 enforce client certificate proof-of-possession for OIDC sessions
- adds verifyCertProofOfPossession() in resolveOidcSession() to require
TLS client cert matching a z_cfs fingerprint or SPIFFE ID when the
OIDC token was issued with cert bindings
- adds z_cae (CertAllowExpired) claim from auth policy, propagated
through token issuance and refresh
- adds TrustCache.VerifyClientCert() with tiered pool matching
(first-party roots, trust anchors, third-party) and TTL cache
- adds WrapIdentityWithCertValidation() on the router to verify client
certs at the TLS level against RDM PublicKeys
- adds IsFirstPartyCert() on the router to gate SPIFFE ID matching by
checking whether the cert chains to the controller root CA
- adds VerifySpiffeId() in common/spiffehlp with SpiffeMatchApiSession,
SpiffeMatchIdentity, and SpiffeMatchNone return types
- adds SPIFFE IDs to OTT and token enrollment certs (/identity/<id>)
- enforces cert expiry by match type: API session certs must be valid,
fingerprint-matched certs respect z_cae, legacy sessions skip checks
- shallow-copies leaf certs before overriding time fields in all cert
verification paths to avoid races on shared x509.Certificate pointers
- fixes controllerRootCache setting inited=true before the ctrl channel
is available, which permanently cached the failure
* fixes#3806 expose OpenZiti endpoints in OIDC discovery document
- adds vendor-specific "openziti_endpoints" field to the
/.well-known/openid-configuration response
- overrides Discovery() on the OIDC server to wrap the standard config
with OpenZiti login and MFA endpoint URLs
- advertises password, cert, ext-jwt, totp, totp enrollment, and
auth query endpoints as absolute URLs derived from the issuer
- adds integration test verifying all openziti_endpoints fields
- adds dual-server integration test confirming endpoint URLs reflect
the correct issuer when edge-oidc is hosted on multiple bind points
* changelog
- registers the service policy type symbol as a string, so queries match the
API's "Dial" and "Bind" names rather than the internal int32 ids
- adds a symbol mapper that converts the stored int32 to its PolicyType name
at query eval time
- updates the posture-checks lookup in EdgeServiceManager, the only direct
caller that went through GetSymbol, to read the mapped string form
- adds a store test covering type = "Dial" and type = "Bind" filtering
- notes the breaking removal of the undocumented type = 1/type = 2 form in
the 2.0 deprecation cleanup list
- migrates from op.NewProvider() to LegacyServer/RegisterLegacyServer,
routing all OIDC endpoint errors through op.WriteError which maps
server_error to HTTP 500 and supports custom status codes via
op.StatusError
- returns oidc.ErrInvalidClient() from AuthorizeClientIDSecret for
unknown clients and bad secrets (HTTP 400 with invalid_client)
- returns oidc.ErrInvalidGrant() from parseRefreshToken,
parseAccessToken, createAccessToken, and renewRefreshToken for
client-supplied token errors (HTTP 400 with invalid_grant)
- fixes copy-paste bug in parseAccessToken that reported "invalid
refresh_token" for access token errors
- plain Go errors from server-side failures (identity read, JSON
marshal, token signing, Raft dispatch) now correctly surface as
HTTP 500 via WriteError's DefaultToServerError handling
- adds integration tests covering error codes for token endpoint,
login endpoint, userinfo endpoint, and end_session endpoint
* fixes#3680 add revocation management API, CLI, and enforcement
- adds Management API endpoints for revocations (POST, GET, LIST) with
type-aware validation (JTI/API_SESSION require UUID, IDENTITY requires
existing identity)
- adds CLI commands: ziti edge create revocation identity|api-session|jti
- adds revocation checks to resolveOidcSession in security_ctx.go so the
REST API returns 401 for revoked OIDC tokens. Previously only
ValidateAccessToken (router ctrl channel path) checked revocations,
so revoked tokens still received 200 OK from the management and client
HTTP APIs
- adds api-session revocation check to ValidateAccessToken, which only
checked JTI and identity revocations
- adds Type field to Revocation model, store, and protobuf message
- adds integration tests covering CRUD, input validation, and token
enforcement for all three revocation types
- use release edge-api@v0.28.1
- adds ErrorType constants for rejected-by-application, DNS resolution failed,
port not allowed, invalid link destination, and resources not available
- adds corresponding CircuitFailureCause strings reported in circuit events
- extracts classifyDialError() in route handler to map dial errors to specific
error codes using typed errors, syscall constants, and string matching
- detects DNS errors via *net.DNSError and string fallback for ER/T hosted
services where errors are serialized through the SDK message protocol
- detects resource exhaustion via EMFILE, ENFILE, ENOBUFS syscall errors
- introduces InvalidLinkDestinationError typed error in forwarder package
- adds unit tests covering all 16 classification cases
- adds integration tests for rejected-by-application (SDK host),
DNS resolution failed, connection refused, and port not allowed
(ER/T host mode) with circuit event verification
- adds CreateEnrollAndStartTunnelerEdgeRouterWithCfgTweaks to test context
- changes lastResponse from plain int64 to atomic.Int64 in both router
and peer heartbeat callbacks, fixing a data race between the heartbeat
response handler and the heartbeat check ticker
- fixes peer heartbeat logger channelType from "router" to "peer"
- uses already-parsed AdvertiseAddress for cert validation instead of
re-reading from the raw map, which panicked when the key was absent
- replaces errors.Wrapf(err, ...) with fmt.Errorf where err is nil, so
routerDataModel.listenerBufferSize validation errors are returned
instead of silently swallowed
- fixes pfxlog.Logger().Warn() call using %w verb, which is not
interpolated outside fmt.Errorf
- prefers the signing cert from the hello header when available, falling
back to the TLS underlay cert
- applies consistently on both dialer and acceptor sides
- removes the dead append on the acceptor side that was immediately
overwritten
- checks for the timeout key in the rateLimiter submap instead of the
parent tls cfgmap, so a user-specified timeout is no longer silently
overwritten by the default
- swaps arguments to strings.HasPrefix so it checks whether the trust
domain starts with "spiffe://" rather than whether "spiffe://" starts
with the trust domain
- fixes both the trustDomain and additionalTrustDomains code paths
- replaces the per-handler goroutine in connectEventsHandler with a shared,
bounded goroutine pool on AppEnv
- the old design spawned a processEvents goroutine per router connection that
only exited on application shutdown, leaking a goroutine on every reconnect
- adds ConnectEventsConfig to the controller config with pool tuning options
(queueSize, minWorkers, maxWorkers, idleTime)
- defaults: queue 16, 0-16 workers, 30s idle timeout
- adds CreateCircuitV3 message type and handler for routers that have
already authorized dials locally via RDM, bypassing service session
tokens in favor of identity ID, service ID, and pre-assigned circuit ID
- renames CreateCircuitRequest/Response to CreateCircuitV2Request/V2Response
for clarity now that V3 exists
- adds CircuitManager.Reserve to atomically claim circuit IDs before routing,
preventing collisions on pre-assigned IDs
- extends CreateCircuitParams with GetCircuitId so V3 can supply a
pre-assigned circuit ID (falls back to UUID generation when empty)
- fixes IsDialableByIdentity which was incorrectly calling IsBindableByIdentity
- extracts V2 handler into its own file create_circuit_v2.go
- adds CreateCircuitV3RequestType/ResponseType (20222/20223) to edge_ctrl protobuf
- registers V3 handler in controller server
- skips certificate DNS names starting with "*." in getPossibleIssuers,
since they produce unusable OIDC URLs (e.g. https://*.example.com/oidc/authorize)
- updates test to verify wildcard DNS SANs are excluded while IP SANs
from the same cert remain valid issuers
- adds DeleteRevocationsBatchCommand so expired-revocation cleanup goes
through raft as a single log entry per batch
- adds CreateRevocationsBatchCommand for batched revocation creation
through raft
- moves refresh-token revocations from synchronous inline creation to a
background batcher that flushes on a configurable interval, removing
the database and raft as a bottleneck on token refreshes
- skips revocation creation for tokens expiring within a configurable
threshold (revocationMinTokenLifetime), since they become invalid on
their own
- validates that revocationMinTokenLifetime is less than 50% of the
configured refresh token lifetime
- makes the revocation enforcer frequency configurable and restricts it
to run only on the raft leader
- adds tests for multi-batch delete, batched create with router RDM
propagation, and skip-threshold behavior
- adds new configuration tunables under edge.oidc: revocationBucketInterval,
revocationMinTokenLifetime, revocationBucketMaxSize, revocationMaxQueued,
revocationEnforcerFrequency
- fixes RevokeToken double-write: adds return after JWTID-keyed revocation
save, preventing fallthrough write with raw JWT string as unreachable key
- fixes TerminateSession key mismatch: stores revocation by identityId alone,
matching the Subject-based lookup in ValidateAccessToken
- fixes RevocationDelete sync action: passes DataState_Delete instead of
DataState_Create so routers evict the entry from their data model
- adds RevocationManager.DeleteExpired: batch-deletes expired revocations in
batches of 500 until none remain
- adds RevocationEnforcer: periodic policy runner (every 1 minute) that calls
DeleteExpired and records metrics
before test fixes
* fixes#3496 allow TOTP enroll mid OIDC
- detects unenrolled TOTP mid-OIDC flow and redirects to enroll endpoints
- adds enroll/verify endpoints in oidc_auth/login.go
- centralizes auth query storage rendering in oidc_auth/storage.go
- adds SDK integration tests for enrollment success, re-auth, cancelled
enrollment, empty/alpha/wrong codes, and missing provider
- removes reduce rp server usage in tests
- refactors some oidc tests to new style where possible
- fixes amr regression in 2.0
- adds test for amr and auth_time
- removes legacy v1 terminator code path; all terminators now use v2 flow
- refactors edgeTerminator.close() to decouple SDK notification from control plane notification
- adds pending SDK close notification queue with retry when channel is busy
- adds post-create inspect mechanism that verifies SDK still holds the bind after terminator creation
- queues second post-create inspect when establishment takes >30s to catch SDK timeout races
- detects and discards stale reordered binds on the same connection by comparing connIds
- re-establishes replacement terminators when a delete/create race is detected
- eliminates IsEntityPresent pre-filter in removeTerminatorsHandler to prevent raft ordering races
- fixes ValidateTerminators to query identities from the correct manager with the correct filter field
- adds postCreate flag to ValidateTerminatorsV2Request so routers skip redundant SDK inspect
- returns retry-later (nil result) from router validation when inspect is temporarily unavailable
- blocks SyncAllSubscribers until completion and guards RouterDataModel replacement with in-progress flag
- fixes InheritLocalData to enable service access tracking for all subscribed identities
- adds Services.Has check in GetServiceAccessPolicies to prevent false policy grants
- validates policy-to-identity associations in ValidateServicePolicies
- adds `ziti agent tunnel dump-sdk` command for SDK context inspection via IPC agent
- adds `ziti fabric inspect sdk` command to query SDK context through routers
- fixes --expected-per-host CLI flag binding in validate terminators command
- changes bind-access-lost retry hint from NotRetriable to RetryStartOver
- moves trace route response and xgress close handling off channel handler goroutine
- fixes listTerminators test helper to URL-encode filter parameter
- improves sdk-hosting-test validation resilience with login and query retries
- adds terminator_create_flow.md documenting the full lifecycle across SDK, router, and controller
- adds detailed logging for data model sync, service access tracking, and subscriber change detection
* fixesopenziti/ziti#3626 omit overlay bind points from /versions apiBaseUrls
- adds BindPointTypeUnderlay and BindPointTypeOverlay constants
- implements Type() on UnderlayBindPoint and OverlayBindPoint
- skips non-underlay bind points when building apiBaseUrls in version_router
- skips non-underlay bind points in GetApiAddresses
- adds unit tests for Type() on both bind point implementations
- updates xweb
Paul Lorenz