- applies the router.link.v1 heartbeat settings (sendInterval, checkInterval,
closeUnresponsiveTimeout) and payload/ack sender queue sizes at runtime,
replacing the hardcoded 10s/1s link heartbeat intervals and the ignored fields;
new links take them at bind time, established links are re-tuned in place
through the channel HeartbeatControl, and queue sizes reach new links only
- publishes heartbeat settings as an immutable generation in one atomic swap, so
no reader pairs one generation's interval with another's timeout, and has each
link enforce the timeout belonging to its own intervals
- pushes a generation and its intervals as one step under the link's lock and
ignores a generation at or below the one applied, so an overtaken update cannot
roll a link back; reconciles every channel of a link, including a split link
whose channels bind on either side of a config change, to the newest generation
- adopts the first registered settings explicitly, since a router with no managed
link config publishes its defaults at generation 0
- re-arms a link's liveness deadline when its generation changes, so age
accumulated under the previous intervals cannot close a healthy link when a
timeout is shortened
- honors a closeUnresponsiveTimeout below 30s, which previously sat behind the
unhealthy threshold; treats a zero timeout as closing nothing, and reports a
closing link unhealthy
- scales the unhealthy threshold with the intervals, to max(30s, 2 x (send +
check)): reporting unhealthy poisons the latency metric the controller routes
on, so a fixed 30s priced healthy links with a long send interval out of
routing every cycle
- validates heartbeat timing on the controller and the router: durations must be
positive, their sum representable, and the timeout at least 10s and greater
than sendInterval + checkInterval; a timeout under two heartbeat cycles is
accepted with a warning
- adds a per-config-type semantic validator registry to ConfigManager, run after
schema validation, and moves the router.link config definition to
common/config/routerlink so the controller applies the same rules
- treats a link section in router YAML, of any content, as local config that
takes precedence over the controller's for the whole router.link type, since
the loader defaults every field and so now records presence separately; warns
at startup when that section has no listeners or dialers
- translates local heartbeat timing leniently, since a local failure stops the
router from starting: keeps an explicit zero, raises a timeout below the floor
to it, and falls back to the defaults for other invalid timing, each with a
warning
- stamps UpdateLinkListeners with the config generation it came from and has the
controller drop superseded sets, which a multi-underlay control channel can
deliver out of order and which would otherwise send peers to withdrawn
addresses
- logs every successful link config apply, including those that dispatch no work,
and pins that a queue-size-only change triggers no GC pass
- deprecates split links, documents the feature in the changelog and the router
config design doc, and brings both up to date
Builds on the channel/v5 HeartbeatControl from openziti/channel#265.
- adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId,
ListenerBinding}) used by all staleness and GC checks, replacing the
pre-rendered key string each side used to pass around
- adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links`
CLI for operator-driven, two-sided staleness verification with optional --gc,
which acts only when both endpoints agree
- implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed):
the router walks its xlinks after each Apply that mutates listeners, dialers,
or gcMode and closes one-sided-stale entries under the configured mode
- sweeps a removal under orphaned whatever the removed config said, including
preserve, since a removal leaves no listeners and no dialers and every
remaining link is one the router can never re-establish
- reports staleness as a three-state verdict (unknown/notStale/stale) so a side
that cannot judge a link abstains instead of guessing; a destination absent
from the registry's listener snapshot means "listeners unknown", not "peer has
none", which keeps a briefly unhealthy peer from losing its live links
- rebuilds the link surface only when a listener or dialer definition actually
moves, so a gcMode-only change no longer rebinds every listen socket
- gates CheckStaleLinks on a new RouterStaleLinkCheck capability and explains
each unqueryable endpoint, including one left unasked by the filter, so a
partial verdict says which of the three causes applies
- detects a peer re-advertising the address a link was dialed to from the
dialing side, not just from the peer's own listener side
- pins every verdict to the link iteration it was computed against, faults that
iteration rather than the current one, and takes the conditional removal's
result as the decision, so a link re-dialed while the sweep was running is
neither closed nor reported as collected
- resolves which side a report speaks for from the sending router against the
controller's link map rather than from the side the report names, so one
router cannot fill both sides and satisfy the both-agree rule --gc depends on
- bounds the query fan-out with a semaphore, matching the other validation
sweeps, since each query carries a minute-long reply timeout
- fails the CLI when the result stream ends before every link is reported, and
drains results already buffered before declaring any missing
- rejects an unrecognized match mode rather than resolving it to changed, the
broadest removal criterion
- consolidates the controller-managed router configuration changelog into one
section, replacing the status that said routers do not yet consume configs at
runtime, and documents the per-config-type opt-in
- restores the Current Beta Features section, dropped when the 2.0 changelog was
archived, and lists controller-managed router configuration in it
- adds unparseable to the codespell ignore list, a spelling variant the
dictionary flags as a typo
For #3743.
BuildRouterLinks runs once per router connect and scanned the whole link table to
find the connecting router's links. The table is proportional to the square of the
router count in a full mesh, so a wave in which every router reconnects walks the
cube of it in link visits. That never shows in steady state; it arrives during a
mass reconnect, when the connect path is already the bottleneck.
Indexing every link under both endpoint router ids fixes the cost, and introduces
the problem that the table and the index are two structures written in sequence.
Three things follow from that, and the destination repair below turned out to
depend on the connect path's ordering as well.
Performance:
- indexes every link under both endpoint router ids, maintained in the add and
remove paths all link mutations funnel through, so a connect costs its own
router's link count rather than the whole table
- prunes a full refresh against the reporting router's own links. That report is
what every reconnect sends, and it walked the whole table to find the router's
stale links, so the cost the connect path shed was still being paid one message
later
- indexes link ids rather than link pointers, so a per-router query resolves each
id against the table and hands back whatever it currently holds. A reconnect
replaces a link's Router object and a higher iteration replaces the Link object,
but neither replaces the id, which is exactly why the scan existed
- holds each router's link ids in a LockedSet rather than a sharded map, which
cost a map and a mutex per shard for an index that is usually empty and is held
for as long as the router has existed: 2.9KB per router id against 150 bytes
- drops a router's index when the router is deleted, guarded on the store still
having no router under that id and checked inside the index map's removal
callback. A router id can come back, since fabric router ids are the enrollment
certificate's common name, and nothing orders the cleanup against that: store
event handlers run after bolt has released the writer lock, so one can execute
after the id was recreated, connected and reported links. Indexing a link takes
the same shard as the guarded read, so an entry for a live router cannot appear
without its create having committed first, which the read then sees, and an entry
arriving later blocks and lands in a freshly created index
- keeps an index when the store cannot be read, since that is not evidence the
router is gone, and dropping one otherwise could discard a link another goroutine
had already fetched the index to record
Keeping the two structures agreeing:
- serializes Add and Remove on the link's id. Without it a removal running between
an add's two writes takes the table entry, finds nothing yet in the index, and
lets the add index a link the table no longer holds, which nothing cleans up.
Only RouterReportedLink held that lock before; the four other removal paths held
nothing
- serializes the connect-time pairing on that id as well, which was the last writer
left outside it. A removal reading no destination could complete while the
pairing was still deciding, leaving the link in the destination's link set after
the table had dropped it, where path computation still routed over it
- unindexes an id only when the table no longer holds a link under it, so a
replacement that has taken the table slot is not unindexed by the older link's
removal. A stale id is the safe direction to err in, since a query filters it,
where an id wrongly dropped leaves a live link no per-router query finds
- has per-router queries resolve ids against the table, so an id it no longer
holds is not handed out as a live link
Pairing a link with its destination:
- re-resolves the destination once the link is in the table, and on every later
report, so a link that reached a bad state heals. A report resolves the
destination before the link is in the table and a router's connect repairs links
already in it, so a router connecting between those two moments was seen by
neither, leaving a link with no adjacency that every operator-facing view still
called healthy
- makes that re-resolve unconditional rather than only for a link holding no
destination. The resolve happens under the source router's connect stripe, not
the destination's, so the destination can be replaced between the resolve and the
report landing, and a link left on a connection that is no longer registered
carries no adjacency just as one holding none does
- registers a connecting router before building its links, which is what makes
that re-resolve close the window rather than narrow it
- adds Link.PointDestAt so the two repair paths cannot both index the link on the
destination, since that index is a slice that does not deduplicate
Also here:
- logs the link count on both sides of the reconnect exchange, and why a report was
discarded. A router announces its links once per reconnect and is not asked
again, so a discarded report left it with no links for a router that believed it
had announced them, with nothing recorded to compare. The discard reports the
state it decided on, read once, since a reconnect landing between the decision
and the log would otherwise have it name a state that discards nothing
- gives test routers version info, which a connected router always has because the
accept path refuses a hello without it, and which the router sync path
dereferences with no nil check
The destination repair is backported from the gossip link-state branch, where the
connect path registers before building links for unrelated reasons. That ordering
is load-bearing here and is now explicit at the call site.
* fixesopenziti/ziti#4321 compare process hashes and fingerprints case-insensitively
- normalizes configured hashes and signer fingerprints to lowercase unseparated hex in the
PROCESS and PROCESS_MULTI stores, on load as well as on save, so values stored before this
change read back normalized without a migration
- replaces the PROCESS store's lowercase-only pass, PROCESS_MULTI stored its values verbatim
- adds store tests for both types covering configured values and values written straight to bolt
- updates the process multi patch round-trip test, the management API now returns normalized
values rather than the submitted strings
- removes the trailing newline from the signer fingerprint literal in the process and
process-multi OIDC posture tests
* fixesopenziti/ziti#4094 accept first-party certs issued by a separate edge signing CA
- adds FirstPartyX509CertValidation and ThirdPartyX509CertValidation usages and an
intermediates field to the router data model public keys, deprecating
ClientX509CertValidation
- publishes config CA bundle roots as first-party anchors with their intermediates and
Ca store entries as third-party anchors; controller certs carry JWT validation only,
since a controller identity is never a CA and anchors no client cert chains
- builds router first-party and client cert trust pools from the published usages,
falling back to the deprecated usage against older controllers
- propagates the full signing cert chain between controllers via a new mesh
SigningCertChainHeader and persists whole chains in controller records
- removes the orphaned InstantStrategy.AddPublicKey, dead since public key sync moved
to controller list data
- gives each command dispatcher its own decoder registry so multiple in-process
controllers no longer decode into the last-started controller's managers
- adds a three-controller in-process HA test harness with a split signing PKI,
cluster formation and first-party cert integration tests
- trusts the edge signing CA when verifying router control channel certs
- adds a variadic additionalRoots parameter to VerifyLeafCertChain, applied to a
clone of the caller's pool so an identity's live tls.Configs are unaffected
- passes the edge enrollment signing CA bundle as additional roots when admitting
a router control channel connection, so a deployment whose signing CA sits
outside the controller's own trust bundle no longer has every router refused
- leaves the fingerprint check bound to the verified leaf, so the wider anchor set
changes which chains verify, not which routers are admitted
- covers the split-root case and the caller-pool guarantee in common/cert tests
- enforces issuer and audience claims inside TokenIssuerExtJwt.VerifyToken, which
previously verified only the token signature and resolved the signing key by kid
- closes the ziti-token-issuer-id header enrollment path accepting a validly-signed
token minted for a different audience or issuer that shares signing keys
- mirrors the runtime ext-jwt authentication and by-inspection enrollment paths so
all paths validate claims consistently
- adds a controller/model unit test covering foreign-audience, foreign-issuer, and
missing-audience rejection
- adds HardenedJwksResolver, a jwks.Resolver that fetches an external jwt signer's
jwksEndpoint with an http/https-only scheme check, a total timeout and a redirect cap
- adds JwksFetchPolicy, gating a fetch on both the URL hostname and the address being
connected to, applied to the first request and to every redirect hop
- hostname gate: deniedHostnames blocks, allowedHostnames is exclusive when set; entries
are an exact hostname or a '*.suffix' wildcard that matches subdomains at any depth but
never the suffix itself, normalized to lower case punycode without a trailing dot
- address gate: built-in blocked (metadata, link-local, link-local multicast,
unspecified), then deniedIPs, then allowedIPs, then blockPrivateAddresses,
first-match-wins with deny over allow
- keeps the gates independent, so neither can authorize what the other refuses; the
address check runs in the dialer against the resolved address, so a hostname that
resolves to a blocked address is refused
- adds the [edge.externalJwtSigners.jwksFetch] config section with compatible defaults:
empty hostname lists, blockPrivateAddresses false, timeout 5s, maxRedirects 5
- takes IP lists as a flat address or a CIDR block, hostname lists as names only, and
rejects an entry belonging to the other list at startup
- rejects a jwksEndpoint the policy refuses when an external jwt signer is created or
updated, and logs an existing signer whose endpoint the configuration now refuses
when the token issuer cache loads
- documents both gates, their deny-wins precedence, the accepted entry forms and the
wildcard matching rules in etc/ctrl.with.edge.yml and CHANGELOG.md
The controller decided which of two racing connections for a router was current
by comparing router instances, but loaded one per connect by evicting the router
cache and reading back through it. Two connects could both evict, and whichever
read second was handed the instance the first had just published. A shared
instance makes the two connections indistinguishable: the connect path cannot
reject the second into an occupied slot, and when either channel dies the
disconnect path finds itself current and tears down the registration the other is
still using. The surviving channel is never re-bound, so the router stays
connected at the transport layer while absent from the model, unable to recover.
Connect and disconnect were also unserialized, so a stale or superseded
disconnect could interleave with a live connection and take its links with it.
- serializes a router's connect and disconnect with a per-router striped lock
- keeps at most one connection per router: a connect into an occupied slot is
rejected via an error from ConnectRouter, so the bind fails and NewChannel
closes it without starting rx or registering it, and the occupant is displaced;
the router redials into the freed slot
- displaces an occupant by closing its channel and also invoking the teardown
directly, since a channel that is already closed never fires its close handler
again; without this a dead but still registered connection holds the slot
forever and every redial is rejected against a slot nothing can free
- refuses a connect whose control channel is already closed rather than
registering it, so a connection no disconnect could ever remove is never
published
- gives every connection its own router instance via RouterManager.NewCtrlChanRouter,
read through readUncached so the cache neither supplies nor receives it, which
is what makes comparing instances meaningful
- moves recording the channel and connect time out of the accept path, so a
caller cannot attach the wrong channel or forget to attach one
- serializes link publication with that teardown on the same per-router stripe.
Validating currency and then publishing without it is a check-then-act: a
report can find the connection current and, by the time it reaches the link
manager, the teardown has already snapshotted and cleared the router's links,
so the link is recreated after everything that would have removed it. It is
then absent from the router's own index while still in the link table with a
disconnected source, and a reconnect reporting the same iteration can adopt
that stale source instead of rebuilding the link
- guards the entire DisconnectRouter teardown by connection currency, all or
nothing, and clears the connected flag and link index only when the
registration was actually given up, with the flag cleared under the same shard
lock as the map removal so the two cannot be observed disagreeing; the connected flag decides whether the
controller accepts a router's link reports, so clearing it for the wrong
connection silences a router that is up and reporting
- reduces MarkConnected to publishing the connection; the takeover-close moves
into ConnectRouter's reject path
- makes the per-router unlock idempotent so callers can defer it as a leak-safety
net and still unlock early before closing a channel outside the lock
- stops the replaced RouterSender in routerTxMap.Add so a takeover does not leak
the old sender's goroutine when the broker's asynchronous RouterDisconnected
loses the race to the redial's RouterConnected
- discards pending peer state changes for a router whose channel has closed,
since sending on one fails immediately and the failed send is retried as soon
as the event loop turns, spinning the loop and flooding the log
- queues the peer-state send-done event on every path, so a missing channel can
no longer leave sendInProgress set and stall that router's updates permanently
- resolves a router's version from its connected instance when validating link
conn info, since the version arrives in the hello and so is absent from an
instance loaded from the database
- normalizes both endpoints to the connected instance in shortestPath, which is
keyed and compared by pointer and so treated an endpoint held as any other
instance of the same router as absent from the graph, reporting a router as
unroutable from itself. That worked before only because the connect path
published its instance into the router cache, so a cache read and the connected
map returned the same object; nothing stated the requirement
- configures test logging once per package in TestMain, so a test no longer
writes global logger state while a previous test's shutdown logging reads it
- swaps testing.TB for require.TestingT in model.NewTestContext, so this
build-included file no longer imports testing
- documents why the narrower interface is used, so it isn't reverted
- leaves hashicorp/raft as the only remaining importer of testing in the
ziti binary's dependency graph
- adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop
- translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup
- adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path
- re-evaluates dialers on link group and listener changes via RescanForDialOpportunities
For #3743.
* fixesopenziti/ziti#4118 disambiguate overlapping ext-jwt-signer kids by issuer
- binds external JWT tokens to signers by exact issuer claim rather than by key ID, so signers drawing from a shared signing-key pool resolve deterministically
- binds controller-issued tokens by key ID first, preserving controller token resolution and preventing an external signer configured with a controller's issuer from capturing controller access tokens
- removes the external key-ID fallback so a token whose issuer matches no configured signer is not bound to an unrelated signer that happens to share its kid
- adds GetControllerIssuerByKid to the TokenIssuerCache interface and implementation
- skips disabled external signers in GetIssuerByKid so a disabled signer sharing a kid cannot poison resolution for an enabled one
- adds an integration test with two HTTPS JWKS providers sharing a key and kid, covering the enabled-collision and disabled-poison cases
- documents that controller issuers are keyed by controller id and that a controller issuer's key ID is the fingerprint of its TLS certificate
- documents that an external kid match is ambiguous because signers can share a signing-key pool, and that a definitive binding requires resolving by issuer claim
- clarifies the overlapping-kid test comment covering why issuer-claim binding is required when a disabled signer shares a kid
* fixesopenziti/ziti#4063 enroll with empty roles when ext-jwt attribute claim is absent
- treats an absent enrollment attribute claim as no role attributes rather than
rejecting the token; a failed jsonPointer.Get means the claim is not present,
since pointer syntax is already validated at jsonpointer.New
- matches the existing unset-selector and empty-claim cases, which already
enroll with an empty attribute set; a present-but-wrong-type claim still errors
- adds unit coverage for resolveStringSliceClaimProperty
- moves the two ext-jwt enrollment integration subtests that asserted the old
failure behavior to success cases asserting an empty role-attribute set
* fixesopenziti/ziti#4063 treat a null attribute claim as no role attributes
- treats an attribute claim sent as JSON null as no role attributes, logging a
warning, matching the behavior of an absent claim
- logs at debug when the attribute claim selector does not resolve
- prints the offending value in the wrong-type error rather than the always-nil
result of the failed array assertion
- corrects the godoc and inline comment: a pointer can fail to resolve through
traversal as well as absence, and the malformed-pointer error is unreachable
- adds unit coverage for null and nested null claims, and asserts the wrong-type
error names the value
- adds enrollment tests for a null attribute claim enrolling with no role
attributes, and for a null name claim rejecting enrollment
- removes the costTags option from router link listeners, which was parsed
from config, advertised to the controller, and stored on the router model
but never used for path selection or any other behavior
- drops GetLinkCostTags from the xlink.Listener interface and its transport
implementation
- reserves the corresponding ctrl_pb.Listener.costTags and
RouterLinks.RouterLink.linkCostTags protobuf fields and regenerates ctrl.pb.go
- documents the removal in the changelog
- reports edge router policy denials with an access-denied error naming the missing policy,
replacing the session error reused on the sessionless ER/T and create-circuit-v3 paths
- adds EdgeRouterManager.GetEdgeRouterAccess, which reports which of the two required policy
links (identity-to-edge-router, service-to-edge-router) is absent, and removes the boolean
IsAccessToEdgeRouterAllowed it replaces
- logs the controller's rejection on the router at warn level, since it is recoverable and
retried by the periodic scan; the router previously discarded the error code and message
- delays a new terminator's first create attempt by a fixed 2s so config applied in quick
succession settles before the router asks, avoiding a 2-3 minute wait for the retry scan; the
delay is a deliberate stopgap until edge router policy visibility lands in the router data model
- propagates the controller's error code and retry hint to SDK clients on the dial paths, which
dropped the code and left every refusal classified as unknown
- adds the retry hint header to controller error replies, grouped with the other error-reply
headers rather than the create-circuit-v3 request headers
- notes that the sync strategy headers alias the edge namespace's 1013-1015 ids and stay
disjoint only by message content type
- tests the per-policy denial reporting, the error code carried with and without a retry hint,
the controller-to-SDK error code mapping at both dial relay sites, and the terminator settle
gate
- waits for terminator establishment in the tunneler dataflow tests instead of a fixed sleep, so
they no longer race the settle delay
- restores the tproxy multiple-lanIf and multiple-resolver changelog entries with keep markers,
which regeneration drops because their commits reference pull requests rather than issues
- releases the per-identity lock before acquiring the cmap shard lock when the scan
loop reaps an entry, establishing a single shard-lock-then-value-lock order
- extracts the entry removal into removeIfEmpty, which decides based on the value
currently in the map, so an identity that reconnects after the scan decided to
remove it is left alone
- takes the per-identity lock in the removal check, fixing an unsynchronized read of
the router map
- documents the lock ordering invariant on identityConnections
- adds tests covering concurrent scanning and connect/disconnect handling, entry
reaping, and reconnection between the scan's decision and the removal
* fixesopenziti/ziti#3990 push service and posture state to subscribed SDKs
- pushes indexed atomic ServiceChangeSet envelopes to subscribed SDK connections: a full snapshot on subscribe, incremental service changes per RDM scan pass, posture check definition changes as their own entries, and identity-resolved config bodies, all serialized so envelopes hit the wire in index order
- pushes per-connection PostureStateChange state (monotonic seq, resync on request) for posture pass/fail, including flips caused by definition edits that mutate no posture data
- registers pending RDM identity subscriptions for identities not yet synced to the router and sends an authoritative full sync plus full posture state when the identity arrives; an active push subscription pins the connection's RDM listener
- advertises service subscriptions and router data model support on the control-channel capability bitmask; the controller persists each router's capabilities mask and version on the EdgeRouter entity via raft and renders them on the edge APIs, so SDKs can select capable routers before connecting
- submits posture per router and corrects MFA posture semantics: pushed expiry is the earliest of timeout and pending wake/unlock grace deadlines, wake/unlock re-pass satisfies the re-prompt, api session tokens whose amr attests TOTP seed the MFA baseline from auth_time only (never iat), and token exchange carries the subject token's auth_time
- sends structured denials on dial and bind refusals: posture failures carry the failing check ids, no-policy denials are access denied, unknown services are invalid service, and session token failures are invalid session; the denial's cause no longer rides the wire as an unserializable error
- hard-closes accepted SDK connections on edge listener shutdown so clients observe a router going away immediately
- adds integration coverage: subscription snapshots and change delivery, poll and push reconciliation as capable routers come and go, posture state and definition-change push, router views over the public SDK API, typed dial errors, MFA baseline seeding, and OIDC token-exchange auth_time preservation
- removed RDM capability from SDK, router/controller only
- keys router capability bits off the sdk-golang RouterCapability enum
- adds a generic capabilities.Mask[T ~int] bitmask, centralizing capability
set/check behind one value-to-bit translation
- adds capabilities.RouterCapability and ControllerCapability types, with
RouterCapabilityMask/ControllerCapabilityMask aliases, so masks and checks are
typed per namespace
- supports control-plane-only router capabilities as negative values that index
down from the top of the mask, collision-free with the SDK's upward-numbered
bits and invisible to the SDK and edge-api
- references the sdk-golang RouterCapability enum as the source of truth for
shared router capability bits
- holds the router's advertised capability mask as an instance on the router env
rather than a global, so in-process test routers do not share state
- advertises PostureChecks and BindSuccess as capability bits on both channels
while still sending the legacy boolean edge headers for backwards compatibility
- routes GetCapabilities/IsCapable and the controller's Router.Capabilities field
through the typed mask
- adds a provenance test that verifies, via go/packages, that every positive
router capability is SDK-sourced, every negative is control-plane-only, and no
two resolve to the same bit
- update sdk to v2.0.0-pre2
- adds a common/servermetrics package that owns the metrics MetricsMessage wire
format and the reporting/usage subsystem (message builder, usage registry,
interval and usage counters), wrapping the openziti/metrics Registry for
metric collection
- moves the controllers metrics reporter into the router package and removes it
from the shared metrics package, breaking a common -> router/env import cycle
- repoints controller and router consumers to common/servermetrics; base metric
collection stays on openziti/metrics
- keeps the proto field numbers and the metrics content-type identical so the
encoding is byte-compatible across the move, and uses a distinct proto package
name so ziti's and the library's messages coexist without a global proto
registry clash
- adds a round-trip test asserting wire compatibility with the library's
MetricsMessage
- leaves openziti/metrics unchanged, so sdk-golang and the shared xgress data
plane are unaffected
* fix#3933 add controller to enrollment response
- adds the cluster's controllers to ott, ottca, updb, and token enrollment
responses with client and OIDC API addresses only
- synthesizes the running controller with its API addresses in non-HA mode
so the list is never empty
- adds --not-before to ziti pki create for backdated test CAs
- replaces the test PKI with a SPIFFE-capable, ziti pki generated and managed
one and rewires the config sets
- tests the controller list across ott/ottca/updb/token, non-HA, and raft
* fix missing wildcard cert from new PKI
* go mod tidy
- adds a striped id locker in common/concurrency for sharded per-entity locking
- switches the link manager to striped per-link locking to reduce contention
- adds a cache-before-transaction read path to the router manager
- adds a RouterReportedLink regression test covering same-link serialization and stale/same-iteration no-op behavior
Bumps the sdk-golang dependency from v1 to the v2 module
(`github.com/openziti/sdk-golang/v2` at v2.0.0-pre1) and updates all
import paths. This is a no-behavior-change precursor that isolates the
dependency migration from the Connect-V2 feature work in #3884.
- Rewrites `github.com/openziti/sdk-golang/...` imports to
`github.com/openziti/sdk-golang/v2/...` across the main and zititest
modules.
- Pins both modules to `github.com/openziti/sdk-golang/v2 v2.0.0-pre1`.
- Adapts `edgeXgressConn.AcceptMessage` to the v2 `MsgSink` signature,
which now takes an `edge.SdkChannel` argument.
- Replaces the removed `edge.Conn.GetRouterId()` with
`RemoteAddr().String()` in the loop4 traffic-test logging.
For openziti/sdk-golang#936.
- moves the channel dependency to channel/v5 v5.0.10 and sdk-golang to v1.9.0 in the root and zititest modules
- mechanically rewrites every channel/v4 import path to channel/v5
This is the import-path-only step; the API-level changes the switch requires land in the following commit. This commit does not build on its own.
- collapses the edge service boltz child store into the unified service store:
EdgeService becomes a type alias for Service, and RoleAttributes, Configs, and
EncryptionRequired become top-level Service fields
- adds an IsFabricOnly discriminator separating pure-fabric ("management")
services from edge services
- adds a forward-only, irreversible migration that moves edge service data out of
the edge child bucket, classifies pre-existing services (fabric -> IsFabricOnly
with encryptionRequired=false; edge -> IsFabricOnly=false), deletes the old edge
bucket and its index, and rebuilds the role-attributes index via CheckIntegrity
- guards every edge-facing path on the unified store against fabric-only services
so they stay invisible to the edge API: by-id Read/ReadByName/Delete, list/query
and the detail lister, ReadForIdentity, identity service-config overrides,
service policy / SERP @id and #all/role denormalization, the association-list
routes (terminators, policies, configs), and the router data-model sync
- rejects edge-surface updates of fabric-only services, and preserves edge fields
on fabric-surface updates of edge services
- marks the fabric-only guards TEMPORARY(fabric-edge-collapse): they exist only to
keep seldom-used management services hidden, and are removed when the fabric/edge
distinction is erased
- adds in-CI migration tests (forward plus a round-trip asserting FK sub-bucket
relocation and forward/reverse refcount preservation), durable fabric-only
behavior tests, and store-level policy-exclusion tests
- adds one-time authentic-migration gate tooling under zititest/migration-test
(create-model.sh, query, verify) and its operator runbook
The router's CheckConnections reaper enforced only JWT expiry, so a revoked OIDC
api-session, or a disabled/deleted identity, kept its live circuits and hosted
terminators until the access token expired. The router now enforces the
RouterDataModel revocations directly, tightening access-loss propagation to the
reaper interval.
- adds a Type field to DataState_Revocation and the raft Revocation command
proto, mirroring rest_model.RevocationTypeEnum (API_SESSION/IDENTITY/JTI) so the
management API, OIDC producers, sync, and router enforcement share one
vocabulary; the common.RevocationType* constants are compile-time bound to the
enum to prevent drift
- adds an IssuedBefore cutoff so an identity revocation invalidates only sessions
issued before it; a session re-authenticated after the cutoff survives the
still-lingering revocation. Persists IssuedBefore on the db and model Revocation
and carries it (plus the Type) through the single and batched raft marshalling
- adds RouterDataModel.IsApiSessionRevoked and IsIdentityRevoked and enforces both
in CheckConnections, closing a revoked session's connections
- revokes a deleted or disabled identity's live OIDC sessions via an
IdentityRevocationConstraint in the db package, run as a store pre-commit
constraint so the revocation is written in the same transaction as the identity
change and cannot be skipped (self-contained OIDC JWTs aren't otherwise
reachable). NewIdentityManager installs it with the revocation type and lifetime
- has the OIDC end-session (TerminateSessionFromRequest) revoke the specific
api-session named by the z_asid claim, with an identity-scoped fallback; sets
IssuedBefore on the identity fallback and the management revocation API; adds
RevocationManager.CreateOrReplace, routed through by both the OIDC paths and the
management revocation API, so a repeat logout/termination/revocation refreshes
the cutoff rather than colliding on the reused id. Expiry derives from the
longest configured token duration via a shared common.MaxTokenDuration helper
- adds tests/revocation_enforcement_oidc_test.go covering api-session revocation
(and a fresh session staying unaffected), identity disable and delete, and the
identity cutoff (a post-cutoff session surviving the lingering revocation)
* fixes#3952 reject invalid externalIdClaim and stop enrollment panic
- moves the error check before the locator assignment in Ca.GetExternalId
so an unsupported matcher/parser no longer nil-derefs and returns HTTP 500
- guards the claim index against negative and out-of-range values and errors
when a matched claim resolves to an empty string, so an empty externalId is
never silently accepted nor mapped to an identity
- adds validateExternalIdClaim, rejecting unsupported locations, unsupported
matcher/parser combos, missing matcher/parser criteria, and negative indexes
with HTTP 400 at CA create and update time
- applies the validation to both CA create and update, which previously had
no externalIdClaim validation on the update path
- tests GetExternalId across the matcher/parser matrix for no-panic and
correct error/value behavior
- tests CA create and update reject invalid externalIdClaim configurations
- tests that an externalIdClaim resolving to empty fails enrollment cleanly
* address pr review
- treats an externalIdClaim with no location as no claim in GetExternalId so an
empty stored claim (e.g. the empty bucket older CLIs leave behind) falls back to
fingerprint enrollment instead of erroring
- validates the merged result on CA update rather than the raw request: a partial
patch overlays only its supplied subfields onto the stored claim, an empty {}
object preserves the stored claim, and a full replace validates as-is
- skips externalIdClaim validation on update when no claim is supplied, fixing a
spurious rejection of updates to CAs that have no claim
- reports the offending location/matcher/parser in the externalIdClaim validation
error instead of the raw struct
- adds a --clear-external-id-claim flag to update ca and only sends the claim object
when a claim flag changed, so existing updates preserve the claim and clearing is
explicit
- tests externalIdClaim patch merge, preserve, and clear behavior, and that an
unconfigured claim reads as no claim
- adds management API endpoints listing role-attribute usage for identities,
edge routers, services, and posture checks, reporting per-source counts
(and optionally ids via withIds) across home-entity collections and the
policies that reference each attribute
- adds RoleAttributeUsage model with QueryRoleAttributeUsage, mapping each
RoleAttributeKind to its contributing sources, with all reads in a single
transaction so counts stay consistent with the attribute list; returns an
empty result (not a panic) for kinds with no attributes and an error for
unknown kinds
- adds boltz.SetIndexValueTransform and AddSetIndexWithTransform, letting a
SetIndex filter and rewrite symbol values without persisting a derived field
- adds derived role-attribute set indexes to the service policy, edge router
policy, and service edge router policy stores, built from existing role
fields via a role-attribute-only transform that excludes the #all wildcard
- adds a db migration that backfills the new indexes via each index's own
CheckIntegrity(fix=true), scoped to the role-attribute indexes only, with a
per-index summary log, stopping at the first error
- tests the new endpoints via the typed edge-api management client
* fixes#3809 support CSR submission during OIDC authentication
- accepts an optional CSR during OIDC login (all auth methods) and
signs it into a session-bound certificate with a SPIFFE ID derived
from the identity and API session
- returns the signed certificate PEM as a top-level "session_cert"
field in the token endpoint JSON response (CodeExchange, RefreshToken,
TokenExchange)
- adds cert-binding verification on RefreshToken and TokenExchange:
if z_cfs is present the peer cert fingerprint must match (strict),
otherwise falls back to SPIFFE ID verification
- supports cert rotation via csr_pem form parameter on refresh and
token exchange; replaces the session cert fingerprint while
preserving the authenticating cert fingerprint
- adds AuthCertFingerprints (z_acfs) claim to track permanent auth
cert fingerprints separately from rotatable session cert fingerprints
- only the leaf certificate fingerprint is added to z_cfs and z_acfs,
intermediates are never included
- invalid CSR returns 400 Bad Request in OIDC error format
- propagates updated CustomClaims (including CertFingerprints) from
access token to renewed refresh token so rotated fingerprints are
enforced on subsequent refreshes
- adds CertGenerated field to ApiSessionEvent
- advertises OIDC_AUTH_WITH_CSR controller capability when OIDC is
enabled
- adds unit tests for verifyCertBinding (fingerprint, SPIFFE ID,
edge cases) and CsrPem field parsing
- adds integration tests for initial CSR auth (updb, cert, ext-jwt),
cert-binding on refresh/exchange, CSR rotation with cert auth,
SPIFFE fallback and z_cfs transition, and CSR property forging
resistance
* address pr concerns
* add z_cfs len tests on junk chain certs
* strip csr subject info, replace w/ santized values
* fix csr rotation rejection/paths during token exchange/refresh
* fixes#3734 enforce client certificate proof-of-possession for OIDC sessions
- adds verifyCertProofOfPossession() in resolveOidcSession() to require
TLS client cert matching a z_cfs fingerprint or SPIFFE ID when the
OIDC token was issued with cert bindings
- adds z_cae (CertAllowExpired) claim from auth policy, propagated
through token issuance and refresh
- adds TrustCache.VerifyClientCert() with tiered pool matching
(first-party roots, trust anchors, third-party) and TTL cache
- adds WrapIdentityWithCertValidation() on the router to verify client
certs at the TLS level against RDM PublicKeys
- adds IsFirstPartyCert() on the router to gate SPIFFE ID matching by
checking whether the cert chains to the controller root CA
- adds VerifySpiffeId() in common/spiffehlp with SpiffeMatchApiSession,
SpiffeMatchIdentity, and SpiffeMatchNone return types
- adds SPIFFE IDs to OTT and token enrollment certs (/identity/<id>)
- enforces cert expiry by match type: API session certs must be valid,
fingerprint-matched certs respect z_cae, legacy sessions skip checks
- shallow-copies leaf certs before overriding time fields in all cert
verification paths to avoid races on shared x509.Certificate pointers
- fixes controllerRootCache setting inited=true before the ctrl channel
is available, which permanently cached the failure
- registers the service policy type symbol as a string, so queries match the
API's "Dial" and "Bind" names rather than the internal int32 ids
- adds a symbol mapper that converts the stored int32 to its PolicyType name
at query eval time
- updates the posture-checks lookup in EdgeServiceManager, the only direct
caller that went through GetSymbol, to read the mapped string form
- adds a store test covering type = "Dial" and type = "Bind" filtering
- notes the breaking removal of the undocumented type = 1/type = 2 form in
the 2.0 deprecation cleanup list
* fixes#3680 add revocation management API, CLI, and enforcement
- adds Management API endpoints for revocations (POST, GET, LIST) with
type-aware validation (JTI/API_SESSION require UUID, IDENTITY requires
existing identity)
- adds CLI commands: ziti edge create revocation identity|api-session|jti
- adds revocation checks to resolveOidcSession in security_ctx.go so the
REST API returns 401 for revoked OIDC tokens. Previously only
ValidateAccessToken (router ctrl channel path) checked revocations,
so revoked tokens still received 200 OK from the management and client
HTTP APIs
- adds api-session revocation check to ValidateAccessToken, which only
checked JTI and identity revocations
- adds Type field to Revocation model, store, and protobuf message
- adds integration tests covering CRUD, input validation, and token
enforcement for all three revocation types
- use release edge-api@v0.28.1
- adds CreateCircuitV3 message type and handler for routers that have
already authorized dials locally via RDM, bypassing service session
tokens in favor of identity ID, service ID, and pre-assigned circuit ID
- renames CreateCircuitRequest/Response to CreateCircuitV2Request/V2Response
for clarity now that V3 exists
- adds CircuitManager.Reserve to atomically claim circuit IDs before routing,
preventing collisions on pre-assigned IDs
- extends CreateCircuitParams with GetCircuitId so V3 can supply a
pre-assigned circuit ID (falls back to UUID generation when empty)
- fixes IsDialableByIdentity which was incorrectly calling IsBindableByIdentity
- extracts V2 handler into its own file create_circuit_v2.go
- adds CreateCircuitV3RequestType/ResponseType (20222/20223) to edge_ctrl protobuf
- registers V3 handler in controller server
- adds DeleteRevocationsBatchCommand so expired-revocation cleanup goes
through raft as a single log entry per batch
- adds CreateRevocationsBatchCommand for batched revocation creation
through raft
- moves refresh-token revocations from synchronous inline creation to a
background batcher that flushes on a configurable interval, removing
the database and raft as a bottleneck on token refreshes
- skips revocation creation for tokens expiring within a configurable
threshold (revocationMinTokenLifetime), since they become invalid on
their own
- validates that revocationMinTokenLifetime is less than 50% of the
configured refresh token lifetime
- makes the revocation enforcer frequency configurable and restricts it
to run only on the raft leader
- adds tests for multi-batch delete, batched create with router RDM
propagation, and skip-threshold behavior
- adds new configuration tunables under edge.oidc: revocationBucketInterval,
revocationMinTokenLifetime, revocationBucketMaxSize, revocationMaxQueued,
revocationEnforcerFrequency
- fixes RevokeToken double-write: adds return after JWTID-keyed revocation
save, preventing fallthrough write with raw JWT string as unreachable key
- fixes TerminateSession key mismatch: stores revocation by identityId alone,
matching the Subject-based lookup in ValidateAccessToken
- fixes RevocationDelete sync action: passes DataState_Delete instead of
DataState_Create so routers evict the entry from their data model
- adds RevocationManager.DeleteExpired: batch-deletes expired revocations in
batches of 500 until none remain
- adds RevocationEnforcer: periodic policy runner (every 1 minute) that calls
DeleteExpired and records metrics
before test fixes
- removes legacy v1 terminator code path; all terminators now use v2 flow
- refactors edgeTerminator.close() to decouple SDK notification from control plane notification
- adds pending SDK close notification queue with retry when channel is busy
- adds post-create inspect mechanism that verifies SDK still holds the bind after terminator creation
- queues second post-create inspect when establishment takes >30s to catch SDK timeout races
- detects and discards stale reordered binds on the same connection by comparing connIds
- re-establishes replacement terminators when a delete/create race is detected
- eliminates IsEntityPresent pre-filter in removeTerminatorsHandler to prevent raft ordering races
- fixes ValidateTerminators to query identities from the correct manager with the correct filter field
- adds postCreate flag to ValidateTerminatorsV2Request so routers skip redundant SDK inspect
- returns retry-later (nil result) from router validation when inspect is temporarily unavailable
- blocks SyncAllSubscribers until completion and guards RouterDataModel replacement with in-progress flag
- fixes InheritLocalData to enable service access tracking for all subscribed identities
- adds Services.Has check in GetServiceAccessPolicies to prevent false policy grants
- validates policy-to-identity associations in ValidateServicePolicies
- adds `ziti agent tunnel dump-sdk` command for SDK context inspection via IPC agent
- adds `ziti fabric inspect sdk` command to query SDK context through routers
- fixes --expected-per-host CLI flag binding in validate terminators command
- changes bind-access-lost retry hint from NotRetriable to RetryStartOver
- moves trace route response and xgress close handling off channel handler goroutine
- fixes listTerminators test helper to URL-encode filter parameter
- improves sdk-hosting-test validation resilience with login and query retries
- adds terminator_create_flow.md documenting the full lifecycle across SDK, router, and controller
- adds detailed logging for data model sync, service access tracking, and subscriber change detection
Paul Lorenz