* The controller and the router wrote the complete service session token, an OIDC
service session JWT or a legacy session token, into structured log fields during
edge control operations. Several of the sites logged at Error or Info, so the
values reached logs at default verbosity during ordinary operation.
* Adds LoggingFields to the session and api session types in common, controller/model
and router/state, so a logging site asks for the session fields or the api session
fields instead of picking values by hand. Ids, type, identity, service and the
token's validity window go in; the token does not. Empty values are dropped, so the
sets merge over each other.
* AddLoggingFields in router/state now delegates to LoggingFields. Its output moves
from a nested field to flat prefixed keys.
* The edge control handlers log the session field set on the shared returnError and
logResult paths, on circuit setup and on terminator update. The terminator removal
paths log the terminator id, since they never load a session.
* Adds unit tests over returnError and logResult that fail if a session token reaches
a log entry, covering the loaded-session, failed-to-load and JWT-claims cases.
- constrains the cpu profiler and router signal monitor to `unix` instead of a list of operating
systems, and turns their windows stubs into `!unix` fallbacks
- replaces the per-OS tproxy and geneve listener stubs with one fallback each, covering every
operating system without an implementation
- replaces survey with local stdin prompts for the pki pickers, since survey's terminal package does
not build on solaris or illumos; an exact name takes precedence over a list number
- limits the interactive database explorer to the platforms its prompt library builds on, and
reports it as unsupported elsewhere
- removes the survey dependency and syncs zititest's go.mod
- tests the pickers
- notes the fix in the changelog
- collects the subscribed identity ids under IterCb when syncing all subscribers and checks each one
after the iteration returns, rather than inside the callback; a deleted or disabled identity closes
its SDK channel, whose close handler unsubscribes through RemoveCb on the same key and
self-deadlocked on the shard lock
- clears each identity's update flag before looking up its live subscription, so a subscription
replaced and updated mid-sync is checked instead of the one collected
- clears the update flag before the subscription lookup in syncSubscriptionIfRequired, for the same
reason
- tests that a subscriber unsubscribing from inside its deleted notification lets the sync complete
- tests that a subscription replaced and updated during the sync delivers the update to its new
listener
- notes the fix in the changelog
- adds RemoveTerminatorsV2, an asynchronous control-channel removal, so a router
no longer holds a request slot open waiting for a synchronous reply; gated on a
new capability so routers fall back to the synchronous path against controllers
that do not advertise it
- paces establish and remove retries with a per-terminator exponential backoff
after a controller rejection, so a rejected operation is not re-sent on the next
event-loop tick, which is what let a rate-limited removal sustain itself
- routes v2 removes to the leader and skips the ordered delete for a terminator the
router confirms it created that no longer exists, so a storm of no-op retries
cannot consume the command rate limiter
- gates that skip on the leader having applied everything committed when it took
leadership, since a leader reports itself elected before those applies finish and
would otherwise read a committed create as absent and drop its delete
- guards the leader term and every write to the caught-up flag with a dedicated
lock, so a barrier completing alongside a leadership change cannot record a result
for a term that has already ended
- scopes v2 removals to the requesting router, so a router can only remove
terminators it owns on the asynchronous path as well as the synchronous one
- tracks in-flight removal batches on the router, resolving or requeueing them when
the response arrives; lost responses self-heal via rate-limiter expiry and the
retry scan
- waits for an outstanding operation before sending a terminator delete, and checks
that before dequeuing, so a delete neither races a create the controller has
committed but not applied nor falls out of the queue until the retry scan finds it
minutes later
- matches a create response to the attempt that produced it, by control channel and
reply sequence, since attempts overlap once one overruns the establishment timeout
and every attempt carries the same terminator id; only the outstanding attempt's
response may confirm the create or report the terminator idle
- records the attempt as the sender assigns its sequence, so a response cannot be
processed before the request it answers is known, and withdraws an attempt whose
send never queued
- reports a removal that succeeds past the establishment timeout as congestion
rather than success, matching how the establish path classifies its own latency,
since both resolve into the same limiter
- extracts the transient-error test both remove handlers share, so the v2 path
treats a leaderless cluster as busy like v1 does instead of as a permanent
failure, and the two cannot drift again
- applies the same delete sequencing and response correlation to the router-hosted
(ER/T) registry, which cleared the same in-flight marker unguarded and dropped a
deferred delete from its queue rather than leaving it there
- bounds the v2 response send with a timeout, so a router that stops draining its
send queue cannot accumulate blocked responder goroutines on the controller
- applies the router.link.v1 heartbeat settings (sendInterval, checkInterval,
closeUnresponsiveTimeout) and payload/ack sender queue sizes at runtime,
replacing the hardcoded 10s/1s link heartbeat intervals and the ignored fields;
new links take them at bind time, established links are re-tuned in place
through the channel HeartbeatControl, and queue sizes reach new links only
- publishes heartbeat settings as an immutable generation in one atomic swap, so
no reader pairs one generation's interval with another's timeout, and has each
link enforce the timeout belonging to its own intervals
- pushes a generation and its intervals as one step under the link's lock and
ignores a generation at or below the one applied, so an overtaken update cannot
roll a link back; reconciles every channel of a link, including a split link
whose channels bind on either side of a config change, to the newest generation
- adopts the first registered settings explicitly, since a router with no managed
link config publishes its defaults at generation 0
- re-arms a link's liveness deadline when its generation changes, so age
accumulated under the previous intervals cannot close a healthy link when a
timeout is shortened
- honors a closeUnresponsiveTimeout below 30s, which previously sat behind the
unhealthy threshold; treats a zero timeout as closing nothing, and reports a
closing link unhealthy
- scales the unhealthy threshold with the intervals, to max(30s, 2 x (send +
check)): reporting unhealthy poisons the latency metric the controller routes
on, so a fixed 30s priced healthy links with a long send interval out of
routing every cycle
- validates heartbeat timing on the controller and the router: durations must be
positive, their sum representable, and the timeout at least 10s and greater
than sendInterval + checkInterval; a timeout under two heartbeat cycles is
accepted with a warning
- adds a per-config-type semantic validator registry to ConfigManager, run after
schema validation, and moves the router.link config definition to
common/config/routerlink so the controller applies the same rules
- treats a link section in router YAML, of any content, as local config that
takes precedence over the controller's for the whole router.link type, since
the loader defaults every field and so now records presence separately; warns
at startup when that section has no listeners or dialers
- translates local heartbeat timing leniently, since a local failure stops the
router from starting: keeps an explicit zero, raises a timeout below the floor
to it, and falls back to the defaults for other invalid timing, each with a
warning
- stamps UpdateLinkListeners with the config generation it came from and has the
controller drop superseded sets, which a multi-underlay control channel can
deliver out of order and which would otherwise send peers to withdrawn
addresses
- logs every successful link config apply, including those that dispatch no work,
and pins that a queue-size-only change triggers no GC pass
- deprecates split links, documents the feature in the changelog and the router
config design doc, and brings both up to date
Builds on the channel/v5 HeartbeatControl from openziti/channel#265.
- adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId,
ListenerBinding}) used by all staleness and GC checks, replacing the
pre-rendered key string each side used to pass around
- adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links`
CLI for operator-driven, two-sided staleness verification with optional --gc,
which acts only when both endpoints agree
- implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed):
the router walks its xlinks after each Apply that mutates listeners, dialers,
or gcMode and closes one-sided-stale entries under the configured mode
- sweeps a removal under orphaned whatever the removed config said, including
preserve, since a removal leaves no listeners and no dialers and every
remaining link is one the router can never re-establish
- reports staleness as a three-state verdict (unknown/notStale/stale) so a side
that cannot judge a link abstains instead of guessing; a destination absent
from the registry's listener snapshot means "listeners unknown", not "peer has
none", which keeps a briefly unhealthy peer from losing its live links
- rebuilds the link surface only when a listener or dialer definition actually
moves, so a gcMode-only change no longer rebinds every listen socket
- gates CheckStaleLinks on a new RouterStaleLinkCheck capability and explains
each unqueryable endpoint, including one left unasked by the filter, so a
partial verdict says which of the three causes applies
- detects a peer re-advertising the address a link was dialed to from the
dialing side, not just from the peer's own listener side
- pins every verdict to the link iteration it was computed against, faults that
iteration rather than the current one, and takes the conditional removal's
result as the decision, so a link re-dialed while the sweep was running is
neither closed nor reported as collected
- resolves which side a report speaks for from the sending router against the
controller's link map rather than from the side the report names, so one
router cannot fill both sides and satisfy the both-agree rule --gc depends on
- bounds the query fan-out with a semaphore, matching the other validation
sweeps, since each query carries a minute-long reply timeout
- fails the CLI when the result stream ends before every link is reported, and
drains results already buffered before declaring any missing
- rejects an unrecognized match mode rather than resolving it to changed, the
broadest removal criterion
- consolidates the controller-managed router configuration changelog into one
section, replacing the status that said routers do not yet consume configs at
runtime, and documents the per-config-type opt-in
- restores the Current Beta Features section, dropped when the 2.0 changelog was
archived, and lists controller-managed router configuration in it
- adds unparseable to the codespell ignore list, a spelling variant the
dictionary flags as a typo
For #3743.
A set guarded by a single RWMutex, for sets small enough or written rarely enough
that one lock is not a bottleneck. The alternatives both cost more in that case: a
sharded map allocates a map and a mutex per shard for every set that exists, and a
copy-on-write set copies every entry per write.
- adds Add, Remove, Contains, and Values, with Values returning a snapshot the
caller owns
- adds RemoveIf, which removes only if the value is present and a predicate agrees.
The predicate runs under the write lock, so a decision that depends on state
outside the set cannot be invalidated between the check and the removal
- resolves zt-session tokens eagerly when the request context is created, so every request carrying one marks activity and gets session lifetime headers, regardless of endpoint
- marks last activity in SecurityCtx.resolveZtSession once the session is loaded
- adds SecurityTokenCtx.HasZtSessionHeader, a header-only check that does not parse bearer tokens
- emits only session lifetime headers from the API wrappers and drops the unused session-error header branch, so a stale token on an anonymous endpoint stays silent as it did on 1.6.x
- tests that requests to authenticated and anonymous endpoints with a zt-session mark activity, carry lifetime headers, and are reflected by current-api-session
* fixesopenziti/ziti#4321 compare process hashes and fingerprints case-insensitively
- normalizes configured hashes and signer fingerprints to lowercase unseparated hex in the
PROCESS and PROCESS_MULTI stores, on load as well as on save, so values stored before this
change read back normalized without a migration
- replaces the PROCESS store's lowercase-only pass, PROCESS_MULTI stored its values verbatim
- adds store tests for both types covering configured values and values written straight to bolt
- updates the process multi patch round-trip test, the management API now returns normalized
values rather than the submitted strings
- removes the trailing newline from the signer fingerprint literal in the process and
process-multi OIDC posture tests
- normalizes reported MAC addresses in Instance.Apply to lowercase unseparated hex,
the form MAC posture check values are persisted in
- stores a new PostureResponse_Macs rather than mutating the incoming protobuf message
- compares the normalized addresses when deciding whether posture data changed, so a
re-reported separated address no longer reads as an update
- moves the normalizer to a new common/posture package, shared by controller/db and
router/posture, and hoists its regexp out of the per-call path
- adds an oidc-auth-test fablab model that exercises OIDC authentication end to end, with an event-forwarder component, an oidc-test-client, a ziti-prox-c component, and OIDC event/gossip/traffic validations
- moves zitirest out of the public API into separate test-only shims: an integration-test shim (tests/restclient.go) and a fablab shim (zititest/zitirest), and repoints upgrade-test at the fablab shim
- adds region-isolation network partitioning and packet capture on all interfaces to the chaos toolkit
- queues oidc-test-client results reporting on a single sender goroutine with a bounded write deadline, so a stalled results circuit can't block the traffic loops
- counts results events dropped when that queue fills and reports them to the collector as errors, so a reporting outage fails validation instead of looking like a clean run
- bounds the event forwarder's event and keepalive writes, and checks for shutdown on every send attempt, so an unreadable destination can't park the forward loop holding its lock
- logs read errors on collector connections instead of ending collection for that client silently
- bounds the oidc-auth-test debug server's header and body reads
- matches ziti-prox-c process matching to the version-agnostic binary name, so a version change still finds the running process
- points upgrade-test's per-service terminator count at the shared validations helper
- adds a fablab design summary doc
* fixesopenziti/ziti#4094 accept first-party certs issued by a separate edge signing CA
- adds FirstPartyX509CertValidation and ThirdPartyX509CertValidation usages and an
intermediates field to the router data model public keys, deprecating
ClientX509CertValidation
- publishes config CA bundle roots as first-party anchors with their intermediates and
Ca store entries as third-party anchors; controller certs carry JWT validation only,
since a controller identity is never a CA and anchors no client cert chains
- builds router first-party and client cert trust pools from the published usages,
falling back to the deprecated usage against older controllers
- propagates the full signing cert chain between controllers via a new mesh
SigningCertChainHeader and persists whole chains in controller records
- removes the orphaned InstantStrategy.AddPublicKey, dead since public key sync moved
to controller list data
- gives each command dispatcher its own decoder registry so multiple in-process
controllers no longer decode into the last-started controller's managers
- adds a three-controller in-process HA test harness with a split signing PKI,
cluster formation and first-party cert integration tests
- trusts the edge signing CA when verifying router control channel certs
- adds a variadic additionalRoots parameter to VerifyLeafCertChain, applied to a
clone of the caller's pool so an identity's live tls.Configs are unaffected
- passes the edge enrollment signing CA bundle as additional roots when admitting
a router control channel connection, so a deployment whose signing CA sits
outside the controller's own trust bundle no longer has every router refused
- leaves the fingerprint check bound to the verified leaf, so the wider anchor set
changes which chains verify, not which routers are admitted
- covers the split-root case and the caller-pool guarantee in common/cert tests
- adds a package-level buildFlags string the linker sets at build time, parsed
into a capped list of [A-Z0-9_] names with blanks, duplicates, and malformed
tokens dropped
- returns those names in the new buildFlags field on /version, separate from
capabilities, and prints them under ziti version -v
- bumps edge-api to v0.36.0 for the buildFlags field
- requires an API session token on CreateCircuitV3 requests and validates it,
covering signature, audience, token type, and revocation by token id,
identity, and api session
- takes the dialing identity from the validated token claims rather than the
router-supplied identity id, and rejects a request whose asserted identity
does not match the token subject
- adds the api session id to the log context, matching the V1 and V2 paths
- adds tests for a missing token, an invalid token, and a token belonging to a
different identity than the one asserted
- notes the advisory in the 2.1.0 release notes
The controller decided which of two racing connections for a router was current
by comparing router instances, but loaded one per connect by evicting the router
cache and reading back through it. Two connects could both evict, and whichever
read second was handed the instance the first had just published. A shared
instance makes the two connections indistinguishable: the connect path cannot
reject the second into an occupied slot, and when either channel dies the
disconnect path finds itself current and tears down the registration the other is
still using. The surviving channel is never re-bound, so the router stays
connected at the transport layer while absent from the model, unable to recover.
Connect and disconnect were also unserialized, so a stale or superseded
disconnect could interleave with a live connection and take its links with it.
- serializes a router's connect and disconnect with a per-router striped lock
- keeps at most one connection per router: a connect into an occupied slot is
rejected via an error from ConnectRouter, so the bind fails and NewChannel
closes it without starting rx or registering it, and the occupant is displaced;
the router redials into the freed slot
- displaces an occupant by closing its channel and also invoking the teardown
directly, since a channel that is already closed never fires its close handler
again; without this a dead but still registered connection holds the slot
forever and every redial is rejected against a slot nothing can free
- refuses a connect whose control channel is already closed rather than
registering it, so a connection no disconnect could ever remove is never
published
- gives every connection its own router instance via RouterManager.NewCtrlChanRouter,
read through readUncached so the cache neither supplies nor receives it, which
is what makes comparing instances meaningful
- moves recording the channel and connect time out of the accept path, so a
caller cannot attach the wrong channel or forget to attach one
- serializes link publication with that teardown on the same per-router stripe.
Validating currency and then publishing without it is a check-then-act: a
report can find the connection current and, by the time it reaches the link
manager, the teardown has already snapshotted and cleared the router's links,
so the link is recreated after everything that would have removed it. It is
then absent from the router's own index while still in the link table with a
disconnected source, and a reconnect reporting the same iteration can adopt
that stale source instead of rebuilding the link
- guards the entire DisconnectRouter teardown by connection currency, all or
nothing, and clears the connected flag and link index only when the
registration was actually given up, with the flag cleared under the same shard
lock as the map removal so the two cannot be observed disagreeing; the connected flag decides whether the
controller accepts a router's link reports, so clearing it for the wrong
connection silences a router that is up and reporting
- reduces MarkConnected to publishing the connection; the takeover-close moves
into ConnectRouter's reject path
- makes the per-router unlock idempotent so callers can defer it as a leak-safety
net and still unlock early before closing a channel outside the lock
- stops the replaced RouterSender in routerTxMap.Add so a takeover does not leak
the old sender's goroutine when the broker's asynchronous RouterDisconnected
loses the race to the redial's RouterConnected
- discards pending peer state changes for a router whose channel has closed,
since sending on one fails immediately and the failed send is retried as soon
as the event loop turns, spinning the loop and flooding the log
- queues the peer-state send-done event on every path, so a missing channel can
no longer leave sendInProgress set and stall that router's updates permanently
- resolves a router's version from its connected instance when validating link
conn info, since the version arrives in the hello and so is absent from an
instance loaded from the database
- normalizes both endpoints to the connected instance in shortestPath, which is
keyed and compared by pointer and so treated an endpoint held as any other
instance of the same router as absent from the graph, reporting a router as
unroutable from itself. That worked before only because the connect path
published its instance into the router cache, so a cache read and the connected
map returned the same object; nothing stated the requirement
- configures test logging once per package in TestMain, so a test no longer
writes global logger state while a previous test's shutdown logging reads it
- adds a link ConfigHandler (router/link FactoryRegistry) that applies router.link.v1 config: Apply rebuilds the listener/dialer set wholesale, and established Xlinks survive because Listener.Close() only closes the accept loop
- translates local link: YAML into router.link.v1 JSON and pushes it through the managed-config registry at startup
- adds the UpdateLinkListeners ctrl message so the router republishes its listener set to the controller on change; the controller re-fans via the existing PeerStateChange path
- re-evaluates dialers on link group and listener changes via RescanForDialOpportunities
For #3743.
- renames the FormatPretty --log-formatter value from "pfxlog" to
"pretty"; the old name described its historical origin rather than its
behavior, and pfxlog is being removed
- accepts "pfxlog" as a deprecated, unadvertised alias for "pretty" via
FormatPrettyAlias so existing configs and scripts keep working
- advertises "pretty" in the --log-formatter help text across the run,
tunnel, controller, router, and demo commands
- adds the "pretty" case alongside "pfxlog" to the legacy pfxlog
formatter switches so --log-formatter accepts the same values on every
command
- tests that the "pfxlog" alias still resolves to pretty output
- points slog.Default() at the async slog sink in Install, gated at the
live global level, so un-named slog output and libraries that log
through slog.Default() land in ziti's configured handler instead of
slog's built-in stderr text handler
- wires it through the registry's HandlerFor under the name "default",
which binds no name attr so records already carrying a channel attr
are not double-tagged
- adds DefaultLoggerName so the gating name is discoverable and
operator-targetable via set-channel-log-level
- tests that after Install, slog.Default() reaches the sink and gates
Debug below the global level
* fixesopenziti/ziti#4118 disambiguate overlapping ext-jwt-signer kids by issuer
- binds external JWT tokens to signers by exact issuer claim rather than by key ID, so signers drawing from a shared signing-key pool resolve deterministically
- binds controller-issued tokens by key ID first, preserving controller token resolution and preventing an external signer configured with a controller's issuer from capturing controller access tokens
- removes the external key-ID fallback so a token whose issuer matches no configured signer is not bound to an unrelated signer that happens to share its kid
- adds GetControllerIssuerByKid to the TokenIssuerCache interface and implementation
- skips disabled external signers in GetIssuerByKid so a disabled signer sharing a kid cannot poison resolution for an enabled one
- adds an integration test with two HTTPS JWKS providers sharing a key and kid, covering the enabled-collision and disabled-poison cases
- documents that controller issuers are keyed by controller id and that a controller issuer's key ID is the fingerprint of its TLS certificate
- documents that an external kid match is ambiguous because signers can share a signing-key pool, and that a definitive binding requires resolving by issuer claim
- clarifies the overlapping-kid test comment covering why issuer-claim binding is required when a disabled signer shares a kid
- removes the costTags option from router link listeners, which was parsed
from config, advertised to the controller, and stored on the router model
but never used for path selection or any other behavior
- drops GetLinkCostTags from the xlink.Listener interface and its transport
implementation
- reserves the corresponding ctrl_pb.Listener.costTags and
RouterLinks.RouterLink.linkCostTags protobuf fields and regenerates ctrl.pb.go
- documents the removal in the changelog
- reports edge router policy denials with an access-denied error naming the missing policy,
replacing the session error reused on the sessionless ER/T and create-circuit-v3 paths
- adds EdgeRouterManager.GetEdgeRouterAccess, which reports which of the two required policy
links (identity-to-edge-router, service-to-edge-router) is absent, and removes the boolean
IsAccessToEdgeRouterAllowed it replaces
- logs the controller's rejection on the router at warn level, since it is recoverable and
retried by the periodic scan; the router previously discarded the error code and message
- delays a new terminator's first create attempt by a fixed 2s so config applied in quick
succession settles before the router asks, avoiding a 2-3 minute wait for the retry scan; the
delay is a deliberate stopgap until edge router policy visibility lands in the router data model
- propagates the controller's error code and retry hint to SDK clients on the dial paths, which
dropped the code and left every refusal classified as unknown
- adds the retry hint header to controller error replies, grouped with the other error-reply
headers rather than the create-circuit-v3 request headers
- notes that the sync strategy headers alias the edge namespace's 1013-1015 ids and stay
disjoint only by message content type
- tests the per-policy denial reporting, the error code carried with and without a retry hint,
the controller-to-SDK error code mapping at both dial relay sites, and the terminator settle
gate
- waits for terminator establishment in the tunneler dataflow tests instead of a fixed sleep, so
they no longer race the settle delay
- restores the tproxy multiple-lanIf and multiple-resolver changelog entries with keep markers,
which regeneration drops because their commits reference pull requests rather than issues
- adds logging.ResolveFormat, which picks the effective --log-formatter
when the flag is unset: ZITI_LOG_NO_JSON (or the deprecated
PFXLOG_NO_JSON) forces pfxlog output, else a terminal stderr gets
pfxlog and a redirected stderr gets json, matching pfxlog's old
GlobalInit behavior
- keeps an explicit --log-formatter value authoritative over the env and
tty defaults
- wires ResolveFormat into the ziti run and ziti tunnel logging setup and
documents the default in the --log-formatter help text
- tests the precedence rules, both env var names, and the pfxlog-style
boolean parsing
Forward ports the GHSA-mrpr-756c-xm47 fix, released in 2.0.2, to main.
- adds a shared cert.VerifyLeafCertChain helper that verifies the presented leaf
(certs[0], whose private key the TLS handshake proved) against the node's full
trusted-CA pool, treating certs[1:] only as candidate intermediates
- uses it for controller cluster mesh peer connections and router link
connections, which previously accepted a connection when any presented
certificate chained to the trusted CA while taking peer identity from the leaf
- matches the pinned metrics scrape certificate against the presented leaf only,
comparing full DER rather than just the signature, and rejects a leaf outside
its validity window
- adds negative-path tests covering rogue leaves paired with CA-chained filler
certificates
- loadFromBolt now accepts a non-JWT (legacy/durable) service session token by
looking the session up via Session.ReadByToken and verifying it belongs to the
api session, so a legacy client's existing session keeps working across a
controller upgrade instead of failing as a malformed JWT
- classifies service-access token validation failures (malformed, expired, bad
claims, mismatched api session, revoked) as InvalidSession so the client
re-creates, while revocation-store/datastore read failures remain internalError
so a transient controller fault does not make clients discard valid sessions
- adds a typed common.InvalidTokenError so ValidateServiceAccessToken can
distinguish token-level failures from infrastructure failures
* fixesopenziti/ziti#3990 push service and posture state to subscribed SDKs
- pushes indexed atomic ServiceChangeSet envelopes to subscribed SDK connections: a full snapshot on subscribe, incremental service changes per RDM scan pass, posture check definition changes as their own entries, and identity-resolved config bodies, all serialized so envelopes hit the wire in index order
- pushes per-connection PostureStateChange state (monotonic seq, resync on request) for posture pass/fail, including flips caused by definition edits that mutate no posture data
- registers pending RDM identity subscriptions for identities not yet synced to the router and sends an authoritative full sync plus full posture state when the identity arrives; an active push subscription pins the connection's RDM listener
- advertises service subscriptions and router data model support on the control-channel capability bitmask; the controller persists each router's capabilities mask and version on the EdgeRouter entity via raft and renders them on the edge APIs, so SDKs can select capable routers before connecting
- submits posture per router and corrects MFA posture semantics: pushed expiry is the earliest of timeout and pending wake/unlock grace deadlines, wake/unlock re-pass satisfies the re-prompt, api session tokens whose amr attests TOTP seed the MFA baseline from auth_time only (never iat), and token exchange carries the subject token's auth_time
- sends structured denials on dial and bind refusals: posture failures carry the failing check ids, no-policy denials are access denied, unknown services are invalid service, and session token failures are invalid session; the denial's cause no longer rides the wire as an unserializable error
- hard-closes accepted SDK connections on edge listener shutdown so clients observe a router going away immediately
- adds integration coverage: subscription snapshots and change delivery, poll and push reconciliation as capable routers come and go, posture state and definition-change push, router views over the public SDK API, typed dial errors, MFA baseline seeding, and OIDC token-exchange auth_time preservation
- removed RDM capability from SDK, router/controller only
- adds a clusterId field to SyncSnapshotCommand and writes it into the database
after the migration snapshot restore, so a controller bootstrapped by
migrating a database ends up with a durable cluster id instead of an empty one
- the snapshot restore replaces the whole FSM database with the migration
source, which carries no cluster id, so without this the id written during
bootstrap was silently wiped and the node came up with an empty, non-durable
cluster id, defeating the mesh cluster-id validation
- persists the raft index after the cluster id in RestoreSnapshot so the index
remains the completion gate: a failure before it halts (SyncSnapshotCommand is
a critical command) and replays/retries on restart rather than skipping the
command with a blank cluster id
- fails RaftRestoreFromBoltDb when the cluster id is blank after bootstrap
- regenerates cmd.pb.go for the new field
- keys router capability bits off the sdk-golang RouterCapability enum
- adds a generic capabilities.Mask[T ~int] bitmask, centralizing capability
set/check behind one value-to-bit translation
- adds capabilities.RouterCapability and ControllerCapability types, with
RouterCapabilityMask/ControllerCapabilityMask aliases, so masks and checks are
typed per namespace
- supports control-plane-only router capabilities as negative values that index
down from the top of the mask, collision-free with the SDK's upward-numbered
bits and invisible to the SDK and edge-api
- references the sdk-golang RouterCapability enum as the source of truth for
shared router capability bits
- holds the router's advertised capability mask as an instance on the router env
rather than a global, so in-process test routers do not share state
- advertises PostureChecks and BindSuccess as capability bits on both channels
while still sending the legacy boolean edge headers for backwards compatibility
- routes GetCapabilities/IsCapable and the controller's Router.Capabilities field
through the typed mask
- adds a provenance test that verifies, via go/packages, that every positive
router capability is SDK-sourced, every negative is control-plane-only, and no
two resolve to the same bit
- update sdk to v2.0.0-pre2
- adds a managed-config registry that tracks available router config per base type by source (controller/local) and version
- reconciles to an effective config with local-config-wins precedence at the base-type level, selecting the highest supported version
- adds a local config-type allow-list so operators bound which config types the controller may enable
- loads the managedConfig section from the router config file
- wires the RDM subscriber to feed router-target config events into the registry
- adds inspect support for the registry
The registry is not yet consumed by any subsystem; routing link management through it is a follow-up. For #3743.
- adds timestamppb.Timestamp to the revocations diff ignore list so
go-cmp does not panic on the unexported fields of the ExpiresAt and
issuedBefore timestamps when validating the router data model
- adds DataState_ServiceConfigs to the identity diff ignore list to
guard the same panic on identity service configs
- applies both ignore-list entries to the RouterDataModelSender diff,
which walks the same nested proto messages and panicked identically
- adds regression tests covering both nested proto-message fields on the
receiver and sender diff paths
- updates Test_SingleRouterPerf for handler_xgress.NewBindHandler's
signature, which takes an env.RouterEnv as its first argument
- adds a minimal testRouterEnv stub supplying the forwarder and xgress
metrics the bind handler uses
- builds the xgress metrics from the test usage registry via
router/metrics.NewXgressMetrics
- adds a common/servermetrics package that owns the metrics MetricsMessage wire
format and the reporting/usage subsystem (message builder, usage registry,
interval and usage counters), wrapping the openziti/metrics Registry for
metric collection
- moves the controllers metrics reporter into the router package and removes it
from the shared metrics package, breaking a common -> router/env import cycle
- repoints controller and router consumers to common/servermetrics; base metric
collection stays on openziti/metrics
- keeps the proto field numbers and the metrics content-type identical so the
encoding is byte-compatible across the move, and uses a distinct proto package
name so ziti's and the library's messages coexist without a global proto
registry clash
- adds a round-trip test asserting wire compatibility with the library's
MetricsMessage
- leaves openziti/metrics unchanged, so sdk-golang and the shared xgress data
plane are unaffected
- adds a striped id locker in common/concurrency for sharded per-entity locking
- switches the link manager to striped per-link locking to reduce contention
- adds a cache-before-transaction read path to the router manager
- adds a RouterReportedLink regression test covering same-link serialization and stale/same-iteration no-op behavior
Implements the router-side Connect-V2 sessionless dial path. Dials are
authorized locally via the RouterDataModel instead of a controller-issued
service session token; circuit creation flows through the existing
`CreateCircuitV3` controller endpoint (#3721). Builds on the sdk-golang
v2 migration.
- Adds `processConnectV2` on `edgeClientConn`: resolves the service by id
or name via the RouterDataModel, checks dial access, and dispatches to
the controller via `sendCreateCircuitV3Msg`. Supports both
`xgEdgeForwarder` (SDK xgress) and `nonXgConnectHandler` flow-control
modes, selected by the SDK's `UseXgressToSdkHeader`.
- Makes `CircuitId` optional in `DecodeCreateCircuitV3Request`. The V2
router path does not pre-assign a circuit ID; the controller generates
it as V1/V2 already do. Without this the decoder rejected the empty
header and every V2 dial hung until timeout. Adds a regression test.
- Splits `checkAccess` to close a posture-check bypass on the V2 path.
The old single `checkAccess` short-circuited to nil for non-OIDC
sessions (V1 ran posture at the controller during `CreateSession`); V2
has no such step, so posture would have been skipped. `checkAccess` now
always runs the RDM `HasAccess` (policy + posture) check;
`checkAccessIfOidc` keeps the OIDC-only gate for the V1 and bind paths.
- Sends the V2 `state_connected` on the default (data) sender rather than
the control sender. On multi-underlay channels the two senders are
independently ordered, so an early terminator payload on the data
sender could beat `state_connected` to the SDK and be dropped (channel/v5
has no message-priority API).
- Updates `xgEdgeForwarder.lastRx` on every forward path, including the
fast `timeout == 0` `TrySend` branch used for normal payload dispatch.
The old code only updated it on the `timeout > 0` path, so active V2
circuits looked idle and could be unrouted prematurely.
- Adds `state.ConnState.ServiceId`, populated by the connect handlers from
the service session token (V1) or the request header (V2). The
non-xgress V2 path previously left this empty, so `handleDialAccessLost`
could not identify and close V2 non-xgress circuits when dial access was
revoked.
- Skips conns with no `ServiceSessionToken` in `RemoveLegacyServiceSession`;
a sessionless V2 conn's token is nil and the cleanup loop previously
dereferenced it unconditionally, which would panic the router.
- Advertises Connect-V2 via the `RouterCapabilityConnectV2` bit in the
listener hello so SDKs can detect V2 support.
- Wires `ContentTypeConnectV2` and `ContentTypeXgControl` handlers in
`Acceptor.BindChannel`, and adds `handleXgControl` for SDK-side xgress
control messages, preserving `ControlUserVal` so trace-route responses
correlate back to the initiator's `SendForReply` waiter.
- Adds `RouterDataModel.serviceNameIndex` for O(1) name->id lookup in the
V2 dial path, maintained with rename safety at the `HandleServiceEvent`
mutation points.
- Adds `tests/connect_v2_test.go` covering end-to-end V2 dataflow and the
V1 fallback (`ForceConnectV1`), asserting the dial path via the SDK
`DialEvent`.
- Propagates a V2 initiator's graceful half-close to legacy hosts via
`edgeXgressConn.FlowFromFabricToXgressClosed`, which emits an edge FIN
when the fabric->app half of the circuit closes. The SDK signals
half-close to its router xgress peer with the native xgress EOF flag;
without translating that to an edge FIN, a legacy host reading to EOF
stalled until teardown.
- Records the dialing identity id as the circuit `ClientId` for
sessionless V2 dials, since there is no dial session to key on; updates
`Test_OidcEvents` to match.
- Adds `tests/connect_v2_teardown_test.go` covering client- and
host-initiated close propagation on both the V2 and forced-V1 paths.
- Polls for the asynchronous conn close in the SDK posture-check tests
(`awaitClientConnClosed`): revocation tears the circuit down out of
band, so checking `IsClosed` immediately after the first read error was
racy.
- Temporarily pins sdk-golang/v2 to the openziti/sdk-golang#959 commit,
which carries the matching xgress conn-close-on-teardown fix the V2
posture tests depend on; to be repointed at the next sdk-golang
pre-release before merge.
For openziti/sdk-golang#936.
Bumps the sdk-golang dependency from v1 to the v2 module
(`github.com/openziti/sdk-golang/v2` at v2.0.0-pre1) and updates all
import paths. This is a no-behavior-change precursor that isolates the
dependency migration from the Connect-V2 feature work in #3884.
- Rewrites `github.com/openziti/sdk-golang/...` imports to
`github.com/openziti/sdk-golang/v2/...` across the main and zititest
modules.
- Pins both modules to `github.com/openziti/sdk-golang/v2 v2.0.0-pre1`.
- Adapts `edgeXgressConn.AcceptMessage` to the v2 `MsgSink` signature,
which now takes an `edge.SdkChannel` argument.
- Replaces the removed `edge.Conn.GetRouterId()` with
`RemoteAddr().String()` in the loop4 traffic-test logging.
For openziti/sdk-golang#936.
- sets Constraints and MinTotalUnderlays: 1 on the listener channel.Config
fixtures so they match the production accept paths and remain multi-underlay-
capable; without them the test listener channels were treated as simple and
rejected additional grouped underlays
- gives ListenerCtrlChannel Min: 0 constraints per underlay type plus
MinTotalUnderlays: 1 in its configs, so the controller accepts the
high/low-priority grouped underlays the router dials while still closing the
channel only when its last underlay is lost
- restores the multi-underlay behavior the v4 listener-side SetMinTotal(1)
provided, which the channel/v5 migration dropped
- works around channel/v5 not yet treating MinTotalUnderlays alone as a
multi-underlay signal
- switches the xlink transport and router ctrl listeners to NewClassicListenerWithAcceptor, passing the MultiListener as a HelloAcceptor
- replaces the controller ctrl channel's NewClassicListener/UnderlayDispatcher wiring with NewClassicListenerWithAcceptor and a TypeRoutingAcceptor, adapting the mesh acceptor via AsHelloAcceptor
- removes the multiListenerAcceptor wrapper now that MultiListener implements HelloAcceptor directly
- removes the xgress_edge Acceptor.Run Create-loop, handing underlays to the MultiListener through the acceptor-based listener
- moves the controller ctrl connect handler into ListenerConfig.ConnectionHandlers
- updates ctrlchan channel tests to the new constructor
- decomposes the ctrlchan, xlink and edge-listener channels onto the v5 Senders, MessageSourceProvider and UnderlayEventListener interfaces, replacing the v4 UnderlayHandler god-interface
- replaces the hand-rolled dial/grouping/backoff machinery with channel.BackoffDialPolicy and declarative Constraints; ctrl keeps survive-to-zero (Min: 0) with MinStableDuration: 0 for prompt reconnect, while xlink and edge default underlays keep Min: 1 so loss closes the channel
- builds grouped channels via channel.NewChannel(*Config) and moves handler retrieval to GetSenders()
- records the channel via InitChannel from each bind handler, before underlay events fire, so handlers registered during bind do not dereference a nil channel
- generates a group secret for ungrouped inbound ctrl underlays on the router accept path, matching the controller, since NewChannel requires one
- preserves link-id-as-channel-id (the link dial policy wraps the cloned link-id identity dialer) and adds a test asserting dialed underlays present the link id
- registers the latency handler explicitly, as it is no longer a self-describing receiver in v5
- channel.MultiChannel -> channel.Channel
- channel.MultiChannelConfig -> channel.Config
- channel.NewMultiChannel(...) -> channel.NewChannel(...)
channel/v5 unifies Channel and MultiChannel into a single Channel abstraction. This is the
mechanical token rename; the Config field changes and handler retrieval that the unification
requires land in the following commit. The two channel.go files that are fully rewritten for
v5 (common/ctrlchan, router/xlink_transport) are excluded here and rewritten in that commit.
Does not build on its own.
- channel.NewChannel(name, factory, bindHandler, opts) -> channel.NewSingleChannel(...)
- channel.NewChannelWithUnderlay(...) -> channel.NewSingleChannelWithUnderlay(...)
channel/v5 repurposes NewChannel for the unified multi-underlay constructor, so the
single-underlay call sites move to the renamed helpers. Mechanical rename only; like the
preceding import commit it does not build on its own.
Andrew Martinez