- applies the router.link.v1 heartbeat settings (sendInterval, checkInterval,
closeUnresponsiveTimeout) and payload/ack sender queue sizes at runtime,
replacing the hardcoded 10s/1s link heartbeat intervals and the ignored fields;
new links take them at bind time, established links are re-tuned in place
through the channel HeartbeatControl, and queue sizes reach new links only
- publishes heartbeat settings as an immutable generation in one atomic swap, so
no reader pairs one generation's interval with another's timeout, and has each
link enforce the timeout belonging to its own intervals
- pushes a generation and its intervals as one step under the link's lock and
ignores a generation at or below the one applied, so an overtaken update cannot
roll a link back; reconciles every channel of a link, including a split link
whose channels bind on either side of a config change, to the newest generation
- adopts the first registered settings explicitly, since a router with no managed
link config publishes its defaults at generation 0
- re-arms a link's liveness deadline when its generation changes, so age
accumulated under the previous intervals cannot close a healthy link when a
timeout is shortened
- honors a closeUnresponsiveTimeout below 30s, which previously sat behind the
unhealthy threshold; treats a zero timeout as closing nothing, and reports a
closing link unhealthy
- scales the unhealthy threshold with the intervals, to max(30s, 2 x (send +
check)): reporting unhealthy poisons the latency metric the controller routes
on, so a fixed 30s priced healthy links with a long send interval out of
routing every cycle
- validates heartbeat timing on the controller and the router: durations must be
positive, their sum representable, and the timeout at least 10s and greater
than sendInterval + checkInterval; a timeout under two heartbeat cycles is
accepted with a warning
- adds a per-config-type semantic validator registry to ConfigManager, run after
schema validation, and moves the router.link config definition to
common/config/routerlink so the controller applies the same rules
- treats a link section in router YAML, of any content, as local config that
takes precedence over the controller's for the whole router.link type, since
the loader defaults every field and so now records presence separately; warns
at startup when that section has no listeners or dialers
- translates local heartbeat timing leniently, since a local failure stops the
router from starting: keeps an explicit zero, raises a timeout below the floor
to it, and falls back to the defaults for other invalid timing, each with a
warning
- stamps UpdateLinkListeners with the config generation it came from and has the
controller drop superseded sets, which a multi-underlay control channel can
deliver out of order and which would otherwise send peers to withdrawn
addresses
- logs every successful link config apply, including those that dispatch no work,
and pins that a queue-size-only change triggers no GC pass
- deprecates split links, documents the feature in the changelog and the router
config design doc, and brings both up to date
Builds on the channel/v5 HeartbeatControl from openziti/channel#265.
- adds a structured xlink.LinkKey ({DialerBinding, Protocol, DestId,
ListenerBinding}) used by all staleness and GC checks, replacing the
pre-rendered key string each side used to pass around
- adds the CheckStaleLinks ctrl message and the `ziti ops verify stale-links`
CLI for operator-driven, two-sided staleness verification with optional --gc,
which acts only when both endpoints agree
- implements auto-GC via the router.link.v1 gcMode (preserve/orphaned/changed):
the router walks its xlinks after each Apply that mutates listeners, dialers,
or gcMode and closes one-sided-stale entries under the configured mode
- sweeps a removal under orphaned whatever the removed config said, including
preserve, since a removal leaves no listeners and no dialers and every
remaining link is one the router can never re-establish
- reports staleness as a three-state verdict (unknown/notStale/stale) so a side
that cannot judge a link abstains instead of guessing; a destination absent
from the registry's listener snapshot means "listeners unknown", not "peer has
none", which keeps a briefly unhealthy peer from losing its live links
- rebuilds the link surface only when a listener or dialer definition actually
moves, so a gcMode-only change no longer rebinds every listen socket
- gates CheckStaleLinks on a new RouterStaleLinkCheck capability and explains
each unqueryable endpoint, including one left unasked by the filter, so a
partial verdict says which of the three causes applies
- detects a peer re-advertising the address a link was dialed to from the
dialing side, not just from the peer's own listener side
- pins every verdict to the link iteration it was computed against, faults that
iteration rather than the current one, and takes the conditional removal's
result as the decision, so a link re-dialed while the sweep was running is
neither closed nor reported as collected
- resolves which side a report speaks for from the sending router against the
controller's link map rather than from the side the report names, so one
router cannot fill both sides and satisfy the both-agree rule --gc depends on
- bounds the query fan-out with a semaphore, matching the other validation
sweeps, since each query carries a minute-long reply timeout
- fails the CLI when the result stream ends before every link is reported, and
drains results already buffered before declaring any missing
- rejects an unrecognized match mode rather than resolving it to changed, the
broadest removal criterion
- consolidates the controller-managed router configuration changelog into one
section, replacing the status that said routers do not yet consume configs at
runtime, and documents the per-config-type opt-in
- restores the Current Beta Features section, dropped when the 2.0 changelog was
archived, and lists controller-managed router configuration in it
- adds unparseable to the codespell ignore list, a spelling variant the
dictionary flags as a typo
For #3743.
- adds an oidc-auth-test fablab model that exercises OIDC authentication end to end, with an event-forwarder component, an oidc-test-client, a ziti-prox-c component, and OIDC event/gossip/traffic validations
- moves zitirest out of the public API into separate test-only shims: an integration-test shim (tests/restclient.go) and a fablab shim (zititest/zitirest), and repoints upgrade-test at the fablab shim
- adds region-isolation network partitioning and packet capture on all interfaces to the chaos toolkit
- queues oidc-test-client results reporting on a single sender goroutine with a bounded write deadline, so a stalled results circuit can't block the traffic loops
- counts results events dropped when that queue fills and reports them to the collector as errors, so a reporting outage fails validation instead of looking like a clean run
- bounds the event forwarder's event and keepalive writes, and checks for shutdown on every send attempt, so an unreadable destination can't park the forward loop holding its lock
- logs read errors on collector connections instead of ending collection for that client silently
- bounds the oidc-auth-test debug server's header and body reads
- matches ziti-prox-c process matching to the version-agnostic binary name, so a version change still finds the running process
- points upgrade-test's per-service terminator count at the shared validations helper
- adds a fablab design summary doc
- adds a `DataState.Router` event variant (id, name, fingerprint,
configs) to the RDM protobuf and the matching ConfigType.Target field
flows
- loads routers into `RouterDataModelSender` at startup and registers
entity-change listeners so router create/update/delete and config
reassignment emit RDM events
- filters `Config` events per-router at `RouterSender`: each router
sees the full `Router` set but only its own router-target Configs
- receiver-side: parses and stores `Router` entities; GCs router-target
Configs locally when they drop off the router's `Configs` list, so
no synthetic remove events are needed on the wire
- extends the `router-data-model-test` fablab model with router-config
distribution scenarios (assignment, reassignment, controller
restart, RDM cache miss)
- updates `validate-router-data-model` to recognize the new event
shape
- refreshes the design doc to describe per-router filtering and the
change-notification flow
* Add router.link.v1 config type. Fixes#3974
- adds the built-in `router.link.v1` ConfigType with a JSON schema
covering listeners, dialers, heartbeats, payload/ack sender queue
sizes, and `gcMode` for auto-GC of stale links
- targets routers (`Target=router`) via the field added in #3743
- registers it for new databases via `createConfigType` in
`initialize`, and for existing databases via a migration step that
bumps the schema version 46 -> 47
- adds config-type-store tests covering registration, the router
target, and migration-driven creation on existing databases
- documents the Phase 1c step in the controller-managed router
configuration design doc
* Address review feedback on router.link.v1 config type. For #3974
- broadens the duration schema pattern to accept compound and fractional values (e.g. 1h30m, 1.5h) matching time.ParseDuration
- adds the gcMode schema property (enum preserve/orphaned/changed) to the built-in config type so its definition is complete where the type is created
- adds duration-format and gcMode validation test cases
- removes the now-unnecessary per-iteration loop variable copy in the reject-cases test
- adds BuildPrettyHandler and BuildHandlerForFormat in common/logging:
pretty output wraps the hand-rolled logging.PrettyHandler (a direct
port of pfxlog's; df/dl was dropped after review found level-label
gaps, so there is no github.com/michaelquigley/df dependency) in the
AsyncHandler chain; the format-aware builder picks pretty / json /
text by --log-formatter so default look matches pre-slog
- adds BuildTextHandler so --log-formatter=text emits logrus-TextFormatter-
style key=value output (level=info msg=...) via a slog TextHandler rather
than the colored pretty handler, restoring the pre-slog meaning of text
- adds logging.Fatal: a slog-world fatal (slog provides none) that emits
at LevelFatal durably via SyncEmit, then exits, so hard-exit paths do
not lose the record to the async queue; converts the controller and
router startup hard-exit sites from Error+os.Exit / Error+panic to it,
dropping the router's startup panic
- rewires agentlog.DefaultLogLevelCallbacks onto common/logging:
SetLogLevel drives logging.SetGlobalLevel (lockstep slog + logrus),
SetChannelLogLevel and ClearChannelLogLevel drive SetNamedLevel /
ClearNamedLevel; per-channel overrides become slog-only per design
- adds agentToSlog mapping across the seven canonical levels with an
Info fallback for forward-compat
- ziti/run Options.PreRun and ziti/tunnel rootPreRun build the slog
handler chain via logging.BuildHandlerForFormat and call Install;
--verbose seeds the initial level instead of mutating logrus
directly; AsyncOptions flags exposed via logging.AddFlags on each
persistent flag set
- hardens the run command's logging flags: PreRun reads --verbose /
--log-formatter across the command chain so they are honored at either
the alias-parent (ziti controller run) or child position, and the
ziti controller / ziti router alias parents skip their legacy
pfxlog/logrus PersistentPreRun setup for the run subcommand (which
installs the slog chain itself), keeping it for sibling subcommands
- adds Phase 7 acceptance tests in common/agentlog: TestInstallInvariant
covers Out=io.Discard, noop formatter, ReportCaller, and the
lockstep level mirror after Install; TestEndToEnd_AgentSetLogLevel
walks the agent set-log-level path end to end across bridged-logrus
and direct-slog routes; TestPerChannelOverride_AppliesToSlogOnly_NotPfxlog
confirms the design's slog-only channel semantics
- adds Fatal/Panic durability subprocess tests in common/logging that
fork the test binary, Install the production handler chain, then
call logrus.Fatal / logrus.Panic and assert the records reach stderr
before exit/panic; proves the bridge's SyncEmit path flushes before
os.Exit
- adds doc/logging.md developer note covering how to write a slog
line, channel-naming convention, the no-Warn/Error-in-hot-paths
rule, the operator surface, the migration checklist, AsyncOptions
tunables, and what's deliberately deferred
- adds doc/design/slog-conversion-plan.md with the code-grounded
per-package channel inventory, the sdk-golang embedder-injection
pattern, conversion order with deep analysis for the first four
chunks, and cross-repo coordination notes
- adds a new common/logging package with custom slog.Level constants for
Trace (-8), Fatal (12), and Panic (16) extending slog's four standard
levels, plus LevelName and ParseLevel as the single source of truth for
canonical lowercase wire names (warn and warning both accepted)
- adds AsyncOptions (QueueSize, BlockThreshold, SummaryInterval) with
Validate, defaults of 4096 / Warn / 5s, and AddFlags / OptionsFromFlags
bindings so the package can wire into cobra via spf13/pflag alone
- adds AsyncHandler, a bounded async slog.Handler that hands records to a
single drain goroutine and onto a downstream handler under a shared
mutex; records at or above the block threshold block (with a closeNotify
escape so shutdown cannot deadlock), records below it drop when the
queue is full and bump a per-level atomic counter
- the drain emits a drop-summary record on each SummaryInterval tick when
any per-level counter is non-zero, and also counts downstream errors in
a drain_errors counter that appears in the same summary line; downstream
errors are also logged once to os.Stderr to avoid slog recursion
- Close signals shutdown and returns immediately; the drain final-flushes
records that beat the close, emits a final summary if drops occurred,
and closes drainDone for tests
- SyncEmit bypasses the queue and writes through the downstream handler
synchronously under the same downstreamMu the drain uses, so fatal/panic
records are durable before the process exits
- adds boundHandler, which prepends bound attrs to every record flowing
through it before delegating to its parent; WithAttrs returns a new
boundHandler whose parent is the receiver's parent (not the receiver
itself), so a chain of slog.Logger.With calls produces sibling
boundHandlers at the same chain depth rather than stacking
wrapper-on-wrapper
- adds groupedHandler, which wraps record attrs in slog.Group(name, ...)
before delegating; a subsequent WithAttrs creates a boundHandler whose
parent is the groupedHandler, so the attrs land inside the group
- AsyncHandler.WithAttrs and WithGroup are the real chain entry points;
empty attrs and empty group names return the receiver so no-op
slog.Logger.With() and WithGroup("") allocate nothing
- covers the lot with -race tests: level round-trip and offset fallback,
defaults validity and bad-value rejection, flag round-trip, async normal
flow, drop-on-full with summary attrs, block at the threshold, Close
idempotent + non-blocking + unblocks Handle, Handle racing Close never
panics, SyncEmit synchronous and serialized with the drain, drain-error
counting, the four worked examples from the design doc for the chain
(with-then-group, group-then-with, nested groups, basic with-attrs),
no-nesting on repeated WithAttrs, sibling-loggers-do-not-leak-attrs,
WithGroup("") and WithAttrs(nil) as no-ops on all three handler types,
and Enabled delegation through the chain
- adds an in-process end-to-end test that starts the agent, registers recording
callbacks, and exercises AppInfoV2, the three v2 channel commands, and the
framed-through-callback path
- adds a developer note (common/agent/README.md) explaining the two-tier
capability model and how to add an agent capability, register an app
capability, add a v2 channel command, and preserve wire compatibility
doc/design/logging-refactor.md captures the design for moving the
codebase to log/slog without forcing a whole-tree migration.
- slog as the API contract at every call site
- logrus.Hook bridge so ~3000 pfxlog/logrus call sites benefit
without code changes
- common/logging.AsyncHandler: bounded queue, level-aware drop
policy, periodic summary emission, never-close-the-queue shutdown
- dl.NewPrettyHandler adopted for dev/console; JSONHandler with
ReplaceAttr for production JSON shape compatibility
- named-logger overrides via logging.For(name); existing
set-channel-log-level IPC dispatches to both pfxlog and slog
registries
- PC-based method/file overrides deferred to a follow-up branch
- one proof-of-pattern call-site conversion in this branch
* wip changes: trying to make the quickstart nothing but ha
* when on windows be nice and use a powershell friendly line continuation char
* remove --clustered usage from config create as that's the default and remove 'ha' from quickstart test
* forgot to update the tests as well
* remove other 'ha' references in the config generation and verify traffic
* remove ha refs from the quickstart ha doc
* was a bit too quick to remove these flags - putting back
* change from fatals to error and RunE. add --spiffed-id to pki creation. remove now defunct TestDatabaseFile*
* updates to docker based test
* pr feedback updates
* missed some test updates
* Remove initialMembers config. Fixes#2745
* Consolidate agent cluster commands. Fixes#2746
* Start removing HA doc, now that it's being added to doc site
* Allow configuring endpoints file full path instead of directory. Fixes#2724
* Write initial router endpoints file based on ctrls in JWT. Fixes#2728
* Add ctrls property to non-ha router enrollment. Fixes#2108
* Enrollment doesn't contain controller which created the enrollment. Fixes#2729
Also fix member events. We were getting add evetts on startup as the log was replayed. Store them in the
DB so we only get events when membership has actually changed.
The path description for the Windows hosts file was updated from:
- Windows: `windows: %windir%\system32\drivers\etc\hosts`
to
- Windows: `%windir%\system32\drivers\etc\hosts`
This change removes the redundant "windows:" prefix, making the path clearer and more accurate for users.
Paul Lorenz