192 Commits

Author SHA1 Message Date
Paul Lorenz e8b42b89ae Add OIDC auth fablab test and supporting test infrastructure
- adds an oidc-auth-test fablab model that exercises OIDC authentication end to end, with an event-forwarder component, an oidc-test-client, a ziti-prox-c component, and OIDC event/gossip/traffic validations
- moves zitirest out of the public API into separate test-only shims: an integration-test shim (tests/restclient.go) and a fablab shim (zititest/zitirest), and repoints upgrade-test at the fablab shim
- adds region-isolation network partitioning and packet capture on all interfaces to the chaos toolkit
- queues oidc-test-client results reporting on a single sender goroutine with a bounded write deadline, so a stalled results circuit can't block the traffic loops
- counts results events dropped when that queue fills and reports them to the collector as errors, so a reporting outage fails validation instead of looking like a clean run
- bounds the event forwarder's event and keepalive writes, and checks for shutdown on every send attempt, so an unreadable destination can't park the forward loop holding its lock
- logs read errors on collector connections instead of ending collection for that client silently
- bounds the oidc-auth-test debug server's header and body reads
- matches ziti-prox-c process matching to the version-agnostic binary name, so a version change still finds the running process
- points upgrade-test's per-service terminator count at the shared validations helper
- adds a fablab design summary doc
2026-08-28 22:00:01 -04:00
Paul Lorenz 0ee288ecbf Distribute routers and router-target configs through the RDM. Fixes #3976 (#3977)
- adds a `DataState.Router` event variant (id, name, fingerprint,
  configs) to the RDM protobuf and the matching ConfigType.Target field
  flows
- loads routers into `RouterDataModelSender` at startup and registers
  entity-change listeners so router create/update/delete and config
  reassignment emit RDM events
- filters `Config` events per-router at `RouterSender`: each router
  sees the full `Router` set but only its own router-target Configs
- receiver-side: parses and stores `Router` entities; GCs router-target
  Configs locally when they drop off the router's `Configs` list, so
  no synthetic remove events are needed on the wire
- extends the `router-data-model-test` fablab model with router-config
  distribution scenarios (assignment, reassignment, controller
  restart, RDM cache miss)
- updates `validate-router-data-model` to recognize the new event
  shape
- refreshes the design doc to describe per-router filtering and the
  change-notification flow
2026-06-16 22:31:43 -04:00
Paul Lorenz 1d6333a6c0 Add router.link.v1 config type. Fixes #3974 (#3975)
* Add router.link.v1 config type. Fixes #3974

- adds the built-in `router.link.v1` ConfigType with a JSON schema
  covering listeners, dialers, heartbeats, payload/ack sender queue
  sizes, and `gcMode` for auto-GC of stale links
- targets routers (`Target=router`) via the field added in #3743
- registers it for new databases via `createConfigType` in
  `initialize`, and for existing databases via a migration step that
  bumps the schema version 46 -> 47
- adds config-type-store tests covering registration, the router
  target, and migration-driven creation on existing databases
- documents the Phase 1c step in the controller-managed router
  configuration design doc

* Address review feedback on router.link.v1 config type. For #3974

- broadens the duration schema pattern to accept compound and fractional values (e.g. 1h30m, 1.5h) matching time.ParseDuration
- adds the gcMode schema property (enum preserve/orphaned/changed) to the built-in config type so its definition is complete where the type is created
- adds duration-format and gcMode validation test cases
- removes the now-unnecessary per-iteration loop variable copy in the reject-cases test
2026-06-16 16:38:13 -04:00
Paul Lorenz 0aefe599c9 Install slog and route agent log-level callbacks through common/logging. Fixes #3910
- adds BuildPrettyHandler and BuildHandlerForFormat in common/logging:
  pretty output wraps the hand-rolled logging.PrettyHandler (a direct
  port of pfxlog's; df/dl was dropped after review found level-label
  gaps, so there is no github.com/michaelquigley/df dependency) in the
  AsyncHandler chain; the format-aware builder picks pretty / json /
  text by --log-formatter so default look matches pre-slog
- adds BuildTextHandler so --log-formatter=text emits logrus-TextFormatter-
  style key=value output (level=info msg=...) via a slog TextHandler rather
  than the colored pretty handler, restoring the pre-slog meaning of text
- adds logging.Fatal: a slog-world fatal (slog provides none) that emits
  at LevelFatal durably via SyncEmit, then exits, so hard-exit paths do
  not lose the record to the async queue; converts the controller and
  router startup hard-exit sites from Error+os.Exit / Error+panic to it,
  dropping the router's startup panic
- rewires agentlog.DefaultLogLevelCallbacks onto common/logging:
  SetLogLevel drives logging.SetGlobalLevel (lockstep slog + logrus),
  SetChannelLogLevel and ClearChannelLogLevel drive SetNamedLevel /
  ClearNamedLevel; per-channel overrides become slog-only per design
- adds agentToSlog mapping across the seven canonical levels with an
  Info fallback for forward-compat
- ziti/run Options.PreRun and ziti/tunnel rootPreRun build the slog
  handler chain via logging.BuildHandlerForFormat and call Install;
  --verbose seeds the initial level instead of mutating logrus
  directly; AsyncOptions flags exposed via logging.AddFlags on each
  persistent flag set
- hardens the run command's logging flags: PreRun reads --verbose /
  --log-formatter across the command chain so they are honored at either
  the alias-parent (ziti controller run) or child position, and the
  ziti controller / ziti router alias parents skip their legacy
  pfxlog/logrus PersistentPreRun setup for the run subcommand (which
  installs the slog chain itself), keeping it for sibling subcommands
- adds Phase 7 acceptance tests in common/agentlog: TestInstallInvariant
  covers Out=io.Discard, noop formatter, ReportCaller, and the
  lockstep level mirror after Install; TestEndToEnd_AgentSetLogLevel
  walks the agent set-log-level path end to end across bridged-logrus
  and direct-slog routes; TestPerChannelOverride_AppliesToSlogOnly_NotPfxlog
  confirms the design's slog-only channel semantics
- adds Fatal/Panic durability subprocess tests in common/logging that
  fork the test binary, Install the production handler chain, then
  call logrus.Fatal / logrus.Panic and assert the records reach stderr
  before exit/panic; proves the bridge's SyncEmit path flushes before
  os.Exit
- adds doc/logging.md developer note covering how to write a slog
  line, channel-naming convention, the no-Warn/Error-in-hot-paths
  rule, the operator surface, the migration checklist, AsyncOptions
  tunables, and what's deliberately deferred
- adds doc/design/slog-conversion-plan.md with the code-grounded
  per-package channel inventory, the sdk-golang embedder-injection
  pattern, conversion order with deep analysis for the first four
  chunks, and cross-repo coordination notes
2026-06-15 15:34:31 -04:00
Paul Lorenz ff8ae7cd45 Add common/logging package: AsyncHandler, level helpers, and chain composition. Fixes #3904
- adds a new common/logging package with custom slog.Level constants for
  Trace (-8), Fatal (12), and Panic (16) extending slog's four standard
  levels, plus LevelName and ParseLevel as the single source of truth for
  canonical lowercase wire names (warn and warning both accepted)
- adds AsyncOptions (QueueSize, BlockThreshold, SummaryInterval) with
  Validate, defaults of 4096 / Warn / 5s, and AddFlags / OptionsFromFlags
  bindings so the package can wire into cobra via spf13/pflag alone
- adds AsyncHandler, a bounded async slog.Handler that hands records to a
  single drain goroutine and onto a downstream handler under a shared
  mutex; records at or above the block threshold block (with a closeNotify
  escape so shutdown cannot deadlock), records below it drop when the
  queue is full and bump a per-level atomic counter
- the drain emits a drop-summary record on each SummaryInterval tick when
  any per-level counter is non-zero, and also counts downstream errors in
  a drain_errors counter that appears in the same summary line; downstream
  errors are also logged once to os.Stderr to avoid slog recursion
- Close signals shutdown and returns immediately; the drain final-flushes
  records that beat the close, emits a final summary if drops occurred,
  and closes drainDone for tests
- SyncEmit bypasses the queue and writes through the downstream handler
  synchronously under the same downstreamMu the drain uses, so fatal/panic
  records are durable before the process exits
- adds boundHandler, which prepends bound attrs to every record flowing
  through it before delegating to its parent; WithAttrs returns a new
  boundHandler whose parent is the receiver's parent (not the receiver
  itself), so a chain of slog.Logger.With calls produces sibling
  boundHandlers at the same chain depth rather than stacking
  wrapper-on-wrapper
- adds groupedHandler, which wraps record attrs in slog.Group(name, ...)
  before delegating; a subsequent WithAttrs creates a boundHandler whose
  parent is the groupedHandler, so the attrs land inside the group
- AsyncHandler.WithAttrs and WithGroup are the real chain entry points;
  empty attrs and empty group names return the receiver so no-op
  slog.Logger.With() and WithGroup("") allocate nothing
- covers the lot with -race tests: level round-trip and offset fallback,
  defaults validity and bad-value rejection, flag round-trip, async normal
  flow, drop-on-full with summary attrs, block at the threshold, Close
  idempotent + non-blocking + unblocks Handle, Handle racing Close never
  panics, SyncEmit synchronous and serialized with the drain, drain-error
  counting, the four worked examples from the design doc for the chain
  (with-then-group, group-then-with, nested groups, basic with-attrs),
  no-nesting on repeated WithAttrs, sibling-loggers-do-not-leak-attrs,
  WithGroup("") and WithAttrs(nil) as no-ops on all three handler types,
  and Enabled delegation through the chain
2026-06-12 19:34:17 -04:00
Paul Lorenz c92de09624 Document the agent capability model and cover commands end-to-end
- adds an in-process end-to-end test that starts the agent, registers recording
  callbacks, and exercises AppInfoV2, the three v2 channel commands, and the
  framed-through-callback path
- adds a developer note (common/agent/README.md) explaining the two-tier
  capability model and how to add an agent capability, register an app
  capability, add a v2 channel command, and preserve wire compatibility
2026-06-12 16:18:28 -04:00
Paul Lorenz 16e8dc2244 Add logging-refactor design proposal
doc/design/logging-refactor.md captures the design for moving the
codebase to log/slog without forcing a whole-tree migration.

- slog as the API contract at every call site
- logrus.Hook bridge so ~3000 pfxlog/logrus call sites benefit
  without code changes
- common/logging.AsyncHandler: bounded queue, level-aware drop
  policy, periodic summary emission, never-close-the-queue shutdown
- dl.NewPrettyHandler adopted for dev/console; JSONHandler with
  ReplaceAttr for production JSON shape compatibility
- named-logger overrides via logging.For(name); existing
  set-channel-log-level IPC dispatches to both pfxlog and slog
  registries
- PC-based method/file overrides deferred to a follow-up branch
- one proof-of-pattern call-site conversion in this branch
2026-06-12 15:46:32 -04:00
Paul Lorenz 2bf9808df6 Add target field to config types. Fixes #3744 2026-05-27 13:17:06 -04:00
Paul Lorenz 82ac2e8060 Rework connect event handling to ensure serialized-per-router handling 2026-04-24 13:22:02 -04:00
Paul Lorenz 6c0cf335d1 Consolidate CLI edge and fabric commands in top level create/update/delete/list/login commands. Fixes #3699 2026-04-15 11:12:55 -04:00
Paul Lorenz 4ee9d94b49 Update docs 2026-04-07 21:49:37 -04:00
Paul Lorenz 46e7ee33e1 Add doc just for metrics and config changes 2026-04-04 14:03:35 -04:00
Clint Dovholuk f6814c117a Quickstart ha only (#3724)
* wip changes: trying to make the quickstart nothing but ha

* when on windows be nice and use a powershell friendly line continuation char

* remove --clustered usage from config create as that's the default and remove 'ha' from quickstart test

* forgot to update the tests as well

* remove other 'ha' references in the config generation and verify traffic

* remove ha refs from the quickstart ha doc

* was a bit too quick to remove these flags - putting back

* change from fatals to error and RunE. add --spiffed-id to pki creation. remove now defunct TestDatabaseFile*

* updates to docker based test

* pr feedback updates

* missed some test updates
2026-03-27 10:13:42 -04:00
Dave Hart 198085ed94 Updated lines to avoid overlaps (#3665) 2026-03-10 09:09:35 -04:00
Dave Hart 5a6e98ded6 README updates 2026-03-08 20:49:43 -04:00
Feng Tu 06d112720b chore: typo at client cert create 2025-10-05 04:23:08 +08:00
Francesco Faraone f0dbcd512a Azure Service Bus logger for controller event streaming 2025-09-03 10:05:37 +02:00
Paul Lorenz f7caf82276 Update for deprecation of HA flag in Go SDK 2025-08-18 19:28:28 -04:00
Andrew Martinez 3995091865 fixes #3068 adds event dispatching for auto ca enrollment (#3069) 2025-05-29 16:05:33 -04:00
Kenneth Bingham c7a95bd1de Update quickstart.md 2025-04-15 15:06:21 -04:00
Paul Lorenz f57006c0b2 Remove init-from-db. Update doc. Fixes #2756 2025-02-06 21:17:39 -05:00
Paul Lorenz d178caefc7 Fix places where controller init is mentioned 2025-02-05 17:10:35 -05:00
Paul Lorenz 6e7c5bf4ac Cluster config/cli tweaks
* Remove initialMembers config. Fixes #2745
* Consolidate agent cluster commands. Fixes #2746
* Start removing HA doc, now that it's being added to doc site
2025-02-05 16:33:05 -05:00
Paul Lorenz 6a01db9e4c Rename raft config to cluster. Fixes #2731 2025-02-04 13:31:15 -05:00
Paul Lorenz fef5327e14 Endpoint management updates
* Allow configuring endpoints file full path instead of directory. Fixes #2724
* Write initial router endpoints file based on ctrls in JWT. Fixes #2728
* Add ctrls property to non-ha router enrollment. Fixes #2108
* Enrollment doesn't contain controller which created the enrollment. Fixes #2729
2025-02-03 18:52:57 -05:00
Paul Lorenz f147fc6b99 Add events for cluster has leader/is leaderless. Change cluster events NS to cluster. Fixes #2639 Fixes #2184
Also fix member events. We were getting add evetts on startup as the log was replayed. Store them in the
DB so we only get events when membership has actually changed.
2025-01-10 13:57:28 -05:00
Kenneth Bingham 22438f470e clarify restarting disconnected members 2024-12-20 17:26:31 -05:00
Kenneth Bingham 98816d53f0 add ha quickstart doc 2024-12-20 17:20:09 -05:00
Paul Lorenz c2162ea9a9 Add cluster id, to prevent merging disparate clusters. Fixes #2541 2024-11-14 12:34:17 -05:00
Paul Lorenz 6e1983991d Fixes from testing 2024-11-01 15:21:01 -04:00
Paul Lorenz fb809263b3 Simplify raft bootstrapping and controller initialization. Fixes #2212 2024-11-01 15:21:01 -04:00
Paul Lorenz 16f0a858e6 Add controller connect events. Fixes #1835. Fixes #2234 2024-11-01 15:21:01 -04:00
Paul Lorenz cfad18d20f Remove use of ziti CLI fabric identities. Fixes #2388 2024-09-09 10:02:25 -04:00
astroicers f5a8cdf725 Update 004-controller-pki.md
The path description for the Windows hosts file was updated from:

- Windows: `windows: %windir%\system32\drivers\etc\hosts`

to

- Windows: `%windir%\system32\drivers\etc\hosts`

This change removes the redundant "windows:" prefix, making the path clearer and more accurate for users.
2024-08-28 16:32:52 +08:00
Paul Lorenz 77d27ca781 Enhance the router data model with config information, fixes #2010 Add subscription model to router data model, fixes #1990 2024-07-17 14:31:39 -04:00
Kenneth Bingham f6a76242d1 begin demolition and remodeling on linux install scripts 2024-06-14 07:45:52 -04:00
Paul Lorenz 97b9b44171 Spelling fixes from codespell 2024-05-28 13:44:54 -04:00
Kenneth Bingham d244882503 Merge pull request #1968 from openziti/de-hyphenate-ziti-router
de-hyphenate old mentions of ziti-router as 'ziti router'
2024-04-30 10:18:00 -04:00
Kenneth Bingham c78ab3d64d Merge pull request #1941 from openziti/de-hyphenate-ziti-controller
de-hyphenate old mentions of ziti-controller as 'ziti controller'
2024-04-30 09:08:31 -04:00
Kenneth Bingham 23651caf20 revert typos 2024-04-22 17:32:27 -04:00
Kenneth Bingham 57f95d6745 de-hyphenate old mentions of ziti-router as 'ziti router' 2024-04-22 17:30:30 -04:00
Kenneth Bingham 0f84d7c7d7 direct quickstart readers to ziti-doc 2024-04-22 17:13:36 -04:00
Andrew Martinez 0ceb7f2714 spellcheck fixes 2024-04-17 16:13:24 -04:00
Andrew Martinez cbca58023a adds feature flag doc for Go SDK and Edge Routers 2024-04-17 13:51:36 -04:00
Kenneth Bingham d16ec86d10 de-hyphenate old mentions of ziti-controller as 'ziti controller' 2024-04-12 20:21:20 -04:00
Paul Lorenz 103cde6d49 Fix SDK dial error when 'token is malformed'. Api session JWT wasn't being forward to controller. Fixes #1916 2024-04-08 22:42:25 -04:00
Andrew Martinez 8c7b3b2e84 reduces prerms from admin to authenticated on list controllers
- updated public key sync to use controller list data instead of mesh peer
- fixes perms on controller list to not be admin only
2024-03-26 10:55:50 -04:00
Andrew Martinez 09542c7728 consolidated ha changes 2024-02-29 09:27:56 -05:00
Tod 8c09409946 fixing missing flag in controller init cmd (#1677) 2024-01-04 14:51:27 -05:00
Mario Trangoni 126ee5d79c codespell: Add spelling check and fix all issues
Signed-off-by: Mario Trangoni <mjtrangoni@gmail.com>
2023-12-19 10:52:42 +01:00