Commit Graph

25 Commits

Author SHA1 Message Date
Paul Lorenz 444999bc5b Break the SDK terminator removal retry storm. Fixes #4156
- adds RemoveTerminatorsV2, an asynchronous control-channel removal, so a router
  no longer holds a request slot open waiting for a synchronous reply; gated on a
  new capability so routers fall back to the synchronous path against controllers
  that do not advertise it
- paces establish and remove retries with a per-terminator exponential backoff
  after a controller rejection, so a rejected operation is not re-sent on the next
  event-loop tick, which is what let a rate-limited removal sustain itself
- routes v2 removes to the leader and skips the ordered delete for a terminator the
  router confirms it created that no longer exists, so a storm of no-op retries
  cannot consume the command rate limiter
- gates that skip on the leader having applied everything committed when it took
  leadership, since a leader reports itself elected before those applies finish and
  would otherwise read a committed create as absent and drop its delete
- guards the leader term and every write to the caught-up flag with a dedicated
  lock, so a barrier completing alongside a leadership change cannot record a result
  for a term that has already ended
- scopes v2 removals to the requesting router, so a router can only remove
  terminators it owns on the asynchronous path as well as the synchronous one
- tracks in-flight removal batches on the router, resolving or requeueing them when
  the response arrives; lost responses self-heal via rate-limiter expiry and the
  retry scan
- waits for an outstanding operation before sending a terminator delete, and checks
  that before dequeuing, so a delete neither races a create the controller has
  committed but not applied nor falls out of the queue until the retry scan finds it
  minutes later
- matches a create response to the attempt that produced it, by control channel and
  reply sequence, since attempts overlap once one overruns the establishment timeout
  and every attempt carries the same terminator id; only the outstanding attempt's
  response may confirm the create or report the terminator idle
- records the attempt as the sender assigns its sequence, so a response cannot be
  processed before the request it answers is known, and withdraws an attempt whose
  send never queued
- reports a removal that succeeds past the establishment timeout as congestion
  rather than success, matching how the establish path classifies its own latency,
  since both resolve into the same limiter
- extracts the transient-error test both remove handlers share, so the v2 path
  treats a leaderless cluster as busy like v1 does instead of as a permanent
  failure, and the two cannot drift again
- applies the same delete sequencing and response correlation to the router-hosted
  (ER/T) registry, which cleared the same in-flight marker unguarded and dropped a
  deferred delete from its queue rather than leaving it there
- bounds the v2 response send with a timeout, so a router that stops draining its
  send queue cannot accumulate blocked responder goroutines on the controller
2026-09-24 13:24:51 -04:00
Andrew Martinez 948735d86c fixes openziti/ziti#4094 accept first-party certs issued by a separat… (#4140)
* fixes openziti/ziti#4094 accept first-party certs issued by a separate edge signing CA

- adds FirstPartyX509CertValidation and ThirdPartyX509CertValidation usages and an
  intermediates field to the router data model public keys, deprecating
  ClientX509CertValidation
- publishes config CA bundle roots as first-party anchors with their intermediates and
  Ca store entries as third-party anchors; controller certs carry JWT validation only,
  since a controller identity is never a CA and anchors no client cert chains
- builds router first-party and client cert trust pools from the published usages,
  falling back to the deprecated usage against older controllers
- propagates the full signing cert chain between controllers via a new mesh
  SigningCertChainHeader and persists whole chains in controller records
- removes the orphaned InstantStrategy.AddPublicKey, dead since public key sync moved
  to controller list data
- gives each command dispatcher its own decoder registry so multiple in-process
  controllers no longer decode into the last-started controller's managers
- adds a three-controller in-process HA test harness with a split signing PKI,
  cluster formation and first-party cert integration tests
- trusts the edge signing CA when verifying router control channel certs
- adds a variadic additionalRoots parameter to VerifyLeafCertChain, applied to a
  clone of the caller's pool so an identity's live tls.Configs are unaffected
- passes the edge enrollment signing CA bundle as additional roots when admitting
  a router control channel connection, so a deployment whose signing CA sits
  outside the controller's own trust bundle no longer has every router refused
- leaves the fingerprint check bound to the verified leaf, so the wider anchor set
  changes which chains verify, not which routers are admitted
- covers the split-root case and the caller-pool guarantee in common/cert tests
2026-08-26 14:07:31 -04:00
Paul Lorenz 11e049a452 Halt on critical-command apply failure instead of advancing. For #4104
- adds a CriticalCommand marker interface for commands that establish base
  state, and marks SyncSnapshotCommand (a full snapshot restore) as critical
- makes BoltDbFsm.Apply halt when a critical command fails to apply, rather
  than logging the error and persisting the advanced raft index; the failed
  apply's in-tx index update is rolled back and left unpersisted, so raft
  replays and retries the command on restart instead of the node running
  caught-up-on-index but empty-on-data
- attaches the command type to all apply log lines so a failure is
  self-contained
2026-07-17 18:07:50 -04:00
Paul Lorenz 1c122af490 Rewrite channel/v4 imports to channel/v5. For #3983
- moves the channel dependency to channel/v5 v5.0.10 and sdk-golang to v1.9.0 in the root and zititest modules
- mechanically rewrites every channel/v4 import path to channel/v5

This is the import-path-only step; the API-level changes the switch requires land in the following commit. This commit does not build on its own.
2026-06-18 12:51:02 -04:00
Paul Lorenz 4df6ae564d Update package paths for newly imported storage and ziti-db-explorer packages 2026-04-16 09:18:55 -04:00
Paul Lorenz 6b869cea9d Add support for ctrlChanListener on router to the model. Fixes #3635 2026-03-07 00:00:51 -05:00
Paul Lorenz 2ffc6e1151 Update ziti version to 2.0 2026-01-28 12:34:10 -05:00
Paul Lorenz c18a594b56 Clustering coordination fixes
* Allow routers to request current cluster membership information. Fixes #3503
* Get cluster membership information from raft directly, rather than trying to cache it in the DB. Fixes #3501
* Set a router data model timeline when initializing a new HA setup, rather than letting it stay blank. Fixes #3500
* Reduce router data model full state updates. Fixes #3504
2026-01-17 02:03:31 -05:00
Paul Lorenz e647d67325 Update to channel/v4 2025-04-02 15:28:59 -04:00
Paul Lorenz 32eddd61ca HA SDK terminators test. Fixes #2217. Fixes #2533 2024-11-12 18:45:08 -05:00
Paul Lorenz fb809263b3 Simplify raft bootstrapping and controller initialization. Fixes #2212 2024-11-01 15:21:01 -04:00
Paul Lorenz cccf0c06af Update to channel/v3. Fixes #2390 2024-09-09 12:23:25 -04:00
Paul Lorenz 0945427320 Migrated edge router tunneler code to use the router data model. Fixes #2121 2024-07-26 17:06:01 -04:00
Paul Lorenz 8f4f9944f5 Add TLS handshake rate limiter. Fixes #1904 2024-04-09 17:25:35 -04:00
Paul Lorenz fab81e4cd4 Add terminator chaos testing and fix issues found. Fixes #1794 Fixes #1369 2024-03-12 10:31:36 -04:00
Paul Lorenz 9bed8a14a8 Add optional command rate limiter. Fixes #1445 2023-10-23 13:45:49 -04:00
Paul Lorenz f3d67b7f49 Update fabric imports 2023-09-28 23:34:28 -04:00
Paul Lorenz 23110fccb2 Implement entity change events. Fixes openziti/fabric#562 2023-04-25 11:38:03 -04:00
Paul Lorenz f0b83d0010 Track raft index in Db. Let's us start without rerunning all cmds at startup. Fixes #677. Fixes openziti/fabric#582 2023-04-25 11:32:35 -04:00
Paul Lorenz d1d0fe781b Update for storage API changes. 2023-04-25 11:32:33 -04:00
Paul Lorenz abb0eb96fb Add more raft configuration options. Fixes openziti/fabric#597
Add more raft config knobs
Refactor peer handles to be consistent with other handlers
Allow inspect to work across controllers
Add config and clusterconfig inspect support
2023-02-15 23:47:57 -05:00
Paul Lorenz b5bae84325 Fix some handler removal. Only propagate crud terminator events on leader by default. Fixes openziti/fabric#573 2023-01-04 09:53:36 -05:00
Paul Lorenz 521c5d2be1 Fixes for command encoding and decoding 2022-08-11 11:24:23 -04:00
Paul Lorenz b26fcf8909 Fix copyright 2022-06-29 16:59:46 -04:00
Paul Lorenz eaa75d48c6 Initial support for distributed model via RAFT 2022-06-02 22:49:55 -04:00