- turns the oidc capability from a config switch into a test gate: OIDC stays on, as a 2.x
controller runs it by default, and tests skip SDK client and SDK-hosted paths through routers older
than 1.6, whose OIDC handling had many bugs that 1.6 fixed
- stops defaulting the oidc flag off for old routers, which did nothing once the controller started
binding OIDC by default
- requires OIDC on the client routers for SDK clients, and on the hosting routers alongside SDK hosting
for SDK-hosted services; ERT client and ERT-hosted paths always run
- adds Support.And for checks that span both ends of a path
- adds an optional labels input to the compat workflow's manual dispatch, appended to every job's
fablab labels, so a run can set capability overrides such as capability_oidc=on
- passes the input to the script through the environment rather than interpolating it
- adds a router capability table to the smoke model: controller-managed link config, SDK hosting
against a 2.x controller, and OIDC api sessions, each with the first release that has it
- keeps link config local on routers without managed link config, and defaults OIDC off when any
edge router lacks it, though an oidc label still decides
- adds capability_<name>=on|off labels that override the version check for every router, so a run
can test a floor or exercise the skips on the local build
- skips the SDK-hosted file, iperf, scp and sim tests with the reason when the hosting routers lack
SDK hosting, and leaves those services out of the terminator waits
- tests the version and override checks
- adds a scheduled sweeper that terminates CI fablab instances (Name gh-*) launched more than 7 hours
ago in every enabled region and deletes their VPCs; EC2 records no VPC creation time, so it ages
each VPC from a tag it writes on first sight, and it reports every find as a workflow warning
- adds a stop-fablab action that interrupts the fablab command and terraform a cancelled step left
running, and waits for terraform to save its state; the smoketest, validation, upgrade and
backward compatibility jobs run it before they archive or tear down their environment
- moves the housekeeper's VPC teardown into a file the sweeper shares, and waits on instances already
shutting down before deleting the subnet they hold
- fixes the HA smoketest teardown's skip condition, which named a job that doesn't exist
- runs the compat matrix against the 2.0 and 1.6 lines, with the old routers on the client side
and, separately, on the hosting side
- resolves each line's newest stable release at run time instead of pinning versions
- makes the compat workflow callable and runs it first in Validation (All); the soak validations
start once it finishes, whatever its result, so the two sets of environments don't share
us-east-1's VPC quota
- keeps link config local on smoke test routers older than the first release that applies
controller-managed link config, since they ignore it and would start with no links
* The controller and the router wrote the complete service session token, an OIDC
service session JWT or a legacy session token, into structured log fields during
edge control operations. Several of the sites logged at Error or Info, so the
values reached logs at default verbosity during ordinary operation.
* Adds LoggingFields to the session and api session types in common, controller/model
and router/state, so a logging site asks for the session fields or the api session
fields instead of picking values by hand. Ids, type, identity, service and the
token's validity window go in; the token does not. Empty values are dropped, so the
sets merge over each other.
* AddLoggingFields in router/state now delegates to LoggingFields. Its output moves
from a nested field to flat prefixed keys.
* The edge control handlers log the session field set on the shared returnError and
logResult paths, on circuit setup and on terminator update. The terminator removal
paths log the terminator id, since they never load a session.
* Adds unit tests over returnError and logResult that fail if a session token reaches
a log entry, covering the loaded-session, failed-to-load and JWT-claims cases.
- constrains the cpu profiler and router signal monitor to `unix` instead of a list of operating
systems, and turns their windows stubs into `!unix` fallbacks
- replaces the per-OS tproxy and geneve listener stubs with one fallback each, covering every
operating system without an implementation
- replaces survey with local stdin prompts for the pki pickers, since survey's terminal package does
not build on solaris or illumos; an exact name takes precedence over a list number
- limits the interactive database explorer to the platforms its prompt library builds on, and
reports it as unsupported elsewhere
- removes the survey dependency and syncs zititest's go.mod
- tests the pickers
- notes the fix in the changelog
- collects the subscribed identity ids under IterCb when syncing all subscribers and checks each one
after the iteration returns, rather than inside the callback; a deleted or disabled identity closes
its SDK channel, whose close handler unsubscribes through RemoveCb on the same key and
self-deadlocked on the shard lock
- clears each identity's update flag before looking up its live subscription, so a subscription
replaced and updated mid-sync is checked instead of the one collected
- clears the update flag before the subscription lookup in syncSubscriptionIfRequired, for the same
reason
- tests that a subscriber unsubscribing from inside its deleted notification lets the sync complete
- tests that a subscription replaced and updated during the sync delivers the update to its new
listener
- notes the fix in the changelog
- closes a disabled router's control channel after the connected map's RemoveCb returns, rather than
inside its callback; the channel's close handler runs DisconnectRouter synchronously, which reads
and removes the same map entry and self-deadlocked on the shard lock, wedging the raft apply
goroutine and every path selection behind it
- logs any error from closing the channel
- adds godoc for UpdateCachedRouter
- tests that disabling a connected router completes and tears the connection down
- notes the fix in the changelog
- replaces the link dialer connectTimeout duration, which older channel options parsers do not recognize, with connectTimeoutMs so dials get 30s instead of the 5s default
- reads the sim's tx.byterate gauge, which was previously dropped as an unknown metric
- asserts throughput on the last non-zero byterate per service instead of the tx.bytes meter's one-minute EWMA, which never converges for a transfer shorter than a minute and reads low depending on where the burst lands against the sampling ticks
- fails a throughput service that reports no transfer rate
- logs the byterate alongside the EWMA per service, and includes it in the failure dump
- tests the gauge parsing and the last non-zero selection
- diagnoses a short link count by unordered router pair, since links are single-dialer, so only pairs with no link in either direction are reported missing and duplicates are found in either direction
- caps missing-pair log output and summarizes link, pair, duplicate and missing counts
- retries a failed link count query until the deadline instead of treating it as success, extending the deadline while the controller host is unreachable
- collects routers the controller could not reach during link validation instead of failing on the first, reporting inconsistent links ahead of unreachable routers
- names the expected link count as a constant
- tracks the link listener set owed to each controller in the link registry, marked on a listener change and after each reconnect
- sends the listener set from the registry loop with wait-for-wire once the controller's reconnect link announcement is written, retrying every tick until the write succeeds, replacing the fire-and-forget send to controllers connected at the time of the change
- fences listener completions with the reconnect generation so a send that reached an earlier connection does not count for a later one
- keeps sync state for controllers absent from a controller snapshot while they still owe the listener set
- stamps the listener set sent to controllers with the config generation read in the same snapshot, matching the hello
- makes the controller skip the peer fan-out for an UpdateLinkListeners message whose set matches the one it replaced, still recording the message generation
- makes Router.SetLinkListeners return the set it held, read under the same lock as the update
- closes a dialed link whose destination or link state is absent from the registry, where it was previously only logged, so a link this router can never report, fault or close does not survive
- skips dialing a queued link state whose destination is no longer registered, since removing a destination leaves its linkMap intact and the existing liveness check cannot see the removal
- leaves an accepted link alone, as the listener side holds no state for one, and leaves the existing fault path for a closed detached link unchanged
- logs each link pairing detached during listener reconciliation, with the reporting controller and whether a peer update or local rescan caused it
- passes the reporting controller id through UpdateLinkDest from the peer state handler
- tests the close, the accepted-link exemption, and the removed destination not being dialed
- treats an unbound payload or ack channel as open in splitImpl.IsClosed, matching Close, which has always skipped a nil channel
- fixes a nil dereference on a split link whose channels are still being bound, since the listener sets the two independently as each arrives
- tests that checking a link with neither channel bound does not panic
- tracks a full link refresh owed to each controller after a reconnect, sent from the link registry loop through the rate limiter pool and retried every tick until written, instead of giving up after three attempts
- holds incremental new-link reports and faults to a connected controller until its refresh is written, so they cannot be pruned by or race ahead of the refresh, while disconnected controllers keep their existing skip and fault expiry handling
- fences refresh completions with a reconnect generation so a refresh written to an earlier connection does not satisfy a later one
- keeps sync state for controllers absent from a controller snapshot while they still owe a refresh
- snapshots the link set under the registry lock and sends outside it
- stops raft before closing the controller database on shutdown, so an in-flight apply cannot fail against a closed store and halt the process
- uses the nil-safe getApiSessionId() in loadServiceForName, matching loadServiceForId; the v2
tunnel circuit handler never establishes an api session, so the lookup-failure branch
dereferenced a nil apiSession and took the controller down on any unresolvable service name
- makes the default admin check log a read failure and return instead of panicking; the check is
advisory and runs in a goroutine that can observe a datastore the shutdown path has closed,
turning a controlled exit into exit 2
- adds godoc for checkEdgeInitialized covering the advisory contract and the remaining fatal case
- notes both in the changelog
- moves the identity and service id lookups in createServicePolicies inside the
parallel task, so a transient failure is retried by the shared retry policy
rather than ending the action; they previously ran serially ahead of the tasks
with no retry, so one expired API session failed the whole run
- records why the create concurrency is left low, since a wider fan-out draws
429s and timeouts from this model's controller that the retry policy cannot
absorb
- corrects the service policy task type and label, which reported policy creation
as service creation
- sets the model defaults through the promoted field rather than naming the
embedded scope
- syncs zititest's go.mod and go.sum, which dependency update automation does not
reach, so the module had drifted from the versions it actually resolves
- adds RemoveTerminatorsV2, an asynchronous control-channel removal, so a router
no longer holds a request slot open waiting for a synchronous reply; gated on a
new capability so routers fall back to the synchronous path against controllers
that do not advertise it
- paces establish and remove retries with a per-terminator exponential backoff
after a controller rejection, so a rejected operation is not re-sent on the next
event-loop tick, which is what let a rate-limited removal sustain itself
- routes v2 removes to the leader and skips the ordered delete for a terminator the
router confirms it created that no longer exists, so a storm of no-op retries
cannot consume the command rate limiter
- gates that skip on the leader having applied everything committed when it took
leadership, since a leader reports itself elected before those applies finish and
would otherwise read a committed create as absent and drop its delete
- guards the leader term and every write to the caught-up flag with a dedicated
lock, so a barrier completing alongside a leadership change cannot record a result
for a term that has already ended
- scopes v2 removals to the requesting router, so a router can only remove
terminators it owns on the asynchronous path as well as the synchronous one
- tracks in-flight removal batches on the router, resolving or requeueing them when
the response arrives; lost responses self-heal via rate-limiter expiry and the
retry scan
- waits for an outstanding operation before sending a terminator delete, and checks
that before dequeuing, so a delete neither races a create the controller has
committed but not applied nor falls out of the queue until the retry scan finds it
minutes later
- matches a create response to the attempt that produced it, by control channel and
reply sequence, since attempts overlap once one overruns the establishment timeout
and every attempt carries the same terminator id; only the outstanding attempt's
response may confirm the create or report the terminator idle
- records the attempt as the sender assigns its sequence, so a response cannot be
processed before the request it answers is known, and withdraws an attempt whose
send never queued
- reports a removal that succeeds past the establishment timeout as congestion
rather than success, matching how the establish path classifies its own latency,
since both resolve into the same limiter
- extracts the transient-error test both remove handlers share, so the v2 path
treats a leaderless cluster as busy like v1 does instead of as a permanent
failure, and the two cannot drift again
- applies the same delete sequencing and response correlation to the router-hosted
(ER/T) registry, which cleared the same in-flight marker unguarded and dropped a
deferred delete from its queue rather than leaving it there
- bounds the v2 response send with a timeout, so a router that stops draining its
send queue cannot accumulate blocked responder goroutines on the controller
- applies the router.link.v1 heartbeat settings (sendInterval, checkInterval,
closeUnresponsiveTimeout) and payload/ack sender queue sizes at runtime,
replacing the hardcoded 10s/1s link heartbeat intervals and the ignored fields;
new links take them at bind time, established links are re-tuned in place
through the channel HeartbeatControl, and queue sizes reach new links only
- publishes heartbeat settings as an immutable generation in one atomic swap, so
no reader pairs one generation's interval with another's timeout, and has each
link enforce the timeout belonging to its own intervals
- pushes a generation and its intervals as one step under the link's lock and
ignores a generation at or below the one applied, so an overtaken update cannot
roll a link back; reconciles every channel of a link, including a split link
whose channels bind on either side of a config change, to the newest generation
- adopts the first registered settings explicitly, since a router with no managed
link config publishes its defaults at generation 0
- re-arms a link's liveness deadline when its generation changes, so age
accumulated under the previous intervals cannot close a healthy link when a
timeout is shortened
- honors a closeUnresponsiveTimeout below 30s, which previously sat behind the
unhealthy threshold; treats a zero timeout as closing nothing, and reports a
closing link unhealthy
- scales the unhealthy threshold with the intervals, to max(30s, 2 x (send +
check)): reporting unhealthy poisons the latency metric the controller routes
on, so a fixed 30s priced healthy links with a long send interval out of
routing every cycle
- validates heartbeat timing on the controller and the router: durations must be
positive, their sum representable, and the timeout at least 10s and greater
than sendInterval + checkInterval; a timeout under two heartbeat cycles is
accepted with a warning
- adds a per-config-type semantic validator registry to ConfigManager, run after
schema validation, and moves the router.link config definition to
common/config/routerlink so the controller applies the same rules
- treats a link section in router YAML, of any content, as local config that
takes precedence over the controller's for the whole router.link type, since
the loader defaults every field and so now records presence separately; warns
at startup when that section has no listeners or dialers
- translates local heartbeat timing leniently, since a local failure stops the
router from starting: keeps an explicit zero, raises a timeout below the floor
to it, and falls back to the defaults for other invalid timing, each with a
warning
- stamps UpdateLinkListeners with the config generation it came from and has the
controller drop superseded sets, which a multi-underlay control channel can
deliver out of order and which would otherwise send peers to withdrawn
addresses
- logs every successful link config apply, including those that dispatch no work,
and pins that a queue-size-only change triggers no GC pass
- deprecates split links, documents the feature in the changelog and the router
config design doc, and brings both up to date
Builds on the channel/v5 HeartbeatControl from openziti/channel#265.
- documents the release: edited trigger that re-points :latest without rebuilding
- documents that :latest and CloudFront only follow GitHub's Latest release, and the force_latest/force_cloudfront overrides
- corrects the image tag format to X.Y.Z (no leading v)
- notes that each release branch must carry the workflow
- removes the minSize lower bound that rejected the router's own default of 1,
which made any ctrl.rateLimiter stanza omitting minSize fail config load
- leaves CtrlRateLimiterMinSizeValue alone, since it also bounds maxSize, and
documents that it does not bound minSize
- relies on AdaptiveRateLimiterConfig.Load, which already rejects minSize below 1
or above maxSize
- tests the stanza shapes that used to fail and the bounds that must still reject
- adds edgeTerminator.establishStart so the post-establish inspect is timed from
the bind the sdk is waiting on rather than the last create re-send, which reset
the clock and skipped the re-inspect for exactly the stalled binds it covers
- gives that gate its own threshold, anchored to the sdk's one minute bind
deadline rather than to controller load
- brings router-hosted terminators to parity: they resolve a superseded rate
limit control instead of orphaning its slot, classify establishment latency
rather than reporting success regardless, and treat a send timeout on the
remove reply as congestion
- shares one establishment timeout between both registries, which resolve into
the same limiter and would otherwise feed one window inconsistent signals
- warns at config load when ctrl.rateLimiter.timeout undercuts that threshold,
where the limiter expires work as congestion before the router can classify it
- describes the remove-path classification accurately: a send-queue timeout is a
busy channel and it backs off
- tests markEstablishedEvent.handle at the call site, pinning the two clocks
against each other
- measures terminator establishment latency from the last attempt, reporting Backoff to the controller rate limiter at or over the establishment timeout and Success below it, so a stalled establish shrinks the window instead of counting as a success
- resolves a superseded establishment's outstanding rate-limit control with Backoff when a stalled attempt is re-sent, rather than orphaning it
- treats a synchronous terminator-removal reply timeout as congestion (Backoff) instead of a neutral failure, so removals shed load when the controller is slow to respond
* fixopenziti/ziti#4413
replaces simple stdin check in the current router and controller postinstall
scripts with typical debconf mechanisms
* change default response for the Configure? diaglog to "true" to maintain
behavioral consistency
Paul Lorenz