mirror of
https://github.com/openziti/ziti.git
synced 2026-10-23 12:15:35 +00:00
Merge pull request #4215 from openziti/router-link-runtime-config
Apply managed link heartbeat and queue settings at runtime
This commit is contained in:
Paul Lorenz
+111
-10
@@ -8,9 +8,8 @@
|
||||
* [Cluster Quorum Recovery](#cluster_quorum_recovery) - A mechanism for recovering clusters that have irrevocably lost the ability to form a quorum
|
||||
* [Quickstart Cluster](#quickstart-cluster) - `ziti run quickstart cluster` brings up a multi-node HA cluster in a single command for testing and development and learning
|
||||
* [Fully Connected Controller Mesh](#fully-connected-controller-mesh) - Controllers now proactively keep the cluster mesh fully connected
|
||||
* [Config Type Target Field](#config-type-target-field) - Config types now have a target field indicating whether they apply to services, routers or other entities
|
||||
* [Wildcard OIDC Issuers](#wildcard-oidc-issuers) - Controllers with a wildcard server-certificate SAN can serve OIDC for explicitly allow-listed hostnames
|
||||
* [Router Configs](#router-configs) - Allow routers to have a list of associated configs
|
||||
* [Controller Managed Router Configuration](#controller-managed-router-configuration) - (beta) Routers can take configuration from the controller and apply it at runtime, starting with link listeners, dialers and heartbeats, with automatic collection of links the new config leaves stale
|
||||
* [Multiple LAN Interfaces for tproxy](#multiple-lan-interfaces-for-tproxy) - `lanIf` now accepts a single interface or a list of interfaces
|
||||
* [Multiple Resolver Addresses for tproxy](#multiple-resolver-addresses-for-tproxy) - `resolver` now accepts a single address or a list of addresses
|
||||
* [DNS Upstream Query Modes](#dns-upstream-query-modes) - choose how multiple DNS upstreams are queried: parallel fan-out (default) or serial fail-through
|
||||
@@ -263,10 +262,8 @@ Config types now have an optional `target` field that indicates what kind of ent
|
||||
is intended for. Valid values are `"service"`, `"router"`, and `"other"`. The field is set on creation
|
||||
and is immutable afterward.
|
||||
|
||||
This is the first step toward controller-managed router configuration. The `target` field lets us
|
||||
distinguish between config types meant for services, config types meant for routers, and config types
|
||||
meant for other purposes, which keeps UIs, APIs, and validation clean. See
|
||||
`doc/design/ctrl-managed-router-config.md` for the full design.
|
||||
The `target` field distinguishes config types meant for services, config types meant for routers,
|
||||
and config types meant for other purposes, which keeps UIs, APIs, and validation clean.
|
||||
|
||||
A database migration sets `target = "service"` on all existing config types. Services and identity
|
||||
service config overrides now require that referenced configs have a config type with
|
||||
@@ -280,8 +277,7 @@ The CLI has been updated to support the new field:
|
||||
### Router Configs
|
||||
|
||||
Routers (edge, transit, and fabric) now have a `configs` field that holds a list of config IDs the
|
||||
router should use. This is the second step toward controller-managed router configuration: routers
|
||||
can now be associated with configs in the same way services already can.
|
||||
router should use, associating routers with configs in the same way services already are.
|
||||
|
||||
Validation rules:
|
||||
|
||||
@@ -344,9 +340,45 @@ rolled back raises an alert to the controller rather than leaving the router in
|
||||
`router.link.v1` is the first controller-managed config type, covering a router's link listeners and
|
||||
dialers. It is built in, so no config type has to be created before use.
|
||||
|
||||
A config looks like this:
|
||||
|
||||
```json
|
||||
{
|
||||
"listeners": [
|
||||
{
|
||||
"binding": "transport",
|
||||
"bind": "tls:0.0.0.0:6004",
|
||||
"advertise": "tls:router-east.example.com:6004",
|
||||
"groups": ["east"]
|
||||
}
|
||||
],
|
||||
"dialers": [
|
||||
{ "binding": "transport", "groups": ["east", "west"] }
|
||||
],
|
||||
"heartbeats": { "sendInterval": "10s", "checkInterval": "1s", "closeUnresponsiveTimeout": "60s" },
|
||||
"gcMode": "orphaned"
|
||||
}
|
||||
```
|
||||
|
||||
To put one on a router, create the config, attach it, and allow the type in the router's own config
|
||||
file (see above):
|
||||
|
||||
```
|
||||
ziti edge create config east-links router.link.v1 "$(cat east-links.json)"
|
||||
ziti edge update edge-router router-east --config east-links
|
||||
```
|
||||
|
||||
Applying one reconciles the router's link surface in place: listeners whose definitions changed are
|
||||
rebuilt, listeners that are unchanged are left running, and dialers are replaced. A change confined
|
||||
to fields that do not describe a listener or dialer does not disturb either.
|
||||
to fields that do not describe a listener or dialer does not disturb either. Every successful apply
|
||||
is logged at info level as `applied link configuration`, with the generation and what changed, so
|
||||
you can confirm a router received a config even when the change asked nothing of its links.
|
||||
|
||||
For links, local config means the router's config file has a `link` section at all. Anything in
|
||||
it, even only heartbeat or queue settings, takes precedence over the controller's `router.link`
|
||||
config for the whole type. A `link` section with no listeners or dialers therefore leaves the
|
||||
router with no links, and the router warns at startup when that happens. To hand a router's links to
|
||||
the controller, remove its `link` section.
|
||||
|
||||
Removing the config from a router removes its link surface entirely.
|
||||
|
||||
@@ -381,6 +413,43 @@ A link is only closed when both of its endpoints agree it is stale and both judg
|
||||
incarnation of it. An endpoint that is offline, too old to answer, or outside `--filter` leaves the
|
||||
link reported as partial and never closes it.
|
||||
|
||||
### Link Heartbeat and Queue Settings
|
||||
|
||||
`router.link.v1` also carries the link heartbeat timing (`sendInterval`, `checkInterval`,
|
||||
`closeUnresponsiveTimeout`) and the `payloadSenderQueueSize` / `ackSenderQueueSize` queue sizes.
|
||||
|
||||
Heartbeat timing is applied to links that are already established, retuning them in place, rather
|
||||
than only reaching links formed after the change. It is applied as a single generation, so a link
|
||||
never pairs one version's interval with another version's timeout.
|
||||
|
||||
**Queue sizes apply to new links only.** `payloadSenderQueueSize` and `ackSenderQueueSize` are read
|
||||
when a link is built, so changing them has no effect on links that are already established. Those
|
||||
keep the queues they were built with until they are re-established for some other reason. A change
|
||||
confined to queue sizes does not trigger stale-link collection, since nothing about an existing link
|
||||
has become stale.
|
||||
|
||||
The timing is validated on both the controller and the router, and some combinations that were
|
||||
accepted before are now rejected. `closeUnresponsiveTimeout` must be greater than `sendInterval +
|
||||
checkInterval`, because a responsive link's own heartbeat gap already reaches that sum, and it must
|
||||
be at least 10s.
|
||||
|
||||
**A `closeUnresponsiveTimeout` under 30s now takes effect.** The close check previously sat behind a
|
||||
fixed 30s threshold, so any shorter timeout silently behaved as 30s. A router configured with a
|
||||
short timeout will start closing unresponsive links sooner than it did before.
|
||||
|
||||
The 10s minimum is there because a healthy connection can legitimately go quiet for several seconds
|
||||
while TCP recovers from ordinary packet loss, and a timeout inside that window turns routine loss
|
||||
into a closed link and the cost of re-establishing it. A controller config below the minimum is
|
||||
rejected. A value below it in a router's own YAML is raised to the minimum with a warning instead,
|
||||
so a router that started yesterday still starts today.
|
||||
|
||||
**Link `sendInterval` and `checkInterval` in a router's own YAML now take effect.** Links
|
||||
previously ran a fixed 10s send and 1s check interval whatever the config said, reading only
|
||||
`closeUnresponsiveTimeout`. Because those values were never in force, local timing that fails the
|
||||
validation above is not allowed to stop a router from starting: the router logs a warning and uses
|
||||
the defaults instead. That includes an explicit `0s`, which was previously treated as if the field
|
||||
were absent.
|
||||
|
||||
## Wildcard OIDC Issuers
|
||||
|
||||
Controllers can serve OIDC for hostnames covered by a wildcard server-certificate SAN. Prior to 2.1,
|
||||
@@ -634,6 +703,15 @@ in a future release. Plan to migrate off them.
|
||||
sessions, is deprecated and will be removed in OpenZiti 3.0, along with the
|
||||
supporting infrastructure built around it.
|
||||
|
||||
* **Split links** - The split-channel link implementation (separate payload and
|
||||
ack connections) is a fallback used only when dialing a router that doesn't
|
||||
support multi-underlay links. The multi-underlay link implementation is a
|
||||
superset of what split links provide, so nothing is lost by moving off them.
|
||||
The deprecation proceeds in two steps: in OpenZiti 3.0 the `split` option will
|
||||
be interpreted as a multi-underlay link with one ack and one payload channel,
|
||||
retiring the separate split implementation; in OpenZiti 4.0 the `split`
|
||||
configuration option will be removed entirely.
|
||||
|
||||
## Removed Features
|
||||
|
||||
* **Link `costTags`** - The `costTags` option on router link listeners has been
|
||||
@@ -676,7 +754,7 @@ Thanks to the community members who contributed to this release.
|
||||
* [Issue #241](https://github.com/openziti/channel/issues/241) - Allow calling LoadOptions on an Options instance
|
||||
* [Issue #238](https://github.com/openziti/channel/issues/238) - Channelv5
|
||||
|
||||
* github.com/openziti/edge-api: [v0.31.0 -> v0.35.2](https://github.com/openziti/edge-api/compare/v0.31.0...v0.35.2)
|
||||
* github.com/openziti/edge-api: [v0.31.0 -> v0.36.0](https://github.com/openziti/edge-api/compare/v0.31.0...v0.36.0)
|
||||
* [Issue #198](https://github.com/openziti/edge-api/issues/198) - Advertise edge router capabilities in the service edge-router list
|
||||
|
||||
* github.com/openziti/foundation/v2: [v2.0.91 -> v2.0.100](https://github.com/openziti/foundation/compare/v2.0.91...v2.0.100)
|
||||
@@ -716,7 +794,30 @@ Thanks to the community members who contributed to this release.
|
||||
|
||||
* github.com/openziti/xweb/v3: [v3.0.4 -> v3.0.5](https://github.com/openziti/xweb/compare/v3.0.4...v3.0.5)
|
||||
* github.com/openziti/ziti/v2: [v2.0.0 -> v2.1.0](https://github.com/openziti/ziti/compare/v2.0.0...v2.1.0)
|
||||
* [Issue #4378](https://github.com/openziti/ziti/issues/4378) - Api session enforcer delete meter only marks when the batch delete fails
|
||||
* [Issue #4434](https://github.com/openziti/ziti/issues/4434) - Update to Go 1.27
|
||||
* [Issue #4413](https://github.com/openziti/ziti/issues/4413) - debian controller & router packages hang on post-install script when installed via ansible
|
||||
* [Issue #4410](https://github.com/openziti/ziti/issues/4410) - REST error responder logs ApiError.Code as a method value
|
||||
* [Issue #4349](https://github.com/openziti/ziti/issues/4349) - Router process check reports configured hashes as the expected signer fingerprints
|
||||
* [Issue #4348](https://github.com/openziti/ziti/issues/4348) - Single-process posture check with no signer fingerprint always fails at the router
|
||||
* [Issue #4005](https://github.com/openziti/ziti/issues/4005) - Auto-GC stale router links
|
||||
* [Issue #4246](https://github.com/openziti/ziti/issues/4246) - Router connect scans the entire link table, making a reconnect wave scale with the cube of the router count
|
||||
* [Issue #4365](https://github.com/openziti/ziti/issues/4365) - REST requests no longer reset the legacy API session timeout
|
||||
* [Issue #4350](https://github.com/openziti/ziti/issues/4350) - Policy enforcers leak a metrics reference on every run
|
||||
* [Issue #4369](https://github.com/openziti/ziti/issues/4369) - Router control channel connectivity events can be delivered out of order
|
||||
* [Issue #4323](https://github.com/openziti/ziti/issues/4323) - Router posture cache grows without bound as clients re-authenticate
|
||||
* [Issue #4321](https://github.com/openziti/ziti/issues/4321) - Process hashes and signer fingerprints are compared case-sensitively at the router
|
||||
* [Issue #4314](https://github.com/openziti/ziti/issues/4314) - Router posture cache corrupts process posture state when applying posture responses
|
||||
* [Issue #4318](https://github.com/openziti/ziti/issues/4318) - Mac and Domain posture checks panic when the client has not reported that data
|
||||
* [Issue #4319](https://github.com/openziti/ziti/issues/4319) - OS posture check with no versions fails at the router, passes at the controller
|
||||
* [Issue #4320](https://github.com/openziti/ziti/issues/4320) - Router does not normalize MAC addresses from posture responses
|
||||
* [Issue #4222](https://github.com/openziti/ziti/issues/4222) - OIDC cert authentication always reports isCertExtendable as true, including for 3rd Party CA certificates
|
||||
* [Issue #4094](https://github.com/openziti/ziti/issues/4094) - Edge router rejects valid first-party client certificates when the edge signing CA differs from the ctrl-channel CA
|
||||
* [Issue #4304](https://github.com/openziti/ziti/issues/4304) - Support build time provided capabilities in the /version capabilities list
|
||||
* [Issue #4296](https://github.com/openziti/ziti/issues/4296) - Bind message with a short cost header panics the router
|
||||
* [Issue #4199](https://github.com/openziti/ziti/issues/4199) - Add in-place upgrade smoke test
|
||||
* [Issue #4291](https://github.com/openziti/ziti/issues/4291) - Admin MFA removal doesn't replicate across HA cluster members
|
||||
* [Issue #4202](https://github.com/openziti/ziti/issues/4202) - wss edge listener rejects all clients on 2.0: certValidatingIdentity forces client-cert verification on a listener that intentionally sets NoClientCert
|
||||
* [Issue #4184](https://github.com/openziti/ziti/issues/4184) - Router leaks LinkSendBuffer goroutines in `drainDeadlines()` — circuits accumulate until the router OOMs
|
||||
* [Issue #4278](https://github.com/openziti/ziti/issues/4278) - fabric inspect data-model-index doesn't move for writes outside the router data model
|
||||
* [Issue #4196](https://github.com/openziti/ziti/issues/4196) - Router control-channel connect/disconnect race can leave a reconnected router de-registered
|
||||
|
||||
@@ -0,0 +1,345 @@
|
||||
/*
|
||||
Copyright NetFoundry Inc.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
https://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package routerlink
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"math"
|
||||
"time"
|
||||
|
||||
"github.com/openziti/channel/v5"
|
||||
)
|
||||
|
||||
// DefaultCloseUnresponsiveTimeout is the fallback when a config supplies no
|
||||
// heartbeats.closeUnresponsiveTimeout, matching the router env default
|
||||
// (DefaultLinkUnresponsiveTimeout).
|
||||
const DefaultCloseUnresponsiveTimeout = time.Minute
|
||||
|
||||
// Heartbeat interval fallbacks, matching the channel package defaults, used when
|
||||
// a config supplies no heartbeats.sendInterval / checkInterval.
|
||||
const (
|
||||
DefaultHeartbeatSendInterval = 10 * time.Second
|
||||
DefaultHeartbeatCheckInterval = time.Second
|
||||
)
|
||||
|
||||
// MinCloseUnresponsiveTimeout is the floor for heartbeats.closeUnresponsiveTimeout.
|
||||
// A healthy connection can go quiet for several seconds while TCP recovers from
|
||||
// ordinary loss, and a shorter timeout closes links that are only doing that.
|
||||
const MinCloseUnresponsiveTimeout = 10 * time.Second
|
||||
|
||||
// ConfigBaseType is the un-versioned config family this package owns.
|
||||
const ConfigBaseType = "router.link"
|
||||
|
||||
// ConfigTypeV1 is the controller-defined ConfigType.Name for version 1.
|
||||
const ConfigTypeV1 = "router.link.v1"
|
||||
|
||||
// Config is the typed view of router.link.v1 JSON. Field tags match the
|
||||
// schema property names. Strings rather than time.Duration are used for
|
||||
// duration fields so they round-trip cleanly into the existing
|
||||
// channel.LoadOptions parser, which already understands "30s" / "100ms".
|
||||
type Config struct {
|
||||
Listeners []ListenerConfig `json:"listeners,omitempty"`
|
||||
Dialers []DialerConfig `json:"dialers,omitempty"`
|
||||
Heartbeats *HeartbeatsConfig `json:"heartbeats,omitempty"`
|
||||
PayloadSenderQueueSize int `json:"payloadSenderQueueSize,omitempty"`
|
||||
AckSenderQueueSize int `json:"ackSenderQueueSize,omitempty"`
|
||||
// GcMode is the auto-GC policy for links this router's own configuration
|
||||
// has made stale: "preserve" (default, never close on that basis),
|
||||
// "orphaned" (close links whose supporting listener/dialer is entirely
|
||||
// gone), or "changed" (close links whose details have shifted). Empty
|
||||
// string is treated as "preserve".
|
||||
//
|
||||
// The policy covers local staleness only. A peer withdrawing or moving a
|
||||
// listener still closes the affected link at once, under every mode,
|
||||
// because the peer's advertised listener set is authoritative about what
|
||||
// it will accept. "preserve" is therefore not a guarantee that links
|
||||
// survive every config change.
|
||||
GcMode string `json:"gcMode,omitempty"`
|
||||
}
|
||||
|
||||
// GcMode names the auto-GC policy applied by the router after each
|
||||
// successful link-config Apply. Mirrors the CLI `--mode` for
|
||||
// `ziti ops verify stale-links`, plus a `Preserve` value that disables
|
||||
// local-staleness GC entirely. It governs only links made stale by this
|
||||
// router's own configuration; peer-driven closures are unaffected.
|
||||
type GcMode int
|
||||
|
||||
const (
|
||||
GcModePreserve GcMode = iota
|
||||
GcModeOrphaned
|
||||
GcModeChanged
|
||||
)
|
||||
|
||||
func (m GcMode) String() string {
|
||||
switch m {
|
||||
case GcModeOrphaned:
|
||||
return "orphaned"
|
||||
case GcModeChanged:
|
||||
return "changed"
|
||||
default:
|
||||
return "preserve"
|
||||
}
|
||||
}
|
||||
|
||||
// ParseGcMode normalizes the string form (as it appears in the JSON
|
||||
// config or local YAML) into the enum. Unknown values return
|
||||
// GcModePreserve and an error; the caller decides whether to fall back
|
||||
// or reject the config.
|
||||
func ParseGcMode(s string) (GcMode, error) {
|
||||
switch s {
|
||||
case "", "preserve":
|
||||
return GcModePreserve, nil
|
||||
case "orphaned":
|
||||
return GcModeOrphaned, nil
|
||||
case "changed":
|
||||
return GcModeChanged, nil
|
||||
default:
|
||||
return GcModePreserve, fmt.Errorf("unknown gcMode %q (expected preserve|orphaned|changed)", s)
|
||||
}
|
||||
}
|
||||
|
||||
// ListenerConfig matches the schema's listener entry. Bind is the only
|
||||
// required field.
|
||||
type ListenerConfig struct {
|
||||
Binding string `json:"binding,omitempty"`
|
||||
Bind string `json:"bind"`
|
||||
Advertise string `json:"advertise,omitempty"`
|
||||
BindInterface string `json:"bindInterface,omitempty"`
|
||||
Groups Groups `json:"groups,omitempty"`
|
||||
Options *ChannelOptions `json:"options,omitempty"`
|
||||
}
|
||||
|
||||
// DialerConfig matches the schema's dialer entry.
|
||||
//
|
||||
// Split is carried here for local-config fidelity only. The router's own YAML
|
||||
// may set the transport's `split` dialer flag (the legacy split-vs-single
|
||||
// fallback used when dialing a router too old for multi-underlay links);
|
||||
// dropping it on translation would silently flip an explicit `split: false`
|
||||
// back to the transport default of true. It is deliberately absent from the
|
||||
// controller-managed router.link schema, so it cannot be set from the
|
||||
// controller. A nil pointer means "unset", letting the transport default apply.
|
||||
type DialerConfig struct {
|
||||
Binding string `json:"binding,omitempty"`
|
||||
MaxDefaultConnections int `json:"maxDefaultConnections,omitempty"`
|
||||
MaxAckConnections *int `json:"maxAckConnections,omitempty"`
|
||||
Split *bool `json:"split,omitempty"`
|
||||
StartupDelay string `json:"startupDelay,omitempty"`
|
||||
BindInterface string `json:"bindInterface,omitempty"`
|
||||
Groups Groups `json:"groups,omitempty"`
|
||||
HealthyDialBackoff *BackoffConfig `json:"healthyDialBackoff,omitempty"`
|
||||
UnhealthyDialBackoff *BackoffConfig `json:"unhealthyDialBackoff,omitempty"`
|
||||
Options *ChannelOptions `json:"options,omitempty"`
|
||||
}
|
||||
|
||||
// ChannelOptions matches the shared channelOptions definition.
|
||||
//
|
||||
// OutQueueSize and MaxQueuedConnects are pointers because zero is a valid
|
||||
// setting for both — each sizes a buffered channel, and zero means unbuffered.
|
||||
// A plain int with omitempty would make an explicit 0 indistinguishable from
|
||||
// unset and silently substitute the default.
|
||||
type ChannelOptions struct {
|
||||
OutQueueSize *int `json:"outQueueSize,omitempty"`
|
||||
MaxQueuedConnects *int `json:"maxQueuedConnects,omitempty"`
|
||||
MaxOutstandingConnects int `json:"maxOutstandingConnects,omitempty"`
|
||||
ConnectTimeout string `json:"connectTimeout,omitempty"`
|
||||
WriteTimeout string `json:"writeTimeout,omitempty"`
|
||||
}
|
||||
|
||||
// BackoffConfig matches the shared backoff definition.
|
||||
type BackoffConfig struct {
|
||||
RetryBackoffFactor float64 `json:"retryBackoffFactor,omitempty"`
|
||||
MinRetryInterval string `json:"minRetryInterval,omitempty"`
|
||||
MaxRetryInterval string `json:"maxRetryInterval,omitempty"`
|
||||
}
|
||||
|
||||
// HeartbeatsConfig matches the heartbeats definition.
|
||||
type HeartbeatsConfig struct {
|
||||
SendInterval string `json:"sendInterval,omitempty"`
|
||||
CheckInterval string `json:"checkInterval,omitempty"`
|
||||
CloseUnresponsiveTimeout string `json:"closeUnresponsiveTimeout,omitempty"`
|
||||
}
|
||||
|
||||
// Groups is a list of group names. The schema allows the JSON value to be
|
||||
// either a single string or an array of strings; UnmarshalJSON normalizes
|
||||
// both to a slice. MarshalJSON always emits an array for downstream
|
||||
// consumers that prefer one shape.
|
||||
type Groups []string
|
||||
|
||||
// UnmarshalJSON accepts either a string or an array of strings per schema.
|
||||
func (g *Groups) UnmarshalJSON(data []byte) error {
|
||||
var single string
|
||||
if err := json.Unmarshal(data, &single); err == nil {
|
||||
*g = Groups{single}
|
||||
return nil
|
||||
}
|
||||
var slice []string
|
||||
if err := json.Unmarshal(data, &slice); err != nil {
|
||||
return err
|
||||
}
|
||||
*g = Groups(slice)
|
||||
return nil
|
||||
}
|
||||
|
||||
// MarshalJSON always emits the array form for stable downstream parsing.
|
||||
func (g Groups) MarshalJSON() ([]byte, error) {
|
||||
if g == nil {
|
||||
return []byte("null"), nil
|
||||
}
|
||||
return json.Marshal([]string(g))
|
||||
}
|
||||
|
||||
// ParseConfig unmarshals raw JSON into a Config. Returns an error if the
|
||||
// JSON is malformed; does not enforce schema semantics (the schema is
|
||||
// enforced at the controller).
|
||||
func ParseConfig(data string) (*Config, error) {
|
||||
var c Config
|
||||
if err := json.Unmarshal([]byte(data), &c); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &c, nil
|
||||
}
|
||||
|
||||
// Validate checks values the controller JSON schema can't express and that
|
||||
// would otherwise fail silently downstream. Callers run it before applying a
|
||||
// config so a bad value fails the apply rather than degrading behavior.
|
||||
func (c *Config) Validate() error {
|
||||
for i := range c.Listeners {
|
||||
if err := validateChannelOptions(c.Listeners[i].Options); err != nil {
|
||||
return fmt.Errorf("listeners[%d].options: %w", i, err)
|
||||
}
|
||||
}
|
||||
for i := range c.Dialers {
|
||||
if err := validateChannelOptions(c.Dialers[i].Options); err != nil {
|
||||
return fmt.Errorf("dialers[%d].options: %w", i, err)
|
||||
}
|
||||
}
|
||||
if _, err := c.EffectiveHeartbeats(); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// HeartbeatDurations is the heartbeat timing a config resolves to, with absent
|
||||
// fields filled from the defaults.
|
||||
type HeartbeatDurations struct {
|
||||
SendInterval time.Duration
|
||||
CheckInterval time.Duration
|
||||
CloseUnresponsiveTimeout time.Duration
|
||||
}
|
||||
|
||||
// HasThinMargin reports whether the timeout leaves less than one further
|
||||
// heartbeat cycle beyond the gap a responsive link already shows. Such a config
|
||||
// is accepted — it works while nothing is lost — but a single dropped response
|
||||
// can close a healthy link, so callers log it.
|
||||
func (h HeartbeatDurations) HasThinMargin() bool {
|
||||
return h.CloseUnresponsiveTimeout < 2*(h.SendInterval+h.CheckInterval)
|
||||
}
|
||||
|
||||
// EffectiveHeartbeats resolves the heartbeat fields against their defaults, so
|
||||
// callers judge the timing the router will actually run with.
|
||||
//
|
||||
// Returns an error when a duration doesn't parse or isn't positive, when
|
||||
// sendInterval + checkInterval isn't representable, or when
|
||||
// closeUnresponsiveTimeout is below MinCloseUnresponsiveTimeout or not greater
|
||||
// than sendInterval + checkInterval, the gap a responsive link already shows.
|
||||
func (c *Config) EffectiveHeartbeats() (HeartbeatDurations, error) {
|
||||
var hb HeartbeatsConfig
|
||||
if c.Heartbeats != nil {
|
||||
hb = *c.Heartbeats
|
||||
}
|
||||
|
||||
send, err := parseHeartbeatDuration("sendInterval", hb.SendInterval, DefaultHeartbeatSendInterval)
|
||||
if err != nil {
|
||||
return HeartbeatDurations{}, err
|
||||
}
|
||||
check, err := parseHeartbeatDuration("checkInterval", hb.CheckInterval, DefaultHeartbeatCheckInterval)
|
||||
if err != nil {
|
||||
return HeartbeatDurations{}, err
|
||||
}
|
||||
closeTimeout, err := parseHeartbeatDuration("closeUnresponsiveTimeout", hb.CloseUnresponsiveTimeout, DefaultCloseUnresponsiveTimeout)
|
||||
if err != nil {
|
||||
return HeartbeatDurations{}, err
|
||||
}
|
||||
|
||||
if closeTimeout < MinCloseUnresponsiveTimeout {
|
||||
return HeartbeatDurations{}, fmt.Errorf(
|
||||
"heartbeats: closeUnresponsiveTimeout %s is below the %s minimum; "+
|
||||
"a shorter timeout closes links that are only recovering from ordinary packet loss",
|
||||
closeTimeout, MinCloseUnresponsiveTimeout)
|
||||
}
|
||||
|
||||
// The sum of two positive durations can only wrap negative, which would make
|
||||
// the check below accept every timeout.
|
||||
if send > math.MaxInt64-check {
|
||||
return HeartbeatDurations{}, fmt.Errorf(
|
||||
"heartbeats: sendInterval %s + checkInterval %s exceeds the largest representable duration (%s)",
|
||||
send, check, time.Duration(math.MaxInt64))
|
||||
}
|
||||
|
||||
if closeTimeout <= send+check {
|
||||
return HeartbeatDurations{}, fmt.Errorf(
|
||||
"heartbeats: closeUnresponsiveTimeout %s must be greater than sendInterval + checkInterval (%s + %s = %s); "+
|
||||
"a responsive link's own heartbeat gap reaches that sum, so links would be closed while healthy",
|
||||
closeTimeout, send, check, send+check)
|
||||
}
|
||||
|
||||
return HeartbeatDurations{
|
||||
SendInterval: send,
|
||||
CheckInterval: check,
|
||||
CloseUnresponsiveTimeout: closeTimeout,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// parseHeartbeatDuration resolves one heartbeat field, substituting def when it
|
||||
// is absent. A non-positive value is rejected rather than defaulted: zero or
|
||||
// negative reaches time.NewTicker in ConfigureHeartbeat, which panics on a bare
|
||||
// goroutine.
|
||||
func parseHeartbeatDuration(name, value string, def time.Duration) (time.Duration, error) {
|
||||
if value == "" {
|
||||
return def, nil
|
||||
}
|
||||
d, err := time.ParseDuration(value)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("invalid heartbeats.%s %q: %w", name, value, err)
|
||||
}
|
||||
if d <= 0 {
|
||||
return 0, fmt.Errorf("heartbeats.%s must be positive, got %s", name, d)
|
||||
}
|
||||
return d, nil
|
||||
}
|
||||
|
||||
// validateChannelOptions rejects a connectTimeout that the downstream channel
|
||||
// can't honor: it must parse as a duration and fall within
|
||||
// [channel.MinConnectTimeout, channel.MaxConnectTimeout], the range the channel
|
||||
// loader accepts.
|
||||
func validateChannelOptions(o *ChannelOptions) error {
|
||||
if o == nil || o.ConnectTimeout == "" {
|
||||
return nil
|
||||
}
|
||||
d, err := time.ParseDuration(o.ConnectTimeout)
|
||||
if err != nil {
|
||||
return fmt.Errorf("invalid connectTimeout %q: %w", o.ConnectTimeout, err)
|
||||
}
|
||||
if d < channel.MinConnectTimeout {
|
||||
return fmt.Errorf("connectTimeout %s is below the minimum of %s", d, channel.MinConnectTimeout)
|
||||
}
|
||||
if d > channel.MaxConnectTimeout {
|
||||
return fmt.Errorf("connectTimeout %s is above the maximum of %s", d, channel.MaxConnectTimeout)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -2178,8 +2178,14 @@ func (x *Listener) GetLocalBinding() string {
|
||||
}
|
||||
|
||||
type Listeners struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
Listeners []*Listener `protobuf:"bytes,1,rep,name=listeners,proto3" json:"listeners,omitempty"`
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
Listeners []*Listener `protobuf:"bytes,1,rep,name=listeners,proto3" json:"listeners,omitempty"`
|
||||
// generation identifies the router link config this set came from. It
|
||||
// increases with each applied config and restarts with the router process, so
|
||||
// a receiver compares it only within one control channel session. Updates can
|
||||
// arrive out of order (a multi-underlay channel gives no ordering across
|
||||
// underlays), so a receiver drops a generation it has already passed.
|
||||
Generation uint64 `protobuf:"varint,2,opt,name=generation,proto3" json:"generation,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
@@ -2221,6 +2227,13 @@ func (x *Listeners) GetListeners() []*Listener {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *Listeners) GetGeneration() uint64 {
|
||||
if x != nil {
|
||||
return x.Generation
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
type CheckStaleLinksRequest struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
Mode StaleLinkMatchMode `protobuf:"varint,1,opt,name=mode,proto3,enum=ziti.ctrl.pb.StaleLinkMatchMode" json:"mode,omitempty"`
|
||||
@@ -3636,9 +3649,12 @@ const file_ctrl_proto_rawDesc = "" +
|
||||
"\aaddress\x18\x01 \x01(\tR\aaddress\x12\x1a\n" +
|
||||
"\bprotocol\x18\x02 \x01(\tR\bprotocol\x12\x16\n" +
|
||||
"\x06groups\x18\x04 \x03(\tR\x06groups\x12\"\n" +
|
||||
"\flocalBinding\x18\x05 \x01(\tR\flocalBindingJ\x04\b\x03\x10\x04R\bcostTags\"A\n" +
|
||||
"\flocalBinding\x18\x05 \x01(\tR\flocalBindingJ\x04\b\x03\x10\x04R\bcostTags\"a\n" +
|
||||
"\tListeners\x124\n" +
|
||||
"\tlisteners\x18\x01 \x03(\v2\x16.ziti.ctrl.pb.ListenerR\tlisteners\"N\n" +
|
||||
"\tlisteners\x18\x01 \x03(\v2\x16.ziti.ctrl.pb.ListenerR\tlisteners\x12\x1e\n" +
|
||||
"\n" +
|
||||
"generation\x18\x02 \x01(\x04R\n" +
|
||||
"generation\"N\n" +
|
||||
"\x16CheckStaleLinksRequest\x124\n" +
|
||||
"\x04mode\x18\x01 \x01(\x0e2 .ziti.ctrl.pb.StaleLinkMatchModeR\x04mode\"\xa6\x01\n" +
|
||||
"\x0fLinkStaleReport\x12\x16\n" +
|
||||
|
||||
@@ -300,6 +300,12 @@ message Listener {
|
||||
|
||||
message Listeners {
|
||||
repeated Listener listeners = 1;
|
||||
// generation identifies the router link config this set came from. It
|
||||
// increases with each applied config and restarts with the router process, so
|
||||
// a receiver compares it only within one control channel session. Updates can
|
||||
// arrive out of order (a multi-underlay channel gives no ordering across
|
||||
// underlays), so a receiver drops a generation it has already passed.
|
||||
uint64 generation = 2;
|
||||
}
|
||||
|
||||
// StaleLinkMatchMode controls when an established link is considered
|
||||
|
||||
@@ -131,7 +131,10 @@ func (self *CtrlAccepter) Bind(binding channel.Binding) error {
|
||||
return errors.New("no version info header, not accepting router connection")
|
||||
}
|
||||
|
||||
r.SetLinkListeners(nil)
|
||||
// A new control channel starts a new session: the router mints listener
|
||||
// generations per process, so anything remembered from a previous one
|
||||
// would reject its first updates.
|
||||
r.ResetLinkListeners()
|
||||
headers := ch.Underlay().Headers()
|
||||
|
||||
// Determine header locations based on router capabilities. 2.0+ routers
|
||||
@@ -151,7 +154,7 @@ func (self *CtrlAccepter) Bind(binding channel.Binding) error {
|
||||
if err = proto.Unmarshal(val, listeners); err != nil {
|
||||
log.WithError(err).Error("unable to unmarshall listeners value")
|
||||
} else {
|
||||
r.SetLinkListeners(listeners.Listeners)
|
||||
r.SetLinkListeners(listeners.Listeners, listeners.Generation)
|
||||
for _, listener := range listeners.Listeners {
|
||||
log.WithField("address", listener.GetAddress()).
|
||||
WithField("protocol", listener.GetProtocol()).
|
||||
|
||||
@@ -52,8 +52,16 @@ func (self *updateLinkListenersHandler) HandleReceive(msg *channel.Message, ch c
|
||||
return
|
||||
}
|
||||
|
||||
self.router.SetLinkListeners(listeners.Listeners)
|
||||
if !self.router.SetLinkListeners(listeners.Listeners, listeners.Generation) {
|
||||
// Superseded by an update that arrived first. Redistributing this set
|
||||
// would push peers back onto listeners the router has already replaced.
|
||||
log.WithField("generation", listeners.Generation).
|
||||
Info("ignoring superseded router link listener update")
|
||||
return
|
||||
}
|
||||
|
||||
log.WithField("listenerCount", len(listeners.Listeners)).
|
||||
WithField("generation", listeners.Generation).
|
||||
Info("updated router link listeners; redistributing to peers")
|
||||
|
||||
// Trigger the existing peer-redistribution path: peers receive a
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
/*
|
||||
Copyright NetFoundry Inc.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
https://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package model
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/openziti/ziti/v2/common/config/routerlink"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestConfigManager_DataValidatorRegistry(t *testing.T) {
|
||||
req := require.New(t)
|
||||
manager := &ConfigManager{dataValidators: map[string]ConfigDataValidator{}}
|
||||
|
||||
// An unregistered type is not an error; most config types have no rules
|
||||
// beyond their schema.
|
||||
req.NoError(manager.validateData("some.other.type", map[string]interface{}{}))
|
||||
|
||||
sentinel := errors.New("nope")
|
||||
manager.RegisterDataValidator("my.type.v1", func(map[string]interface{}) error {
|
||||
return sentinel
|
||||
})
|
||||
|
||||
req.ErrorIs(manager.validateData("my.type.v1", nil), sentinel)
|
||||
req.NoError(manager.validateData("my.other.type.v1", nil), "validators are scoped to their type")
|
||||
|
||||
// Re-registering replaces, so init order can't leave two validators fighting.
|
||||
manager.RegisterDataValidator("my.type.v1", func(map[string]interface{}) error { return nil })
|
||||
req.NoError(manager.validateData("my.type.v1", nil))
|
||||
}
|
||||
|
||||
func TestValidateRouterLinkConfigData(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
data map[string]interface{}
|
||||
expectErr string
|
||||
}{
|
||||
{
|
||||
name: "empty config is valid",
|
||||
data: map[string]interface{}{},
|
||||
},
|
||||
{
|
||||
name: "sane heartbeats",
|
||||
data: map[string]interface{}{
|
||||
"heartbeats": map[string]interface{}{
|
||||
"sendInterval": "10s",
|
||||
"checkInterval": "1s",
|
||||
"closeUnresponsiveTimeout": "60s",
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
// The check interval outruns the timeout, so every pulse would send a
|
||||
// heartbeat and condemn the link before a response could arrive.
|
||||
name: "check interval past the timeout",
|
||||
data: map[string]interface{}{
|
||||
"heartbeats": map[string]interface{}{
|
||||
"checkInterval": "2m",
|
||||
"closeUnresponsiveTimeout": "30s",
|
||||
},
|
||||
},
|
||||
expectErr: "closeUnresponsiveTimeout",
|
||||
},
|
||||
{
|
||||
name: "non-positive interval",
|
||||
data: map[string]interface{}{
|
||||
"heartbeats": map[string]interface{}{"checkInterval": "0s"},
|
||||
},
|
||||
expectErr: "must be positive",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
req := require.New(t)
|
||||
err := validateRouterLinkConfigData(tt.data)
|
||||
if tt.expectErr == "" {
|
||||
req.NoError(err)
|
||||
return
|
||||
}
|
||||
req.Error(err)
|
||||
req.Contains(err.Error(), tt.expectErr)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewConfigManager_RegistersRouterLinkValidator(t *testing.T) {
|
||||
// The built-in validator is wired at construction rather than by a separate
|
||||
// init step, so guard that the wiring exists and is keyed correctly.
|
||||
req := require.New(t)
|
||||
manager := &ConfigManager{dataValidators: map[string]ConfigDataValidator{}}
|
||||
manager.RegisterDataValidator(routerlink.ConfigTypeV1, validateRouterLinkConfigData)
|
||||
|
||||
err := manager.validateData(routerlink.ConfigTypeV1, map[string]interface{}{
|
||||
"heartbeats": map[string]interface{}{
|
||||
"checkInterval": "2m",
|
||||
"closeUnresponsiveTimeout": "30s",
|
||||
},
|
||||
})
|
||||
req.Error(err, "a router.link.v1 config with unsafe timing must be rejected")
|
||||
}
|
||||
@@ -18,32 +18,80 @@ package model
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/openziti/ziti/v2/controller/storage/boltz"
|
||||
"github.com/openziti/ziti/v2/common/config/routerlink"
|
||||
"github.com/openziti/ziti/v2/common/pb/edge_cmd_pb"
|
||||
"github.com/openziti/ziti/v2/controller/change"
|
||||
"github.com/openziti/ziti/v2/controller/command"
|
||||
"github.com/openziti/ziti/v2/controller/db"
|
||||
"github.com/openziti/ziti/v2/controller/fields"
|
||||
"github.com/openziti/ziti/v2/controller/models"
|
||||
"github.com/openziti/ziti/v2/controller/storage/boltz"
|
||||
"go.etcd.io/bbolt"
|
||||
"google.golang.org/protobuf/proto"
|
||||
)
|
||||
|
||||
// ConfigDataValidator applies rules to a config's data that its JSON schema
|
||||
// can't express: relationships between fields, combinations of values, anything
|
||||
// needing more than per-field checks. Registered against a config type name and
|
||||
// run on create and update, after schema validation has passed.
|
||||
//
|
||||
// Return a field error (errorz.NewFieldError) to have the REST layer report the
|
||||
// offending field; a plain error is reported against the data as a whole.
|
||||
type ConfigDataValidator func(data map[string]interface{}) error
|
||||
|
||||
func NewConfigManager(env Env) *ConfigManager {
|
||||
manager := &ConfigManager{
|
||||
baseEntityManager: newBaseEntityManager[*Config, *db.Config](env, env.GetStores().Config),
|
||||
dataValidators: map[string]ConfigDataValidator{},
|
||||
}
|
||||
manager.impl = manager
|
||||
|
||||
RegisterManagerDecoder[*Config](env, manager)
|
||||
|
||||
manager.RegisterDataValidator(routerlink.ConfigTypeV1, validateRouterLinkConfigData)
|
||||
|
||||
return manager
|
||||
}
|
||||
|
||||
// validateRouterLinkConfigData applies the router.link rules its JSON schema
|
||||
// can't express, notably heartbeat timing, using the router's own validation.
|
||||
func validateRouterLinkConfigData(data map[string]interface{}) error {
|
||||
js, err := json.Marshal(data)
|
||||
if err != nil {
|
||||
return fmt.Errorf("unable to re-encode config data: %w", err)
|
||||
}
|
||||
cfg, err := routerlink.ParseConfig(string(js))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return cfg.Validate()
|
||||
}
|
||||
|
||||
type ConfigManager struct {
|
||||
baseEntityManager[*Config, *db.Config]
|
||||
|
||||
// dataValidators is keyed by config type name, which is stable across networks
|
||||
// where ids are not. Written only during controller init, so reads need no lock.
|
||||
dataValidators map[string]ConfigDataValidator
|
||||
}
|
||||
|
||||
// RegisterDataValidator installs a semantic validator for the named config
|
||||
// type, to run after schema validation on every create and update. Call during
|
||||
// controller init; it is not safe to call once requests are being served.
|
||||
// Registering a second validator for the same type replaces the first.
|
||||
func (self *ConfigManager) RegisterDataValidator(configTypeName string, v ConfigDataValidator) {
|
||||
self.dataValidators[configTypeName] = v
|
||||
}
|
||||
|
||||
// validateData runs the validator registered for configTypeName, if any.
|
||||
func (self *ConfigManager) validateData(configTypeName string, data map[string]interface{}) error {
|
||||
if v, found := self.dataValidators[configTypeName]; found {
|
||||
return v(data)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (self *ConfigManager) NewModelEntity() *Config {
|
||||
|
||||
@@ -17,11 +17,13 @@
|
||||
package model
|
||||
|
||||
import (
|
||||
"errors"
|
||||
|
||||
"github.com/openziti/foundation/v2/errorz"
|
||||
"github.com/openziti/ziti/v2/controller/storage/boltz"
|
||||
"github.com/openziti/ziti/v2/controller/apierror"
|
||||
"github.com/openziti/ziti/v2/controller/db"
|
||||
"github.com/openziti/ziti/v2/controller/models"
|
||||
"github.com/openziti/ziti/v2/controller/storage/boltz"
|
||||
"github.com/xeipuuv/gojsonschema"
|
||||
"go.etcd.io/bbolt"
|
||||
)
|
||||
@@ -50,7 +52,9 @@ func (entity *Config) toBoltEntity(tx *bbolt.Tx, env Env) (*db.Config, error) {
|
||||
entity.TypeId = currentConfig.TypeId
|
||||
}
|
||||
|
||||
if configType, _ := env.GetManagers().ConfigType.readInTx(tx, entity.TypeId); configType != nil && len(configType.Schema) > 0 {
|
||||
configType, _ := env.GetManagers().ConfigType.readInTx(tx, entity.TypeId)
|
||||
|
||||
if configType != nil && len(configType.Schema) > 0 {
|
||||
compileSchema, err := configType.GetCompiledSchema()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -65,6 +69,19 @@ func (entity *Config) toBoltEntity(tx *bbolt.Tx, env Env) (*db.Config, error) {
|
||||
}
|
||||
}
|
||||
|
||||
// Semantic validation runs after the schema, so a validator can assume the
|
||||
// shape is already right and only judge the values. Keyed on the type name,
|
||||
// which is stable across networks in a way the id is not.
|
||||
if configType != nil {
|
||||
if err := env.GetManagers().Config.validateData(configType.Name, entity.Data); err != nil {
|
||||
var fieldErr *errorz.FieldError
|
||||
if errors.As(err, &fieldErr) {
|
||||
return nil, fieldErr
|
||||
}
|
||||
return nil, errorz.NewFieldError(err.Error(), db.FieldConfigData, entity.Data)
|
||||
}
|
||||
}
|
||||
|
||||
return &db.Config{
|
||||
BaseExtEntity: *boltz.NewExtEntity(entity.Id, entity.Tags),
|
||||
Name: entity.Name,
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
/*
|
||||
Copyright NetFoundry Inc.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
https://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package model
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/openziti/ziti/v2/common/pb/ctrl_pb"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func listenerSet(addr string) []*ctrl_pb.Listener {
|
||||
return []*ctrl_pb.Listener{{Address: addr, Protocol: "tls"}}
|
||||
}
|
||||
|
||||
func TestRouter_SetLinkListeners_DropsSupersededGeneration(t *testing.T) {
|
||||
req := require.New(t)
|
||||
r := &Router{}
|
||||
|
||||
req.True(r.SetLinkListeners(listenerSet("a"), 1))
|
||||
req.Equal("a", r.GetLinkListeners()[0].Address)
|
||||
|
||||
req.True(r.SetLinkListeners(listenerSet("b"), 2))
|
||||
req.Equal("b", r.GetLinkListeners()[0].Address)
|
||||
|
||||
// An update that lost the race must not push the router back onto listeners
|
||||
// it has already replaced, since the set is redistributed to every peer.
|
||||
req.False(r.SetLinkListeners(listenerSet("a"), 1))
|
||||
req.Equal("b", r.GetLinkListeners()[0].Address, "the newer set must survive")
|
||||
|
||||
// The same generation is also refused: nothing new to record.
|
||||
req.False(r.SetLinkListeners(listenerSet("c"), 2))
|
||||
req.Equal("b", r.GetLinkListeners()[0].Address)
|
||||
|
||||
req.True(r.SetLinkListeners(listenerSet("d"), 3))
|
||||
req.Equal("d", r.GetLinkListeners()[0].Address)
|
||||
}
|
||||
|
||||
func TestRouter_ResetLinkListeners_AllowsRestartedRouter(t *testing.T) {
|
||||
// A router mints generations per process, so a restart begins at 1 again. If
|
||||
// the controller kept the generation from the previous session, every update
|
||||
// from the restarted router would be refused as superseded.
|
||||
req := require.New(t)
|
||||
r := &Router{}
|
||||
|
||||
req.True(r.SetLinkListeners(listenerSet("old"), 47))
|
||||
req.False(r.SetLinkListeners(listenerSet("new"), 1), "without a reset, a restart looks superseded")
|
||||
|
||||
r.ResetLinkListeners()
|
||||
req.Empty(r.GetLinkListeners(), "reset clears the recorded set")
|
||||
req.True(r.SetLinkListeners(listenerSet("new"), 1), "after a reset the new session starts fresh")
|
||||
req.Equal("new", r.GetLinkListeners()[0].Address)
|
||||
}
|
||||
|
||||
func TestRouter_SetLinkListeners_FirstUpdateAlwaysRecorded(t *testing.T) {
|
||||
// Generation zero from a router that predates the field must still land,
|
||||
// rather than being read as "already superseded".
|
||||
req := require.New(t)
|
||||
r := &Router{}
|
||||
req.True(r.SetLinkListeners(listenerSet("a"), 0))
|
||||
req.Equal("a", r.GetLinkListeners()[0].Address)
|
||||
}
|
||||
@@ -42,6 +42,9 @@ type Router struct {
|
||||
// mid-session changes from the router. Guarded by mu, so it is
|
||||
// unexported: reach it through SetLinkListeners / GetLinkListeners.
|
||||
listeners []*ctrl_pb.Listener
|
||||
// listenerGeneration is the generation of the recorded listener set, scoped
|
||||
// to the current control channel session. Zero means nothing recorded yet.
|
||||
listenerGeneration uint64
|
||||
|
||||
// mu guards the fields a connected Router mutates mid-session, which
|
||||
// today is just listeners. Fields added later should share it rather
|
||||
@@ -167,10 +170,30 @@ func (entity *Router) addLinkListener(addr, linkProtocol string, groups []string
|
||||
// SetLinkListeners atomically replaces the router's link listener slice.
|
||||
// Callers do not mutate the previous slice — readers may still hold and
|
||||
// iterate it safely after a Set.
|
||||
func (entity *Router) SetLinkListeners(listeners []*ctrl_pb.Listener) {
|
||||
// SetLinkListeners records listeners when generation is newer than the last
|
||||
// recorded for this control channel session, and reports whether it did. Older
|
||||
// generations are dropped: a multi-underlay control channel doesn't order updates
|
||||
// across underlays, and a stale set would be redistributed to peers.
|
||||
func (entity *Router) SetLinkListeners(listeners []*ctrl_pb.Listener, generation uint64) bool {
|
||||
entity.mu.Lock()
|
||||
defer entity.mu.Unlock()
|
||||
if generation <= entity.listenerGeneration && entity.listenerGeneration != 0 {
|
||||
return false
|
||||
}
|
||||
entity.listeners = listeners
|
||||
entity.listenerGeneration = generation
|
||||
return true
|
||||
}
|
||||
|
||||
// ResetLinkListeners clears the listener set and the generation it was recorded
|
||||
// at, for a newly-connected control channel. The router mints generations per
|
||||
// process, so they restart when it does; a generation remembered from a
|
||||
// previous session would reject a restarted router's first updates.
|
||||
func (entity *Router) ResetLinkListeners() {
|
||||
entity.mu.Lock()
|
||||
defer entity.mu.Unlock()
|
||||
entity.listeners = nil
|
||||
entity.listenerGeneration = 0
|
||||
}
|
||||
|
||||
// GetLinkListeners returns the current link listener slice under a read
|
||||
|
||||
@@ -214,8 +214,8 @@ the router has final say on what it applies.
|
||||
If a router has both local and controller-managed config for the same subsystem, local always wins.
|
||||
Routers are often deployed on systems owned by someone other than the network operator (one entity
|
||||
runs the network and uses it to manage devices for their customers). The customer needs control
|
||||
over what the router is capable of. Controller-managed config fills in gaps where local config is
|
||||
silent; it doesn't override what the operator has explicitly set.
|
||||
over what the router is capable of. Controller-managed config configures the subsystems local config
|
||||
leaves alone; it never overrides a subsystem the operator has configured.
|
||||
|
||||
### Source Tracking and Selection
|
||||
|
||||
@@ -230,8 +230,14 @@ Local takes precedence at the **base type level**. If the operator set anything
|
||||
`router.link`, controller versions are ignored for that base. Matches operator intent: someone who
|
||||
set v1 locally probably doesn't want the controller silently upgrading them to v2.
|
||||
|
||||
Per-field merge — the richest interpretation of "fills in gaps where local config is silent" — is
|
||||
out of scope for now. Wait for operator demand.
|
||||
For `router.link`, "set anything locally" means the router's config file has a `link` section at
|
||||
all. The env loader fills every link field with a default whether or not the section is present, so
|
||||
it records presence separately, and a section that only tunes heartbeats or queue sizes still counts.
|
||||
Such a section leaves the router with no listeners or dialers of its own while ignoring the
|
||||
controller's, so the router warns at startup when a `link` section has neither.
|
||||
|
||||
Per-field merge, where local config would win only for the fields it sets, is out of scope for now.
|
||||
Wait for operator demand.
|
||||
|
||||
### YAML → JSON Translation
|
||||
|
||||
@@ -243,6 +249,13 @@ default `binding` if absent, and preserve the deprecated `split` dialer flag as
|
||||
from the controller). `costTags` was unused and has been removed from the router outright rather
|
||||
than translated. Per-type code, lives next to each handler. The registry itself is agnostic.
|
||||
|
||||
Heartbeat timing is translated leniently, because local link config is applied synchronously and a
|
||||
failure stops the router from starting. An explicit zero is carried through as `0s` rather than
|
||||
dropped, so it isn't mistaken for an absent field. A `closeUnresponsiveTimeout` below the floor is
|
||||
raised to it, and any other timing that fails validation is replaced by the defaults, each with a
|
||||
warning. The controller path rejects the same values outright, where the error comes back on the
|
||||
API call.
|
||||
|
||||
### Hot Reconfiguration
|
||||
|
||||
Every subsystem must handle config updates at runtime: accept new config, tear down and recreate
|
||||
@@ -258,8 +271,9 @@ General approach:
|
||||
|
||||
### Reconciliation Strategy
|
||||
|
||||
The link handler takes the simple route: on Apply, build the new listener / dialer set, close the
|
||||
old listeners, swap in the new state, then `Listen()` on the new listeners. No per-item diffing.
|
||||
The link handler takes the simple route: on Apply, if the listener or dialer definitions changed,
|
||||
build the new listener / dialer set, close the old listeners, swap in the new state, then `Listen()`
|
||||
on the new listeners. No per-item diffing.
|
||||
|
||||
This works without disturbing traffic because `xlink.Listener.Close()` only closes the listener's
|
||||
accept loop — already-accepted Xlinks survive. So the operator sees:
|
||||
@@ -269,14 +283,59 @@ accept loop — already-accepted Xlinks survive. So the operator sees:
|
||||
- **Remove a listener**: same; established links on it stay up.
|
||||
- **Modify a listener** (advertise change, etc.): the new accept loop is bound with the new
|
||||
config; old established links unaffected.
|
||||
- **No actual change**: the deep-equality check in `notifyChange` no-ops the post-apply notify, but
|
||||
the listener rebuild still runs. Cheap enough that we don't optimize for it.
|
||||
- **No change to listeners or dialers**: the rebuild is skipped, so a change confined to heartbeats,
|
||||
queue sizes, or `gcMode` never rebinds a listen socket. Re-applying identical config returns
|
||||
before doing anything at all.
|
||||
|
||||
Per-item diffing (a `ReconcileSet[K, V]` helper that fires add/update/remove callbacks) is a
|
||||
plausible refinement when we have multiple subsystems with item-level identity (xgress listeners,
|
||||
ctrl-channel listeners), but the link subsystem doesn't need it. Add it when a second consumer
|
||||
shows up and the simple-rebuild approach has actual cost there.
|
||||
|
||||
### Heartbeats and Queue Sizes
|
||||
|
||||
Beyond listeners and dialers, the link handler applies the `router.link.v1` `heartbeats` settings
|
||||
and sender queue sizes from the effective config. These follow full-config (PUT) semantics: a field
|
||||
absent from the applied config reverts to its default rather than retaining a previously-applied
|
||||
value.
|
||||
|
||||
- **Heartbeat timing** (`sendInterval`, `checkInterval`, `closeUnresponsiveTimeout`) is published as
|
||||
one generation, stamped with the applied config's generation. New links take the current
|
||||
generation at bind time; established links are pushed it in place through the channel
|
||||
`HeartbeatControl` handle, with no teardown. Every channel backing a link is re-tuned, including
|
||||
both channels of a split link, and a channel bound on the far side of a config change is
|
||||
reconciled to the newest generation when it registers. A link ignores a generation at or below the
|
||||
one it holds, so an overtaken update cannot roll it back.
|
||||
- **`payloadSenderQueueSize` / `ackSenderQueueSize`** are read when a link channel is built, so a
|
||||
change applies to links established afterward; an established channel's send queues aren't resized.
|
||||
A change confined to them asks nothing of established links and so triggers no notification or
|
||||
GC pass, though the apply is still logged.
|
||||
|
||||
A new generation re-arms each link's liveness deadline. Under the previous intervals a healthy
|
||||
link's last response can be most of a send + check cycle old, and a newly shortened timeout would
|
||||
read that accumulated age as unresponsiveness. The re-arm grants one fresh timeout window, not
|
||||
standing immunity.
|
||||
|
||||
Two thresholds are evaluated against time since the last response, independently:
|
||||
|
||||
- **Close**: `closeUnresponsiveTimeout`, as configured. A zero timeout closes nothing, since it
|
||||
means no settings have been published yet.
|
||||
- **Unhealthy**: `max(30s, 2 × (sendInterval + checkInterval))`, meaning a response has actually
|
||||
been missed. Reporting unhealthy poisons the link's latency metric, which the controller routes on,
|
||||
so the threshold has to scale with the intervals; a fixed 30s would price a healthy link with a
|
||||
long send interval out of routing every cycle. A link being closed is always reported unhealthy.
|
||||
|
||||
Timing is validated on both the controller and the router: every duration must be positive,
|
||||
`sendInterval + checkInterval` must be representable, and `closeUnresponsiveTimeout` must be at least
|
||||
10s and greater than `sendInterval + checkInterval`, since a responsive link's own gap already reaches
|
||||
that sum. The floor exists because a healthy connection can go quiet for several seconds while TCP
|
||||
recovers from ordinary loss. A timeout under two heartbeat cycles is accepted with a warning, since
|
||||
a single lost response can close the link.
|
||||
|
||||
Split links (the legacy fallback for peers without multi-underlay support) are deprecated;
|
||||
multi-underlay is a superset of split links, and the `split` option is slated for removal in
|
||||
OpenZiti 4.0.
|
||||
|
||||
### Config Application Strategy
|
||||
|
||||
The router maintains a config handler registry keyed by config type. When a config event arrives:
|
||||
@@ -339,6 +398,11 @@ rather than requiring peer agreement, because if the local side can no longer su
|
||||
the peer was going to see it disconnect anyway. Each closure logs `linkId`, `linkKey`, `side`,
|
||||
`mode`, and `reason`.
|
||||
|
||||
Removing the config always sweeps, under `orphaned`, whatever mode the removed config carried. A
|
||||
removal leaves no listeners and no dialers, so every remaining link is one the router can never
|
||||
re-establish; `gcMode` governs links the router's configuration made stale, and after a removal there
|
||||
is no configuration.
|
||||
|
||||
Operators who want a two-sided, controller-aggregated sweep use the `ziti ops verify stale-links`
|
||||
CLI (which fans CheckStaleLinks out to all routers and only GCs when both endpoints agree).
|
||||
|
||||
@@ -353,8 +417,8 @@ is this subsystem configured this way?":
|
||||
- What's currently applied (source + version).
|
||||
- Recent alerts (parse failures, rollbacks, offline subsystems).
|
||||
|
||||
Wired into the existing inspect framework: `ziti fabric inspect <router> managed-config` returns
|
||||
the registry view.
|
||||
Wired into the existing inspect framework:
|
||||
`ziti fabric inspect router-config-registry [router id regex]` returns the registry view.
|
||||
|
||||
### Startup Behavior
|
||||
|
||||
|
||||
Vendored
+5
@@ -152,6 +152,10 @@ type Config struct {
|
||||
Listeners []*CtrlListenerConfig
|
||||
}
|
||||
Link struct {
|
||||
// Configured reports that the config file has a link section at all. The
|
||||
// other fields are filled with defaults either way, so this is the only
|
||||
// record of whether the operator wrote one.
|
||||
Configured bool
|
||||
Listeners []map[interface{}]interface{}
|
||||
Dialers []map[interface{}]interface{}
|
||||
Heartbeats channel.HeartbeatOptions
|
||||
@@ -586,6 +590,7 @@ func LoadConfigWithOptions(path string, loadIdentity bool) (*Config, error) {
|
||||
cfg.Link.AckSenderQueueSize = DefaultLinkAckSenderQueueSize
|
||||
|
||||
if value, found := cfgmap["link"]; found {
|
||||
cfg.Link.Configured = true
|
||||
if submap, ok := value.(map[interface{}]interface{}); ok {
|
||||
if value, found := submap["listeners"]; found {
|
||||
if subarr, ok := value.([]interface{}); ok {
|
||||
|
||||
Vendored
+71
@@ -0,0 +1,71 @@
|
||||
/*
|
||||
Copyright NetFoundry Inc.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
https://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package env
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
const minimalRouterConfig = `v: 3
|
||||
identity:
|
||||
cert: c.pem
|
||||
server_cert: s.pem
|
||||
key: k.pem
|
||||
ca: ca.pem
|
||||
ctrl:
|
||||
endpoint: tls:127.0.0.1:6262
|
||||
csr:
|
||||
country: US
|
||||
province: NC
|
||||
locality: Charlotte
|
||||
organization: NetFoundry
|
||||
organizationalUnit: Ops
|
||||
sans:
|
||||
dns: [localhost]
|
||||
`
|
||||
|
||||
func loadRouterConfig(t *testing.T, yaml string) *Config {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), "router.yml")
|
||||
require.NoError(t, os.WriteFile(path, []byte(yaml), 0600))
|
||||
cfg, err := LoadConfigWithOptions(path, false)
|
||||
require.NoError(t, err)
|
||||
return cfg
|
||||
}
|
||||
|
||||
func Test_LoadConfig_RecordsWhetherLinkSectionIsPresent(t *testing.T) {
|
||||
// Every link field is filled with a default whether or not the section was
|
||||
// written, so Configured is the only way to tell them apart.
|
||||
t.Run("absent", func(t *testing.T) {
|
||||
req := require.New(t)
|
||||
cfg := loadRouterConfig(t, minimalRouterConfig)
|
||||
req.False(cfg.Link.Configured)
|
||||
req.NotZero(cfg.Link.Heartbeats.SendInterval, "defaults are filled in regardless")
|
||||
})
|
||||
|
||||
t.Run("present with only heartbeats", func(t *testing.T) {
|
||||
req := require.New(t)
|
||||
cfg := loadRouterConfig(t, minimalRouterConfig+"link:\n heartbeats:\n sendInterval: 5s\n")
|
||||
req.True(cfg.Link.Configured)
|
||||
req.Empty(cfg.Link.Listeners)
|
||||
req.Empty(cfg.Link.Dialers)
|
||||
})
|
||||
}
|
||||
+100
-23
@@ -2,6 +2,7 @@ package handler_link
|
||||
|
||||
import (
|
||||
"crypto/x509"
|
||||
"math"
|
||||
"time"
|
||||
|
||||
"github.com/michaelquigley/pfxlog"
|
||||
@@ -13,6 +14,7 @@ import (
|
||||
"github.com/openziti/metrics"
|
||||
"github.com/openziti/sdk-golang/v2/xgress"
|
||||
"github.com/openziti/ziti/v2/common/pb/ctrl_pb"
|
||||
"github.com/openziti/ziti/v2/common/servermetrics"
|
||||
"github.com/openziti/ziti/v2/common/trace"
|
||||
"github.com/openziti/ziti/v2/router/env"
|
||||
"github.com/openziti/ziti/v2/router/forwarder"
|
||||
@@ -22,22 +24,43 @@ import (
|
||||
"github.com/sirupsen/logrus"
|
||||
)
|
||||
|
||||
func NewBindHandlerFactory(c env.NetworkControllers, f *forwarder.Forwarder, hbo *channel.HeartbeatOptions, mr metrics.Registry, registry xlink.Registry) *bindHandlerFactory {
|
||||
// LinkSettings supplies the link settings the bind handler reads, which can
|
||||
// change at runtime through controller-managed config. An interface so
|
||||
// handler_link doesn't depend on the link package, whose Subsystem implements it.
|
||||
// Implementations must be safe for concurrent reads.
|
||||
type LinkSettings interface {
|
||||
// Heartbeats is the heartbeat timing currently in force, as one snapshot.
|
||||
// Taken whole so a link is never configured from a mix of generations.
|
||||
Heartbeats() xlink.HeartbeatSettings
|
||||
}
|
||||
|
||||
// LinkChannelEnv bundles what the link bind handler needs. The router satisfies
|
||||
// it directly; GetForwarderImpl returns the concrete *forwarder.Forwarder, which
|
||||
// GetForwarder can't without an env->forwarder import cycle.
|
||||
type LinkChannelEnv interface {
|
||||
GetNetworkControllers() env.NetworkControllers
|
||||
GetForwarderImpl() *forwarder.Forwarder
|
||||
GetMetricsRegistry() servermetrics.UsageRegistry
|
||||
GetXlinkRegistry() xlink.Registry
|
||||
GetLinkSettings() LinkSettings
|
||||
}
|
||||
|
||||
func NewBindHandlerFactory(linkEnv LinkChannelEnv) *bindHandlerFactory {
|
||||
return &bindHandlerFactory{
|
||||
ctrl: c,
|
||||
forwarder: f,
|
||||
metricsRegistry: mr,
|
||||
xlinkRegistry: registry,
|
||||
heartbeatOptions: hbo,
|
||||
ctrl: linkEnv.GetNetworkControllers(),
|
||||
forwarder: linkEnv.GetForwarderImpl(),
|
||||
metricsRegistry: linkEnv.GetMetricsRegistry(),
|
||||
xlinkRegistry: linkEnv.GetXlinkRegistry(),
|
||||
linkSettings: linkEnv.GetLinkSettings(),
|
||||
}
|
||||
}
|
||||
|
||||
type bindHandlerFactory struct {
|
||||
ctrl env.NetworkControllers
|
||||
forwarder *forwarder.Forwarder
|
||||
metricsRegistry metrics.Registry
|
||||
xlinkRegistry xlink.Registry
|
||||
heartbeatOptions *channel.HeartbeatOptions
|
||||
ctrl env.NetworkControllers
|
||||
forwarder *forwarder.Forwarder
|
||||
metricsRegistry metrics.Registry
|
||||
xlinkRegistry xlink.Registry
|
||||
linkSettings LinkSettings
|
||||
}
|
||||
|
||||
func (self *bindHandlerFactory) NewBindHandler(link xlink.Xlink, latency bool, listenerSide bool) channel.BindHandler {
|
||||
@@ -97,16 +120,23 @@ func (self *bindHandler) BindChannel(binding channel.Binding) error {
|
||||
}))
|
||||
|
||||
log.Info("link destination support heartbeats")
|
||||
|
||||
// One sample, used for both the ticker and the seed, and handed to the link
|
||||
// so its callbacks enforce the timeout belonging to these intervals.
|
||||
settings := self.linkSettings.Heartbeats()
|
||||
|
||||
cb := &heartbeatCallback{
|
||||
linkId: self.xlink.Id(),
|
||||
latencyMetric: latencyMetric,
|
||||
queueTimeMetric: queueTimeMetric,
|
||||
ch: binding.GetChannel(),
|
||||
heartbeatOptions: self.heartbeatOptions,
|
||||
link: self.xlink,
|
||||
latencySemaphore: concurrenz.NewSemaphore(2),
|
||||
lastResponse: time.Now().Add(self.heartbeatOptions.CloseUnresponsiveTimeout * 2).UnixMilli(),
|
||||
lastResponse: time.Now().Add(settings.CloseUnresponsiveTimeout * 2).UnixMilli(),
|
||||
generation: settings.Generation,
|
||||
}
|
||||
channel.ConfigureHeartbeat(binding, 10*time.Second, time.Second, cb)
|
||||
hc := channel.ConfigureHeartbeat(binding, settings.SendInterval, settings.CheckInterval, cb)
|
||||
self.xlink.SetHeartbeatControl(hc, settings)
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -162,12 +192,35 @@ func leafFingerprint(certs []*x509.Certificate) (string, error) {
|
||||
return nfpem.FingerprintFromCertificate(certs[0]), nil
|
||||
}
|
||||
|
||||
// minUnhealthyThreshold is the floor for reporting a silent link as unhealthy.
|
||||
const minUnhealthyThreshold = 30 * time.Second
|
||||
|
||||
// unhealthyThresholdMs is how long a link can go without a heartbeat response
|
||||
// before it is reported unhealthy, which poisons its latency metric and so its
|
||||
// routing cost: twice the send + check gap a healthy link already shows, so a
|
||||
// response has actually been missed, and never less than minUnhealthyThreshold.
|
||||
func unhealthyThresholdMs(settings xlink.HeartbeatSettings) int64 {
|
||||
threshold := minUnhealthyThreshold
|
||||
if gap := settings.SendInterval + settings.CheckInterval; gap > threshold/2 {
|
||||
threshold = 2 * gap
|
||||
if threshold < gap { // validation bounds the gap, not its double
|
||||
threshold = math.MaxInt64
|
||||
}
|
||||
}
|
||||
return threshold.Milliseconds()
|
||||
}
|
||||
|
||||
// heartbeatCallback enforces one channel's heartbeat liveness. Every method runs
|
||||
// on that channel's heartbeat pulse goroutine, including the response callbacks,
|
||||
// which the heartbeater queues rather than calling on the rxer, so lastResponse
|
||||
// and generation need no synchronization.
|
||||
type heartbeatCallback struct {
|
||||
linkId string
|
||||
latencyMetric metrics.Histogram
|
||||
queueTimeMetric metrics.Histogram
|
||||
lastResponse int64
|
||||
heartbeatOptions *channel.HeartbeatOptions
|
||||
generation uint64
|
||||
link xlink.Xlink
|
||||
ch channel.Channel
|
||||
latencySemaphore concurrenz.Semaphore
|
||||
}
|
||||
@@ -186,23 +239,47 @@ func (self *heartbeatCallback) HeartbeatRespRx(ts int64) {
|
||||
|
||||
func (self *heartbeatCallback) CheckHeartBeat() {
|
||||
log := pfxlog.Logger().WithField("channelId", self.ch.Label())
|
||||
now := time.Now().UnixMilli()
|
||||
if delta := now - self.lastResponse; delta > 30000 {
|
||||
|
||||
unhealthy, shouldClose := self.evaluate(time.Now().UnixMilli(), self.link.HeartbeatSettings())
|
||||
if unhealthy {
|
||||
log.Warn("heartbeat not received in time, link may be unhealthy")
|
||||
self.latencyMetric.Clear()
|
||||
self.latencyMetric.Update(8888888888888)
|
||||
|
||||
if delta > self.heartbeatOptions.CloseUnresponsiveTimeout.Milliseconds() {
|
||||
log.Error("heartbeat not received in time, closing router link connection")
|
||||
if err := self.ch.Close(); err != nil {
|
||||
log.WithError(err).Error("error while closing router link connection")
|
||||
}
|
||||
}
|
||||
if shouldClose {
|
||||
log.Error("heartbeat not received in time, closing router link connection")
|
||||
if err := self.ch.Close(); err != nil {
|
||||
log.WithError(err).Error("error while closing router link connection")
|
||||
}
|
||||
}
|
||||
|
||||
go self.checkQueueTime()
|
||||
}
|
||||
|
||||
// evaluate advances the deadline state for a check run at now and reports what
|
||||
// that check should do. Split from CheckHeartBeat so the timing rules can be
|
||||
// exercised without a live channel.
|
||||
//
|
||||
// A change of generation re-arms the deadline, granting one fresh timeout window:
|
||||
// under the previous intervals a healthy link's last response can be most of a
|
||||
// send+check cycle old, and the channel only picks up a new check interval on its
|
||||
// next pulse.
|
||||
func (self *heartbeatCallback) evaluate(now int64, settings xlink.HeartbeatSettings) (unhealthy, shouldClose bool) {
|
||||
if settings.Generation != self.generation {
|
||||
self.generation = settings.Generation
|
||||
self.lastResponse = now
|
||||
}
|
||||
|
||||
delta := now - self.lastResponse
|
||||
|
||||
// The thresholds are independent. A zero timeout means no settings have been
|
||||
// published yet, so it closes nothing.
|
||||
if timeout := settings.CloseUnresponsiveTimeout.Milliseconds(); timeout > 0 && delta > timeout {
|
||||
return true, true
|
||||
}
|
||||
return delta > unhealthyThresholdMs(settings), false
|
||||
}
|
||||
|
||||
func (self *heartbeatCallback) checkQueueTime() {
|
||||
log := pfxlog.Logger().WithField("linkId", self.linkId)
|
||||
if !self.latencySemaphore.TryAcquire() {
|
||||
|
||||
@@ -19,9 +19,12 @@ package handler_link
|
||||
import (
|
||||
"crypto/x509"
|
||||
"crypto/x509/pkix"
|
||||
"math"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
nfpem "github.com/openziti/foundation/v2/pem"
|
||||
"github.com/openziti/ziti/v2/router/xlink"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
@@ -61,3 +64,157 @@ func Test_leafFingerprint(t *testing.T) {
|
||||
req.Error(err)
|
||||
})
|
||||
}
|
||||
|
||||
func Test_heartbeatCallback_GenerationChangeRearmsDeadline(t *testing.T) {
|
||||
hb := func(gen uint64, send, check, closeTimeout time.Duration) xlink.HeartbeatSettings {
|
||||
return xlink.HeartbeatSettings{
|
||||
Generation: gen,
|
||||
SendInterval: send,
|
||||
CheckInterval: check,
|
||||
CloseUnresponsiveTimeout: closeTimeout,
|
||||
}
|
||||
}
|
||||
|
||||
// Under send 30s / check 20s a healthy link's last response is legitimately
|
||||
// up to ~50s old, and the heartbeat sent on this very pulse cannot have been
|
||||
// answered yet.
|
||||
const healthyAgeMs = 50_000
|
||||
now := time.Now().UnixMilli()
|
||||
|
||||
t.Run("tightening does not close a healthy link", func(t *testing.T) {
|
||||
req := require.New(t)
|
||||
cb := &heartbeatCallback{generation: 1, lastResponse: now - healthyAgeMs}
|
||||
|
||||
_, shouldClose := cb.evaluate(now, hb(2, time.Second, 500*time.Millisecond, 10*time.Second))
|
||||
req.False(shouldClose, "accumulated age under the old intervals must not condemn the link")
|
||||
req.Equal(now, cb.lastResponse, "the deadline is re-armed to this check")
|
||||
req.Equal(uint64(2), cb.generation)
|
||||
})
|
||||
|
||||
t.Run("the grace is one window, not immunity", func(t *testing.T) {
|
||||
req := require.New(t)
|
||||
cb := &heartbeatCallback{generation: 1, lastResponse: now - healthyAgeMs}
|
||||
// Above the unhealthy threshold on purpose: a timeout below it is inert
|
||||
// today, which is a separate defect this test should not depend on.
|
||||
settings := hb(2, 30*time.Second, 10*time.Second, 90*time.Second)
|
||||
|
||||
cb.evaluate(now, settings)
|
||||
|
||||
// Still silent a full new timeout later: detection must resume.
|
||||
later := now + settings.CloseUnresponsiveTimeout.Milliseconds() + 1
|
||||
unhealthy, shouldClose := cb.evaluate(later, settings)
|
||||
req.True(unhealthy)
|
||||
req.True(shouldClose, "a genuinely dead link still closes once the new timeout elapses")
|
||||
})
|
||||
|
||||
t.Run("no generation change, no re-arm", func(t *testing.T) {
|
||||
req := require.New(t)
|
||||
cb := &heartbeatCallback{generation: 2, lastResponse: now - 90_000}
|
||||
|
||||
unhealthy, shouldClose := cb.evaluate(now, hb(2, 30*time.Second, 20*time.Second, time.Minute))
|
||||
req.True(unhealthy)
|
||||
req.True(shouldClose, "an unresponsive link on a steady generation is still closed")
|
||||
})
|
||||
}
|
||||
|
||||
func Test_heartbeatCallback_HonorsConfiguredTimeout(t *testing.T) {
|
||||
hb := func(closeTimeout time.Duration) xlink.HeartbeatSettings {
|
||||
return xlink.HeartbeatSettings{
|
||||
Generation: 1,
|
||||
SendInterval: 5 * time.Second,
|
||||
CheckInterval: time.Second,
|
||||
CloseUnresponsiveTimeout: closeTimeout,
|
||||
}
|
||||
}
|
||||
now := time.Now().UnixMilli()
|
||||
silentFor := func(d time.Duration) *heartbeatCallback {
|
||||
return &heartbeatCallback{generation: 1, lastResponse: now - d.Milliseconds()}
|
||||
}
|
||||
|
||||
t.Run("a timeout under the unhealthy threshold still closes", func(t *testing.T) {
|
||||
req := require.New(t)
|
||||
// The close timeout is evaluated on its own, not behind the 30s unhealthy
|
||||
// threshold.
|
||||
_, shouldClose := silentFor(11*time.Second).evaluate(now, hb(10*time.Second))
|
||||
req.True(shouldClose, "a 10s timeout must close at 11s, not wait for 30s")
|
||||
})
|
||||
|
||||
t.Run("closing always reports unhealthy", func(t *testing.T) {
|
||||
req := require.New(t)
|
||||
// Otherwise a short timeout closes the link with no preceding warning
|
||||
// and without poisoning the latency metric watchers key off.
|
||||
unhealthy, shouldClose := silentFor(11*time.Second).evaluate(now, hb(10*time.Second))
|
||||
req.True(shouldClose)
|
||||
req.True(unhealthy)
|
||||
})
|
||||
|
||||
t.Run("still healthy before the timeout", func(t *testing.T) {
|
||||
req := require.New(t)
|
||||
unhealthy, shouldClose := silentFor(9*time.Second).evaluate(now, hb(10*time.Second))
|
||||
req.False(shouldClose)
|
||||
req.False(unhealthy)
|
||||
})
|
||||
|
||||
t.Run("a zero timeout closes nothing", func(t *testing.T) {
|
||||
req := require.New(t)
|
||||
// Zero means no settings have been published yet, not that the link
|
||||
// should go immediately. Without the guard, decoupling the thresholds
|
||||
// turns "unset" into "close on the first check".
|
||||
_, shouldClose := silentFor(5*time.Minute).evaluate(now, hb(0))
|
||||
req.False(shouldClose, "an unset timeout must never close a link")
|
||||
})
|
||||
|
||||
t.Run("unhealthy is still reported on a long timeout", func(t *testing.T) {
|
||||
req := require.New(t)
|
||||
unhealthy, shouldClose := silentFor(31*time.Second).evaluate(now, hb(60*time.Second))
|
||||
req.True(unhealthy, "the 30s unhealthy signal survives the decoupling")
|
||||
req.False(shouldClose)
|
||||
})
|
||||
}
|
||||
|
||||
func Test_heartbeatCallback_UnhealthyThresholdScalesWithIntervals(t *testing.T) {
|
||||
hb := func(send, check, closeTimeout time.Duration) xlink.HeartbeatSettings {
|
||||
return xlink.HeartbeatSettings{Generation: 1, SendInterval: send, CheckInterval: check, CloseUnresponsiveTimeout: closeTimeout}
|
||||
}
|
||||
now := time.Now().UnixMilli()
|
||||
silentFor := func(d time.Duration) *heartbeatCallback {
|
||||
return &heartbeatCallback{generation: 1, lastResponse: now - d.Milliseconds()}
|
||||
}
|
||||
|
||||
t.Run("a long send interval is not unhealthy between heartbeats", func(t *testing.T) {
|
||||
req := require.New(t)
|
||||
// Reporting unhealthy poisons the latency metric the controller routes on, so
|
||||
// a healthy link must not trip it while waiting for a heartbeat that isn't due.
|
||||
settings := hb(45*time.Second, time.Second, 2*time.Minute)
|
||||
unhealthy, _ := silentFor(44*time.Second).evaluate(now, settings)
|
||||
req.False(unhealthy, "the next heartbeat is not even due yet")
|
||||
|
||||
unhealthy, shouldClose := silentFor(93*time.Second).evaluate(now, settings)
|
||||
req.True(unhealthy, "a whole cycle with no response is a missed response")
|
||||
req.False(shouldClose)
|
||||
})
|
||||
|
||||
t.Run("the defaults keep the 30s floor", func(t *testing.T) {
|
||||
req := require.New(t)
|
||||
settings := hb(10*time.Second, time.Second, time.Minute)
|
||||
unhealthy, _ := silentFor(29*time.Second).evaluate(now, settings)
|
||||
req.False(unhealthy)
|
||||
unhealthy, _ = silentFor(31*time.Second).evaluate(now, settings)
|
||||
req.True(unhealthy)
|
||||
})
|
||||
|
||||
t.Run("a thin margin still reports unhealthy on close", func(t *testing.T) {
|
||||
req := require.New(t)
|
||||
// The threshold (92s) lands past the timeout (90s), leaving no early
|
||||
// warning, but a close is always reported unhealthy.
|
||||
unhealthy, shouldClose := silentFor(91*time.Second).evaluate(now, hb(45*time.Second, time.Second, 90*time.Second))
|
||||
req.True(shouldClose)
|
||||
req.True(unhealthy)
|
||||
})
|
||||
|
||||
t.Run("doubling a huge gap does not wrap", func(t *testing.T) {
|
||||
req := require.New(t)
|
||||
huge := time.Duration(math.MaxInt64 / 2)
|
||||
req.Equal(time.Duration(math.MaxInt64).Milliseconds(), unhealthyThresholdMs(hb(huge, time.Second, time.Duration(math.MaxInt64))))
|
||||
})
|
||||
}
|
||||
|
||||
+25
-207
@@ -17,219 +17,37 @@
|
||||
package link
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/openziti/channel/v5"
|
||||
"github.com/openziti/ziti/v2/common/config/routerlink"
|
||||
)
|
||||
|
||||
// ConfigBaseType is the un-versioned config family this package owns.
|
||||
const ConfigBaseType = "router.link"
|
||||
|
||||
// ConfigTypeV1 is the controller-defined ConfigType.Name for version 1.
|
||||
const ConfigTypeV1 = "router.link.v1"
|
||||
|
||||
// Config is the typed view of router.link.v1 JSON. Field tags match the
|
||||
// schema property names. Strings rather than time.Duration are used for
|
||||
// duration fields so they round-trip cleanly into the existing
|
||||
// channel.LoadOptions parser, which already understands "30s" / "100ms".
|
||||
type Config struct {
|
||||
Listeners []ListenerConfig `json:"listeners,omitempty"`
|
||||
Dialers []DialerConfig `json:"dialers,omitempty"`
|
||||
Heartbeats *HeartbeatsConfig `json:"heartbeats,omitempty"`
|
||||
PayloadSenderQueueSize int `json:"payloadSenderQueueSize,omitempty"`
|
||||
AckSenderQueueSize int `json:"ackSenderQueueSize,omitempty"`
|
||||
// GcMode is the auto-GC policy for links this router's own configuration
|
||||
// has made stale: "preserve" (default, never close on that basis),
|
||||
// "orphaned" (close links whose supporting listener/dialer is entirely
|
||||
// gone), or "changed" (close links whose details have shifted). Empty
|
||||
// string is treated as "preserve".
|
||||
//
|
||||
// The policy covers local staleness only. A peer withdrawing or moving a
|
||||
// listener still closes the affected link at once, under every mode,
|
||||
// because the peer's advertised listener set is authoritative about what
|
||||
// it will accept. "preserve" is therefore not a guarantee that links
|
||||
// survive every config change.
|
||||
GcMode string `json:"gcMode,omitempty"`
|
||||
}
|
||||
|
||||
// GcMode names the auto-GC policy applied by the router after each
|
||||
// successful link-config Apply. Mirrors the CLI `--mode` for
|
||||
// `ziti ops verify stale-links`, plus a `Preserve` value that disables
|
||||
// local-staleness GC entirely. It governs only links made stale by this
|
||||
// router's own configuration; peer-driven closures are unaffected.
|
||||
type GcMode int
|
||||
// The router.link config definition and validation live in common/config/routerlink
|
||||
// so the controller applies the same rules without importing the link subsystem.
|
||||
|
||||
const (
|
||||
GcModePreserve GcMode = iota
|
||||
GcModeOrphaned
|
||||
GcModeChanged
|
||||
ConfigBaseType = routerlink.ConfigBaseType
|
||||
ConfigTypeV1 = routerlink.ConfigTypeV1
|
||||
)
|
||||
|
||||
func (m GcMode) String() string {
|
||||
switch m {
|
||||
case GcModeOrphaned:
|
||||
return "orphaned"
|
||||
case GcModeChanged:
|
||||
return "changed"
|
||||
default:
|
||||
return "preserve"
|
||||
}
|
||||
}
|
||||
type (
|
||||
Config = routerlink.Config
|
||||
ListenerConfig = routerlink.ListenerConfig
|
||||
DialerConfig = routerlink.DialerConfig
|
||||
ChannelOptions = routerlink.ChannelOptions
|
||||
BackoffConfig = routerlink.BackoffConfig
|
||||
HeartbeatsConfig = routerlink.HeartbeatsConfig
|
||||
HeartbeatDurations = routerlink.HeartbeatDurations
|
||||
Groups = routerlink.Groups
|
||||
GcMode = routerlink.GcMode
|
||||
)
|
||||
|
||||
// ParseGcMode normalizes the string form (as it appears in the JSON
|
||||
// config or local YAML) into the enum. Unknown values return
|
||||
// GcModePreserve and an error; the caller decides whether to fall back
|
||||
// or reject the config.
|
||||
func ParseGcMode(s string) (GcMode, error) {
|
||||
switch s {
|
||||
case "", "preserve":
|
||||
return GcModePreserve, nil
|
||||
case "orphaned":
|
||||
return GcModeOrphaned, nil
|
||||
case "changed":
|
||||
return GcModeChanged, nil
|
||||
default:
|
||||
return GcModePreserve, fmt.Errorf("unknown gcMode %q (expected preserve|orphaned|changed)", s)
|
||||
}
|
||||
}
|
||||
const (
|
||||
GcModePreserve = routerlink.GcModePreserve
|
||||
GcModeOrphaned = routerlink.GcModeOrphaned
|
||||
GcModeChanged = routerlink.GcModeChanged
|
||||
)
|
||||
|
||||
// ListenerConfig matches the schema's listener entry. Bind is the only
|
||||
// required field.
|
||||
type ListenerConfig struct {
|
||||
Binding string `json:"binding,omitempty"`
|
||||
Bind string `json:"bind"`
|
||||
Advertise string `json:"advertise,omitempty"`
|
||||
BindInterface string `json:"bindInterface,omitempty"`
|
||||
Groups Groups `json:"groups,omitempty"`
|
||||
Options *ChannelOptions `json:"options,omitempty"`
|
||||
}
|
||||
// ParseConfig unmarshals raw JSON into a Config.
|
||||
func ParseConfig(data string) (*Config, error) { return routerlink.ParseConfig(data) }
|
||||
|
||||
// DialerConfig matches the schema's dialer entry.
|
||||
//
|
||||
// Split is carried here for local-config fidelity only. The router's own YAML
|
||||
// may set the transport's `split` dialer flag (the legacy split-vs-single
|
||||
// fallback used when dialing a router too old for multi-underlay links);
|
||||
// dropping it on translation would silently flip an explicit `split: false`
|
||||
// back to the transport default of true. It is deliberately absent from the
|
||||
// controller-managed router.link schema, so it cannot be set from the
|
||||
// controller. A nil pointer means "unset", letting the transport default apply.
|
||||
type DialerConfig struct {
|
||||
Binding string `json:"binding,omitempty"`
|
||||
MaxDefaultConnections int `json:"maxDefaultConnections,omitempty"`
|
||||
MaxAckConnections *int `json:"maxAckConnections,omitempty"`
|
||||
Split *bool `json:"split,omitempty"`
|
||||
StartupDelay string `json:"startupDelay,omitempty"`
|
||||
BindInterface string `json:"bindInterface,omitempty"`
|
||||
Groups Groups `json:"groups,omitempty"`
|
||||
HealthyDialBackoff *BackoffConfig `json:"healthyDialBackoff,omitempty"`
|
||||
UnhealthyDialBackoff *BackoffConfig `json:"unhealthyDialBackoff,omitempty"`
|
||||
Options *ChannelOptions `json:"options,omitempty"`
|
||||
}
|
||||
|
||||
// ChannelOptions matches the shared channelOptions definition.
|
||||
//
|
||||
// OutQueueSize and MaxQueuedConnects are pointers because zero is a valid
|
||||
// setting for both — each sizes a buffered channel, and zero means unbuffered.
|
||||
// A plain int with omitempty would make an explicit 0 indistinguishable from
|
||||
// unset and silently substitute the default.
|
||||
type ChannelOptions struct {
|
||||
OutQueueSize *int `json:"outQueueSize,omitempty"`
|
||||
MaxQueuedConnects *int `json:"maxQueuedConnects,omitempty"`
|
||||
MaxOutstandingConnects int `json:"maxOutstandingConnects,omitempty"`
|
||||
ConnectTimeout string `json:"connectTimeout,omitempty"`
|
||||
WriteTimeout string `json:"writeTimeout,omitempty"`
|
||||
}
|
||||
|
||||
// BackoffConfig matches the shared backoff definition.
|
||||
type BackoffConfig struct {
|
||||
RetryBackoffFactor float64 `json:"retryBackoffFactor,omitempty"`
|
||||
MinRetryInterval string `json:"minRetryInterval,omitempty"`
|
||||
MaxRetryInterval string `json:"maxRetryInterval,omitempty"`
|
||||
}
|
||||
|
||||
// HeartbeatsConfig matches the heartbeats definition.
|
||||
type HeartbeatsConfig struct {
|
||||
SendInterval string `json:"sendInterval,omitempty"`
|
||||
CheckInterval string `json:"checkInterval,omitempty"`
|
||||
CloseUnresponsiveTimeout string `json:"closeUnresponsiveTimeout,omitempty"`
|
||||
}
|
||||
|
||||
// Groups is a list of group names. The schema allows the JSON value to be
|
||||
// either a single string or an array of strings; UnmarshalJSON normalizes
|
||||
// both to a slice. MarshalJSON always emits an array for downstream
|
||||
// consumers that prefer one shape.
|
||||
type Groups []string
|
||||
|
||||
// UnmarshalJSON accepts either a string or an array of strings per schema.
|
||||
func (g *Groups) UnmarshalJSON(data []byte) error {
|
||||
var single string
|
||||
if err := json.Unmarshal(data, &single); err == nil {
|
||||
*g = Groups{single}
|
||||
return nil
|
||||
}
|
||||
var slice []string
|
||||
if err := json.Unmarshal(data, &slice); err != nil {
|
||||
return err
|
||||
}
|
||||
*g = Groups(slice)
|
||||
return nil
|
||||
}
|
||||
|
||||
// MarshalJSON always emits the array form for stable downstream parsing.
|
||||
func (g Groups) MarshalJSON() ([]byte, error) {
|
||||
if g == nil {
|
||||
return []byte("null"), nil
|
||||
}
|
||||
return json.Marshal([]string(g))
|
||||
}
|
||||
|
||||
// ParseConfig unmarshals raw JSON into a Config. Returns an error if the
|
||||
// JSON is malformed; does not enforce schema semantics (the schema is
|
||||
// enforced at the controller).
|
||||
func ParseConfig(data string) (*Config, error) {
|
||||
var c Config
|
||||
if err := json.Unmarshal([]byte(data), &c); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &c, nil
|
||||
}
|
||||
|
||||
// Validate checks values the controller JSON schema can't express and that
|
||||
// would otherwise fail silently downstream. Callers run it before applying a
|
||||
// config so a bad value fails the apply rather than degrading behavior.
|
||||
func (c *Config) Validate() error {
|
||||
for i := range c.Listeners {
|
||||
if err := validateChannelOptions(c.Listeners[i].Options); err != nil {
|
||||
return fmt.Errorf("listeners[%d].options: %w", i, err)
|
||||
}
|
||||
}
|
||||
for i := range c.Dialers {
|
||||
if err := validateChannelOptions(c.Dialers[i].Options); err != nil {
|
||||
return fmt.Errorf("dialers[%d].options: %w", i, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// validateChannelOptions rejects a connectTimeout that the downstream channel
|
||||
// can't honor: it must parse as a duration and fall within
|
||||
// [channel.MinConnectTimeout, channel.MaxConnectTimeout], the range the channel
|
||||
// loader accepts.
|
||||
func validateChannelOptions(o *ChannelOptions) error {
|
||||
if o == nil || o.ConnectTimeout == "" {
|
||||
return nil
|
||||
}
|
||||
d, err := time.ParseDuration(o.ConnectTimeout)
|
||||
if err != nil {
|
||||
return fmt.Errorf("invalid connectTimeout %q: %w", o.ConnectTimeout, err)
|
||||
}
|
||||
if d < channel.MinConnectTimeout {
|
||||
return fmt.Errorf("connectTimeout %s is below the minimum of %s", d, channel.MinConnectTimeout)
|
||||
}
|
||||
if d > channel.MaxConnectTimeout {
|
||||
return fmt.Errorf("connectTimeout %s is above the maximum of %s", d, channel.MaxConnectTimeout)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
// ParseGcMode normalizes the string form of gcMode into the enum.
|
||||
func ParseGcMode(s string) (GcMode, error) { return routerlink.ParseGcMode(s) }
|
||||
|
||||
@@ -46,6 +46,7 @@ type Env interface {
|
||||
GetLinkDialerPool() goroutines.Pool
|
||||
GetRateLimiterPool() goroutines.Pool
|
||||
GetMetricsRegistry() servermetrics.UsageRegistry
|
||||
GetHeartbeatSettings() xlink.HeartbeatSettings
|
||||
}
|
||||
|
||||
func NewLinkRegistry(routerEnv Env) xlink.Registry {
|
||||
@@ -289,6 +290,11 @@ func (self *linkRegistryImpl) applyLink(link xlink.Xlink) (xlink.Xlink, bool) {
|
||||
|
||||
self.updateLinkStateEstablished(link)
|
||||
|
||||
// A link samples its heartbeat settings while binding and only joins the
|
||||
// registry afterwards, so a change pushed in between missed it. A no-op when the
|
||||
// link already has the current generation.
|
||||
link.UpdateHeartbeat(self.env.GetHeartbeatSettings())
|
||||
|
||||
log.Info("link registered")
|
||||
|
||||
return nil, true
|
||||
|
||||
@@ -110,14 +110,25 @@ func (self *testEnv) GetRateLimiterPool() goroutines.Pool {
|
||||
panic("implement me")
|
||||
}
|
||||
|
||||
func (self *testEnv) GetHeartbeatSettings() xlink.HeartbeatSettings {
|
||||
return xlink.HeartbeatSettings{
|
||||
Generation: 1,
|
||||
SendInterval: 10 * time.Second,
|
||||
CheckInterval: time.Second,
|
||||
CloseUnresponsiveTimeout: time.Minute,
|
||||
}
|
||||
}
|
||||
|
||||
func (self *testEnv) GetMetricsRegistry() servermetrics.UsageRegistry {
|
||||
return self.metricsRegistry
|
||||
}
|
||||
|
||||
type testLink struct {
|
||||
id string
|
||||
key string
|
||||
destId string
|
||||
id string
|
||||
key string
|
||||
destId string
|
||||
hbLock sync.Mutex
|
||||
hbPushed []xlink.HeartbeatSettings
|
||||
}
|
||||
|
||||
func (self *testLink) GetDestinationType() string {
|
||||
@@ -164,6 +175,28 @@ func (self *testLink) DestinationId() string {
|
||||
return self.destId
|
||||
}
|
||||
|
||||
func (self *testLink) SetHeartbeatControl(channel.HeartbeatControl, xlink.HeartbeatSettings) {}
|
||||
func (self *testLink) HeartbeatSettings() xlink.HeartbeatSettings {
|
||||
self.hbLock.Lock()
|
||||
defer self.hbLock.Unlock()
|
||||
if len(self.hbPushed) == 0 {
|
||||
return xlink.HeartbeatSettings{}
|
||||
}
|
||||
return self.hbPushed[len(self.hbPushed)-1]
|
||||
}
|
||||
|
||||
func (self *testLink) heartbeatPushes() []xlink.HeartbeatSettings {
|
||||
self.hbLock.Lock()
|
||||
defer self.hbLock.Unlock()
|
||||
return append([]xlink.HeartbeatSettings(nil), self.hbPushed...)
|
||||
}
|
||||
|
||||
func (self *testLink) UpdateHeartbeat(settings xlink.HeartbeatSettings) {
|
||||
self.hbLock.Lock()
|
||||
defer self.hbLock.Unlock()
|
||||
self.hbPushed = append(self.hbPushed, settings)
|
||||
}
|
||||
|
||||
func (self *testLink) DestVersion() string {
|
||||
panic("implement me")
|
||||
}
|
||||
@@ -185,7 +218,7 @@ func (self *testLink) CloseOnce(func()) {
|
||||
}
|
||||
|
||||
func (self *testLink) IsClosed() bool {
|
||||
panic("implement me")
|
||||
return false
|
||||
}
|
||||
|
||||
func (self *testLink) InspectLink() *inspect.LinkInspectDetail {
|
||||
@@ -197,11 +230,11 @@ func (self *testLink) GetLinkConnState() *ctrl_pb.LinkConnState {
|
||||
}
|
||||
|
||||
func (self *testLink) IsDialed() bool {
|
||||
panic("implement me")
|
||||
return false
|
||||
}
|
||||
|
||||
func (self *testLink) Iteration() uint32 {
|
||||
panic("implement me")
|
||||
return 0
|
||||
}
|
||||
|
||||
func (self *testLink) AreFaultsSent() bool {
|
||||
@@ -804,3 +837,24 @@ func Test_LinkRegistry_GetDestinationListeners_OmitsUnhealthyDestinations(t *tes
|
||||
return !present
|
||||
}, 2*time.Second, 25*time.Millisecond, "unhealthy destination should drop out of the snapshot")
|
||||
}
|
||||
|
||||
func Test_LinkRegistry_ReconcilesHeartbeatsOnRegistration(t *testing.T) {
|
||||
// A link samples its heartbeat settings while binding and only reaches the
|
||||
// registry afterwards. A change pushed in that window iterates the registry
|
||||
// and cannot see it, so registration has to reconcile or the link runs the
|
||||
// previous generation forever while reading the current timeout.
|
||||
req := require.New(t)
|
||||
tenv := newTestEnv()
|
||||
defer close(tenv.closeNotify)
|
||||
|
||||
reg := NewLinkRegistry(tenv).(*linkRegistryImpl)
|
||||
|
||||
link := &testLink{id: "l1", key: "k1", destId: "dest1"}
|
||||
existing, applied := reg.DialSucceeded(link)
|
||||
req.Nil(existing)
|
||||
req.True(applied)
|
||||
|
||||
pushes := link.heartbeatPushes()
|
||||
req.Len(pushes, 1, "registration must push the current generation")
|
||||
req.Equal(tenv.GetHeartbeatSettings(), pushes[0])
|
||||
}
|
||||
|
||||
+36
-31
@@ -22,7 +22,9 @@ import (
|
||||
"math"
|
||||
"time"
|
||||
|
||||
"github.com/michaelquigley/pfxlog"
|
||||
"github.com/openziti/channel/v5"
|
||||
"github.com/openziti/ziti/v2/common/config/routerlink"
|
||||
)
|
||||
|
||||
// LocalYamlConfig is the slice of router env config we need to translate
|
||||
@@ -30,6 +32,8 @@ import (
|
||||
// from router/env (which would create a circular import via router/link's
|
||||
// other consumers).
|
||||
type LocalYamlConfig struct {
|
||||
// Configured reports that the router's config file has a link section.
|
||||
Configured bool
|
||||
Listeners []map[interface{}]interface{}
|
||||
Dialers []map[interface{}]interface{}
|
||||
Heartbeats channel.HeartbeatOptions
|
||||
@@ -40,20 +44,21 @@ type LocalYamlConfig struct {
|
||||
|
||||
// ConfigFromLocalYaml converts the YAML-decoded router config slice into a
|
||||
// typed Config and serializes it to JSON, ready for
|
||||
// managedconfig.Registry.ApplyLocal. Returns ("", nil) when no listeners
|
||||
// or dialers are configured locally — caller treats that as "no local
|
||||
// config; controller may manage."
|
||||
// managedconfig.Registry.ApplyLocal. Returns ("", nil) when the router config
|
||||
// has no link section, which leaves the router's links to the controller.
|
||||
//
|
||||
// Heartbeats and queue sizes alone don't count as local content: the
|
||||
// router env loader auto-fills heartbeat defaults regardless of whether
|
||||
// the YAML has a `link:` section, so treating those as "operator set
|
||||
// local config" would suppress controller management for every router.
|
||||
// Operator intent for local-wins requires explicit listeners or dialers.
|
||||
// A link section of any content is local config, and local config takes
|
||||
// precedence over the controller's for the whole router.link type, so one that
|
||||
// sets only heartbeats or queue sizes still leaves the router with no listeners
|
||||
// or dialers of its own. That is warned about rather than rejected.
|
||||
func ConfigFromLocalYaml(in LocalYamlConfig) (string, error) {
|
||||
hasContent := len(in.Listeners) > 0 || len(in.Dialers) > 0
|
||||
if !hasContent {
|
||||
if !in.Configured && len(in.Listeners) == 0 && len(in.Dialers) == 0 {
|
||||
return "", nil
|
||||
}
|
||||
if len(in.Listeners) == 0 && len(in.Dialers) == 0 {
|
||||
pfxlog.Logger().Warn("router config has a link section with no listeners or dialers; it takes precedence " +
|
||||
"over any controller-managed link config, so this router will have no links")
|
||||
}
|
||||
|
||||
cfg := Config{
|
||||
PayloadSenderQueueSize: in.PayloadSenderQueueSize,
|
||||
@@ -214,30 +219,30 @@ func yamlBackoff(m map[interface{}]interface{}, key string) (*BackoffConfig, err
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// heartbeatsFromYamlOptions converts the channel-package HeartbeatOptions
|
||||
// (already parsed by the router env loader) into the typed
|
||||
// HeartbeatsConfig in duration-string form for round-trip through the
|
||||
// router.link.v1 JSON shape.
|
||||
// heartbeatsFromYamlOptions converts env-loaded HeartbeatOptions into a
|
||||
// HeartbeatsConfig. The env loader defaults absent fields, so a zero is explicit
|
||||
// and is kept as "0s". A closeUnresponsiveTimeout below the floor is raised to it,
|
||||
// and any other invalid timing is replaced by the defaults, both with a warning,
|
||||
// since local link config that fails validation stops the router from starting.
|
||||
// Returns nil when the defaults apply.
|
||||
func heartbeatsFromYamlOptions(h channel.HeartbeatOptions) *HeartbeatsConfig {
|
||||
if !heartbeatsConfigured(h) {
|
||||
closeTimeout := h.CloseUnresponsiveTimeout
|
||||
if closeTimeout > 0 && closeTimeout < routerlink.MinCloseUnresponsiveTimeout {
|
||||
pfxlog.Logger().Warnf("link heartbeat closeUnresponsiveTimeout %s in router config is below the %s minimum, using the minimum",
|
||||
closeTimeout, routerlink.MinCloseUnresponsiveTimeout)
|
||||
closeTimeout = routerlink.MinCloseUnresponsiveTimeout
|
||||
}
|
||||
|
||||
hb := &HeartbeatsConfig{
|
||||
SendInterval: h.SendInterval.String(),
|
||||
CheckInterval: h.CheckInterval.String(),
|
||||
CloseUnresponsiveTimeout: closeTimeout.String(),
|
||||
}
|
||||
if _, err := (&Config{Heartbeats: hb}).EffectiveHeartbeats(); err != nil {
|
||||
pfxlog.Logger().WithError(err).Warn("link heartbeat timing in router config is invalid, using the defaults")
|
||||
return nil
|
||||
}
|
||||
return &HeartbeatsConfig{
|
||||
SendInterval: durationString(h.SendInterval),
|
||||
CheckInterval: durationString(h.CheckInterval),
|
||||
CloseUnresponsiveTimeout: durationString(h.CloseUnresponsiveTimeout),
|
||||
}
|
||||
}
|
||||
|
||||
func heartbeatsConfigured(h channel.HeartbeatOptions) bool {
|
||||
return h.SendInterval != 0 || h.CheckInterval != 0 || h.CloseUnresponsiveTimeout != 0
|
||||
}
|
||||
|
||||
func durationString(d time.Duration) string {
|
||||
if d == 0 {
|
||||
return ""
|
||||
}
|
||||
return d.String()
|
||||
return hb
|
||||
}
|
||||
|
||||
// yamlString reads m[key] as string; returns "" if absent.
|
||||
|
||||
@@ -20,6 +20,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/openziti/channel/v5"
|
||||
"github.com/openziti/metrics"
|
||||
"github.com/openziti/sdk-golang/v2/xgress"
|
||||
"github.com/openziti/ziti/v2/common/inspect"
|
||||
@@ -40,14 +41,19 @@ type stubXlink struct {
|
||||
closed bool
|
||||
}
|
||||
|
||||
func (*stubXlink) GetDestinationType() string { return "link" }
|
||||
func (s *stubXlink) Id() string { return s.id }
|
||||
func (s *stubXlink) Key() string { return s.linkKey.String() }
|
||||
func (s *stubXlink) DialAddress() string { return s.dialAddress }
|
||||
func (s *stubXlink) LinkKey() xlink.LinkKey { return s.linkKey }
|
||||
func (s *stubXlink) LinkProtocol() string { return s.linkProtocol }
|
||||
func (s *stubXlink) IsDialed() bool { return s.dialed }
|
||||
func (s *stubXlink) IsClosed() bool { return s.closed }
|
||||
func (*stubXlink) GetDestinationType() string { return "link" }
|
||||
func (*stubXlink) SetHeartbeatControl(channel.HeartbeatControl, xlink.HeartbeatSettings) {}
|
||||
func (*stubXlink) UpdateHeartbeat(xlink.HeartbeatSettings) {}
|
||||
func (*stubXlink) HeartbeatSettings() xlink.HeartbeatSettings {
|
||||
return xlink.HeartbeatSettings{}
|
||||
}
|
||||
func (s *stubXlink) Id() string { return s.id }
|
||||
func (s *stubXlink) Key() string { return s.linkKey.String() }
|
||||
func (s *stubXlink) DialAddress() string { return s.dialAddress }
|
||||
func (s *stubXlink) LinkKey() xlink.LinkKey { return s.linkKey }
|
||||
func (s *stubXlink) LinkProtocol() string { return s.linkProtocol }
|
||||
func (s *stubXlink) IsDialed() bool { return s.dialed }
|
||||
func (s *stubXlink) IsClosed() bool { return s.closed }
|
||||
|
||||
// Required by the interface but not exercised by these tests.
|
||||
func (*stubXlink) Iteration() uint32 { return 0 }
|
||||
|
||||
+176
-17
@@ -22,11 +22,29 @@ import (
|
||||
"sync"
|
||||
|
||||
"github.com/michaelquigley/pfxlog"
|
||||
"github.com/openziti/foundation/v2/concurrenz"
|
||||
"github.com/openziti/identity"
|
||||
"github.com/openziti/transport/v2"
|
||||
"github.com/openziti/ziti/v2/common/config/routerlink"
|
||||
"github.com/openziti/ziti/v2/router/xlink"
|
||||
)
|
||||
|
||||
// Aliases for the heartbeat fallbacks, which routerlink owns as part of the
|
||||
// config contract.
|
||||
const (
|
||||
defaultCloseUnresponsiveTimeout = routerlink.DefaultCloseUnresponsiveTimeout
|
||||
defaultHeartbeatSendInterval = routerlink.DefaultHeartbeatSendInterval
|
||||
defaultHeartbeatCheckInterval = routerlink.DefaultHeartbeatCheckInterval
|
||||
)
|
||||
|
||||
// Link sender queue size fallbacks, matching the router env defaults
|
||||
// (DefaultLinkPayloadSenderQueueSize / DefaultLinkAckSenderQueueSize), used
|
||||
// when no link config supplies them.
|
||||
const (
|
||||
defaultPayloadSenderQueueSize = 128
|
||||
defaultAckSenderQueueSize = 64
|
||||
)
|
||||
|
||||
// Subsystem owns the router's configurable link surface: the set of
|
||||
// xlink.Factory implementations registered for each binding name, the
|
||||
// currently-applied router.link.v1 config, and the listener / dialer
|
||||
@@ -57,9 +75,84 @@ type Subsystem struct {
|
||||
listeners []xlink.Listener
|
||||
dialers []xlink.Dialer
|
||||
|
||||
// heartbeats is the heartbeat timing in force, swapped whole on Apply and
|
||||
// Remove so a reader never pairs one generation's interval with another's
|
||||
// timeout.
|
||||
heartbeats concurrenz.AtomicValue[xlink.HeartbeatSettings]
|
||||
// generation identifies the applied config. Bumped under mu with the listener
|
||||
// and dialer swap, so a reader holding mu sees a generation and the state it
|
||||
// describes together.
|
||||
generation uint64
|
||||
|
||||
changeHandler ConfigurationChangeHandler
|
||||
}
|
||||
|
||||
// Heartbeats returns the heartbeat timing currently in force as one snapshot.
|
||||
// Lock-free. Until a config has been applied it returns the defaults at
|
||||
// generation zero.
|
||||
func (self *Subsystem) Heartbeats() xlink.HeartbeatSettings {
|
||||
settings := self.heartbeats.Load()
|
||||
if settings.Generation == 0 {
|
||||
return xlink.HeartbeatSettings{
|
||||
SendInterval: defaultHeartbeatSendInterval,
|
||||
CheckInterval: defaultHeartbeatCheckInterval,
|
||||
CloseUnresponsiveTimeout: defaultCloseUnresponsiveTimeout,
|
||||
}
|
||||
}
|
||||
return settings
|
||||
}
|
||||
|
||||
// applyHeartbeats resolves cfg's heartbeat timing and publishes it as the next
|
||||
// generation; a nil cfg, on Remove, publishes the defaults. Validate has already
|
||||
// accepted cfg, so a resolution failure falls back to the defaults.
|
||||
func (self *Subsystem) applyHeartbeats(cfg *Config, generation uint64) {
|
||||
timings := HeartbeatDurations{
|
||||
SendInterval: defaultHeartbeatSendInterval,
|
||||
CheckInterval: defaultHeartbeatCheckInterval,
|
||||
CloseUnresponsiveTimeout: defaultCloseUnresponsiveTimeout,
|
||||
}
|
||||
if cfg != nil {
|
||||
if resolved, err := cfg.EffectiveHeartbeats(); err == nil {
|
||||
timings = resolved
|
||||
} else {
|
||||
pfxlog.Logger().WithError(err).
|
||||
Error("applied link config has unresolvable heartbeat timing; using defaults")
|
||||
}
|
||||
}
|
||||
|
||||
self.heartbeats.Store(xlink.HeartbeatSettings{
|
||||
Generation: generation,
|
||||
SendInterval: timings.SendInterval,
|
||||
CheckInterval: timings.CheckInterval,
|
||||
CloseUnresponsiveTimeout: timings.CloseUnresponsiveTimeout,
|
||||
})
|
||||
}
|
||||
|
||||
// PayloadSenderQueueSize returns the effective link payload sender queue size
|
||||
// from the applied config, falling back to the default when unset. Read when a
|
||||
// link channel is built, so a change takes effect on links established after
|
||||
// it; existing links keep the size they were built with.
|
||||
func (self *Subsystem) PayloadSenderQueueSize() int {
|
||||
self.mu.RLock()
|
||||
defer self.mu.RUnlock()
|
||||
if self.config != nil && self.config.PayloadSenderQueueSize > 0 {
|
||||
return self.config.PayloadSenderQueueSize
|
||||
}
|
||||
return defaultPayloadSenderQueueSize
|
||||
}
|
||||
|
||||
// AckSenderQueueSize returns the effective link ack sender queue size from the
|
||||
// applied config, falling back to the default when unset. Like
|
||||
// PayloadSenderQueueSize, it applies to links established after a change.
|
||||
func (self *Subsystem) AckSenderQueueSize() int {
|
||||
self.mu.RLock()
|
||||
defer self.mu.RUnlock()
|
||||
if self.config != nil && self.config.AckSenderQueueSize > 0 {
|
||||
return self.config.AckSenderQueueSize
|
||||
}
|
||||
return defaultAckSenderQueueSize
|
||||
}
|
||||
|
||||
// ConfigurationChange describes which parts of the link configuration
|
||||
// changed during an Apply / Remove. Any flag may be true; consumers
|
||||
// inspect them to decide what work to do.
|
||||
@@ -81,6 +174,9 @@ type ConfigurationChange struct {
|
||||
// to read. A removal uses the policy of the config being removed, which is
|
||||
// the last one the operator set.
|
||||
GcMode GcMode
|
||||
// HeartbeatsChanged is true when any heartbeat field differs from the
|
||||
// previous state; the handler pushes the new generation to established links.
|
||||
HeartbeatsChanged bool
|
||||
}
|
||||
|
||||
// ConfigurationChangeHandler is invoked asynchronously after a successful
|
||||
@@ -199,9 +295,25 @@ func (self *Subsystem) Apply(version int, data string) error {
|
||||
}
|
||||
self.config = cfg
|
||||
self.appliedData = data
|
||||
self.generation++
|
||||
generation := self.generation
|
||||
handler := self.changeHandler
|
||||
self.mu.Unlock()
|
||||
|
||||
// Published whether or not the listener and dialer set moved, since it applies
|
||||
// to established links too.
|
||||
self.applyHeartbeats(cfg, generation)
|
||||
|
||||
// A thin margin is warned about rather than rejected: the config works until a
|
||||
// response is lost, and on the local path a rejection stops startup.
|
||||
if timings, err := cfg.EffectiveHeartbeats(); err == nil && timings.HasThinMargin() {
|
||||
pfxlog.Logger().
|
||||
WithField("sendInterval", timings.SendInterval).
|
||||
WithField("checkInterval", timings.CheckInterval).
|
||||
WithField("closeUnresponsiveTimeout", timings.CloseUnresponsiveTimeout).
|
||||
Warn("link heartbeat timeout leaves less than one heartbeat cycle of margin; a single lost response may close a healthy link")
|
||||
}
|
||||
|
||||
if rebuild {
|
||||
closeListeners(oldListeners)
|
||||
if err := startListeners(newListeners); err != nil {
|
||||
@@ -209,7 +321,21 @@ func (self *Subsystem) Apply(version int, data string) error {
|
||||
}
|
||||
}
|
||||
|
||||
notifyChange(handler, prevConfig, cfg)
|
||||
change := newConfigurationChange(prevConfig, cfg)
|
||||
|
||||
// Logged on every apply, including one that dispatches nothing, so there is
|
||||
// always a record that the router received the config.
|
||||
pfxlog.Logger().
|
||||
WithField("generation", generation).
|
||||
WithField("listenersChanged", change.ListenersChanged).
|
||||
WithField("dialersChanged", change.DialersChanged).
|
||||
WithField("gcModeChanged", change.GcModeChanged).
|
||||
WithField("heartbeatsChanged", change.HeartbeatsChanged).
|
||||
WithField("payloadSenderQueueSize", cfg.PayloadSenderQueueSize).
|
||||
WithField("ackSenderQueueSize", cfg.AckSenderQueueSize).
|
||||
Info("applied link configuration")
|
||||
|
||||
notifyChange(handler, change)
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -224,35 +350,58 @@ func (self *Subsystem) Remove() error {
|
||||
self.dialers = nil
|
||||
self.config = nil
|
||||
self.appliedData = ""
|
||||
self.generation++
|
||||
generation := self.generation
|
||||
handler := self.changeHandler
|
||||
self.mu.Unlock()
|
||||
|
||||
// A removed config's heartbeat settings revert to the defaults, published as a
|
||||
// new generation so established links are re-tuned off the removed values.
|
||||
self.applyHeartbeats(nil, generation)
|
||||
|
||||
closeListeners(oldListeners)
|
||||
notifyChange(handler, prevConfig, nil)
|
||||
notifyChange(handler, newConfigurationChange(prevConfig, nil))
|
||||
return nil
|
||||
}
|
||||
|
||||
// notifyChange invokes handler asynchronously when prev and next differ.
|
||||
// nil handler is a no-op. Compares listeners and dialers slices by
|
||||
// deep-equality; identical state is a no-op (avoids spurious peer
|
||||
// notifications and dialer rescans when an Apply was just re-applying
|
||||
// the same config).
|
||||
func notifyChange(handler ConfigurationChangeHandler, prev, next *Config) {
|
||||
if handler == nil {
|
||||
return
|
||||
// newConfigurationChange describes how next differs from prev. Listeners and
|
||||
// dialers are compared by deep equality. Queue sizes are not represented, since
|
||||
// they only affect links created later, so a change confined to them reports no
|
||||
// work.
|
||||
func newConfigurationChange(prev, next *Config) ConfigurationChange {
|
||||
return ConfigurationChange{
|
||||
ListenersChanged: !listenerSlicesEqual(getListeners(prev), getListeners(next)),
|
||||
DialersChanged: !dialerSlicesEqual(getEffectiveDialers(prev), getEffectiveDialers(next)),
|
||||
GcModeChanged: getGcMode(prev) != getGcMode(next),
|
||||
GcMode: effectiveGcMode(next),
|
||||
HeartbeatsChanged: getHeartbeats(prev) != getHeartbeats(next),
|
||||
}
|
||||
change := ConfigurationChange{
|
||||
ListenersChanged: !listenerSlicesEqual(getListeners(prev), getListeners(next)),
|
||||
DialersChanged: !dialerSlicesEqual(getEffectiveDialers(prev), getEffectiveDialers(next)),
|
||||
GcModeChanged: getGcMode(prev) != getGcMode(next),
|
||||
GcMode: effectiveGcMode(next),
|
||||
}
|
||||
if !change.ListenersChanged && !change.DialersChanged && !change.GcModeChanged {
|
||||
}
|
||||
|
||||
// hasWork reports whether the change asks anything of the handler. A change
|
||||
// that asks nothing is not dispatched.
|
||||
func (self ConfigurationChange) hasWork() bool {
|
||||
return self.ListenersChanged || self.DialersChanged || self.GcModeChanged || self.HeartbeatsChanged
|
||||
}
|
||||
|
||||
// notifyChange invokes handler asynchronously when change asks for work. A nil
|
||||
// handler is a no-op.
|
||||
func notifyChange(handler ConfigurationChangeHandler, change ConfigurationChange) {
|
||||
if handler == nil || !change.hasWork() {
|
||||
return
|
||||
}
|
||||
go handler(change)
|
||||
}
|
||||
|
||||
// getHeartbeats returns a comparable value of the config's heartbeat settings
|
||||
// (zero value when unset), so notifyChange can detect heartbeat changes.
|
||||
func getHeartbeats(c *Config) HeartbeatsConfig {
|
||||
if c == nil || c.Heartbeats == nil {
|
||||
return HeartbeatsConfig{}
|
||||
}
|
||||
return *c.Heartbeats
|
||||
}
|
||||
|
||||
func getGcMode(c *Config) string {
|
||||
if c == nil {
|
||||
return ""
|
||||
@@ -358,6 +507,16 @@ func (self *Subsystem) Listeners() []xlink.Listener {
|
||||
return out
|
||||
}
|
||||
|
||||
// ListenerSnapshot returns the current listener set and the config generation it
|
||||
// belongs to, read under one lock so the pair is consistent.
|
||||
func (self *Subsystem) ListenerSnapshot() (uint64, []xlink.Listener) {
|
||||
self.mu.RLock()
|
||||
defer self.mu.RUnlock()
|
||||
out := make([]xlink.Listener, len(self.listeners))
|
||||
copy(out, self.listeners)
|
||||
return self.generation, out
|
||||
}
|
||||
|
||||
// Dialers returns a snapshot of the current dialer slice.
|
||||
func (self *Subsystem) Dialers() []xlink.Dialer {
|
||||
self.mu.RLock()
|
||||
|
||||
@@ -26,6 +26,7 @@ import (
|
||||
"github.com/openziti/foundation/v2/util"
|
||||
"github.com/openziti/identity"
|
||||
"github.com/openziti/transport/v2"
|
||||
"github.com/openziti/ziti/v2/common/config/routerlink"
|
||||
"github.com/openziti/ziti/v2/router/xlink"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
@@ -151,6 +152,80 @@ func Test_Subsystem_Apply_BuildsListenersAndDialers(t *testing.T) {
|
||||
req.Equal("tls:0.0.0.0:6262", f.createdListeners[0].bind)
|
||||
}
|
||||
|
||||
func Test_Subsystem_CloseUnresponsiveTimeout(t *testing.T) {
|
||||
req := require.New(t)
|
||||
r, _ := newTestRegistry(t)
|
||||
|
||||
// before any apply, falls back to the default
|
||||
req.Equal(defaultCloseUnresponsiveTimeout, r.Heartbeats().CloseUnresponsiveTimeout)
|
||||
|
||||
// an applied config with the field takes effect (for established links too)
|
||||
req.NoError(r.Apply(1, `{"heartbeats":{"closeUnresponsiveTimeout":"45s"}}`))
|
||||
req.Equal(45*time.Second, r.Heartbeats().CloseUnresponsiveTimeout)
|
||||
|
||||
// the value comes from the active config, so a later apply that omits it
|
||||
// reverts to the default
|
||||
req.NoError(r.Apply(1, `{"listeners":[{"bind":"tls:0.0.0.0:6262"}]}`))
|
||||
req.Equal(defaultCloseUnresponsiveTimeout, r.Heartbeats().CloseUnresponsiveTimeout)
|
||||
}
|
||||
|
||||
func Test_Subsystem_HeartbeatIntervals(t *testing.T) {
|
||||
req := require.New(t)
|
||||
r, _ := newTestRegistry(t)
|
||||
|
||||
// before any apply, fall back to the channel defaults
|
||||
req.Equal(defaultHeartbeatSendInterval, r.Heartbeats().SendInterval)
|
||||
req.Equal(defaultHeartbeatCheckInterval, r.Heartbeats().CheckInterval)
|
||||
|
||||
// applied intervals take effect
|
||||
req.NoError(r.Apply(1, `{"heartbeats":{"sendInterval":"3s","checkInterval":"250ms"}}`))
|
||||
req.Equal(3*time.Second, r.Heartbeats().SendInterval)
|
||||
req.Equal(250*time.Millisecond, r.Heartbeats().CheckInterval)
|
||||
|
||||
// the settings come from the active config, so a later apply that omits
|
||||
// them reverts to the defaults
|
||||
req.NoError(r.Apply(1, `{"listeners":[{"bind":"tls:0.0.0.0:6262"}]}`))
|
||||
req.Equal(defaultHeartbeatSendInterval, r.Heartbeats().SendInterval)
|
||||
req.Equal(defaultHeartbeatCheckInterval, r.Heartbeats().CheckInterval)
|
||||
}
|
||||
|
||||
func Test_Subsystem_Remove_RevertsHeartbeatsToDefaults(t *testing.T) {
|
||||
req := require.New(t)
|
||||
r, _ := newTestRegistry(t)
|
||||
|
||||
req.NoError(r.Apply(1, `{"heartbeats":{"sendInterval":"3s","checkInterval":"250ms","closeUnresponsiveTimeout":"45s"}}`))
|
||||
req.Equal(3*time.Second, r.Heartbeats().SendInterval)
|
||||
req.Equal(250*time.Millisecond, r.Heartbeats().CheckInterval)
|
||||
req.Equal(45*time.Second, r.Heartbeats().CloseUnresponsiveTimeout)
|
||||
|
||||
// removing the config drops its heartbeat settings, reverting to the
|
||||
// defaults just as a later apply that omits them would
|
||||
req.NoError(r.Remove())
|
||||
req.Equal(defaultHeartbeatSendInterval, r.Heartbeats().SendInterval)
|
||||
req.Equal(defaultHeartbeatCheckInterval, r.Heartbeats().CheckInterval)
|
||||
req.Equal(defaultCloseUnresponsiveTimeout, r.Heartbeats().CloseUnresponsiveTimeout)
|
||||
}
|
||||
|
||||
func Test_Subsystem_QueueSizes(t *testing.T) {
|
||||
req := require.New(t)
|
||||
r, _ := newTestRegistry(t)
|
||||
|
||||
// before any apply, fall back to the defaults
|
||||
req.Equal(defaultPayloadSenderQueueSize, r.PayloadSenderQueueSize())
|
||||
req.Equal(defaultAckSenderQueueSize, r.AckSenderQueueSize())
|
||||
|
||||
// applied sizes take effect (for links established afterward)
|
||||
req.NoError(r.Apply(1, `{"payloadSenderQueueSize":256,"ackSenderQueueSize":96}`))
|
||||
req.Equal(256, r.PayloadSenderQueueSize())
|
||||
req.Equal(96, r.AckSenderQueueSize())
|
||||
|
||||
// the sizes come from the active config, so a later apply that omits them
|
||||
// reverts to the defaults
|
||||
req.NoError(r.Apply(1, `{"listeners":[{"bind":"tls:0.0.0.0:6262"}]}`))
|
||||
req.Equal(defaultPayloadSenderQueueSize, r.PayloadSenderQueueSize())
|
||||
req.Equal(defaultAckSenderQueueSize, r.AckSenderQueueSize())
|
||||
}
|
||||
|
||||
func Test_Subsystem_Apply_MapsDialerBindInterfaceToTransportBind(t *testing.T) {
|
||||
req := require.New(t)
|
||||
r, f := newTestRegistry(t)
|
||||
@@ -991,3 +1066,392 @@ func Test_EffectiveGcMode(t *testing.T) {
|
||||
// policy could apply.
|
||||
req.Equal(GcModeOrphaned, effectiveGcMode(nil))
|
||||
}
|
||||
|
||||
func Test_Config_Validate_HeartbeatTimings(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
data string
|
||||
expectErr string
|
||||
}{
|
||||
{
|
||||
name: "no heartbeats section uses defaults",
|
||||
data: `{"listeners":[{"bind":"tls:0.0.0.0:6262"}]}`,
|
||||
},
|
||||
{
|
||||
name: "sane explicit timings",
|
||||
data: `{"heartbeats":{"sendInterval":"10s","checkInterval":"1s","closeUnresponsiveTimeout":"60s"}}`,
|
||||
},
|
||||
{
|
||||
// The finding's case: the check interval outruns the timeout, so the
|
||||
// pulse sends a heartbeat and condemns the link on the same tick.
|
||||
name: "check interval past the timeout",
|
||||
data: `{"heartbeats":{"checkInterval":"2m","closeUnresponsiveTimeout":"30s"}}`,
|
||||
expectErr: "closeUnresponsiveTimeout",
|
||||
},
|
||||
{
|
||||
name: "timeout equal to send plus check",
|
||||
data: `{"heartbeats":{"sendInterval":"20s","checkInterval":"10s","closeUnresponsiveTimeout":"30s"}}`,
|
||||
expectErr: "must be greater than sendInterval + checkInterval",
|
||||
},
|
||||
{
|
||||
// A timeout below the default send interval is broken even though the
|
||||
// config never names sendInterval. Caught by the floor first.
|
||||
name: "timeout under the default send interval",
|
||||
data: `{"heartbeats":{"closeUnresponsiveTimeout":"5s"}}`,
|
||||
expectErr: "below the 10s minimum",
|
||||
},
|
||||
{
|
||||
// Intervals tight enough that the sum rule would pass: only the
|
||||
// absolute floor catches it. A timeout this short closes links that
|
||||
// are merely recovering from ordinary packet loss.
|
||||
name: "timeout below the floor despite roomy intervals",
|
||||
data: `{"heartbeats":{"sendInterval":"100ms","checkInterval":"100ms","closeUnresponsiveTimeout":"1s"}}`,
|
||||
expectErr: "below the 10s minimum",
|
||||
},
|
||||
{
|
||||
name: "timeout exactly at the floor is accepted",
|
||||
data: `{"heartbeats":{"sendInterval":"5s","checkInterval":"1s","closeUnresponsiveTimeout":"10s"}}`,
|
||||
},
|
||||
{
|
||||
// Durations are int64 nanoseconds and the sum of two positive ones can
|
||||
// only wrap negative, so the sum rule would compare against a negative
|
||||
// and accept every timeout.
|
||||
name: "interval sum overflows",
|
||||
data: `{"heartbeats":{"sendInterval":"2562047h","checkInterval":"1h","closeUnresponsiveTimeout":"60s"}}`,
|
||||
expectErr: "exceeds the largest representable duration",
|
||||
},
|
||||
{
|
||||
// Representable, so the overflow guard stands aside and the sum rule
|
||||
// judges it.
|
||||
name: "interval sum at the maximum is judged on its merits",
|
||||
data: `{"heartbeats":{"sendInterval":"2562047h47m16.854775806s","checkInterval":"1ns","closeUnresponsiveTimeout":"60s"}}`,
|
||||
expectErr: "must be greater than sendInterval + checkInterval",
|
||||
},
|
||||
{
|
||||
name: "zero check interval",
|
||||
data: `{"heartbeats":{"checkInterval":"0s"}}`,
|
||||
expectErr: "checkInterval must be positive",
|
||||
},
|
||||
{
|
||||
name: "negative send interval",
|
||||
data: `{"heartbeats":{"sendInterval":"-5s"}}`,
|
||||
expectErr: "sendInterval must be positive",
|
||||
},
|
||||
{
|
||||
name: "unparseable duration",
|
||||
data: `{"heartbeats":{"checkInterval":"soon"}}`,
|
||||
expectErr: "invalid heartbeats.checkInterval",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
req := require.New(t)
|
||||
cfg, err := ParseConfig(tt.data)
|
||||
req.NoError(err)
|
||||
|
||||
err = cfg.Validate()
|
||||
if tt.expectErr == "" {
|
||||
req.NoError(err)
|
||||
return
|
||||
}
|
||||
req.Error(err)
|
||||
req.Contains(err.Error(), tt.expectErr)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func Test_Subsystem_Apply_RejectsUnsafeHeartbeatTimings(t *testing.T) {
|
||||
// Validation has to run on the apply path, not just as a standalone call,
|
||||
// and a rejected apply must leave the previous settings in place.
|
||||
req := require.New(t)
|
||||
r, _ := newTestRegistry(t)
|
||||
|
||||
req.NoError(r.Apply(1, `{"heartbeats":{"sendInterval":"10s","checkInterval":"1s","closeUnresponsiveTimeout":"60s"}}`))
|
||||
req.Equal(time.Second, r.Heartbeats().CheckInterval)
|
||||
req.Equal(60*time.Second, r.Heartbeats().CloseUnresponsiveTimeout)
|
||||
|
||||
err := r.Apply(1, `{"heartbeats":{"checkInterval":"2m","closeUnresponsiveTimeout":"30s"}}`)
|
||||
req.Error(err)
|
||||
req.Contains(err.Error(), "closeUnresponsiveTimeout")
|
||||
|
||||
req.Equal(time.Second, r.Heartbeats().CheckInterval, "a rejected apply must not change the live intervals")
|
||||
req.Equal(60*time.Second, r.Heartbeats().CloseUnresponsiveTimeout)
|
||||
}
|
||||
|
||||
func Test_HeartbeatDurations_HasThinMargin(t *testing.T) {
|
||||
req := require.New(t)
|
||||
|
||||
mk := func(data string) HeartbeatDurations {
|
||||
cfg, err := ParseConfig(data)
|
||||
req.NoError(err)
|
||||
timings, err := cfg.EffectiveHeartbeats()
|
||||
req.NoError(err)
|
||||
return timings
|
||||
}
|
||||
|
||||
// Defaults: 60s over a 11s healthy gap, comfortably more than one cycle.
|
||||
req.False(mk(`{}`).HasThinMargin())
|
||||
|
||||
// Valid, but one lost response reaches the timeout.
|
||||
req.True(mk(`{"heartbeats":{"sendInterval":"30s","checkInterval":"5s","closeUnresponsiveTimeout":"60s"}}`).HasThinMargin())
|
||||
}
|
||||
|
||||
func Test_Subsystem_Heartbeats_GenerationAdvancesPerApply(t *testing.T) {
|
||||
req := require.New(t)
|
||||
r, _ := newTestRegistry(t)
|
||||
|
||||
// Before any apply: defaults at generation zero, so a link binding now and
|
||||
// a link binding after the first apply are distinguishable.
|
||||
initial := r.Heartbeats()
|
||||
req.Zero(initial.Generation)
|
||||
req.Equal(defaultHeartbeatSendInterval, initial.SendInterval)
|
||||
req.Equal(defaultCloseUnresponsiveTimeout, initial.CloseUnresponsiveTimeout)
|
||||
|
||||
req.NoError(r.Apply(1, `{"heartbeats":{"sendInterval":"3s","checkInterval":"250ms","closeUnresponsiveTimeout":"45s"}}`))
|
||||
first := r.Heartbeats()
|
||||
req.Greater(first.Generation, initial.Generation)
|
||||
req.Equal(3*time.Second, first.SendInterval)
|
||||
req.Equal(250*time.Millisecond, first.CheckInterval)
|
||||
req.Equal(45*time.Second, first.CloseUnresponsiveTimeout)
|
||||
|
||||
// A later apply that omits them reverts to defaults, as a new generation.
|
||||
req.NoError(r.Apply(1, `{"listeners":[{"bind":"tls:0.0.0.0:6262"}]}`))
|
||||
second := r.Heartbeats()
|
||||
req.Greater(second.Generation, first.Generation)
|
||||
req.Equal(defaultHeartbeatSendInterval, second.SendInterval)
|
||||
|
||||
// Remove also publishes a generation, so links are re-tuned off the removed
|
||||
// values rather than keeping them.
|
||||
req.NoError(r.Remove())
|
||||
third := r.Heartbeats()
|
||||
req.Greater(third.Generation, second.Generation)
|
||||
req.Equal(defaultCloseUnresponsiveTimeout, third.CloseUnresponsiveTimeout)
|
||||
}
|
||||
|
||||
func Test_Subsystem_Heartbeats_SnapshotIsCoherent(t *testing.T) {
|
||||
// The point of one atomic swap: a reader can never see one generation's
|
||||
// interval with another's timeout, however many applies race with it.
|
||||
req := require.New(t)
|
||||
r, _ := newTestRegistry(t)
|
||||
|
||||
configs := []string{
|
||||
`{"heartbeats":{"sendInterval":"1s","checkInterval":"100ms","closeUnresponsiveTimeout":"10s"}}`,
|
||||
`{"heartbeats":{"sendInterval":"20s","checkInterval":"5s","closeUnresponsiveTimeout":"5m"}}`,
|
||||
}
|
||||
valid := map[xlink.HeartbeatSettings]bool{}
|
||||
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
for i := 0; i < 200; i++ {
|
||||
_ = r.Apply(1, configs[i%len(configs)])
|
||||
}
|
||||
}()
|
||||
|
||||
for i := 0; i < 2000; i++ {
|
||||
s := r.Heartbeats()
|
||||
// Every observed snapshot must be one of the coherent combinations,
|
||||
// never a mix.
|
||||
coherent := (s.SendInterval == time.Second && s.CheckInterval == 100*time.Millisecond && s.CloseUnresponsiveTimeout == 10*time.Second) ||
|
||||
(s.SendInterval == 20*time.Second && s.CheckInterval == 5*time.Second && s.CloseUnresponsiveTimeout == 5*time.Minute) ||
|
||||
(s.SendInterval == defaultHeartbeatSendInterval && s.CheckInterval == defaultHeartbeatCheckInterval && s.CloseUnresponsiveTimeout == defaultCloseUnresponsiveTimeout)
|
||||
req.True(coherent, "observed a mixed snapshot: %+v", s)
|
||||
valid[s] = true
|
||||
}
|
||||
<-done
|
||||
req.NotEmpty(valid)
|
||||
}
|
||||
|
||||
func Test_Subsystem_ListenerSnapshot_PairsGenerationWithItsListeners(t *testing.T) {
|
||||
// Sampling the generation and the listeners separately would let a publisher
|
||||
// stamp one generation's listeners with another's number, which a receiver
|
||||
// discriminating on generation would then accept as the newest state.
|
||||
req := require.New(t)
|
||||
r, _ := newTestRegistry(t)
|
||||
|
||||
gen0, listeners0 := r.ListenerSnapshot()
|
||||
req.Zero(gen0)
|
||||
req.Empty(listeners0)
|
||||
|
||||
req.NoError(r.Apply(1, `{"listeners":[{"bind":"tls:0.0.0.0:6262"}]}`))
|
||||
gen1, listeners1 := r.ListenerSnapshot()
|
||||
req.Greater(gen1, gen0)
|
||||
req.Len(listeners1, 1)
|
||||
|
||||
req.NoError(r.Apply(1, `{"listeners":[{"bind":"tls:0.0.0.0:6263"},{"bind":"tls:0.0.0.0:6264"}]}`))
|
||||
gen2, listeners2 := r.ListenerSnapshot()
|
||||
req.Greater(gen2, gen1)
|
||||
req.Len(listeners2, 2)
|
||||
|
||||
// Under concurrent applies, every snapshot must be a real pairing: the
|
||||
// two-listener set never appears stamped with the one-listener generation.
|
||||
seen := map[uint64]int{}
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
for i := 0; i < 200; i++ {
|
||||
if i%2 == 0 {
|
||||
_ = r.Apply(1, `{"listeners":[{"bind":"tls:0.0.0.0:6262"}]}`)
|
||||
} else {
|
||||
_ = r.Apply(1, `{"listeners":[{"bind":"tls:0.0.0.0:6263"},{"bind":"tls:0.0.0.0:6264"}]}`)
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
for i := 0; i < 2000; i++ {
|
||||
gen, listeners := r.ListenerSnapshot()
|
||||
if prev, found := seen[gen]; found {
|
||||
req.Equal(prev, len(listeners),
|
||||
"generation %d reported %d listeners and then %d", gen, prev, len(listeners))
|
||||
}
|
||||
seen[gen] = len(listeners)
|
||||
}
|
||||
<-done
|
||||
req.NotEmpty(seen)
|
||||
}
|
||||
|
||||
func Test_LocalYaml_ClampsTimeoutBelowFloor(t *testing.T) {
|
||||
// Local link config is applied synchronously and a failure stops the router
|
||||
// from starting, so a below-floor value from router YAML is raised to the
|
||||
// floor rather than rejected. The managed path rejects it instead, where the
|
||||
// operator sees the error on the API call and nothing is already running.
|
||||
req := require.New(t)
|
||||
|
||||
hb := heartbeatsFromYamlOptions(channel.HeartbeatOptions{
|
||||
SendInterval: time.Second,
|
||||
CheckInterval: time.Second,
|
||||
CloseUnresponsiveTimeout: 5 * time.Second,
|
||||
})
|
||||
req.NotNil(hb)
|
||||
req.Equal(routerlink.MinCloseUnresponsiveTimeout.String(), hb.CloseUnresponsiveTimeout)
|
||||
|
||||
// And the clamped result must survive the validation it was clamped to pass.
|
||||
cfg := &Config{Heartbeats: hb}
|
||||
req.NoError(cfg.Validate())
|
||||
|
||||
// A value at or above the floor is passed through untouched.
|
||||
hb = heartbeatsFromYamlOptions(channel.HeartbeatOptions{
|
||||
SendInterval: 10 * time.Second,
|
||||
CheckInterval: time.Second,
|
||||
CloseUnresponsiveTimeout: 45 * time.Second,
|
||||
})
|
||||
req.NotNil(hb)
|
||||
req.Equal("45s", hb.CloseUnresponsiveTimeout)
|
||||
}
|
||||
|
||||
func Test_LocalYaml_InvalidTimingFallsBackToDefaults(t *testing.T) {
|
||||
// The env loader fills absent fields with non-zero defaults, so every input
|
||||
// here is timing an operator wrote. Local link config failing validation
|
||||
// stops the router from starting, and the send and check intervals were once
|
||||
// ignored for links entirely, so timing that was never in force must not
|
||||
// start blocking boot on upgrade: it falls back to the defaults instead.
|
||||
cases := []struct {
|
||||
name string
|
||||
in channel.HeartbeatOptions
|
||||
}{
|
||||
{
|
||||
// Kept as "0s" so validation sees it, as the controller does, rather than
|
||||
// read as an absent field and quietly defaulted.
|
||||
name: "explicit zero check interval",
|
||||
in: channel.HeartbeatOptions{SendInterval: 10 * time.Second, CheckInterval: 0, CloseUnresponsiveTimeout: time.Minute},
|
||||
},
|
||||
{
|
||||
name: "every field explicitly zero",
|
||||
in: channel.HeartbeatOptions{},
|
||||
},
|
||||
{
|
||||
// Ignored before these settings reached links, and so harmless then.
|
||||
name: "check interval past the timeout",
|
||||
in: channel.HeartbeatOptions{SendInterval: 10 * time.Second, CheckInterval: 2 * time.Minute, CloseUnresponsiveTimeout: 30 * time.Second},
|
||||
},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
req := require.New(t)
|
||||
req.Nil(heartbeatsFromYamlOptions(c.in), "invalid local timing must yield the defaults")
|
||||
|
||||
// And the router must still start: the translated config validates.
|
||||
js, err := ConfigFromLocalYaml(LocalYamlConfig{
|
||||
Listeners: []map[interface{}]interface{}{{"binding": "transport", "bind": "tls:0.0.0.0:6262"}},
|
||||
Heartbeats: c.in,
|
||||
})
|
||||
req.NoError(err)
|
||||
cfg, err := ParseConfig(js)
|
||||
req.NoError(err)
|
||||
req.NoError(cfg.Validate())
|
||||
timings, err := cfg.EffectiveHeartbeats()
|
||||
req.NoError(err)
|
||||
req.Equal(routerlink.DefaultCloseUnresponsiveTimeout, timings.CloseUnresponsiveTimeout)
|
||||
req.Equal(routerlink.DefaultHeartbeatCheckInterval, timings.CheckInterval)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func Test_LocalYaml_ValidTimingPassesThrough(t *testing.T) {
|
||||
req := require.New(t)
|
||||
hb := heartbeatsFromYamlOptions(channel.HeartbeatOptions{
|
||||
SendInterval: 5 * time.Second,
|
||||
CheckInterval: 500 * time.Millisecond,
|
||||
CloseUnresponsiveTimeout: 20 * time.Second,
|
||||
})
|
||||
req.NotNil(hb)
|
||||
req.Equal(&HeartbeatsConfig{SendInterval: "5s", CheckInterval: "500ms", CloseUnresponsiveTimeout: "20s"}, hb)
|
||||
}
|
||||
|
||||
func Test_Subsystem_QueueSizeOnlyChange_AppliesWithoutDispatching(t *testing.T) {
|
||||
// Queue sizes are read when a link is created, so a change to them reaches
|
||||
// new links only. There is nothing to do for established links and nothing
|
||||
// about them has become stale, so the change must not reach the handler:
|
||||
// that would run a GC pass and, under a non-preserve gcMode, close healthy
|
||||
// links to resize a buffer.
|
||||
//
|
||||
// This is emergent from which flags hasWork consults, so it is pinned here
|
||||
// rather than left for someone adding a flag to rediscover.
|
||||
req := require.New(t)
|
||||
r, _ := newTestRegistry(t)
|
||||
|
||||
const listener = `{"bind":"tls:0.0.0.0:6262"}`
|
||||
req.NoError(r.Apply(1, fmt.Sprintf(`{"listeners":[%s],"gcMode":"changed","payloadSenderQueueSize":128}`, listener)))
|
||||
|
||||
rec := newChangeRecorder()
|
||||
r.SetConfigurationChangeHandler(rec.handle)
|
||||
|
||||
req.NoError(r.Apply(1, fmt.Sprintf(`{"listeners":[%s],"gcMode":"changed","payloadSenderQueueSize":512,"ackSenderQueueSize":256}`, listener)))
|
||||
|
||||
req.False(rec.waitForChange(250*time.Millisecond),
|
||||
"a queue-size-only change must not dispatch, or a tuning edit would close healthy links")
|
||||
|
||||
// Applied all the same: the next link built picks the new sizes up.
|
||||
req.Equal(512, r.PayloadSenderQueueSize())
|
||||
req.Equal(256, r.AckSenderQueueSize())
|
||||
req.Equal(512, r.GetConfig().PayloadSenderQueueSize)
|
||||
}
|
||||
|
||||
func Test_ConfigFromLocalYaml_LinkSectionWithoutSurfaceIsLocalConfig(t *testing.T) {
|
||||
// A link section of any content takes precedence over the controller's
|
||||
// router.link config, including one that only tunes heartbeats, so it is applied
|
||||
// as local config rather than dropped.
|
||||
req := require.New(t)
|
||||
|
||||
js, err := ConfigFromLocalYaml(LocalYamlConfig{
|
||||
Configured: true,
|
||||
Heartbeats: channel.HeartbeatOptions{
|
||||
SendInterval: 5 * time.Second,
|
||||
CheckInterval: time.Second,
|
||||
CloseUnresponsiveTimeout: 45 * time.Second,
|
||||
},
|
||||
PayloadSenderQueueSize: 256,
|
||||
AckSenderQueueSize: 64,
|
||||
})
|
||||
req.NoError(err)
|
||||
req.NotEmpty(js, "a link section must count as local config even with no listeners or dialers")
|
||||
|
||||
cfg, err := ParseConfig(js)
|
||||
req.NoError(err)
|
||||
req.NoError(cfg.Validate())
|
||||
req.Empty(cfg.Listeners)
|
||||
req.Empty(cfg.Dialers)
|
||||
req.Equal("45s", cfg.Heartbeats.CloseUnresponsiveTimeout)
|
||||
req.Equal(256, cfg.PayloadSenderQueueSize)
|
||||
}
|
||||
|
||||
+39
-11
@@ -156,6 +156,12 @@ func (self *Router) GetXlinkDialers() []xlink.Dialer {
|
||||
return self.linkSubsystem.Dialers()
|
||||
}
|
||||
|
||||
// GetHeartbeatSettings returns the heartbeat generation currently in force,
|
||||
// for reconciling a link that registered while a change was being pushed.
|
||||
func (self *Router) GetHeartbeatSettings() xlink.HeartbeatSettings {
|
||||
return self.linkSubsystem.Heartbeats()
|
||||
}
|
||||
|
||||
func (self *Router) GetXlinkListeners() []xlink.Listener {
|
||||
return self.linkSubsystem.Listeners()
|
||||
}
|
||||
@@ -181,11 +187,11 @@ func (self *Router) GetMetricsRegistry() servermetrics.UsageRegistry {
|
||||
}
|
||||
|
||||
func (self *Router) GetLinkPayloadSenderQueueSize() int {
|
||||
return self.config.Link.PayloadSenderQueueSize
|
||||
return self.linkSubsystem.PayloadSenderQueueSize()
|
||||
}
|
||||
|
||||
func (self *Router) GetLinkAckSenderQueueSize() int {
|
||||
return self.config.Link.AckSenderQueueSize
|
||||
return self.linkSubsystem.AckSenderQueueSize()
|
||||
}
|
||||
|
||||
func (self *Router) GetXgressRegistry() *env.Registry {
|
||||
@@ -244,6 +250,20 @@ func (self *Router) GetForwarder() env.Forwarder {
|
||||
return self.forwarder
|
||||
}
|
||||
|
||||
// GetForwarderImpl returns the concrete forwarder. It exists alongside
|
||||
// GetForwarder (which returns the env.Forwarder interface to avoid an
|
||||
// env->forwarder import cycle) so callers needing the concrete type — like the
|
||||
// link bind handler — can get it without a separate adapter.
|
||||
func (self *Router) GetForwarderImpl() *forwarder.Forwarder {
|
||||
return self.forwarder
|
||||
}
|
||||
|
||||
// GetLinkSettings returns the live, managed-config-driven link settings (backed
|
||||
// by the link subsystem) for the link bind handler.
|
||||
func (self *Router) GetLinkSettings() handler_link.LinkSettings {
|
||||
return self.linkSubsystem
|
||||
}
|
||||
|
||||
func (self *Router) GetXgressMetrics() env.XgressMetrics {
|
||||
return self.xgMetrics
|
||||
}
|
||||
@@ -752,13 +772,7 @@ func (self *Router) NotifyCertsUpdated() {
|
||||
|
||||
func (self *Router) registerComponents() error {
|
||||
acceptor := newXlinkAccepter(self.forwarder)
|
||||
xlinkChAccepter := handler_link.NewBindHandlerFactory(
|
||||
self.ctrls,
|
||||
self.forwarder,
|
||||
&self.config.Link.Heartbeats,
|
||||
self.metricsRegistry,
|
||||
self.xlinkRegistry,
|
||||
)
|
||||
xlinkChAccepter := handler_link.NewBindHandlerFactory(self)
|
||||
|
||||
linkTransportConfig := map[interface{}]interface{}{}
|
||||
for k, v := range self.config.Transport {
|
||||
@@ -852,8 +866,12 @@ func (self *Router) registerPlugins() error {
|
||||
// header packing and the post-Apply publish path so both wire shapes
|
||||
// stay aligned.
|
||||
func (self *Router) currentLinkListeners() *ctrl_pb.Listeners {
|
||||
listeners := &ctrl_pb.Listeners{}
|
||||
for _, listener := range self.linkSubsystem.Listeners() {
|
||||
// Generation and listeners come from one read: an update carries the
|
||||
// generation of the set it actually contains, so a controller comparing
|
||||
// generations can drop a superseded one.
|
||||
generation, current := self.linkSubsystem.ListenerSnapshot()
|
||||
listeners := &ctrl_pb.Listeners{Generation: generation}
|
||||
for _, listener := range current {
|
||||
listeners.Listeners = append(listeners.Listeners, &ctrl_pb.Listener{
|
||||
Address: listener.GetAdvertisement(),
|
||||
Protocol: listener.GetLinkProtocol(),
|
||||
@@ -883,6 +901,15 @@ func (self *Router) onLinkSubsystemChanged(change link.ConfigurationChange) {
|
||||
if change.DialersChanged {
|
||||
self.xlinkRegistry.RescanForDialOpportunities()
|
||||
}
|
||||
if change.HeartbeatsChanged {
|
||||
// Push the whole generation, so intervals and the timeout they pair with
|
||||
// arrive together. Links ignore generations they have already passed, and new
|
||||
// links take the current one at bind time.
|
||||
settings := self.linkSubsystem.Heartbeats()
|
||||
for xl := range self.xlinkRegistry.Iter() {
|
||||
xl.UpdateHeartbeat(settings)
|
||||
}
|
||||
}
|
||||
link.RunStaleLinkGc(self, change.GcMode)
|
||||
}
|
||||
|
||||
@@ -917,6 +944,7 @@ func (self *Router) publishLinkListeners() {
|
||||
// bind address) surfaces as a startup error rather than just a log line.
|
||||
func (self *Router) applyLocalLinkConfig() error {
|
||||
js, err := link.ConfigFromLocalYaml(link.LocalYamlConfig{
|
||||
Configured: self.config.Link.Configured,
|
||||
Listeners: self.config.Link.Listeners,
|
||||
Dialers: self.config.Link.Dialers,
|
||||
Heartbeats: self.config.Link.Heartbeats,
|
||||
|
||||
@@ -18,6 +18,7 @@ import (
|
||||
metrics2 "github.com/openziti/ziti/v2/router/metrics"
|
||||
"github.com/openziti/ziti/v2/router/state"
|
||||
"github.com/openziti/ziti/v2/router/xgress_router"
|
||||
"github.com/openziti/ziti/v2/router/xlink"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
@@ -79,6 +80,13 @@ func (link *mirrorLink) DestinationId() string {
|
||||
return "test"
|
||||
}
|
||||
|
||||
func (link *mirrorLink) SetHeartbeatControl(channel.HeartbeatControl, xlink.HeartbeatSettings) {}
|
||||
|
||||
func (link *mirrorLink) UpdateHeartbeat(xlink.HeartbeatSettings) {}
|
||||
func (link *mirrorLink) HeartbeatSettings() xlink.HeartbeatSettings {
|
||||
return xlink.HeartbeatSettings{}
|
||||
}
|
||||
|
||||
func (link *mirrorLink) Id() string {
|
||||
return "router1"
|
||||
}
|
||||
|
||||
@@ -82,6 +82,17 @@ type Registry interface {
|
||||
RescanForDialOpportunities()
|
||||
}
|
||||
|
||||
// HeartbeatSettings is the heartbeat timing a link runs with, as one immutable
|
||||
// snapshot. Generation increases with each applied link config. The values
|
||||
// travel together because they are only valid as a set: the timeout must exceed
|
||||
// the gap a responsive link shows, roughly sendInterval + checkInterval.
|
||||
type HeartbeatSettings struct {
|
||||
Generation uint64
|
||||
SendInterval time.Duration
|
||||
CheckInterval time.Duration
|
||||
CloseUnresponsiveTimeout time.Duration
|
||||
}
|
||||
|
||||
type Forwarder interface {
|
||||
ForwardPayload(srcAddr xgress.Address, payload *xgress.Payload, timeout time.Duration) error
|
||||
ForwardAcknowledgement(srcAddr xgress.Address, acknowledgement *xgress.Acknowledgement) error
|
||||
@@ -164,6 +175,19 @@ type Xlink interface {
|
||||
LinkKey() LinkKey
|
||||
CloseOnce(f func())
|
||||
IsClosed() bool
|
||||
// SetHeartbeatControl registers one channel's heartbeat control handle and the
|
||||
// settings that channel was configured from. Call once per channel backing the
|
||||
// link (a split link has two); handles accumulate.
|
||||
SetHeartbeatControl(hc channel.HeartbeatControl, settings HeartbeatSettings)
|
||||
// UpdateHeartbeat applies settings to every channel backing the link,
|
||||
// retuning intervals and publishing the timeout its callbacks read, so a
|
||||
// managed-config change reaches an established link without rebuilding it.
|
||||
// Settings at or below the generation already applied are ignored.
|
||||
UpdateHeartbeat(settings HeartbeatSettings)
|
||||
// HeartbeatSettings returns the settings currently in force for this link.
|
||||
// Read by each channel's heartbeat callback, so the timeout it enforces is
|
||||
// always the one belonging to the intervals its ticker is running.
|
||||
HeartbeatSettings() HeartbeatSettings
|
||||
InspectLink() *inspect.LinkInspectDetail
|
||||
GetLinkConnState() *ctrl_pb.LinkConnState
|
||||
IsDialed() bool
|
||||
|
||||
@@ -0,0 +1,287 @@
|
||||
/*
|
||||
Copyright NetFoundry Inc.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
https://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package xlink_transport
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/openziti/ziti/v2/router/xlink"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// settings builds a generation with intervals whose sum stays under the
|
||||
// timeout, the relationship the real config validation enforces.
|
||||
func settings(gen uint64, send, check, closeTimeout time.Duration) xlink.HeartbeatSettings {
|
||||
return xlink.HeartbeatSettings{
|
||||
Generation: gen,
|
||||
SendInterval: send,
|
||||
CheckInterval: check,
|
||||
CloseUnresponsiveTimeout: closeTimeout,
|
||||
}
|
||||
}
|
||||
|
||||
type recordingHeartbeatControl struct {
|
||||
send time.Duration
|
||||
check time.Duration
|
||||
calls int
|
||||
}
|
||||
|
||||
func (self *recordingHeartbeatControl) UpdateIntervals(send, check time.Duration) {
|
||||
self.send = send
|
||||
self.check = check
|
||||
self.calls++
|
||||
}
|
||||
|
||||
func Test_heartbeatControl_UpdatesCurrent(t *testing.T) {
|
||||
req := require.New(t)
|
||||
|
||||
var hc heartbeatControl
|
||||
a := &recordingHeartbeatControl{}
|
||||
hc.SetHeartbeatControl(a, settings(1, 10*time.Second, time.Second, time.Minute))
|
||||
|
||||
next := settings(2, 2*time.Second, 200*time.Millisecond, 30*time.Second)
|
||||
hc.UpdateHeartbeat(next)
|
||||
|
||||
req.Equal(1, a.calls)
|
||||
req.Equal(2*time.Second, a.send)
|
||||
req.Equal(200*time.Millisecond, a.check)
|
||||
req.Equal(next, hc.HeartbeatSettings(), "the link adopts the pushed generation")
|
||||
}
|
||||
|
||||
// A split link binds a control per channel, payload and ack. Both must be
|
||||
// re-tuned: every channel's callback reads the close-unresponsive timeout live,
|
||||
// so one left at an older, longer interval can close a healthy link when the
|
||||
// timeout shortens.
|
||||
func Test_heartbeatControl_RetunesEveryRegisteredControl(t *testing.T) {
|
||||
req := require.New(t)
|
||||
|
||||
var hc heartbeatControl
|
||||
payload := &recordingHeartbeatControl{}
|
||||
ack := &recordingHeartbeatControl{}
|
||||
base := settings(1, 10*time.Second, time.Second, time.Minute)
|
||||
hc.SetHeartbeatControl(payload, base)
|
||||
hc.SetHeartbeatControl(ack, base)
|
||||
|
||||
hc.UpdateHeartbeat(settings(2, 3*time.Second, 300*time.Millisecond, 30*time.Second))
|
||||
|
||||
for name, ctrl := range map[string]*recordingHeartbeatControl{"payload": payload, "ack": ack} {
|
||||
req.Equal(1, ctrl.calls, "%s channel must be re-tuned", name)
|
||||
req.Equal(3*time.Second, ctrl.send, "%s channel", name)
|
||||
req.Equal(300*time.Millisecond, ctrl.check, "%s channel", name)
|
||||
}
|
||||
|
||||
// A second retune reaches both again, rather than only the newest.
|
||||
hc.UpdateHeartbeat(settings(3, time.Second, time.Second, 10*time.Second))
|
||||
req.Equal(2, payload.calls)
|
||||
req.Equal(2, ack.calls)
|
||||
}
|
||||
|
||||
func Test_heartbeatControl_IgnoresNilControl(t *testing.T) {
|
||||
req := require.New(t)
|
||||
|
||||
var hc heartbeatControl
|
||||
hc.SetHeartbeatControl(nil, settings(1, 10*time.Second, time.Second, time.Minute))
|
||||
a := &recordingHeartbeatControl{}
|
||||
hc.SetHeartbeatControl(a, settings(1, 10*time.Second, time.Second, time.Minute))
|
||||
|
||||
// A nil handle must not be stored, or the retune loop would panic on it.
|
||||
hc.UpdateHeartbeat(settings(2, time.Second, time.Second, 10*time.Second))
|
||||
req.Equal(1, a.calls)
|
||||
}
|
||||
|
||||
func Test_heartbeatControl_NoControlIsNoop(t *testing.T) {
|
||||
var hc heartbeatControl
|
||||
// must not panic with nothing registered
|
||||
hc.UpdateHeartbeat(settings(1, time.Second, time.Second, 10*time.Second))
|
||||
}
|
||||
|
||||
func Test_heartbeatControl_IgnoresStaleGeneration(t *testing.T) {
|
||||
// Change notifications run on independent goroutines, so an older one can
|
||||
// land after a newer. It must not roll the link back onto older settings.
|
||||
req := require.New(t)
|
||||
|
||||
var hc heartbeatControl
|
||||
a := &recordingHeartbeatControl{}
|
||||
hc.SetHeartbeatControl(a, settings(1, 10*time.Second, time.Second, time.Minute))
|
||||
|
||||
newer := settings(5, time.Second, 100*time.Millisecond, 10*time.Second)
|
||||
hc.UpdateHeartbeat(newer)
|
||||
req.Equal(newer, hc.HeartbeatSettings())
|
||||
|
||||
older := settings(4, 30*time.Second, 20*time.Second, 5*time.Minute)
|
||||
hc.UpdateHeartbeat(older)
|
||||
req.Equal(newer, hc.HeartbeatSettings(), "an overtaken update must not be applied")
|
||||
req.Equal(1, a.calls, "and must not retune the ticker either")
|
||||
|
||||
// The same generation is also a no-op, which is what makes the
|
||||
// post-registration reconcile free in the common case.
|
||||
hc.UpdateHeartbeat(newer)
|
||||
req.Equal(1, a.calls)
|
||||
}
|
||||
|
||||
// gatedControl records the generation implied by the intervals it was re-tuned
|
||||
// to, and holds the first caller inside UpdateIntervals until released. That is
|
||||
// how a second update gets the chance to overtake one already dispatching.
|
||||
type gatedControl struct {
|
||||
lock sync.Mutex
|
||||
recorded []uint64
|
||||
entered chan struct{}
|
||||
gate chan struct{}
|
||||
seen bool
|
||||
}
|
||||
|
||||
func newGatedControl() *gatedControl {
|
||||
return &gatedControl{
|
||||
entered: make(chan struct{}),
|
||||
gate: make(chan struct{}),
|
||||
}
|
||||
}
|
||||
|
||||
func (self *gatedControl) UpdateIntervals(send, _ time.Duration) {
|
||||
self.lock.Lock()
|
||||
first := !self.seen
|
||||
self.seen = true
|
||||
self.lock.Unlock()
|
||||
|
||||
if first {
|
||||
close(self.entered)
|
||||
<-self.gate
|
||||
}
|
||||
|
||||
self.lock.Lock()
|
||||
defer self.lock.Unlock()
|
||||
self.recorded = append(self.recorded, uint64(send/time.Millisecond))
|
||||
}
|
||||
|
||||
func (self *gatedControl) lastRecorded() uint64 {
|
||||
self.lock.Lock()
|
||||
defer self.lock.Unlock()
|
||||
if len(self.recorded) == 0 {
|
||||
return 0
|
||||
}
|
||||
return self.recorded[len(self.recorded)-1]
|
||||
}
|
||||
|
||||
// generationSettings encodes the generation into the send interval so a control
|
||||
// can report which generation its ticker was last re-tuned to.
|
||||
func generationSettings(gen uint64) xlink.HeartbeatSettings {
|
||||
return xlink.HeartbeatSettings{
|
||||
Generation: gen,
|
||||
SendInterval: time.Duration(gen) * time.Millisecond,
|
||||
CheckInterval: time.Millisecond,
|
||||
CloseUnresponsiveTimeout: 5 * time.Minute,
|
||||
}
|
||||
}
|
||||
|
||||
func Test_heartbeatControl_OvertakenUpdateCannotDragControlsBack(t *testing.T) {
|
||||
// Config changes are notified on independent goroutines, so two updates can
|
||||
// be in flight at once. Whichever generation the callbacks end up reporting
|
||||
// must be the one the tickers were last re-tuned to: an update that publishes
|
||||
// its generation and dispatches separately can be overtaken in between and
|
||||
// then retune the controls back to its own intervals.
|
||||
req := require.New(t)
|
||||
|
||||
var hc heartbeatControl
|
||||
ctrl := newGatedControl()
|
||||
hc.SetHeartbeatControl(ctrl, generationSettings(1))
|
||||
|
||||
secondDone := make(chan struct{})
|
||||
go func() {
|
||||
defer close(secondDone)
|
||||
hc.UpdateHeartbeat(generationSettings(2))
|
||||
}()
|
||||
<-ctrl.entered // generation 2 is now inside the dispatch
|
||||
|
||||
thirdDone := make(chan struct{})
|
||||
go func() {
|
||||
defer close(thirdDone)
|
||||
hc.UpdateHeartbeat(generationSettings(3))
|
||||
}()
|
||||
|
||||
// Give generation 3 the chance to overtake. With the dispatch serialized it
|
||||
// cannot, which is the property under test; without that it runs to
|
||||
// completion here and generation 2 then overwrites its intervals.
|
||||
select {
|
||||
case <-thirdDone:
|
||||
case <-time.After(50 * time.Millisecond):
|
||||
}
|
||||
|
||||
close(ctrl.gate)
|
||||
<-secondDone
|
||||
<-thirdDone
|
||||
|
||||
reported := hc.HeartbeatSettings().Generation
|
||||
req.Equal(reported, ctrl.lastRecorded(),
|
||||
"callbacks report generation %d while the ticker was last re-tuned to %d",
|
||||
reported, ctrl.lastRecorded())
|
||||
req.Equal(uint64(3), reported, "the newest generation must win")
|
||||
}
|
||||
|
||||
func Test_heartbeatControl_FirstRegistrationAdoptsGenerationZero(t *testing.T) {
|
||||
// A router with no managed link config hands out the real defaults carrying
|
||||
// generation 0. Treating that as "nothing registered" leaves the link on
|
||||
// zero-valued settings, and a zero timeout condemns the first late
|
||||
// heartbeat instead of waiting out the configured minute.
|
||||
req := require.New(t)
|
||||
|
||||
var hc heartbeatControl
|
||||
defaults := settings(0, 10*time.Second, time.Second, time.Minute)
|
||||
hc.SetHeartbeatControl(&recordingHeartbeatControl{}, defaults)
|
||||
|
||||
req.Equal(defaults, hc.HeartbeatSettings())
|
||||
}
|
||||
|
||||
func Test_heartbeatControl_LaterChannelReconcilesGenerations(t *testing.T) {
|
||||
// A split link binds its payload and ack channels one at a time, so config
|
||||
// can move in between and the two arrive carrying different generations.
|
||||
// Whichever order that happens in, the link must end up holding the newer
|
||||
// generation with every channel tuned to it.
|
||||
older := settings(1, 30*time.Second, 5*time.Second, 2*time.Minute)
|
||||
newer := settings(2, 2*time.Second, 500*time.Millisecond, 30*time.Second)
|
||||
|
||||
t.Run("second channel is newer", func(t *testing.T) {
|
||||
req := require.New(t)
|
||||
var hc heartbeatControl
|
||||
first, second := &recordingHeartbeatControl{}, &recordingHeartbeatControl{}
|
||||
hc.SetHeartbeatControl(first, older)
|
||||
hc.SetHeartbeatControl(second, newer)
|
||||
|
||||
req.Equal(newer, hc.HeartbeatSettings())
|
||||
req.Equal(newer.SendInterval, first.send, "the already-bound channel must be brought forward")
|
||||
req.Equal(newer.CheckInterval, first.check)
|
||||
|
||||
// An update at the adopted generation is a no-op, so registration is the only
|
||||
// chance to bring the first channel forward.
|
||||
hc.UpdateHeartbeat(newer)
|
||||
req.Equal(newer.SendInterval, first.send)
|
||||
})
|
||||
|
||||
t.Run("second channel is older", func(t *testing.T) {
|
||||
req := require.New(t)
|
||||
var hc heartbeatControl
|
||||
first, second := &recordingHeartbeatControl{}, &recordingHeartbeatControl{}
|
||||
hc.SetHeartbeatControl(first, newer)
|
||||
hc.SetHeartbeatControl(second, older)
|
||||
|
||||
req.Equal(newer, hc.HeartbeatSettings(), "an older registration must not roll the link back")
|
||||
req.Equal(newer.SendInterval, second.send, "the late channel must be brought up to current")
|
||||
req.Equal(newer.CheckInterval, second.check)
|
||||
req.Equal(0, first.calls, "and the channel already on that generation is left alone")
|
||||
})
|
||||
}
|
||||
@@ -17,6 +17,7 @@
|
||||
package xlink_transport
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
@@ -28,6 +29,80 @@ import (
|
||||
"github.com/openziti/ziti/v2/router/xlink"
|
||||
)
|
||||
|
||||
// heartbeatControl holds a link's channel heartbeat handles and the settings
|
||||
// generation they run. Embedded into the link impls to satisfy xlink.Xlink's
|
||||
// SetHeartbeatControl / UpdateHeartbeat. A link has one control per channel: one
|
||||
// for a single or multi-underlay link, two for a split link.
|
||||
type heartbeatControl struct {
|
||||
lock sync.Mutex
|
||||
controls []channel.HeartbeatControl
|
||||
settings xlink.HeartbeatSettings
|
||||
// haveSettings distinguishes "nothing registered yet" from settings at
|
||||
// generation 0, which is what a router with no managed link config publishes.
|
||||
haveSettings bool
|
||||
}
|
||||
|
||||
// SetHeartbeatControl registers one channel's heartbeat handle and the settings
|
||||
// it was configured from. Calls accumulate, one per channel. Registration
|
||||
// reconciles: the link keeps the newest generation any channel was configured
|
||||
// from, and every channel is re-tuned to it.
|
||||
func (self *heartbeatControl) SetHeartbeatControl(hc channel.HeartbeatControl, settings xlink.HeartbeatSettings) {
|
||||
if hc == nil {
|
||||
return
|
||||
}
|
||||
self.lock.Lock()
|
||||
defer self.lock.Unlock()
|
||||
self.controls = append(self.controls, hc)
|
||||
|
||||
if !self.haveSettings {
|
||||
// Nothing to reconcile against, and hc is already tuned to these.
|
||||
self.settings = settings
|
||||
self.haveSettings = true
|
||||
return
|
||||
}
|
||||
if settings.Generation > self.settings.Generation {
|
||||
self.applyLocked(settings)
|
||||
return
|
||||
}
|
||||
if settings.Generation < self.settings.Generation {
|
||||
hc.UpdateIntervals(self.settings.SendInterval, self.settings.CheckInterval)
|
||||
}
|
||||
}
|
||||
|
||||
// HeartbeatSettings implements xlink.Xlink.
|
||||
func (self *heartbeatControl) HeartbeatSettings() xlink.HeartbeatSettings {
|
||||
self.lock.Lock()
|
||||
defer self.lock.Unlock()
|
||||
return self.settings
|
||||
}
|
||||
|
||||
// UpdateHeartbeat implements xlink.Xlink. A generation at or below the one
|
||||
// applied is ignored, so a late or duplicate update is harmless. Runs under the
|
||||
// lock, so a generation and its intervals are applied as one step; UpdateIntervals
|
||||
// never blocks, so holding the lock is cheap.
|
||||
func (self *heartbeatControl) UpdateHeartbeat(next xlink.HeartbeatSettings) {
|
||||
self.lock.Lock()
|
||||
defer self.lock.Unlock()
|
||||
|
||||
if next.Generation <= self.settings.Generation {
|
||||
return
|
||||
}
|
||||
self.applyLocked(next)
|
||||
}
|
||||
|
||||
// applyLocked publishes next and re-tunes every registered control. Caller holds
|
||||
// the lock. The order of the two steps doesn't matter: a retune lands on the
|
||||
// channel's next pulse regardless, and the callback re-arms its deadline when the
|
||||
// generation changes.
|
||||
func (self *heartbeatControl) applyLocked(next xlink.HeartbeatSettings) {
|
||||
self.settings = next
|
||||
self.haveSettings = true
|
||||
|
||||
for _, hc := range self.controls {
|
||||
hc.UpdateIntervals(next.SendInterval, next.CheckInterval)
|
||||
}
|
||||
}
|
||||
|
||||
type impl struct {
|
||||
id string
|
||||
ch LinkChannel
|
||||
@@ -45,6 +120,7 @@ type impl struct {
|
||||
dialed bool
|
||||
iteration uint32
|
||||
dupsRejected uint32
|
||||
heartbeatControl
|
||||
|
||||
droppedMsgMeter metrics.Meter
|
||||
droppedXgMsgMeter metrics.Meter
|
||||
|
||||
@@ -50,6 +50,7 @@ type splitImpl struct {
|
||||
iteration uint32
|
||||
dupsRejected uint32
|
||||
lock sync.Mutex
|
||||
heartbeatControl
|
||||
|
||||
droppedMsgMeter metrics.Meter
|
||||
droppedXgMsgMeter metrics.Meter
|
||||
|
||||
@@ -87,6 +87,16 @@ func (self *testRegistryEnv) GetMetricsRegistry() servermetrics.UsageRegistry {
|
||||
return self.metricsRegistry
|
||||
}
|
||||
|
||||
// GetHeartbeatSettings mirrors what a router reports before any link config has
|
||||
// been applied: the defaults, at generation zero.
|
||||
func (self *testRegistryEnv) GetHeartbeatSettings() xlink.HeartbeatSettings {
|
||||
return xlink.HeartbeatSettings{
|
||||
SendInterval: 10 * time.Second,
|
||||
CheckInterval: time.Second,
|
||||
CloseUnresponsiveTimeout: time.Minute,
|
||||
}
|
||||
}
|
||||
|
||||
type testDial struct {
|
||||
Key string
|
||||
LinkId string
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
//go:build apitests
|
||||
|
||||
/*
|
||||
Copyright NetFoundry Inc.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
https://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package tests
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/openziti/foundation/v2/errorz"
|
||||
"github.com/openziti/ziti/v2/common/config/routerlink"
|
||||
)
|
||||
|
||||
// Semantic validation of router.link.v1 has to happen where the operator wrote
|
||||
// the config. A JSON schema can't express a relationship between two fields, so
|
||||
// without this the controller accepts a config that every router then rejects,
|
||||
// and the only sign is a line in each router's log.
|
||||
func Test_RouterLinkConfigValidation(t *testing.T) {
|
||||
ctx := NewTestContext(t)
|
||||
defer ctx.Teardown()
|
||||
ctx.StartServer()
|
||||
ctx.RequireAdminManagementApiLogin()
|
||||
|
||||
heartbeats := func(fields map[string]interface{}) map[string]interface{} {
|
||||
return map[string]interface{}{"heartbeats": fields}
|
||||
}
|
||||
|
||||
t.Run("accepts sane heartbeat timing", func(t *testing.T) {
|
||||
ctx.testContextChanged(t)
|
||||
config := ctx.newConfig(routerlink.ConfigTypeV1, heartbeats(map[string]interface{}{
|
||||
"sendInterval": "10s",
|
||||
"checkInterval": "1s",
|
||||
"closeUnresponsiveTimeout": "60s",
|
||||
}))
|
||||
config.Id = ctx.AdminManagementSession.requireCreateEntity(config)
|
||||
ctx.Req.NotEmpty(config.Id)
|
||||
})
|
||||
|
||||
t.Run("rejects a check interval past the timeout", func(t *testing.T) {
|
||||
ctx.testContextChanged(t)
|
||||
config := ctx.newConfig(routerlink.ConfigTypeV1, heartbeats(map[string]interface{}{
|
||||
"checkInterval": "2m",
|
||||
"closeUnresponsiveTimeout": "30s",
|
||||
}))
|
||||
resp := ctx.AdminManagementSession.createEntity(config)
|
||||
ctx.requireFieldError(resp.StatusCode(), resp.Body(), errorz.CouldNotValidateCode, "data")
|
||||
})
|
||||
|
||||
t.Run("rejects a non-positive interval", func(t *testing.T) {
|
||||
ctx.testContextChanged(t)
|
||||
config := ctx.newConfig(routerlink.ConfigTypeV1, heartbeats(map[string]interface{}{
|
||||
"checkInterval": "0s",
|
||||
}))
|
||||
resp := ctx.AdminManagementSession.createEntity(config)
|
||||
ctx.requireFieldError(resp.StatusCode(), resp.Body(), errorz.CouldNotValidateCode, "data")
|
||||
})
|
||||
|
||||
t.Run("rejects a timeout under the default send interval", func(t *testing.T) {
|
||||
// The config names no intervals, so validation has to resolve the
|
||||
// defaults to see that this combination is broken.
|
||||
ctx.testContextChanged(t)
|
||||
config := ctx.newConfig(routerlink.ConfigTypeV1, heartbeats(map[string]interface{}{
|
||||
"closeUnresponsiveTimeout": "5s",
|
||||
}))
|
||||
resp := ctx.AdminManagementSession.createEntity(config)
|
||||
ctx.requireFieldError(resp.StatusCode(), resp.Body(), errorz.CouldNotValidateCode, "data")
|
||||
})
|
||||
|
||||
t.Run("rejects an unsafe update to a valid config", func(t *testing.T) {
|
||||
ctx.testContextChanged(t)
|
||||
config := ctx.newConfig(routerlink.ConfigTypeV1, heartbeats(map[string]interface{}{
|
||||
"closeUnresponsiveTimeout": "60s",
|
||||
}))
|
||||
config.Id = ctx.AdminManagementSession.requireCreateEntity(config)
|
||||
|
||||
// Validation runs on update as well as create, so a config can't be
|
||||
// walked into an unsafe state after the fact.
|
||||
config.Data = heartbeats(map[string]interface{}{"closeUnresponsiveTimeout": "1s"})
|
||||
resp := ctx.AdminManagementSession.updateEntity(config)
|
||||
ctx.requireFieldError(resp.StatusCode(), resp.Body(), errorz.CouldNotValidateCode, "data")
|
||||
})
|
||||
}
|
||||
Reference in New Issue
Block a user