The reset added in b45bd66b9 only fired when the temperature SSH key
file on disk changed (mtime/size). An operator who repairs SSH access
any other way — fixing authorized_keys on the host, repairing
known_hosts, restoring network reachability — still waited out a
backoff window that may have compounded toward fifteen minutes.
A system-settings save is the natural operator touchpoint after such a
repair, so the settings handler now fans ResetSSHFailureBackoff out to
every live tenant monitor after a successful save, clearing the
per-host temperature SSH backoff and the knownhosts keyscan backoff.
The reset touches in-memory retry timing only; nothing is persisted
and no request field controls it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The discovery-policy check resolved endpoint hostnames with a bare
net.LookupIP while the actual dials went through pkg/tlsutil's process-global
cached resolver, so the policy and the connection reasoned about two different
DNS views. That split is why 108aa4e20 had to skip resolution entirely for the
default policy, which left the injected 169.254.0.0/16 blocklist enforced only
against literal IPs: a hostname endpoint pointed at the metadata range walked
straight through.
Resolve through tlsutil.LookupHostCached instead. The shared resolver caches
answers and lookup failures alike until its next refresh, so repeat poll cycles
cost a cache hit rather than a query and the per-poll DNS volume that opened
#1638 stays gone. With that in place the default-policy skip is removed and the
blocklist applies to resolved addresses again, and the five-minute decision
cache is dropped rather than kept: it bought nothing on top of the resolver
cache, made the verdict trail the configuration, and memoized the fail-open
"resolution failed, allow" outcome for minutes even with an explicit allowlist
configured. Its claim to match a DNS refresh interval that operators configure
through DNS_CACHE_TIMEOUT goes with it.
The SSH backoffs now only escalate for work that ran. A knownhosts manager
suppressing a call inside its own window reports ErrKeyscanSuppressed, and the
temperature layer neither records a failure nor pays for the RPi fallback in
that case. An expired collection deadline is our own budget rather than
evidence about the host, so it holds the window at the floor. Both backoffs
decay once a retry deadline is more than one window past, and replacing the
temperature SSH key on disk clears both maps so a repaired key is tried on the
next cycle instead of after fifteen minutes.
Refs discussion #1638.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The cluster-endpoint discovery-policy check ran a raw net.LookupIP per node
per poll cycle since c5f5af7ab, bypassing the process-global cached resolver,
and the injected default subnet blocklist (169.254.0.0/16) made the
zero-policy fast path unreachable so even unconfigured installs generated
that DNS volume. Evaluate the default link-local-only policy against literal
endpoint IPs without resolution, and memoize custom-policy verdicts per
endpoint for the shared 5-minute DNS-cache TTL so repeat polls stay off the
resolver. Also cache ssh-keyscan failures with doubling backoff in the
knownhosts manager and back off temperature SSH collection per host after
failures instead of re-executing ssh twice per node every 10s cycle.
Refs discussion #1638.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
courtmanr@gmail.com