mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-09 10:21:03 +00:00
fix(mesh): re-evaluate data plane every 10s and add opt-in auto-recreate (#1184)
* fix(mesh): re-evaluate data plane every 10s and add opt-in auto-recreate MeshService.dataPlaneStatus was written exactly once at boot in setupMeshNetwork() and never re-evaluated. After the operator removed sencho_mesh at runtime (or it was recreated externally, or Sencho was disconnected from it), /api/health and the dashboard banner kept returning the stale boot-time discriminator until the next process restart. Adds a 10s revalidator that inspects the current Docker truth in one network-inspect call and transitions dataPlaneStatus to reflect it. Short-circuits in not_started / not_in_docker / subnet_invalid (states that cannot change within this process) and in concurrent ticks. Transitions are idempotent on stable state, so the timer can tick indefinitely on a healthy mesh without log noise. New 'not_found' reason value for the network-was-removed-at-runtime case. Existing reasons (subnet_mismatch, subnet_overlap, attach_failed) also surface from the revalidator when their underlying conditions arise post-boot. transitionDataPlane keeps message and subnet fields fresh across consecutive observations even when reason is unchanged, so /api/health never reports stale numbers (e.g. two consecutive subnet_mismatch observations against different external subnets). Adds an opt-in mesh_auto_recreate global setting (default off). When on, the revalidator additionally calls attemptInPlaceRecreate() after surfacing not_found. The helper hard-prefers the boot-chosen subnet (this.meshSubnet) and never iterates candidates, because changing the subnet here would invalidate every existing extra_hosts override on disk. A real conflict on the original subnet is reported as subnet_overlap and preserved during the 60s recreate throttle window so the operator-actionable reason is not flapped back to not_found between attempts. Self-attachment is checked via Name match (operator --hostname X matches container Name /X) or full container-ID prefix for hex HOSTNAMEs >= 12 chars (Docker default short ID). Non-hex HOSTNAMEs cannot collide with container IDs at all so a Name miss is conclusive; short hex HOSTNAMEs preserve the prior status as 'unknown' rather than risking a false-positive prefix match. Frontend surfaces: - types/mesh.ts: 'not_found' added to MeshDataPlaneReason. - MeshDataPlaneBanner: 'not_found' headline copy. - Settings > System > Mesh data plane: TogglePill bound to mesh_auto_recreate, default off, helper text explains the tradeoff. Backend coverage in backend/src/__tests__/mesh-data-plane-revalidate.test.ts (25 cases): short-circuits, idempotent stable-state, recovery from subnet_mismatch / subnet_overlap, transition to not_found / subnet_mismatch / attach_failed, transient-Docker anti-flap, re-entrancy guard, name match path, ID-prefix path, short hex hostname ambiguity, non-hex hostname certainty, transition message refresh on observation drift, auto-recreate off (default), auto-recreate success with senchoIp preservation, auto-recreate overlap classification with no subnet drift, throttle window preserves classified reason, throttle release. Lifecycle test covers timer wiring in start()/stop(). Existing mesh-setup-error-classification suite (27 cases) still green. Resolves: F-4 in the v1.0 audit tracker. * fix(mesh): address Codex review of PR #1184 Three findings from the independent review: BLOCKER: attemptInPlaceRecreate() called recordSetupFailure() on create / attach failures, which clears this.senchoIp. The next revalidator tick's attachment check is guarded on senchoIp, so with it null the check is skipped and the snapshot path can silently flip the status back to ok against a network where Sencho is in fact not attached. Also: a later successful recreate would call ensureSelfAttached() with senchoIp null, which short-circuits, so the network gets recreated without binding Sencho. Replaced the recordSetupFailure() calls in attemptInPlaceRecreate with a new recordRecreateFailure() that uses transitionDataPlane and preserves senchoIp. Added two tests: create-fails-then-succeeds (verifies senchoIp survives the failure and the later retry binds Sencho correctly) and create-succeeds-attach-fails-then-next-tick (verifies the snapshot path surfaces attach_failed on the next tick instead of falsely reporting ok). SHOULD-FIX 1: single-key POST /api/settings wrote String(value) without re-validating against the per-key schema, so an allowlisted enum-shaped key like mesh_auto_recreate could persist arbitrary strings ('banana', 'true') that the bulk PATCH would later refuse. Routed the single-key path through SettingsPatchSchema.safeParse so both write paths validate identically. Added regression tests for an invalid mesh_auto_recreate value, a valid mesh_auto_recreate write, and an out-of-range numeric value. SHOULD-FIX 2: the new Mesh data plane subsection lived inside a section the registry exposes to non-admins, who would see the toggle and only learn it was admin-only after the save 403'd. Gated the subsection on `isAdmin` from useAuth so non-admins do not see the control. The other system controls keep their existing visibility pattern (read-only for non-admins). 71/71 backend tests green (revalidate + mesh-setup + settings-routes). 276/276 frontend tests green. tsc clean on backend + frontend.
This commit is contained in:
Anso
@@ -1237,6 +1237,7 @@ export class DatabaseService {
|
||||
stmt.run('log_retention_days', '30');
|
||||
stmt.run('trivy_auto_update', '0');
|
||||
stmt.run('trivy_last_notified_version', '');
|
||||
stmt.run('mesh_auto_recreate', '0');
|
||||
|
||||
// Seed the default local node if none exists
|
||||
const nodeCount = (this.db.prepare('SELECT COUNT(*) as count FROM nodes').get() as any)?.count || 0;
|
||||
|
||||
@@ -26,6 +26,16 @@ const ALIAS_REFRESH_INTERVAL_MS = 60_000;
|
||||
const PROBE_TIMEOUT_MS = 5_000;
|
||||
const SLOW_PROBE_THRESHOLD_MS = 500;
|
||||
const DEFAULT_MESH_SUBNET = '172.30.0.0/24';
|
||||
// Cadence at which `revalidateDataPlane` re-evaluates the Docker network and
|
||||
// Sencho's attachment. Bounds the staleness of `getDataPlaneStatus()` between
|
||||
// runtime mesh changes (operator `docker network rm`, external recreate at a
|
||||
// different subnet, container detach) and the next /api/health response.
|
||||
export const DATA_PLANE_REVALIDATE_INTERVAL_MS = 10_000;
|
||||
// After a failed in-place recreate attempt, refuse to retry within this
|
||||
// window so a persistent CIDR overlap does not spam the Docker daemon on
|
||||
// every 10s tick. Matches the cadence pattern used by other notification
|
||||
// dedup paths (e.g. PolicyEnforcement.notifyTrivyMissingOnce).
|
||||
export const MESH_RECREATE_THROTTLE_MS = 60_000;
|
||||
|
||||
/**
|
||||
* Subnets attempted in order when SENCHO_MESH_SUBNET is unset and no
|
||||
@@ -129,7 +139,8 @@ export type MeshDataPlaneReason =
|
||||
| 'subnet_mismatch' // sencho_mesh already exists with a different subnet
|
||||
| 'ip_in_use' // another container squats <network>+2
|
||||
| 'attach_failed' // self-attach failed for any other reason
|
||||
| 'not_in_docker'; // HOSTNAME unset or self-container lookup returned 404
|
||||
| 'not_in_docker' // HOSTNAME unset or self-container lookup returned 404
|
||||
| 'not_found'; // sencho_mesh was removed after boot (revalidator-only)
|
||||
|
||||
export interface MeshDataPlaneStatus {
|
||||
ok: boolean;
|
||||
@@ -362,6 +373,19 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
// upstream tunnel is the most authoritative source. Cleared on tunnel
|
||||
// close.
|
||||
private proxyTunnelSelfCentralNodeId: number | null = null;
|
||||
// 10s revalidator timer that re-evaluates the mesh data plane against
|
||||
// Docker's current state so `/api/health` and the dashboard banner do
|
||||
// not serve stale values after the operator removes or recreates
|
||||
// sencho_mesh while Sencho is running.
|
||||
private dataPlaneRevalidateTimer?: NodeJS.Timeout;
|
||||
// Re-entrancy guard: two ticks of the revalidator can in principle
|
||||
// overlap if a Docker inspect takes longer than the cadence. Skip the
|
||||
// overlapping tick rather than fire concurrent network inspects.
|
||||
private dataPlaneRevalidateInFlight = false;
|
||||
// Wall-clock of the last auto-recreate attempt (success or failure).
|
||||
// Used by `attemptInPlaceRecreate` to enforce MESH_RECREATE_THROTTLE_MS
|
||||
// so a persistent overlap does not spam createNetwork on every tick.
|
||||
private lastRecreateAttemptAt = 0;
|
||||
|
||||
private constructor() {
|
||||
super();
|
||||
@@ -454,6 +478,17 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
}
|
||||
})();
|
||||
}, ALIAS_REFRESH_INTERVAL_MS);
|
||||
// Data-plane revalidator: bounds /api/health staleness at
|
||||
// DATA_PLANE_REVALIDATE_INTERVAL_MS when the operator removes,
|
||||
// recreates, or disconnects Sencho from sencho_mesh at runtime.
|
||||
this.dataPlaneRevalidateTimer = setInterval(() => {
|
||||
void this.revalidateDataPlane().catch((err) => {
|
||||
console.warn(
|
||||
'[MeshService] data plane revalidate failed:',
|
||||
sanitizeForLog((err as Error).message),
|
||||
);
|
||||
});
|
||||
}, DATA_PLANE_REVALIDATE_INTERVAL_MS);
|
||||
|
||||
const dpReason = this.dataPlaneStatus.reason;
|
||||
const dataPlane = this.senchoIp ? 'ok' : `unavailable (${dpReason}: ${this.networkSetupError ?? 'unknown'})`;
|
||||
@@ -483,6 +518,10 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
clearInterval(this.aliasRefreshTimer);
|
||||
this.aliasRefreshTimer = undefined;
|
||||
}
|
||||
if (this.dataPlaneRevalidateTimer) {
|
||||
clearInterval(this.dataPlaneRevalidateTimer);
|
||||
this.dataPlaneRevalidateTimer = undefined;
|
||||
}
|
||||
this.stopBridgeReconcileLoop();
|
||||
await this.forwarder.shutdown();
|
||||
}
|
||||
@@ -586,7 +625,7 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
* real failures.
|
||||
*/
|
||||
private recordSetupFailure(
|
||||
reason: Exclude<MeshDataPlaneReason, 'ok' | 'not_started'>,
|
||||
reason: Exclude<MeshDataPlaneReason, 'ok' | 'not_started' | 'not_found'>,
|
||||
err: unknown,
|
||||
level: MeshActivityLevel,
|
||||
// The subnet_invalid path fires before `this.meshSubnet` is assigned,
|
||||
@@ -995,6 +1034,370 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* One-shot snapshot of `sencho_mesh`: subnet, IPRange, and the set of
|
||||
* attached containers (full ID + Name). Returns `null` when the
|
||||
* network does not exist (Docker inspect 404). Used by the
|
||||
* revalidator to keep `/api/health` fresh after the operator
|
||||
* removes, recreates, or disconnects Sencho from the mesh at
|
||||
* runtime. Real daemon errors propagate so the revalidator can
|
||||
* preserve the prior status rather than flap it on a single
|
||||
* transient.
|
||||
*/
|
||||
private async inspectMeshNetworkSnapshot(): Promise<
|
||||
{
|
||||
subnet: string;
|
||||
ipRange: string | null;
|
||||
containers: Array<{ id: string; name: string | null }>;
|
||||
} | null
|
||||
> {
|
||||
const dc = DockerController.getInstance(NodeRegistry.getInstance().getDefaultNodeId());
|
||||
try {
|
||||
const info = await dc.inspectNetwork(SENCHO_MESH_NETWORK) as {
|
||||
IPAM?: { Config?: Array<{ Subnet?: string; IPRange?: string }> };
|
||||
Containers?: Record<string, { Name?: string }>;
|
||||
} | undefined;
|
||||
const cfg = info?.IPAM?.Config?.[0];
|
||||
if (!cfg?.Subnet) return null;
|
||||
const containers = Object.entries(info?.Containers ?? {}).map(([id, c]) => ({
|
||||
id,
|
||||
name: c?.Name ?? null,
|
||||
}));
|
||||
return {
|
||||
subnet: cfg.Subnet,
|
||||
ipRange: cfg.IPRange ?? null,
|
||||
containers,
|
||||
};
|
||||
} catch (err) {
|
||||
const e = err as { statusCode?: number };
|
||||
if (e?.statusCode === 404) return null;
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Result of the self-attachment check during revalidation:
|
||||
* - `attached`: Sencho's container is in the network's Containers map.
|
||||
* - `detached`: Sencho's identity was determinable AND not present.
|
||||
* - `unknown`: cannot determine (HOSTNAME unset / too short to be a
|
||||
* safe container-ID prefix). Caller preserves prior status.
|
||||
*/
|
||||
private isSelfStillAttached(
|
||||
containers: Array<{ id: string; name: string | null }>,
|
||||
): 'attached' | 'detached' | 'unknown' {
|
||||
const hostname = process.env.HOSTNAME;
|
||||
if (!hostname) return 'unknown';
|
||||
// Match either by container Name (operator set `--hostname`, the
|
||||
// container's `Name` in the network's Containers map equals that
|
||||
// value with a leading `/`) or by full container-ID prefix
|
||||
// (Docker's default HOSTNAME is the 12-char short ID; the
|
||||
// Containers map keys are the full 64-char IDs).
|
||||
//
|
||||
// The ID-prefix path is gated by two checks to avoid false
|
||||
// positives: HOSTNAME must be all hex (`0-9a-f`) AND at least 12
|
||||
// chars. Container IDs are pure hex, so a non-hex HOSTNAME (e.g.
|
||||
// operator-set `--hostname sencho`) cannot collide with any
|
||||
// container ID regardless of length, and a short hex HOSTNAME
|
||||
// (e.g. `--hostname ab`) could match unrelated containers.
|
||||
//
|
||||
// Network-inspect's `Name` field carries a leading `/`; normalize
|
||||
// before comparing against HOSTNAME which never has it.
|
||||
const matchByName = (name: string | null): boolean => {
|
||||
if (!name) return false;
|
||||
const normalized = name.startsWith('/') ? name.slice(1) : name;
|
||||
return normalized === hostname;
|
||||
};
|
||||
const isHexOnly = /^[0-9a-f]+$/.test(hostname);
|
||||
const idPrefixUsable = isHexOnly && hostname.length >= 12;
|
||||
for (const c of containers) {
|
||||
if (matchByName(c.name)) return 'attached';
|
||||
if (idPrefixUsable && c.id.startsWith(hostname)) return 'attached';
|
||||
}
|
||||
// No match.
|
||||
// - Non-hex HOSTNAME (e.g. `sencho`, `mynode`): the operator set
|
||||
// it explicitly. It cannot be a container-ID prefix. The Name
|
||||
// path was the only reliable signal and it failed, so we are
|
||||
// detached.
|
||||
// - Hex HOSTNAME >= 12 chars: ID-prefix path was run and missed.
|
||||
// Detached.
|
||||
// - Hex HOSTNAME < 12 chars (rare, only when an operator passes
|
||||
// `--hostname` with a hex string shorter than 12 chars): we
|
||||
// could not safely ID-prefix-match and Name did not match. We
|
||||
// genuinely do not know; preserve the prior status.
|
||||
if (!isHexOnly) return 'detached';
|
||||
if (hostname.length >= 12) return 'detached';
|
||||
return 'unknown';
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads the persisted `mesh_auto_recreate` global setting. Defaults to
|
||||
* off (`'0'`); only `'1'` enables the auto-recreate behaviour in
|
||||
* `attemptInPlaceRecreate`. Wrapped in try/catch so a transient DB
|
||||
* failure (e.g. SQLite briefly locked during a backup) cannot crash
|
||||
* the revalidator timer; on failure we default to off so a missing
|
||||
* read never accidentally triggers a Docker mutation.
|
||||
*/
|
||||
private isMeshAutoRecreateEnabled(): boolean {
|
||||
try {
|
||||
const settings = DatabaseService.getInstance().getGlobalSettings();
|
||||
return settings['mesh_auto_recreate'] === '1';
|
||||
} catch (err) {
|
||||
console.warn(
|
||||
'[MeshService] could not read mesh_auto_recreate setting:',
|
||||
sanitizeForLog((err as Error).message),
|
||||
);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Single transition path for `dataPlaneStatus`. Two-tier semantics:
|
||||
* - Fields (`message`, `subnet`) are always refreshed when they
|
||||
* drift, so `/api/health` never carries stale numbers (e.g. two
|
||||
* consecutive `subnet_mismatch` observations against different
|
||||
* external subnets both show their own remote subnet).
|
||||
* - Activity ring + `console.{log,warn}` mirrors fire only when
|
||||
* the `reason` discriminator actually changes, so the 10s timer
|
||||
* can tick indefinitely on a stable mesh without spamming the
|
||||
* log surface.
|
||||
* `networkSetupError` follows the same coherence rule it had before
|
||||
* the revalidator landed: cleared on recovery to `ok`, otherwise
|
||||
* tracks the typed status's `message`.
|
||||
*/
|
||||
private transitionDataPlane(next: MeshDataPlaneStatus): void {
|
||||
const prev = this.dataPlaneStatus;
|
||||
const reasonChanged = prev.reason !== next.reason;
|
||||
const fieldsChanged =
|
||||
reasonChanged ||
|
||||
prev.message !== next.message ||
|
||||
prev.subnet !== next.subnet ||
|
||||
prev.ok !== next.ok;
|
||||
if (!fieldsChanged) return;
|
||||
this.dataPlaneStatus = next;
|
||||
if (next.ok) {
|
||||
this.networkSetupError = null;
|
||||
} else if (next.message) {
|
||||
// Keep the legacy raw-error string in step with the typed
|
||||
// discriminator so callers that still read `networkSetupError`
|
||||
// (optInStack, applyLocalOverride, regenerateAllOverrides) see
|
||||
// the same message the typed status surfaces.
|
||||
this.networkSetupError = next.message;
|
||||
}
|
||||
if (!reasonChanged) return;
|
||||
const line = next.ok
|
||||
? `[Mesh] data plane recovered (reason ${prev.reason} -> ${next.reason}, subnet ${next.subnet})`
|
||||
: `[Mesh] data plane changed (reason ${prev.reason} -> ${next.reason}, subnet ${next.subnet}): ${next.message ?? 'no detail'}`;
|
||||
if (next.ok) console.log(line);
|
||||
else console.warn(line);
|
||||
this.logActivity({
|
||||
source: 'mesh',
|
||||
level: next.ok ? 'info' : 'warn',
|
||||
type: next.ok ? 'mesh.enable' : 'mesh.disable',
|
||||
message: line,
|
||||
details: { prevReason: prev.reason, nextReason: next.reason, subnet: next.subnet },
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* 10s tick body. Reports current Docker reality into
|
||||
* `dataPlaneStatus`; never mutates `senchoIp` or `meshSubnet`
|
||||
* (those are pinned at boot to keep override files stable). The
|
||||
* `mesh_auto_recreate` setting opts into a single bounded
|
||||
* recreate-on-the-same-subnet attempt via `attemptInPlaceRecreate`
|
||||
* when the network has been removed. Public so unit tests can call
|
||||
* it directly without manipulating timers.
|
||||
*
|
||||
* Short-circuits in states where the truth cannot have changed
|
||||
* within this process: `not_started` (boot still in flight),
|
||||
* `not_in_docker` (dev mode; HOSTNAME unset for the whole process),
|
||||
* and `subnet_invalid` (env config error; resolved only by restart
|
||||
* with new env).
|
||||
*/
|
||||
public async revalidateDataPlane(): Promise<void> {
|
||||
if (this.dataPlaneRevalidateInFlight) return;
|
||||
const reason = this.dataPlaneStatus.reason;
|
||||
if (reason === 'not_started' || reason === 'not_in_docker' || reason === 'subnet_invalid') {
|
||||
return;
|
||||
}
|
||||
this.dataPlaneRevalidateInFlight = true;
|
||||
try {
|
||||
let snapshot: Awaited<ReturnType<typeof this.inspectMeshNetworkSnapshot>>;
|
||||
try {
|
||||
snapshot = await this.inspectMeshNetworkSnapshot();
|
||||
} catch {
|
||||
// Transient Docker daemon failure. Preserve current status;
|
||||
// next tick retries. Anti-flap by design.
|
||||
return;
|
||||
}
|
||||
|
||||
if (snapshot === null) {
|
||||
// Anti-flap during the auto-recreate throttle window:
|
||||
// `attemptInPlaceRecreate` may have just classified the
|
||||
// failure (`subnet_overlap`, `ip_in_use`, `attach_failed`).
|
||||
// Those reasons are more actionable than the generic
|
||||
// `not_found`, and the network is still missing because the
|
||||
// recreate could not finish, not because the operator just
|
||||
// removed it. Preserve the recreate-failure status until
|
||||
// the throttle elapses and the next attempt re-classifies.
|
||||
const prev = this.dataPlaneStatus;
|
||||
const inThrottleWindow =
|
||||
this.lastRecreateAttemptAt > 0 &&
|
||||
Date.now() - this.lastRecreateAttemptAt < MESH_RECREATE_THROTTLE_MS;
|
||||
const isRecreateFailureReason =
|
||||
prev.reason === 'subnet_overlap' ||
|
||||
prev.reason === 'subnet_mismatch' ||
|
||||
prev.reason === 'ip_in_use' ||
|
||||
prev.reason === 'attach_failed';
|
||||
if (inThrottleWindow && isRecreateFailureReason) {
|
||||
return;
|
||||
}
|
||||
this.transitionDataPlane({
|
||||
ok: false,
|
||||
reason: 'not_found',
|
||||
message: `${SENCHO_MESH_NETWORK} is not present on this host. Mesh routing is offline; restart Sencho to recreate the network.`,
|
||||
subnet: this.meshSubnet,
|
||||
});
|
||||
if (this.isMeshAutoRecreateEnabled()) {
|
||||
await this.attemptInPlaceRecreate();
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if (this.meshSubnet && snapshot.subnet !== this.meshSubnet) {
|
||||
this.transitionDataPlane({
|
||||
ok: false,
|
||||
reason: 'subnet_mismatch',
|
||||
message: `${SENCHO_MESH_NETWORK} now uses ${snapshot.subnet}, Sencho is configured for ${this.meshSubnet}. Restart Sencho to adopt the new subnet.`,
|
||||
subnet: this.meshSubnet,
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (this.senchoIp) {
|
||||
const attachment = this.isSelfStillAttached(snapshot.containers);
|
||||
if (attachment === 'unknown') {
|
||||
// Cannot determine our own identity (HOSTNAME unset or
|
||||
// too short to be a safe ID prefix). Preserve the prior
|
||||
// status; the boot-time `ensureSelfAttached` already
|
||||
// classified the not-in-Docker case.
|
||||
return;
|
||||
}
|
||||
if (attachment === 'detached') {
|
||||
this.transitionDataPlane({
|
||||
ok: false,
|
||||
reason: 'attach_failed',
|
||||
message: `Sencho is no longer attached to ${SENCHO_MESH_NETWORK}; restart Sencho to re-attach.`,
|
||||
subnet: this.meshSubnet,
|
||||
});
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
this.transitionDataPlane({
|
||||
ok: true,
|
||||
reason: 'ok',
|
||||
message: null,
|
||||
subnet: this.meshSubnet,
|
||||
});
|
||||
} finally {
|
||||
this.dataPlaneRevalidateInFlight = false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Records a runtime recreate failure WITHOUT clearing `senchoIp`.
|
||||
* Distinct from `recordSetupFailure`, which is boot-only and clears
|
||||
* the IP as part of the failure-disables-mesh contract. At runtime
|
||||
* the boot-resolved IP must survive a transient failure so a later
|
||||
* successful attempt can drive `ensureSelfAttached` and re-bind
|
||||
* Sencho to the freshly created network. Otherwise the next
|
||||
* revalidator tick that observes the new network would skip the
|
||||
* attachment check (gated on `this.senchoIp`) and silently report
|
||||
* `ok` while Sencho is in fact detached.
|
||||
*/
|
||||
private recordRecreateFailure(
|
||||
reason: Exclude<MeshDataPlaneReason, 'ok' | 'not_started' | 'not_found'>,
|
||||
err: unknown,
|
||||
subnet: string,
|
||||
): void {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
this.transitionDataPlane({
|
||||
ok: false,
|
||||
reason,
|
||||
message,
|
||||
subnet,
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Opt-in auto-recreate path. Only invoked from the revalidator when
|
||||
* the network was observed missing AND the operator has flipped
|
||||
* `mesh_auto_recreate` to `'1'` in Settings -> System. Hard-prefers
|
||||
* `this.meshSubnet` (the subnet chosen at boot) and never iterates
|
||||
* the candidate list, because changing the chosen subnet here would
|
||||
* invalidate every existing `extra_hosts` override on disk and
|
||||
* silently break cross-node routing for opted-in stacks. A real
|
||||
* overlap on the prior subnet is reported via `subnet_overlap` so
|
||||
* the operator can take action.
|
||||
*
|
||||
* Throttled by MESH_RECREATE_THROTTLE_MS to keep a persistent
|
||||
* conflict from spamming the daemon on every 10s tick.
|
||||
*/
|
||||
private async attemptInPlaceRecreate(): Promise<void> {
|
||||
const now = Date.now();
|
||||
if (now - this.lastRecreateAttemptAt < MESH_RECREATE_THROTTLE_MS) {
|
||||
return;
|
||||
}
|
||||
this.lastRecreateAttemptAt = now;
|
||||
// A revalidator-driven recreate is only meaningful when the
|
||||
// boot-resolved Sencho IP is known. If `senchoIp` is null,
|
||||
// setupMeshNetwork either never ran (impossible: revalidator
|
||||
// short-circuits on `not_started`) or recorded a hard failure
|
||||
// (`subnet_invalid` / `not_in_docker`, both of which also
|
||||
// short-circuit). Defensive bail.
|
||||
if (!this.senchoIp) return;
|
||||
this.logActivity({
|
||||
source: 'mesh',
|
||||
level: 'info',
|
||||
type: 'mesh.enable',
|
||||
message: `attempting auto-recreate of ${SENCHO_MESH_NETWORK} at ${this.meshSubnet}`,
|
||||
details: { subnet: this.meshSubnet },
|
||||
});
|
||||
try {
|
||||
await this.createMeshNetwork(this.meshSubnet);
|
||||
} catch (err) {
|
||||
// Preserve `senchoIp` so a later successful retry (after the
|
||||
// throttle elapses + the overlap clears) can re-attach.
|
||||
this.recordRecreateFailure(
|
||||
this.classifyMeshNetworkError(err),
|
||||
err,
|
||||
this.meshSubnet,
|
||||
);
|
||||
return;
|
||||
}
|
||||
try {
|
||||
await this.ensureSelfAttached();
|
||||
} catch (err) {
|
||||
// Same preservation rationale; the attach can succeed on a
|
||||
// later attempt (e.g. when whatever was squatting the IP
|
||||
// gets out of the way). The network now exists, so the next
|
||||
// revalidator tick will surface the attachment failure
|
||||
// honestly through the snapshot path.
|
||||
this.recordRecreateFailure(
|
||||
this.classifySelfAttachError(err),
|
||||
err,
|
||||
this.meshSubnet,
|
||||
);
|
||||
return;
|
||||
}
|
||||
this.transitionDataPlane({
|
||||
ok: true,
|
||||
reason: 'ok',
|
||||
message: null,
|
||||
subnet: this.meshSubnet,
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Bind the forwarder's listeners to every alias port across the fleet
|
||||
* and release any listeners no longer in the alias set. Called from
|
||||
|
||||
Reference in New Issue
Block a user