fix(mesh): cascade override regen across all meshed nodes on opt-in/opt-out (#1079)

Opting a stack into mesh on node A now propagates the new alias to every
meshed node's override file, not just stacks on node A. Same on opt-out.

Pre-fix, optInStack and optOutStack called regenerateOverridesForNode
which only walks db.listMeshStacks(nodeId). Other meshed stacks on other
nodes did not learn about the new alias until something else re-pushed
their overrides; operators worked around it with a manual
POST /api/mesh/regen-overrides plus a redeploy of each affected stack.

New regenerateOverridesAcrossFleet helper iterates db.listMeshStacks()
(no arg, fleet-wide) under Promise.allSettled, skipping the
(nodeId, stackName) tuple opt-in just pushed loudly. Per-stack failures
surface as forwarder.error activity events, matching the pattern used by
regenerateAllOverrides. Offline remote nodes leave stale overrides until
the next opt-in/opt-out, the next tunnel reconnect, or a manual rerun of
the regen-overrides endpoint.

regenerateOverridesForNode stays for the tunnel-up retry listener: a
reconnecting pilot only needs its own stacks re-pushed, not the fleet.

Tests: redirect five existing spies in mesh-service.test.ts to the new
method; add two new cases asserting cross-node cascade for opt-in and
opt-out. 52 tests in mesh-service.test.ts pass.
This commit is contained in:
Anso
2026-05-17 03:48:58 -04:00
committed by GitHub
parent de725cc4c1
commit a32183d198
2 changed files with 107 additions and 11 deletions
+45 -6
View File
@@ -675,11 +675,12 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
await this.syncForwarderListeners();
throw err;
}
// Regenerate OTHER meshed stacks' overrides on the same node so
// they pick up the new alias entry. The just-opted-in stack was
// already pushed above; skip it to avoid a duplicate round-trip.
// Best-effort; per-stack failures are logged inside the helper.
await this.regenerateOverridesForNode(nodeId, stackName);
// Regenerate every other meshed stack's override across the fleet
// so they pick up the new alias entry. The just-opted-in stack was
// already pushed above; skip the (nodeId, stackName) tuple to
// avoid a duplicate round-trip. Best-effort; per-stack failures
// surface as forwarder.error activity events.
await this.regenerateOverridesAcrossFleet(nodeId, stackName);
this.triggerRedeploy(nodeId, stackName, actor);
this.logActivity({
@@ -704,7 +705,10 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
await this.removeOverrideFromNode(nodeId, stackName);
await this.refreshAliasCache();
await this.syncForwarderListeners();
await this.regenerateOverridesForNode(nodeId);
// The opted-out row is already deleted, so listMeshStacks() will not
// include it. Walk the remaining fleet-wide rows so every other
// meshed stack regenerates its override without the dropped alias.
await this.regenerateOverridesAcrossFleet();
this.triggerRedeploy(nodeId, stackName, actor);
this.logActivity({
@@ -927,6 +931,41 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
);
}
/**
* Walk every `mesh_stacks` row across the fleet and re-push each override.
* Called from optInStack / optOutStack so a new or removed alias
* propagates to every meshed node's override file in one pass, not just
* the node whose row changed. Best-effort: per-stack failures emit a
* forwarder.error activity event and the other nodes still get
* regenerated. An offline remote node leaves stale overrides until the
* next opt-in / opt-out, the next tunnel reconnect, or a manual
* `POST /api/mesh/regen-overrides`.
*/
private async regenerateOverridesAcrossFleet(
skipNodeId?: number,
skipStack?: string,
): Promise<void> {
const db = DatabaseService.getInstance();
const stacks = db.listMeshStacks();
await Promise.allSettled(
stacks
.filter((s) => !(s.node_id === skipNodeId && s.stack_name === skipStack))
.map(async (s) => {
try {
await this.pushOverrideToNode(s.node_id, s.stack_name);
} catch (err) {
const message = sanitizeForLog((err as Error).message);
this.logActivity({
source: 'mesh', level: 'warn', type: 'forwarder.error',
nodeId: s.node_id,
message: `cascade override push failed for ${s.stack_name}: ${message}`,
details: { stackName: s.stack_name },
});
}
}),
);
}
/**
* Walk every `mesh_stacks` row across the fleet and re-push each override
* to its owning node. Called once at boot so on-disk override files