fix(mesh): recompose previously-meshed containers when alias set changes (#1090)

When a stack is opted into the mesh, every previously-meshed stack's
override.yml on disk gains the new alias, but the running containers
keep their stale /etc/hosts because extra_hosts is read at container
creation time. Cross-stack DNS for the new alias silently fails until
an operator manually redeploys each prior stack.

Complete the cascade: after regenerateOverridesAcrossFleet finishes,
fan out triggerRedeploy across every (node_id, stack_name) tuple in
mesh_stacks, skipping the just-opted-in pair (the explicit
triggerRedeploy at the end of optInStack covers it). Mirror the same
cascade in optOutStack so the dropped alias exits every container's
/etc/hosts.

triggerRedeploy already dispatches local vs remote, already wraps
runRedeploy in fire-and-forget error logging to the mesh activity
ring and audit log, and already enforces compose policy gates. The
cascade just reuses it.

regenerateAllOverrides (boot and manual /regen-overrides) is
intentionally NOT covered: a Sencho restart must not force a
fleet-wide recompose; override files alone are sufficient there.

Adds four unit tests in mesh-service.test.ts: cascade fires for every
prior meshed stack on opt-in, no double-redeploy of the just-opted-in
tuple, cascade is a no-op when no peers exist, cascade fires for
every survivor on opt-out.
This commit is contained in:
Anso
2026-05-17 19:02:02 -04:00
committed by GitHub
parent c2a23378aa
commit 1d7418a233
2 changed files with 186 additions and 0 deletions
+71
View File
@@ -795,6 +795,12 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
// avoid a duplicate round-trip. Best-effort; per-stack failures
// surface as forwarder.error activity events.
await this.regenerateOverridesAcrossFleet(nodeId, stackName);
// Recompose every previously-meshed container so the new alias
// actually lands in /etc/hosts. The override file alone is not
// enough: extra_hosts is read at container creation, so prior
// containers need to be recreated. Skip the just-opted-in tuple
// because the explicit triggerRedeploy below already covers it.
this.cascadeRecomposeAcrossFleet(nodeId, stackName, actor);
this.triggerRedeploy(nodeId, stackName, actor);
this.logActivity({
@@ -823,6 +829,13 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
// include it. Walk the remaining fleet-wide rows so every other
// meshed stack regenerates its override without the dropped alias.
await this.regenerateOverridesAcrossFleet();
// Recompose every still-meshed container so the dropped alias
// exits /etc/hosts. Skip args are absent because the opted-out
// row is already gone from listMeshStacks; the explicit
// triggerRedeploy below recomposes the opted-out stack itself
// (with the override file removed) so its container drops the
// entries it owned.
this.cascadeRecomposeAcrossFleet(undefined, undefined, actor);
this.triggerRedeploy(nodeId, stackName, actor);
this.logActivity({
@@ -1080,6 +1093,64 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
);
}
/**
* Walk every `mesh_stacks` row across the fleet and fire a redeploy for
* each, skipping the (skipNodeId, skipStack) tuple. Called from
* optInStack / optOutStack after `regenerateOverridesAcrossFleet` so
* previously-meshed containers actually pick up the new or removed
* alias entries in `/etc/hosts`. Without this, override `.yml` files on
* disk reflect the new alias set but the running containers still hold
* the alias set they had at last compose, so cross-stack DNS silently
* fails until an operator redeploys every prior stack by hand.
*
* Each `triggerRedeploy` call is fire-and-forget. Local stacks route
* through `ComposeService.deployStack`, remote stacks through
* `POST /api/stacks/:name/deploy` via the proxy chain. Failures land in
* the mesh activity ring buffer and the audit log, so one slow or
* offline peer cannot block other targets.
*
* This is intentionally not invoked from `regenerateAllOverrides`
* (boot and manual `/regen-overrides`). A Sencho restart must not
* force a fleet-wide recompose of every meshed stack; the override
* files alone are sufficient there.
*
* Pacing: the cascade fans out in parallel. For the v1 mesh-stack
* counts (single-digit to low teens per host) this is fine; Docker's
* daemon serializes the contention that matters. If real-world fleets
* routinely exceed ~20 meshed stacks on one host, swap the loop for a
* `p-limit(4)` semaphore keyed on `node_id` (no test rewiring needed).
*/
private cascadeRecomposeAcrossFleet(
skipNodeId: number | undefined,
skipStack: string | undefined,
actor: string,
): void {
const db = DatabaseService.getInstance();
const targets = db.listMeshStacks().filter(
(s) => !(s.node_id === skipNodeId && s.stack_name === skipStack),
);
if (targets.length === 0) return;
for (const t of targets) {
this.triggerRedeploy(t.node_id, t.stack_name, actor);
}
const nodeIds = new Set(targets.map((t) => t.node_id));
const skippedNote = skipNodeId !== undefined && skipStack !== undefined
? ` (skipped ${skipStack} on node ${skipNodeId})`
: '';
this.logActivity({
source: 'mesh', level: 'info', type: 'mesh.enable',
message: `mesh cascade recompose: ${targets.length} stack${targets.length === 1 ? '' : 's'} across ${nodeIds.size} node${nodeIds.size === 1 ? '' : 's'}${skippedNote}`,
details: {
cascadeRecomposes: targets.length,
nodeCount: nodeIds.size,
skipNodeId: skipNodeId ?? null,
skipStack: skipStack ?? null,
},
});
}
/**
* Walk every `mesh_stacks` row across the fleet and re-push each override
* to its owning node. Called once at boot so on-disk override files