mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-27 18:57:09 +00:00
fix(mesh): recompose previously-meshed containers when alias set changes (#1090)
When a stack is opted into the mesh, every previously-meshed stack's override.yml on disk gains the new alias, but the running containers keep their stale /etc/hosts because extra_hosts is read at container creation time. Cross-stack DNS for the new alias silently fails until an operator manually redeploys each prior stack. Complete the cascade: after regenerateOverridesAcrossFleet finishes, fan out triggerRedeploy across every (node_id, stack_name) tuple in mesh_stacks, skipping the just-opted-in pair (the explicit triggerRedeploy at the end of optInStack covers it). Mirror the same cascade in optOutStack so the dropped alias exits every container's /etc/hosts. triggerRedeploy already dispatches local vs remote, already wraps runRedeploy in fire-and-forget error logging to the mesh activity ring and audit log, and already enforces compose policy gates. The cascade just reuses it. regenerateAllOverrides (boot and manual /regen-overrides) is intentionally NOT covered: a Sencho restart must not force a fleet-wide recompose; override files alone are sufficient there. Adds four unit tests in mesh-service.test.ts: cascade fires for every prior meshed stack on opt-in, no double-redeploy of the just-opted-in tuple, cascade is a no-op when no peers exist, cascade fires for every survivor on opt-out.
This commit is contained in:
@@ -795,6 +795,12 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
// avoid a duplicate round-trip. Best-effort; per-stack failures
|
||||
// surface as forwarder.error activity events.
|
||||
await this.regenerateOverridesAcrossFleet(nodeId, stackName);
|
||||
// Recompose every previously-meshed container so the new alias
|
||||
// actually lands in /etc/hosts. The override file alone is not
|
||||
// enough: extra_hosts is read at container creation, so prior
|
||||
// containers need to be recreated. Skip the just-opted-in tuple
|
||||
// because the explicit triggerRedeploy below already covers it.
|
||||
this.cascadeRecomposeAcrossFleet(nodeId, stackName, actor);
|
||||
this.triggerRedeploy(nodeId, stackName, actor);
|
||||
|
||||
this.logActivity({
|
||||
@@ -823,6 +829,13 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
// include it. Walk the remaining fleet-wide rows so every other
|
||||
// meshed stack regenerates its override without the dropped alias.
|
||||
await this.regenerateOverridesAcrossFleet();
|
||||
// Recompose every still-meshed container so the dropped alias
|
||||
// exits /etc/hosts. Skip args are absent because the opted-out
|
||||
// row is already gone from listMeshStacks; the explicit
|
||||
// triggerRedeploy below recomposes the opted-out stack itself
|
||||
// (with the override file removed) so its container drops the
|
||||
// entries it owned.
|
||||
this.cascadeRecomposeAcrossFleet(undefined, undefined, actor);
|
||||
this.triggerRedeploy(nodeId, stackName, actor);
|
||||
|
||||
this.logActivity({
|
||||
@@ -1080,6 +1093,64 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Walk every `mesh_stacks` row across the fleet and fire a redeploy for
|
||||
* each, skipping the (skipNodeId, skipStack) tuple. Called from
|
||||
* optInStack / optOutStack after `regenerateOverridesAcrossFleet` so
|
||||
* previously-meshed containers actually pick up the new or removed
|
||||
* alias entries in `/etc/hosts`. Without this, override `.yml` files on
|
||||
* disk reflect the new alias set but the running containers still hold
|
||||
* the alias set they had at last compose, so cross-stack DNS silently
|
||||
* fails until an operator redeploys every prior stack by hand.
|
||||
*
|
||||
* Each `triggerRedeploy` call is fire-and-forget. Local stacks route
|
||||
* through `ComposeService.deployStack`, remote stacks through
|
||||
* `POST /api/stacks/:name/deploy` via the proxy chain. Failures land in
|
||||
* the mesh activity ring buffer and the audit log, so one slow or
|
||||
* offline peer cannot block other targets.
|
||||
*
|
||||
* This is intentionally not invoked from `regenerateAllOverrides`
|
||||
* (boot and manual `/regen-overrides`). A Sencho restart must not
|
||||
* force a fleet-wide recompose of every meshed stack; the override
|
||||
* files alone are sufficient there.
|
||||
*
|
||||
* Pacing: the cascade fans out in parallel. For the v1 mesh-stack
|
||||
* counts (single-digit to low teens per host) this is fine; Docker's
|
||||
* daemon serializes the contention that matters. If real-world fleets
|
||||
* routinely exceed ~20 meshed stacks on one host, swap the loop for a
|
||||
* `p-limit(4)` semaphore keyed on `node_id` (no test rewiring needed).
|
||||
*/
|
||||
private cascadeRecomposeAcrossFleet(
|
||||
skipNodeId: number | undefined,
|
||||
skipStack: string | undefined,
|
||||
actor: string,
|
||||
): void {
|
||||
const db = DatabaseService.getInstance();
|
||||
const targets = db.listMeshStacks().filter(
|
||||
(s) => !(s.node_id === skipNodeId && s.stack_name === skipStack),
|
||||
);
|
||||
if (targets.length === 0) return;
|
||||
|
||||
for (const t of targets) {
|
||||
this.triggerRedeploy(t.node_id, t.stack_name, actor);
|
||||
}
|
||||
|
||||
const nodeIds = new Set(targets.map((t) => t.node_id));
|
||||
const skippedNote = skipNodeId !== undefined && skipStack !== undefined
|
||||
? ` (skipped ${skipStack} on node ${skipNodeId})`
|
||||
: '';
|
||||
this.logActivity({
|
||||
source: 'mesh', level: 'info', type: 'mesh.enable',
|
||||
message: `mesh cascade recompose: ${targets.length} stack${targets.length === 1 ? '' : 's'} across ${nodeIds.size} node${nodeIds.size === 1 ? '' : 's'}${skippedNote}`,
|
||||
details: {
|
||||
cascadeRecomposes: targets.length,
|
||||
nodeCount: nodeIds.size,
|
||||
skipNodeId: skipNodeId ?? null,
|
||||
skipStack: skipStack ?? null,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Walk every `mesh_stacks` row across the fleet and re-push each override
|
||||
* to its owning node. Called once at boot so on-disk override files
|
||||
|
||||
Reference in New Issue
Block a user