mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-06 00:47:52 +00:00
Anso
9dbce9c3c7
Operators previously saw "spawn docker ENOENT" or "spawn /bin/sh ENOENT" when the host was under memory pressure, which sent them down a missing-binary debugging path. Linux libuv's posix_spawn can fail to allocate its argv / path-search arena under low free memory and surface the underlying ENOMEM as ENOENT. Centralizes spawn-error mapping in a new utils/spawnErrors.ts helper: - Explicit ENOMEM is rewritten to "Out of memory while launching <command> (host free memory: X MiB of Y MiB)". - ENOENT under the 128 MiB free-memory floor is rewritten with the same wording plus a "reported as ENOENT under memory pressure" hint. - ENOENT for docker on a healthy host preserves the existing "Docker CLI unavailable on this node" mapping. - Other errors pass through unchanged. Applied at the four named offenders: ComposeService.execute(), ComposeService.captureCompose(), DockerController.getContainersByStack(), and FileSystemService.getStacks() (which gets an ENOMEM-aware log line for the scandir failure). Startup also logs host free/total MiB once and warns when free memory is below the 128 MiB floor, so the diagnostic surfaces before the first spawn attempt rather than after it fails. 37 tests cover the mapping function directly and the ComposeService / FileSystemService integration paths.
64 lines
2.3 KiB
TypeScript
64 lines
2.3 KiB
TypeScript
import os from 'os';
|
|
|
|
/**
|
|
* Below this floor of free host memory, treat an ENOENT or ENOMEM
|
|
* spawn failure as a memory-pressure event rather than a missing binary.
|
|
* Linux libuv's posix_spawn can fail to allocate its argv/path-search arena
|
|
* under memory pressure and surface the underlying ENOMEM as ENOENT, sending
|
|
* operators down a "missing binary" debugging path when the real cause is
|
|
* host OOM. 128 MiB is well above the point where posix_spawn starts dropping
|
|
* arenas yet low enough that a healthy homelab host never trips it.
|
|
*/
|
|
export const LOW_MEMORY_FLOOR_BYTES = 128 * 1024 * 1024;
|
|
|
|
export interface SpawnErrorContext {
|
|
/** Literal first argument passed to spawn/exec (e.g. "docker", "/bin/sh"). */
|
|
command: string;
|
|
}
|
|
|
|
export interface MappedSpawnError {
|
|
/** Operator-facing message. Safe to send through WS / log / throw. */
|
|
message: string;
|
|
/** True when the failure was attributed to host memory pressure. */
|
|
isLowMemory: boolean;
|
|
}
|
|
|
|
/**
|
|
* Rewrite a spawn / exec error into an operator-facing message.
|
|
*
|
|
* Ordering matters: an explicit ENOMEM always wins, then an ENOENT under low
|
|
* free memory is treated as the libuv masquerade case, and only after that
|
|
* does the genuine "docker CLI missing" mapping fire. Other errors pass
|
|
* through unchanged.
|
|
*/
|
|
export function describeSpawnError(
|
|
error: NodeJS.ErrnoException,
|
|
ctx: SpawnErrorContext,
|
|
): MappedSpawnError {
|
|
const free = os.freemem();
|
|
const total = os.totalmem();
|
|
const freeMiB = Math.round(free / (1024 * 1024));
|
|
const totalMiB = Math.round(total / (1024 * 1024));
|
|
const lowMem = free < LOW_MEMORY_FLOOR_BYTES;
|
|
|
|
if (error.code === 'ENOMEM') {
|
|
return {
|
|
message: `Out of memory while launching ${ctx.command} (host free memory: ${freeMiB} MiB of ${totalMiB} MiB)`,
|
|
isLowMemory: true,
|
|
};
|
|
}
|
|
|
|
if (error.code === 'ENOENT' && lowMem) {
|
|
return {
|
|
message: `Out of memory while launching ${ctx.command} (host free memory: ${freeMiB} MiB of ${totalMiB} MiB; reported as ENOENT under memory pressure)`,
|
|
isLowMemory: true,
|
|
};
|
|
}
|
|
|
|
if (error.code === 'ENOENT' && /^spawn docker(?:$| )/.test(error.message ?? '')) {
|
|
return { message: 'Docker CLI unavailable on this node', isLowMemory: false };
|
|
}
|
|
|
|
return { message: error.message ?? String(error), isLowMemory: false };
|
|
}
|