mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-16 21:48:45 +00:00
fix(spawn): attribute ENOMEM and ENOENT-under-memory-pressure spawn failures to host OOM (#1111)
Operators previously saw "spawn docker ENOENT" or "spawn /bin/sh ENOENT" when the host was under memory pressure, which sent them down a missing-binary debugging path. Linux libuv's posix_spawn can fail to allocate its argv / path-search arena under low free memory and surface the underlying ENOMEM as ENOENT. Centralizes spawn-error mapping in a new utils/spawnErrors.ts helper: - Explicit ENOMEM is rewritten to "Out of memory while launching <command> (host free memory: X MiB of Y MiB)". - ENOENT under the 128 MiB free-memory floor is rewritten with the same wording plus a "reported as ENOENT under memory pressure" hint. - ENOENT for docker on a healthy host preserves the existing "Docker CLI unavailable on this node" mapping. - Other errors pass through unchanged. Applied at the four named offenders: ComposeService.execute(), ComposeService.captureCompose(), DockerController.getContainersByStack(), and FileSystemService.getStacks() (which gets an ENOMEM-aware log line for the scandir failure). Startup also logs host free/total MiB once and warns when free memory is below the 128 MiB floor, so the diagnostic surfaces before the first spawn attempt rather than after it fails. 37 tests cover the mapping function directly and the ComposeService / FileSystemService integration paths.
This commit is contained in:
@@ -1,5 +1,6 @@
|
||||
import type { Server } from 'http';
|
||||
import crypto from 'crypto';
|
||||
import os from 'os';
|
||||
import { FileSystemService } from '../services/FileSystemService';
|
||||
import { NodeRegistry } from '../services/NodeRegistry';
|
||||
import { DatabaseService } from '../services/DatabaseService';
|
||||
@@ -21,6 +22,7 @@ import { PilotMetrics } from '../services/PilotMetrics';
|
||||
import { invalidateRemoteMetaCache } from '../helpers/cacheInvalidation';
|
||||
import { sweepStaleTempDirs as sweepStaleGitTempDirs } from '../services/GitSourceService';
|
||||
import { PORT } from '../helpers/constants';
|
||||
import { LOW_MEMORY_FLOOR_BYTES } from '../utils/spawnErrors';
|
||||
|
||||
function isPilotMode(): boolean {
|
||||
return process.env.SENCHO_MODE === 'pilot';
|
||||
@@ -56,6 +58,19 @@ export function ensurePilotJwtSecret(): boolean {
|
||||
* port.
|
||||
*/
|
||||
export async function startServer(server: Server): Promise<void> {
|
||||
const freeBytes = os.freemem();
|
||||
const freeMiB = Math.round(freeBytes / (1024 * 1024));
|
||||
const totalMiB = Math.round(os.totalmem() / (1024 * 1024));
|
||||
const floorMiB = Math.round(LOW_MEMORY_FLOOR_BYTES / (1024 * 1024));
|
||||
console.log(`[Startup] Host memory: ${freeMiB} MiB free of ${totalMiB} MiB`);
|
||||
if (freeBytes < LOW_MEMORY_FLOOR_BYTES) {
|
||||
console.warn(
|
||||
`[Startup] Free host memory is ${freeMiB} MiB (below ${floorMiB} MiB floor). ` +
|
||||
'Sencho operations that spawn child processes (docker, /bin/sh) may fail ' +
|
||||
'with misleading ENOENT errors under memory pressure.'
|
||||
);
|
||||
}
|
||||
|
||||
try {
|
||||
console.log('Running stack migration check...');
|
||||
const defaultFsService = FileSystemService.getInstance(NodeRegistry.getInstance().getDefaultNodeId());
|
||||
|
||||
Reference in New Issue
Block a user