gsadmin/sencho
1
0
Fork 0
mirror of https://github.com/Studio-Saelix/sencho.git synced 2026-08-27 18:57:09 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(monitor): decouple janitor disk-usage check from 30s cycle (F-6) (#1164)

Browse Source 
* fix(monitor): decouple janitor disk-usage check from 30s cycle (F-6)

`docker system df` (called by the MonitorService janitor check) can take
30+ seconds on Docker Desktop with many volumes. Running it on the 30s
evaluate cycle compounded with the per-container stats fan-out and pushed
the cycle to 140s+, blocking subsequent monitoring work.

This change:

- Moves the janitor disk-usage check into its own 15-minute cycle with
  a tight 8s timeout. A circuit breaker opens after 3 consecutive
  timeouts (60-minute cooldown) so a sick daemon stops pinning Dockerode
  sockets every tick. The first janitor tick is deferred 45 seconds past
  boot to avoid head-of-line collision with the initial monitor cycle's
  stats fan-out.
- Adds a paired 8s `withTimeout` wrap to the admin prune-estimate routes
  (`/api/system/prune/estimate` and the dry-run path of
  `/api/system/prune/system`) so a slow df does not hang the admin tab.
  Both routes respond 503 with code `docker_df_slow` on timeout.
- Factors `withTimeout` and `TimeoutError` into `utils/withTimeout.ts`
  so the route layer does not have to import from a service module.
- Adds 10 unit tests covering the decoupling guardrail, breaker
  open/close, cooldown, threshold gate, the 100 MB reclaimable floor,
  re-entrancy, recovery logging, non-timeout error handling, and the
  full timer-cleanup contract of `stop()`.
- Adds 4 integration tests for the prune routes covering the 503
  timeout response, the success path, and the non-timeout 5xx path.

* fix(fleet,monitor): extend F-6 timeout to fleet prune routes; close breaker-recovery log gap

Codex audit findings on PR #1164:

Major. The fleet routes that fan out prune-estimate work on local nodes
(`POST /api/fleet/labels/fleet-prune` dry-run path and
`POST /api/fleet/prune/estimate`) called `estimateSystemReclaim` without
a timeout, so a slow local Docker daemon could still hang the fleet
admin tab even though the system-maintenance routes were already
bounded. Wrap both call sites with the shared `withTimeout(..., 8s)`
and surface a "Docker daemon is busy" message via the per-target and
per-node error channels the routes already used for other failures.
The destructive (non-dry-run) prune path stays unwrapped because it
calls `pruneSystem` / `pruneManagedOnly`, not `df`.

Minor. The janitor circuit breaker zeroed `janitorConsecutiveTimeouts`
when it opened, so a successful call after a full breaker-open cooldown
slipped past the `if (counter > 0)` recovery-log branch and never
emitted `[Monitor] Janitor disk-usage check recovered`. The operator
observability signal was missing exactly when it mattered most.
Extend the predicate to also trip on `janitorBreakerUntil > 0` (which
stays set to its past timestamp after cooldown until the next success
clears it), so recovery logs symmetrically for both partial-failure
and post-breaker recovery paths. Added a dedicated test.

Three new integration tests cover the fleet routes (timeout, success,
and the estimate endpoint's per-node unreachable shape).
This commit is contained in:
Anso
2026-05-23 00:04:05 -04:00
committed by GitHub
parent da3e7adc30
commit a51547a158
7 changed files with 702 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/src/routes/fleet.ts
+29 -5
View File
@@ -23,6 +23,11 @@ import { isValidStackName } from '../utils/validation';
import { isDebugEnabled } from '../utils/debug';
import { getErrorMessage } from '../utils/errors';
import { parseIntParam } from '../utils/parseIntParam';
import { withTimeout, TimeoutError } from '../utils/withTimeout';
// Mirror the system-maintenance route timeout so fleet's local-node prune
// paths cap the slow `docker system df` call at the same 8s budget (F-6).
const FLEET_DF_TIMEOUT_MS = 8_000;
import { POLICY_SEVERITIES } from '../utils/severity';
import { sanitizeForLog } from '../utils/safeLog';
import { formatNoTargetError } from '../utils/remoteTarget';
@@ -1197,9 +1202,17 @@ fleetRouter.post('/labels/fleet-prune', authMiddleware, async (req: Request, res
for (const target of targets) {
try {
if (isDryRun) {
// estimateSystemReclaim hits `docker system df`; bound it
// so a slow local daemon doesn't hang the fleet admin tab
// (F-6). estimateManagedReclaim is fast (no df) and stays
// unwrapped.
const estimate = scope === 'managed'
? await dockerController.estimateManagedReclaim(target, knownStacks)
: await dockerController.estimateSystemReclaim(target, knownStacks);
: await withTimeout(
dockerController.estimateSystemReclaim(target, knownStacks),
FLEET_DF_TIMEOUT_MS,
'docker disk usage',
);
targetResults.push({ target, success: true, reclaimedBytes: estimate.reclaimableBytes, dryRun: true });
continue;
}
@@ -1209,7 +1222,10 @@ fleetRouter.post('/labels/fleet-prune', authMiddleware, async (req: Request, res
targetResults.push({ target, success: true, reclaimedBytes: result.reclaimedBytes });
if (result.reclaimedBytes > 0 || result.success) anySuccess = true;
} catch (err) {
targetResults.push({ target, success: false, reclaimedBytes: 0, error: getErrorMessage(err, 'Prune failed') });
const error = err instanceof TimeoutError
? 'Docker daemon is busy. Please try again in a moment.'
: getErrorMessage(err, 'Prune failed');
targetResults.push({ target, success: false, reclaimedBytes: 0, error });
}
}
if (anySuccess && !isDryRun) invalidateNodeCaches(node.id);
@@ -1370,16 +1386,24 @@ fleetRouter.post('/prune/estimate', authMiddleware, async (req: Request, res: Re
const dockerController = DockerController.getInstance(node.id);
let nodeBytes = 0;
for (const target of targets) {
// estimateSystemReclaim hits `docker system df`; bound it so a
// slow local daemon doesn't hang the fleet estimate (F-6).
const result = scope === 'managed'
? await dockerController.estimateManagedReclaim(target, knownStacks)
: await dockerController.estimateSystemReclaim(target, knownStacks);
: await withTimeout(
dockerController.estimateSystemReclaim(target, knownStacks),
FLEET_DF_TIMEOUT_MS,
'docker disk usage',
);
nodeBytes += result.reclaimableBytes;
}
return { nodeId: node.id, nodeName: node.name, reclaimableBytes: nodeBytes, reachable: true };
} catch (err) {
const error = err instanceof TimeoutError
? 'Docker daemon is busy. Please try again in a moment.'
: getErrorMessage(err, 'Failed to estimate locally');
return {
nodeId: node.id, nodeName: node.name, reclaimableBytes: 0, reachable: false,
error: getErrorMessage(err, 'Failed to estimate locally'),
nodeId: node.id, nodeName: node.name, reclaimableBytes: 0, reachable: false, error,
};
}
}
backend/src/routes/systemMaintenance.ts
+40 -6
View File
@@ -8,6 +8,20 @@ import { isValidDockerResourceId, isValidCidr, isValidIPv4 } from '../utils/vali
import { isDebugEnabled } from '../utils/debug';
import { getErrorMessage } from '../utils/errors';
import { sanitizeForLog } from '../utils/safeLog';
import { withTimeout, TimeoutError } from '../utils/withTimeout';
// `docker system df` (the call backing estimateSystemReclaim) can take 30+
// seconds on Docker Desktop with many volumes; 8s matches the MonitorService
// janitor timeout so the daemon never has more than ~16s of concurrent
// pressure from Sencho's own paths even when prune and janitor collide.
const PRUNE_ESTIMATE_TIMEOUT_MS = 8_000;
function respondDfSlow(res: Response): Response {
return res.status(503).json({
error: 'Docker daemon is busy. Please try again in a moment.',
code: 'docker_df_slow',
});
}
export const systemMaintenanceRouter = Router();
@@ -95,9 +109,15 @@ systemMaintenanceRouter.post('/prune/system', async (req: Request, res: Response
knownStacks,
);
} else {
estimate = await dockerController.estimateSystemReclaim(
target as 'containers' | 'images' | 'networks' | 'volumes',
knownStacks,
// estimateSystemReclaim calls `docker system df`; bound it so a slow
// daemon doesn't hang the admin's tab (F-6).
estimate = await withTimeout(
dockerController.estimateSystemReclaim(
target as 'containers' | 'images' | 'networks' | 'volumes',
knownStacks,
),
PRUNE_ESTIMATE_TIMEOUT_MS,
'docker disk usage',
);
}
res.json({ message: 'Dry run', success: true, dryRun: true, reclaimedBytes: estimate.reclaimableBytes });
@@ -122,6 +142,10 @@ systemMaintenanceRouter.post('/prune/system', async (req: Request, res: Response
}
res.json({ message: 'Prune completed', ...result });
} catch (error: unknown) {
if (error instanceof TimeoutError) {
console.warn('System prune: docker disk usage timed out');
return respondDfSlow(res);
}
console.error('System prune error:', error);
res.status(500).json({ error: 'System prune failed' });
}
@@ -150,13 +174,23 @@ systemMaintenanceRouter.post('/prune/estimate', async (req: Request, res: Respon
knownStacks,
);
} else {
result = await dockerController.estimateSystemReclaim(
target as 'containers' | 'images' | 'networks' | 'volumes',
knownStacks,
// estimateSystemReclaim calls `docker system df`; bound it so a slow
// daemon doesn't hang the admin's tab (F-6).
result = await withTimeout(
dockerController.estimateSystemReclaim(
target as 'containers' | 'images' | 'networks' | 'volumes',
knownStacks,
),
PRUNE_ESTIMATE_TIMEOUT_MS,
'docker disk usage',
);
}
res.json({ reclaimableBytes: result.reclaimableBytes });
} catch (error: unknown) {
if (error instanceof TimeoutError) {
console.warn('Prune estimate: docker disk usage timed out');
return respondDfSlow(res);
}
console.error('Prune estimate error:', error);
res.status(500).json({ error: 'Failed to estimate reclaimable bytes' });
}