mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-06 08:58:05 +00:00
38aabe7064
* feat: classify stack deploy and update failures with suggested next actions Failed deploy and update responses now carry a failure classification (cause category, headline, and suggested next step) derived from the compose error output. The recovery panel and chip render the classification and include it in copied diagnostics, and gateway-style failures surface as a node-unreachable cause. * feat: add update and rollback readiness reports for stacks Before a manual update, Sencho now shows an advisory readiness verdict computed from the stored preflight result, open drift findings, live container health, the pending image change, the rollback backup slot, and node disk headroom. The Stack Dossier gains a rollback readiness section that states what a rollback can restore and explicitly discloses that volume and bind-mounted data are not covered. Toolbar and sidebar updates now share one update path, and admins can create a fleet snapshot from the readiness dialog before updating. Nodes that do not advertise the capability keep the direct update flow. * feat: observe stack health after updates with a post-deploy health gate After a deploy or update succeeds, Sencho now watches the stack for a configurable observation window and records a passed, failed, or unknown verdict: containers must stay running, healthchecks must report healthy, and restart loops or disappearing containers fail the gate. The deploy panel shows the observation live and holds off auto-closing until the verdict lands, a failed gate surfaces the existing recovery actions including rollback, and the stack timeline records update started and gate verdict events. Scheduled, webhook, bulk, and git-source updates are gated the same way; rollbacks and installs are deliberately not. The gate is observational only and can be tuned or disabled per node under host alert settings. * docs: document health-gated updates and rollback readiness New operator page covering the update readiness dialog, the post-update health gate and its settings, the rollback readiness disclosure, and classified failures, with cross-links from the atomic deployments and deploy progress pages. The API reference gains the readiness and health-gate endpoints, the healthGateId success field, and the failure classification schema on deploy and update error responses. * feat: withhold the success verdict while the health gate observes An update used to show a green Succeeded that a failed health gate then contradicted moments later. The deploy modal now reports Verifying health while the gate observes, shows success only when the gate passes, and makes a failed or unknown gate the headline result; success toasts soften to a verifying message while a gate runs. The mobile recovery card groups its actions behind one bottom-right Take action menu so it stays compact on a phone, with the classified cause still visible on the card. A successful image update now also counts as the last known-good marker in rollback readiness, and the docs gain screenshots of the readiness dialog, gate states, dossier section, and settings. * fix: harden log format strings and the env existence path check Log calls that interpolated the stack name into the console format string now use constant format strings with placeholder arguments, and envExists validates path containment inline at its filesystem access, matching the established patterns used elsewhere in the same files. * test: adapt deploy modal success specs to the post-deploy health gate The deploy feedback modal now withholds its success verdict while the health gate observes the new containers, showing "Verifying health" until the gate passes. The two success-path E2E tests waited for "Succeeded" within the gate's 90s default window and timed out. Shorten the observation window to the 15s minimum for these tests via the settings API, assert the verify-then-succeed sequence the modal actually renders, and restore the default window afterward so the test value does not leak into later runs. * fix: serialize health gate polling and harden gate observation Address race conditions in the post-update health gate found in review. Backend: the gate poller used setInterval, so a Docker observe slower than the 5s tick could overlap the next poll and corrupt the restart and missing-container accounting, and a wedged socket could leave a poll pending forever. Polling is now single-flight: each cycle self-schedules the next only after it settles, and the observe is bounded by an 8s timeout so a hung probe counts as a poll error and resolves the gate unknown after three in a row. Frontend: the gate poller could overlap requests, letting a slow earlier "observing" response overwrite an already-applied terminal verdict. It is now single-flight with a terminal latch, so a late response can never roll the UI back from passed or failed. Also reject a non-digit nodeId on the snapshot coverage route instead of letting parseInt coerce it, document that turning off the deploy progress panel opts out of the live gate UI while the gate still runs server-side, and add gate-coverage tests for the webhook, git source, and auto-update apply paths plus the new single-flight, observe-timeout, and recovery cases.
155 lines
5.7 KiB
TypeScript
155 lines
5.7 KiB
TypeScript
import { describe, it, expect } from 'vitest';
|
|
import { classifyFailure } from '../services/updateGuard/failureClassifier';
|
|
import type { FailureReason } from '../services/updateGuard/types';
|
|
|
|
describe('classifyFailure', () => {
|
|
const cases: Array<{ name: string; message: string; reason: FailureReason }> = [
|
|
{
|
|
name: 'container crash sentinel',
|
|
message: 'CONTAINER_CRASHED\nExit Code: 137\nContainer exited after deployment. Check container logs for details.',
|
|
reason: 'container_exited',
|
|
},
|
|
{
|
|
name: 'idle stall sentinel',
|
|
message: 'STACK_STALLED_OUTPUT: no output for 600s',
|
|
reason: 'unknown',
|
|
},
|
|
{
|
|
name: 'docker daemon down (message)',
|
|
message: 'Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?',
|
|
reason: 'node_unreachable',
|
|
},
|
|
{
|
|
name: 'docker desktop connect error',
|
|
message: 'error during connect: this error may indicate that the docker daemon is not running',
|
|
reason: 'node_unreachable',
|
|
},
|
|
{
|
|
name: 'required variable missing',
|
|
message: 'error while interpolating services.web.environment.TOKEN: required variable REQ_TOKEN is missing a value: must be provided',
|
|
reason: 'env_missing',
|
|
},
|
|
{
|
|
name: 'variable not set (logfmt warning escalated)',
|
|
message: 'level=warning msg="The \\"DB_HOST\\" variable is not set. Defaulting to a blank string."',
|
|
reason: 'env_missing',
|
|
},
|
|
{
|
|
name: 'env file not found',
|
|
message: "env file /compose/app/.env not found: stat /compose/app/.env: no such file or directory",
|
|
reason: 'env_missing',
|
|
},
|
|
{
|
|
name: 'pull access denied',
|
|
message: 'Error response from daemon: pull access denied for ghost/missing, repository does not exist or may require docker login',
|
|
reason: 'image_pull_failed',
|
|
},
|
|
{
|
|
name: 'manifest unknown',
|
|
message: 'manifest unknown: manifest unknown',
|
|
reason: 'image_pull_failed',
|
|
},
|
|
{
|
|
name: 'registry rate limited',
|
|
message: 'toomanyrequests: You have reached your pull rate limit.',
|
|
reason: 'image_pull_failed',
|
|
},
|
|
{
|
|
name: 'port already allocated',
|
|
message: 'Error response from daemon: driver failed programming external connectivity: Bind for 0.0.0.0:8080 failed: port is already allocated',
|
|
reason: 'port_conflict',
|
|
},
|
|
{
|
|
name: 'address already in use',
|
|
message: 'listen tcp4 0.0.0.0:443: bind: address already in use',
|
|
reason: 'port_conflict',
|
|
},
|
|
{
|
|
name: 'windows ports not available',
|
|
message: 'ports are not available: exposing port TCP 0.0.0.0:5432 -> 0.0.0.0:0',
|
|
reason: 'port_conflict',
|
|
},
|
|
{
|
|
name: 'bind source missing',
|
|
message: 'Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /srv/missing',
|
|
reason: 'bind_path_missing',
|
|
},
|
|
{
|
|
name: 'permission denied',
|
|
message: 'open /compose/app/data: permission denied',
|
|
reason: 'permission_denied',
|
|
},
|
|
{
|
|
name: 'unhealthy dependency',
|
|
message: 'dependency failed to start: container app-db-1 is unhealthy',
|
|
reason: 'healthcheck_failed',
|
|
},
|
|
{
|
|
name: 'dependency exited',
|
|
message: 'dependency failed to start: container app-db-1 exited (1)',
|
|
reason: 'dependency_unavailable',
|
|
},
|
|
{
|
|
name: 'external network missing',
|
|
message: 'network proxy_net declared as external, but could not be found',
|
|
reason: 'dependency_unavailable',
|
|
},
|
|
{
|
|
name: 'yaml syntax error',
|
|
message: 'yaml: line 14: mapping values are not allowed in this context',
|
|
reason: 'compose_render_failed',
|
|
},
|
|
{
|
|
name: 'undefined volume',
|
|
message: 'service "web" refers to undefined volume data: invalid compose project',
|
|
reason: 'compose_render_failed',
|
|
},
|
|
{
|
|
name: 'unmatched output falls back to unknown',
|
|
message: 'something completely unexpected happened',
|
|
reason: 'unknown',
|
|
},
|
|
];
|
|
|
|
it.each(cases)('classifies $name as $reason', ({ message, reason }) => {
|
|
const result = classifyFailure(message);
|
|
expect(result.reason).toBe(reason);
|
|
expect(result.label.length).toBeGreaterThan(0);
|
|
expect(result.suggestion.length).toBeGreaterThan(0);
|
|
});
|
|
|
|
it('short-circuits to node_unreachable when the route flags a dead daemon', () => {
|
|
expect(classifyFailure('arbitrary text', { dockerUnavailable: true }).reason).toBe('node_unreachable');
|
|
});
|
|
|
|
it('prefers env_missing over compose_render_failed when a render fails on a missing variable', () => {
|
|
const result = classifyFailure(
|
|
'invalid compose project: required variable DB_PASS is missing a value',
|
|
);
|
|
expect(result.reason).toBe('env_missing');
|
|
});
|
|
|
|
it('prefers the crash sentinel over any other pattern in the same output', () => {
|
|
const result = classifyFailure(
|
|
'CONTAINER_CRASHED\nExit Code: 1\nport is already allocated',
|
|
);
|
|
expect(result.reason).toBe('container_exited');
|
|
});
|
|
|
|
it('classifies the underlying cause of a rolled-back update from its message', () => {
|
|
// ComposeRollbackError copies the original error message, so the route can
|
|
// pass getErrorMessage(error) unchanged for wrapped failures.
|
|
const causeMessage = 'pull access denied for private/app';
|
|
expect(classifyFailure(causeMessage).reason).toBe('image_pull_failed');
|
|
});
|
|
|
|
it('always returns a classification (total over arbitrary input)', () => {
|
|
for (const message of ['', ' ', 'x'.repeat(10_000)]) {
|
|
const result = classifyFailure(message);
|
|
expect(result.reason).toBeDefined();
|
|
expect(result.label).toBeDefined();
|
|
expect(result.suggestion).toBeDefined();
|
|
}
|
|
});
|
|
});
|