mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-20 15:22:59 +00:00
feat: health-gated updates and rollback readiness (#1354)
* feat: classify stack deploy and update failures with suggested next actions Failed deploy and update responses now carry a failure classification (cause category, headline, and suggested next step) derived from the compose error output. The recovery panel and chip render the classification and include it in copied diagnostics, and gateway-style failures surface as a node-unreachable cause. * feat: add update and rollback readiness reports for stacks Before a manual update, Sencho now shows an advisory readiness verdict computed from the stored preflight result, open drift findings, live container health, the pending image change, the rollback backup slot, and node disk headroom. The Stack Dossier gains a rollback readiness section that states what a rollback can restore and explicitly discloses that volume and bind-mounted data are not covered. Toolbar and sidebar updates now share one update path, and admins can create a fleet snapshot from the readiness dialog before updating. Nodes that do not advertise the capability keep the direct update flow. * feat: observe stack health after updates with a post-deploy health gate After a deploy or update succeeds, Sencho now watches the stack for a configurable observation window and records a passed, failed, or unknown verdict: containers must stay running, healthchecks must report healthy, and restart loops or disappearing containers fail the gate. The deploy panel shows the observation live and holds off auto-closing until the verdict lands, a failed gate surfaces the existing recovery actions including rollback, and the stack timeline records update started and gate verdict events. Scheduled, webhook, bulk, and git-source updates are gated the same way; rollbacks and installs are deliberately not. The gate is observational only and can be tuned or disabled per node under host alert settings. * docs: document health-gated updates and rollback readiness New operator page covering the update readiness dialog, the post-update health gate and its settings, the rollback readiness disclosure, and classified failures, with cross-links from the atomic deployments and deploy progress pages. The API reference gains the readiness and health-gate endpoints, the healthGateId success field, and the failure classification schema on deploy and update error responses. * feat: withhold the success verdict while the health gate observes An update used to show a green Succeeded that a failed health gate then contradicted moments later. The deploy modal now reports Verifying health while the gate observes, shows success only when the gate passes, and makes a failed or unknown gate the headline result; success toasts soften to a verifying message while a gate runs. The mobile recovery card groups its actions behind one bottom-right Take action menu so it stays compact on a phone, with the classified cause still visible on the card. A successful image update now also counts as the last known-good marker in rollback readiness, and the docs gain screenshots of the readiness dialog, gate states, dossier section, and settings. * fix: harden log format strings and the env existence path check Log calls that interpolated the stack name into the console format string now use constant format strings with placeholder arguments, and envExists validates path containment inline at its filesystem access, matching the established patterns used elsewhere in the same files. * test: adapt deploy modal success specs to the post-deploy health gate The deploy feedback modal now withholds its success verdict while the health gate observes the new containers, showing "Verifying health" until the gate passes. The two success-path E2E tests waited for "Succeeded" within the gate's 90s default window and timed out. Shorten the observation window to the 15s minimum for these tests via the settings API, assert the verify-then-succeed sequence the modal actually renders, and restore the default window afterward so the test value does not leak into later runs. * fix: serialize health gate polling and harden gate observation Address race conditions in the post-update health gate found in review. Backend: the gate poller used setInterval, so a Docker observe slower than the 5s tick could overlap the next poll and corrupt the restart and missing-container accounting, and a wedged socket could leave a poll pending forever. Polling is now single-flight: each cycle self-schedules the next only after it settles, and the observe is bounded by an 8s timeout so a hung probe counts as a poll error and resolves the gate unknown after three in a row. Frontend: the gate poller could overlap requests, letting a slow earlier "observing" response overwrite an already-applied terminal verdict. It is now single-flight with a terminal latch, so a late response can never roll the UI back from passed or failed. Also reject a non-digit nodeId on the snapshot coverage route instead of letting parseInt coerce it, document that turning off the deploy progress panel opts out of the live gate UI while the gate still runs server-side, and add gate-coverage tests for the webhook, git source, and auto-update apply paths plus the new single-flight, observe-timeout, and recovery cases.
This commit is contained in:
Anso
@@ -72,6 +72,17 @@ export type StackAction =
|
||||
*/
|
||||
export type RecoverableAction = Extract<StackAction, 'deploy' | 'update' | 'restart' | 'rollback'>;
|
||||
|
||||
/**
|
||||
* Server-side classification of a failed deploy/update: a cause headline and a
|
||||
* suggested next step. `reason` stays a plain string here; the backend owns the
|
||||
* category vocabulary and the UI only displays it.
|
||||
*/
|
||||
export interface FailureClassification {
|
||||
reason: string;
|
||||
label: string;
|
||||
suggestion: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Terminal record of a failed stack operation, kept in memory per stack so the
|
||||
* recovery panel can offer safe next steps after an update/deploy fails or
|
||||
@@ -88,6 +99,9 @@ export interface StackActionResult {
|
||||
// session was streaming this stack at failure time; omitted otherwise so a
|
||||
// line from another stack/session never leaks into diagnostics.
|
||||
lastOutputLine?: string;
|
||||
// Classified cause + suggested next action from the failed response body,
|
||||
// when the backend (or the unreachable-node fallback) provided one.
|
||||
failure?: FailureClassification;
|
||||
}
|
||||
|
||||
export interface ContainerStatsEntry {
|
||||
|
||||
@@ -22,6 +22,18 @@ interface RecoveryActionsProps {
|
||||
variant?: 'inline' | 'list';
|
||||
}
|
||||
|
||||
// The classified cause + suggested next step, rendered by both recovery
|
||||
// surfaces above their action sets. Nothing renders without a classification.
|
||||
export function RecoveryClassification({ result }: { result: StackActionResult }) {
|
||||
if (!result.failure) return null;
|
||||
return (
|
||||
<div data-testid="recovery-classification">
|
||||
<p className="text-xs font-medium text-foreground">{result.failure.label}</p>
|
||||
<p className="mt-0.5 text-xs text-muted-foreground">{result.failure.suggestion}</p>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
// The recovery action set shared by the mobile inline panel and the desktop
|
||||
// chip popover, so retry/restart/rollback/refresh/copy have one implementation.
|
||||
export function RecoveryActions({
|
||||
|
||||
@@ -2,7 +2,7 @@ import { useState } from 'react';
|
||||
import { AlertTriangle, ChevronDown, X } from 'lucide-react';
|
||||
import { Popover, PopoverContent, PopoverTrigger } from '@/components/ui/popover';
|
||||
import { Button } from '../ui/button';
|
||||
import { RecoveryActions } from './RecoveryActions';
|
||||
import { RecoveryActions, RecoveryClassification } from './RecoveryActions';
|
||||
import { capitalize, formatElapsed } from './recovery-format';
|
||||
import type { Node } from '@/context/NodeContext';
|
||||
import type { StackActionResult } from './EditorView';
|
||||
@@ -64,6 +64,9 @@ export function RecoveryChip({
|
||||
{result.errorMessage ?? 'The operation did not complete.'}
|
||||
{result.rolledBack && ' · rolled back to previous version'}
|
||||
</p>
|
||||
<div className="mt-1.5">
|
||||
<RecoveryClassification result={result} />
|
||||
</div>
|
||||
</div>
|
||||
<div className="border-t border-glass-border p-1">
|
||||
<RecoveryActions
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
import { AlertTriangle, X } from 'lucide-react';
|
||||
import { useState } from 'react';
|
||||
import { AlertTriangle, ChevronDown, X } from 'lucide-react';
|
||||
import { Popover, PopoverContent, PopoverTrigger } from '@/components/ui/popover';
|
||||
import { Button } from '../ui/button';
|
||||
import { RecoveryActions } from './RecoveryActions';
|
||||
import { RecoveryActions, RecoveryClassification } from './RecoveryActions';
|
||||
import { capitalize, formatElapsed } from './recovery-format';
|
||||
import type { Node } from '@/context/NodeContext';
|
||||
import type { StackActionResult } from './EditorView';
|
||||
@@ -23,7 +25,9 @@ interface RecoveryPanelProps {
|
||||
// update/deploy/restart/rollback fails or stalls. Styled as a quiet card with a
|
||||
// thin destructive rail (the toast accent language) so it blends with the
|
||||
// surrounding detail rather than shouting; the desktop surface uses RecoveryChip
|
||||
// instead. The full failure output stays in the deploy modal.
|
||||
// instead. The error and the classified cause stay visible on the card; the
|
||||
// actions collapse behind one Take action menu so the card stays small on a
|
||||
// phone. The full failure output stays in the deploy modal.
|
||||
export function RecoveryPanel({
|
||||
stackName,
|
||||
result,
|
||||
@@ -36,6 +40,7 @@ export function RecoveryPanel({
|
||||
onRefreshState,
|
||||
onDismiss,
|
||||
}: RecoveryPanelProps) {
|
||||
const [actionsOpen, setActionsOpen] = useState(false);
|
||||
const elapsed = formatElapsed(result.endedAt - result.startedAt);
|
||||
|
||||
return (
|
||||
@@ -60,6 +65,9 @@ export function RecoveryPanel({
|
||||
{result.errorMessage ?? 'The operation did not complete.'}
|
||||
{result.rolledBack && ' · rolled back to previous version'}
|
||||
</p>
|
||||
<div className="mt-1.5">
|
||||
<RecoveryClassification result={result} />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<Button
|
||||
@@ -74,18 +82,29 @@ export function RecoveryPanel({
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
<div className="mt-2.5 pl-2">
|
||||
<RecoveryActions
|
||||
stackName={stackName}
|
||||
result={result}
|
||||
activeNode={activeNode}
|
||||
backupInfo={backupInfo}
|
||||
canDeploy={canDeploy}
|
||||
onRetry={onRetry}
|
||||
onRestart={onRestart}
|
||||
onRollback={onRollback}
|
||||
onRefreshState={onRefreshState}
|
||||
/>
|
||||
<div className="mt-2.5 flex justify-end pl-2">
|
||||
<Popover open={actionsOpen} onOpenChange={setActionsOpen}>
|
||||
<PopoverTrigger asChild>
|
||||
<Button variant="outline" size="sm" className="h-8 gap-1.5 text-xs">
|
||||
Take action
|
||||
<ChevronDown className="h-3 w-3 opacity-70" />
|
||||
</Button>
|
||||
</PopoverTrigger>
|
||||
<PopoverContent align="end" className="w-60 p-1" data-testid="recovery-actions-menu">
|
||||
<RecoveryActions
|
||||
variant="list"
|
||||
stackName={stackName}
|
||||
result={result}
|
||||
activeNode={activeNode}
|
||||
backupInfo={backupInfo}
|
||||
canDeploy={canDeploy}
|
||||
onRetry={onRetry}
|
||||
onRestart={onRestart}
|
||||
onRollback={onRollback}
|
||||
onRefreshState={onRefreshState}
|
||||
/>
|
||||
</PopoverContent>
|
||||
</Popover>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
|
||||
@@ -2,6 +2,7 @@ import { lazy, Suspense } from 'react';
|
||||
import BashExecModal from '../BashExecModal';
|
||||
import LazyBoundary from '../LazyBoundary';
|
||||
import { PolicyBlockDialog } from '../stack/PolicyBlockDialog';
|
||||
import { UpdateReadinessDialog } from '../stack/UpdateReadinessDialog';
|
||||
import { DeleteStackDialog } from './DeleteStackDialog';
|
||||
import { UnsavedChangesDialog } from './UnsavedChangesDialog';
|
||||
import { StackAlertSheet } from '../StackAlertSheet';
|
||||
@@ -55,6 +56,7 @@ export function ShellOverlays({
|
||||
logViewerOpen, logContainer,
|
||||
stackMonitor, closeStackMonitor,
|
||||
policyBlock, setPolicyBlock, policyBypassing,
|
||||
updateReadiness, setUpdateReadiness,
|
||||
stackMisconfigScanId, setStackMisconfigScanId,
|
||||
diffPreview, setDiffPreview, diffPreviewConfirming, setDiffPreviewConfirming,
|
||||
} = overlayState;
|
||||
@@ -102,6 +104,14 @@ export function ShellOverlays({
|
||||
initialTab={stackMonitor?.tab ?? 'alerts'}
|
||||
/>
|
||||
|
||||
{/* Pre-update readiness check */}
|
||||
<UpdateReadinessDialog
|
||||
open={updateReadiness !== null}
|
||||
stackName={updateReadiness?.stackName ?? ''}
|
||||
onCancel={() => setUpdateReadiness(null)}
|
||||
onProceed={() => updateReadiness?.proceed()}
|
||||
/>
|
||||
|
||||
{/* Pre-deploy policy block */}
|
||||
<PolicyBlockDialog
|
||||
open={policyBlock !== null}
|
||||
|
||||
@@ -34,23 +34,37 @@ function setup(over: Partial<Parameters<typeof RecoveryPanel>[0]> = {}) {
|
||||
return props;
|
||||
}
|
||||
|
||||
// The actions live behind the Take action menu so the mobile card stays small.
|
||||
function openActions() {
|
||||
fireEvent.click(screen.getByText('Take action'));
|
||||
}
|
||||
|
||||
describe('RecoveryPanel', () => {
|
||||
beforeEach(() => vi.clearAllMocks());
|
||||
|
||||
it('shows the failed action title and error message', () => {
|
||||
it('shows the failed action title and error message on the card', () => {
|
||||
setup();
|
||||
expect(screen.getByText(/Update failed/)).toBeInTheDocument();
|
||||
expect(screen.getByText(/pull failed: connection reset/)).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('calls onRetry from the retry button', () => {
|
||||
it('keeps the actions collapsed behind the Take action menu', () => {
|
||||
setup();
|
||||
expect(screen.queryByText('Retry update')).not.toBeInTheDocument();
|
||||
openActions();
|
||||
expect(screen.getByText('Retry update')).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('calls onRetry from the retry action', () => {
|
||||
const props = setup();
|
||||
openActions();
|
||||
fireEvent.click(screen.getByText('Retry update'));
|
||||
expect(props.onRetry).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('hides retry/restart/rollback without the deploy permission', () => {
|
||||
setup({ canDeploy: false, backupInfo: { exists: true, timestamp: 1 } });
|
||||
openActions();
|
||||
expect(screen.queryByText('Retry update')).not.toBeInTheDocument();
|
||||
expect(screen.queryByText('Restart')).not.toBeInTheDocument();
|
||||
expect(screen.queryByText('Roll back')).not.toBeInTheDocument();
|
||||
@@ -61,19 +75,23 @@ describe('RecoveryPanel', () => {
|
||||
|
||||
it('offers rollback only when a backup exists', () => {
|
||||
setup({ backupInfo: { exists: false, timestamp: null } });
|
||||
openActions();
|
||||
expect(screen.queryByText('Roll back')).not.toBeInTheDocument();
|
||||
setup({ backupInfo: { exists: true, timestamp: 123 } });
|
||||
fireEvent.click(screen.getAllByText('Take action')[1]);
|
||||
expect(screen.getByText('Roll back')).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('does not show a redundant restart button when the failed action was a restart', () => {
|
||||
setup({ result: { ...baseResult, action: 'restart' } });
|
||||
openActions();
|
||||
expect(screen.getByText('Retry restart')).toBeInTheDocument();
|
||||
expect(screen.queryByText('Restart')).not.toBeInTheDocument();
|
||||
expect(screen.queryByText('Restart', { exact: true })).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('copies session-safe diagnostics including stack and error', () => {
|
||||
setup({ result: { ...baseResult, lastOutputLine: 'pulling app ...' } });
|
||||
openActions();
|
||||
fireEvent.click(screen.getByText('Copy details'));
|
||||
expect(copyToClipboard).toHaveBeenCalledTimes(1);
|
||||
const blob = vi.mocked(copyToClipboard).mock.calls[0][0];
|
||||
@@ -82,8 +100,47 @@ describe('RecoveryPanel', () => {
|
||||
expect(blob).toContain('Last output: pulling app ...');
|
||||
});
|
||||
|
||||
it('renders the failure classification on the card without opening the menu', () => {
|
||||
setup({
|
||||
result: {
|
||||
...baseResult,
|
||||
failure: {
|
||||
reason: 'port_conflict',
|
||||
label: 'Host port conflict',
|
||||
suggestion: 'Free the conflicting host port, then retry.',
|
||||
},
|
||||
},
|
||||
});
|
||||
expect(screen.getByText('Host port conflict')).toBeInTheDocument();
|
||||
expect(screen.getByText('Free the conflicting host port, then retry.')).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('renders no classification block without a failure field', () => {
|
||||
setup();
|
||||
expect(screen.queryByTestId('recovery-classification')).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('includes the classification in copied diagnostics', () => {
|
||||
setup({
|
||||
result: {
|
||||
...baseResult,
|
||||
failure: {
|
||||
reason: 'env_missing',
|
||||
label: 'Missing environment variable',
|
||||
suggestion: 'Define the missing variable, then retry.',
|
||||
},
|
||||
},
|
||||
});
|
||||
openActions();
|
||||
fireEvent.click(screen.getByText('Copy details'));
|
||||
const blob = vi.mocked(copyToClipboard).mock.calls[0][0];
|
||||
expect(blob).toContain('Classified: Missing environment variable');
|
||||
expect(blob).toContain('Suggestion: Define the missing variable, then retry.');
|
||||
});
|
||||
|
||||
it('wires refresh and dismiss callbacks', () => {
|
||||
const props = setup();
|
||||
openActions();
|
||||
fireEvent.click(screen.getByText('Refresh'));
|
||||
fireEvent.click(screen.getByLabelText('Dismiss recovery panel'));
|
||||
expect(props.onRefreshState).toHaveBeenCalledTimes(1);
|
||||
|
||||
@@ -88,6 +88,14 @@ export function useOverlayState() {
|
||||
const [policyBlock, setPolicyBlock] = useState<PolicyBlock | null>(null);
|
||||
const [policyBypassing, setPolicyBypassing] = useState(false);
|
||||
|
||||
// Pre-update readiness dialog. `proceed` runs the actual update when the
|
||||
// user confirms; opened by useStackActions.requestStackUpdate.
|
||||
const [updateReadiness, setUpdateReadiness] = useState<{
|
||||
stackName: string;
|
||||
stackFile: string;
|
||||
proceed: () => void;
|
||||
} | null>(null);
|
||||
|
||||
const [stackMisconfigScanId, setStackMisconfigScanId] = useState<number | null>(null);
|
||||
|
||||
const [diffPreview, setDiffPreview] = useState<DiffPreview | null>(null);
|
||||
@@ -103,6 +111,7 @@ export function useOverlayState() {
|
||||
logViewerOpen, logContainer, openLogViewer, closeLogViewer,
|
||||
stackMonitor, openAlertSheet, openAutoHeal, closeStackMonitor,
|
||||
policyBlock, setPolicyBlock, policyBypassing, setPolicyBypassing,
|
||||
updateReadiness, setUpdateReadiness,
|
||||
stackMisconfigScanId, setStackMisconfigScanId,
|
||||
diffPreview, setDiffPreview, diffPreviewConfirming, setDiffPreviewConfirming,
|
||||
} as const;
|
||||
|
||||
@@ -83,6 +83,8 @@ function makeOverlay(over: Partial<OverlayState> = {}): OverlayState {
|
||||
policyBlock: null,
|
||||
setPolicyBlock: vi.fn(),
|
||||
setPolicyBypassing: vi.fn(),
|
||||
updateReadiness: null,
|
||||
setUpdateReadiness: vi.fn(),
|
||||
setDiffPreview: vi.fn(),
|
||||
...over,
|
||||
} as unknown as OverlayState;
|
||||
@@ -96,6 +98,7 @@ function setup(over: {
|
||||
overlay?: Partial<OverlayState>;
|
||||
stackList?: Partial<StackListState>;
|
||||
getLastDeployOutputLine?: (stackName: string) => string | undefined;
|
||||
hasUpdateGuard?: boolean;
|
||||
} = {}) {
|
||||
const editorState = makeEditorState(over.editorState);
|
||||
const stackListState = makeStackListState(over.stackList);
|
||||
@@ -114,6 +117,7 @@ function setup(over: {
|
||||
runWithLog,
|
||||
getLastDeployOutputLine: over.getLastDeployOutputLine ?? (() => undefined),
|
||||
diffPreviewEnabled: false,
|
||||
hasUpdateGuard: over.hasUpdateGuard ?? false,
|
||||
}),
|
||||
);
|
||||
return { result, editorState, stackListState, overlayState };
|
||||
@@ -365,6 +369,75 @@ describe('useStackActions.attemptLeaveEditor (mobile back / nav guard)', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('useStackActions update readiness routing', () => {
|
||||
beforeEach(() => {
|
||||
vi.mocked(apiFetch).mockReset();
|
||||
});
|
||||
|
||||
function routeUpdateOk() {
|
||||
vi.mocked(apiFetch).mockImplementation((url: string) => {
|
||||
const u = String(url);
|
||||
if (u.includes('/update')) return Promise.resolve(new Response('', { status: 200 }));
|
||||
return Promise.resolve(new Response('[]', { status: 200 }));
|
||||
});
|
||||
}
|
||||
|
||||
it('opens the readiness dialog instead of posting when the node has update-guard', async () => {
|
||||
routeUpdateOk();
|
||||
const { result, overlayState } = setup({ hasUpdateGuard: true });
|
||||
await act(async () => { await result.current.updateStack(); });
|
||||
expect(overlayState.setUpdateReadiness).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ stackName: 'web', stackFile: 'web.yml' }),
|
||||
);
|
||||
expect(apiFetch).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('routes a sidebar/context-menu update through the readiness dialog too', async () => {
|
||||
routeUpdateOk();
|
||||
const { result, overlayState } = setup({ hasUpdateGuard: true });
|
||||
await act(async () => { await result.current.executeStackActionByFile('web.yml', 'update', 'update'); });
|
||||
expect(overlayState.setUpdateReadiness).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ stackName: 'web', stackFile: 'web.yml' }),
|
||||
);
|
||||
expect(apiFetch).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('runs the shared update executor when the dialog proceeds', async () => {
|
||||
routeUpdateOk();
|
||||
const { result, overlayState, stackListState } = setup({ hasUpdateGuard: true });
|
||||
await act(async () => { await result.current.updateStack(); });
|
||||
const pending = vi.mocked(overlayState.setUpdateReadiness).mock.calls[0][0] as
|
||||
{ stackName: string; stackFile: string; proceed: () => void };
|
||||
expect(pending).not.toBeNull();
|
||||
await act(async () => { pending.proceed(); });
|
||||
expect(overlayState.setUpdateReadiness).toHaveBeenLastCalledWith(null);
|
||||
const urls = vi.mocked(apiFetch).mock.calls.map(c => String(c[0]));
|
||||
expect(urls).toContain('/stacks/web/update');
|
||||
expect(stackListState.recordActionSuccess).toHaveBeenCalledWith('web.yml');
|
||||
});
|
||||
|
||||
it('does nothing while the stack is busy, with or without the dialog', async () => {
|
||||
routeUpdateOk();
|
||||
const { result, overlayState } = setup({
|
||||
hasUpdateGuard: true,
|
||||
stackList: { isStackBusy: vi.fn().mockReturnValue(true) as never },
|
||||
});
|
||||
await act(async () => { await result.current.updateStack(); });
|
||||
expect(overlayState.setUpdateReadiness).not.toHaveBeenCalled();
|
||||
expect(apiFetch).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('updates directly without the capability, from both entry points', async () => {
|
||||
routeUpdateOk();
|
||||
const { result, overlayState } = setup();
|
||||
await act(async () => { await result.current.updateStack(); });
|
||||
await act(async () => { await result.current.executeStackActionByFile('web.yml', 'update', 'update'); });
|
||||
expect(overlayState.setUpdateReadiness).not.toHaveBeenCalled();
|
||||
const updatePosts = vi.mocked(apiFetch).mock.calls.filter(c => String(c[0]) === '/stacks/web/update');
|
||||
expect(updatePosts).toHaveLength(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe('useStackActions recovery records', () => {
|
||||
beforeEach(() => {
|
||||
vi.mocked(apiFetch).mockReset();
|
||||
@@ -453,6 +526,67 @@ describe('useStackActions recovery records', () => {
|
||||
);
|
||||
});
|
||||
|
||||
it('carries the server failure classification into the recovery record', async () => {
|
||||
const body = JSON.stringify({
|
||||
error: 'port is already allocated',
|
||||
rolledBack: false,
|
||||
failure: { reason: 'port_conflict', label: 'Host port conflict', suggestion: 'Free the port, then retry.' },
|
||||
});
|
||||
routeApi(500, body);
|
||||
const { result, stackListState } = setup();
|
||||
await act(async () => { await result.current.updateStack(); });
|
||||
expect(stackListState.recordActionFailure).toHaveBeenCalledWith(
|
||||
'web.yml',
|
||||
expect.objectContaining({
|
||||
failure: { reason: 'port_conflict', label: 'Host port conflict', suggestion: 'Free the port, then retry.' },
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it('ignores a malformed failure field in the response body', async () => {
|
||||
routeApi(500, JSON.stringify({ error: 'boom', failure: { reason: 42 } }));
|
||||
const { result, stackListState } = setup();
|
||||
await act(async () => { await result.current.updateStack(); });
|
||||
expect(stackListState.recordActionFailure).toHaveBeenCalledWith(
|
||||
'web.yml',
|
||||
expect.objectContaining({ failure: undefined }),
|
||||
);
|
||||
});
|
||||
|
||||
it('synthesizes a node_unreachable classification for a gateway 502 with no body', async () => {
|
||||
routeApi(502, 'Bad Gateway');
|
||||
const { result, stackListState } = setup();
|
||||
await act(async () => { await result.current.updateStack(); });
|
||||
expect(stackListState.recordActionFailure).toHaveBeenCalledWith(
|
||||
'web.yml',
|
||||
expect.objectContaining({
|
||||
failure: expect.objectContaining({ reason: 'node_unreachable' }),
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it('does not mislabel an unrelated JSON 503 as node_unreachable', async () => {
|
||||
routeApi(503, JSON.stringify({ error: 'maintenance window' }));
|
||||
const { result, stackListState } = setup();
|
||||
await act(async () => { await result.current.updateStack(); });
|
||||
expect(stackListState.recordActionFailure).toHaveBeenCalledWith(
|
||||
'web.yml',
|
||||
expect.objectContaining({ failure: undefined }),
|
||||
);
|
||||
});
|
||||
|
||||
it('synthesizes node_unreachable for a docker_unavailable 503 without a classified body', async () => {
|
||||
routeApi(503, JSON.stringify({ error: 'daemon gone', code: 'docker_unavailable' }));
|
||||
const { result, stackListState } = setup();
|
||||
await act(async () => { await result.current.updateStack(); });
|
||||
expect(stackListState.recordActionFailure).toHaveBeenCalledWith(
|
||||
'web.yml',
|
||||
expect.objectContaining({
|
||||
failure: expect.objectContaining({ reason: 'node_unreachable' }),
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it('records a rollback failure', async () => {
|
||||
vi.mocked(apiFetch).mockImplementation((url: string) => {
|
||||
const u = String(url);
|
||||
|
||||
@@ -7,7 +7,7 @@ import type { useViewNavigationState } from './useViewNavigationState';
|
||||
import type { OverlayState } from './useOverlayState';
|
||||
import type { Node } from '@/context/NodeContext';
|
||||
import type { ActionVerb } from '@/context/DeployFeedbackContext';
|
||||
import type { StackAction, RecoverableAction } from '../EditorView';
|
||||
import type { StackAction, RecoverableAction, FailureClassification } from '../EditorView';
|
||||
import type { NotificationItem } from '../../dashboard/types';
|
||||
import type { PolicyBlockPayload, PolicyBlockableAction } from '../../stack/PolicyBlockDialog';
|
||||
|
||||
@@ -15,15 +15,57 @@ interface RunResult {
|
||||
ok: boolean;
|
||||
errorMessage?: string;
|
||||
rolledBack?: boolean;
|
||||
/** Health gate run id from the success body, when the backend started one. */
|
||||
healthGateId?: string | null;
|
||||
}
|
||||
|
||||
/** healthGateId from a success body, or null when absent or unreadable. */
|
||||
const parseHealthGateId = async (response: Response): Promise<string | null> => {
|
||||
try {
|
||||
const body: unknown = await response.json();
|
||||
if (isRecord(body) && typeof body.healthGateId === 'string') return body.healthGateId;
|
||||
} catch (e) {
|
||||
// A success body should always parse; the warn surfaces a future
|
||||
// double-read bug instead of silently disabling the gate UI.
|
||||
console.warn('[HealthGate] could not read the success body:', e);
|
||||
}
|
||||
return null;
|
||||
};
|
||||
|
||||
// Sentinel stored in overlayState.pendingUnsavedLoad to mark that the pending
|
||||
// confirmation is a node switch (not a stack load). When the user confirms the
|
||||
// discard, discardAndLoadPending calls setActiveNode(targetNode) and skips the
|
||||
// stack-load branch.
|
||||
export const NODE_SWITCH_PENDING_TOKEN = '__node-switch-pending__';
|
||||
|
||||
type StackActionError = Error & { rolledBack?: boolean };
|
||||
type StackActionError = Error & { rolledBack?: boolean; failure?: FailureClassification };
|
||||
|
||||
// Fallback classification when the response never reached a Sencho backend
|
||||
// (proxy 502/504 for a dead remote, or a 503 with no classified body).
|
||||
const NODE_UNREACHABLE_FAILURE: FailureClassification = {
|
||||
reason: 'node_unreachable',
|
||||
label: 'Node or Docker unreachable',
|
||||
suggestion: 'Check that the node is online and Docker is running, then retry.',
|
||||
};
|
||||
|
||||
const UNREACHABLE_STATUSES: ReadonlySet<number> = new Set([502, 503, 504]);
|
||||
|
||||
const parseFailureClassification = (value: unknown): FailureClassification | undefined => {
|
||||
if (
|
||||
isRecord(value) &&
|
||||
typeof value.reason === 'string' &&
|
||||
typeof value.label === 'string' && value.label.trim() &&
|
||||
typeof value.suggestion === 'string' && value.suggestion.trim()
|
||||
) {
|
||||
return { reason: value.reason, label: value.label, suggestion: value.suggestion };
|
||||
}
|
||||
if (value !== undefined) {
|
||||
// Likely hub/node version skew or a mangled proxy body; the raw error
|
||||
// message still renders, only the classification panel is degraded.
|
||||
console.warn('Unrecognized failure classification shape in error response:', value);
|
||||
}
|
||||
return undefined;
|
||||
};
|
||||
|
||||
type StackOpAction = 'deploy' | 'down' | 'restart' | 'stop' | 'start' | 'update';
|
||||
|
||||
@@ -66,6 +108,10 @@ interface UseStackActionsOptions {
|
||||
// is streaming that exact stack; used to enrich failure diagnostics safely.
|
||||
getLastDeployOutputLine: (stackName: string) => string | undefined;
|
||||
diffPreviewEnabled: boolean;
|
||||
// Active node advertises the update-guard capability, so manual updates show
|
||||
// the pre-update readiness dialog. Defaults to false: without the
|
||||
// capability, updates run directly with no dialog.
|
||||
hasUpdateGuard?: boolean;
|
||||
}
|
||||
|
||||
const isRecord = (value: unknown): value is Record<string, unknown> =>
|
||||
@@ -100,24 +146,40 @@ const stackOpInProgressMessage = (stackName: string, info: StackOpInProgressInfo
|
||||
return `${stackName} is already ${verb}${actor}.`;
|
||||
};
|
||||
|
||||
const parseStackActionError = (rawBody: string, fallback: string): StackActionError => {
|
||||
const parseStackActionError = (rawBody: string, fallback: string, status?: number): StackActionError => {
|
||||
let message = rawBody || fallback;
|
||||
let rolledBack = false;
|
||||
let failure: FailureClassification | undefined;
|
||||
let parsedCode: string | undefined;
|
||||
let bodyWasJson = false;
|
||||
|
||||
try {
|
||||
const parsed: unknown = JSON.parse(rawBody);
|
||||
bodyWasJson = true;
|
||||
if (isRecord(parsed)) {
|
||||
if (typeof parsed.error === 'string' && parsed.error.trim()) {
|
||||
message = parsed.error;
|
||||
}
|
||||
rolledBack = parsed.rolledBack === true;
|
||||
failure = parseFailureClassification(parsed.failure);
|
||||
if (typeof parsed.code === 'string') parsedCode = parsed.code;
|
||||
}
|
||||
} catch {
|
||||
/* not JSON */
|
||||
}
|
||||
|
||||
// A gateway-style status with no classified body means the request likely
|
||||
// never reached the owning node's backend; surface that as the cause. A 503
|
||||
// qualifies only when it is body-less (proxy generated) or the backend's own
|
||||
// docker_unavailable shape, so an unrelated future 503 is not mislabeled.
|
||||
if (!failure && status !== undefined && UNREACHABLE_STATUSES.has(status)) {
|
||||
const qualifies = status !== 503 || !bodyWasJson || parsedCode === 'docker_unavailable';
|
||||
if (qualifies) failure = { ...NODE_UNREACHABLE_FAILURE };
|
||||
}
|
||||
|
||||
const error = new Error(message) as StackActionError;
|
||||
error.rolledBack = rolledBack;
|
||||
error.failure = failure;
|
||||
return error;
|
||||
};
|
||||
|
||||
@@ -133,6 +195,7 @@ export function useStackActions(options: UseStackActionsOptions) {
|
||||
runWithLog,
|
||||
getLastDeployOutputLine,
|
||||
diffPreviewEnabled,
|
||||
hasUpdateGuard = false,
|
||||
} = options;
|
||||
|
||||
const pendingStackLoadRef = useRef<string | null>(null);
|
||||
@@ -235,6 +298,7 @@ export function useStackActions(options: UseStackActionsOptions) {
|
||||
startedAt: number,
|
||||
errorMessage: string | undefined,
|
||||
rolledBack: boolean,
|
||||
failure?: FailureClassification,
|
||||
) => {
|
||||
if (!isRecoverableAction(action)) return;
|
||||
stackListState.recordActionFailure(stackFile, {
|
||||
@@ -244,6 +308,7 @@ export function useStackActions(options: UseStackActionsOptions) {
|
||||
startedAt,
|
||||
endedAt: Date.now(),
|
||||
lastOutputLine: getLastDeployOutputLine(stackName),
|
||||
failure,
|
||||
});
|
||||
};
|
||||
|
||||
@@ -619,12 +684,17 @@ export function useStackActions(options: UseStackActionsOptions) {
|
||||
return { ok: false, errorMessage: message };
|
||||
}
|
||||
}
|
||||
throw parseStackActionError(rawBody, 'Deploy failed');
|
||||
throw parseStackActionError(rawBody, 'Deploy failed', response.status);
|
||||
}
|
||||
overlayState.setPolicyBlock(null);
|
||||
toast.success(
|
||||
ignorePolicy ? 'Stack deployed (policy bypassed).' : 'Stack deployed successfully!',
|
||||
);
|
||||
const healthGateId = await parseHealthGateId(response);
|
||||
// With a health gate observing, the operation finishing is not the
|
||||
// final verdict yet; soften the toast so success is not claimed twice.
|
||||
if (healthGateId) {
|
||||
toast.info(ignorePolicy ? 'Stack deployed (policy bypassed). Verifying health...' : 'Stack deployed. Verifying health...');
|
||||
} else {
|
||||
toast.success(ignorePolicy ? 'Stack deployed (policy bypassed).' : 'Stack deployed successfully!');
|
||||
}
|
||||
await refreshSelectedContainers(stackName, stackFile);
|
||||
try {
|
||||
const backupRes = await apiFetch(`/stacks/${stackName}/backup`);
|
||||
@@ -633,7 +703,7 @@ export function useStackActions(options: UseStackActionsOptions) {
|
||||
/* ignore */
|
||||
}
|
||||
stackListState.recordActionSuccess(stackFile);
|
||||
return { ok: true };
|
||||
return { ok: true, healthGateId };
|
||||
} catch (error) {
|
||||
console.error('Failed to deploy:', error);
|
||||
if (previousStatus !== undefined)
|
||||
@@ -645,7 +715,7 @@ export function useStackActions(options: UseStackActionsOptions) {
|
||||
? `${errorMessage} - automatically rolled back to previous version.`
|
||||
: errorMessage,
|
||||
);
|
||||
recordActionFailureFor(stackFile, stackName, 'deploy', startedAt, errorMessage, deployError.rolledBack === true);
|
||||
recordActionFailureFor(stackFile, stackName, 'deploy', startedAt, errorMessage, deployError.rolledBack === true, deployError.failure);
|
||||
await refreshSelectedContainers(stackName, stackFile);
|
||||
return { ok: false, errorMessage, rolledBack: deployError.rolledBack };
|
||||
}
|
||||
@@ -736,7 +806,7 @@ export function useStackActions(options: UseStackActionsOptions) {
|
||||
return;
|
||||
}
|
||||
}
|
||||
throw parseStackActionError(rawBody, 'Rollback failed');
|
||||
throw parseStackActionError(rawBody, 'Rollback failed', res.status);
|
||||
}
|
||||
overlayState.setPolicyBlock(null);
|
||||
toast.success('Stack rolled back successfully.');
|
||||
@@ -758,7 +828,8 @@ export function useStackActions(options: UseStackActionsOptions) {
|
||||
} catch (error: unknown) {
|
||||
const msg = error instanceof Error ? error.message : 'Rollback failed';
|
||||
toast.error(msg);
|
||||
recordActionFailureFor(stackFile, stackName, 'rollback', startedAt, msg, false);
|
||||
recordActionFailureFor(stackFile, stackName, 'rollback', startedAt, msg, false,
|
||||
error instanceof Error ? (error as StackActionError).failure : undefined);
|
||||
await refreshSelectedContainers(stackName, stackFile);
|
||||
} finally {
|
||||
stackListState.clearStackAction(stackFile);
|
||||
@@ -850,8 +921,8 @@ export function useStackActions(options: UseStackActionsOptions) {
|
||||
}
|
||||
}
|
||||
}
|
||||
const actionError = parseStackActionError(errText, `${action} failed`);
|
||||
recordActionFailureFor(stackFile, stackName, action, startedAt, actionError.message, actionError.rolledBack === true);
|
||||
const actionError = parseStackActionError(errText, `${action} failed`, response.status);
|
||||
recordActionFailureFor(stackFile, stackName, action, startedAt, actionError.message, actionError.rolledBack === true, actionError.failure);
|
||||
await refreshSelectedContainers(stackName, stackFile);
|
||||
return {
|
||||
ok: false as const,
|
||||
@@ -860,11 +931,18 @@ export function useStackActions(options: UseStackActionsOptions) {
|
||||
};
|
||||
}
|
||||
overlayState.setPolicyBlock(null);
|
||||
toast.success(successMessage);
|
||||
const healthGateId = await parseHealthGateId(response);
|
||||
// With a health gate observing, the operation finishing is not the
|
||||
// final verdict yet; soften the toast so success is not claimed twice.
|
||||
if (healthGateId && action === 'update') {
|
||||
toast.info('Stack updated. Verifying health...');
|
||||
} else {
|
||||
toast.success(successMessage);
|
||||
}
|
||||
if (action === 'update') stackListState.fetchImageUpdates();
|
||||
await refreshSelectedContainers(stackName, stackFile);
|
||||
stackListState.recordActionSuccess(stackFile);
|
||||
return { ok: true as const };
|
||||
return { ok: true as const, healthGateId };
|
||||
} catch (err) {
|
||||
const message = (err as Error).message || `${action} failed`;
|
||||
recordActionFailureFor(stackFile, stackName, action, startedAt, message, false);
|
||||
@@ -923,11 +1001,33 @@ export function useStackActions(options: UseStackActionsOptions) {
|
||||
}
|
||||
};
|
||||
|
||||
// Single entry point for every manual update trigger (toolbar, sidebar
|
||||
// context menu, recovery retry). With the update-guard capability it opens
|
||||
// the readiness dialog first; the dialog's proceed runs the same
|
||||
// runWithLog-backed executor either way, so there is exactly one update path.
|
||||
const requestStackUpdate = async (stackFile: string): Promise<void> => {
|
||||
if (stackListState.isStackBusy(stackFile)) return;
|
||||
const stackName = stackFile.replace(/\.(yml|yaml)$/, '');
|
||||
const run = () => runStackAction(stackFile, 'update', 'update', 'running', 'Stack updated successfully!');
|
||||
if (hasUpdateGuard) {
|
||||
overlayState.setUpdateReadiness({
|
||||
stackName,
|
||||
stackFile,
|
||||
proceed: () => {
|
||||
overlayState.setUpdateReadiness(null);
|
||||
void run();
|
||||
},
|
||||
});
|
||||
return;
|
||||
}
|
||||
await run();
|
||||
};
|
||||
|
||||
const updateStack = async (e?: React.MouseEvent) => {
|
||||
e?.preventDefault();
|
||||
e?.stopPropagation();
|
||||
if (!stackListState.selectedFile) return;
|
||||
await runStackAction(stackListState.selectedFile, 'update', 'update', 'running', 'Stack updated successfully!');
|
||||
await requestStackUpdate(stackListState.selectedFile);
|
||||
};
|
||||
|
||||
const deleteStack = async (pruneVolumes: boolean) => {
|
||||
@@ -1016,13 +1116,20 @@ export function useStackActions(options: UseStackActionsOptions) {
|
||||
endpoint: string,
|
||||
) => {
|
||||
if (stackListState.isStackBusy(stackFile)) return;
|
||||
// Updates route through the shared update path so the sidebar gets the
|
||||
// readiness dialog, the deploy-feedback modal, and the same failure
|
||||
// handling as the toolbar.
|
||||
if (action === 'update') {
|
||||
await requestStackUpdate(stackFile);
|
||||
return;
|
||||
}
|
||||
const stackName = stackFile.replace(/\.(yml|yaml)$/, '');
|
||||
const startedAt = Date.now();
|
||||
stackListState.setStackAction(stackFile, action);
|
||||
|
||||
if (action === 'stop') {
|
||||
stackListState.setOptimisticStatus(stackFile, 'exited');
|
||||
} else if (action === 'deploy' || action === 'restart' || action === 'update') {
|
||||
} else if (action === 'deploy' || action === 'restart') {
|
||||
stackListState.setOptimisticStatus(stackFile, 'running');
|
||||
}
|
||||
|
||||
@@ -1036,19 +1143,18 @@ export function useStackActions(options: UseStackActionsOptions) {
|
||||
toast.error(stackOpInProgressMessage(stackName, inProgress));
|
||||
return;
|
||||
}
|
||||
if (action === 'deploy' || action === 'update') {
|
||||
if (action === 'deploy') {
|
||||
const blockedBy = tryOpenPolicyBlock(errText, stackName, stackFile, action);
|
||||
if (blockedBy) {
|
||||
toast.error(`${action === 'update' ? 'Update' : 'Deploy'} blocked by policy "${blockedBy}"`);
|
||||
toast.error(`Deploy blocked by policy "${blockedBy}"`);
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
throw parseStackActionError(errText, `${action} failed`);
|
||||
throw parseStackActionError(errText, `${action} failed`, response.status);
|
||||
}
|
||||
toast.success(`Stack ${action}ed successfully!`);
|
||||
await refreshSelectedContainers(stackName, stackFile);
|
||||
if (action === 'update') stackListState.fetchImageUpdates();
|
||||
if (action === 'deploy') {
|
||||
try {
|
||||
const backupRes = await apiFetch(`/stacks/${stackName}/backup`);
|
||||
@@ -1067,7 +1173,7 @@ export function useStackActions(options: UseStackActionsOptions) {
|
||||
? `${msg} - automatically rolled back to previous version.`
|
||||
: msg,
|
||||
);
|
||||
recordActionFailureFor(stackFile, stackName, action, startedAt, msg, actionError.rolledBack === true);
|
||||
recordActionFailureFor(stackFile, stackName, action, startedAt, msg, actionError.rolledBack === true, actionError.failure);
|
||||
await refreshSelectedContainers(stackName, stackFile);
|
||||
} finally {
|
||||
stackListState.clearStackAction(stackFile);
|
||||
@@ -1154,6 +1260,7 @@ export function useStackActions(options: UseStackActionsOptions) {
|
||||
restartStack,
|
||||
serviceAction,
|
||||
updateStack,
|
||||
requestStackUpdate,
|
||||
deleteStack,
|
||||
attemptLeaveEditor,
|
||||
cancelPendingUnsavedLoad,
|
||||
|
||||
@@ -34,6 +34,10 @@ export function buildDiagnostics(
|
||||
? `available${backupInfo.timestamp ? ` (${new Date(backupInfo.timestamp).toISOString()})` : ''}`
|
||||
: 'none'}`,
|
||||
];
|
||||
if (result.failure) {
|
||||
lines.push(`Classified: ${result.failure.label}`);
|
||||
lines.push(`Suggestion: ${result.failure.suggestion}`);
|
||||
}
|
||||
if (result.lastOutputLine) lines.push(`Last output: ${result.lastOutputLine}`);
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user