mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-11 11:16:55 +00:00
Anso
5bb4b01953
* feat(db): add auto_heal_policies and auto_heal_history schema and CRUD Adds two new SQLite tables (auto_heal_policies, auto_heal_history) to DatabaseService.initSchema() and exposes CRUD methods: getAutoHealPolicies, getAutoHealPolicy, addAutoHealPolicy, updateAutoHealPolicy, deleteAutoHealPolicy, recordAutoHealHistory, getAutoHealHistory, incrementConsecutiveFailures, resetConsecutiveFailures, setPolicyEnabled. Also adds AutoHealPolicy and AutoHealHistoryEntry TypeScript interfaces. * feat(events): track health-status duration and expose state accessors - Add healthStatus and unhealthySince fields to InternalContainerState - onHealthStatus now records unhealthySince timestamp on first transition to unhealthy, and clears it when the container recovers or restarts - onStart resets both fields so a restarted container begins from 'starting' - Add listContainerStates() and getContainerState() public accessors for use by the upcoming AutoHealService evaluator * fix(auto-heal): key allowlist in updateAutoHealPolicy, cascade delete, extract ContainerHealthSnapshot * feat: add AutoHealService evaluator singleton Polls every 30 s, matches containers to enabled policies via Compose labels, and restarts containers that have been unhealthy beyond the configured threshold. Enforces cooldown, per-hour rate cap, and recent-user-action suppression; auto-disables policies after repeated consecutive failures. Also adds DockerEventManager.getService() accessor required by the evaluator. * fix(auto-heal): prune stale restartTimestamps, guard undefined policy id - Prune restartTimestamps entries for containers no longer running after each container list fetch, preventing unbounded map growth from dead container IDs. - Guard against policies with undefined id at the start of the per-policy loop; warn and skip rather than proceed with a non-null assertion. - Extract handleAutoDisable private helper to bring executeHeal under 30 lines and isolate the auto-disable side-effect sequence. - Move ContainerInfo type to module scope. * feat: add auto-heal API routes and wire AutoHealService lifecycle Registers five REST endpoints under /api/auto-heal/policies (list, create, patch, delete, history) with requirePaid + requireAdmin guards and Zod validation. Wires AutoHealService.start()/stop() into the server startup and graceful-shutdown blocks alongside MonitorService. * test: add AutoHealService and DatabaseService auto-heal unit tests - 15 unit tests for AutoHealService.shouldHeal covering all decision branches (healthy state, duration threshold, user-action suppression, cooldown, rate limiting, and correct skipReason values) - 13 integration tests for DatabaseService auto-heal CRUD: policy round-trip, stack-name filter, partial update, cascade delete, history ordering/limit, consecutive failure counters, and setPolicyEnabled toggle * fix: log AutoHealService shutdown errors consistently * fix(api): requireAdmin-first guard order and try/catch on auto-heal routes * feat(ui): add StackAutoHealSheet component * feat(ui): add Auto-Heal context menu item to EditorLayout * fix(ui): StackAutoHealSheet label, token, a11y, and useEffect fixes - Rename 'All services in stack' to 'All services' in combobox options and placeholder - Replace text-green-600 with text-success design token in actionColorClass - Add htmlFor/id pairs to all four numeric form inputs for accessibility - Inline fetch logic into useEffect, removing stale closure risk and eslint-disable comment - Remove now-unused fetchPolicies and fetchServices standalone functions - Update 'Auto-disable after' label to 'Auto-disable after (failures)' for clarity - Add toast.error in policy fetch failure path; services fetch silently skips as before * docs: add auto-heal-policies feature documentation * test(e2e): add auto-heal policies CRUD spec * fix(docs): correct auto-heal-policies nav position in docs.json
78 lines
4.8 KiB
Plaintext
78 lines
4.8 KiB
Plaintext
---
|
|
title: "Auto-Heal Policies"
|
|
description: "Automatically restart containers that fail Docker healthchecks."
|
|
---
|
|
|
|
<Note>
|
|
Auto-Heal Policies require a **Skipper** or **Admiral** license.
|
|
</Note>
|
|
|
|
## Overview
|
|
|
|
Auto-Heal Policies let you define rules that restart containers when they have been in an `unhealthy` Docker healthcheck state for longer than a specified threshold. This keeps long-running services recoverable without manual intervention.
|
|
|
|
## Prerequisites
|
|
|
|
- Your containers must define a `HEALTHCHECK` instruction in their `Dockerfile` or in the `healthcheck` section of your `docker-compose.yml`.
|
|
- You must be an admin user.
|
|
- A Skipper or Admiral license.
|
|
|
|
## Creating a Policy
|
|
|
|
1. In the sidebar, right-click the stack you want to protect.
|
|
2. Select **Auto-Heal** from the context menu.
|
|
3. In the sheet that opens, fill in the form:
|
|
- **Service** — Select a specific service from your stack, or choose **All services** to apply the policy to every container.
|
|
- **Unhealthy for (minutes)** — How long a container must be continuously unhealthy before it is restarted.
|
|
- **Cooldown (minutes)** — How long to wait after a restart before evaluating the container again.
|
|
- **Max restarts per hour** — The maximum number of times this container can be restarted within a rolling hour window.
|
|
- **Auto-disable after (failures)** — How many consecutive failed restart attempts disable the policy automatically.
|
|
4. Click **Add Policy**.
|
|
|
|
<Frame>
|
|
<img src="/images/auto-heal-policies/policy-sheet.png" alt="Auto-Heal Policies sheet showing a policy for the web service" />
|
|
</Frame>
|
|
|
|
## Stack vs Service Scope
|
|
|
|
- **All services** — The policy applies to every container in the stack that reports `unhealthy` status.
|
|
- **Named service** — The policy targets only the containers for that specific Compose service (matched by the `com.docker.compose.service` label).
|
|
|
|
Multiple policies can coexist on the same stack. Each policy is evaluated independently.
|
|
|
|
## Safety Rails
|
|
|
|
Each policy includes four built-in safety mechanisms:
|
|
|
|
**Cooldown period** — After a restart is triggered, the policy pauses evaluation for the configured number of minutes. This gives the container time to recover before being evaluated again.
|
|
|
|
**Hourly restart cap** — If a container has been restarted the configured maximum number of times within the last hour, further restarts are skipped until the window clears. This prevents a persistently broken container from being restarted in a tight loop.
|
|
|
|
**Recent user action suppression** — If you or another operator has manually stopped or restarted the container in the last 60 seconds, the policy skips evaluation for that container. This avoids interfering with in-progress manual interventions.
|
|
|
|
**Auto-disable on repeated failures** — If the restart attempt itself fails (for example, because the Docker daemon is temporarily unavailable) the configured number of times in a row, the policy is automatically disabled. A notification is sent, and you can re-enable the policy from the sheet once the underlying issue is resolved.
|
|
|
|
## Policy History
|
|
|
|
Each policy row in the sheet can be expanded to show recent activity: restarts, skipped evaluations, and any auto-disable events. The history shows the container name, action taken, and the reason.
|
|
|
|
## Troubleshooting
|
|
|
|
<AccordionGroup>
|
|
<Accordion title="The policy was auto-disabled">
|
|
The policy disabled itself after the configured number of consecutive restart failures. Check the container logs to understand why the restart is failing. Common causes include the Docker daemon being unavailable, insufficient system resources, or a misconfigured compose file. Once the issue is resolved, re-enable the policy from the Auto-Heal sheet.
|
|
</Accordion>
|
|
|
|
<Accordion title="The policy did not fire when expected">
|
|
Verify that the container's Docker healthcheck actually reports `unhealthy`. A container can fail to start (exit code non-zero) without ever reaching the `unhealthy` state. Use `docker inspect <container>` and check `State.Health.Status`.
|
|
|
|
If a manual restart was performed recently, the policy suppresses evaluation for 60 seconds after the restart to avoid conflicting with operator actions.
|
|
|
|
Also check that the **Unhealthy for (minutes)** threshold has elapsed. The policy evaluates every 30 seconds, so there may be up to a 30-second delay between the threshold being crossed and the restart firing.
|
|
</Accordion>
|
|
|
|
<Accordion title="The container keeps getting restarted in a loop">
|
|
If the container becomes unhealthy immediately after each restart, the hourly restart cap and the auto-disable-after-failures setting limit how many times this can occur. Review the container logs to address the root cause. You can lower the **Max restarts per hour** value or increase the **Unhealthy for (minutes)** threshold to reduce restart frequency while you investigate.
|
|
</Accordion>
|
|
</AccordionGroup>
|