---
title: "Auto-Heal Policies"
sidebarTitle: Auto-heal
description: "Restart containers that fail their Docker healthcheck or crash, with per-policy thresholds and a built-in safety rail set."
---

## Overview

Auto-Heal Policies restart a container when it stays broken for longer than you allow. A policy acts on two conditions:

- The container reports Docker's `unhealthy` status continuously past your threshold.
- The container crashes (exits with a non-zero code) and stays down past your threshold.

Policies are scoped to a stack and can target every container in the stack or a single Compose service. Each policy runs with its own thresholds and four built-in safety rails so a persistently broken container cannot be restarted in a tight loop.

A container that you stop yourself, or that exits cleanly (a one-shot job finishing with exit code 0), is never restarted. Auto-Heal only acts on failures, not on intentional shutdowns.

Policies live next to your stack-level alert rules in the stack's **Monitor** sheet, which has two tabs: **Alerts** and **Auto-heal**.

## Prerequisites

- For **healthcheck-based** healing, containers must declare a `HEALTHCHECK` in the Dockerfile or a `healthcheck` block in `docker-compose.yml` so they report a Docker health status. **Crash-based** healing needs no healthcheck: any container that exits with a non-zero code qualifies.
- Viewing existing policies and their history is open to every signed-in role. Creating, toggling, or deleting a policy requires the admin role.

<Note>
  Crash healing acts on crashes Sencho observes while it is running. A container that crashed before Sencho started is left for you to inspect rather than restarted automatically.
</Note>

## Workflow

1. Open the stack's **Monitor** sheet any of these ways:
   - Stack header **More actions** → **Monitor**, then switch to the **Auto-heal** tab
   - Container card **Monitor** (prefers that Compose service in **Add new policy**), then switch to **Auto-heal**
   - Sidebar: right-click the stack (or focus it and press **H**) → **Auto-Heal** in the **inspect** group (opens directly on the **Auto-heal** tab)
2. In **Add new policy**, pick a **Service** (or leave the combobox on **All services** to cover every container in the stack) and tune the four thresholds.
3. Click **Add Policy**. The new policy appears in **Active policies** above the form, with its enable toggle already on.

<Frame>
  <img src="/images/auto-heal-policies/sheet.png" alt="The Stack › plex › Monitor sheet on the Auto-heal tab, showing one policy scoped to the plex service in the Active policies section with its ON toggle, and the Add new policy form below with Service, Unhealthy for, Cooldown, Max restarts / hr, and Auto-disable after fields, plus the Add Policy button" />
</Frame>

You can add as many policies to a stack as you need. Each policy is evaluated independently.

## Form fields

| Field | Meaning |
|-------|---------|
| **Service** | The Compose service this policy applies to. Leave on **All services** for a stack-wide policy, or pick a specific service to limit the policy to that service's containers. |
| **Unhealthy for (minutes)** | How long a container must stay broken before Auto-Heal restarts it. This applies to both conditions: continuously reporting `unhealthy`, or staying down after a crash. |
| **Cooldown (minutes)** | After a restart fires, the policy pauses evaluation for this many minutes so the container has time to come back up. |
| **Max restarts / hr** | The most times this policy will restart a given container in a rolling one-hour window. Once the cap is reached, further restarts are skipped until the window clears. |
| **Auto-disable after (failures)** | If the restart call itself fails this many times in a row, the policy disables itself so it stops looping on a broken setup. The counter resets after any successful restart. |

## Stack vs service scope

- **All services** applies the policy to every container in the stack that reports `unhealthy`. Containers are matched by the `com.docker.compose.service` label.
- A **named service** scopes the policy to the containers for that Compose service only. Multiple policies can coexist on the same stack, so you can keep an aggressive policy on one service while running a more permissive default for the rest.
- If an **All services** policy and a named-service policy both cover the same container, only one restart fires for that container per evaluation pass. Give the two policies different thresholds deliberately if you want predictable behavior; otherwise whichever policy is evaluated first claims the restart.

## Safety rails

Each policy has four safety rails built in. They run before any restart.

**Cooldown.** After a restart, the policy ignores the container for the configured number of minutes. This gives the container time to start up and pass its healthcheck before the next evaluation. The cooldown is per container, so other services in the same stack are not blocked.

**Hourly cap.** The **Max restarts / hr** value is enforced as a rolling one-hour window per container. When the cap is reached, evaluations continue but restarts are skipped until the window clears.

**Recent operator action.** If you or another operator manually stops or restarts the container, the policy suppresses evaluation for that container for 60 seconds. This avoids fighting an operator who is intentionally power-cycling a service. A clean stop is also never seen as a crash, so a service you take down stays down.

**Auto-disable.** If the restart call itself fails (for example, the Docker daemon is not reachable, or the container is in a state that cannot be restarted), the policy increments a consecutive-failure counter. Failed attempts are spaced by the cooldown, so a broken setup is retried on the cooldown interval rather than on every evaluation tick. When the counter reaches the configured **Auto-disable after (failures)** value, the policy disables itself and emits a notification. A successful restart resets the counter to zero.

## Managing policies

Each row in **Active policies** shows the service it targets and a compact summary of the thresholds. Every signed-in role sees the row and can expand its history; the **ON/OFF** toggle and the delete icon are visible only to admins.

<Frame>
  <img src="/images/auto-heal-policies/policy-row.png" alt="Single Auto-heal policy row showing the plex threshold summary, ON toggle, history chevron, and delete button" />
</Frame>

- **Toggle a policy off** to pause evaluation without losing its configuration. This is useful when you need to take a service down for maintenance without auto-heal interfering.
- **Delete a policy** to remove it and its history. The policy stops evaluating immediately.
- When a policy has accumulated restart failures but has not yet hit the auto-disable threshold, a red **N failures** pill appears next to its summary so you can see trouble building before the policy disables itself.

## Recent activity

Expand the chevron on any policy to reveal its history. Each entry shows the timestamp, the container that was acted on, a colored action label, and a one-line reason.

<Frame>
  <img src="/images/auto-heal-policies/recent-activity.png" alt="The plex policy row with the Recent activity panel expanded, showing No history yet" />
</Frame>

The action label tells you what Auto-Heal did or, if it chose not to act, why it skipped.

| Action | When you see it |
|--------|-----------------|
| **Restarted** | A restart fired successfully. |
| **Skipped (user action)** | The container was manually stopped or restarted in the last 60 seconds. |
| **Skipped (cooldown)** | The policy fired recently and is still inside its cooldown window. |
| **Skipped (rate limit)** | The hourly cap for this container has been hit. |
| **Failed** | The restart call was attempted and failed. Counts toward the auto-disable threshold. |
| **Auto-disabled** | The consecutive-failure threshold was reached and the policy disabled itself. |
| **Docker unavailable** | The Docker daemon was not reachable at evaluation time. The policy is not penalized and will retry on the next 30-second tick. |

## Multi-node

Auto-heal is per node. Policies you create against a remote node are stored on and evaluated by that remote Sencho instance, so the policy keeps running even if the central node is offline. The **Monitor** sheet operates against whichever node is currently selected in the sidebar; switch nodes to see and edit that node's policies.

## Notifications

Auto-Heal dispatches a notification through your configured channels when:

- A restart fires successfully (info severity).
- A restart call fails (warning).
- A policy auto-disables itself (warning).

Each notification includes the stack and container names. Configure delivery channels under **Settings → Notifications → Channels**. See [Alerts & Notifications](/features/alerts-notifications) for setup details.

## Dashboard visibility

The dashboard's **Configuration status** card surfaces an **Auto-heal policies** entry that reads `N / M active` across the current node, so you can see at a glance whether stacks are covered. The entry shows **None** when no policies exist on the active node.

## Troubleshooting

<AccordionGroup>
  <Accordion title="The policy was auto-disabled">
    The policy hit its **Auto-disable after (failures)** threshold. Open the policy's **Recent activity** to see which container the restart calls were failing on and what reason was reported. Common causes are a corrupted container that cannot be restarted, missing image layers after a registry change, or a stack that needs to be recreated rather than restarted. Resolve the underlying issue, then flip the policy back to **ON**.
  </Accordion>

  <Accordion title="The policy did not fire when expected">
    Auto-Heal acts on two conditions: a container that stays `unhealthy`, and a container that crashes (non-zero exit) and stays down. It does **not** act on a container that exited cleanly (exit code 0), one you stopped yourself, or a crash that happened before Sencho was running. For the healthcheck path, use `docker inspect <container>` and check `State.Health.Status` to confirm the container actually reaches `unhealthy`; for the crash path, check `State.ExitCode` is non-zero.

    Evaluation runs every 30 seconds, so there is up to a 30-second delay between the **Unhealthy for (minutes)** threshold being crossed and the restart firing. The 60-second operator-action window also suppresses evaluation right after a manual stop or restart.
  </Accordion>

  <Accordion title="History shows 'Docker unavailable'">
    The Docker daemon was not reachable when Auto-Heal tried to read or restart a container. This is treated as transient: the policy is not penalized, the consecutive-failures counter is not incremented, and the next 30-second tick will retry. If you see this entry repeatedly, check that the Docker socket is mounted and the daemon is running on that node.
  </Accordion>

  <Accordion title="The container keeps getting restarted in a loop">
    Two safety rails are designed to stop this: the **Max restarts / hr** cap and the **Auto-disable after (failures)** threshold. If a container is unhealthy immediately after each restart, lower the cap and the failure threshold while you investigate the root cause in the container's logs. You can also raise **Unhealthy for (minutes)** so transient blips do not count against the policy.
  </Accordion>

  <Accordion title="I don't see the Add new policy form, or the ON/OFF toggle and delete icon">
    The **Auto-heal** tab itself is visible to every signed-in role, including viewers, so you can always check what is configured and browse history. Creating a policy, toggling it on or off, and deleting it are admin-only actions; a non-admin role sees the policy list and its history but not those controls. Confirm you are signed in as an admin under **Settings → Users** if you expect to make changes.
  </Accordion>
</AccordionGroup>
