sencho/docs/features/scheduled-operations.mdx

---
title: Scheduled Operations
description: Automate stack lifecycle, image updates, vulnerability scans, fleet snapshots, and system prunes on a cron schedule.
---

Schedules is a unified surface for every recurring maintenance operation Sencho knows how to run: stack restarts, per-node and fleet-wide image updates, lifecycle events (stop, take down, start, backup), system prunes, and vulnerability scans. The default view is a next-24-hour Timeline of upcoming runs across five lanes; an All tasks table view lists every schedule regardless of when it next fires.

<Note>
  Available to admins.
</Note>

<Note>
  Schedules is hub-only and is hidden from the nav strip when a remote node is the active selection. See [Multi-Node Management](/features/multi-node#what-top-level-views-show-when-a-remote-node-is-active).
</Note>

## Anatomy at a glance

Open the **Schedules** tab from the top navigation bar. The page opens on the Timeline view with the masthead, the five lane track, and a bottom time axis.

<Frame>
  <img src="/images/scheduled-operations/timeline.png" alt="Schedules page in Timeline view. Header reads 'Scheduled Operations' on the left with a clock icon; on the right are a Timeline / All tasks toggle, a Refresh button, and a 'New Schedule' primary button. Below, a kicker 'NEXT 24 HOURS' sits above an italic display heading 'Next 24 hours' and a monospace date range 'Thu, May 14 11:21 → Fri, May 15 11:21'. A right-aligned 'Next' pill shows '03:00' with the subtitle 'Vul Scan · in 15h 38m'. Five lanes (Stack lifecycle, Updates, Security, Maintenance, Backups) run horizontally; a purple '03:00 hub' pill sits on the Security lane and a cyan '04:00 Entire fleet' pill sits on the Backups lane. A cyan vertical 'now' rail glows at the left edge. Six time ticks (11:21, 16:09, 20:57, 01:45, 06:33, 11:21) run along the bottom axis." />
</Frame>

## Timeline view

The Timeline plots every firing of every enabled task across a rolling 24-hour window starting from the current minute.

- **Masthead.** A `NEXT 24 HOURS` kicker, an italic display heading, the window's start and end timestamps in a monospace range, and a right-anchored **Next** pill that reads out the time and task name of the next firing and a relative countdown.
- **Five lanes.** Lifecycle (label blue), Updates (success green), Security (label purple), Upkeep (warning amber), and Backups (brand cyan). The Lifecycle lane holds stack lifecycle actions (Backup Stack Compose Files, Start / Bring Up Stack, Restart Stack, Stop Stack, Take Stack Down) and standalone container actions (Restart Container, Stop Container, Start Container); Updates holds per-node and fleet image updates; Security holds vulnerability scans; Upkeep holds node resource prunes; Backups holds fleet snapshots.
- **Pills.** One pill per firing within the window, positioned proportionally to the firing's time. Each pill shows the firing time and a target: the stack for stack actions, the container name for container actions, the selected node for prune and scan, and "Entire fleet" for a fleet snapshot. Hover a pill for the full detail (action, task name, and node). Pills are color-matched to their lane. Click a pill to open the run history sheet for that task.
- **Now rail.** A glowing vertical rail at the current minute, anchored to the left of the track at page open and drifting right as time passes (the page recomputes positions periodically).
- **Axis.** Six monospace time ticks run along the bottom, evenly spaced through the window.

Multi-fire crons (for example, `0 */6 * * *`) render a pill for each firing inside the window. Disabled tasks do not appear on the Timeline. If nothing fires in the next 24 hours, the lanes still render, and an empty-state line points you at the All tasks view.

Toggle to **All tasks** from the masthead to see every schedule in a table regardless of when it next fires.

## All tasks view

The All tasks toggle swaps the lane track for a sortable table.

<Frame>
  <img src="/images/scheduled-operations/all-tasks.png" alt="All tasks table view of Scheduled Operations. Columns are Name, Action, Target, Schedule, Status, Next Run, Enabled, Actions. Two rows are visible: 'Vul Scan' with a Scan Node Images badge, target 'system', schedule 'At 03:00 AM / 0 3 * * *', Success status, next run '5/15/2026, 3:00:00 AM', ON toggle, and an action group of Run now / Execution history / Edit / Delete (Delete in destructive red). 'Nightly Snapshot' with a Create Fleet Snapshot badge, target 'fleet', schedule 'At 04:00 AM / 0 4 * * *', Success, next run '5/15/2026, 4:00:00 AM', ON, same action group." />
</Frame>

| Column | What it shows |
|---|---|
| **Name** | The task name. |
| **Action** | A badge labelling the operation (e.g. Restart Stack, Scan Node Images, Create Fleet Snapshot). |
| **Target** | The stack the task targets (with a service list in parentheses when restart is scoped to specific services), the container name for container actions, or the target type for other non-stack actions (`system`, `fleet`). |
| **Schedule** | A human-readable description of the cron with the raw expression on a second line. |
| **Status** | The last run result: **Success** (green), **Failed** (red), or `Never run` if the task has not fired yet. |
| **Next Run** | The timestamp of the next firing, or a dash if the task is disabled or has no upcoming runs. |
| **Enabled** | An on/off toggle that pauses or resumes the schedule without deleting it. |
| **Actions** | Per-row controls: see [Managing tasks](#managing-tasks). |

## Supported actions

| Action | Target | What it does |
|---|---|---|
| **Restart Stack** | A specific stack (optionally specific services) on a specific node | Restarts all or selected containers in the stack. |
| **Auto-update Stack** | A specific stack on a specific node | Checks each image in the stack for a newer tag and recreates the stack if any image has an update. See [Auto-Update Policies](/features/auto-update-policies) for the companion review board. |
| **Auto-update All Stacks on Node** | A specific node | Runs the auto-update check across every stack on the node. Pair with **Auto-update Stack** rows when you want different cadences for specific stacks. |
| **Create Fleet Snapshot** | The whole fleet | Creates a versioned, fleet-wide snapshot of every node's compose files and `.env` files. See [Fleet Backups](/features/fleet-backups). |
| **Prune Node Resources** | A local node | Prunes containers, images, networks, and volumes (any subset), optionally filtered by a Docker label. Runs on local nodes only. |
| **Scan Node Images** | A local node | Runs Trivy against every image on the node and persists the results. Requires Trivy to be installed on the target node ([Installing Trivy](/operations/trivy-setup)). Runs on local nodes only. |
| **Backup Stack Compose Files** | A specific stack on a specific node | Copies the stack's compose file and `.env` to `<DATA_DIR>/backups/<stack>/`. One slot per stack; each run overwrites the previous backup. For a versioned archive use a Create Fleet Snapshot instead. |
| **Stop Stack** | A specific stack on a specific node | Runs `docker compose stop`. Containers are stopped but preserved. Use for off-hours power saving when you want a fast restart later. |
| **Take Stack Down** | A specific stack on a specific node | Runs `docker compose down`. Containers are removed. Use to fully release resources when the stack is not needed for an extended period. |
| **Start / Bring Up Stack** | A specific stack on a specific node | Runs `docker compose up -d`. Works for both stopped and removed containers: if they exist they are started, if not they are created from the compose file. |
| **Restart Container** | A specific container by name on a specific node | Restarts one container directly through Docker. Use for third-party or standalone containers that are not managed as a Sencho stack. |
| **Stop Container** | A specific container by name on a specific node | Stops one container. The container remains on disk for a faster start later. |
| **Start Container** | A specific container by name on a specific node | Starts one stopped container by name. |

## Creating a scheduled task

Click **New Schedule** in the header. The form opens in a centered modal. The Action picker lists every supported operation, grouped by category: Lifecycle, Updates, Security, Upkeep, and Backups.

<Frame>
  <img src="/images/scheduled-operations/action-picker.png" alt="The New scheduled task modal with the Action combobox expanded. The dropdown groups actions under category headers: Stack lifecycle (Backup Stack Compose Files, Start / Bring Up Stack, Restart Stack, Stop Stack, Take Stack Down), Updates (Auto-update Stack, Auto-update All Stacks on Node), Security (Scan Node Images), Maintenance (Prune Node Resources), and Backups (Create Fleet Snapshot). Below the picker, partly visible, sit a Services row with 'echo' and 'prober' checkboxes, the Cron Expression input, the Enabled toggle, and the Delete after successful run checkbox." />
</Frame>

Common fields:

- **Name.** A descriptive label.
- **Action.** Pick the operation. Once selected, the form shows a risk badge and a one-line helper text below the picker so you can see the blast radius at a glance. The six risk levels are Safe (informational, no risk), Read-only (observation only), Interruptive (temporarily disrupts running state), Runtime change (changes running state, not destructive), Removes containers (tears down containers), and Destructive (irreversibly deletes resources).
- **Schedule.** Choose when the task runs. The section opens in **Simple** mode, where you pick a frequency (Once, Hourly, Daily, Weekly, or Monthly) and a time, and Sencho builds the cron expression for you. Switch to **Advanced** to type a raw 5-field cron by hand. Either way a human-readable preview appears below the controls (`At 03:00 AM`, `At 04:00 AM, only on Sunday`, and so on). See [Simple and Advanced modes](#simple-and-advanced-modes) and the [Cron expression reference](#cron-expression-reference).
- **Enabled.** Toggle the task on or off without deleting it.
- **Delete after successful run.** When enabled, the task removes itself from the schedule after its first successful execution. Failures keep the task so you can inspect the error and retry. See [Delete after successful run](#delete-after-successful-run).

Conditional fields per action:

- **Stack actions** (Backup Stack Compose Files, Start / Bring Up Stack, Restart Stack, Auto-update Stack, Stop Stack, Take Stack Down) add a **Node** combobox and a **Stack** combobox. Restart Stack additionally renders a **Services** checkbox grid sourced from the stack's compose services on the selected node, so you can scope the restart to a subset instead of restarting the entire stack.
- **Container actions** (Restart Container, Stop Container, Start Container) add a **Node** combobox and a **Container** combobox listing every container on that node (running and stopped). The picker shows each container's name, state, and image. When the container is not part of a Sencho stack, helper text explains that the schedule targets the container by node and name.
- **Auto-update All Stacks on Node** adds a **Node** combobox. The helper text "Checks every stack on the selected node and updates stacks with newer images" appears above, next to the Runtime change badge.
- **Scan Node Images** adds a **Node** combobox listing local nodes only. The helper text "Runs Trivy against images on the selected local node and records the findings" and Read-only badge appear above.
- **Prune Node Resources** adds a **Node** combobox listing local nodes only, then a **Prune Targets** group (Containers, Images, Networks, Volumes; all selected by default) and a **Label Filter** input for scoping the prune to resources matching a Docker label.
- **Create Fleet Snapshot** shows a read-only **Scope: Entire fleet** summary instead of a Node or Stack picker, because it captures every node.

<Frame>
  <img src="/images/scheduled-operations/create-restart.png" alt="The New scheduled task modal configured for a Restart Stack task. Name reads 'Nightly staging restart'. Action is Restart Stack. Node is Local. Stack is 'audit-mesh-prod'. A Services group with helper text '(leave empty for all)' shows two unchecked checkboxes labelled 'echo' and 'prober'. Cron Expression is '0 3 * * *' with the preview 'At 03:00 AM'. The Enabled toggle reads ON. The Delete after successful run checkbox is unchecked. Cancel and Create buttons sit at the bottom right." />
</Frame>

### Simple and Advanced modes

The **Schedule** section has two modes, toggled at the top right of the section.

**Simple** (the default) turns a frequency and a time into a cron expression so you do not have to write one by hand:

- **Once** runs the task a single time on a date you pick. Selecting Once turns on **Delete after successful run** and locks it on (see [One-time schedules](#one-time-schedules)).
- **Hourly** runs at a chosen minute past every hour.
- **Daily** runs once a day at a chosen time.
- **Weekly** runs at a chosen time on the weekdays you select.
- **Monthly** runs at a chosen time on a chosen day of the month.

The generated cron and its plain-language description appear beneath the controls, and times run in the node's local timezone.

**Advanced** swaps the controls for a single cron input where you type a raw 5-field expression. Use it for cadences Simple mode does not cover, such as "every 6 hours" (`0 */6 * * *`). Editing a task whose schedule is not expressible in Simple mode opens directly in Advanced; switching such a task to Simple replaces the custom expression and tells you so before you save.

#### One-time schedules

Because cron has no year field, a one-time schedule is stored as a fully pinned `minute hour day month *` expression that would otherwise repeat on that date every year. To keep it genuinely one-time, selecting **Once** turns on **Delete after successful run** and locks it on: the task fires on the chosen date and then removes itself once it succeeds. A failed run is kept so you can retry or debug. To build a recurring schedule instead, choose a different frequency or use Advanced cron.

## Granular targeting

### Per-service restart

When creating a Restart Stack task and a stack is selected, Sencho reads the compose file and renders a checkbox per defined service under **Services (leave empty for all)**. Check one or more boxes to restart only those services. Leave them all unchecked to restart the entire stack.

### Scheduled vulnerability scans

A Scan Node Images task runs Trivy against every image on the selected node and persists the findings. Trivy must be installed on that node ([Installing Trivy](/operations/trivy-setup)). Manual and scheduled scans share the same digest-based 24-hour cache, so unchanged images that were scanned recently are reused instead of rescanned on every run, which keeps execution time low.

When a scheduled scan finishes, Sencho dispatches a completion notification. The severity reflects the outcome (info on a clean run, warning when findings are present); the category is `scan_finding`. The full message format and how it surfaces in the bell is documented in [Alerts & Notifications · Vulnerability scanning](/features/alerts-notifications#vulnerability-scanning).

<Frame>
  <img src="/images/scheduled-operations/create-scan.png" alt="The New scheduled task modal configured for a Scan Node Images task. Name reads 'Nightly vulnerability scan'. Action is Scan Node Images. Node is Local. A Read-only risk badge and the helper text 'Runs Trivy against images on the selected local node and records the findings.' appear below the action picker. Cron Expression is '0 3 * * *' with the preview 'At 03:00 AM'. The Enabled toggle reads ON. Cancel and Create buttons sit at the bottom right." />
</Frame>

### Prune label filter

When creating a Prune Node Resources task, the **Label Filter (optional)** input scopes the prune to resources matching a specific Docker label. Use `key=value` format (for example, `com.docker.compose.project=staging`). Leave the field empty to prune every unused resource of the selected types.

<Frame>
  <img src="/images/scheduled-operations/create-prune.png" alt="The New scheduled task modal configured for a Prune Node Resources. Name reads 'Weekly cleanup'. Action is Prune Node Resources. Node is Local. The Prune Targets group shows four checked boxes: Containers, Images, Networks, Volumes. The Label Filter (optional) input reads 'com.docker.compose.project=staging' with the helper text 'Only prune resources matching this Docker label.' Cron Expression is '0 4 * * 0' with the preview 'At 04:00 AM, only on Sunday'. The Enabled toggle reads ON. The Delete after successful run checkbox is unchecked. Cancel and Create buttons sit at the bottom right." />
</Frame>

### Stack lifecycle scheduling

Backup Stack Compose Files, Start / Bring Up Stack, Stop Stack, and Take Stack Down let you schedule lifecycle events for individual stacks on a per-node basis. A common pattern for development or staging stacks is:

- A **Stop Stack** or **Take Stack Down** task at the end of the working day (for example, `0 19 * * 1-5`, 7 PM on weekdays).
- A **Start / Bring Up Stack** task at the start of the working day (for example, `0 8 * * 1-5`, 8 AM on weekdays).

**Stop vs Take Down.** Use Stop Stack when you want containers ready to resume quickly: Docker keeps the container filesystem in place and the matching Start Stack simply restarts the existing containers. Use Take Stack Down to fully release resources; Start Stack then recreates the containers from the compose file on the next run.

**Backup Stack Compose Files** keeps a single slot per stack under `<DATA_DIR>/backups/<stack>/`. Each run overwrites the previous backup. For a versioned, point-in-time archive across all nodes, use a [Create Fleet Snapshot](/features/fleet-backups) task instead.

Pick any node in your fleet from the schedule's **Node** combobox: the four lifecycle actions run against the node you select, whether it is the hub or a connected remote. The schedule itself is stored on the hub, so you manage every node's lifecycle tasks from one place.

### Delete after successful run

When **Delete after successful run** is enabled, the task removes itself, including its run history, after its first successful execution. Use this for one-shot operations: a pre-deploy preparation step, a one-off backup before a config change, or a stop request that should fire exactly once.

If the run fails, the task stays in the schedule so you can inspect the error in the run history and retry with **Run now** or by re-enabling the schedule. Disabling the option at any time before the successful run prevents auto-deletion.

<Note>
  Manual runs via **Run now** also trigger the delete-after-run logic. If you run a one-shot task manually and it succeeds, the task removes itself.
</Note>

## Cron expression reference

Sencho uses standard 5-field cron expressions:

```
┌───────────── minute (0–59)
│ ┌─────────── hour (0–23)
│ │ ┌───────── day of month (1–31)
│ │ │ ┌─────── month (1–12)
│ │ │ │ ┌───── day of week (0–7, 0 and 7 = Sunday)
│ │ │ │ │
* * * * *
```

Schedules run on a one-minute granularity, so a six-field expression with a leading seconds field is not accepted: enter exactly five fields, or the create and edit forms reject it.

Common examples:

| Expression | Meaning |
|---|---|
| `0 3 * * *` | Every day at 3:00 AM |
| `0 */6 * * *` | Every 6 hours |
| `0 3 * * 0` | Every Sunday at 3:00 AM |
| `30 2 1 * *` | The 1st of every month at 2:30 AM |
| `0 0 * * 1-5` | Midnight on weekdays |

## Filtering by node

When you manage a multi-node fleet, you can filter the schedule list to show only tasks targeting a specific node. There are two entry points:

- **From the Nodes table:** click the calendar icon on any node row in **Settings · Nodes** to jump straight to Schedules filtered to that node.
- **From the Schedules view:** a filter chip appears at the top of the page showing the active node filter, with a **Clear filter** button to return to the full list.

When you create a new task while a node filter is active, Sencho pre-selects that node in the create modal.

## Managing tasks

Each row in the All tasks table has four per-row buttons:

- **Run now** (play icon). Immediately execute the task without waiting for the next firing. While the run is in flight, the button pulses; once it completes, the row's Status badge updates and the run appears in the history as a `Manual` source.
- **Execution history** (clock icon). Open the run history sheet for the task. See [Execution history](#execution-history).
- **Edit** (pencil icon). Open the task in the same modal used to create it. All fields are editable; the modal keeps the existing target unless you change the Action.
- **Delete** (trash icon). Open a destructive confirmation dialog that permanently removes the task and its run history. There is no soft-delete.

Use the **Enabled** toggle in the row to pause a schedule without losing its configuration or history. Disabled tasks still appear in the table; they are excluded from the Timeline.

## Failure notifications

When a scheduled task fails, Sencho dispatches an `error`-level notification in the `system` category through your configured channels (Discord, Slack, Apprise, custom webhooks, and the in-app bell). The notification carries the task name, the action, and the error message so you can diagnose without opening the run history.

When a task that previously failed succeeds again, Sencho dispatches an `info`-level recovery notification in the same category to confirm the issue is resolved. Tasks that succeed on every run do not produce per-run notifications: the recovery-only approach keeps the channel quiet during steady-state operation.

Scan Node Images tasks are an exception: they always dispatch a completion notification, even on a clean run, because the message body carries severity counts you may want to react to. The severity is `info` on a clean run and `warning` when findings are present; the category is `scan_finding`, not `system`. The full message format and routing behavior is documented in [Alerts & Notifications · Vulnerability scanning](/features/alerts-notifications#vulnerability-scanning).

Configure delivery channels in **Settings · Notifications · Channels**.

## Execution history

The Execution history button on any row opens a right-side sheet with the breadcrumb **Schedules › `<task name>` › Runs**. The sheet header reports the total run count, and a **Download CSV** secondary action exports the full history. The footer reads **Next run `<timestamp>`** when one is scheduled.

<Frame>
  <img src="/images/scheduled-operations/run-history.png" alt="The Execution history sheet for a task named 'Vul Scan'. Breadcrumb reads 'Schedules › VUL SCAN › RUNS'. The italic display heading 'Vul Scan' sits above a '28 runs' meta line and a 'Download CSV' button. A table lists nineteen runs with columns Time, Source (every row a Scheduled badge), Status (every row a green Success badge), Duration (mixing fast cache-hit runs of 0.0s with full scans up to 21.9s), and Details (mixing 'Scanned 15 image(s). Found 27 critical…' rows with 'All 15 image(s) already scanned recently (cache hit)' rows). The footer reads 'NEXT RUN 5/15/2026, 3:00:00 AM'." />
</Frame>

The table columns are:

| Column | What it shows |
|---|---|
| **Time** | When the run started, in your local timezone. |
| **Source** | A badge: **Scheduled** for cron firings, **Manual** for Run now triggers. |
| **Status** | **Success** (green), **Failed** (red), or **Running** (in-flight). |
| **Duration** | How long the run took. |
| **Details** | A summary of the outcome for successful runs, or the captured error for failures. |

The sheet paginates at 20 rows per page; navigate with the **Previous** / **Next** controls in the footer. Run history is retained for 30 days; older runs are purged automatically.

The CSV export includes every run in the history with columns: Timestamp, Source, Status, Duration (s), Details.

## How it works

A background Scheduler Service evaluates due tasks every 60 seconds. When a task's next run time has passed:

1. The scheduler checks that your license tier still allows the action; if not, the run is logged as failed and skipped.
2. It executes the configured action using the same internal services that power the equivalent UI buttons (Restart, Snapshot, Prune, and so on).
3. The result is logged to the execution history.
4. On failure, an `error`-level notification goes out through the configured channels. On recovery from a previously failing state, an `info`-level recovery notification is dispatched.
5. The next run time is recalculated from the cron expression.

Schedules are stored on the hub. Stack-targeted actions execute against the selected node, so lifecycle and per-stack update schedules can target either the hub or a connected remote node. Prune Node Resources and Scan Node Images run on local nodes only. Create Fleet Snapshot captures every node from the hub schedule.

If a task is still running from a previous firing, the scheduler skips the new firing to prevent overlap.

If Sencho restarts while a task is mid-execution, the orphaned run record is marked as failed with the message `Server restarted during execution` on the next startup. No manual cleanup is needed.

## Troubleshooting

<AccordionGroup>
  <Accordion title="A task was automatically disabled">
    If a task's cron expression stops being parseable (for example, due to a corrupt database edit), the scheduler disables the task and records the validation error in its last-error field. Open the task in Edit mode, re-enter a valid cron expression, and flip the Enabled toggle back on. The next firing recalculates from the new expression on the next tick.
  </Accordion>
  <Accordion title="A run shows 'Server restarted during execution'">
    Sencho was restarted (or the process crashed) while this run was in flight, so the scheduler marked it failed on startup to avoid a stuck `Running` row. The task itself is still enabled and will fire again at its next cron tick. If you want to re-execute immediately, click **Run now** on the row.
  </Accordion>
  <Accordion title="A scheduled scan completed but no notification arrived">
    Scan notifications share the same delivery path as every other Sencho notification. Check, in order: 1) at least one channel is enabled in **Settings · Notifications · Channels** and its **Test** button succeeds; 2) if the stack the scan is associated with has notification routes defined, at least one matching route is enabled (the routing layer takes priority over the global channels); 3) open the notification bell, where every notification is recorded regardless of channel configuration, and look for a delivery-error entry; 4) on a remote node, channels must be configured on that node itself because channel settings are per-node.
  </Accordion>
  <Accordion title="A Scan Node Images fails with 'Trivy binary is not available'">
    Trivy must be installed on the node that runs the scan, not just on the gateway. Follow [Installing Trivy](/operations/trivy-setup) on the target node, then click **Run now** on the schedule to confirm the scan succeeds before waiting for the next cron tick.
  </Accordion>
  <Accordion title="A Start / Bring Up Stack run fails with a compose file error">
    Start / Bring Up Stack runs `docker compose up -d` against the stack's compose directory. If the stack folder has been removed or its compose file is missing, the run fails. Re-create or restore the stack from the [Editor](/features/editor), then click **Run now** on the schedule to confirm it succeeds before the next cron tick.
  </Accordion>
  <Accordion title="A Backup Stack Compose Files run overwrote the previous backup">
    Backup Stack Compose Files keeps a single slot per stack under `<DATA_DIR>/backups/<stack>/`. Each run overwrites the previous backup. This is by design: for a versioned archive of compose files across the fleet, use a [Create Fleet Snapshot](/features/fleet-backups) task, which retains every snapshot under the snapshots directory.
  </Accordion>
  <Accordion title="A one-shot task disappeared from the list">
    A task with **Delete after successful run** enabled removes itself automatically after the first successful execution, along with its run history. This is expected behavior. If you need a record of the run before deletion, click **Download CSV** in the run history sheet before the next successful run.
  </Accordion>
</AccordionGroup>