mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-06 00:47:52 +00:00
Anso
95278843cf
* feat(backend): add stack update-preview endpoint for readiness board Adds GET /api/stacks/:stackName/update-preview that returns per-image semver diff, bump classification, and a stack-level summary powering the Auto-Update readiness board. - New UpdatePreviewService parses compose images, inspects local digests, fetches remote digests and tag lists, and finds the highest compatible semver tag. - Major bumps are flagged blocked until human review; unknown bumps rank below real semver so they cannot mask a major. - Rollback target is reconstructed through parseImageRef to preserve registry ports and drop the Docker Hub library/ prefix. - Registry helpers (httpGet, auth token, digest, tag list, ref parse) are extracted into registry-api.ts and shared with ImageUpdateService. - 28 Vitest cases cover parse, selection, bump math, digest rebuilds, blocked policy, and rollback target construction. * feat(schedules): next-24h timeline, merge auto-update crud, add readiness board Replace the flat task table with a Timeline view as the default, showing the next 24 hours of scheduled work across four lanes (Restart, Update, Scan, Prune) with a live now rail and per-firing pills. The All tasks tab preserves the existing CRUD surface. Merge Auto-update Stack into Schedules as a first-class action and replace the standalone Auto-Update Policies view with a per-stack Readiness board that surfaces version diffs, risk tags, changelog previews, and rollback targets sourced from the stack update-preview endpoint.
222 lines
13 KiB
Plaintext
222 lines
13 KiB
Plaintext
---
|
||
title: Scheduled Operations
|
||
description: Automate recurring Docker operations like stack restarts, fleet snapshots, and system prunes on a cron schedule.
|
||
---
|
||
|
||
<Note>
|
||
Scheduled Operations requires a Sencho **Admiral** license. Skipper users see only the **Auto-update Stack** action; Admiral users see every action.
|
||
</Note>
|
||
|
||
## Overview
|
||
|
||
Scheduled Operations lets you automate recurring maintenance tasks across your infrastructure. Define a cron schedule, choose an action, and Sencho handles the rest, including a full execution history log so you always know what ran and when.
|
||
|
||
The view opens on a **Timeline** that plots the next 24 hours of scheduled work across four lanes (Restart, Update, Scan, Prune) so you can see, at a glance, what is about to fire and when. Toggle to **All tasks** for the full CRUD table.
|
||
|
||
<Frame>
|
||
<img src="/images/scheduled-operations/timeline.png" alt="Schedules timeline showing the next 24 hours across four lanes with a cyan now rail" />
|
||
</Frame>
|
||
|
||
## Timeline view
|
||
|
||
The timeline is the default view. It shows:
|
||
|
||
- A hero with the current 24-hour window as a date range, and the **next firing** on the right (time, task name, and relative countdown).
|
||
- Four color-coded lanes: **Restart** (cyan), **Update** (green), **Scan** (purple), and **Prune** (amber). Snapshot tasks share the Prune lane.
|
||
- One pill per firing within the window, positioned proportionally to the task's next run time. Click any pill to open that task's execution history.
|
||
- A vertical cyan **now rail** at the left edge, and six mono time ticks along the bottom axis.
|
||
|
||
Tasks that fire more than once in the window (e.g. an hourly cron) render a pill for each firing. Disabled tasks do not appear on the timeline.
|
||
|
||
Toggle to **All tasks** from the header to see every schedule in a table, regardless of whether it fires in the next 24 hours.
|
||
|
||
## Supported Actions
|
||
|
||
| Action | Target | Description |
|
||
|--------|--------|-------------|
|
||
| **Restart Stack** | A specific stack (or specific services within it) on a specific node | Restarts all or selected containers in the stack |
|
||
| **Auto-update Stack** | A specific stack on a specific node | Checks each image for updates and recreates the stack if any image has a newer version. See [Auto-Update Readiness](/features/auto-update-policies) for the companion board. Available on Skipper and Admiral. |
|
||
| **Fleet Snapshot** | All nodes | Creates a fleet-wide backup of all compose files and `.env` files |
|
||
| **System Prune** | The default node | Prunes selected resources, optionally filtered by Docker label |
|
||
| **Vulnerability Scan** | All images on a specific node | Runs Trivy against every image on the target node and records the results. Requires Trivy to be installed, see [Installing Trivy](/operations/trivy-setup). Available on Skipper and Admiral. |
|
||
|
||
## Creating a Scheduled Task
|
||
|
||
1. Navigate to the **Schedules** tab in the top navigation bar (visible to Admiral admins).
|
||
2. Click **New Schedule**.
|
||
3. Fill in the form:
|
||
- **Name**: A descriptive label (e.g. "Nightly staging restart").
|
||
- **Action**: Choose Restart Stack, Auto-update Stack, Fleet Snapshot, System Prune, or Vulnerability Scan. The form fields below change based on your selection.
|
||
- **Node**: (Restart Stack, Auto-update Stack, and Vulnerability Scan) Select the node to run against. For stack actions it determines where the target stack lives; for Vulnerability Scan it determines which node's images are scanned.
|
||
- **Stack**: (Restart Stack and Auto-update Stack) Select the stack to target. Becomes available after choosing a node.
|
||
- **Services**: (Restart Stack only) Optionally select specific services within the stack to restart. Leave empty to restart all services.
|
||
- **Prune Targets**: (System Prune only) Select which resources to prune: containers, images, networks, volumes. All are selected by default.
|
||
- **Label Filter**: (System Prune only) Optionally filter prune operations to resources matching a specific Docker label (e.g. `com.docker.compose.project=mystack`).
|
||
- **Cron Expression**: Standard 5-field cron format. A human-readable preview appears below the input.
|
||
- **Enabled**: Toggle the task on or off.
|
||
4. Click **Create**.
|
||
|
||
<Frame>
|
||
<img src="/images/scheduled-operations/create-dialog.png" alt="Create scheduled task dialog with action, cron expression, and prune target options" />
|
||
</Frame>
|
||
|
||
## Task List
|
||
|
||
The task list is displayed as a table with the following columns:
|
||
|
||
| Column | Description |
|
||
|--------|-------------|
|
||
| **Name** | The task name |
|
||
| **Action** | Task type badge: Restart Stack, Fleet Snapshot, System Prune, or Vulnerability Scan |
|
||
| **Target** | Stack name (with selected services, if any), or the target type for non-stack actions |
|
||
| **Schedule** | Human-readable description with the raw cron expression below |
|
||
| **Status** | Last run result: **Success** (green), **Failed** (red), or "Never run" |
|
||
| **Next Run** | When the task will next execute |
|
||
| **Enabled** | Toggle switch to enable or disable the task |
|
||
| **Actions** | Action buttons (see [Managing Tasks](#managing-tasks)) |
|
||
|
||
## Granular Targeting
|
||
|
||
### Per-Service Restart
|
||
|
||
When creating a Restart Stack schedule, you can target individual services instead of restarting the entire stack. After selecting a stack, Sencho reads the compose file and displays checkboxes for each defined service. Select the services you want to restart, or leave all unchecked to restart every service in the stack.
|
||
|
||
<Frame>
|
||
<img src="/images/scheduled-operations/per-service-restart.png" alt="Service checkboxes displayed when creating a per-service restart schedule" />
|
||
</Frame>
|
||
|
||
### Scheduled Vulnerability Scans
|
||
|
||
A Vulnerability Scan task runs Trivy against every image on the selected node and persists the results. The scan uses the same digest-based 24-hour cache as manual scans, so unchanged images are not rescanned on every run. See [Vulnerability Scanning](/features/vulnerability-scanning) for how results are surfaced in the UI and [Installing Trivy](/operations/trivy-setup) for setup on each node.
|
||
|
||
When a scheduled scan finishes, Sencho dispatches a completion notification with a summary of what was scanned and a breakdown of findings by severity. The full message format is documented in [Alerts & Notifications → Scheduled scan completion](/features/alerts-notifications#scheduled-scan-completion).
|
||
|
||
### Prune Label Filter
|
||
|
||
When creating a System Prune schedule, you can scope the prune to resources matching a specific Docker label. This lets you target resources from a particular stack or project without affecting unrelated containers, images, or volumes.
|
||
|
||
Enter a label in `key=value` format (e.g. `com.docker.compose.project=mystack`). Leave the field empty to prune all unused resources of the selected types.
|
||
|
||
<Frame>
|
||
<img src="/images/scheduled-operations/prune-label-filter.png" alt="Label filter input for scoping prune operations to specific Docker labels" />
|
||
</Frame>
|
||
|
||
## Cron Expression Reference
|
||
|
||
Sencho uses standard 5-field cron expressions:
|
||
|
||
```
|
||
┌───────────── minute (0–59)
|
||
│ ┌─────────── hour (0–23)
|
||
│ │ ┌───────── day of month (1–31)
|
||
│ │ │ ┌─────── month (1–12)
|
||
│ │ │ │ ┌───── day of week (0–7, 0 and 7 = Sunday)
|
||
│ │ │ │ │
|
||
* * * * *
|
||
```
|
||
|
||
### Common Examples
|
||
|
||
| Expression | Description |
|
||
|-----------|-------------|
|
||
| `0 3 * * *` | Every day at 3:00 AM |
|
||
| `0 */6 * * *` | Every 6 hours |
|
||
| `0 3 * * 0` | Every Sunday at 3:00 AM |
|
||
| `30 2 1 * *` | 1st of every month at 2:30 AM |
|
||
| `0 0 * * 1-5` | Midnight on weekdays |
|
||
|
||
## Filtering by Node
|
||
|
||
When managing a multi-node fleet, you can filter the schedule list to show only tasks targeting a specific node. There are two ways to access this:
|
||
|
||
- **From the Nodes table:** Click the **calendar icon** on any node row in **Settings → Nodes** to jump directly to the Schedules view filtered to that node.
|
||
- **From the Schedules view:** A filter bar appears at the top showing which node you're viewing, with a **Clear filter** button to return to the full list.
|
||
|
||
When creating a new task while a node filter is active, Sencho pre-selects that node in the create dialog.
|
||
|
||
## Managing Tasks
|
||
|
||
Each task row has four action buttons:
|
||
|
||
- **Run Now** (play icon): Immediately execute the task without waiting for the next scheduled run. Manual runs are labeled "Manual" in the execution history.
|
||
- **Execution History** (clock icon): Open the run history panel for this task.
|
||
- **Edit** (pencil icon): Modify the task name, action, target, or schedule.
|
||
- **Delete** (trash icon): Permanently remove the task and all its execution history after confirmation.
|
||
|
||
Use the **Enabled** toggle switch in the task list to pause or resume a schedule without deleting it.
|
||
|
||
## Failure Notifications
|
||
|
||
When a scheduled task fails, Sencho automatically dispatches an **error-level alert** through your configured notification channels (Discord, Slack, or custom webhooks). The alert includes the task name, action type, and error message so you can diagnose the issue immediately.
|
||
|
||
When a previously failing task succeeds again, Sencho sends an **info-level recovery notification** to confirm the issue is resolved. This recovery-only approach avoids notification noise from tasks that succeed on every run.
|
||
|
||
Vulnerability Scan tasks always send a completion notification, even on a clean run, because the message carries severity counts you may want to react to. See [Alerts & Notifications → Scheduled scan completion](/features/alerts-notifications#scheduled-scan-completion) for the full message format.
|
||
|
||
To configure notification channels, go to **Settings > Notifications**.
|
||
|
||
<Frame>
|
||
<img src="/images/scheduled-operations/failure-notification.png" alt="Failure notification alert shown in the notification bell popover" />
|
||
</Frame>
|
||
|
||
## Execution History
|
||
|
||
Click the clock icon on any task to open the run history panel. The history is displayed as a table with the following columns:
|
||
|
||
| Column | Description |
|
||
|--------|-------------|
|
||
| **Time** | When the run started |
|
||
| **Source** | Whether the run was triggered by the **Scheduler** or **Manual** (via Run Now) |
|
||
| **Status** | Success, Failed, or Running |
|
||
| **Duration** | How long the run took (in seconds) |
|
||
| **Details** | Output summary or error message |
|
||
|
||
Run history is paginated at 20 entries per page. You can export the full history as CSV using the download button in the panel header.
|
||
|
||
Execution history is retained for 30 days.
|
||
|
||
<Frame>
|
||
<img src="/images/scheduled-operations/run-history.png" alt="Execution history showing run source, status, duration, and details with export button" />
|
||
</Frame>
|
||
|
||
## How It Works
|
||
|
||
The Scheduler Service runs in the background and checks for due tasks every 60 seconds. When a task's next run time has passed:
|
||
|
||
1. The scheduler verifies your Admiral license is active.
|
||
2. It executes the configured action using the same internal services that power the UI buttons (restart, snapshot, prune).
|
||
3. Results are logged to the execution history.
|
||
4. On failure, an alert is dispatched via your configured notification channels.
|
||
5. The next run time is recalculated from the cron expression.
|
||
|
||
If a task is still running from a previous execution, the scheduler skips it to prevent overlap.
|
||
|
||
If the server restarts while a task is mid-execution, the orphaned run record is automatically marked as failed with a "Server restarted during execution" message on the next startup. No manual cleanup is needed.
|
||
|
||
## Troubleshooting
|
||
|
||
### Task was automatically disabled
|
||
|
||
If a task's cron expression becomes invalid after creation (for example, due to a corrupt database edit), the scheduler disables the task and records the reason in the last error column. To fix this:
|
||
|
||
1. Open the task in edit mode.
|
||
2. Re-enter a valid cron expression.
|
||
3. Re-enable the task using the toggle switch.
|
||
|
||
### Run shows "Server restarted during execution"
|
||
|
||
This means Sencho was restarted (or crashed) while this task was mid-execution. The run was marked as failed automatically on startup. The task itself is still enabled and will run at its next scheduled time. If you want to re-run it immediately, use the **Run Now** button.
|
||
|
||
### Scheduled scan completed but no notification arrived
|
||
|
||
The scan notification shares the same delivery path as every other Sencho alert. Check, in order:
|
||
|
||
1. At least one channel is enabled in **Settings > Notifications** and the **Test** button succeeds for it.
|
||
2. If the stack the scan is associated with has notification routes defined, make sure at least one matching route is enabled; the routing layer takes priority over the global channels.
|
||
3. Open the notification bell. The in-app bell receives every notification regardless of channel configuration, and a failed external delivery is logged there as an error entry you can inspect.
|
||
4. On a remote node, notification channels must be configured on the remote instance itself; channel settings are per-node.
|
||
|
||
### Scan task fails with "Trivy binary is not available"
|
||
|
||
Trivy must be installed on the node that runs the scan, not just on the primary instance. Follow [Installing Trivy](/operations/trivy-setup) on the target node, then trigger **Run Now** from the schedule to confirm the scan succeeds before waiting for the next cron tick.
|