mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-06 00:47:52 +00:00
Anso
44e8fdfba9
* fix(scheduler): clean up stale runs on startup and auto-disable invalid cron tasks
- Add markStaleRunsAsFailed() bulk DB method with status index
- Clean up orphaned 'running' records on scheduler startup
- Auto-disable tasks when cron expression becomes invalid at execution time
- Promote CRUD debug logs to standard logs for scheduled task admin actions
- Add diagnostic logging for task pre-checks, action timing, and prune fallback
* refactor(scheduling): extract shared types and fix design system violations
- Extract ScheduledTask, TaskRun, NodeOption to shared types file
- Extract getCronDescription and formatTimestamp to shared utilities
- Fix formatTimestamp falsy-zero null check
- Tighten last_status type to 'success' | 'failure' | null
- Add strokeWidth={1.5} to all action icons per design system
- Add sr-only DialogDescription for accessibility
- Wrap Sheet run history in ScrollArea
- Fix delete button styling to match design system pattern
- Change manual trigger toast from "executed" to "triggered"
* test(scheduler): add tests for snapshot, remote update, stale cleanup, and cron invalidation
- Add stale run cleanup tests (bulk markStaleRunsAsFailed, logging)
- Add cron invalidation test (auto-disable, error message)
- Add executeSnapshot tests (fleet capture, empty stacks)
- Add executeUpdateRemote tests (proxy success, remote error)
- Document stale run cleanup and cron auto-disable in troubleshooting docs
185 lines
9.2 KiB
Plaintext
185 lines
9.2 KiB
Plaintext
---
|
||
title: Scheduled Operations
|
||
description: Automate recurring Docker operations like stack restarts, fleet snapshots, and system prunes on a cron schedule.
|
||
---
|
||
|
||
<Note>
|
||
Scheduled Operations requires a Sencho **Admiral** license.
|
||
Skipper and Community Edition do not include this feature.
|
||
</Note>
|
||
|
||
## Overview
|
||
|
||
Scheduled Operations lets you automate recurring maintenance tasks across your infrastructure. Define a cron schedule, choose an action, and Sencho handles the rest, including a full execution history log so you always know what ran and when.
|
||
|
||
<Frame>
|
||
<img src="/images/scheduled-operations/overview.png" alt="Scheduled operations list view showing tasks with status, schedule, and actions" />
|
||
</Frame>
|
||
|
||
## Supported Actions
|
||
|
||
| Action | Target | Description |
|
||
|--------|--------|-------------|
|
||
| **Restart Stack** | A specific stack (or specific services within it) on a specific node | Restarts all or selected containers in the stack |
|
||
| **Fleet Snapshot** | All nodes | Creates a fleet-wide backup of all compose files and `.env` files |
|
||
| **System Prune** | The default node | Prunes selected resources, optionally filtered by Docker label |
|
||
|
||
## Creating a Scheduled Task
|
||
|
||
1. Navigate to the **Schedules** tab in the top navigation bar (visible to Admiral admins).
|
||
2. Click **New Schedule**.
|
||
3. Fill in the form:
|
||
- **Name**: A descriptive label (e.g. "Nightly staging restart").
|
||
- **Action**: Choose Restart Stack, Fleet Snapshot, or System Prune. The form fields below change based on your selection.
|
||
- **Node**: (Restart Stack only) Select the node where the target stack runs.
|
||
- **Stack**: (Restart Stack only) Select the stack to restart. Becomes available after choosing a node.
|
||
- **Services**: (Restart Stack only) Optionally select specific services within the stack to restart. Leave empty to restart all services.
|
||
- **Prune Targets**: (System Prune only) Select which resources to prune: containers, images, networks, volumes. All are selected by default.
|
||
- **Label Filter**: (System Prune only) Optionally filter prune operations to resources matching a specific Docker label (e.g. `com.docker.compose.project=mystack`).
|
||
- **Cron Expression**: Standard 5-field cron format. A human-readable preview appears below the input.
|
||
- **Enabled**: Toggle the task on or off.
|
||
4. Click **Create**.
|
||
|
||
<Frame>
|
||
<img src="/images/scheduled-operations/create-dialog.png" alt="Create scheduled task dialog with action, cron expression, and prune target options" />
|
||
</Frame>
|
||
|
||
## Task List
|
||
|
||
The task list is displayed as a table with the following columns:
|
||
|
||
| Column | Description |
|
||
|--------|-------------|
|
||
| **Name** | The task name |
|
||
| **Action** | Task type badge: Restart Stack, Fleet Snapshot, or System Prune |
|
||
| **Target** | Stack name (with selected services, if any), or the target type for non-stack actions |
|
||
| **Schedule** | Human-readable description with the raw cron expression below |
|
||
| **Status** | Last run result: **Success** (green), **Failed** (red), or "Never run" |
|
||
| **Next Run** | When the task will next execute |
|
||
| **Enabled** | Toggle switch to enable or disable the task |
|
||
| **Actions** | Action buttons (see [Managing Tasks](#managing-tasks)) |
|
||
|
||
## Granular Targeting
|
||
|
||
### Per-Service Restart
|
||
|
||
When creating a Restart Stack schedule, you can target individual services instead of restarting the entire stack. After selecting a stack, Sencho reads the compose file and displays checkboxes for each defined service. Select the services you want to restart, or leave all unchecked to restart every service in the stack.
|
||
|
||
<Frame>
|
||
<img src="/images/scheduled-operations/per-service-restart.png" alt="Service checkboxes displayed when creating a per-service restart schedule" />
|
||
</Frame>
|
||
|
||
### Prune Label Filter
|
||
|
||
When creating a System Prune schedule, you can scope the prune to resources matching a specific Docker label. This lets you target resources from a particular stack or project without affecting unrelated containers, images, or volumes.
|
||
|
||
Enter a label in `key=value` format (e.g. `com.docker.compose.project=mystack`). Leave the field empty to prune all unused resources of the selected types.
|
||
|
||
<Frame>
|
||
<img src="/images/scheduled-operations/prune-label-filter.png" alt="Label filter input for scoping prune operations to specific Docker labels" />
|
||
</Frame>
|
||
|
||
## Cron Expression Reference
|
||
|
||
Sencho uses standard 5-field cron expressions:
|
||
|
||
```
|
||
┌───────────── minute (0–59)
|
||
│ ┌─────────── hour (0–23)
|
||
│ │ ┌───────── day of month (1–31)
|
||
│ │ │ ┌─────── month (1–12)
|
||
│ │ │ │ ┌───── day of week (0–7, 0 and 7 = Sunday)
|
||
│ │ │ │ │
|
||
* * * * *
|
||
```
|
||
|
||
### Common Examples
|
||
|
||
| Expression | Description |
|
||
|-----------|-------------|
|
||
| `0 3 * * *` | Every day at 3:00 AM |
|
||
| `0 */6 * * *` | Every 6 hours |
|
||
| `0 3 * * 0` | Every Sunday at 3:00 AM |
|
||
| `30 2 1 * *` | 1st of every month at 2:30 AM |
|
||
| `0 0 * * 1-5` | Midnight on weekdays |
|
||
|
||
## Filtering by Node
|
||
|
||
When managing a multi-node fleet, you can filter the schedule list to show only tasks targeting a specific node. There are two ways to access this:
|
||
|
||
- **From the Nodes table:** Click the **calendar icon** on any node row in **Settings → Nodes** to jump directly to the Schedules view filtered to that node.
|
||
- **From the Schedules view:** A filter bar appears at the top showing which node you're viewing, with a **Clear filter** button to return to the full list.
|
||
|
||
When creating a new task while a node filter is active, Sencho pre-selects that node in the create dialog.
|
||
|
||
## Managing Tasks
|
||
|
||
Each task row has four action buttons:
|
||
|
||
- **Run Now** (play icon): Immediately execute the task without waiting for the next scheduled run. Manual runs are labeled "Manual" in the execution history.
|
||
- **Execution History** (clock icon): Open the run history panel for this task.
|
||
- **Edit** (pencil icon): Modify the task name, action, target, or schedule.
|
||
- **Delete** (trash icon): Permanently remove the task and all its execution history after confirmation.
|
||
|
||
Use the **Enabled** toggle switch in the task list to pause or resume a schedule without deleting it.
|
||
|
||
## Failure Notifications
|
||
|
||
When a scheduled task fails, Sencho automatically dispatches an **error-level alert** through your configured notification channels (Discord, Slack, or custom webhooks). The alert includes the task name, action type, and error message so you can diagnose the issue immediately.
|
||
|
||
When a previously failing task succeeds again, Sencho sends an **info-level recovery notification** to confirm the issue is resolved. This recovery-only approach avoids notification noise from tasks that succeed on every run.
|
||
|
||
To configure notification channels, go to **Settings > Notifications**.
|
||
|
||
<Frame>
|
||
<img src="/images/scheduled-operations/failure-notification.png" alt="Failure notification alert shown in the notification bell popover" />
|
||
</Frame>
|
||
|
||
## Execution History
|
||
|
||
Click the clock icon on any task to open the run history panel. The history is displayed as a table with the following columns:
|
||
|
||
| Column | Description |
|
||
|--------|-------------|
|
||
| **Time** | When the run started |
|
||
| **Source** | Whether the run was triggered by the **Scheduler** or **Manual** (via Run Now) |
|
||
| **Status** | Success, Failed, or Running |
|
||
| **Duration** | How long the run took (in seconds) |
|
||
| **Details** | Output summary or error message |
|
||
|
||
Run history is paginated at 20 entries per page. You can export the full history as CSV using the download button in the panel header.
|
||
|
||
Execution history is retained for 30 days.
|
||
|
||
<Frame>
|
||
<img src="/images/scheduled-operations/run-history.png" alt="Execution history showing run source, status, duration, and details with export button" />
|
||
</Frame>
|
||
|
||
## How It Works
|
||
|
||
The Scheduler Service runs in the background and checks for due tasks every 60 seconds. When a task's next run time has passed:
|
||
|
||
1. The scheduler verifies your Admiral license is active.
|
||
2. It executes the configured action using the same internal services that power the UI buttons (restart, snapshot, prune).
|
||
3. Results are logged to the execution history.
|
||
4. On failure, an alert is dispatched via your configured notification channels.
|
||
5. The next run time is recalculated from the cron expression.
|
||
|
||
If a task is still running from a previous execution, the scheduler skips it to prevent overlap.
|
||
|
||
If the server restarts while a task is mid-execution, the orphaned run record is automatically marked as failed with a "Server restarted during execution" message on the next startup. No manual cleanup is needed.
|
||
|
||
## Troubleshooting
|
||
|
||
### Task was automatically disabled
|
||
|
||
If a task's cron expression becomes invalid after creation (for example, due to a corrupt database edit or an expression that was valid in a previous version), the scheduler disables the task and records the reason in the last error column. To fix this:
|
||
|
||
1. Open the task in edit mode.
|
||
2. Re-enter a valid cron expression.
|
||
3. Re-enable the task using the toggle switch.
|
||
|
||
### Run shows "Server restarted during execution"
|
||
|
||
This means Sencho was restarted (or crashed) while this task was mid-execution. The run was marked as failed automatically on startup. The task itself is still enabled and will run at its next scheduled time. If you want to re-run it immediately, use the **Run Now** button.
|