mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-07-26 11:49:16 +00:00
Anso
809bf76c20
Extract duplicated snapshot capture functions from index.ts and SchedulerService.ts into a shared module (snapshot-capture.ts). Fix audit log route patterns that used singular 'snapshot' instead of plural 'snapshots'. Apply design system to FleetSnapshots component (card styling, strokeWidth, font-mono, tabular-nums, ScrollArea). Add safePage pagination, loading toast for creation, and fix the restore dialog race condition with a controlled AlertDialog. Add diagnostic logging gated behind Developer Mode. Add tests for restore endpoint, admin role enforcement, and edge cases. Update docs with troubleshooting section and refresh screenshots.
123 lines
6.6 KiB
Plaintext
123 lines
6.6 KiB
Plaintext
---
|
|
title: Fleet-Wide Backups
|
|
description: Snapshot compose files across all nodes for disaster recovery and auditing.
|
|
---
|
|
|
|
<Note>
|
|
Fleet-Wide Backups require a Skipper or Admiral license. The feature is available to admins in the Fleet View.
|
|
</Note>
|
|
|
|
Create point-in-time snapshots of every `compose.yaml` and `.env` file across your entire fleet, local and remote nodes alike. Snapshots are stored centrally in Sencho's database and can be browsed, previewed, and restored at any time.
|
|
|
|
## Creating a snapshot
|
|
|
|
1. Navigate to **Fleet View** and select the **Snapshots** tab
|
|
2. Click **Create Snapshot** in the top-right corner
|
|
3. An inline form appears with an optional description field (e.g. "Before v2 migration")
|
|
4. Click **Create** to capture files from every reachable node
|
|
|
|
<Frame>
|
|
<img src="/images/fleet-backups/create-snapshot.png" alt="Inline form for creating a fleet snapshot with an optional description field" />
|
|
</Frame>
|
|
|
|
During creation, Sencho connects to each node in parallel:
|
|
- **Local nodes** are read directly from the compose directory
|
|
- **Remote nodes** are fetched via the Distributed API proxy using the node's API token
|
|
|
|
If a remote node is offline or unreachable, it is **skipped gracefully**. The snapshot still captures data from all reachable nodes, and skipped nodes are recorded with the reason for the failure.
|
|
|
|
### Scheduled snapshots
|
|
|
|
Fleet snapshots can also be created automatically on a recurring schedule. Navigate to the **Schedules** view, create a new scheduled operation, and select **Fleet Snapshot** as the action type. Scheduled snapshots appear in the snapshot list with a "Scheduled snapshot" prefix in their description.
|
|
|
|
## Browsing snapshots
|
|
|
|
The snapshot list shows each snapshot in a table with the following columns:
|
|
|
|
- **Date** - when the snapshot was taken
|
|
- **Description** - your optional label, or a prefix like "Scheduled snapshot" for automated ones
|
|
- **Scope** - how many nodes and stacks were captured (e.g. "2 nodes, 22 stacks")
|
|
- **Warnings** - a warning icon with a count if any nodes were skipped, or "None"
|
|
- **Actions** - **View** to open the detail view, and a delete button for admins
|
|
|
|
<Frame>
|
|
<img src="/images/fleet-backups/browse-snapshots.png" alt="Snapshot list showing date, description, scope, warnings, and action buttons" />
|
|
</Frame>
|
|
|
|
## Snapshot detail view
|
|
|
|
Click **View** on any snapshot to open the detail view. The header card displays the snapshot's title (or "Untitled Snapshot"), who created it, when, and badge counts for nodes and stacks captured.
|
|
|
|
Below the header, each node appears as a collapsible card. Expand a node to see its stacks, then expand a stack to see individual files. Each file has a **Preview** button that renders the file contents inline.
|
|
|
|
<Frame>
|
|
<img src="/images/fleet-backups/snapshot-detail.png" alt="Snapshot detail view showing a collapsible tree of nodes, stacks, and files with preview buttons" />
|
|
</Frame>
|
|
|
|
If any nodes were unreachable during snapshot creation, a warning banner appears at the top of the detail view listing each skipped node and the reason it was skipped.
|
|
|
|
## Restoring from a snapshot
|
|
|
|
Admins can restore individual stacks from any snapshot:
|
|
|
|
1. Open a snapshot's detail view
|
|
2. Expand the node and stack you want to restore
|
|
3. Click **Restore** below the file list
|
|
4. A confirmation dialog appears. Optionally check **Redeploy stack after restore** to immediately apply the restored configuration.
|
|
5. Click **Restore** to confirm
|
|
|
|
<Frame>
|
|
<img src="/images/fleet-backups/restore-dialog.png" alt="Restore confirmation dialog with a redeploy checkbox" />
|
|
</Frame>
|
|
|
|
Sencho writes the snapshot's files back to the target node:
|
|
- **Local nodes** have files written directly, with the current files backed up first (creating a rollback point via the atomic deployment system)
|
|
- **Remote nodes** receive files via the Distributed API proxy
|
|
|
|
<Warning>
|
|
Restoring overwrites the current compose and environment files on the target node. If atomic deployments are enabled, the current files are backed up before restoration.
|
|
</Warning>
|
|
|
|
## Deleting snapshots
|
|
|
|
Admins can delete snapshots from the list view by clicking the trash icon on the right side of each row. A confirmation dialog asks you to confirm before the snapshot is permanently removed. Deleting a snapshot removes all captured file data from the database. This action cannot be undone.
|
|
|
|
## Access control
|
|
|
|
| Action | Admin | Node Admin | Deployer | Auditor | Viewer |
|
|
|--------|-------|------------|----------|---------|--------|
|
|
| View snapshot list | Yes | Yes | Yes | Yes | Yes |
|
|
| Browse snapshot contents | Yes | Yes | Yes | Yes | Yes |
|
|
| Create snapshot | Yes | No | No | No | No |
|
|
| Restore from snapshot | Yes | No | No | No | No |
|
|
| Delete snapshot | Yes | No | No | No | No |
|
|
|
|
## Storage
|
|
|
|
Snapshots are stored in Sencho's SQLite database. Compose files are typically small (under 10 KB each), so even hundreds of snapshots consume minimal disk space. For very large fleets, consider periodically deleting old snapshots to keep the database lean.
|
|
|
|
## Troubleshooting
|
|
|
|
### Snapshot shows skipped nodes
|
|
|
|
If a remote node is offline, unreachable, or its API token has expired, the node is skipped during snapshot creation. The snapshot list shows a warning icon with a count of skipped nodes. Open the snapshot's detail view to see which nodes were skipped and the reason for each.
|
|
|
|
Common causes:
|
|
- The remote Sencho instance is stopped or restarting
|
|
- The node's API URL or token was changed after it was added
|
|
- A firewall or network issue is blocking the connection between nodes
|
|
|
|
To resolve, verify the remote node is running and reachable, then update the node's API URL and token in the Fleet settings if needed. Create a new snapshot after fixing the connectivity issue.
|
|
|
|
### Restore fails with "Target node no longer exists"
|
|
|
|
This error occurs when the node recorded in the snapshot has been removed from your fleet since the snapshot was taken. You cannot restore to a node that is no longer registered. Re-add the node first, then retry the restore.
|
|
|
|
### Restore fails with "No files found for this stack"
|
|
|
|
The stack you're trying to restore may not have been captured in the snapshot (for example, if the compose file was missing or unreadable at the time the snapshot was taken). Open the snapshot's detail view to verify which stacks and files are available.
|
|
|
|
### Diagnostic logging
|
|
|
|
If you need to investigate snapshot operations in detail, enable **Developer Mode** in **Settings > Developer**. This activates diagnostic logging for snapshot creation (per-node capture timing and file counts), restore operations, and scheduled snapshot execution. Diagnostic logs appear in the server's standard output with a `:debug` suffix.
|