mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-07-26 11:49:16 +00:00
Anso
ecf4dd5d52
Realign tier guards to the user-stated philosophy: Community covers
deploy/monitor at scale plus security basics, Skipper adds automation
and advanced fleet management, Admiral keeps enterprise control.
Community now includes:
- Trivy install / uninstall / update from the Settings Hub (admin role)
- CVE suppressions CRUD (admin role; replicates fleet-wide)
- Manual image scan with vuln, secret, and misconfig results
- Stack-config scan, scan comparison
- Manual fleet snapshots: create, list, view, restore, delete
- Per-node Sencho self-update (Check Updates + per-node Update)
- Fleet Overview search, sort, filters, node-card expand, auto-refresh
Stays paid:
- Scan policies with block_on_deploy enforcement (Skipper+)
- SBOM (SPDX, CycloneDX), SARIF export (Skipper+)
- Bulk Update All across the fleet (Skipper+)
- Scheduled snapshot create (now Skipper, was Admiral)
- Trivy auto-update toggle, fleet-wide policy push (Admiral)
The Settings -> Security tab is unhidden by setting the registry tier to
null. The SecuritySection no longer early-returns a PaidGate; the policy
list, Add Policy button, and policy dialogs are wrapped in {isPaid && }.
The Fleet view drops isPaid gates on the Snapshots tab, Check Updates
button, per-node update handlers, OverviewToolbar grid controls, the
NodeCard expand affordance, and the auto-refresh notice. The
NodeUpdatesSheet receives a canBulkUpdate prop and gates the Update All
button on it. useFleetUpdateStatus and useFleetPolling drop their isPaid
guards so polling runs for Community; useFleetOverview drops the isPaid
wrap on the filter and sort path.
Backend route guards are flipped per the matrix above. The scheduler
tick and requireScheduledTaskTier add 'snapshot' to the Skipper+ branch.
Backend test assertions are inverted for the now-Community endpoints
and a positive Skipper-snapshot-task test is added.
Documentation across features/, api-reference/, and operations/ is
updated to reflect the new tier mapping.
176 lines
10 KiB
Plaintext
176 lines
10 KiB
Plaintext
---
|
|
title: Fleet-Wide Backups
|
|
description: Snapshot compose files across all nodes for disaster recovery and auditing.
|
|
---
|
|
|
|
<Note>
|
|
Manual fleet snapshots are available on every tier (admin role required). Scheduled fleet snapshots are a Skipper or Admiral feature.
|
|
</Note>
|
|
|
|
Create point-in-time snapshots of every `compose.yaml` and `.env` file across your entire fleet, local and remote nodes alike. Snapshots are stored centrally in Sencho's database and can be browsed, previewed, and restored at any time.
|
|
|
|
## Creating a snapshot
|
|
|
|
1. Navigate to **Fleet View** and select the **Snapshots** tab
|
|
2. Click **Create Snapshot** in the top-right corner
|
|
3. An inline form appears with an optional description field (e.g. "Before v2 migration")
|
|
4. Click **Create** to capture files from every reachable node
|
|
|
|
<Frame>
|
|
<img src="/images/fleet-backups/create-snapshot.png" alt="Inline form for creating a fleet snapshot with an optional description field" />
|
|
</Frame>
|
|
|
|
During creation, Sencho connects to each node in parallel:
|
|
- **Local nodes** are read directly from the compose directory
|
|
- **Remote nodes** are fetched via the Distributed API proxy using the node's API token
|
|
|
|
If a remote node is offline or unreachable, it is **skipped gracefully**. The snapshot still captures data from all reachable nodes, and skipped nodes are recorded with the reason for the failure.
|
|
|
|
### Scheduled snapshots
|
|
|
|
Fleet snapshots can also be created automatically on a recurring schedule. Navigate to the **Schedules** view, create a new scheduled operation, and select **Fleet Snapshot** as the action type. Scheduled snapshots appear in the snapshot list with a "Scheduled snapshot" prefix in their description.
|
|
|
|
## Browsing snapshots
|
|
|
|
The snapshot list shows each snapshot in a table with the following columns:
|
|
|
|
- **Date** - when the snapshot was taken
|
|
- **Description** - your optional label, or a prefix like "Scheduled snapshot" for automated ones
|
|
- **Scope** - how many nodes and stacks were captured (e.g. "2 nodes, 22 stacks")
|
|
- **Warnings** - a warning icon with a count if any nodes were skipped, or "None"
|
|
- **Actions** - **View** to open the detail view, and a delete button for admins
|
|
|
|
<Frame>
|
|
<img src="/images/fleet-backups/browse-snapshots.png" alt="Snapshot list showing date, description, scope, warnings, and action buttons" />
|
|
</Frame>
|
|
|
|
## Snapshot detail view
|
|
|
|
Click **View** on any snapshot to open the detail view. The header card displays the snapshot's title (or "Untitled Snapshot"), who created it, when, and badge counts for nodes and stacks captured.
|
|
|
|
Below the header, each node appears as a collapsible card. Expand a node to see its stacks, then expand a stack to see individual files. Each file has a **Preview** button that renders the file contents inline.
|
|
|
|
<Frame>
|
|
<img src="/images/fleet-backups/snapshot-detail.png" alt="Snapshot detail view showing a collapsible tree of nodes, stacks, and files with preview buttons" />
|
|
</Frame>
|
|
|
|
If any nodes were unreachable during snapshot creation, a warning banner appears at the top of the detail view listing each skipped node and the reason it was skipped.
|
|
|
|
## Restoring from a snapshot
|
|
|
|
Admins can restore individual stacks from any snapshot:
|
|
|
|
1. Open a snapshot's detail view
|
|
2. Expand the node and stack you want to restore
|
|
3. Click **Restore** below the file list
|
|
4. A confirmation dialog appears. Optionally check **Redeploy stack after restore** to immediately apply the restored configuration.
|
|
5. Click **Restore** to confirm
|
|
|
|
<Frame>
|
|
<img src="/images/fleet-backups/restore-dialog.png" alt="Restore confirmation dialog with a redeploy checkbox" />
|
|
</Frame>
|
|
|
|
Sencho writes the snapshot's files back to the target node:
|
|
- **Local nodes** have files written directly, with the current files backed up first (creating a rollback point via the atomic deployment system)
|
|
- **Remote nodes** receive files via the Distributed API proxy
|
|
|
|
<Warning>
|
|
Restoring overwrites the current compose and environment files on the target node. If atomic deployments are enabled, the current files are backed up before restoration.
|
|
</Warning>
|
|
|
|
## Deleting snapshots
|
|
|
|
Admins can delete snapshots from the list view by clicking the trash icon on the right side of each row. A confirmation dialog asks you to confirm before the snapshot is permanently removed. Deleting a snapshot removes all captured file data from the database. This action cannot be undone.
|
|
|
|
## Cloud Backup
|
|
|
|
<Note>
|
|
Cloud Backup requires an Admiral license. Configure it in **Settings → Cloud Backup**.
|
|
</Note>
|
|
|
|
Cloud Backup mirrors every fleet snapshot to off-site storage so your snapshots survive local disk failure. Two storage modes are supported.
|
|
|
|
### Sencho Cloud Backup (included)
|
|
|
|
A managed 500 MB allowance backed by Cloudflare R2, included with every Admiral license. Open **Settings → Cloud Backup**, choose **Sencho Cloud Backup**, and click **Activate**. Sencho exchanges your license key for scoped storage credentials and starts replicating new snapshots automatically. The settings panel shows your storage usage and lets you reprovision credentials if needed.
|
|
|
|
### Custom S3 (BYOB)
|
|
|
|
Bring any S3-compatible bucket: AWS S3, MinIO, Backblaze B2, Wasabi, or your own Cloudflare R2 token. Choose **Custom S3** in the storage-mode dropdown and fill in:
|
|
|
|
- **Endpoint URL** (e.g. `https://s3.us-east-1.amazonaws.com`, `https://my-minio.example.com:9000`)
|
|
- **Region** (e.g. `us-east-1`, or `auto` for R2)
|
|
- **Bucket** name
|
|
- **Path Prefix** (default `sencho/`)
|
|
- **Access Key ID** and **Secret Access Key**
|
|
- **Auto-upload** toggle
|
|
|
|
Click **Test** to verify connectivity, then **Save**. Secret keys are encrypted at rest. Sencho only sends them to your configured endpoint.
|
|
|
|
### Manual upload vs auto-upload
|
|
|
|
When auto-upload is on, every fleet snapshot is replicated as soon as it is created. Manual snapshots from the **Fleet → Snapshots** view upload asynchronously so the UI returns immediately; scheduled snapshots block on the upload so the task's success status reflects cloud durability.
|
|
|
|
To upload a single snapshot on demand, open the **Snapshots** tab in Fleet View. Each row that hasn't been mirrored yet shows a cloud-upload action next to the **View** button. Once a snapshot is in the cloud, a small cloud icon appears next to its description.
|
|
|
|
### Browsing and downloading cloud snapshots
|
|
|
|
The **Cloud Snapshots** panel in **Settings → Cloud Backup** lists every archive currently in your bucket, with size and last-modified timestamp. Click the download icon to save a `.tar.gz` archive locally for off-host disaster recovery. Each archive contains a `metadata.json` describing the snapshot and a `nodes/` tree with the captured compose and environment files, organised by node and stack.
|
|
|
|
### Restoring from a cloud snapshot
|
|
|
|
For in-place rollback, use the **Restore** action on the snapshot detail view as described above; the local copy is the source of truth for live restore. Cloud snapshots cover the disaster-recovery case where the local disk is gone: download the archive, extract it, and bring up a fresh Sencho instance pointed at the recovered files.
|
|
|
|
### Cloud Backup troubleshooting
|
|
|
|
#### Bad credentials
|
|
|
|
If **Test** reports an authentication error, double-check the Access Key ID, Secret Access Key, and bucket. Some providers require you to enable S3-compatible API access on the bucket separately. For MinIO, verify the user has read/write permission on the target bucket.
|
|
|
|
#### Over quota (Sencho Cloud Backup)
|
|
|
|
Sencho Cloud Backup has a 500 MB allowance per license. When you hit the cap, new uploads fail with a quota error. Delete older cloud snapshots from the **Cloud Snapshots** panel to free space. The local copies are unaffected.
|
|
|
|
#### Network timeout or 5xx error
|
|
|
|
Transient errors surface as a notification. Retry by clicking the cloud-upload action on the snapshot row, or wait for the next scheduled snapshot which will retry on its own. Persistent failures usually indicate an endpoint outage; verify the storage provider is reachable from your Sencho host.
|
|
|
|
## Access control
|
|
|
|
| Action | Admin | Node Admin | Deployer | Auditor | Viewer |
|
|
|--------|-------|------------|----------|---------|--------|
|
|
| View snapshot list | Yes | Yes | Yes | Yes | Yes |
|
|
| Browse snapshot contents | Yes | Yes | Yes | Yes | Yes |
|
|
| Create snapshot | Yes | No | No | No | No |
|
|
| Restore from snapshot | Yes | No | No | No | No |
|
|
| Delete snapshot | Yes | No | No | No | No |
|
|
|
|
## Storage
|
|
|
|
Snapshots are stored in Sencho's SQLite database. Compose files are typically small (under 10 KB each), so even hundreds of snapshots consume minimal disk space. For very large fleets, consider periodically deleting old snapshots to keep the database lean.
|
|
|
|
## Troubleshooting
|
|
|
|
### Snapshot shows skipped nodes
|
|
|
|
If a remote node is offline, unreachable, or its API token has expired, the node is skipped during snapshot creation. The snapshot list shows a warning icon with a count of skipped nodes. Open the snapshot's detail view to see which nodes were skipped and the reason for each.
|
|
|
|
Common causes:
|
|
- The remote Sencho instance is stopped or restarting
|
|
- The node's API URL or token was changed after it was added
|
|
- A firewall or network issue is blocking the connection between nodes
|
|
|
|
To resolve, verify the remote node is running and reachable, then update the node's API URL and token in the Fleet settings if needed. Create a new snapshot after fixing the connectivity issue.
|
|
|
|
### Restore fails with "Target node no longer exists"
|
|
|
|
This error occurs when the node recorded in the snapshot has been removed from your fleet since the snapshot was taken. You cannot restore to a node that is no longer registered. Re-add the node first, then retry the restore.
|
|
|
|
### Restore fails with "No files found for this stack"
|
|
|
|
The stack you're trying to restore may not have been captured in the snapshot (for example, if the compose file was missing or unreadable at the time the snapshot was taken). Open the snapshot's detail view to verify which stacks and files are available.
|
|
|
|
### Diagnostic logging
|
|
|
|
If you need to investigate snapshot operations in detail, enable **Developer Mode** in **Settings > Developer**. This activates diagnostic logging for snapshot creation (per-node capture timing and file counts), restore operations, and scheduled snapshot execution. Diagnostic logs appear in the server's standard output with a `:debug` suffix.
|