mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-07-26 11:49:16 +00:00
Anso
380ed6fd50
* feat(cloud-backup): make Custom S3-compatible target available on every tier Sencho Cloud Backup remains an Admiral feature; the bring-your-own-bucket Custom S3 target is now reachable on Community and Skipper as well. Backend splits the per-route Admiral gate into two helpers: operations that touch the saved provider use gateForCurrentProvider, PUT /config uses gateForRequestedProvider against the body. /provision and /usage stay requireAdmiral because they are Sencho-only by definition; GET /config is ungated so any tier can read its own stored configuration. Frontend drops the AdmiralGate wrapper on the Cloud Backup section, filters the Sencho provider option out of the dropdown for non-Admiral users, and gates the per-snapshot cloud-upload affordance on "cloud-backup configured" instead of Admiral tier. Dashboard Configuration row is no longer locked on lower tiers. Sidebar registry tier on cloud-backup goes from 'admiral' to null. Docs and licensing breakdown restate the rule once per page without fence-spec. * fix(cloud-backup): keep downgraded sencho config off the upload surface If an Admiral configured Sencho Cloud Backup and the license later drops to Skipper or Community, the saved provider is still 'sencho'. The FleetSnapshots cloud-upload affordance now requires either provider= custom (every tier) or provider=sencho with an active Admiral license, so a downgraded admin never sees an upload button that the backend would 403 on click. Also tidies the Fleet Backups doc, which still claimed the cloud-upload icon was Admiral only; the icon now renders whenever a Cloud Backup target is configured.
181 lines
13 KiB
Plaintext
181 lines
13 KiB
Plaintext
---
|
|
title: Fleet-Wide Backups
|
|
description: Snapshot compose files across all nodes for disaster recovery and auditing.
|
|
---
|
|
|
|
<Note>
|
|
Manual fleet snapshots are available on every tier (admin role required). Scheduled fleet snapshots are a Skipper or Admiral feature.
|
|
</Note>
|
|
|
|
Create point-in-time snapshots of every `compose.yaml` and `.env` file across your entire fleet, local and remote nodes alike. Snapshots are stored centrally in Sencho's database and can be browsed, previewed, and restored at any time.
|
|
|
|
## Creating a snapshot
|
|
|
|
1. Navigate to **Fleet** and select the **Snapshots** tab
|
|
2. Click **Create Snapshot**
|
|
3. An inline form appears above the snapshot list with an optional description field (e.g. "Before v2 migration")
|
|
4. Click **Create** to capture files from every reachable node
|
|
|
|
<Frame>
|
|
<img src="/images/fleet-backups/create-snapshot.png" alt="Inline create snapshot form with optional description field, showing Create and Cancel buttons above the snapshot table" />
|
|
</Frame>
|
|
|
|
During creation, Sencho connects to each node in parallel:
|
|
- **Local nodes** are read directly from the compose directory
|
|
- **Remote nodes** are fetched via the Distributed API proxy using the node's API token
|
|
|
|
If a remote node is offline or unreachable, it is **skipped gracefully**. The snapshot still captures data from all reachable nodes, and skipped nodes are recorded with the reason for the failure.
|
|
|
|
### Scheduled snapshots
|
|
|
|
Fleet snapshots can also be created automatically on a recurring schedule. Navigate to the **Schedules** view, create a new scheduled operation, and select **Fleet Snapshot** as the action type. Scheduled snapshots appear in the snapshot list with a "Scheduled snapshot" prefix in their description.
|
|
|
|
## Browsing snapshots
|
|
|
|
The snapshot list shows each snapshot in a table with the following columns:
|
|
|
|
- **Date** - when the snapshot was taken
|
|
- **Description** - your optional label, or a prefix like "Scheduled snapshot" for automated ones. If Cloud Backup is configured, an upload icon in this column marks snapshots that have been mirrored off-site.
|
|
- **Scope** - how many nodes and stacks were captured (e.g. "3 nodes, 21 stacks")
|
|
- **Warnings** - a warning icon with a count if any nodes were skipped, or "None"
|
|
- **Actions** - **View** to open the detail view, a cloud-upload icon for snapshots not yet mirrored to a configured Cloud Backup target, and a delete button for admins
|
|
|
|
<Frame>
|
|
<img src="/images/fleet-backups/browse-snapshots.png" alt="Snapshot list showing paginated rows with date, description with cloud upload indicator, scope, warnings, and action buttons" />
|
|
</Frame>
|
|
|
|
## Snapshot detail view
|
|
|
|
Click **View** on any snapshot to open the detail view. Use **Back to Snapshots** in the top-left to return to the list.
|
|
|
|
The header shows the snapshot's title (or "Untitled Snapshot"), who created it, when, and badge counts for nodes and stacks captured.
|
|
|
|
Below the header, each node appears as a collapsible card. Expand a node to see its stacks, then expand a stack to see individual files. Each file has a **Preview** button that renders the file contents inline.
|
|
|
|
<Frame>
|
|
<img src="/images/fleet-backups/snapshot-detail.png" alt="Snapshot detail view with a node expanded, showing a stack expanded with a compose file, Preview button, and Restore button" />
|
|
</Frame>
|
|
|
|
If any nodes were unreachable during snapshot creation, a warning banner appears at the top of the detail view listing each skipped node and the reason it was skipped.
|
|
|
|
## Restoring from a snapshot
|
|
|
|
Admins can restore individual stacks from any snapshot:
|
|
|
|
1. Open a snapshot's detail view
|
|
2. Expand the node and stack you want to restore
|
|
3. Click **Restore** below the stack's file list
|
|
4. A confirmation dialog appears. Optionally check **Redeploy stack after restore** to immediately apply the restored configuration.
|
|
5. Click **Restore** to confirm
|
|
|
|
<Frame>
|
|
<img src="/images/fleet-backups/restore-dialog.png" alt="Restore confirmation dialog showing the target stack and node name, a redeploy checkbox, and Cancel and Restore buttons" />
|
|
</Frame>
|
|
|
|
Sencho writes the snapshot's files back to the target node:
|
|
- **Local nodes** have files written directly, with the current files backed up first (creating a rollback point via the atomic deployment system)
|
|
- **Remote nodes** receive files via the Distributed API proxy
|
|
|
|
<Warning>
|
|
Restoring overwrites the current compose and environment files on the target node. If atomic deployments are enabled, the current files are backed up before restoration.
|
|
</Warning>
|
|
|
|
## Deleting snapshots
|
|
|
|
Admins can delete snapshots from the list view by clicking the trash icon on the right side of each row. A confirmation dialog asks you to confirm before the snapshot is permanently removed. Deleting a snapshot removes all captured file data from the database. This action cannot be undone.
|
|
|
|
## Cloud Backup
|
|
|
|
<Note>
|
|
Custom S3-compatible storage is available on every tier. Sencho Cloud Backup is an Admiral feature. Configure either in **Settings → System → Cloud Backup**.
|
|
</Note>
|
|
|
|
Cloud Backup mirrors every fleet snapshot to off-site storage so your snapshots survive local disk failure. The Cloud Backup settings page (reached via **Settings → System → Cloud Backup**) shows a header with your current scope, provider, storage used, and total snapshot count in the cloud. Two storage modes are supported.
|
|
|
|
### Sencho Cloud Backup (included)
|
|
|
|
A managed 500 MB allowance backed by Cloudflare R2, included with every Admiral license. Open **Settings → System → Cloud Backup**, choose **Sencho Cloud Backup (Included)**, and click **Activate**. Sencho exchanges your license key for scoped storage credentials and starts replicating new snapshots automatically.
|
|
|
|
Once active, the settings page shows a storage gauge (used / 500 MB and object count), a status message confirming auto-upload is on, and a **Reprovision** button to refresh credentials if needed. You can verify connectivity at any time with the **Test** button.
|
|
|
|
<Frame>
|
|
<img src="/images/cloud-backup/cloud-backup-active.png" alt="Cloud Backup settings page with Sencho Cloud Backup active, showing the storage gauge, auto-upload status, and the Cloud Snapshots list" />
|
|
</Frame>
|
|
|
|
### Custom S3 (BYOB)
|
|
|
|
Bring any S3-compatible bucket: AWS S3, MinIO, Backblaze B2, Wasabi, or your own Cloudflare R2 token. Choose **Custom S3 (BYOB)** in the storage-mode dropdown and fill in:
|
|
|
|
- **Endpoint URL** (e.g. `https://s3.us-east-1.amazonaws.com`, `https://my-minio.example.com:9000`)
|
|
- **Region** (e.g. `us-east-1`, or `auto` for R2)
|
|
- **Bucket** name
|
|
- **Path Prefix** (default `sencho/`)
|
|
- **Access Key ID** and **Secret Access Key**
|
|
- **Auto-upload** toggle
|
|
|
|
Click **Test** to verify connectivity, then **Save**. Secret keys are encrypted at rest. Sencho only sends them to your configured endpoint.
|
|
|
|
<Frame>
|
|
<img src="/images/cloud-backup/cloud-backup-custom.png" alt="Custom S3 configuration form with fields for Endpoint URL, Region, Bucket, Path Prefix, Access Key ID, Secret Access Key, and an Auto-upload toggle" />
|
|
</Frame>
|
|
|
|
### Manual upload vs auto-upload
|
|
|
|
When auto-upload is on, every fleet snapshot is replicated as soon as it is created. Manual snapshots from the **Fleet → Snapshots** view upload asynchronously so the UI returns immediately; scheduled snapshots block on the upload so the task's success status reflects cloud durability.
|
|
|
|
To upload a single snapshot on demand, open the **Snapshots** tab in Fleet View. Each row that hasn't been mirrored yet shows a cloud-upload action in the Actions column. Once a snapshot is in the cloud, an upload icon appears next to its description.
|
|
|
|
### Browsing and downloading cloud snapshots
|
|
|
|
The **Cloud Snapshots** panel in **Settings → System → Cloud Backup** lists every archive currently in your bucket, with size and last-modified timestamp. Click the download icon to save a `.tar.gz` archive locally for off-host disaster recovery. Each archive contains a `metadata.json` describing the snapshot and a `nodes/` tree with the captured compose and environment files, organised by node and stack.
|
|
|
|
### Restoring from a cloud snapshot
|
|
|
|
For in-place rollback, use the **Restore** action on the snapshot detail view as described above; the local copy is the source of truth for live restore. Cloud snapshots cover the disaster-recovery case where the local disk is gone: download the archive, extract it, and bring up a fresh Sencho instance pointed at the recovered files.
|
|
|
|
## Access control
|
|
|
|
| Action | Admin | Node Admin | Deployer | Auditor | Viewer |
|
|
|--------|-------|------------|----------|---------|--------|
|
|
| View snapshot list | Yes | Yes | Yes | Yes | Yes |
|
|
| Browse snapshot contents | Yes | Yes | Yes | Yes | Yes |
|
|
| Create snapshot | Yes | No | No | No | No |
|
|
| Restore from snapshot | Yes | No | No | No | No |
|
|
| Delete snapshot | Yes | No | No | No | No |
|
|
| Upload snapshot to cloud | Yes | No | No | No | No |
|
|
| Delete cloud snapshot | Yes | No | No | No | No |
|
|
|
|
<Note>
|
|
Cloud backup actions to Sencho Cloud Backup require an Admiral license. Cloud backup actions to a Custom S3-compatible target work on every tier.
|
|
</Note>
|
|
|
|
## Storage
|
|
|
|
Snapshots are stored in Sencho's SQLite database. Compose files are typically small (under 10 KB each), so even hundreds of snapshots consume minimal disk space. For very large fleets, consider periodically deleting old snapshots to keep the database lean.
|
|
|
|
## Troubleshooting
|
|
|
|
<AccordionGroup>
|
|
<Accordion title="A snapshot shows skipped nodes">
|
|
If a remote node is offline, unreachable, or its API token has expired, the node is skipped during snapshot creation. The list shows a warning icon with a count of skipped nodes, and the snapshot's detail view names each one along with the reason. Common causes are the remote Sencho instance being stopped or restarting, the node's API URL or token having been changed after it was added, or a firewall or network issue blocking the connection. Verify the remote is running and reachable, update the node's API URL and token in **Settings → System → Nodes** if needed, then create a new snapshot.
|
|
</Accordion>
|
|
<Accordion title="Restore fails with 'Target node no longer exists'">
|
|
The node recorded in the snapshot has been removed from the fleet since the snapshot was taken. Snapshots reference nodes by registry ID, so a node that was deleted and re-added is treated as a different target. Re-add the node first, then retry the restore against the freshly registered row.
|
|
</Accordion>
|
|
<Accordion title="Restore fails with 'No files found for this stack'">
|
|
The stack was not captured in the snapshot, usually because its compose file was missing or unreadable on disk at the time the snapshot was taken. Open the snapshot's detail view to verify which stacks and files are available, and pick a different snapshot if the one you have is incomplete.
|
|
</Accordion>
|
|
<Accordion title="Cloud Backup Test reports an authentication error">
|
|
Double-check the Access Key ID, Secret Access Key, and bucket name; one wrong character is the most common cause. Some providers require S3-compatible API access to be enabled on the bucket separately from the credentials. For MinIO, confirm the user has read/write permission on the target bucket. After correcting the values, click **Test** again before saving.
|
|
</Accordion>
|
|
<Accordion title="Cloud uploads fail after hitting the 500 MB quota">
|
|
Sencho Cloud Backup carries a 500 MB allowance per license. Once you hit the cap, new uploads fail with a quota error and the storage gauge in the settings page reads at or near 500 MB. Free space by deleting older archives from the **Cloud Snapshots** panel in **Settings → System → Cloud Backup**. Local snapshots are unaffected by cloud deletions, so the in-place restore path stays intact. To raise the ceiling, switch the storage mode to **Custom S3 (BYOB)** and point at a bucket you control.
|
|
</Accordion>
|
|
<Accordion title="Cloud upload returns a network timeout or 5xx error">
|
|
Transient errors surface as a notification and leave the local snapshot in place. Retry by clicking the cloud-upload action on the snapshot row, or wait for the next scheduled snapshot which retries on its own. Persistent failures usually indicate an endpoint outage; verify the storage provider is reachable from your Sencho host (custom S3 endpoints often sit behind a different DNS or firewall path than the rest of your traffic).
|
|
</Accordion>
|
|
<Accordion title="I need more detail about a snapshot or restore operation">
|
|
Enable **Developer Mode** under **Settings → Developer** to activate diagnostic logging for snapshot creation (per-node capture timing and file counts), restore operations, and scheduled snapshot execution. Diagnostic logs appear in the server's standard output with a `:debug` suffix.
|
|
</Accordion>
|
|
</AccordionGroup>
|