---
title: Fleet-Wide Backups
sidebarTitle: Fleet backups
description: Snapshot compose files across all nodes for disaster recovery and auditing.
---

<Note>
  Manual and scheduled fleet snapshots are available on every tier (admin role required).
</Note>

Create point-in-time snapshots of every `compose.yaml` and `.env` file across your entire fleet, local and remote nodes alike. Snapshots are stored centrally in Sencho's database and can be browsed, previewed, and restored at any time.

## Creating a snapshot

1. Navigate to **Fleet** and select the **Snapshots** tab
2. Click **Create Snapshot**
3. An inline form appears above the snapshot list with an optional description field (e.g. "Before v2 migration")
4. Click **Create** to capture files from every reachable node

<Frame>
  <img src="/images/fleet-backups/create-snapshot.png" alt="Inline create snapshot form with optional description field, showing Create and Cancel buttons above the snapshot table" />
</Frame>

During creation, Sencho connects to each node in parallel:
- **Local nodes** are read directly from the compose directory
- **Remote nodes** are fetched via the Distributed API proxy using the node's API token

If a remote node is offline or unreachable, it is **skipped gracefully**. The snapshot still captures data from all reachable nodes, and skipped nodes are recorded with the reason for the failure. Individual stacks that could not be captured (for example, a compose file that failed to read, or a file too large to store) are recorded the same way, so a snapshot is never silently incomplete.

### Scheduled snapshots

Fleet snapshots can also be created automatically on a recurring schedule. Navigate to the **Schedules** view, create a new scheduled operation, and select **Create Fleet Snapshot** as the action type. Scheduled snapshots appear in the snapshot list with a "Scheduled snapshot" prefix in their description.

### Capturing stack documentation

Snapshots can also preserve each stack's [Dossier](/features/stack-dossier) notes (purpose, owner, access URLs, firewall and recovery notes, and the rest) alongside the captured files, so a recovery restores the operational knowledge around a stack, not just its configuration.

This is off by default. Turn it on in **Settings → Infrastructure → Fleet** with **Capture stack documentation in snapshots**. Once enabled, every manual and scheduled snapshot records the current Dossier notes for each captured stack. Stacks with no notes are simply omitted, and a stack's notes are never required. Secret values are never included: a Dossier only ever holds the notes an operator typed, never the contents of a `.env`.

## Browsing snapshots

The snapshot list shows each snapshot in a table with the following columns:

- **Date** - when the snapshot was taken
- **Description** - your optional label, or a prefix like "Scheduled snapshot" for automated ones. If Recovery Vault (or a custom off-site target) is configured, an upload icon in this column marks snapshots that have been mirrored off-site.
- **Scope** - how many nodes and stacks were captured (e.g. "3 nodes, 21 stacks")
- **Warnings** - a warning icon with a count if any nodes or stacks were skipped, or "None"
- **Actions** - **View** to open the detail view, a cloud-upload icon for snapshots not yet mirrored to a configured Recovery Vault or custom off-site target, and a delete button for admins

<Frame>
  <img src="/images/fleet-backups/browse-snapshots.png" alt="Snapshot list showing paginated rows with date, description with cloud upload indicator, scope, warnings, and action buttons" />
</Frame>

## Snapshot detail view

Click **View** on any snapshot to open the detail view. Use **Back to Snapshots** in the top-left to return to the list.

The header shows the snapshot's title (or "Untitled Snapshot"), who created it, when, and badge counts for nodes and stacks captured. When the snapshot preserved Dossier notes, a **Documentation captured** badge appears alongside those counts.

Below the header, each node appears as a collapsible card. Expand a node to see its stacks, then expand a stack to see individual files. If documentation was captured for a stack, its preserved Dossier notes are shown read-only beneath the file list. Each file has a **Preview** button that renders the full file contents inline in a scrollable panel, and a **Download** button that saves that single file to your machine (named `<stack>-<filename>`, for example `web-compose.yaml`).

<Frame>
  <img src="/images/fleet-backups/snapshot-detail.png" alt="Snapshot detail view with a node expanded, showing a stack expanded with a compose file, Preview button, and Restore button" />
</Frame>

If any nodes were unreachable during snapshot creation, a warning banner appears at the top of the detail view listing each skipped node and the reason it was skipped. A second banner lists any individual stacks that were only partially captured, naming the node, the stack, and the reason.

## Restoring from a snapshot

Admins can restore individual stacks from any snapshot:

1. Open a snapshot's detail view
2. Expand the node and stack you want to restore
3. Click **Restore** below the stack's file list
4. A confirmation dialog appears. Optionally check **Redeploy stack after restore** to immediately apply the restored configuration. When the snapshot preserved Dossier notes for this stack, a **Restore documentation notes** option also appears.
5. Click **Restore** to confirm

By default, restoring a stack only writes back its files and leaves the stack's current Dossier notes untouched. Notes are restored only when you explicitly check **Restore documentation notes**, which overwrites the current notes with the captured ones. The same option appears on **Restore all** when the snapshot has documentation.

<Frame>
  <img src="/images/fleet-backups/restore-dialog.png" alt="Restore confirmation dialog showing the target stack and node name, a redeploy checkbox, and Cancel and Restore buttons" />
</Frame>

Sencho writes the snapshot's files back to the target node:
- **Local nodes** have files written directly, with the current files backed up first (creating a rollback point via the atomic deployment system)
- **Remote nodes** receive files via the Distributed API proxy

<Warning>
  Restoring overwrites the current compose and environment files on the target node. If atomic deployments are enabled, the current files are backed up before restoration.
</Warning>

### Restoring an entire snapshot

To roll back every captured stack at once, click **Restore all** in the snapshot detail header. A confirmation dialog appears with an optional **Redeploy all stacks after restore** checkbox. On confirm, Sencho restores each stack across the fleet and reports how many succeeded.

Each stack is restored independently: if one stack cannot be restored (for example, its node has since been removed, or a remote node is offline), that stack is reported as failed and the rest still proceed. The result is summarised as a success, a partial result naming how many failed, or a full failure.

## Deleting snapshots

Admins can delete snapshots from the list view by clicking the trash icon on the right side of each row. A confirmation dialog asks you to confirm before the snapshot is permanently removed. Deleting a snapshot removes all captured file data from the database. This action cannot be undone.

## Recovery Vault

<Note>
  Custom S3-compatible storage is available on every tier. Recovery Vault is an Admiral feature. Configure either in **Settings → Infrastructure → Recovery Vault**.
</Note>

Recovery Vault mirrors every fleet snapshot to off-site storage so your snapshots survive local disk failure. The Recovery Vault settings page (reached via **Settings → Infrastructure → Recovery Vault**) shows a header with your current scope, provider, storage used, and total snapshot count in the cloud. Two storage modes are supported.

### Recovery Vault (included)

A managed 500 MB allowance, included with every Admiral subscription. Open **Settings → Infrastructure → Recovery Vault**, choose **Recovery Vault (Included)**, and click **Activate**. Sencho exchanges your license key for scoped storage credentials and starts replicating new snapshots automatically.

Once active, the settings page shows a storage gauge (used / 500 MB and object count), a status message confirming auto-upload is on, and a **Reprovision** button to refresh credentials if needed. You can verify connectivity at any time with the **Test** button.

<Frame>
  <img src="/images/cloud-backup/cloud-backup-active.png" alt="Recovery Vault settings page with Recovery Vault active, showing the storage gauge, auto-upload status, and the Cloud Snapshots list" />
</Frame>

### Custom S3 (BYOB)

Bring any S3-compatible bucket: AWS S3, MinIO, Backblaze B2, Wasabi, or your own Cloudflare R2 token. Choose **Custom S3 (BYOB)** in the storage-mode dropdown and fill in:

- **Endpoint URL** (e.g. `https://s3.us-east-1.amazonaws.com`, `https://my-minio.example.com:9000`)
- **Region** (e.g. `us-east-1`, or `auto` for R2)
- **Bucket** name
- **Path Prefix** (default `sencho/`)
- **Access Key ID** and **Secret Access Key**
- **Auto-upload** toggle

Click **Test** to verify connectivity, then **Save**. Secret keys are encrypted at rest. Sencho only sends them to your configured endpoint.

<Frame>
  <img src="/images/cloud-backup/cloud-backup-custom.png" alt="Custom S3 configuration form with fields for Endpoint URL, Region, Bucket, Path Prefix, Access Key ID, Secret Access Key, and an Auto-upload toggle" />
</Frame>

### Manual upload vs auto-upload

When auto-upload is on, every fleet snapshot is replicated as soon as it is created. Manual snapshots from the **Fleet → Snapshots** view upload asynchronously so the UI returns immediately; scheduled snapshots block on the upload so the task's success status reflects cloud durability.

To upload a single snapshot on demand, open the **Snapshots** tab in Fleet View. Each row that hasn't been mirrored yet shows a cloud-upload action in the Actions column. Once a snapshot is in the cloud, an upload icon appears next to its description.

### Browsing and downloading cloud snapshots

The **Cloud Snapshots** panel in **Settings → Infrastructure → Recovery Vault** lists every archive currently in your bucket, with size and last-modified timestamp. Click the download icon to save a `.tar.gz` archive locally for off-host disaster recovery. Each archive contains a `metadata.json` describing the snapshot and a `nodes/` tree with the captured compose and environment files, organised by node and stack. When the snapshot preserved Dossier notes, the archive also includes a `documentation.json` with those notes.

### Restoring from a cloud snapshot

For in-place rollback, use the **Restore** action on the snapshot detail view as described above; the local copy is the source of truth for live restore. Cloud snapshots cover the disaster-recovery case where the local disk is gone: download the archive, extract it, and bring up a fresh Sencho instance pointed at the recovered files.

## Access control

Every fleet snapshot action requires the **admin** role: viewing the snapshot list, browsing snapshot contents, creating, restoring, deleting, and uploading to the cloud. Because a snapshot captures the `.env` file of every stack, the snapshot list and detail views are restricted to administrators rather than read-only roles.

Recovery Vault configuration is also admin-only. Mirroring to Recovery Vault additionally requires an Admiral subscription; a Custom S3-compatible target works on every tier.

## Storage

Snapshots are stored in Sencho's SQLite database. Captured file contents, including `.env` files, are encrypted at rest with the instance key, so a database copy never exposes stack secrets in plaintext. Compose files are typically small (under 10 KB each), so even hundreds of snapshots consume minimal disk space. Individual files larger than 1 MB are skipped and recorded as a warning to keep snapshots bounded. For very large fleets, consider periodically deleting old snapshots to keep the database lean.

## Troubleshooting

<AccordionGroup>
  <Accordion title="A snapshot shows skipped nodes">
    If a remote node is offline, unreachable, or its API token has expired, the node is skipped during snapshot creation. The list shows a warning icon with a count of skipped nodes, and the snapshot's detail view names each one along with the reason. Common causes are the remote Sencho instance being stopped or restarting, the node's API URL or token having been changed after it was added, or a firewall or network issue blocking the connection. Verify the remote is running and reachable, update the node's API URL and token in **Settings → Infrastructure → Nodes** if needed, then create a new snapshot.
  </Accordion>
  <Accordion title="Restore fails with 'Target node no longer exists'">
    The node recorded in the snapshot has been removed from the fleet since the snapshot was taken. Snapshots reference nodes by registry ID, so a node that was deleted and re-added is treated as a different target. Re-add the node first, then retry the restore against the freshly registered row.
  </Accordion>
  <Accordion title="Restore fails with 'No files found for this stack'">
    The stack was not captured in the snapshot, usually because its compose file was missing or unreadable on disk at the time the snapshot was taken. Open the snapshot's detail view to verify which stacks and files are available, and pick a different snapshot if the one you have is incomplete.
  </Accordion>
  <Accordion title="Restore all reports that some stacks failed">
    Restore all applies each stack independently, so a failure on one stack does not stop the others. A stack is reported as failed when its node has been removed from the fleet since the snapshot was taken, when a remote node is offline or unreachable, or when its existing files cannot be written. The successful stacks are fully restored regardless. Resolve the underlying cause (re-add a removed node, bring an offline node back online) and run Restore all again, or restore the remaining stacks individually from the same snapshot.
  </Accordion>
  <Accordion title="Recovery Vault Test reports an authentication error">
    Double-check the Access Key ID, Secret Access Key, and bucket name; one wrong character is the most common cause. Some providers require S3-compatible API access to be enabled on the bucket separately from the credentials. For MinIO, confirm the user has read/write permission on the target bucket. After correcting the values, click **Test** again before saving.
  </Accordion>
  <Accordion title="Cloud uploads fail after hitting the 500 MB quota">
    Recovery Vault carries a 500 MB allowance per subscription. Once you hit the cap, new uploads fail with a quota error and the storage gauge in the settings page reads at or near 500 MB. Free space by deleting older archives from the **Cloud Snapshots** panel in **Settings → Infrastructure → Recovery Vault**. Local snapshots are unaffected by cloud deletions, so the in-place restore path stays intact. To raise the ceiling, switch the storage mode to **Custom S3 (BYOB)** and point at a bucket you control.
  </Accordion>
  <Accordion title="Cloud upload returns a network timeout or 5xx error">
    Transient errors surface as a notification and leave the local snapshot in place. Retry by clicking the cloud-upload action on the snapshot row, or wait for the next scheduled snapshot which retries on its own. Persistent failures usually indicate an endpoint outage; verify the storage provider is reachable from your Sencho host (custom S3 endpoints often sit behind a different DNS or firewall path than the rest of your traffic).
  </Accordion>
  <Accordion title="I need more detail about a snapshot or restore operation">
    Enable **Developer Mode** under **Settings → Operations → Developer Diagnostics** to activate diagnostic logging for snapshot creation (per-node capture timing and file counts), restore operations, and scheduled snapshot execution. Diagnostic logs appear in the server's standard output with a `:debug` suffix.
  </Accordion>
</AccordionGroup>
