mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-07-26 20:00:08 +00:00
Anso
d8e8d500c9
The fleet-sync.mdx page reflected an earlier shape of the feature. Refreshed to cover everything the receiver and sender now do: - Both scan policies AND CVE suppressions replicate over the same channel today (was previously framed as "future" for suppressions). - Control anchor: replicas bind to the first control fingerprint and reject pushes from a different control until an admin reanchors. Documented the reanchor curl call. - Push ordering: monotonic pushedAt rejects strictly-older pushes with 409 STALE_SYNC_PUSH; legacy controls without timestamps still accepted for back-compat. - Automatic retry: the control retries failed pushes every 5 minutes for 24h and emits one warning notification per hour-long failure window for previously-working remotes. - Demote to control: documented the admin-only button and what it wipes. Replaced the "remove the node from the control" workaround with the proper flow. - Pilot-agent nodes: explicitly documented as out of fleet-sync scope. - Tiebreaker: documented lowest-id wins for ties. - Replica policy filtering: identity-scoped policies for other replicas no longer surface in this replica's UI. - Troubleshooting expanded with 409 codes (STALE_SYNC_PUSH, CONTROL_IDENTITY_MISMATCH). Cross-link added from vulnerability-scanning.mdx (Scan policies section) so anyone reading about policies finds the replication docs.
123 lines
7.3 KiB
Plaintext
123 lines
7.3 KiB
Plaintext
---
|
|
title: "Fleet Sync"
|
|
description: "How security rules replicate from a control Sencho instance to remote nodes."
|
|
---
|
|
|
|
When you manage several Sencho instances as a fleet, the control instance (the one where you added remote nodes in **Settings → Nodes**) acts as the source of truth for security configuration. Rules you create on the control replicate automatically to every remote so the entire fleet enforces the same policies.
|
|
|
|
Today this covers **vulnerability scan policies** and **CVE suppressions**. Both replicate over the same channel and follow the same rules.
|
|
|
|
## Control vs replica
|
|
|
|
Every Sencho instance has a **role** that determines whether it accepts writes for replicated resources:
|
|
|
|
| Role | Behavior |
|
|
|------|----------|
|
|
| **Control** | Default for any standalone instance and for the node you point your browser at when managing a fleet. Accepts create, edit, and delete for security rules. Pushes changes to every remote. |
|
|
| **Replica** | An instance that has received at least one sync push from a control. Shows rules read-only with a banner indicating they are managed upstream. Returns `403 Forbidden` for direct write attempts. |
|
|
|
|
Role detection is automatic: a Sencho instance becomes a replica the first time it accepts a sync push. To switch a replica back to a standalone control, an admin uses **Demote to control** in **Settings → Security** (described below).
|
|
|
|
<Frame>
|
|
<img src="/images/fleet-sync/replica-banner.png" alt="Settings page on a replica showing the 'Managed by control node' banner above the policies list" />
|
|
</Frame>
|
|
|
|
## How replication works
|
|
|
|
1. You create, edit, or delete a security rule on the control.
|
|
2. The control commits the change to its local database.
|
|
3. The control iterates every remote node registered in **Settings → Nodes** that has an API URL and token configured.
|
|
4. For each remote, the control posts the full current rule list to `/api/fleet/sync/<resource>`, tagged with the remote's own identity, a monotonic timestamp, and a fingerprint that identifies the control.
|
|
5. The remote validates the payload, checks the control fingerprint and timestamp, and replaces its replicated rows in a single transaction.
|
|
|
|
Local-only rules created directly on a remote coexist with the replicated set. Sync only replaces rows flagged as coming from the control.
|
|
|
|
## Control anchor
|
|
|
|
A replica binds to the first control that pushes to it. The fingerprint is a hash derived from the control's persistent install ID, so a hostname change does not flag drift.
|
|
|
|
After the first sync, pushes from any other control are rejected with `409 CONTROL_IDENTITY_MISMATCH`. This prevents an operator from accidentally pointing a second control at an existing replica and overwriting its mirrored state.
|
|
|
|
To re-bind a replica to a different control (for example, after a control rebuild), an admin sends:
|
|
|
|
```bash
|
|
curl -X POST https://<replica-url>/api/fleet/role/reanchor \
|
|
-H "Authorization: Bearer <admin-jwt>" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"override": true}'
|
|
```
|
|
|
|
The next push from any control becomes the new anchor.
|
|
|
|
## Push ordering
|
|
|
|
Each push carries a strictly-increasing timestamp. The receiver rejects strictly-older pushes with `409 STALE_SYNC_PUSH` so a slow push that arrives after a faster one cannot overwrite the newer state. The control's normal write path generates a fresh timestamp on every change.
|
|
|
|
Controls that predate this protocol (older Sencho versions in mixed-version fleets) send no timestamp, and the receiver treats those pushes as legacy and accepts them.
|
|
|
|
## Automatic retry
|
|
|
|
If a remote is offline when a write happens, the control records the failure on the **Settings → Security** sync-status panel and retries every 5 minutes for the next 24 hours. Once the remote comes back online, the next retry catches it up to the latest state.
|
|
|
|
If a previously-working remote stays unreachable for more than an hour, the control dispatches a **warning notification** so the operator knows the node has fallen behind. The notification fires once per hour-long failure window.
|
|
|
|
## Policy scope across a fleet
|
|
|
|
When you create a policy on the control, its scope is preserved during replication:
|
|
|
|
- **Fleet-wide** (no specific node selected): applies on every instance in the fleet.
|
|
- **Specific node**: applies only on the instance whose identity matches. A policy targeting `Node A` evaluates only during scans that run on Node A, even after it has replicated to every other remote.
|
|
|
|
Only one policy is evaluated per deploy. The most specific match wins, in this order:
|
|
|
|
1. Node-scoped and stack-scoped policies.
|
|
2. Node-scoped, stack-wildcard policies.
|
|
3. Fleet-wide, stack-scoped policies.
|
|
4. Fleet-wide, stack-wildcard policies.
|
|
|
|
When two rules tie on scope class, the lowest policy id wins so every replica resolves the same winner.
|
|
|
|
On a replica, the **Settings → Security** panel shows local rules, fleet-wide replicated rules, and replicated rules that target this replica. Identity-scoped rules meant for a different replica do not appear.
|
|
|
|
## Demote to control
|
|
|
|
A replica admin can demote the instance back to a standalone control from **Settings → Security**. The button sits next to the "Managed by control node" banner. Demote:
|
|
|
|
- Drops every replicated scan policy and CVE suppression mirrored from the control.
|
|
- Clears the cached fleet identity and control fingerprint.
|
|
- Re-enables local edits on this instance.
|
|
|
|
The control loses this remote as a replica. Re-adding the remote in the control's **Settings → Nodes** restarts replication from scratch on the next write.
|
|
|
|
## Pilot-agent nodes
|
|
|
|
Pilot-agent nodes are not part of fleet sync today. The control logs a one-time warning per pilot node and skips it during pushes. Pilot-tunnel-based replication is on the post-1.0 roadmap.
|
|
|
|
## Troubleshooting
|
|
|
|
### Security rules are missing on this node
|
|
|
|
You are looking at a replica. Navigate to the control Sencho instance and view its **Settings → Security** page; rules you create there replicate here automatically.
|
|
|
|
If rules are expected to be present but are not, check **Settings → Security** on the control: it surfaces sync status per node and shows the most recent error. The retry loop catches up automatically once the underlying issue (network, expired token) is resolved.
|
|
|
|
### A remote rejects the sync
|
|
|
|
A remote rejects a push if its bearer token is invalid or the request does not reach it. Verify the node's API URL and token in **Settings → Nodes** on the control, then use **Test Connection** to confirm the remote is reachable.
|
|
|
|
### A push returns 409 CONTROL_IDENTITY_MISMATCH
|
|
|
|
The replica is anchored to a different control. Either point your browser at the original control, or call the reanchor endpoint on the replica with `{"override": true}` to re-bind.
|
|
|
|
### A push returns 409 STALE_SYNC_PUSH
|
|
|
|
A newer push has already landed on the replica for the same resource. The control's retry loop will skip the stale outcome on the next tick. No action needed.
|
|
|
|
### A rule I deleted on the control is still visible on a replica
|
|
|
|
Sync happens on every write. If you deleted a rule but the replica still shows it, the push probably failed. Check the sync-status panel on the control or wait for the next retry tick (within 5 minutes).
|
|
|
|
### I want this replica to be a standalone control
|
|
|
|
Use **Demote to control** in **Settings → Security** on the replica. The button is admin-only and asks for explicit confirmation because demote wipes mirrored rules.
|