mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-10-26 05:30:18 +00:00
Anso
1efaa13492
* refactor(stacks,security): extract status and overview builders into services
Move the bodies of `GET /api/stacks/statuses` and `GET /api/security/overview`
into `services/stackStatusesEvidence.ts` and `services/securityOverview.ts`.
Both routes keep their guards, response shapes, error handling, and timing
logs; they now delegate to the new builders.
The moved logic is verbatim apart from `req.nodeId` becoming a `nodeId`
parameter and `res.json(...)` becoming a return, so the existing route suites
cover it unchanged.
The move does restructure one thing: a builder rejection is now caught by a
`try` in the route that no longer wraps the code which throws. That propagation
was not covered by any test, so `cache-endpoints.test.ts` now pins it, including
that the real error still reaches the console rather than the 500 standing alone.
This is groundwork: the fleet readiness surface needs to read the same evidence
from a batch context rather than through one Express request per node.
Extracting first keeps that change from also being a refactor.
One accepted deviation, on the error path only. `dockerMs` and `enrichmentMs`
were route-scope bindings mutated by the cache closure, so a rejection after a
timing was recorded still let the route log that partial value. They are now
builder-scope, so a rejection logs the initial nulls instead. The HTTP
response is unchanged (500, same message); only the debug timing line differs,
and only when an unexpected throw lands between the Docker call and the
enrichment block with no stale cache entry present.
* feat(readiness): add the fleet readiness wire contract
Define the types the fleet readiness surface speaks in: the six domains,
the per-node domain cells the matrix renders, the prioritized findings
list, and the per-stack summary the tier-2 node route returns. No producer
or consumer lands in this commit; the routes and the aggregator follow.
The contract carries no scoring and no prose headlines. A finding's display
payload is a reason code the frontend maps to copy, and the two free-text
fields (finding detail and per-stack top reason) carry an explicit
redaction requirement, because the canonical contracts they pass through
can embed a materialized input path.
Also exports the reachability classifier from the blueprint preview
projection, along with the contact-source union it already returned and a
closed union for its note, so the fleet surface classifies reachability
with the same cached logic rather than writing a second classifier. The
three existing consumers are unaffected: both substitutions are in type
positions only, and 39 tests across the preview, binding-preview, and
reconciler-gate suites pass unchanged.
* feat(readiness): add the node-local per-stack readiness rollup
Adds `GET /api/stacks/readiness-summary`, the node-local half of the fleet
readiness surface. It reports the canonical per-stack update and rollback
verdicts for one node, because the Docker socket is on that node: the update
preview's remote path aborts at a hard 90s ceiling per stack, which a hub
fan-out cannot wait on.
The route adds no scoring. It selects, bounds, and reports what
`UpdateGuardService` already computes:
- one pass, aborted by a 6s deadline that starts before the compose directory
scan so the scan is charged to the same budget
- concurrency 3, and a 25-stack cap that keeps the most operationally
interesting stacks (failed checks, then updates available, then open drift
findings, then alphabetical)
- a 120s cache entry, invalidated on stack mutation through
`invalidateNodeCaches` so a fan-out does not re-run the Docker-heavy pass per
request
- the two verdicts settle independently, so a failing rollback compute cannot
erase a good update verdict; a stack whose work throws degrades to an
unavailable row rather than failing the pass
Each row carries the strongest reason behind its verdict, taken from the
canonical signals and items and passed through `redactSensitiveText`, because a
detail there can carry a materialized input path on a `node:read`-gated surface.
The served payload carries `stale`, derived from the pass timestamp rather than
from the cache outcome. The outcome is not truthful for a caller that joined
someone else's in-flight fetch: when that fetch fails and an expired entry is
served, the caller that ran it is told `stale` while every joiner is told
`inflight`, on the same old bytes. An expired entry's own timestamp names the
condition for every reader. The pass type omits the field, so a pass cannot
assert freshness it cannot know.
The contract also names the aggregate's row types (`StackReadinessRow`,
`StackUpdateReadinessRow`, `StackRollbackReadinessRow`), so the producer and its
tests refer to them rather than restating the shape at each call site.
Two registrations move together and each has its own test: the route is mounted
ahead of the `/:stackName` catch-all, and `/stacks/readiness-summary` joins
`STATIC_STACK_PATHS`. Missing the first resolves the request as a stack named
`readiness-summary`; missing the second demands exact `stack:read` on a stack of
that name, for the router gate and for the hub's own fan-out call.
`fleet-readiness-v1` joins the capability registry on both sides, following the
append-only convention there. The hub reader that gates on it lands with the
aggregator.
The route is guarded by `stack:read`, which every role holding `node:read`
already has.
* feat(fleet): aggregate fleet readiness on the hub
Add GET /api/fleet/readiness, a hub-owned aggregate over six domains
(connectivity, workloads, updates, recovery, security, control) that answers
what across the fleet needs attention before an operator acts on it.
The hub answers from its own SQLite for connectivity, stored update facts,
snapshot gaps, and Policy Sync state, and fans out two bounded node-local
reads: GET /api/readiness/evidence (3s budget) for workloads, security, and
stored recovery rows, and GET /api/stacks/readiness-summary (8s budget) for
the canonical per-stack update and rollback verdicts. Every node is wrapped
in its own try/catch into a per-node row, so one slow or unreachable node
degrades one row instead of holding the page, and a disconnected Pilot
target costs no request at all. A build without the evidence route answers
404, which is reported as a missing capability rather than as a fault.
Findings carry a stable id, an enumerated code, and a drill-down target.
Verdicts are read from the existing update and rollback readiness functions
rather than recomputed here, and nothing is scored: a check that could not
run is reported unknown or unavailable, never healthy.
The Control domain is admin-only, matching the sibling Policy Sync read. The
response omits the domain for an account that cannot see it rather than
sending it and relying on the client to hide it.
Retire GET /api/fleet/configuration and the backend normalizer that served
only it. Home's Configuration Summary keeps its own builder, and
GET /api/dashboard/configuration is unchanged.
Document the three endpoints in docs/openapi.yaml.
* feat(fleet): replace the Status tab with Readiness
Swap the Fleet tab strip's configuration inventory for a Readiness tab: a
worst-first fleet summary, a prioritized findings worklist, and a per-node
matrix over the six readiness domains. States, reason codes, and findings
all arrive decided from the hub, so the surface renders the payload and
dispatches each drill-down to the surface that already owns that
remediation.
The matrix lays out the domains the payload carries, so the Control column
is simply absent for an account the server withheld it from. Tier-2 columns
are read from each node's own cells rather than from hasCapability(), which
describes the active node and would misreport every other row.
Findings that restate an update or a rollback verdict reuse the mappings
those surfaces already publish instead of re-declaring them.
Normalize the legacy `configuration` segment to `readiness` in the route
parser so an existing deep link still resolves.
Delete the retired configuration inventory component.
* docs(fleet): document the Readiness tab
Follow the tab from Status to Readiness on the Fleet view page: the summary,
the findings worklist, the per-domain matrix, what happens when a node cannot
be reached, and the admin-only Control column. Update the tab table and the
tab-strip prose, and drop the tab list from the Overview screenshot alt text
so it stops describing a strip the screenshot no longer shows.
Replace the tab screenshot and follow the rename through Fleet Sync, the
deep-link reference, the introduction, and the API reference contents.
* test(fleet): compare readiness mock hosts by origin
The readiness fan-out tests route each mocked fetch by asking whether the
request targeted the proxy host. Decide that on the parsed origin rather
than on a prefix match against the raw URL string, which is the shape the
static analysis flags as incomplete URL handling.
No behavior change: every URL reaching a dispatcher is absolute, and each
of the four tests exercises both sides of the branch.
* fix(readiness): keep ordinary configuration and repeated causes out of findings
The aggregate reported several conditions that are not problems, and one
unreachable node could hold its row for the full verdict budget.
- A stack with no containers (not deployed) is a known state, not missing
evidence: it is no longer a Workloads concern, and its update and
rollback verdicts are left out. A stale or degraded status read never
hides a verdict.
- A fleet that never took a snapshot adds no Recovery concern; snapshots
are optional. An unreadable skip column is a check that could not run.
- Policy sync (Control) is classified only for the nodes it pushes to.
The hub's own node and Pilot-agent nodes carry no Control cell instead
of a permanent "unknown".
- When the reachability probe gets no answer, the slower verdict read to
the same remote is cancelled, so a dead remote costs the 3s probe
budget instead of the 8s verdict budget. A verdict read that failed on
its own keeps its own cause.
- When Connectivity already reports why a node could not be read, the
gated domains keep their cells but add no finding restating the same
cause. The hub's own node, whose Connectivity is always healthy, keeps
every finding.
- Rows are sorted worst first and carry their worst state.
* feat(fleet): rebuild the Readiness tab on the shared page patterns
The tab used a bespoke layout that did not match the rest of the app. It
now uses the same building blocks as the other Fleet tabs and the
Security and GitOps tables:
- The shared Fleet tab heading, then a summary strip with the worst state
as a short phrase and tiles for nodes needing attention, unverified,
healthy, and when the check last ran.
- Findings on the shared table shell with the standard filter row
(search, domain, state, verdict, node) and pagination. The verdict
filter has presets for rollback partial or not ready, and for updates
that are blocked or need review.
- A node matrix whose cells name each state in the domain's own words
("online", "2 blocked", "never scanned", "in sync", "--" where policy
sync does not apply) instead of generic state words. Clicking a problem
cell narrows the findings to it.
- The Fleet toolbar Refresh now re-runs the readiness check.
- A Security finding switches to the finding's node before opening
Security, using the same pending node-view intent the Networking
shortcut uses.
* docs(fleet): describe the rebuilt Readiness tab
Rewrite the Readiness section for the new summary strip, findings
filters, and domain-specific matrix values, refresh the screenshot, and
update the Fleet Sync references and the OpenAPI node contract (worst
state per row, Control only for policy sync targets, one finding per
unreachable node).
197 lines
22 KiB
Plaintext
197 lines
22 KiB
Plaintext
---
|
|
title: "Fleet Sync"
|
|
description: "Push-only replication of security rules from a control Sencho instance to its replicas."
|
|
---
|
|
|
|
When you manage several Sencho instances as a fleet, the control instance (the one whose **Settings → Nodes** holds the rest of the fleet as remotes) owns the authoritative copy of your security rules. Every time you create, edit, or delete one of those rules on the control, it streams to every reachable remote so the whole fleet enforces the same posture. The remotes are *replicas*: they hold a mirrored read-only copy until you demote them.
|
|
|
|
Fleet Sync replicates three resources today, all over the same channel and with the same lifecycle:
|
|
|
|
- **Scan policies** ([Vulnerability Scanning](/features/vulnerability-scanning)).
|
|
- **CVE suppressions** ([CVE Suppressions](/features/cve-suppressions)).
|
|
- **Misconfig acknowledgements**.
|
|
|
|
Fleet Sync lives on the **Security** page (the **Policies** and **Suppressions** tabs, the latter also holding misconfig acknowledgements) on both the control (where you author rules) and the replica (where you see them, read-only).
|
|
|
|
<Frame>
|
|
<img src="/images/fleet-sync/fleet-sync-policies-control.png" alt="The Security page Policies tab on the control instance. The header reads 'Deploy enforcement policies' with an Add policy button, the body shows the empty 'No scan policies configured' state, and a Honor suppressions in deploy blocks toggle sits below, switched off." />
|
|
</Frame>
|
|
|
|
## Control and replica roles
|
|
|
|
Every Sencho instance carries a `fleet_role` flag that is either `control` or `replica`. The flag is consulted on every write path for the replicated resources and by the **Security** page UI when it decides whether to render the edit controls.
|
|
|
|
| Role | Behaviour |
|
|
|---|---|
|
|
| **Control** | The default for any fresh install and for the instance whose **Settings → Nodes** lists the rest of the fleet. Accepts create, edit, and delete on scan policies, CVE suppressions, and misconfig acknowledgements. Pushes the full current state of each resource to every reachable remote on every write. |
|
|
| **Replica** | An instance that has received at least one Fleet Sync push. Renders replicated rules as read-only with a "Managed by control node" banner above the policy editor on the **Security** page → **Policies** tab. Returns `403 Forbidden` for any direct write attempt against the replicated tables. |
|
|
|
|
The transition from control to replica happens automatically the first time a replica accepts a push: the apply transaction sets `fleet_role = 'replica'` atomically with the row replacement, so the role flip and the new rows land together or not at all. Going the other way is explicit: an admin clicks **Demote to control** on the replica (see [Demote a replica](#demote-a-replica) below).
|
|
|
|
<Frame>
|
|
<img src="/images/fleet-sync/fleet-sync-policies-replica.png" alt="The Security page Policies tab on a replica, signed in directly. A callout reads 'Managed by control node: Security policies replicate from the control Sencho instance. View them here for audit; edit them on the control,' with a Demote to control button, above the empty 'No scan policies configured' state." />
|
|
</Frame>
|
|
|
|
### Viewing a replica through the control's fleet view
|
|
|
|
The read-only banner above only appears when you sign in **directly** to the replica's own URL. Scan policies, CVE suppressions, and misconfig acknowledgements are fetched with a `localOnly` request, so they are not proxied through the Distributed API the way most other per-node pages are. If you instead browse a remote node's **Security** page from the control by switching nodes in the top-left node switcher, the Policies and Suppressions tabs show a different message and no replicated data at all:
|
|
|
|
<Frame>
|
|
<img src="/images/fleet-sync/fleet-sync-proxy-notice.png" alt="The Security page Policies tab on the control, with the node switcher set to a remote. A callout reads 'Managed on the local instance: Scan policies are managed on the local Sencho instance. Switch to the local node to manage them,' with no policy list below it." />
|
|
</Frame>
|
|
|
|
To see a replica's mirrored rules, sign in to that instance's own URL instead of viewing it through the control's node switcher. This is different from most other Security page tabs (Images, Secrets, Scanner setup) and most other Sencho pages generally, which do proxy through the node switcher; scan policies, CVE suppressions, and misconfig acknowledgements are the exception.
|
|
|
|
## What replicates
|
|
|
|
The three replicated resources share one wire protocol, one retry queue, and one anchor. Each push carries the full current state of one resource (not a delta); the receiver replaces every `replicated_from_control = 1` row in a single transaction. On the replica, each mirrored row carries a **replicated** tag so you can tell it apart from a local one at a glance:
|
|
|
|
<Frame>
|
|
<img src="/images/fleet-sync/fleet-sync-suppressions-replica.png" alt="The Security page Suppressions tab on a replica. A CVE suppression is listed, tagged 'Accepted risk' and 'replicated,' with no Add suppression or Export VEX buttons visible. A Misconfig acknowledgements section below is empty." />
|
|
</Frame>
|
|
|
|
What does *not* replicate:
|
|
|
|
- **Local rules created directly on a replica.** A replica can still hold its own local rules; sync only touches rows that originated on the control. The two coexist in the same table and are distinguished by an internal flag.
|
|
- **Trivy itself.** The scanner binary is installed independently on each instance. The Security page's **Scanner setup** tab (unlike Policies and Suppressions, this one does proxy through the node switcher) shows a "Scanner is per-node" callout in place of the full editor when you view it on a remote, since the scanner lifecycle is a node concern, not a fleet concern.
|
|
- **Everything outside the three resources above.** API tokens, audit logs, blueprints, secrets, alert rules, users, SSO config, and general settings stay per-instance.
|
|
- **Pilot-agent nodes.** Sync over the [pilot tunnel](/features/pilot-agent) is not part of v1; the control logs a one-time warning per pilot node and skips it during fanout. The pilot node's local rules are unaffected.
|
|
|
|
## How replication works
|
|
|
|
Fleet Sync is a push-only fan-out from the control on every relevant write. There is no pull, no polling, no schedule, no autonomous reconciler.
|
|
|
|
1. The control commits a change locally (the operator clicks **Save** in the UI; the row is written in a database transaction).
|
|
2. The write handler invokes `pushResourceAsync(resource)`, which loads the current full state of that resource from the local database.
|
|
3. The control iterates every remote node in **Settings → Nodes** that is a proxy-mode remote with an `api_url` and `api_token` configured, then posts the payload to `POST /api/fleet/sync/<resource>` on each one in parallel.
|
|
4. Each remote validates the payload, checks the control fingerprint, checks the per-resource watermark, replaces its replicated rows, sets `fleet_role = 'replica'`, and records an audit-log entry. All five steps run inside one SQLite transaction.
|
|
5. Failures are queued for retry (see [Push ordering and retry](#push-ordering-and-retry)).
|
|
|
|
Pushes to the same remote are serialised: a per-`(node, resource)` mutex guarantees that two payloads for the same node never overlap, so a slow earlier push cannot land on top of a faster later one. Different resources fan out independently and can travel in parallel.
|
|
|
|
When you add a new remote to **Settings → Nodes**, the control performs an immediate backfill push for every resource so the new replica catches up to the current state without waiting for the next write event.
|
|
|
|
The control identifies itself in every payload with a fingerprint derived from its persistent install ID (SHA-256 truncated to 16 hex characters). The fingerprint is stable across hostname or `api_url` changes; only a SQLite reset or an explicit re-anchor on the replica resets it.
|
|
|
|
## Anchor and re-anchor
|
|
|
|
A replica binds to the first non-empty fingerprint it sees. After that:
|
|
|
|
- A push carrying the same fingerprint is accepted.
|
|
- A push carrying a different non-empty fingerprint is rejected with `HTTP 409` and the body code `CONTROL_IDENTITY_MISMATCH`. This prevents an operator from accidentally pointing a second control at an existing replica and overwriting its mirrored state.
|
|
- A push with no fingerprint at all is accepted (this is the legacy code path; the receiver treats absence as "unknown control" and applies the rows without binding).
|
|
|
|
When you need to re-bind a replica to a different control (for example, the original control was rebuilt and now carries a fresh install ID), the control itself surfaces the mismatch: **Settings → Nodes** shows a banner on the affected node reading "Node is anchored to another central," with a **Reset anchor on peer** button. An admin clicks it and the control proxies the reset to the peer over its existing bearer token, no direct access to the replica required.
|
|
|
|
If you need to script it or the node is not reachable from the control's own Settings → Nodes list, call the endpoint directly on the replica with an admin JWT and an explicit override flag:
|
|
|
|
```bash
|
|
curl -X POST https://<replica-url>/api/fleet/role/reanchor \
|
|
-H "Authorization: Bearer <admin-jwt>" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"override": true}'
|
|
```
|
|
|
|
Either path clears the cached fingerprint, clears the per-resource watermarks, and drops every replicated row inside one transaction. The replica stays a replica (it remains a passive receiver) and the next push from any control becomes the new anchor.
|
|
|
|
## Push ordering and retry
|
|
|
|
Each push carries a `pushedAt` timestamp generated by the control. The control keeps a process-wide monotonic counter so two rapid writes never produce the same number, and so a clock skew that would otherwise rewind time still emits a strictly-increasing value.
|
|
|
|
On the receiver, the watermark is stored per resource. An incoming `pushedAt` that is strictly older than the current watermark for that resource is rejected with `HTTP 409 STALE_SYNC_PUSH`. Watermarks are independent: a stale push for scan policies does not block a newer push for CVE suppressions.
|
|
|
|
If a push fails (network error, replica offline, `409 STALE_SYNC_PUSH`, `409 CONTROL_IDENTITY_MISMATCH`, or any other non-2xx response), the failure is recorded against `(node, resource)` in the `fleet_sync_status` table. The retry service then takes over:
|
|
|
|
- **Tick:** every 5 minutes.
|
|
- **Window:** any failure within the last 24 hours is eligible for retry.
|
|
- **Stale-target warning:** once a previously-working remote has been failing continuously for more than 1 hour, the control dispatches a `warning`-level notification. The warning fires at most once per hour-long failure window so it does not spam alert routing.
|
|
- **Identity drift:** when a replica's view of its own URL (as seen by the control) changes between pushes, the replica dispatches a `warning` notification flagging the new identity. Identity-scoped rules are reapplied automatically on the next sync.
|
|
|
|
Stale pushes (`409 STALE_SYNC_PUSH`) are not counted as failures, since a newer push has already landed for the same resource on that node. The retry loop treats those targets as healthy.
|
|
|
|
## Demote a replica
|
|
|
|
An admin on a replica can take the instance back to a standalone control from the **Security** page → **Policies** tab. The button is "Demote to control" and sits inside the "Managed by control node" callout. The exact modal copy:
|
|
|
|
> **Demote replica to control**
|
|
>
|
|
> Removes every replicated scan policy and CVE suppression mirrored from the control. Local edits to security policies on this instance become available again.
|
|
|
|
Demote runs inside a transaction. It:
|
|
|
|
- Drops every row marked `replicated_from_control = 1` across all three resources.
|
|
- Clears the cached control fingerprint and the three per-resource watermarks.
|
|
- Flips `fleet_role` back to `control`.
|
|
|
|
Local rules authored on this instance directly (rows that did not come from the control) are untouched. From the control's side, the demoted instance is still in **Settings → Nodes**; the next write event triggers another push, which the now-standalone control rejects until the operator removes it from the original control's node list or accepts a new anchor.
|
|
|
|
Demote requires `{"confirm": true}` in the request body to prevent a misclick from wiping mirrored state.
|
|
|
|
## Prerequisites
|
|
|
|
Fleet Sync works on every tier: authoring scan policies, CVE suppressions, and misconfig acknowledgements, and the replication, anchor, and demote mechanics that ship them to remotes, all run on Community. Replicas accept pushes regardless of their own tier.
|
|
|
|
| Requirement | Why it matters |
|
|
|---|---|
|
|
| **Admin user role on the control** | Authoring the rules that replicate, and operating the re-anchor and demote endpoints on a replica, are all admin-only actions. Operator and viewer roles can read rules but cannot create or remove them. |
|
|
| **Proxy-mode remotes with `api_url` and `api_token` configured in Settings → Nodes** | Fleet Sync pushes over HTTPS to each remote's Sencho API using its long-lived bearer token. Remotes without an `api_url` or `api_token`, or remotes that connect over the pilot tunnel, are skipped. |
|
|
| **Network reachability from the control to each remote** | Pushes are HTTP requests originating on the control. A remote that is firewalled off, behind NAT without a forwarded port, or otherwise unreachable will queue retries until it returns. |
|
|
|
|
## Limitations
|
|
|
|
Fleet Sync v1 ships the three replicated resources and the control mechanics described above. The following are deliberately out of scope today:
|
|
|
|
- **The built-in status signal is a rollup, not a push-by-push history.** [Fleet View](/features/fleet-view)'s **Readiness** tab carries a **Policy sync** cell per proxy-mode remote: `in sync`, `sync failing` when pushes are failing, `paused` when the remote's anchor no longer matches the control, or `never synced`. It tells you the current state only, never the history of individual pushes. For that detail, use the receiving replica's audit log (every applied push records `POST /api/fleet/sync/<resource>` with username `system` and the source fingerprint in the summary) or the control's server logs.
|
|
- **Stack-pattern scope only in the form UI.** The scan policy and CVE suppression forms expose a stack-pattern glob (e.g. `prod-*`) but no node selector. The underlying data model supports node-scoped rules, and they replicate faithfully when present, but authoring them today requires writing directly to the control's API.
|
|
- **5,000 rows per resource per push.** If the control accumulates more than 5,000 active rules in any one resource, the sender truncates to the first 5,000 and emits a `warning` notification (cool-down: 6 hours). In practice no fleet should approach this cap; if you see the warning, treat it as a signal to consolidate rules.
|
|
- **Pilot-agent nodes are not part of Fleet Sync today.** The control logs a one-time warning per pilot node and skips it during fanout. Replicating security rules over the [pilot tunnel](/features/pilot-agent) is on the post-1.0 roadmap.
|
|
- **One control per replica at a time.** The anchor binds on the first non-empty fingerprint received and stays bound until re-anchor. Two controls pushing to the same replica is not a supported topology; the second control will receive `409 CONTROL_IDENTITY_MISMATCH` for every push.
|
|
- **Pushes carry the full current state, not deltas.** Each push replaces every replicated row for that resource on the receiver. The cost is bounded by the row cap above; the benefit is that the replica always converges to the control's current state without needing to reconcile a long edit history.
|
|
|
|
## Troubleshooting
|
|
|
|
<AccordionGroup>
|
|
<Accordion title="Rules I created on the control are missing on a remote">
|
|
The remote is either offline, was offline at the moment of the last write and has not yet been retried, or its bearer token in **Settings → Nodes** is invalid. Open **Settings → Nodes** on the control and click **Test connection** for the remote in question. If the connection check passes, wait up to 5 minutes for the next retry tick; the next successful push will reconcile the remote to the current state. If the connection check fails, fix the underlying reachability or token issue and Fleet Sync will catch up on the next retry.
|
|
</Accordion>
|
|
<Accordion title="The Security panel on a remote is empty even though the control has rules">
|
|
This usually means the remote has not yet received its first push (it is still a fresh control from its own perspective). Any write on the control triggers a fresh push, or you can add the remote again in **Settings → Nodes** to trigger the add-node backfill. Once the first push lands, the remote flips to replica and shows the "Managed by control node" callout above the policy list.
|
|
</Accordion>
|
|
<Accordion title="A push returns 409 CONTROL_IDENTITY_MISMATCH">
|
|
The replica is already anchored to a different control's fingerprint. **Settings → Nodes** on the control shows a banner on the affected node with a **Reset anchor on peer** button; click it to clear the anchor without leaving the control. Alternatively, point your browser at the original control and continue authoring there, or sign in as admin on the replica and call the re-anchor endpoint with `{"override": true}` directly. The next push from any control then becomes the new anchor.
|
|
</Accordion>
|
|
<Accordion title="A push returns 409 STALE_SYNC_PUSH">
|
|
A newer push for the same resource has already landed on this replica, so the older retry is silently dropped. No action needed; the next write on the control will produce a fresher timestamp and succeed.
|
|
</Accordion>
|
|
<Accordion title="I deleted a rule on the control but it is still visible on a replica">
|
|
The push for the delete probably failed. Check **Settings → Nodes** on the control; a remote that is offline or whose token is invalid will not have received the delete. Once the underlying issue is resolved, the next retry catches the remote up to the current state (which no longer includes the deleted rule).
|
|
</Accordion>
|
|
<Accordion title="I cannot see whether sync succeeded">
|
|
Open the [Fleet View](/features/fleet-view) **Readiness** tab and read the **Policy sync** column: `in sync`, `sync failing` when pushes are failing, `paused` when the remote's anchor no longer matches the control, or `never synced`. A failing or paused remote also lists a finding that carries the last push error; a paused remote's finding reads `Policy sync is paused on this node`. For push-by-push detail beyond that rollup (see [Limitations](#limitations)), two surfaces give you the information directly: the replica's **Audit** view records every applied push as a `POST /api/fleet/sync/<resource>` entry with username `system` and the source fingerprint in the summary, and the control's server logs record the push outcome per remote. The administrative API endpoint `GET /api/fleet/sync-status` returns the per-resource per-node status table for tooling that needs to consume it directly.
|
|
</Accordion>
|
|
<Accordion title="A warning says my suppression list was truncated">
|
|
The control side of Fleet Sync caps each push at 5,000 rows per resource (see [Limitations](#limitations)). If your suppression list, scan policy list, or acknowledgement list exceeds the cap, the sender truncates to the first 5,000 and emits a single warning every 6 hours. Consolidate rules by replacing repetitive entries with pattern-scoped ones, or remove suppressions that are no longer needed.
|
|
</Accordion>
|
|
<Accordion title="Misconfig acknowledgements did not replicate">
|
|
Misconfig acknowledgements ride the same channel as scan policies and CVE suppressions; if one resource replicates and another does not, it is almost always a per-resource watermark race. Re-save the acknowledgement on the control to produce a fresh `pushedAt`, then the next push catches the remote up. If the symptom persists, check the control's server logs for the `[FleetSync]` push outcome on that remote.
|
|
</Accordion>
|
|
<Accordion title="I want this replica to be a standalone control again">
|
|
Open the **Security** page → **Policies** tab on the replica and click **Demote to control** in the "Managed by control node" callout. The action requires confirmation and drops every replicated row from this instance. Local rules authored directly on this instance are kept. The control loses this remote as a replica; remove it from the control's **Settings → Nodes** as well if you no longer want the control to push to it.
|
|
</Accordion>
|
|
</AccordionGroup>
|
|
|
|
## Where Fleet Sync fits
|
|
|
|
Fleet Sync is one of several features that span multiple nodes in a Sencho fleet. Each handles a different concern; they do not overlap.
|
|
|
|
| Related feature | What it covers | Why it is not Fleet Sync |
|
|
|---|---|---|
|
|
| [Fleet View](/features/fleet-view) | The masthead, tab strip, and node-grid that show fleet status at a glance. | Fleet View is observability; Fleet Sync is state replication. |
|
|
| [Multi-Node Management](/features/multi-node) | Adding remotes to **Settings → Nodes**, proxy versus pilot mode, license inheritance across the fleet. | Multi-Node Management is how nodes become reachable; Fleet Sync uses the same plumbing to ship security rules. |
|
|
| [Pilot Agent](/features/pilot-agent) | The outbound WebSocket tunnel mode used by remotes behind NAT. | Fleet Sync does not flow over the pilot tunnel today. |
|
|
| [Vulnerability Scanning](/features/vulnerability-scanning) | Trivy installation, scan execution, and posture grading on a single node. | Scanning runs per node; Fleet Sync replicates the policies and acknowledgements that shape what scans surface across the fleet. |
|
|
| [CVE Suppressions](/features/cve-suppressions) | Marking specific CVEs as known-benign so they stop triggering alerts. | Suppressions are one of the three resources Fleet Sync replicates; this page documents the replication mechanics. |
|
|
| [Fleet Federation](/features/fleet-federation) | Operator-driven placement controls for blueprints (cordon and pin). | Federation steers blueprint placement; Fleet Sync replicates security state. The two do not interact. |
|
|
| [Fleet Actions](/features/fleet-actions) | Bulk imperative operations (fleet stop, bulk label) across labelled nodes. | Fleet Actions is imperative and operator-driven; Fleet Sync is declarative and automatic. |
|
|
| [Licensing](/features/licensing) | Community and Admiral plans. | How fleet capabilities relate to plans. |
|