Files
sencho/docs/features/vulnerability-scanning.mdx
T
Anso 3b650523c1 Audit-hardening pass for secret and misconfiguration scanning (#977)
* fix(security): dedupe concurrent compose-stack scans

Track stack scans in scanningImages keyed stack:<nodeId>:<stackName>.
The /scan/stack route returns 409 when an in-flight scan exists, and
the service-side check is the real correctness barrier (the route
pre-check is a fast-path optimization that mirrors scanImage). The
dedup key release lives in a try/finally so failed scans free the
slot for retry.

Why: scanComposeStack had no equivalent of scanImage's scanningImages
guard, so two simultaneous calls for the same stack would both run
trivy config, both insert a vulnerability_scans row, and double-
process the result.

* feat(security): acknowledge misconfig findings

Adds a parallel acknowledgement system for Trivy misconfig findings
that mirrors cve_suppressions: a new misconfig_acknowledgements table,
read-time enrichment via the new misconfig-ack-filter utility, REST
CRUD endpoints, fleet-sync replication from control to replicas, a
Settings panel, and an Acknowledge button on the Misconfigs tab.

Schema and behavior parity with cve_suppressions:
  - UNIQUE(rule_id, COALESCE(stack_pattern, '')) so fleet-wide acks
    collide as expected
  - blockIfReplica on every write
  - Audit-log entries name the scope (rule_id, stack_pattern) but
    never the reason text
  - replicated_from_control flag controls UI delete affordance and
    drives clearReplicatedRows on demote/reanchor
  - Validators reused: validateStackPatternForRedos for glob safety,
    sanitizeForLog for log fragments

SARIF export emits an external/accepted suppression entry per
acknowledged misconfig, matching the CVE pattern.

Per-row Acknowledge dialog prefills stack_pattern with the scan's
stack_context so the default scope is "rule + this stack only" and an
operator must broaden explicitly.

Tests: misconfig-ack-filter (15) and misconfig-ack-routes (23)
including the duplicate-409 case for both pinned and fleet-wide acks.

* fix(security): reap orphaned trivy tmp dirs at startup

When the buildEnv path writes a per-scan DOCKER_CONFIG dir under
os.tmpdir() and the process crashes before the finally block runs,
the dir leaks. Mirrors GitSourceService.sweepStaleTempDirs:
exported sweepStaleTrivyTempDirs is fire-and-forget at boot,
removes prefix-matching dirs older than 1 hour, swallows
permission/race failures, logs a single line if any were reaped.

* perf(security): emit per-batch summary for scanAllNodeImages

Adds one diag() line at the end of scanAllNodeImages summarising
unique image count, scanned, skipped, failed, violation count, and
elapsed time. Per-image diag inside scanImage stays useful for
debugging individual scans; the summary gives operators a single
fleet-level checkpoint when developer_mode is on.

* perf(security): cap SARIF export at 5000 findings per type

Replace the unbounded fetchAllPages walk on /scans/:id/sarif with a
hard limit of 5000 findings per type. When any type trips the cap,
emit run-level properties.truncated=true plus row_limit and per-type
totals so downstream tooling can flag the export as partial.
Console-warns for ops visibility.

A scan with 50k vulns previously streamed every row into memory
before serialising; the cap bounds memory and serialisation time at
the cost of completeness on pathological scans.

* docs(env): document TRIVY_BIN host-binary override

The env var is honored by TrivyService.detectTrivy as a fallback when
no managed install is present, but it was undocumented in
.env.example. Adds the var with a comment explaining precedence
(managed > TRIVY_BIN > PATH).

* test(security): cover scanComposeStack failure modes

Two new cases drive the existing try/catch through real failure
paths:
  - Malformed Trivy stdout: row flips to status='failed' with the
    parser error preserved on `error`.
  - execFile rejection: row flips to status='failed' with a string
    error message.

Pairs with the existing dedup tests so the failure path now also
verifies the scan row state, not just the thrown exception.

* test(e2e): security scanner + misconfig acknowledgement flow

Seven Playwright tests covering the scanner UI and the new
acknowledgement system end-to-end:
  - Trivy availability gate (skips suite when binary absent so CI
    without Trivy can opt out via E2E_SKIP_TRIVY=1)
  - Stack config scan completes and records misconfig findings
  - Concurrent stack scan returns 409 from the dedup gate
  - Misconfig ack POST creates and lists on Settings
  - Duplicate (rule_id, stack_pattern) returns 409
  - Malformed rule_id (shell metacharacters) returns 400
  - Misconfigs tab renders against a real stack scan

Tests drive the API for behaviour assertions and the UI only for
shell-rendering checks; the visual snapshot suite owns screenshots.

* docs(features): add misconfig acknowledgement workflow and SARIF cap

Refreshes vulnerability-scanning.mdx with:
  - Misconfig acknowledgements section covering the per-row dialog,
    Settings panel, scope/matching rules, and SARIF emission
  - Tier table row for the new feature
  - SARIF section note on the 5000 row-per-type cap and the
    properties.truncated marker for partial exports
  - Troubleshooting entries: SARIF cap, hidden Acknowledge button,
    findings resurfacing after delete, Trivy DB phone-home, and
    409 on concurrent compose-stack scans

* fix(ci): clear backend lint and CodeQL alerts

- Remove the dead fetchAllPages helper in routes/security.ts. It lost
  its callers when the SARIF endpoint switched to direct paged reads
  for the truncation cap. ESLint flagged it as unused.
- Switch the trivy-tmp-cleanup test helper to fs.mkdtempSync. Building
  paths under os.tmpdir() with predictable names tripped CodeQL's
  js/insecure-temporary-file rule (high severity), which warns about
  symlink-pre-creation attacks even in test code. mkdtempSync appends
  a process-random suffix and creates the dir atomically; the
  sencho-trivy- prefix is preserved so the production sweep still
  matches the test fixtures.
2026-05-07 19:23:11 -04:00

476 lines
32 KiB
Plaintext

---
title: "Vulnerability Scanning"
description: "Scan container images for known CVEs, surface severity badges in the Resources Hub, and alert on policy violations."
---
Sencho integrates with [Trivy](https://trivy.dev) to scan container images for known vulnerabilities (CVEs), surface severity badges next to your images, and alert when a scan result exceeds a configured threshold. Manual scanning, secret and misconfiguration detection, scan comparison, and CVE suppressions are all available on every tier. Skipper and Admiral add automation, policy enforcement, and compliance exports.
<Frame>
<img src="/images/vulnerability-scanning/resources-badges.png" alt="Resources Hub showing vulnerability severity badges next to image tags" />
</Frame>
## Prerequisites
The Trivy CLI must be available on the machine running Sencho. Trivy is not bundled with the Sencho Docker image; see [Installing Trivy](/operations/trivy-setup) for mount and installation options. Sencho checks for Trivy on startup and hides scanning UI when the binary is not available.
<Note>
Trivy is installed independently on each Sencho instance. When you select a remote node, **Settings > Security** shows only the scanner status for that node; install, update, or uninstall Trivy from there to manage the remote's binary. Scan policies and CVE suppressions are managed on the control node and apply fleet-wide at read time.
</Note>
## Tier availability
| Feature | Community | Skipper | Admiral |
|---------|:---------:|:-------:|:-------:|
| Install / update / uninstall Trivy from Settings | ✓ | ✓ | ✓ |
| On-demand image scanning (vulnerabilities) | ✓ | ✓ | ✓ |
| Severity badges in the Resources Hub | ✓ | ✓ | ✓ |
| Scan results drawer with vulnerability table | ✓ | ✓ | ✓ |
| Post-deploy automated scanning | ✓ | ✓ | ✓ |
| Secret detection in image filesystems | ✓ | ✓ | ✓ |
| Compose file misconfiguration scanning | ✓ | ✓ | ✓ |
| Scan history and comparison | ✓ | ✓ | ✓ |
| CVE suppressions | ✓ | ✓ | ✓ |
| Misconfig acknowledgements | ✓ | ✓ | ✓ |
| Scheduled fleet scans (all images on a node) | | ✓ | ✓ |
| Scan policies with `block_on_deploy` enforcement | | ✓ | ✓ |
| SBOM generation (SPDX, CycloneDX) | | ✓ | ✓ |
| SARIF export (code scanning integration) | | ✓ | ✓ |
| Auto-update of the managed Trivy binary | | | ✓ |
## On-demand scanning
Navigate to the **Resources** tab and open the **Images** panel. When Trivy is available, every image row shows a shield icon alongside the delete action.
1. Click the shield icon on any image row. A menu appears with two options:
- **Scan (vulnerabilities)**: the default, fastest path. Trivy inspects package metadata only.
- **Full scan (vulnerabilities + secrets)**: additionally walks the image filesystem for hardcoded credentials, tokens, and keys. This takes noticeably longer.
2. The row shows a loading spinner while Trivy runs. Most vulnerability scans finish in 10 to 60 seconds depending on image size and whether the Trivy database is already cached. Full scans add the time needed to read the filesystem.
3. When the scan completes, a severity badge appears next to the image status (e.g. `CRITICAL`, `HIGH`, `MEDIUM`, `LOW`, or `CLEAN`).
4. Click the badge to open the scan results drawer.
<Frame>
<img src="/images/vulnerability-scanning/scan-details-sheet.png" alt="Vulnerability scan results drawer showing CVE table with severity, package, and fix columns alongside a policy violation banner" />
</Frame>
### Reading severity badges
Hovering over a severity badge reveals the breakdown of vulnerabilities by severity and the timestamp of the last scan. The badge color reflects the highest severity found:
| Badge | Meaning |
|-------|---------|
| **CRITICAL** (red) | At least one vulnerability with a CVSS score of 9.0 or higher |
| **HIGH** (amber) | At least one high-severity vulnerability |
| **MEDIUM** (blue) | Only medium and lower vulnerabilities |
| **LOW** (muted) | Only low-severity vulnerabilities |
| **CLEAN** (green) | Scan completed with zero findings |
Scan results are cached by image digest. If the same digest is scanned again within 24 hours, Sencho returns the cached result instantly instead of re-running Trivy.
## The scan results drawer
The drawer shows a full breakdown of the most recent scan for an image and groups findings across three tabs: **Vulnerabilities**, **Secrets**, and **Misconfigs**. The summary header shows counts across all three so you can see the full risk picture at a glance.
- **Summary**: counts per severity (critical, high, medium, low), total vulnerabilities, how many have a fix available, the Trivy version used, and when the scan ran.
- **Vulnerabilities tab**: severity filter pills narrow the table, paginated list of every CVE found, including:
- **CVE ID** (CVE-prefixed identifiers link to [cve.org](https://www.cve.org); GHSA identifiers link to the GitHub Advisory Database)
- **Package** name and installed version
- **Severity** badge
- **Fixed version** with a green indicator if a fix is available
Critical and high rows in the table are tinted with a left accent rail so the rows that need attention catch the eye even before you read the severity column.
If the scan was evaluated against a [scan policy](#scan-policies) and the highest severity meets or exceeds the policy threshold, a destructive **Policy violation** banner appears at the top of the drawer naming the policy and the threshold it crossed.
- **Secrets tab**: hardcoded credentials or keys detected in the image filesystem, with severity, rule, title, and the file/line location. Secret values are redacted: only the first eight characters of the match are stored.
- **Misconfigs tab**: misconfiguration findings with severity, check ID, title, target file, and a suggested resolution. For image scans this tab is empty; for stack config scans (see below) it is the primary view.
### Actions
From the drawer header you can:
- **Re-scan**: kick off a fresh scan, ignoring the digest cache.
- **Download SBOM**: export a Software Bill of Materials in SPDX JSON or CycloneDX format (Skipper and Admiral).
- **Export CSV**: export the full vulnerability list for offline review.
- **Export SARIF**: download the full scan (vulnerabilities, secrets, and misconfigs) as SARIF 2.1.0 for upload to GitHub code scanning or other SARIF-aware tooling (Skipper and Admiral).
## Post-deploy automated scanning
When Trivy is available, Sencho automatically scans every deployed image in the background after a successful deploy. This applies to all deploy paths:
- Stack deploy and redeploy
- Stack update
- Template deploy from the App Store
- Git source apply
- Git source create
The deploy itself is never blocked by scanning; scans run asynchronously and surface their results via the severity badges in the Resources Hub. If high or critical vulnerabilities are found, an alert is dispatched through your configured [notification channels](/features/alerts-notifications).
### Opting out per deployment
The App Store deploy sheet includes an **Scan images for vulnerabilities after deploy** toggle (enabled by default). Uncheck it to skip the post-deploy scan for that single deployment. This does not disable scan policies globally; it simply opts this deployment out of scanning.
<Frame>
<img src="/images/vulnerability-scanning/app-store-toggle.png" alt="App Store deploy sheet showing the auto-scan checkbox enabled by default" />
</Frame>
## Scheduled fleet scans
<Note>
Scheduled scans require a **Skipper** or **Admiral** license.
</Note>
You can run recurring scans of every image on a node through the standard [Scheduled Operations](/features/scheduled-operations) system. Create a new scheduled task with action **Scan** and a cron expression. The scheduler iterates every image on the target node with a short delay between scans and records the result in the task's run history.
Use scheduled scans to keep CVE badges fresh even for images that are rarely redeployed: nightly (`0 3 * * *`) is a good default for most fleets.
### Completion notifications
When a scheduled scan finishes, Sencho dispatches an alert through your configured [notification channels](/features/alerts-notifications). The alert includes the task name and a summary of scanned, cached, and failed images:
- **Info** when every image scanned successfully.
- **Warning** when one or more images in the run failed to scan.
Failures are typically transient (registry timeouts, missing credentials) and do not stop the rest of the run from completing. Check the task's run history for the detailed output.
## Scan policies
<Note>
Scan policies require a **Skipper** or **Admiral** license.
</Note>
Policies let you define severity thresholds that govern whether a stack can deploy at all. A policy with **Block on deploy** enabled runs a pre-flight scan on every image in the stack before `docker compose up` executes; if any image meets or exceeds the threshold, the deploy is rejected with a dialog listing the offending images. Policies with **Block on deploy** disabled still evaluate every post-deploy and scheduled scan, and dispatch warning alerts when the threshold is exceeded.
See [Deploy Enforcement](/features/deploy-enforcement) for the full pre-flight flow, admin bypass path, and audit-log behavior.
<Frame>
<img src="/images/vulnerability-scanning/security-settings.png" alt="Security section of Settings showing the scan policies list with add policy button" />
</Frame>
### Creating a policy
Go to **Settings → Security** and click **Add Policy**.
| Field | Description |
|-------|-------------|
| **Name** | A descriptive label (e.g. "Production critical block"). |
| **Stack pattern** | Optional glob against stack names (e.g. `prod-*`). Leave empty to match every stack. |
| **Max severity** | The threshold. If a scan finds any vulnerability at or above this severity, the policy fires. |
| **Block on deploy** | When enabled, deploys are rejected before `docker compose up` runs if any image violates the threshold. When disabled, the policy still evaluates post-deploy and scheduled scans and dispatches warning alerts on violations. |
| **Enabled** | Disabled policies are skipped during evaluation. |
### Policy scoping
When multiple policies match a deploy, Sencho picks the most specific one:
1. Policies scoped to a specific node win over global policies.
2. Policies with a stack pattern win over wildcard policies.
3. Disabled policies are never applied.
Only one policy is evaluated per deploy; use a single tight pattern rather than overlapping policies for clarity.
<Note>
Policies created on a control instance replicate to every remote in the fleet automatically. See [Fleet Sync](/features/fleet-sync) for the replication, push retry, and replica demote behavior.
</Note>
### Example policies
**Block criticals and highs in production.** A tight gate that keeps known vulnerable base images out of your production fleet.
- **Name:** `prod-high-gate`
- **Stack pattern:** `prod-*`
- **Max severity:** `HIGH`
- **Block on deploy:** On
- **Enabled:** On
**Alert on criticals in staging, never block.** Lets engineers iterate without friction while still surfacing critical findings for triage.
- **Name:** `staging-critical-alert`
- **Stack pattern:** `staging-*`
- **Max severity:** `CRITICAL`
- **Block on deploy:** Off
- **Enabled:** On
### Creating a policy via the API
Policy CRUD endpoints are documented in the [Security API reference](/api-reference/security). A typical create call from CI looks like this:
```bash
curl -X POST https://your-sencho-instance:1852/api/security/policies \
-H "Authorization: Bearer YOUR_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "prod-high-gate",
"stack_pattern": "prod-*",
"max_severity": "HIGH",
"block_on_deploy": 1,
"enabled": 1
}'
```
## SBOM generation
<Note>
SBOM generation requires a **Skipper** or **Admiral** license.
</Note>
A Software Bill of Materials (SBOM) is a machine-readable inventory of every package present in a container image. SBOMs are required by an increasing number of security frameworks (SLSA, Executive Order 14028, EU Cyber Resilience Act) and are useful for offline compliance reviews.
From the scan results drawer, click **Download SBOM** and choose a format:
| Format | Use case |
|--------|----------|
| **SPDX JSON** | Widely supported, best for tooling integration. |
| **CycloneDX** | Richer dependency metadata, better for supply-chain analysis. |
The download starts immediately and uses the image's digest (when available) in the filename.
## Secret detection
<Note>
Secret detection requires a **Skipper** or **Admiral** license.
</Note>
Full scans ask Trivy to walk the image filesystem for patterns that look like hardcoded credentials, API tokens, cloud access keys, or private keys. Detection rules cover common providers (AWS, GCP, GitHub, Slack, Stripe) plus generic high-entropy strings.
To run a full scan, click the shield icon in the Resources Hub and pick **Full scan (vulnerabilities + secrets)**. Findings appear on the **Secrets** tab of the scan drawer:
| Column | Description |
|--------|-------------|
| **Severity** | Trivy-assigned severity for the rule that matched. |
| **Rule** | The detection rule identifier (e.g. `aws-access-key-id`). |
| **Title** | A short description of what was detected. The second line shows a redacted excerpt of the match. |
| **Target** | The file path inside the image filesystem, including the line number range when available. |
Only the first eight characters of any matched secret are stored, followed by an ellipsis. The full value is never written to the database, so exporting a scan drawer or comparing scans cannot leak the underlying credential.
Full scans take longer than vulnerability-only scans because Trivy reads every file in the image. If runtime is a concern, schedule full scans overnight and keep deploy-time scans on the default vulnerability-only setting.
## Compose misconfiguration scanning
<Note>
Compose misconfiguration scanning requires a **Skipper** or **Admiral** license.
</Note>
Beyond package CVEs, Sencho can run `trivy config` against a stack's Compose file to flag insecure defaults before you deploy. Typical checks include containers running as root, missing resource limits, privileged mode, host network, and mounted Docker sockets.
From any stack page, click **Scan config** next to the Deploy controls. Sencho runs the scanner against the stack's working directory and opens the scan drawer on the **Misconfigs** tab:
| Column | Description |
|--------|-------------|
| **Severity** | Rule severity (CRITICAL/HIGH/MEDIUM/LOW). |
| **Check** | Rule ID or AVD identifier for the violated check. |
| **Title** | Short summary, linked to the upstream advisory when available. The second line shows Trivy's detailed message. |
| **Target** | The file that triggered the finding. |
| **Fix** | The recommended resolution. |
Config scans are stored in the same history as image scans with an `image_ref` of `stack:<name>`, so they appear on the Scan history page and can be exported as CSV.
## Misconfig acknowledgements
Some misconfigurations are intentional. A reverse-proxy stack legitimately needs root to bind privileged ports; a network monitor might require host networking; an `--privileged` Docker socket mount might be exactly what your janitor service expects. Sencho lets admins acknowledge a rule so it stops triggering alerts without lowering the policy bar for every other stack.
Acknowledgements never modify stored finding rows. They are applied at read time, so deleting an acknowledgement immediately resurfaces the finding wherever it appears.
### Acknowledging from a scan result
1. Open a stack config scan that contains the finding.
2. Click the shield-check icon at the right edge of the misconfig row. The dialog opens with the rule id prefilled and the stack pattern set to the current stack name (so a single click acknowledges *only this stack* — narrowest possible scope by default).
3. Add a reason explaining why the misconfiguration is accepted. The reason is stored locally and replicated fleet-wide; it never appears in audit-log summaries to avoid leaking incident-tracker IDs or vendor secrets.
4. Optionally set an expiry in days. After expiry the acknowledgement stops applying and the finding resurfaces.
The acknowledged row renders dimmed with a strikethrough title; hovering surfaces the acknowledgement reason.
### Managing acknowledgements
**Settings > Security** has a panel listing every acknowledgement on this control: rule id, optional stack pattern (glob), creator, expiry date, and a delete button. The same `replicated` badge that appears on CVE suppressions appears here for rows pushed from the control to a replica.
Replicas show the panel read-only — write operations return 403 with a "managed by control" message so configuration drift cannot accumulate on the leaf nodes.
### Scope and matching
An acknowledgement matches a misconfig finding when:
- `rule_id` equals the finding's `rule_id` (exact match), and
- `stack_pattern` is null **or** matches the scan's stack name via the same glob syntax used elsewhere (e.g. `traefik`, `web-*`), and
- the acknowledgement has not expired.
When more than one acknowledgement could match, Sencho picks the most specific: a stack-pinned ack beats a fleet-wide ack for the same rule.
### SARIF emission
Acknowledged misconfigs are emitted in the SARIF export with a `suppressions` entry of kind `external` and status `accepted`, mirroring CVE suppressions. Code-scanning dashboards that respect SARIF suppressions will dismiss them with the recorded justification.
## SARIF export
<Note>
SARIF export requires a **Skipper** or **Admiral** license.
</Note>
SARIF (Static Analysis Results Interchange Format) is the standard format supported by GitHub code scanning, Microsoft Defender for Cloud, and most security dashboards. Sencho generates SARIF 2.1.0 documents directly from the stored scan results so the download matches what you see in the drawer (same findings, same suppression state) without re-running Trivy.
From the scan drawer header, click **SARIF** to download the report. The file is named after the image reference with a `.sarif.json` extension.
What the export contains:
- **Vulnerabilities**: one SARIF result per CVE, with `security-severity` scored 9.8 (CRITICAL), 7.5 (HIGH), 5.0 (MEDIUM), 2.5 (LOW), or 0.0 (UNKNOWN). The affected package appears as a logical location (`<pkg>@<version>`).
- **Secrets**: rule IDs are namespaced as `SECRET:<rule>`. Results point at the file and line number where the match was found.
- **Misconfigs**: rule IDs are namespaced as `MISCONFIG:<rule>`. Results point at the Compose file that triggered the check.
- **Suppressions**: CVEs you have suppressed in Sencho are emitted with a SARIF `suppressions` entry of kind `external` and status `accepted`, so code-scanning dashboards can dismiss them with the justification you recorded.
- **Acknowledged misconfigs**: misconfigs you have acknowledged in Sencho are emitted with the same `suppressions` shape so dashboards apply the same dismissal logic.
The export caps each finding type at 5000 rows to bound memory and serialisation time on pathological scans. When any type trips the cap, the SARIF run carries `properties.truncated = true`, `properties.row_limit`, and a `properties.totals` object with the original counts so downstream tooling can flag the export as partial.
Typical upload flow for GitHub code scanning:
```yaml
- name: Upload SARIF to GitHub
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: sencho-scan.sarif.json
```
## Scan history
Every scan Sencho runs is stored with its full vulnerability detail. Scan records are automatically pruned after 90 days to keep the database compact. The history is used to power:
- **Digest caching**: skip re-scanning an image that has already been scanned within 24 hours.
- **Trend badges**: surface whether the latest scan added or resolved vulnerabilities compared to the previous scan for the same image.
Click **Scan history** from the top of the Resources Hub to open a right-side sheet layered over the current page. The sheet lists completed scans grouped by image, lets you search by image reference, and lets you pick two scans to compare. Close the sheet by pressing Escape, clicking the overlay, or clicking the close button in the header.
<Frame>
<img src="/images/vulnerability-scanning/scan-history-sheet.png" alt="Scan history sheet overlaid on the Resources Hub, with search box, pagination, and scan rows grouped by image" />
</Frame>
## Comparing scans <Badge>Skipper</Badge>
Compare any two completed scans for an image to see what changed between them.
**From the Scan history page**: select two scans via the checkboxes (one baseline, one newer) and click **Compare**. Selecting a third scan replaces the oldest selection.
**From an open scan**: click **Compare** in the drawer header, then pick a baseline scan from the dropdown. Only completed scans for the same image appear.
The comparison sheet shows:
- A **delta ribbon** summarizing the net change per severity (CRITICAL, HIGH, MEDIUM, LOW). Net-positive deltas on CRITICAL render in destructive red so a regression on the worst tier is immediately visible.
- Filter pills to switch between **Added** (new findings since the baseline), **Removed** (resolved findings), and **Unchanged** (findings present in both).
- A sorted table of CVEs with severity, affected package, and direct links to the upstream advisory. CVE-prefixed identifiers resolve to [cve.org](https://www.cve.org); GHSA identifiers resolve to the GitHub Advisory Database. Critical and high rows carry the same left-rail tint as the scan results drawer for visual continuity.
Comparisons are scoped to a single node; scans taken on different nodes cannot be compared against each other.
Cross-image comparisons (picking scans from two different image references) are allowed but flagged with a warning, since package-level changes may reflect image differences rather than CVE drift. In that mode, the **Unchanged** pill is relabeled to **Shared** to reflect that same CVE + package matches across different images are not necessarily the same finding.
Up to 1000 findings per scan are loaded for comparison. When a scan exceeds this limit, the sheet shows a banner indicating the comparison may be incomplete.
<Frame>
<img src="/images/vulnerability-scanning/scan-compare-sheet.png" alt="Compare scans sheet with delta ribbon, Added/Removed/Unchanged filter pills, and a diff table tinted by severity" />
</Frame>
## How it works
1. On startup, Sencho looks for the `trivy` binary on `PATH` and caches its availability.
2. When a scan is triggered, Sencho resolves the image digest via Docker and checks the 24-hour cache. If a completed scan exists for that digest, it is returned instantly.
3. Otherwise, Sencho spawns `trivy image --format json --quiet <image-ref>` and parses the JSON output into the vulnerability database.
4. Private registry credentials are forwarded automatically by writing a temporary `DOCKER_CONFIG` for the Trivy subprocess, then deleting it when the scan completes.
5. Scan status, counts per severity, and the full vulnerability list are persisted for display in the drawer and for policy evaluation.
## Troubleshooting
### Scan button is not visible
Sencho hides scanning UI when the Trivy binary is not detected. Check **Settings → Support** for the Trivy availability status, then follow [Installing Trivy](/operations/trivy-setup) if it is missing.
### Scans time out
The default scan timeout is 5 minutes. Very large images (2+ GB) over a slow connection may exceed this; pre-pulling the image to the host speeds up the scan significantly because Trivy works against the local image store.
### Private registry images fail to scan
Sencho forwards the same registry credentials configured under **Settings → Registries** to Trivy during a scan. If a pull works in Sencho but a scan fails, make sure the image has been pulled at least once (Trivy can then work against the cached local image).
### Badge is out of date after an image update
Post-deploy scanning only runs on deploy actions. For long-running images that aren't redeployed, schedule a recurring scan (Skipper+) or click the shield icon in the Resources Hub to re-scan on demand.
### A scan shows "in progress" for a long time
Scans have a 5-minute internal timeout. The scheduler also sweeps every tick and marks any scan that has been `in progress` for more than 15 minutes as failed, so the UI always recovers on its own. Wait for the sweep to run, then click the shield icon again to start a fresh scan.
### Trivy is not detected after installing it
Sencho re-checks for the Trivy binary every ten minutes from the scheduler. Install Trivy on the host, wait for the next window, and the scanning UI lights up automatically.
### Collecting diagnostic logs for support
Enable **Developer Mode** under **Settings → Developer** and trigger the failing scan again. The backend logs verbose `[Trivy:diag]` entries covering detection timing, cache hits, Trivy invocation, and parse statistics. Attach these lines when filing a support issue. Turn Developer Mode off once you have captured the output.
### Post-deploy scan failure notifications
When a post-deploy scan fails for a specific image (for example because Trivy could not resolve a private registry pull), Sencho dispatches a warning-level alert through your configured notification channels. The deploy itself is never blocked by a scan failure.
### A deploy was blocked by a policy I did not expect
The block dialog names the policy that fired and lists every image that violated the threshold. Open **Settings → Security → Scan Policies** and review the matching policy: check the stack pattern glob and the max severity. If the policy should not apply, tighten the pattern (for example `staging-*` instead of `*`) or turn **Block on deploy** off to keep the evaluation in alert-only mode. Admins can also bypass a single deploy with the **Deploy anyway** button; every bypass is recorded in the [Audit Log](/features/audit-log) with the actor, policy, and violation list.
### Trivy is not installed and a deploy with a block policy went through
Sencho fails open when Trivy is not installed on the target node, so users are never locked out by tooling state. A warning alert is dispatched through your configured notification channels with the message `Pre-deploy scan for "<stack>" skipped: Trivy not installed on this node`. Install Trivy from **Settings → Security** to enforce the policy; see [Installing Trivy](/operations/trivy-setup) for options.
### Compare button is disabled
Scan comparison is a Skipper feature; on Community, the Compare button stays disabled with a tooltip explaining the upgrade path. If your license is Skipper or Admiral, make sure you have ticked exactly two completed scans: selecting zero, one, or three scans leaves the button disabled. If you have only one scan for an image, trigger a second scan from the Resources Hub (or wait for a scheduled scan), then return to the Scan history page and tick both.
### Comparison shows unexpected results
Two common causes:
- **Cross-image comparison.** If the Baseline and Current rows at the top of the sheet point at different image references, the warning banner is shown and the "Unchanged" pill is labeled "Shared". Items in that bucket match on CVE + package name but may not be the same finding across two distinct images. Stick to scans of the same image reference for apples-to-apples drift analysis.
- **Truncated scans.** When either scan has more than 1000 stored findings, the sheet shows a truncation banner. In that case, items past the 1000-row cap do not contribute to the Added / Removed / Unchanged buckets and the totals may be misleading. Re-run the scan with a tighter image (or scope the investigation to the most severe findings) to avoid truncation.
### An older scan is missing from Scan history
The Scan history page uses server-driven pagination. If you know the scan exists but cannot see it, use the search box to filter by image reference, or page forward with the arrows in the card header. Scans older than 90 days are pruned automatically to keep the database compact.
### Scan policies are missing on one of my nodes
Scan policies are managed from the control Sencho instance and replicate to every remote. When you view **Settings → Security** on a remote Sencho (a replica), you will see a banner explaining that rules are managed upstream. See [Fleet Sync](/features/fleet-sync) for how replication works and how to investigate push failures.
### I suppressed a CVE but the scan badge count is unchanged
Badge counts reflect the raw findings so alerting and policy evaluation stay accurate. Open the scan drawer to confirm the row is dimmed with a shield-off icon. See [CVE Suppressions](/features/cve-suppressions) for how the filter is applied across the drawer, compare sheet, and other read surfaces.
### The Secrets tab is empty on an image I expect to contain credentials
Secret detection matches against Trivy's built-in rule set, which focuses on well-known provider patterns. Plain text passwords, custom token formats, or values that do not match any published rule will not appear. Ensure you picked **Full scan (vulnerabilities + secrets)** from the shield-icon menu; a plain vulnerability scan does not walk the filesystem.
### Scan config button is disabled on a stack
The button is only shown when Trivy is available on the stack's node, the current user is an admin, and the license is Skipper or Admiral. If all three conditions are met but the button stays disabled, another stack action (deploy, update, rollback) is still in progress; wait for it to finish.
### Compose misconfiguration scan returns 404
The scanner needs to locate a Compose file in the stack directory. If the stack was created outside Sencho and the working directory does not contain a file named `compose.yml`, `compose.yaml`, `docker-compose.yml`, or `docker-compose.yaml`, the scan returns 404. Name the file accordingly or keep the stack under Sencho's managed compose directory.
### SARIF download returns 409 "Scan not complete"
SARIF export requires a completed scan. If a scan failed, timed out, or is still running, the button downloads nothing and the server returns a 409. Trigger a fresh scan from the Resources Hub or the stack page, wait for the drawer to populate, then export again.
### SARIF download is missing findings I see in the drawer
Each finding type (vulnerabilities, secrets, misconfigs) is capped at 5000 rows in the SARIF export. When any type trips the cap, the SARIF run includes `properties.truncated: true` along with the original counts, and the server logs a warning. The drawer paginates beyond 5000 so it shows everything, but the export is bounded for memory safety. If you need every row, narrow the scope (per-stack scan, per-image scan) before exporting.
### Acknowledge button is missing on a misconfig finding
The button is admin-only and hidden on replica nodes. Replicas read acknowledgements from the control via fleet sync; mutations must happen on the control. If you are an admin on the control and still do not see the button, the row is likely already acknowledged — look for the dimmed/strikethrough rendering and hover for the recorded reason.
### Findings reappeared after deleting a CVE suppression or misconfig acknowledgement
Suppressions and acknowledgements are applied at read time and never modify the stored finding rows. Removing one immediately resurfaces the underlying finding everywhere it appears (drawer, compare sheet, badge counts, SARIF export). The behaviour is intentional: deleting an acknowledgement is meant to revert the operator decision, not to mask history.
### Outbound traffic to ghcr.io / aquasecurity from the scanner host
Trivy itself fetches its CVE and secret-rule database from public registries on first scan and refreshes periodically; that egress is required for vulnerability scanning to work. Sencho does not emit telemetry of its own. If your environment forbids egress, see Trivy's [air-gapped scanning guide](https://aquasecurity.github.io/trivy/latest/docs/advanced/air-gap/) for how to pre-seed the database and run scans with `--offline-scan`.
### Compose stack scan returns 409 "Already scanning this stack"
Sencho deduplicates concurrent scans of the same stack so two simultaneous calls cannot double-process the result. Wait for the in-flight scan to finish (its row appears in Scan history with status `in_progress` and flips to `completed` or `failed` shortly after) and trigger again.