AutopilotMonitor/docs/backend/vulnerability-correlation-sources.md

type, title, description, resource, tags, timestamp

type	title	description	resource	tags	timestamp
Concept	Vulnerability Correlation Sources (NVD, KEV, MSRC, EPSS)	How the software inventory is turned into CVE findings - CPE mapping, the per-CPE NVD cache with its budget and TTLs, KEV/MSRC/EPSS enrichment, the throttle contract that keeps a refused NVD answer out of the cache, and the act/attend/track priority.	src/Backend/AutopilotMonitor.Functions/Services/Vulnerability/VulnerabilityCorrelationService.cs	backend vulnerability nvd epss kev msrc cache	2026-08-29T18:00:00+02:00

Vulnerability Correlation Sources

The Software Inventory & Vulnerability Analyzer (tenant opt-in) ships the installed-software list with the session; the backend turns it into CVE findings entirely from cached public reference data. Four sources, all trustworthy primaries, no aggregators:

Source	What it contributes	Fetch shape	Cache
NVD 2.0 (NIST)	CVE records per product: CVSS v3.1 (v3.0 fallback) score, severity, vector, description, affected version ranges; CPE dictionary for auto-mapping	cves/2.0?virtualMatchString={cpe} per product, cpes/2.0?keywordSearch= for unmapped titles; optional API key (AdminConfiguration.NvdApiKey)	Azure Table VulnerabilityCache, one row per CPE (cve_{cpe} / cache_result), CveDataJson chunked; soft TTL 7 d (serve + background refresh), hard 14 d
CISA KEV	"Exploited in the wild" flag, due date, required action, ransomware use	Whole catalog JSON	In-memory dictionary, 24 h, blob mirror kev-catalog.json
MSRC	Microsoft-side confirmation: document id/title, release date, Microsoft severity / score / vector	CVRF v3.0 index of the last six Patch Tuesdays	In-memory index, blob mirror msrc-index.json
FIRST EPSS	Probability (0..1) that the CVE is exploited within 30 days + percentile + scoring date	api.first.org/data/v1/epss?cve=a,b,c - 100 ids per request, unauthenticated	Stored on the cached CVE (no separate table): filled when a CPE row is (re)fetched, re-scored for every row by the nightly sync

Nothing about a tenant leaves the backend. The one outbound query that carries content is the EPSS lookup, and its payload is the list of public CVE ids held in the shared, tenant-agnostic CPE cache. Customer-facing wording: autopilotmonitor-docs/trust/data-flows.md and the terms page attribution block.

Schema

Cached CVE (CachedCve, serialised into CveDataJson)

CveId, CvssScore, CvssSeverity, CvssVector?, Description (<=500), PublishedDate,
EpssScore?, EpssPercentile?, EpssDate?, AffectedVersions[] {Criteria, VersionStart/End Incl/Excl}

JsonOptions is WhenWritingNull + case-insensitive, so rows written before the vector / EPSS properties existed deserialise with those as null, and a row is never rewritten just to add nulls. CveCacheBudget bounds the row (24 months, or CVSS >= 7 within 60 months; max 100 CVEs per CPE, 50 ranges per CVE) - which is also why one CPE fill is exactly one EPSS request.

ProjectNvdResponse is the single NVD -> CachedCve projection; the live fill and the nightly refresh both call it, so both write the same shape. BuildCveCacheRow(cpe, cves, cachedAt) is the single row builder; cachedAt is the NVD freshness stamp, an EPSS-only rewrite passes the existing value through so it does not look like a fresh NVD fetch to the 7-day TTL.

Per-finding output (report findings[].vulnerabilities[])

cveId, cvssScore, cvssSeverity, cvssVector?, description, publishedDate, source ("NVD" | "NVD+MSRC"),
isKev, kevDueDate, kevRequiredAction, knownRansomware,
epssScore?, epssPercentile?, epssDate?,
priority ("act" | "attend" | "track"),
msrc* (when MSRC confirmed), versionMatchConfidence ("low" on wildcard CPE matches)

Finding level: riskLevel (unchanged CVSS+KEV bands), priority (highest band), maxEpssScore?. scan_summary additionally carries act_cves / attend_cves. MergeReports (Part 1 + Part 2) recomputes all of these. The CveIndex row (one per tenant/CVE/session) carries EpssScore (max over occurrences, absent when unscored) and Priority (highest), which the fleet summary (TopCve.epssScore / priority) reads.

Priority (CvePriority.Compute)

• act - in CISA KEV.
• attend - EPSS >= 0.1 or CVSS >= 9.0.
• track - everything else.

Three visible rules instead of a weighted composite: severity (riskLevel) keeps saying how bad, priority says how soon. A CVE without an EPSS score is "unknown likelihood" - it can still reach attend via CVSS, never via a defaulted 0. The thresholds are constants; changing them is a product decision, not a tuning knob.

Throttle contract (NVD)

The typed HttpClient runs under the ExternalDataApi Polly policy: 429/5xx/timeouts are retried three times (1/2/4 s + jitter) with a circuit breaker. A 429 that reaches NvdApiClient therefore means NVD is refusing us for real. Two things then happen, and one deliberately does not:

1. NvdFetchResult<T> reports Succeeded = false, Throttled = true. Data is an empty object so enumerating callers keep working, but nothing that persists may treat it as an answer: the CPE cache fill returns null and writes no row; the negative auto-resolve cache is not touched; the admin auto-resolve reports nvd_throttled. Before this contract an empty response was cached as "0 CVEs" for seven days - a product NVD happened to throttle disappeared from every report for a week.
2. A process-wide cooldown opens: Retry-After when present (delta or date, capped at 120 s), otherwise 30 s. During the cooldown every NVD call fails fast with Throttled - no wait, no retry chain. Worst case added latency per cooldown window is one Polly chain (~7.5 s), never a per-item stall; the nightly refresh breaks out of its loop on the first throttled answer and reports the remainder as cveSkippedThrottled.
3. Not done on purpose: no in-process retry beyond Polly, no VulnCheck-style mirror, no separate "throttled" cache state. Hard-expired cache rows keep serving while NVD is unavailable (better than nothing, logged); a product with no row at all is reported without CVE data for that run and gets a real attempt on the next correlation.

Nightly sync (VulnerabilityDataSync, 03:00 UTC) and the manual buttons

1. KEV catalog refresh (blob mirror updated).
2. MSRC index refresh (last six Patch Tuesdays).
3. NVD refresh of cache rows older than 7 days (sequential, breaks on throttle).
4. EPSS re-score of every cve_* row - all ids batched through FIRST, only rows whose score / percentile / date changed are rewritten (with their original CachedAt). A missing id in the response keeps yesterday's score: a failed batch never erases data.

Steps 3 and 4 live in VulnerabilityCacheRefreshService (singleton, one in-flight slot per walk), which the timer and the Global-Admin buttons share, so a manual run writes exactly what the nightly run writes. Four endpoints, one per source, each with its own "Last … Sync" line on the admin page (GET vulnerability/sync-status):

Button	Endpoint	Shape	Timestamp
Sync KEV	POST vulnerability/sync	synchronous	VulnerabilityDataLastSyncUtc
Sync MSRC	POST vulnerability/sync-msrc	synchronous	MsrcLastSyncUtc
Refresh NVD Cache	POST vulnerability/sync-nvd	202 + background (minutes, NVD-rate-limited); 409 while running; the page polls nvdRefreshRunning / nvdLastRun every 10 s	NvdCacheLastRefreshUtc - stamped only for a walk that reached every stale row (no throttle cut-off)
Sync EPSS	POST vulnerability/sync-epss	synchronous (seconds); 409 while running	EpssLastSyncUtc

The timer skips a walk whose slot a manual run currently holds (cveSkippedBusy in telemetry) rather than queueing a duplicate. nvdLastRun / epssLastRun are in-memory on the answering instance (null after a cold start); the persisted timestamps are the durable signal.

Telemetry event VulnerabilityDataSyncCompleted carries epssRows / epssCves / epssScored / epssRowsRewritten / epssError next to the KEV / MSRC / CVE counters; the manual EPSS run emits ManualEpssSyncCompleted.

Examples

• Chrome 126 finding: CVE-2024-4947 CVSS 8.8 HIGH, AV:N/AC:L/PR:N/UI:R, KEV yes -> priority = act, riskLevel critical (KEV) - unchanged severity semantics, priority now explicit.
• A 5.5 MEDIUM local-privilege CVE with EPSS 0.31 -> attend; the same CVE with EPSS 0.004 -> track. Same severity band, different order of work.
• Nightly sync while NVD throttles: [VulnSync] NVD throttled - leaving 37 stale entries for the next run, then [VulnSync] EPSS refresh: 1412/1450 CVEs scored across 212 rows, 180 rows rewritten - EPSS is independent of NVD's mood.

Not done (recorded)

• ExploitDB CSV as a "public exploit exists" flag (static file, daily, same shape as the KEV mirror) - useful, cheap, deliberately not added: four primary sources are enough for now and each additional outbound dependency is a trust-page line. Revisit if customers ask "is there a public exploit?".
• GitHub / Nuclei PoC search, composite 0-100 risk scores, CWE -> ATT&CK mapping - noise for an Intune admin, not adopted.

Citations

• src/Backend/AutopilotMonitor.Functions/Services/Vulnerability/NvdApiClient.cs - NvdFetchResult, cooldown, ComputeCooldown
• src/Backend/AutopilotMonitor.Functions/Services/Vulnerability/EpssApiClient.cs - batching, ParseResponse
• src/Backend/AutopilotMonitor.Functions/Services/Vulnerability/CvePriority.cs
• src/Backend/AutopilotMonitor.Functions/Services/Vulnerability/VulnerabilityCorrelationService.cs - ProjectNvdResponse, ApplyEpssScores, BuildCveCacheRow, StampFindingPriority, FetchAndCacheCvesFromNvdAsync
• src/Backend/AutopilotMonitor.Functions/Functions/Vulnerability/VulnerabilityDataSyncFunction.cs - RefreshStaleCveCacheAsync, RefreshEpssScoresAsync
• src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.AgentApi.cs - BuildCveIndexEntities, ScanCveIndexAsync
• src/Backend/AutopilotMonitor.Functions.Tests/EpssAndNvdThrottleTests.cs
• FIRST EPSS API: https://api.first.org/epss/ - CISA KEV: https://www.cisa.gov/known-exploited-vulnerabilities-catalog