From cf76c4338f394637c37b5ca32fc2c23bd2f8e9c4 Mon Sep 17 00:00:00 2001 From: Oliver Kieselbach Date: Sat, 29 Aug 2026 11:57:55 +0200 Subject: [PATCH] feat(vuln): manual "Refresh NVD Cache" and "Sync EPSS" with their own last-sync lines MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The admin Data Sync Status only offered KEV and MSRC; the NVD stale-row refresh and the EPSS re-score ran on the nightly timer alone. Both walks move out of the timer into VulnerabilityCacheRefreshService (singleton, one in-flight slot per walk) so the buttons and the nightly run share one code path and stamp the same timestamps (NvdCacheLastRefreshUtc — only for a walk that reached every stale row; EpssLastSyncUtc). POST vulnerability/sync-nvd starts the minutes-long, rate-limited NVD walk in the background and answers 202 (409 while running); the page polls sync-status (nvdRefreshRunning, nvdLastRun) every 10 s meanwhile. POST vulnerability/sync-epss runs synchronously and returns the counts. The timer skips a slot a manual run holds instead of queueing a duplicate. Shared manifest regenerated for the two new AdminConfiguration fields. Co-Authored-By: Claude Fable 5 --- .../vulnerability-correlation-sources.md | 21 +- docs/log.md | 4 + .../TableStorage/TableConfigRepository.cs | 4 + .../GetVulnerabilitySyncStatusFunction.cs | 27 +- .../Vulnerability/TriggerEpssSyncFunction.cs | 86 +++++ .../TriggerNvdCacheRefreshFunction.cs | 51 +++ .../VulnerabilityDataSyncFunction.cs | 271 ++------------- .../AutopilotMonitor.Functions/Program.cs | 1 + .../Security/EndpointAccessPolicyCatalog.cs | 2 + .../VulnerabilityCacheRefreshService.cs | 328 ++++++++++++++++++ .../Models/Config/AdminConfiguration.cs | 13 + .../components/VulnerabilityDataSection.tsx | 145 ++++++++ src/Web/autopilot-monitor-web/lib/api.ts | 2 + .../types/adminConfig.ts | 2 + .../utils/shared-manifests.generated.ts | 4 +- .../utils/shared-manifests.json | 4 +- 16 files changed, 715 insertions(+), 250 deletions(-) create mode 100644 src/Backend/AutopilotMonitor.Functions/Functions/Vulnerability/TriggerEpssSyncFunction.cs create mode 100644 src/Backend/AutopilotMonitor.Functions/Functions/Vulnerability/TriggerNvdCacheRefreshFunction.cs create mode 100644 src/Backend/AutopilotMonitor.Functions/Services/Vulnerability/VulnerabilityCacheRefreshService.cs diff --git a/docs/backend/vulnerability-correlation-sources.md b/docs/backend/vulnerability-correlation-sources.md index 1adaf8b6..7f3ae8ad 100644 --- a/docs/backend/vulnerability-correlation-sources.md +++ b/docs/backend/vulnerability-correlation-sources.md @@ -100,7 +100,7 @@ means NVD is refusing us for real. Two things then happen, and one deliberately (better than nothing, logged); a product with no row at all is reported without CVE data for that run and gets a real attempt on the next correlation. -# Nightly sync (`VulnerabilityDataSync`, 03:00 UTC) +# Nightly sync (`VulnerabilityDataSync`, 03:00 UTC) and the manual buttons 1. KEV catalog refresh (blob mirror updated). 2. MSRC index refresh (last six Patch Tuesdays). @@ -109,8 +109,25 @@ means NVD is refusing us for real. Two things then happen, and one deliberately percentile / date changed are rewritten (with their original `CachedAt`). A missing id in the response keeps yesterday's score: a failed batch never erases data. +Steps 3 and 4 live in `VulnerabilityCacheRefreshService` (singleton, one in-flight slot per walk), +which the timer and the Global-Admin buttons share, so a manual run writes exactly what the nightly +run writes. Four endpoints, one per source, each with its own "Last … Sync" line on the admin page +(`GET vulnerability/sync-status`): + +| Button | Endpoint | Shape | Timestamp | +| --- | --- | --- | --- | +| Sync KEV | `POST vulnerability/sync` | synchronous | `VulnerabilityDataLastSyncUtc` | +| Sync MSRC | `POST vulnerability/sync-msrc` | synchronous | `MsrcLastSyncUtc` | +| Refresh NVD Cache | `POST vulnerability/sync-nvd` | **202 + background** (minutes, NVD-rate-limited); 409 while running; the page polls `nvdRefreshRunning` / `nvdLastRun` every 10 s | `NvdCacheLastRefreshUtc` - stamped only for a walk that reached every stale row (no throttle cut-off) | +| Sync EPSS | `POST vulnerability/sync-epss` | synchronous (seconds); 409 while running | `EpssLastSyncUtc` | + +The timer skips a walk whose slot a manual run currently holds (`cveSkippedBusy` in telemetry) +rather than queueing a duplicate. `nvdLastRun` / `epssLastRun` are in-memory on the answering +instance (null after a cold start); the persisted timestamps are the durable signal. + Telemetry event `VulnerabilityDataSyncCompleted` carries `epssRows / epssCves / epssScored / -epssRowsRewritten / epssError` next to the KEV / MSRC / CVE counters. +epssRowsRewritten / epssError` next to the KEV / MSRC / CVE counters; the manual EPSS run emits +`ManualEpssSyncCompleted`. # Examples diff --git a/docs/log.md b/docs/log.md index a21b1918..50bc69cd 100644 --- a/docs/log.md +++ b/docs/log.md @@ -1,5 +1,9 @@ # Log +## 2026-08-29 (3) + +* **Update**: `backend/vulnerability-correlation-sources.md` - the admin page's Data Sync Status is complete: next to Sync KEV / Sync MSRC there are now Refresh NVD Cache and Sync EPSS, each with its own last-sync line (`NvdCacheLastRefreshUtc`, `EpssLastSyncUtc` on AdminConfiguration; shared manifest regenerated). The NVD and EPSS walks moved out of the timer into `VulnerabilityCacheRefreshService` (singleton, one in-flight slot per walk) so the button and the nightly run are literally the same code; the NVD walk is minutes long and rate-limited, so `POST vulnerability/sync-nvd` answers 202 and the page polls `sync-status` (`nvdRefreshRunning`, `nvdLastRun`) while it runs, `sync-epss` runs synchronously. 409 while a walk is in flight; the timer skips a slot a manual run holds. Operator-only surface - no customer docs. + ## 2026-08-29 (2) * **New**: `backend/vulnerability-correlation-sources.md` - FIRST EPSS joins NVD / CISA KEV / MSRC as the fourth vulnerability source: the probability + percentile + date live on the cached CVE (one batch of <= 100 ids per CPE fill, every `cve_*` row re-scored by the nightly sync, only changed rows rewritten and with their original `CachedAt`), the CVSS vector is persisted instead of dropped, and each CVE carries an act / attend / track priority (KEV; EPSS >= 0.1 or CVSS >= 9.0; rest) next to the unchanged severity bands - two visible signals, no composite score. The comparison with an open-source CVE MCP server surfaced the real 429 defect: Polly already retried, but a refused NVD answer came back as an empty response and was cached as "0 CVEs" for seven days. `NvdFetchResult` now separates "answered" from "refused" (nothing persists a refusal - CPE cache, negative auto-resolve cache, admin auto-resolve), and a 429 opens a process-wide cooldown (`Retry-After` capped at 120 s, else 30 s) so the run fails fast instead of paying a retry chain per software item; hard-expired rows keep serving meanwhile. NVD -> cache projection and the row builder are now single functions shared by the live fill and the sync. CveIndex / fleet summary carry `epssScore` + `priority`; portal shows EPSS pill, priority pill, vector; MCP tool descriptions and the CVE-audit prompt name the fields. ExploitDB recorded as a deliberate not-now. Backend before web; trust pages (terms attribution, privacy, data-flows), settings and portal-guide docs updated in the same change. diff --git a/src/Backend/AutopilotMonitor.Functions/DataAccess/TableStorage/TableConfigRepository.cs b/src/Backend/AutopilotMonitor.Functions/DataAccess/TableStorage/TableConfigRepository.cs index fbc29785..5a6f65f0 100644 --- a/src/Backend/AutopilotMonitor.Functions/DataAccess/TableStorage/TableConfigRepository.cs +++ b/src/Backend/AutopilotMonitor.Functions/DataAccess/TableStorage/TableConfigRepository.cs @@ -900,6 +900,8 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage { "SessionDeletionKillSwitch", config.SessionDeletionKillSwitch }, { "VulnerabilityDataLastSyncUtc", config.VulnerabilityDataLastSyncUtc }, { "MsrcLastSyncUtc", config.MsrcLastSyncUtc }, + { "NvdCacheLastRefreshUtc", config.NvdCacheLastRefreshUtc }, + { "EpssLastSyncUtc", config.EpssLastSyncUtc }, { "MaxDiagnosticsDownloadSizeMB", config.MaxDiagnosticsDownloadSizeMB }, { "DiagnosticsDownloadTimeoutSeconds", config.DiagnosticsDownloadTimeoutSeconds }, { "OpsEventRetentionDays", config.OpsEventRetentionDays }, @@ -969,6 +971,8 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage SessionDeletionKillSwitch = entity.GetBoolean("SessionDeletionKillSwitch") ?? false, VulnerabilityDataLastSyncUtc = entity.GetString("VulnerabilityDataLastSyncUtc"), MsrcLastSyncUtc = entity.GetString("MsrcLastSyncUtc"), + NvdCacheLastRefreshUtc = entity.GetString("NvdCacheLastRefreshUtc"), + EpssLastSyncUtc = entity.GetString("EpssLastSyncUtc"), MaxDiagnosticsDownloadSizeMB = entity.GetInt32("MaxDiagnosticsDownloadSizeMB") ?? 500, DiagnosticsDownloadTimeoutSeconds = entity.GetInt32("DiagnosticsDownloadTimeoutSeconds") ?? 120, OpsEventRetentionDays = entity.GetInt32("OpsEventRetentionDays") ?? 90, diff --git a/src/Backend/AutopilotMonitor.Functions/Functions/Vulnerability/GetVulnerabilitySyncStatusFunction.cs b/src/Backend/AutopilotMonitor.Functions/Functions/Vulnerability/GetVulnerabilitySyncStatusFunction.cs index 0929652d..91ad0fa6 100644 --- a/src/Backend/AutopilotMonitor.Functions/Functions/Vulnerability/GetVulnerabilitySyncStatusFunction.cs +++ b/src/Backend/AutopilotMonitor.Functions/Functions/Vulnerability/GetVulnerabilitySyncStatusFunction.cs @@ -19,17 +19,20 @@ namespace AutopilotMonitor.Functions.Functions.Vulnerability { private readonly KevDataService _kevService; private readonly MsrcCorrelationService _msrcService; + private readonly VulnerabilityCacheRefreshService _refresh; private readonly AdminConfigurationService _adminConfigService; private readonly ILogger _logger; public GetVulnerabilitySyncStatusFunction( KevDataService kevService, MsrcCorrelationService msrcService, + VulnerabilityCacheRefreshService refresh, AdminConfigurationService adminConfigService, ILogger logger) { _kevService = kevService; _msrcService = msrcService; + _refresh = refresh; _adminConfigService = adminConfigService; _logger = logger; } @@ -46,6 +49,9 @@ namespace AutopilotMonitor.Functions.Functions.Vulnerability await _msrcService.EnsureHydratedAsync(); var config = await _adminConfigService.GetConfigurationAsync(); + var nvdCacheEntries = await _refresh.CountCveCacheRowsAsync(); + var nvd = _refresh.LastNvdRefresh; + var epss = _refresh.LastEpssRefresh; var response = req.CreateResponse(HttpStatusCode.OK); await response.WriteAsJsonAsync(new @@ -54,7 +60,26 @@ namespace AutopilotMonitor.Functions.Functions.Vulnerability kevLastSyncUtc = config?.VulnerabilityDataLastSyncUtc, msrcIndexedCves = _msrcService.IndexSize, msrcCoveredDocuments = _msrcService.CoveredDocuments.Count, - msrcLastSyncUtc = config?.MsrcLastSyncUtc + msrcLastSyncUtc = config?.MsrcLastSyncUtc, + nvdCacheEntries, + nvdCacheLastRefreshUtc = config?.NvdCacheLastRefreshUtc, + nvdRefreshRunning = _refresh.NvdRefreshRunning, + // Last run on THIS instance (in-memory; null after a cold start) — the + // persisted timestamp above is the durable "last completed" signal. + nvdLastRun = nvd == null ? null : new + { + stale = nvd.Stale, refreshed = nvd.Refreshed, failed = nvd.Failed, + skippedThrottled = nvd.SkippedThrottled, error = nvd.Error, + startedUtc = nvd.StartedUtc.ToString("o"), finishedUtc = nvd.FinishedUtc?.ToString("o"), + }, + epssLastSyncUtc = config?.EpssLastSyncUtc, + epssRefreshRunning = _refresh.EpssRefreshRunning, + epssLastRun = epss == null ? null : new + { + rows = epss.Rows, cves = epss.Cves, scored = epss.Scored, + rowsRewritten = epss.RowsRewritten, error = epss.Error, + startedUtc = epss.StartedUtc.ToString("o"), finishedUtc = epss.FinishedUtc?.ToString("o"), + }, }); return response; } diff --git a/src/Backend/AutopilotMonitor.Functions/Functions/Vulnerability/TriggerEpssSyncFunction.cs b/src/Backend/AutopilotMonitor.Functions/Functions/Vulnerability/TriggerEpssSyncFunction.cs new file mode 100644 index 00000000..972d7c54 --- /dev/null +++ b/src/Backend/AutopilotMonitor.Functions/Functions/Vulnerability/TriggerEpssSyncFunction.cs @@ -0,0 +1,86 @@ +using System; +using System.Collections.Generic; +using System.Net; +using System.Threading.Tasks; +using AutopilotMonitor.Functions.Services.Vulnerability; +using Microsoft.ApplicationInsights; +using Microsoft.Azure.Functions.Worker; +using Microsoft.Azure.Functions.Worker.Http; +using Microsoft.Extensions.Logging; + +namespace AutopilotMonitor.Functions.Functions.Vulnerability +{ + /// + /// Manual FIRST EPSS re-score of every cached CVE (the EPSS phase of the daily timer). A + /// handful of batched requests — runs synchronously and answers with the counts. 409 when a + /// re-score is already in flight. Global Admin only. + /// + public class TriggerEpssSyncFunction + { + private readonly VulnerabilityCacheRefreshService _refresh; + private readonly TelemetryClient? _telemetryClient; + private readonly ILogger _logger; + + public TriggerEpssSyncFunction( + VulnerabilityCacheRefreshService refresh, + TelemetryClient? telemetryClient, + ILogger logger) + { + _refresh = refresh; + _telemetryClient = telemetryClient; + _logger = logger; + } + + [Function("TriggerEpssSync")] + public async Task Run( + [HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = "vulnerability/sync-epss")] HttpRequestData req) + { + try + { + var result = await _refresh.RefreshEpssScoresAsync("manual"); + if (result == null) + { + var busy = req.CreateResponse(HttpStatusCode.Conflict); + await busy.WriteAsJsonAsync(new { success = false, message = "An EPSS sync is already running" }); + return busy; + } + + var durationMs = (int)((result.FinishedUtc ?? DateTime.UtcNow) - result.StartedUtc).TotalMilliseconds; + _telemetryClient?.TrackEvent("ManualEpssSyncCompleted", new Dictionary + { + { "rows", result.Rows.ToString() }, + { "cves", result.Cves.ToString() }, + { "scored", result.Scored.ToString() }, + { "rowsRewritten", result.RowsRewritten.ToString() }, + { "error", result.Error ?? "" }, + { "durationMs", durationMs.ToString() } + }); + + var ok = result.Error == null; + var response = req.CreateResponse(HttpStatusCode.OK); + await response.WriteAsJsonAsync(new + { + success = ok, + message = ok + ? $"EPSS scores refreshed: {result.Scored:N0} of {result.Cves:N0} cached CVEs scored, {result.RowsRewritten:N0} rows updated" + : $"EPSS sync failed: {result.Error}", + epssRows = result.Rows, + epssCves = result.Cves, + epssScored = result.Scored, + epssRowsRewritten = result.RowsRewritten, + epssError = result.Error, + durationMs, + syncedAt = DateTime.UtcNow.ToString("o") + }); + return response; + } + catch (Exception ex) + { + _logger.LogError(ex, "Manual EPSS sync failed"); + var errorResponse = req.CreateResponse(HttpStatusCode.InternalServerError); + await errorResponse.WriteAsJsonAsync(new { success = false, message = "EPSS sync failed: " + ex.Message }); + return errorResponse; + } + } + } +} diff --git a/src/Backend/AutopilotMonitor.Functions/Functions/Vulnerability/TriggerNvdCacheRefreshFunction.cs b/src/Backend/AutopilotMonitor.Functions/Functions/Vulnerability/TriggerNvdCacheRefreshFunction.cs new file mode 100644 index 00000000..ca2c3d4b --- /dev/null +++ b/src/Backend/AutopilotMonitor.Functions/Functions/Vulnerability/TriggerNvdCacheRefreshFunction.cs @@ -0,0 +1,51 @@ +using System; +using System.Net; +using System.Threading.Tasks; +using AutopilotMonitor.Functions.Services.Vulnerability; +using Microsoft.Azure.Functions.Worker; +using Microsoft.Azure.Functions.Worker.Http; +using Microsoft.Extensions.Logging; + +namespace AutopilotMonitor.Functions.Functions.Vulnerability +{ + /// + /// Manual NVD cache refresh (the stale-row phase of the daily timer). The walk is sequential + /// and NVD-rate-limited, so it can run for minutes: this endpoint only STARTS it and answers + /// 202; progress and the outcome are visible on /api/vulnerability/sync-status + /// (nvdRefreshRunning, nvdLastRun). 409 when a walk is already in flight. + /// Global Admin only. + /// + public class TriggerNvdCacheRefreshFunction + { + private readonly VulnerabilityCacheRefreshService _refresh; + private readonly ILogger _logger; + + public TriggerNvdCacheRefreshFunction(VulnerabilityCacheRefreshService refresh, ILogger logger) + { + _refresh = refresh; + _logger = logger; + } + + [Function("TriggerNvdCacheRefresh")] + public async Task Run( + [HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = "vulnerability/sync-nvd")] HttpRequestData req) + { + if (!_refresh.TryStartNvdRefreshInBackground("manual")) + { + var busy = req.CreateResponse(HttpStatusCode.Conflict); + await busy.WriteAsJsonAsync(new { success = false, message = "An NVD cache refresh is already running" }); + return busy; + } + + _logger.LogInformation("Manual NVD cache refresh started"); + var response = req.CreateResponse(HttpStatusCode.Accepted); + await response.WriteAsJsonAsync(new + { + success = true, + message = "NVD cache refresh started — stale entries are re-queried sequentially; check the sync status for progress", + startedAt = DateTime.UtcNow.ToString("o") + }); + return response; + } + } +} diff --git a/src/Backend/AutopilotMonitor.Functions/Functions/Vulnerability/VulnerabilityDataSyncFunction.cs b/src/Backend/AutopilotMonitor.Functions/Functions/Vulnerability/VulnerabilityDataSyncFunction.cs index 8e5fdd96..5c7afa3b 100644 --- a/src/Backend/AutopilotMonitor.Functions/Functions/Vulnerability/VulnerabilityDataSyncFunction.cs +++ b/src/Backend/AutopilotMonitor.Functions/Functions/Vulnerability/VulnerabilityDataSyncFunction.cs @@ -1,23 +1,23 @@ using System; using System.Collections.Generic; -using System.Linq; -using System.Threading; using System.Threading.Tasks; -using AutopilotMonitor.Functions.DataAccess.TableStorage; using AutopilotMonitor.Functions.Services; using AutopilotMonitor.Functions.Services.Vulnerability; -using AutopilotMonitor.Shared.DataAccess; using Microsoft.ApplicationInsights; using Microsoft.Azure.Functions.Worker; using Microsoft.Extensions.Logging; -using static AutopilotMonitor.Functions.Services.Vulnerability.VulnerabilityCorrelationService; namespace AutopilotMonitor.Functions.Functions.Vulnerability { /// /// Timer-triggered function that refreshes vulnerability data caches daily. /// - CISA KEV catalog (full in-memory refresh) + /// - MSRC CVRF index (last 6 Patch Tuesdays) /// - Stale NVD CVE cache entries in VulnerabilityCache table (older than 7 days) + /// - FIRST EPSS re-score of every cached CVE + /// + /// The NVD and EPSS walks live in , shared with + /// the admin's manual buttons, so both paths behave identically. /// /// All operations are logged to ILogger → visible in Application Insights under /// "traces" with customDimensions.Category = "VulnerabilityDataSyncFunction". @@ -26,29 +26,23 @@ namespace AutopilotMonitor.Functions.Functions.Vulnerability public class VulnerabilityDataSyncFunction { private readonly KevDataService _kevService; - private readonly NvdApiClient _nvdApi; - private readonly EpssApiClient _epssApi; private readonly MsrcCorrelationService _msrcCorrelation; - private readonly IVulnerabilityRepository _vulnRepo; + private readonly VulnerabilityCacheRefreshService _refresh; private readonly AdminConfigurationService _adminConfigService; private readonly TelemetryClient? _telemetryClient; private readonly ILogger _logger; public VulnerabilityDataSyncFunction( KevDataService kevService, - NvdApiClient nvdApi, - EpssApiClient epssApi, MsrcCorrelationService msrcCorrelation, - IVulnerabilityRepository vulnRepo, + VulnerabilityCacheRefreshService refresh, AdminConfigurationService adminConfigService, TelemetryClient? telemetryClient, ILogger logger) { _kevService = kevService; - _nvdApi = nvdApi; - _epssApi = epssApi; _msrcCorrelation = msrcCorrelation; - _vulnRepo = vulnRepo; + _refresh = refresh; _adminConfigService = adminConfigService; _telemetryClient = telemetryClient; _logger = logger; @@ -66,9 +60,6 @@ namespace AutopilotMonitor.Functions.Functions.Vulnerability int kevCount = 0; string? kevError = null; - int cveRefreshed = 0; - int cveFailed = 0; - int cveStale = 0; int msrcDocs = 0; int msrcCves = 0; string? msrcError = null; @@ -110,19 +101,20 @@ namespace AutopilotMonitor.Functions.Functions.Vulnerability _logger.LogWarning(ex, "[VulnSync] MSRC index refresh failed"); } - // 3. Refresh stale CVE cache entries - var cveStats = await RefreshStaleCveCacheAsync(); - cveStale = cveStats.Stale; - cveRefreshed = cveStats.Refreshed; - cveFailed = cveStats.Failed; + // 3. Refresh stale CVE cache entries (null = a manual walk is in flight; skip, it does the same work). + var cveStats = await _refresh.RefreshStaleCveCacheAsync("timer"); + if (cveStats == null) + _logger.LogInformation("[VulnSync] CVE cache refresh skipped: a manual refresh is already running"); // 3b. Re-score EVERY cached CVE row with today's EPSS. Independent of the 7-day NVD // TTL: EPSS is republished daily and the calls are cheap (one batch per 100 ids). - var epssStats = await RefreshEpssScoresAsync(); + var epssStats = await _refresh.RefreshEpssScoresAsync("timer"); + if (epssStats == null) + _logger.LogInformation("[VulnSync] EPSS refresh skipped: a manual sync is already running"); // 4. Update per-source sync timestamps. Only the sources that succeeded get // bumped — a failed phase keeps its prior timestamp so the UI can surface - // "Last KEV/MSRC Sync" accurately. + // "Last KEV/MSRC Sync" accurately. (NVD/EPSS stamps are written by the service.) try { var config = await _adminConfigService.GetConfigurationAsync(); @@ -160,229 +152,18 @@ namespace AutopilotMonitor.Functions.Functions.Vulnerability { "msrcDocs", msrcDocs.ToString() }, { "msrcCves", msrcCves.ToString() }, { "msrcError", msrcError ?? "" }, - { "cveStale", cveStale.ToString() }, - { "cveRefreshed", cveRefreshed.ToString() }, - { "cveFailed", cveFailed.ToString() }, - { "cveSkippedThrottled", cveStats.SkippedThrottled.ToString() }, - { "epssRows", epssStats.Rows.ToString() }, - { "epssCves", epssStats.Cves.ToString() }, - { "epssScored", epssStats.Scored.ToString() }, - { "epssRowsRewritten", epssStats.RowsRewritten.ToString() }, - { "epssError", epssStats.Error ?? "" }, + { "cveStale", (cveStats?.Stale ?? 0).ToString() }, + { "cveRefreshed", (cveStats?.Refreshed ?? 0).ToString() }, + { "cveFailed", (cveStats?.Failed ?? 0).ToString() }, + { "cveSkippedThrottled", (cveStats?.SkippedThrottled ?? 0).ToString() }, + { "cveSkippedBusy", (cveStats == null).ToString() }, + { "epssRows", (epssStats?.Rows ?? 0).ToString() }, + { "epssCves", (epssStats?.Cves ?? 0).ToString() }, + { "epssScored", (epssStats?.Scored ?? 0).ToString() }, + { "epssRowsRewritten", (epssStats?.RowsRewritten ?? 0).ToString() }, + { "epssError", epssStats?.Error ?? "" }, { "durationMs", durationMs.ToString("F0", System.Globalization.CultureInfo.InvariantCulture) } }); } - - private struct CveCacheStats - { - public int Stale; - public int Refreshed; - public int Failed; - /// Stale entries left untouched because NVD opened its throttle cooldown mid-run. - public int SkippedThrottled; - } - - private struct EpssRefreshStats - { - public int Rows; - public int Cves; - public int Scored; - public int RowsRewritten; - public string? Error; - } - - /// - /// Walks every cve_* cache row, batches all CVE ids through the EPSS API and rewrites - /// only the rows whose scores actually changed — with the row's ORIGINAL CachedAt, so - /// an EPSS rewrite never counts as a fresh NVD fetch for the 7-day TTL. - /// - private async Task RefreshEpssScoresAsync() - { - var stats = new EpssRefreshStats(); - try - { - var index = await _vulnRepo.GetStaleCveCacheEntriesAsync(); - var rows = new List<(string CacheKey, string CpeUri, DateTime CachedAt, List Cves)>(); - - foreach (var entry in index) - { - var pk = entry.ContainsKey("PartitionKey") ? entry["PartitionKey"]?.ToString() : null; - if (pk == null || !pk.StartsWith("cve_")) continue; - - var full = await _vulnRepo.GetCveCacheEntryAsync(pk, "cache_result"); - if (full == null) continue; - - var cachedAtStr = full.ContainsKey("CachedAt") ? full["CachedAt"]?.ToString() : null; - if (!DateTime.TryParse(cachedAtStr, null, System.Globalization.DateTimeStyles.RoundtripKind, out var cachedAt)) - continue; - - var json = TableStorageChunking.ReassembleProperty(full, "CveDataJson", _logger, $"{pk}/cache_result") ?? "[]"; - var cves = System.Text.Json.JsonSerializer.Deserialize>(json, JsonOptions) ?? new List(); - if (cves.Count == 0) continue; - - var cpeUri = full.ContainsKey("CpeUri") ? full["CpeUri"]?.ToString() ?? pk.Substring(4) : pk.Substring(4); - rows.Add((pk, cpeUri, cachedAt, cves)); - } - - stats.Rows = rows.Count; - var allIds = rows.SelectMany(r => r.Cves.Select(c => c.CveId)).Distinct(StringComparer.OrdinalIgnoreCase).ToList(); - stats.Cves = allIds.Count; - if (allIds.Count == 0) - { - _logger.LogInformation("[VulnSync] EPSS refresh: no cached CVEs to score"); - return stats; - } - - var scores = await _epssApi.GetScoresAsync(allIds, CancellationToken.None); - stats.Scored = scores.Count; - - foreach (var (cacheKey, cpeUri, cachedAt, cves) in rows) - { - var changed = VulnerabilityCorrelationService.ApplyEpssScores(cves, scores); - if (changed == 0) continue; - try - { - await _vulnRepo.UpsertCveCacheEntryAsync(cacheKey, "cache_result", - VulnerabilityCorrelationService.BuildCveCacheRow(cpeUri, cves, cachedAt)); - stats.RowsRewritten++; - } - catch (Exception ex) - { - _logger.LogWarning(ex, "[VulnSync] EPSS rewrite failed for {CpeUri}", cpeUri); - } - } - - _logger.LogInformation( - "[VulnSync] EPSS refresh: {Scored}/{Cves} CVEs scored across {Rows} rows, {Rewritten} rows rewritten", - stats.Scored, stats.Cves, stats.Rows, stats.RowsRewritten); - } - catch (Exception ex) - { - stats.Error = ex.Message; - _logger.LogWarning(ex, "[VulnSync] EPSS refresh failed"); - } - return stats; - } - - /// - /// Scans the VulnerabilityCache table for CVE cache entries older than 7 days - /// and refreshes them by re-querying the NVD API. Processes sequentially with - /// rate limiting to avoid NVD API throttling. - /// - private async Task RefreshStaleCveCacheAsync() - { - var stats = new CveCacheStats(); - try - { - var staleCutoff = DateTime.UtcNow.AddDays(-7).ToString("o"); - - var cacheEntries = await _vulnRepo.GetStaleCveCacheEntriesAsync(); - - var staleEntries = new List<(string CacheKey, string CpeUri, string CachedAt, int TotalCves)>(); - - foreach (var entity in cacheEntries) - { - var pk = entity.ContainsKey("PartitionKey") ? entity["PartitionKey"]?.ToString() : null; - if (pk == null || !pk.StartsWith("cve_")) continue; - - var cachedAt = entity.ContainsKey("CachedAt") ? entity["CachedAt"]?.ToString() ?? "" : ""; - if (string.Compare(cachedAt, staleCutoff, StringComparison.OrdinalIgnoreCase) < 0) - { - staleEntries.Add(( - pk, - entity.ContainsKey("CpeUri") ? entity["CpeUri"]?.ToString() ?? pk.Substring(4) : pk.Substring(4), - cachedAt, - entity.ContainsKey("TotalCves") && entity["TotalCves"] is int tc ? tc : 0 - )); - } - } - - stats.Stale = staleEntries.Count; - - if (staleEntries.Count == 0) - { - _logger.LogInformation("[VulnSync] CVE cache: no stale entries found"); - return stats; - } - - _logger.LogInformation("[VulnSync] CVE cache: {StaleCount} stale entries to refresh", staleEntries.Count); - - int refreshed = 0; - int failed = 0; - int skippedThrottled = 0; - - for (int i = 0; i < staleEntries.Count; i++) - { - var (cacheKey, cpeUri, cachedAt, totalCves) = staleEntries[i]; - try - { - _logger.LogInformation("[VulnSync] Refreshing CVE cache for {CpeUri} (cached: {CachedAt}, had {TotalCves} CVEs)", - cpeUri, cachedAt, totalCves); - - // Query NVD API (rate-limited internally by NvdApiClient) - var fetch = await _nvdApi.SearchCvesByCpeAsync(cpeUri, CancellationToken.None); - if (fetch.Throttled) - { - // NVD is refusing us; the stale rows keep serving (soft-stale reads - // still return data) and get another attempt tomorrow. Bail out of - // the loop instead of paying the cooldown skip per remaining entry. - skippedThrottled = staleEntries.Count - i; - _logger.LogWarning("[VulnSync] NVD throttled — leaving {Remaining} stale entries for the next run", skippedThrottled); - break; - } - if (!fetch.Succeeded) - { - failed++; - continue; // row untouched: never overwrite real data with a refusal - } - - // Project NVD response → CachedCve list (same shape the read path expects), - // under the same cache budget the live FetchAndCache path uses. - var cves = VulnerabilityCorrelationService.ProjectNvdResponse(fetch.Data); - var budget = CveCacheBudget.Apply(cves, DateTime.UtcNow); - cves = budget.Cves; - - // EPSS is re-applied for all rows in step 3b; scoring here keeps the row - // complete in case that step fails. - try - { - var scores = await _epssApi.GetScoresAsync(cves.Select(c => c.CveId), CancellationToken.None); - VulnerabilityCorrelationService.ApplyEpssScores(cves, scores); - } - catch (Exception ex) - { - _logger.LogWarning(ex, "[VulnSync] EPSS enrichment failed for {CpeUri}", cpeUri); - } - - var row = VulnerabilityCorrelationService.BuildCveCacheRow(cpeUri, cves, DateTime.UtcNow); - await _vulnRepo.UpsertCveCacheEntryAsync(cacheKey, "cache_result", row); - refreshed++; - - _logger.LogInformation( - "[VulnSync] Refreshed {CpeUri}: {OldCount} → {NewCount} CVEs (NVD={Total}, afterDate={AfterDate}, truncRanges={TruncCves})", - cpeUri, totalCves, cves.Count, - budget.TotalIn, budget.KeptAfterDateFilter, budget.CvesWithTruncatedRanges); - } - catch (Exception ex) - { - failed++; - _logger.LogWarning(ex, "[VulnSync] Failed to refresh CVE cache for {CpeUri}", cpeUri); - } - } - - _logger.LogInformation("[VulnSync] CVE cache refresh complete: {Refreshed} refreshed, {Failed} failed, {Skipped} skipped (throttled) out of {Total} stale entries", - refreshed, failed, skippedThrottled, staleEntries.Count); - - stats.Refreshed = refreshed; - stats.Failed = failed; - stats.SkippedThrottled = skippedThrottled; - } - catch (Exception ex) - { - _logger.LogWarning(ex, "[VulnSync] CVE cache refresh scan failed"); - } - - return stats; - } } } diff --git a/src/Backend/AutopilotMonitor.Functions/Program.cs b/src/Backend/AutopilotMonitor.Functions/Program.cs index 3106e2d7..4e555e9f 100644 --- a/src/Backend/AutopilotMonitor.Functions/Program.cs +++ b/src/Backend/AutopilotMonitor.Functions/Program.cs @@ -410,6 +410,7 @@ builder.Services.AddHttpClient() .AddPolicyHandler((sp, _) => sp.GetRequiredService().ExternalDataApi); builder.Services.AddSingleton(); +builder.Services.AddSingleton(); builder.Services.AddSingleton(); // Hydrate MSRC + KEV in-memory caches from blob snapshots at app startup (fire-and-forget; // keeps cold-start fast for re-deploys, see VulnerabilityCacheWarmer for the contract). diff --git a/src/Backend/AutopilotMonitor.Functions/Security/EndpointAccessPolicyCatalog.cs b/src/Backend/AutopilotMonitor.Functions/Security/EndpointAccessPolicyCatalog.cs index a4b93a38..31880df9 100644 --- a/src/Backend/AutopilotMonitor.Functions/Security/EndpointAccessPolicyCatalog.cs +++ b/src/Backend/AutopilotMonitor.Functions/Security/EndpointAccessPolicyCatalog.cs @@ -547,6 +547,8 @@ public static class EndpointAccessPolicyCatalog new("GET", "vulnerability/software-inventory", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam), new("POST", "vulnerability/sync", EndpointPolicy.GlobalAdminOnly), new("POST", "vulnerability/sync-msrc", EndpointPolicy.GlobalAdminOnly), + new("POST", "vulnerability/sync-nvd", EndpointPolicy.GlobalAdminOnly), + new("POST", "vulnerability/sync-epss", EndpointPolicy.GlobalAdminOnly), new("GET", "vulnerability/sync-status", EndpointPolicy.GlobalReadOrAdmin), new("POST", "vulnerability/cpe-mapping", EndpointPolicy.GlobalAdminOnly), new("POST", "vulnerability/cpe-mapping/auto-resolve", EndpointPolicy.GlobalAdminOnly), diff --git a/src/Backend/AutopilotMonitor.Functions/Services/Vulnerability/VulnerabilityCacheRefreshService.cs b/src/Backend/AutopilotMonitor.Functions/Services/Vulnerability/VulnerabilityCacheRefreshService.cs new file mode 100644 index 00000000..51e63d2b --- /dev/null +++ b/src/Backend/AutopilotMonitor.Functions/Services/Vulnerability/VulnerabilityCacheRefreshService.cs @@ -0,0 +1,328 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using AutopilotMonitor.Functions.DataAccess.TableStorage; +using AutopilotMonitor.Shared.DataAccess; +using Microsoft.Extensions.Logging; +using static AutopilotMonitor.Functions.Services.Vulnerability.VulnerabilityCorrelationService; + +namespace AutopilotMonitor.Functions.Services.Vulnerability +{ + /// Outcome of one NVD stale-cache refresh walk. + public sealed class NvdCacheRefreshResult + { + public int Stale { get; set; } + public int Refreshed { get; set; } + public int Failed { get; set; } + /// Stale entries left untouched because NVD opened its throttle cooldown mid-run. + public int SkippedThrottled { get; set; } + public string? Error { get; set; } + public DateTime StartedUtc { get; set; } + public DateTime? FinishedUtc { get; set; } + /// True when every stale row got a real attempt (no throttle cut-off, no scan error). + public bool Completed => Error == null && SkippedThrottled == 0; + } + + /// Outcome of one EPSS re-score over all cached CVE rows. + public sealed class EpssRefreshResult + { + public int Rows { get; set; } + public int Cves { get; set; } + public int Scored { get; set; } + public int RowsRewritten { get; set; } + public string? Error { get; set; } + public DateTime StartedUtc { get; set; } + public DateTime? FinishedUtc { get; set; } + } + + /// + /// The two VulnerabilityCache maintenance walks — NVD stale-row refresh and FIRST EPSS re-score — + /// shared by the daily timer (VulnerabilityDataSyncFunction) and the admin's manual + /// buttons (/api/vulnerability/sync-nvd, /api/vulnerability/sync-epss), so both + /// paths write identical rows and stamp the same AdminConfiguration timestamps. + /// + /// Singleton with one in-flight slot per walk: a second caller gets null instead of a + /// concurrent duplicate (two NVD walks would just fight over the same rate limit). The NVD walk + /// is sequential and rate-limited (2–8 s per CPE) and therefore may run for minutes — the manual + /// endpoint starts it in the background () and the + /// status endpoint reports / . + /// The EPSS walk is a handful of batched requests and finishes in seconds. + /// + /// + public class VulnerabilityCacheRefreshService + { + private readonly IVulnerabilityRepository _vulnRepo; + private readonly NvdApiClient _nvdApi; + private readonly EpssApiClient _epssApi; + private readonly AdminConfigurationService _adminConfigService; + private readonly ILogger _logger; + + private readonly SemaphoreSlim _nvdSlot = new(1, 1); + private readonly SemaphoreSlim _epssSlot = new(1, 1); + + public VulnerabilityCacheRefreshService( + IVulnerabilityRepository vulnRepo, + NvdApiClient nvdApi, + EpssApiClient epssApi, + AdminConfigurationService adminConfigService, + ILogger logger) + { + _vulnRepo = vulnRepo; + _nvdApi = nvdApi; + _epssApi = epssApi; + _adminConfigService = adminConfigService; + _logger = logger; + } + + public bool NvdRefreshRunning => _nvdSlot.CurrentCount == 0; + public bool EpssRefreshRunning => _epssSlot.CurrentCount == 0; + public NvdCacheRefreshResult? LastNvdRefresh { get; private set; } + public EpssRefreshResult? LastEpssRefresh { get; private set; } + + /// Number of per-CPE CVE cache rows (one point scan over the index columns). + public async Task CountCveCacheRowsAsync() + { + try + { + var entries = await _vulnRepo.GetStaleCveCacheEntriesAsync(); + return entries.Count(e => e.ContainsKey("PartitionKey") && (e["PartitionKey"]?.ToString() ?? "").StartsWith("cve_")); + } + catch (Exception ex) + { + _logger.LogDebug(ex, "CVE cache row count failed"); + return 0; + } + } + + // ----------------------------------------------------------------------- + // NVD stale-row refresh + // ----------------------------------------------------------------------- + + /// + /// Starts on the thread pool. Returns false when a + /// walk is already in flight (nothing started). + /// + public bool TryStartNvdRefreshInBackground(string trigger) + { + if (NvdRefreshRunning) return false; + _ = Task.Run(async () => + { + try { await RefreshStaleCveCacheAsync(trigger); } + catch (Exception ex) { _logger.LogWarning(ex, "[VulnSync] Background NVD cache refresh crashed"); } + }); + return true; + } + + /// + /// Re-queries NVD for every VulnerabilityCache row older than 7 days, sequentially (the + /// client rate-limits). Breaks out on the first throttled answer — the stale rows keep + /// serving and get another attempt next time — and never overwrites a row with a refusal. + /// Stamps AdminConfiguration.NvdCacheLastRefreshUtc only for a completed walk. + /// Returns null when a walk is already running. + /// + public async Task RefreshStaleCveCacheAsync(string trigger) + { + if (!await _nvdSlot.WaitAsync(0)) return null; + var result = new NvdCacheRefreshResult { StartedUtc = DateTime.UtcNow }; + LastNvdRefresh = result; + try + { + var staleCutoff = DateTime.UtcNow.AddDays(-7).ToString("o"); + var cacheEntries = await _vulnRepo.GetStaleCveCacheEntriesAsync(); + + var staleEntries = new List<(string CacheKey, string CpeUri, string CachedAt, int TotalCves)>(); + foreach (var entity in cacheEntries) + { + var pk = entity.ContainsKey("PartitionKey") ? entity["PartitionKey"]?.ToString() : null; + if (pk == null || !pk.StartsWith("cve_")) continue; + + var cachedAt = entity.ContainsKey("CachedAt") ? entity["CachedAt"]?.ToString() ?? "" : ""; + if (string.Compare(cachedAt, staleCutoff, StringComparison.OrdinalIgnoreCase) < 0) + { + staleEntries.Add(( + pk, + entity.ContainsKey("CpeUri") ? entity["CpeUri"]?.ToString() ?? pk.Substring(4) : pk.Substring(4), + cachedAt, + entity.ContainsKey("TotalCves") && entity["TotalCves"] is int tc ? tc : 0)); + } + } + + result.Stale = staleEntries.Count; + _logger.LogInformation("[VulnSync] CVE cache ({Trigger}): {StaleCount} stale entries to refresh", trigger, staleEntries.Count); + + for (int i = 0; i < staleEntries.Count; i++) + { + var (cacheKey, cpeUri, cachedAt, totalCves) = staleEntries[i]; + try + { + _logger.LogInformation("[VulnSync] Refreshing CVE cache for {CpeUri} (cached: {CachedAt}, had {TotalCves} CVEs)", + cpeUri, cachedAt, totalCves); + + var fetch = await _nvdApi.SearchCvesByCpeAsync(cpeUri, CancellationToken.None); + if (fetch.Throttled) + { + result.SkippedThrottled = staleEntries.Count - i; + _logger.LogWarning("[VulnSync] NVD throttled — leaving {Remaining} stale entries for the next run", result.SkippedThrottled); + break; + } + if (!fetch.Succeeded) + { + result.Failed++; + continue; // row untouched: never overwrite real data with a refusal + } + + var cves = ProjectNvdResponse(fetch.Data); + var budget = CveCacheBudget.Apply(cves, DateTime.UtcNow); + cves = budget.Cves; + + // EPSS is re-applied for all rows by the EPSS walk; scoring here keeps the + // row complete in case that walk fails. + try + { + var scores = await _epssApi.GetScoresAsync(cves.Select(c => c.CveId), CancellationToken.None); + ApplyEpssScores(cves, scores); + } + catch (Exception ex) + { + _logger.LogWarning(ex, "[VulnSync] EPSS enrichment failed for {CpeUri}", cpeUri); + } + + await _vulnRepo.UpsertCveCacheEntryAsync(cacheKey, "cache_result", BuildCveCacheRow(cpeUri, cves, DateTime.UtcNow)); + result.Refreshed++; + + _logger.LogInformation( + "[VulnSync] Refreshed {CpeUri}: {OldCount} → {NewCount} CVEs (NVD={Total}, afterDate={AfterDate}, truncRanges={TruncCves})", + cpeUri, totalCves, cves.Count, budget.TotalIn, budget.KeptAfterDateFilter, budget.CvesWithTruncatedRanges); + } + catch (Exception ex) + { + result.Failed++; + _logger.LogWarning(ex, "[VulnSync] Failed to refresh CVE cache for {CpeUri}", cpeUri); + } + } + + _logger.LogInformation( + "[VulnSync] CVE cache refresh complete ({Trigger}): {Refreshed} refreshed, {Failed} failed, {Skipped} skipped (throttled) out of {Total} stale entries", + trigger, result.Refreshed, result.Failed, result.SkippedThrottled, staleEntries.Count); + } + catch (Exception ex) + { + result.Error = ex.Message; + _logger.LogWarning(ex, "[VulnSync] CVE cache refresh scan failed"); + } + finally + { + result.FinishedUtc = DateTime.UtcNow; + _nvdSlot.Release(); + } + + if (result.Completed) + await StampAsync(c => c.NvdCacheLastRefreshUtc = DateTime.UtcNow.ToString("o"), "NvdCacheLastRefreshUtc"); + + return result; + } + + // ----------------------------------------------------------------------- + // EPSS re-score + // ----------------------------------------------------------------------- + + /// + /// Walks every cve_* cache row, batches all CVE ids through the EPSS API and rewrites + /// only the rows whose scores actually changed — with the row's ORIGINAL CachedAt, so + /// an EPSS rewrite never counts as a fresh NVD fetch for the 7-day TTL. Stamps + /// AdminConfiguration.EpssLastSyncUtc on success. Returns null when already running. + /// + public async Task RefreshEpssScoresAsync(string trigger) + { + if (!await _epssSlot.WaitAsync(0)) return null; + var stats = new EpssRefreshResult { StartedUtc = DateTime.UtcNow }; + LastEpssRefresh = stats; + try + { + var index = await _vulnRepo.GetStaleCveCacheEntriesAsync(); + var rows = new List<(string CacheKey, string CpeUri, DateTime CachedAt, List Cves)>(); + + foreach (var entry in index) + { + var pk = entry.ContainsKey("PartitionKey") ? entry["PartitionKey"]?.ToString() : null; + if (pk == null || !pk.StartsWith("cve_")) continue; + + var full = await _vulnRepo.GetCveCacheEntryAsync(pk, "cache_result"); + if (full == null) continue; + + var cachedAtStr = full.ContainsKey("CachedAt") ? full["CachedAt"]?.ToString() : null; + if (!DateTime.TryParse(cachedAtStr, null, System.Globalization.DateTimeStyles.RoundtripKind, out var cachedAt)) + continue; + + var json = TableStorageChunking.ReassembleProperty(full, "CveDataJson", _logger, $"{pk}/cache_result") ?? "[]"; + var cves = System.Text.Json.JsonSerializer.Deserialize>(json, JsonOptions) ?? new List(); + if (cves.Count == 0) continue; + + var cpeUri = full.ContainsKey("CpeUri") ? full["CpeUri"]?.ToString() ?? pk.Substring(4) : pk.Substring(4); + rows.Add((pk, cpeUri, cachedAt, cves)); + } + + stats.Rows = rows.Count; + var allIds = rows.SelectMany(r => r.Cves.Select(c => c.CveId)).Distinct(StringComparer.OrdinalIgnoreCase).ToList(); + stats.Cves = allIds.Count; + + if (allIds.Count > 0) + { + var scores = await _epssApi.GetScoresAsync(allIds, CancellationToken.None); + stats.Scored = scores.Count; + + foreach (var (cacheKey, cpeUri, cachedAt, cves) in rows) + { + var changed = ApplyEpssScores(cves, scores); + if (changed == 0) continue; + try + { + await _vulnRepo.UpsertCveCacheEntryAsync(cacheKey, "cache_result", BuildCveCacheRow(cpeUri, cves, cachedAt)); + stats.RowsRewritten++; + } + catch (Exception ex) + { + _logger.LogWarning(ex, "[VulnSync] EPSS rewrite failed for {CpeUri}", cpeUri); + } + } + } + + _logger.LogInformation( + "[VulnSync] EPSS refresh ({Trigger}): {Scored}/{Cves} CVEs scored across {Rows} rows, {Rewritten} rows rewritten", + trigger, stats.Scored, stats.Cves, stats.Rows, stats.RowsRewritten); + } + catch (Exception ex) + { + stats.Error = ex.Message; + _logger.LogWarning(ex, "[VulnSync] EPSS refresh failed"); + } + finally + { + stats.FinishedUtc = DateTime.UtcNow; + _epssSlot.Release(); + } + + if (stats.Error == null) + await StampAsync(c => c.EpssLastSyncUtc = DateTime.UtcNow.ToString("o"), "EpssLastSyncUtc"); + + return stats; + } + + private async Task StampAsync(Action set, string field) + { + try + { + var config = await _adminConfigService.GetConfigurationAsync(); + if (config == null) return; + set(config); + await _adminConfigService.SaveConfigurationAsync(config); + } + catch (Exception ex) + { + _logger.LogWarning(ex, "[VulnSync] Failed to update {Field}", field); + } + } + } +} diff --git a/src/Shared/AutopilotMonitor.Shared/Models/Config/AdminConfiguration.cs b/src/Shared/AutopilotMonitor.Shared/Models/Config/AdminConfiguration.cs index 45631a15..ca8c6e83 100644 --- a/src/Shared/AutopilotMonitor.Shared/Models/Config/AdminConfiguration.cs +++ b/src/Shared/AutopilotMonitor.Shared/Models/Config/AdminConfiguration.cs @@ -521,6 +521,19 @@ namespace AutopilotMonitor.Shared.Models /// public string MsrcLastSyncUtc { get; set; } = default!; + /// + /// Last COMPLETED refresh of the stale NVD CVE cache rows (UTC ISO 8601) — a run that walked + /// every stale row without being cut short by an NVD throttle cooldown. Updated by + /// VulnerabilityDataSyncFunction (daily timer) and the manual /api/vulnerability/sync-nvd. + /// + public string NvdCacheLastRefreshUtc { get; set; } = default!; + + /// + /// Last successful FIRST EPSS re-score of the cached CVEs (UTC ISO 8601). Updated by + /// VulnerabilityDataSyncFunction (daily timer) and the manual /api/vulnerability/sync-epss. + /// + public string EpssLastSyncUtc { get; set; } = default!; + /// /// Returns a shallow copy with all secret-bearing string fields replaced by /// (empty values are left empty). Used when diff --git a/src/Web/autopilot-monitor-web/app/admin/components/VulnerabilityDataSection.tsx b/src/Web/autopilot-monitor-web/app/admin/components/VulnerabilityDataSection.tsx index 4bb2bafb..9b566b89 100644 --- a/src/Web/autopilot-monitor-web/app/admin/components/VulnerabilityDataSection.tsx +++ b/src/Web/autopilot-monitor-web/app/admin/components/VulnerabilityDataSection.tsx @@ -68,11 +68,19 @@ export function VulnerabilityDataSection({ const [reseedingSeed, setReseedingSeed] = useState(false); const [reseedingCommunity, setReseedingCommunity] = useState(false); + const [syncingNvd, setSyncingNvd] = useState(false); + const [syncingEpss, setSyncingEpss] = useState(false); + // Live catalog/index sizes (read-only; fetched from /api/vulnerability/sync-status) const [syncStatus, setSyncStatus] = useState<{ kevCatalogEntries?: number; msrcIndexedCves?: number; msrcCoveredDocuments?: number; + nvdCacheEntries?: number; + nvdRefreshRunning?: boolean; + nvdLastRun?: { stale: number; refreshed: number; failed: number; skippedThrottled: number; error?: string | null; finishedUtc?: string | null } | null; + epssRefreshRunning?: boolean; + epssLastRun?: { cves: number; scored: number; rowsRewritten: number; error?: string | null } | null; } | null>(null); // Fleet exposure overview (read-only; cross-tenant via /api/global/metrics/vulnerability) @@ -89,6 +97,11 @@ export function VulnerabilityDataSection({ kevCatalogEntries: data.kevCatalogEntries, msrcIndexedCves: data.msrcIndexedCves, msrcCoveredDocuments: data.msrcCoveredDocuments, + nvdCacheEntries: data.nvdCacheEntries, + nvdRefreshRunning: data.nvdRefreshRunning, + nvdLastRun: data.nvdLastRun, + epssRefreshRunning: data.epssRefreshRunning, + epssLastRun: data.epssLastRun, }); } catch { // Non-critical: counts are auxiliary; timestamp still renders from admin config @@ -102,6 +115,15 @@ export function VulnerabilityDataSection({ void run(); }, [refreshSyncStatus]); + // The NVD walk runs in the backend background (minutes, rate-limited): poll the status + // while it is running so the "refreshing…" state and the final counts arrive on their own. + const nvdRunning = !!syncStatus?.nvdRefreshRunning; + useEffect(() => { + if (!nvdRunning) return; + const id = window.setInterval(() => void refreshSyncStatus(), 10_000); + return () => window.clearInterval(id); + }, [nvdRunning, refreshSyncStatus]); + const refreshSummary = useCallback(async () => { setLoadingSummary(true); try { @@ -281,6 +303,57 @@ export function VulnerabilityDataSection({ } }, [getAccessToken, setError, onAdminConfigUpdated, refreshSyncStatus]); + // NVD cache refresh + EPSS re-score share one shape: POST, show the backend's message, + // pull the admin config (timestamps) and the sync status (counts) afterwards. + const runCacheSync = useCallback(async ( + kind: "nvd" | "epss", + setBusy: (b: boolean) => void, + ) => { + const label = kind === "nvd" ? "NVD cache refresh" : "EPSS sync"; + setBusy(true); + setSyncResult(null); + setError(null); + try { + const response = await authenticatedFetch( + kind === "nvd" ? api.vulnerability.syncNvd() : api.vulnerability.syncEpss(), + getAccessToken, + { method: "POST" } + ); + const data = await response.json().catch(() => ({})); + if (!response.ok) { + // 409 = already running: informational, not an error banner. + throw new Error(data.message || `${label} failed: ${response.statusText}`); + } + setSyncResult({ + success: !!data.success, + message: data.message || `${label} completed`, + syncedAt: data.syncedAt ?? data.startedAt, + }); + try { + const configResponse = await authenticatedFetch(api.globalConfig.get(), getAccessToken); + if (configResponse.ok) { + onAdminConfigUpdated(await configResponse.json()); + } + } catch { + // Non-critical: config refresh after sync + } + void refreshSyncStatus(); + } catch (err) { + if (err instanceof TokenExpiredError) { + console.error(`Session expired during ${label}`); + } + setSyncResult({ + success: false, + message: err instanceof Error ? err.message : `${label} failed`, + }); + } finally { + setBusy(false); + } + }, [getAccessToken, setError, onAdminConfigUpdated, refreshSyncStatus]); + + const handleSyncNvd = useCallback(() => runCacheSync("nvd", setSyncingNvd), [runCacheSync]); + const handleSyncEpss = useCallback(() => runCacheSync("epss", setSyncingEpss), [runCacheSync]); + const handleReseed = useCallback(async (type: "seed" | "community") => { const isSeed = type === "seed"; if (isSeed) setReseedingSeed(true); else setReseedingCommunity(true); @@ -471,6 +544,38 @@ export function VulnerabilityDataSection({ )} +
+ Last NVD Cache Refresh + + {formatTimestamp(adminConfig?.nvdCacheLastRefreshUtc)} + {syncStatus?.nvdCacheEntries !== undefined && ( + + · {syncStatus.nvdCacheEntries.toLocaleString()} products + + )} + {syncStatus?.nvdRefreshRunning && ( + · refreshing… + )} + {!syncStatus?.nvdRefreshRunning && syncStatus?.nvdLastRun && ( + + · last run: {syncStatus.nvdLastRun.refreshed}/{syncStatus.nvdLastRun.stale} refreshed + {syncStatus.nvdLastRun.skippedThrottled > 0 ? `, ${syncStatus.nvdLastRun.skippedThrottled} throttled` : ""} + {syncStatus.nvdLastRun.failed > 0 ? `, ${syncStatus.nvdLastRun.failed} failed` : ""} + + )} + +
+
+ Last EPSS Sync + + {formatTimestamp(adminConfig?.epssLastSyncUtc)} + {syncStatus?.epssLastRun && !syncStatus.epssLastRun.error && ( + + · {syncStatus.epssLastRun.scored.toLocaleString()} / {syncStatus.epssLastRun.cves.toLocaleString()} CVEs scored + + )} + +
{/* Sync Result Message */} {syncResult && ( @@ -540,6 +645,46 @@ export function VulnerabilityDataSection({ )} + +