feat(vuln): manual "Refresh NVD Cache" and "Sync EPSS" with their own last-sync lines

The admin Data Sync Status only offered KEV and MSRC; the NVD stale-row refresh and the EPSS
re-score ran on the nightly timer alone. Both walks move out of the timer into
VulnerabilityCacheRefreshService (singleton, one in-flight slot per walk) so the buttons and the
nightly run share one code path and stamp the same timestamps (NvdCacheLastRefreshUtc — only for
a walk that reached every stale row; EpssLastSyncUtc). POST vulnerability/sync-nvd starts the
minutes-long, rate-limited NVD walk in the background and answers 202 (409 while running); the
page polls sync-status (nvdRefreshRunning, nvdLastRun) every 10 s meanwhile. POST
vulnerability/sync-epss runs synchronously and returns the counts. The timer skips a slot a
manual run holds instead of queueing a duplicate. Shared manifest regenerated for the two new
AdminConfiguration fields.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Oliver Kieselbach
2026-08-29 11:57:55 +02:00
parent a6184f165b
commit cf76c4338f
16 changed files with 715 additions and 250 deletions
@@ -100,7 +100,7 @@ means NVD is refusing us for real. Two things then happen, and one deliberately
(better than nothing, logged); a product with no row at all is reported without CVE data for
that run and gets a real attempt on the next correlation.
# Nightly sync (`VulnerabilityDataSync`, 03:00 UTC)
# Nightly sync (`VulnerabilityDataSync`, 03:00 UTC) and the manual buttons
1. KEV catalog refresh (blob mirror updated).
2. MSRC index refresh (last six Patch Tuesdays).
@@ -109,8 +109,25 @@ means NVD is refusing us for real. Two things then happen, and one deliberately
percentile / date changed are rewritten (with their original `CachedAt`). A missing id in the
response keeps yesterday's score: a failed batch never erases data.
Steps 3 and 4 live in `VulnerabilityCacheRefreshService` (singleton, one in-flight slot per walk),
which the timer and the Global-Admin buttons share, so a manual run writes exactly what the nightly
run writes. Four endpoints, one per source, each with its own "Last … Sync" line on the admin page
(`GET vulnerability/sync-status`):
| Button | Endpoint | Shape | Timestamp |
| --- | --- | --- | --- |
| Sync KEV | `POST vulnerability/sync` | synchronous | `VulnerabilityDataLastSyncUtc` |
| Sync MSRC | `POST vulnerability/sync-msrc` | synchronous | `MsrcLastSyncUtc` |
| Refresh NVD Cache | `POST vulnerability/sync-nvd` | **202 + background** (minutes, NVD-rate-limited); 409 while running; the page polls `nvdRefreshRunning` / `nvdLastRun` every 10 s | `NvdCacheLastRefreshUtc` - stamped only for a walk that reached every stale row (no throttle cut-off) |
| Sync EPSS | `POST vulnerability/sync-epss` | synchronous (seconds); 409 while running | `EpssLastSyncUtc` |
The timer skips a walk whose slot a manual run currently holds (`cveSkippedBusy` in telemetry)
rather than queueing a duplicate. `nvdLastRun` / `epssLastRun` are in-memory on the answering
instance (null after a cold start); the persisted timestamps are the durable signal.
Telemetry event `VulnerabilityDataSyncCompleted` carries `epssRows / epssCves / epssScored /
epssRowsRewritten / epssError` next to the KEV / MSRC / CVE counters.
epssRowsRewritten / epssError` next to the KEV / MSRC / CVE counters; the manual EPSS run emits
`ManualEpssSyncCompleted`.
# Examples
+4
View File
@@ -1,5 +1,9 @@
# Log
## 2026-08-29 (3)
* **Update**: `backend/vulnerability-correlation-sources.md` - the admin page's Data Sync Status is complete: next to Sync KEV / Sync MSRC there are now Refresh NVD Cache and Sync EPSS, each with its own last-sync line (`NvdCacheLastRefreshUtc`, `EpssLastSyncUtc` on AdminConfiguration; shared manifest regenerated). The NVD and EPSS walks moved out of the timer into `VulnerabilityCacheRefreshService` (singleton, one in-flight slot per walk) so the button and the nightly run are literally the same code; the NVD walk is minutes long and rate-limited, so `POST vulnerability/sync-nvd` answers 202 and the page polls `sync-status` (`nvdRefreshRunning`, `nvdLastRun`) while it runs, `sync-epss` runs synchronously. 409 while a walk is in flight; the timer skips a slot a manual run holds. Operator-only surface - no customer docs.
## 2026-08-29 (2)
* **New**: `backend/vulnerability-correlation-sources.md` - FIRST EPSS joins NVD / CISA KEV / MSRC as the fourth vulnerability source: the probability + percentile + date live on the cached CVE (one batch of <= 100 ids per CPE fill, every `cve_*` row re-scored by the nightly sync, only changed rows rewritten and with their original `CachedAt`), the CVSS vector is persisted instead of dropped, and each CVE carries an act / attend / track priority (KEV; EPSS >= 0.1 or CVSS >= 9.0; rest) next to the unchanged severity bands - two visible signals, no composite score. The comparison with an open-source CVE MCP server surfaced the real 429 defect: Polly already retried, but a refused NVD answer came back as an empty response and was cached as "0 CVEs" for seven days. `NvdFetchResult` now separates "answered" from "refused" (nothing persists a refusal - CPE cache, negative auto-resolve cache, admin auto-resolve), and a 429 opens a process-wide cooldown (`Retry-After` capped at 120 s, else 30 s) so the run fails fast instead of paying a retry chain per software item; hard-expired rows keep serving meanwhile. NVD -> cache projection and the row builder are now single functions shared by the live fill and the sync. CveIndex / fleet summary carry `epssScore` + `priority`; portal shows EPSS pill, priority pill, vector; MCP tool descriptions and the CVE-audit prompt name the fields. ExploitDB recorded as a deliberate not-now. Backend before web; trust pages (terms attribution, privacy, data-flows), settings and portal-guide docs updated in the same change.
@@ -900,6 +900,8 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
{ "SessionDeletionKillSwitch", config.SessionDeletionKillSwitch },
{ "VulnerabilityDataLastSyncUtc", config.VulnerabilityDataLastSyncUtc },
{ "MsrcLastSyncUtc", config.MsrcLastSyncUtc },
{ "NvdCacheLastRefreshUtc", config.NvdCacheLastRefreshUtc },
{ "EpssLastSyncUtc", config.EpssLastSyncUtc },
{ "MaxDiagnosticsDownloadSizeMB", config.MaxDiagnosticsDownloadSizeMB },
{ "DiagnosticsDownloadTimeoutSeconds", config.DiagnosticsDownloadTimeoutSeconds },
{ "OpsEventRetentionDays", config.OpsEventRetentionDays },
@@ -969,6 +971,8 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
SessionDeletionKillSwitch = entity.GetBoolean("SessionDeletionKillSwitch") ?? false,
VulnerabilityDataLastSyncUtc = entity.GetString("VulnerabilityDataLastSyncUtc"),
MsrcLastSyncUtc = entity.GetString("MsrcLastSyncUtc"),
NvdCacheLastRefreshUtc = entity.GetString("NvdCacheLastRefreshUtc"),
EpssLastSyncUtc = entity.GetString("EpssLastSyncUtc"),
MaxDiagnosticsDownloadSizeMB = entity.GetInt32("MaxDiagnosticsDownloadSizeMB") ?? 500,
DiagnosticsDownloadTimeoutSeconds = entity.GetInt32("DiagnosticsDownloadTimeoutSeconds") ?? 120,
OpsEventRetentionDays = entity.GetInt32("OpsEventRetentionDays") ?? 90,
@@ -19,17 +19,20 @@ namespace AutopilotMonitor.Functions.Functions.Vulnerability
{
private readonly KevDataService _kevService;
private readonly MsrcCorrelationService _msrcService;
private readonly VulnerabilityCacheRefreshService _refresh;
private readonly AdminConfigurationService _adminConfigService;
private readonly ILogger<GetVulnerabilitySyncStatusFunction> _logger;
public GetVulnerabilitySyncStatusFunction(
KevDataService kevService,
MsrcCorrelationService msrcService,
VulnerabilityCacheRefreshService refresh,
AdminConfigurationService adminConfigService,
ILogger<GetVulnerabilitySyncStatusFunction> logger)
{
_kevService = kevService;
_msrcService = msrcService;
_refresh = refresh;
_adminConfigService = adminConfigService;
_logger = logger;
}
@@ -46,6 +49,9 @@ namespace AutopilotMonitor.Functions.Functions.Vulnerability
await _msrcService.EnsureHydratedAsync();
var config = await _adminConfigService.GetConfigurationAsync();
var nvdCacheEntries = await _refresh.CountCveCacheRowsAsync();
var nvd = _refresh.LastNvdRefresh;
var epss = _refresh.LastEpssRefresh;
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(new
@@ -54,7 +60,26 @@ namespace AutopilotMonitor.Functions.Functions.Vulnerability
kevLastSyncUtc = config?.VulnerabilityDataLastSyncUtc,
msrcIndexedCves = _msrcService.IndexSize,
msrcCoveredDocuments = _msrcService.CoveredDocuments.Count,
msrcLastSyncUtc = config?.MsrcLastSyncUtc
msrcLastSyncUtc = config?.MsrcLastSyncUtc,
nvdCacheEntries,
nvdCacheLastRefreshUtc = config?.NvdCacheLastRefreshUtc,
nvdRefreshRunning = _refresh.NvdRefreshRunning,
// Last run on THIS instance (in-memory; null after a cold start) — the
// persisted timestamp above is the durable "last completed" signal.
nvdLastRun = nvd == null ? null : new
{
stale = nvd.Stale, refreshed = nvd.Refreshed, failed = nvd.Failed,
skippedThrottled = nvd.SkippedThrottled, error = nvd.Error,
startedUtc = nvd.StartedUtc.ToString("o"), finishedUtc = nvd.FinishedUtc?.ToString("o"),
},
epssLastSyncUtc = config?.EpssLastSyncUtc,
epssRefreshRunning = _refresh.EpssRefreshRunning,
epssLastRun = epss == null ? null : new
{
rows = epss.Rows, cves = epss.Cves, scored = epss.Scored,
rowsRewritten = epss.RowsRewritten, error = epss.Error,
startedUtc = epss.StartedUtc.ToString("o"), finishedUtc = epss.FinishedUtc?.ToString("o"),
},
});
return response;
}
@@ -0,0 +1,86 @@
using System;
using System.Collections.Generic;
using System.Net;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Services.Vulnerability;
using Microsoft.ApplicationInsights;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Functions.Vulnerability
{
/// <summary>
/// Manual FIRST EPSS re-score of every cached CVE (the EPSS phase of the daily timer). A
/// handful of batched requests — runs synchronously and answers with the counts. 409 when a
/// re-score is already in flight. Global Admin only.
/// </summary>
public class TriggerEpssSyncFunction
{
private readonly VulnerabilityCacheRefreshService _refresh;
private readonly TelemetryClient? _telemetryClient;
private readonly ILogger<TriggerEpssSyncFunction> _logger;
public TriggerEpssSyncFunction(
VulnerabilityCacheRefreshService refresh,
TelemetryClient? telemetryClient,
ILogger<TriggerEpssSyncFunction> logger)
{
_refresh = refresh;
_telemetryClient = telemetryClient;
_logger = logger;
}
[Function("TriggerEpssSync")]
public async Task<HttpResponseData> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = "vulnerability/sync-epss")] HttpRequestData req)
{
try
{
var result = await _refresh.RefreshEpssScoresAsync("manual");
if (result == null)
{
var busy = req.CreateResponse(HttpStatusCode.Conflict);
await busy.WriteAsJsonAsync(new { success = false, message = "An EPSS sync is already running" });
return busy;
}
var durationMs = (int)((result.FinishedUtc ?? DateTime.UtcNow) - result.StartedUtc).TotalMilliseconds;
_telemetryClient?.TrackEvent("ManualEpssSyncCompleted", new Dictionary<string, string>
{
{ "rows", result.Rows.ToString() },
{ "cves", result.Cves.ToString() },
{ "scored", result.Scored.ToString() },
{ "rowsRewritten", result.RowsRewritten.ToString() },
{ "error", result.Error ?? "" },
{ "durationMs", durationMs.ToString() }
});
var ok = result.Error == null;
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(new
{
success = ok,
message = ok
? $"EPSS scores refreshed: {result.Scored:N0} of {result.Cves:N0} cached CVEs scored, {result.RowsRewritten:N0} rows updated"
: $"EPSS sync failed: {result.Error}",
epssRows = result.Rows,
epssCves = result.Cves,
epssScored = result.Scored,
epssRowsRewritten = result.RowsRewritten,
epssError = result.Error,
durationMs,
syncedAt = DateTime.UtcNow.ToString("o")
});
return response;
}
catch (Exception ex)
{
_logger.LogError(ex, "Manual EPSS sync failed");
var errorResponse = req.CreateResponse(HttpStatusCode.InternalServerError);
await errorResponse.WriteAsJsonAsync(new { success = false, message = "EPSS sync failed: " + ex.Message });
return errorResponse;
}
}
}
}
@@ -0,0 +1,51 @@
using System;
using System.Net;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Services.Vulnerability;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Functions.Vulnerability
{
/// <summary>
/// Manual NVD cache refresh (the stale-row phase of the daily timer). The walk is sequential
/// and NVD-rate-limited, so it can run for minutes: this endpoint only STARTS it and answers
/// 202; progress and the outcome are visible on <c>/api/vulnerability/sync-status</c>
/// (<c>nvdRefreshRunning</c>, <c>nvdLastRun</c>). 409 when a walk is already in flight.
/// Global Admin only.
/// </summary>
public class TriggerNvdCacheRefreshFunction
{
private readonly VulnerabilityCacheRefreshService _refresh;
private readonly ILogger<TriggerNvdCacheRefreshFunction> _logger;
public TriggerNvdCacheRefreshFunction(VulnerabilityCacheRefreshService refresh, ILogger<TriggerNvdCacheRefreshFunction> logger)
{
_refresh = refresh;
_logger = logger;
}
[Function("TriggerNvdCacheRefresh")]
public async Task<HttpResponseData> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = "vulnerability/sync-nvd")] HttpRequestData req)
{
if (!_refresh.TryStartNvdRefreshInBackground("manual"))
{
var busy = req.CreateResponse(HttpStatusCode.Conflict);
await busy.WriteAsJsonAsync(new { success = false, message = "An NVD cache refresh is already running" });
return busy;
}
_logger.LogInformation("Manual NVD cache refresh started");
var response = req.CreateResponse(HttpStatusCode.Accepted);
await response.WriteAsJsonAsync(new
{
success = true,
message = "NVD cache refresh started — stale entries are re-queried sequentially; check the sync status for progress",
startedAt = DateTime.UtcNow.ToString("o")
});
return response;
}
}
}
@@ -1,23 +1,23 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.DataAccess.TableStorage;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Functions.Services.Vulnerability;
using AutopilotMonitor.Shared.DataAccess;
using Microsoft.ApplicationInsights;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.Logging;
using static AutopilotMonitor.Functions.Services.Vulnerability.VulnerabilityCorrelationService;
namespace AutopilotMonitor.Functions.Functions.Vulnerability
{
/// <summary>
/// Timer-triggered function that refreshes vulnerability data caches daily.
/// - CISA KEV catalog (full in-memory refresh)
/// - MSRC CVRF index (last 6 Patch Tuesdays)
/// - Stale NVD CVE cache entries in VulnerabilityCache table (older than 7 days)
/// - FIRST EPSS re-score of every cached CVE
///
/// The NVD and EPSS walks live in <see cref="VulnerabilityCacheRefreshService"/>, shared with
/// the admin's manual buttons, so both paths behave identically.
///
/// All operations are logged to ILogger → visible in Application Insights under
/// "traces" with customDimensions.Category = "VulnerabilityDataSyncFunction".
@@ -26,29 +26,23 @@ namespace AutopilotMonitor.Functions.Functions.Vulnerability
public class VulnerabilityDataSyncFunction
{
private readonly KevDataService _kevService;
private readonly NvdApiClient _nvdApi;
private readonly EpssApiClient _epssApi;
private readonly MsrcCorrelationService _msrcCorrelation;
private readonly IVulnerabilityRepository _vulnRepo;
private readonly VulnerabilityCacheRefreshService _refresh;
private readonly AdminConfigurationService _adminConfigService;
private readonly TelemetryClient? _telemetryClient;
private readonly ILogger<VulnerabilityDataSyncFunction> _logger;
public VulnerabilityDataSyncFunction(
KevDataService kevService,
NvdApiClient nvdApi,
EpssApiClient epssApi,
MsrcCorrelationService msrcCorrelation,
IVulnerabilityRepository vulnRepo,
VulnerabilityCacheRefreshService refresh,
AdminConfigurationService adminConfigService,
TelemetryClient? telemetryClient,
ILogger<VulnerabilityDataSyncFunction> logger)
{
_kevService = kevService;
_nvdApi = nvdApi;
_epssApi = epssApi;
_msrcCorrelation = msrcCorrelation;
_vulnRepo = vulnRepo;
_refresh = refresh;
_adminConfigService = adminConfigService;
_telemetryClient = telemetryClient;
_logger = logger;
@@ -66,9 +60,6 @@ namespace AutopilotMonitor.Functions.Functions.Vulnerability
int kevCount = 0;
string? kevError = null;
int cveRefreshed = 0;
int cveFailed = 0;
int cveStale = 0;
int msrcDocs = 0;
int msrcCves = 0;
string? msrcError = null;
@@ -110,19 +101,20 @@ namespace AutopilotMonitor.Functions.Functions.Vulnerability
_logger.LogWarning(ex, "[VulnSync] MSRC index refresh failed");
}
// 3. Refresh stale CVE cache entries
var cveStats = await RefreshStaleCveCacheAsync();
cveStale = cveStats.Stale;
cveRefreshed = cveStats.Refreshed;
cveFailed = cveStats.Failed;
// 3. Refresh stale CVE cache entries (null = a manual walk is in flight; skip, it does the same work).
var cveStats = await _refresh.RefreshStaleCveCacheAsync("timer");
if (cveStats == null)
_logger.LogInformation("[VulnSync] CVE cache refresh skipped: a manual refresh is already running");
// 3b. Re-score EVERY cached CVE row with today's EPSS. Independent of the 7-day NVD
// TTL: EPSS is republished daily and the calls are cheap (one batch per 100 ids).
var epssStats = await RefreshEpssScoresAsync();
var epssStats = await _refresh.RefreshEpssScoresAsync("timer");
if (epssStats == null)
_logger.LogInformation("[VulnSync] EPSS refresh skipped: a manual sync is already running");
// 4. Update per-source sync timestamps. Only the sources that succeeded get
// bumped — a failed phase keeps its prior timestamp so the UI can surface
// "Last KEV/MSRC Sync" accurately.
// "Last KEV/MSRC Sync" accurately. (NVD/EPSS stamps are written by the service.)
try
{
var config = await _adminConfigService.GetConfigurationAsync();
@@ -160,229 +152,18 @@ namespace AutopilotMonitor.Functions.Functions.Vulnerability
{ "msrcDocs", msrcDocs.ToString() },
{ "msrcCves", msrcCves.ToString() },
{ "msrcError", msrcError ?? "" },
{ "cveStale", cveStale.ToString() },
{ "cveRefreshed", cveRefreshed.ToString() },
{ "cveFailed", cveFailed.ToString() },
{ "cveSkippedThrottled", cveStats.SkippedThrottled.ToString() },
{ "epssRows", epssStats.Rows.ToString() },
{ "epssCves", epssStats.Cves.ToString() },
{ "epssScored", epssStats.Scored.ToString() },
{ "epssRowsRewritten", epssStats.RowsRewritten.ToString() },
{ "epssError", epssStats.Error ?? "" },
{ "cveStale", (cveStats?.Stale ?? 0).ToString() },
{ "cveRefreshed", (cveStats?.Refreshed ?? 0).ToString() },
{ "cveFailed", (cveStats?.Failed ?? 0).ToString() },
{ "cveSkippedThrottled", (cveStats?.SkippedThrottled ?? 0).ToString() },
{ "cveSkippedBusy", (cveStats == null).ToString() },
{ "epssRows", (epssStats?.Rows ?? 0).ToString() },
{ "epssCves", (epssStats?.Cves ?? 0).ToString() },
{ "epssScored", (epssStats?.Scored ?? 0).ToString() },
{ "epssRowsRewritten", (epssStats?.RowsRewritten ?? 0).ToString() },
{ "epssError", epssStats?.Error ?? "" },
{ "durationMs", durationMs.ToString("F0", System.Globalization.CultureInfo.InvariantCulture) }
});
}
private struct CveCacheStats
{
public int Stale;
public int Refreshed;
public int Failed;
/// <summary>Stale entries left untouched because NVD opened its throttle cooldown mid-run.</summary>
public int SkippedThrottled;
}
private struct EpssRefreshStats
{
public int Rows;
public int Cves;
public int Scored;
public int RowsRewritten;
public string? Error;
}
/// <summary>
/// Walks every <c>cve_*</c> cache row, batches all CVE ids through the EPSS API and rewrites
/// only the rows whose scores actually changed — with the row's ORIGINAL <c>CachedAt</c>, so
/// an EPSS rewrite never counts as a fresh NVD fetch for the 7-day TTL.
/// </summary>
private async Task<EpssRefreshStats> RefreshEpssScoresAsync()
{
var stats = new EpssRefreshStats();
try
{
var index = await _vulnRepo.GetStaleCveCacheEntriesAsync();
var rows = new List<(string CacheKey, string CpeUri, DateTime CachedAt, List<CachedCve> Cves)>();
foreach (var entry in index)
{
var pk = entry.ContainsKey("PartitionKey") ? entry["PartitionKey"]?.ToString() : null;
if (pk == null || !pk.StartsWith("cve_")) continue;
var full = await _vulnRepo.GetCveCacheEntryAsync(pk, "cache_result");
if (full == null) continue;
var cachedAtStr = full.ContainsKey("CachedAt") ? full["CachedAt"]?.ToString() : null;
if (!DateTime.TryParse(cachedAtStr, null, System.Globalization.DateTimeStyles.RoundtripKind, out var cachedAt))
continue;
var json = TableStorageChunking.ReassembleProperty(full, "CveDataJson", _logger, $"{pk}/cache_result") ?? "[]";
var cves = System.Text.Json.JsonSerializer.Deserialize<List<CachedCve>>(json, JsonOptions) ?? new List<CachedCve>();
if (cves.Count == 0) continue;
var cpeUri = full.ContainsKey("CpeUri") ? full["CpeUri"]?.ToString() ?? pk.Substring(4) : pk.Substring(4);
rows.Add((pk, cpeUri, cachedAt, cves));
}
stats.Rows = rows.Count;
var allIds = rows.SelectMany(r => r.Cves.Select(c => c.CveId)).Distinct(StringComparer.OrdinalIgnoreCase).ToList();
stats.Cves = allIds.Count;
if (allIds.Count == 0)
{
_logger.LogInformation("[VulnSync] EPSS refresh: no cached CVEs to score");
return stats;
}
var scores = await _epssApi.GetScoresAsync(allIds, CancellationToken.None);
stats.Scored = scores.Count;
foreach (var (cacheKey, cpeUri, cachedAt, cves) in rows)
{
var changed = VulnerabilityCorrelationService.ApplyEpssScores(cves, scores);
if (changed == 0) continue;
try
{
await _vulnRepo.UpsertCveCacheEntryAsync(cacheKey, "cache_result",
VulnerabilityCorrelationService.BuildCveCacheRow(cpeUri, cves, cachedAt));
stats.RowsRewritten++;
}
catch (Exception ex)
{
_logger.LogWarning(ex, "[VulnSync] EPSS rewrite failed for {CpeUri}", cpeUri);
}
}
_logger.LogInformation(
"[VulnSync] EPSS refresh: {Scored}/{Cves} CVEs scored across {Rows} rows, {Rewritten} rows rewritten",
stats.Scored, stats.Cves, stats.Rows, stats.RowsRewritten);
}
catch (Exception ex)
{
stats.Error = ex.Message;
_logger.LogWarning(ex, "[VulnSync] EPSS refresh failed");
}
return stats;
}
/// <summary>
/// Scans the VulnerabilityCache table for CVE cache entries older than 7 days
/// and refreshes them by re-querying the NVD API. Processes sequentially with
/// rate limiting to avoid NVD API throttling.
/// </summary>
private async Task<CveCacheStats> RefreshStaleCveCacheAsync()
{
var stats = new CveCacheStats();
try
{
var staleCutoff = DateTime.UtcNow.AddDays(-7).ToString("o");
var cacheEntries = await _vulnRepo.GetStaleCveCacheEntriesAsync();
var staleEntries = new List<(string CacheKey, string CpeUri, string CachedAt, int TotalCves)>();
foreach (var entity in cacheEntries)
{
var pk = entity.ContainsKey("PartitionKey") ? entity["PartitionKey"]?.ToString() : null;
if (pk == null || !pk.StartsWith("cve_")) continue;
var cachedAt = entity.ContainsKey("CachedAt") ? entity["CachedAt"]?.ToString() ?? "" : "";
if (string.Compare(cachedAt, staleCutoff, StringComparison.OrdinalIgnoreCase) < 0)
{
staleEntries.Add((
pk,
entity.ContainsKey("CpeUri") ? entity["CpeUri"]?.ToString() ?? pk.Substring(4) : pk.Substring(4),
cachedAt,
entity.ContainsKey("TotalCves") && entity["TotalCves"] is int tc ? tc : 0
));
}
}
stats.Stale = staleEntries.Count;
if (staleEntries.Count == 0)
{
_logger.LogInformation("[VulnSync] CVE cache: no stale entries found");
return stats;
}
_logger.LogInformation("[VulnSync] CVE cache: {StaleCount} stale entries to refresh", staleEntries.Count);
int refreshed = 0;
int failed = 0;
int skippedThrottled = 0;
for (int i = 0; i < staleEntries.Count; i++)
{
var (cacheKey, cpeUri, cachedAt, totalCves) = staleEntries[i];
try
{
_logger.LogInformation("[VulnSync] Refreshing CVE cache for {CpeUri} (cached: {CachedAt}, had {TotalCves} CVEs)",
cpeUri, cachedAt, totalCves);
// Query NVD API (rate-limited internally by NvdApiClient)
var fetch = await _nvdApi.SearchCvesByCpeAsync(cpeUri, CancellationToken.None);
if (fetch.Throttled)
{
// NVD is refusing us; the stale rows keep serving (soft-stale reads
// still return data) and get another attempt tomorrow. Bail out of
// the loop instead of paying the cooldown skip per remaining entry.
skippedThrottled = staleEntries.Count - i;
_logger.LogWarning("[VulnSync] NVD throttled — leaving {Remaining} stale entries for the next run", skippedThrottled);
break;
}
if (!fetch.Succeeded)
{
failed++;
continue; // row untouched: never overwrite real data with a refusal
}
// Project NVD response → CachedCve list (same shape the read path expects),
// under the same cache budget the live FetchAndCache path uses.
var cves = VulnerabilityCorrelationService.ProjectNvdResponse(fetch.Data);
var budget = CveCacheBudget.Apply(cves, DateTime.UtcNow);
cves = budget.Cves;
// EPSS is re-applied for all rows in step 3b; scoring here keeps the row
// complete in case that step fails.
try
{
var scores = await _epssApi.GetScoresAsync(cves.Select(c => c.CveId), CancellationToken.None);
VulnerabilityCorrelationService.ApplyEpssScores(cves, scores);
}
catch (Exception ex)
{
_logger.LogWarning(ex, "[VulnSync] EPSS enrichment failed for {CpeUri}", cpeUri);
}
var row = VulnerabilityCorrelationService.BuildCveCacheRow(cpeUri, cves, DateTime.UtcNow);
await _vulnRepo.UpsertCveCacheEntryAsync(cacheKey, "cache_result", row);
refreshed++;
_logger.LogInformation(
"[VulnSync] Refreshed {CpeUri}: {OldCount} → {NewCount} CVEs (NVD={Total}, afterDate={AfterDate}, truncRanges={TruncCves})",
cpeUri, totalCves, cves.Count,
budget.TotalIn, budget.KeptAfterDateFilter, budget.CvesWithTruncatedRanges);
}
catch (Exception ex)
{
failed++;
_logger.LogWarning(ex, "[VulnSync] Failed to refresh CVE cache for {CpeUri}", cpeUri);
}
}
_logger.LogInformation("[VulnSync] CVE cache refresh complete: {Refreshed} refreshed, {Failed} failed, {Skipped} skipped (throttled) out of {Total} stale entries",
refreshed, failed, skippedThrottled, staleEntries.Count);
stats.Refreshed = refreshed;
stats.Failed = failed;
stats.SkippedThrottled = skippedThrottled;
}
catch (Exception ex)
{
_logger.LogWarning(ex, "[VulnSync] CVE cache refresh scan failed");
}
return stats;
}
}
}
@@ -410,6 +410,7 @@ builder.Services.AddHttpClient<AutopilotMonitor.Functions.Services.Vulnerability
builder.Services.AddHttpClient<AutopilotMonitor.Functions.Services.Vulnerability.EpssApiClient>()
.AddPolicyHandler((sp, _) => sp.GetRequiredService<ResiliencePolicies>().ExternalDataApi);
builder.Services.AddSingleton<AutopilotMonitor.Functions.Services.Vulnerability.MsrcCorrelationService>();
builder.Services.AddSingleton<AutopilotMonitor.Functions.Services.Vulnerability.VulnerabilityCacheRefreshService>();
builder.Services.AddSingleton<AutopilotMonitor.Functions.Services.Vulnerability.VulnerabilityCorrelationService>();
// Hydrate MSRC + KEV in-memory caches from blob snapshots at app startup (fire-and-forget;
// keeps cold-start fast for re-deploys, see VulnerabilityCacheWarmer for the contract).
@@ -547,6 +547,8 @@ public static class EndpointAccessPolicyCatalog
new("GET", "vulnerability/software-inventory", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
new("POST", "vulnerability/sync", EndpointPolicy.GlobalAdminOnly),
new("POST", "vulnerability/sync-msrc", EndpointPolicy.GlobalAdminOnly),
new("POST", "vulnerability/sync-nvd", EndpointPolicy.GlobalAdminOnly),
new("POST", "vulnerability/sync-epss", EndpointPolicy.GlobalAdminOnly),
new("GET", "vulnerability/sync-status", EndpointPolicy.GlobalReadOrAdmin),
new("POST", "vulnerability/cpe-mapping", EndpointPolicy.GlobalAdminOnly),
new("POST", "vulnerability/cpe-mapping/auto-resolve", EndpointPolicy.GlobalAdminOnly),
@@ -0,0 +1,328 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.DataAccess.TableStorage;
using AutopilotMonitor.Shared.DataAccess;
using Microsoft.Extensions.Logging;
using static AutopilotMonitor.Functions.Services.Vulnerability.VulnerabilityCorrelationService;
namespace AutopilotMonitor.Functions.Services.Vulnerability
{
/// <summary>Outcome of one NVD stale-cache refresh walk.</summary>
public sealed class NvdCacheRefreshResult
{
public int Stale { get; set; }
public int Refreshed { get; set; }
public int Failed { get; set; }
/// <summary>Stale entries left untouched because NVD opened its throttle cooldown mid-run.</summary>
public int SkippedThrottled { get; set; }
public string? Error { get; set; }
public DateTime StartedUtc { get; set; }
public DateTime? FinishedUtc { get; set; }
/// <summary>True when every stale row got a real attempt (no throttle cut-off, no scan error).</summary>
public bool Completed => Error == null && SkippedThrottled == 0;
}
/// <summary>Outcome of one EPSS re-score over all cached CVE rows.</summary>
public sealed class EpssRefreshResult
{
public int Rows { get; set; }
public int Cves { get; set; }
public int Scored { get; set; }
public int RowsRewritten { get; set; }
public string? Error { get; set; }
public DateTime StartedUtc { get; set; }
public DateTime? FinishedUtc { get; set; }
}
/// <summary>
/// The two VulnerabilityCache maintenance walks — NVD stale-row refresh and FIRST EPSS re-score —
/// shared by the daily timer (<c>VulnerabilityDataSyncFunction</c>) and the admin's manual
/// buttons (<c>/api/vulnerability/sync-nvd</c>, <c>/api/vulnerability/sync-epss</c>), so both
/// paths write identical rows and stamp the same AdminConfiguration timestamps.
/// <para>
/// Singleton with one in-flight slot per walk: a second caller gets <c>null</c> instead of a
/// concurrent duplicate (two NVD walks would just fight over the same rate limit). The NVD walk
/// is sequential and rate-limited (2–8 s per CPE) and therefore may run for minutes — the manual
/// endpoint starts it in the background (<see cref="TryStartNvdRefreshInBackground"/>) and the
/// status endpoint reports <see cref="NvdRefreshRunning"/> / <see cref="LastNvdRefresh"/>.
/// The EPSS walk is a handful of batched requests and finishes in seconds.
/// </para>
/// </summary>
public class VulnerabilityCacheRefreshService
{
private readonly IVulnerabilityRepository _vulnRepo;
private readonly NvdApiClient _nvdApi;
private readonly EpssApiClient _epssApi;
private readonly AdminConfigurationService _adminConfigService;
private readonly ILogger<VulnerabilityCacheRefreshService> _logger;
private readonly SemaphoreSlim _nvdSlot = new(1, 1);
private readonly SemaphoreSlim _epssSlot = new(1, 1);
public VulnerabilityCacheRefreshService(
IVulnerabilityRepository vulnRepo,
NvdApiClient nvdApi,
EpssApiClient epssApi,
AdminConfigurationService adminConfigService,
ILogger<VulnerabilityCacheRefreshService> logger)
{
_vulnRepo = vulnRepo;
_nvdApi = nvdApi;
_epssApi = epssApi;
_adminConfigService = adminConfigService;
_logger = logger;
}
public bool NvdRefreshRunning => _nvdSlot.CurrentCount == 0;
public bool EpssRefreshRunning => _epssSlot.CurrentCount == 0;
public NvdCacheRefreshResult? LastNvdRefresh { get; private set; }
public EpssRefreshResult? LastEpssRefresh { get; private set; }
/// <summary>Number of per-CPE CVE cache rows (one point scan over the index columns).</summary>
public async Task<int> CountCveCacheRowsAsync()
{
try
{
var entries = await _vulnRepo.GetStaleCveCacheEntriesAsync();
return entries.Count(e => e.ContainsKey("PartitionKey") && (e["PartitionKey"]?.ToString() ?? "").StartsWith("cve_"));
}
catch (Exception ex)
{
_logger.LogDebug(ex, "CVE cache row count failed");
return 0;
}
}
// -----------------------------------------------------------------------
// NVD stale-row refresh
// -----------------------------------------------------------------------
/// <summary>
/// Starts <see cref="RefreshStaleCveCacheAsync"/> on the thread pool. Returns false when a
/// walk is already in flight (nothing started).
/// </summary>
public bool TryStartNvdRefreshInBackground(string trigger)
{
if (NvdRefreshRunning) return false;
_ = Task.Run(async () =>
{
try { await RefreshStaleCveCacheAsync(trigger); }
catch (Exception ex) { _logger.LogWarning(ex, "[VulnSync] Background NVD cache refresh crashed"); }
});
return true;
}
/// <summary>
/// Re-queries NVD for every VulnerabilityCache row older than 7 days, sequentially (the
/// client rate-limits). Breaks out on the first throttled answer — the stale rows keep
/// serving and get another attempt next time — and never overwrites a row with a refusal.
/// Stamps <c>AdminConfiguration.NvdCacheLastRefreshUtc</c> only for a completed walk.
/// Returns null when a walk is already running.
/// </summary>
public async Task<NvdCacheRefreshResult?> RefreshStaleCveCacheAsync(string trigger)
{
if (!await _nvdSlot.WaitAsync(0)) return null;
var result = new NvdCacheRefreshResult { StartedUtc = DateTime.UtcNow };
LastNvdRefresh = result;
try
{
var staleCutoff = DateTime.UtcNow.AddDays(-7).ToString("o");
var cacheEntries = await _vulnRepo.GetStaleCveCacheEntriesAsync();
var staleEntries = new List<(string CacheKey, string CpeUri, string CachedAt, int TotalCves)>();
foreach (var entity in cacheEntries)
{
var pk = entity.ContainsKey("PartitionKey") ? entity["PartitionKey"]?.ToString() : null;
if (pk == null || !pk.StartsWith("cve_")) continue;
var cachedAt = entity.ContainsKey("CachedAt") ? entity["CachedAt"]?.ToString() ?? "" : "";
if (string.Compare(cachedAt, staleCutoff, StringComparison.OrdinalIgnoreCase) < 0)
{
staleEntries.Add((
pk,
entity.ContainsKey("CpeUri") ? entity["CpeUri"]?.ToString() ?? pk.Substring(4) : pk.Substring(4),
cachedAt,
entity.ContainsKey("TotalCves") && entity["TotalCves"] is int tc ? tc : 0));
}
}
result.Stale = staleEntries.Count;
_logger.LogInformation("[VulnSync] CVE cache ({Trigger}): {StaleCount} stale entries to refresh", trigger, staleEntries.Count);
for (int i = 0; i < staleEntries.Count; i++)
{
var (cacheKey, cpeUri, cachedAt, totalCves) = staleEntries[i];
try
{
_logger.LogInformation("[VulnSync] Refreshing CVE cache for {CpeUri} (cached: {CachedAt}, had {TotalCves} CVEs)",
cpeUri, cachedAt, totalCves);
var fetch = await _nvdApi.SearchCvesByCpeAsync(cpeUri, CancellationToken.None);
if (fetch.Throttled)
{
result.SkippedThrottled = staleEntries.Count - i;
_logger.LogWarning("[VulnSync] NVD throttled — leaving {Remaining} stale entries for the next run", result.SkippedThrottled);
break;
}
if (!fetch.Succeeded)
{
result.Failed++;
continue; // row untouched: never overwrite real data with a refusal
}
var cves = ProjectNvdResponse(fetch.Data);
var budget = CveCacheBudget.Apply(cves, DateTime.UtcNow);
cves = budget.Cves;
// EPSS is re-applied for all rows by the EPSS walk; scoring here keeps the
// row complete in case that walk fails.
try
{
var scores = await _epssApi.GetScoresAsync(cves.Select(c => c.CveId), CancellationToken.None);
ApplyEpssScores(cves, scores);
}
catch (Exception ex)
{
_logger.LogWarning(ex, "[VulnSync] EPSS enrichment failed for {CpeUri}", cpeUri);
}
await _vulnRepo.UpsertCveCacheEntryAsync(cacheKey, "cache_result", BuildCveCacheRow(cpeUri, cves, DateTime.UtcNow));
result.Refreshed++;
_logger.LogInformation(
"[VulnSync] Refreshed {CpeUri}: {OldCount} → {NewCount} CVEs (NVD={Total}, afterDate={AfterDate}, truncRanges={TruncCves})",
cpeUri, totalCves, cves.Count, budget.TotalIn, budget.KeptAfterDateFilter, budget.CvesWithTruncatedRanges);
}
catch (Exception ex)
{
result.Failed++;
_logger.LogWarning(ex, "[VulnSync] Failed to refresh CVE cache for {CpeUri}", cpeUri);
}
}
_logger.LogInformation(
"[VulnSync] CVE cache refresh complete ({Trigger}): {Refreshed} refreshed, {Failed} failed, {Skipped} skipped (throttled) out of {Total} stale entries",
trigger, result.Refreshed, result.Failed, result.SkippedThrottled, staleEntries.Count);
}
catch (Exception ex)
{
result.Error = ex.Message;
_logger.LogWarning(ex, "[VulnSync] CVE cache refresh scan failed");
}
finally
{
result.FinishedUtc = DateTime.UtcNow;
_nvdSlot.Release();
}
if (result.Completed)
await StampAsync(c => c.NvdCacheLastRefreshUtc = DateTime.UtcNow.ToString("o"), "NvdCacheLastRefreshUtc");
return result;
}
// -----------------------------------------------------------------------
// EPSS re-score
// -----------------------------------------------------------------------
/// <summary>
/// Walks every <c>cve_*</c> cache row, batches all CVE ids through the EPSS API and rewrites
/// only the rows whose scores actually changed — with the row's ORIGINAL <c>CachedAt</c>, so
/// an EPSS rewrite never counts as a fresh NVD fetch for the 7-day TTL. Stamps
/// <c>AdminConfiguration.EpssLastSyncUtc</c> on success. Returns null when already running.
/// </summary>
public async Task<EpssRefreshResult?> RefreshEpssScoresAsync(string trigger)
{
if (!await _epssSlot.WaitAsync(0)) return null;
var stats = new EpssRefreshResult { StartedUtc = DateTime.UtcNow };
LastEpssRefresh = stats;
try
{
var index = await _vulnRepo.GetStaleCveCacheEntriesAsync();
var rows = new List<(string CacheKey, string CpeUri, DateTime CachedAt, List<CachedCve> Cves)>();
foreach (var entry in index)
{
var pk = entry.ContainsKey("PartitionKey") ? entry["PartitionKey"]?.ToString() : null;
if (pk == null || !pk.StartsWith("cve_")) continue;
var full = await _vulnRepo.GetCveCacheEntryAsync(pk, "cache_result");
if (full == null) continue;
var cachedAtStr = full.ContainsKey("CachedAt") ? full["CachedAt"]?.ToString() : null;
if (!DateTime.TryParse(cachedAtStr, null, System.Globalization.DateTimeStyles.RoundtripKind, out var cachedAt))
continue;
var json = TableStorageChunking.ReassembleProperty(full, "CveDataJson", _logger, $"{pk}/cache_result") ?? "[]";
var cves = System.Text.Json.JsonSerializer.Deserialize<List<CachedCve>>(json, JsonOptions) ?? new List<CachedCve>();
if (cves.Count == 0) continue;
var cpeUri = full.ContainsKey("CpeUri") ? full["CpeUri"]?.ToString() ?? pk.Substring(4) : pk.Substring(4);
rows.Add((pk, cpeUri, cachedAt, cves));
}
stats.Rows = rows.Count;
var allIds = rows.SelectMany(r => r.Cves.Select(c => c.CveId)).Distinct(StringComparer.OrdinalIgnoreCase).ToList();
stats.Cves = allIds.Count;
if (allIds.Count > 0)
{
var scores = await _epssApi.GetScoresAsync(allIds, CancellationToken.None);
stats.Scored = scores.Count;
foreach (var (cacheKey, cpeUri, cachedAt, cves) in rows)
{
var changed = ApplyEpssScores(cves, scores);
if (changed == 0) continue;
try
{
await _vulnRepo.UpsertCveCacheEntryAsync(cacheKey, "cache_result", BuildCveCacheRow(cpeUri, cves, cachedAt));
stats.RowsRewritten++;
}
catch (Exception ex)
{
_logger.LogWarning(ex, "[VulnSync] EPSS rewrite failed for {CpeUri}", cpeUri);
}
}
}
_logger.LogInformation(
"[VulnSync] EPSS refresh ({Trigger}): {Scored}/{Cves} CVEs scored across {Rows} rows, {Rewritten} rows rewritten",
trigger, stats.Scored, stats.Cves, stats.Rows, stats.RowsRewritten);
}
catch (Exception ex)
{
stats.Error = ex.Message;
_logger.LogWarning(ex, "[VulnSync] EPSS refresh failed");
}
finally
{
stats.FinishedUtc = DateTime.UtcNow;
_epssSlot.Release();
}
if (stats.Error == null)
await StampAsync(c => c.EpssLastSyncUtc = DateTime.UtcNow.ToString("o"), "EpssLastSyncUtc");
return stats;
}
private async Task StampAsync(Action<global::AutopilotMonitor.Shared.Models.AdminConfiguration> set, string field)
{
try
{
var config = await _adminConfigService.GetConfigurationAsync();
if (config == null) return;
set(config);
await _adminConfigService.SaveConfigurationAsync(config);
}
catch (Exception ex)
{
_logger.LogWarning(ex, "[VulnSync] Failed to update {Field}", field);
}
}
}
}
@@ -521,6 +521,19 @@ namespace AutopilotMonitor.Shared.Models
/// </summary>
public string MsrcLastSyncUtc { get; set; } = default!;
/// <summary>
/// Last COMPLETED refresh of the stale NVD CVE cache rows (UTC ISO 8601) — a run that walked
/// every stale row without being cut short by an NVD throttle cooldown. Updated by
/// VulnerabilityDataSyncFunction (daily timer) and the manual /api/vulnerability/sync-nvd.
/// </summary>
public string NvdCacheLastRefreshUtc { get; set; } = default!;
/// <summary>
/// Last successful FIRST EPSS re-score of the cached CVEs (UTC ISO 8601). Updated by
/// VulnerabilityDataSyncFunction (daily timer) and the manual /api/vulnerability/sync-epss.
/// </summary>
public string EpssLastSyncUtc { get; set; } = default!;
/// <summary>
/// Returns a shallow copy with all secret-bearing string fields replaced by
/// <see cref="Constants.RedactedSecretPlaceholder"/> (empty values are left empty). Used when
@@ -68,11 +68,19 @@ export function VulnerabilityDataSection({
const [reseedingSeed, setReseedingSeed] = useState(false);
const [reseedingCommunity, setReseedingCommunity] = useState(false);
const [syncingNvd, setSyncingNvd] = useState(false);
const [syncingEpss, setSyncingEpss] = useState(false);
// Live catalog/index sizes (read-only; fetched from /api/vulnerability/sync-status)
const [syncStatus, setSyncStatus] = useState<{
kevCatalogEntries?: number;
msrcIndexedCves?: number;
msrcCoveredDocuments?: number;
nvdCacheEntries?: number;
nvdRefreshRunning?: boolean;
nvdLastRun?: { stale: number; refreshed: number; failed: number; skippedThrottled: number; error?: string | null; finishedUtc?: string | null } | null;
epssRefreshRunning?: boolean;
epssLastRun?: { cves: number; scored: number; rowsRewritten: number; error?: string | null } | null;
} | null>(null);
// Fleet exposure overview (read-only; cross-tenant via /api/global/metrics/vulnerability)
@@ -89,6 +97,11 @@ export function VulnerabilityDataSection({
kevCatalogEntries: data.kevCatalogEntries,
msrcIndexedCves: data.msrcIndexedCves,
msrcCoveredDocuments: data.msrcCoveredDocuments,
nvdCacheEntries: data.nvdCacheEntries,
nvdRefreshRunning: data.nvdRefreshRunning,
nvdLastRun: data.nvdLastRun,
epssRefreshRunning: data.epssRefreshRunning,
epssLastRun: data.epssLastRun,
});
} catch {
// Non-critical: counts are auxiliary; timestamp still renders from admin config
@@ -102,6 +115,15 @@ export function VulnerabilityDataSection({
void run();
}, [refreshSyncStatus]);
// The NVD walk runs in the backend background (minutes, rate-limited): poll the status
// while it is running so the "refreshing…" state and the final counts arrive on their own.
const nvdRunning = !!syncStatus?.nvdRefreshRunning;
useEffect(() => {
if (!nvdRunning) return;
const id = window.setInterval(() => void refreshSyncStatus(), 10_000);
return () => window.clearInterval(id);
}, [nvdRunning, refreshSyncStatus]);
const refreshSummary = useCallback(async () => {
setLoadingSummary(true);
try {
@@ -281,6 +303,57 @@ export function VulnerabilityDataSection({
}
}, [getAccessToken, setError, onAdminConfigUpdated, refreshSyncStatus]);
// NVD cache refresh + EPSS re-score share one shape: POST, show the backend's message,
// pull the admin config (timestamps) and the sync status (counts) afterwards.
const runCacheSync = useCallback(async (
kind: "nvd" | "epss",
setBusy: (b: boolean) => void,
) => {
const label = kind === "nvd" ? "NVD cache refresh" : "EPSS sync";
setBusy(true);
setSyncResult(null);
setError(null);
try {
const response = await authenticatedFetch(
kind === "nvd" ? api.vulnerability.syncNvd() : api.vulnerability.syncEpss(),
getAccessToken,
{ method: "POST" }
);
const data = await response.json().catch(() => ({}));
if (!response.ok) {
// 409 = already running: informational, not an error banner.
throw new Error(data.message || `${label} failed: ${response.statusText}`);
}
setSyncResult({
success: !!data.success,
message: data.message || `${label} completed`,
syncedAt: data.syncedAt ?? data.startedAt,
});
try {
const configResponse = await authenticatedFetch(api.globalConfig.get(), getAccessToken);
if (configResponse.ok) {
onAdminConfigUpdated(await configResponse.json());
}
} catch {
// Non-critical: config refresh after sync
}
void refreshSyncStatus();
} catch (err) {
if (err instanceof TokenExpiredError) {
console.error(`Session expired during ${label}`);
}
setSyncResult({
success: false,
message: err instanceof Error ? err.message : `${label} failed`,
});
} finally {
setBusy(false);
}
}, [getAccessToken, setError, onAdminConfigUpdated, refreshSyncStatus]);
const handleSyncNvd = useCallback(() => runCacheSync("nvd", setSyncingNvd), [runCacheSync]);
const handleSyncEpss = useCallback(() => runCacheSync("epss", setSyncingEpss), [runCacheSync]);
const handleReseed = useCallback(async (type: "seed" | "community") => {
const isSeed = type === "seed";
if (isSeed) setReseedingSeed(true); else setReseedingCommunity(true);
@@ -471,6 +544,38 @@ export function VulnerabilityDataSection({
)}
</span>
</div>
<div className="flex items-center justify-between text-sm">
<span className="text-emerald-800 dark:text-gray-300" title="Last complete re-query of cache rows older than 7 days (NVD, sequential, rate-limited)">Last NVD Cache Refresh</span>
<span className="text-gray-900 dark:text-gray-100 font-mono text-xs">
{formatTimestamp(adminConfig?.nvdCacheLastRefreshUtc)}
{syncStatus?.nvdCacheEntries !== undefined && (
<span className="ml-2 text-emerald-700 dark:text-emerald-300 font-semibold">
· {syncStatus.nvdCacheEntries.toLocaleString()} products
</span>
)}
{syncStatus?.nvdRefreshRunning && (
<span className="ml-2 text-emerald-700 dark:text-emerald-300 italic">· refreshing…</span>
)}
{!syncStatus?.nvdRefreshRunning && syncStatus?.nvdLastRun && (
<span className="ml-2 text-gray-500 dark:text-gray-400" title={syncStatus.nvdLastRun.error || undefined}>
· last run: {syncStatus.nvdLastRun.refreshed}/{syncStatus.nvdLastRun.stale} refreshed
{syncStatus.nvdLastRun.skippedThrottled > 0 ? `, ${syncStatus.nvdLastRun.skippedThrottled} throttled` : ""}
{syncStatus.nvdLastRun.failed > 0 ? `, ${syncStatus.nvdLastRun.failed} failed` : ""}
</span>
)}
</span>
</div>
<div className="flex items-center justify-between text-sm">
<span className="text-emerald-800 dark:text-gray-300" title="Last FIRST EPSS re-score of every cached CVE">Last EPSS Sync</span>
<span className="text-gray-900 dark:text-gray-100 font-mono text-xs">
{formatTimestamp(adminConfig?.epssLastSyncUtc)}
{syncStatus?.epssLastRun && !syncStatus.epssLastRun.error && (
<span className="ml-2 text-emerald-700 dark:text-emerald-300 font-semibold">
· {syncStatus.epssLastRun.scored.toLocaleString()} / {syncStatus.epssLastRun.cves.toLocaleString()} CVEs scored
</span>
)}
</span>
</div>
{/* Sync Result Message */}
{syncResult && (
@@ -540,6 +645,46 @@ export function VulnerabilityDataSection({
</>
)}
</button>
<button
onClick={handleSyncNvd}
disabled={!canMutate || syncingNvd || nvdRunning || syncing || syncingMsrc || syncingEpss || reseedingSeed || reseedingCommunity}
className="inline-flex items-center px-4 py-2 text-sm font-medium rounded-md text-white bg-emerald-700 hover:bg-emerald-800 disabled:opacity-50 disabled:cursor-not-allowed transition-colors shadow-sm"
title="Re-queries NVD for every cached product older than 7 days. Runs in the background (sequential, NVD rate-limited) — the status line above updates while it runs"
>
{syncingNvd || nvdRunning ? (
<>
<div className="animate-spin rounded-full h-4 w-4 border-b-2 border-white mr-2"></div>
<span>Refreshing...</span>
</>
) : (
<>
<svg className="w-4 h-4 mr-2" fill="none" stroke="currentColor" viewBox="0 0 24 24" strokeWidth={1.5}>
<path strokeLinecap="round" strokeLinejoin="round" d="M20.25 7.5l-.625 10.632a2.25 2.25 0 01-2.247 2.118H6.622a2.25 2.25 0 01-2.247-2.118L3.75 7.5M10 11.25h4M3.375 7.5h17.25c.621 0 1.125-.504 1.125-1.125v-1.5c0-.621-.504-1.125-1.125-1.125H3.375c-.621 0-1.125.504-1.125 1.125v1.5c0 .621.504 1.125 1.125 1.125z" />
</svg>
<span>Refresh NVD Cache</span>
</>
)}
</button>
<button
onClick={handleSyncEpss}
disabled={!canMutate || syncingEpss || syncing || syncingMsrc || syncingNvd || reseedingSeed || reseedingCommunity}
className="inline-flex items-center px-4 py-2 text-sm font-medium rounded-md text-white bg-teal-600 hover:bg-teal-700 disabled:opacity-50 disabled:cursor-not-allowed transition-colors shadow-sm"
title="Re-scores every cached CVE with today's FIRST EPSS exploitation probability (api.first.org)"
>
{syncingEpss ? (
<>
<div className="animate-spin rounded-full h-4 w-4 border-b-2 border-white mr-2"></div>
<span>Syncing...</span>
</>
) : (
<>
<svg className="w-4 h-4 mr-2" fill="none" stroke="currentColor" viewBox="0 0 24 24" strokeWidth={1.5}>
<path strokeLinecap="round" strokeLinejoin="round" d="M3 13.125C3 12.504 3.504 12 4.125 12h2.25c.621 0 1.125.504 1.125 1.125v6.75C7.5 20.496 6.996 21 6.375 21h-2.25A1.125 1.125 0 013 19.875v-6.75zM9.75 8.625c0-.621.504-1.125 1.125-1.125h2.25c.621 0 1.125.504 1.125 1.125v11.25c0 .621-.504 1.125-1.125 1.125h-2.25a1.125 1.125 0 01-1.125-1.125V8.625zM16.5 4.125c0-.621.504-1.125 1.125-1.125h2.25C20.496 3 21 3.504 21 4.125v15.75c0 .621-.504 1.125-1.125 1.125h-2.25a1.125 1.125 0 01-1.125-1.125V4.125z" />
</svg>
<span>Sync EPSS</span>
</>
)}
</button>
<button
onClick={() => handleReseed("seed")}
disabled={!canMutate || reseedingSeed || reseedingCommunity || syncing || syncingMsrc}
+2
View File
@@ -618,6 +618,8 @@ export const api = {
vulnerability: {
sync: () => `${API_BASE_URL}/api/vulnerability/sync`,
syncMsrc: () => `${API_BASE_URL}/api/vulnerability/sync-msrc`,
syncNvd: () => `${API_BASE_URL}/api/vulnerability/sync-nvd`,
syncEpss: () => `${API_BASE_URL}/api/vulnerability/sync-epss`,
syncStatus: () => `${API_BASE_URL}/api/vulnerability/sync-status`,
syncReseed: (type: string) =>
`${API_BASE_URL}/api/vulnerability/sync${qs({ reseed: type })}`,
@@ -58,6 +58,8 @@ export interface AdminConfiguration {
vulnerabilityCorrelationEnabled?: boolean;
vulnerabilityDataLastSyncUtc?: string;
msrcLastSyncUtc?: string;
nvdCacheLastRefreshUtc?: string;
epssLastSyncUtc?: string;
opsAlertRulesJson?: string;
opsAlertTelegramEnabled?: boolean;
opsAlertTelegramChatId?: string;
@@ -55,7 +55,9 @@ export const SHARED_MANIFEST = {
"selfServiceAppHomingEnabled",
"sessionDeletionKillSwitch",
"vulnerabilityDataLastSyncUtc",
"msrcLastSyncUtc"
"msrcLastSyncUtc",
"nvdCacheLastRefreshUtc",
"epssLastSyncUtc"
]
},
"tenantConfiguration": {
@@ -51,7 +51,9 @@
"selfServiceAppHomingEnabled",
"sessionDeletionKillSwitch",
"vulnerabilityDataLastSyncUtc",
"msrcLastSyncUtc"
"msrcLastSyncUtc",
"nvdCacheLastRefreshUtc",
"epssLastSyncUtc"
]
},
"tenantConfiguration": {