feat(vuln): queue-driven correlation + WhiteGlove Part-1 inventory trigger

Replaces the fire-and-forget Task.Run inside EventIngestProcessor — which
Azure Functions scale-in could kill mid-flight, leaving sessions with no
vulnerability_report and forcing users to click "Analyze Now" — with a
dedicated queue + BackgroundService worker (vulnerability-correlate),
mirroring the analyze-on-enrollment-end pattern. Adds a V2-agent trigger
that emits a Part-1 software-inventory snapshot the moment Windows Event
62407 fires (WhiteGlove sealing success), so Part-1 vuln correlation
happens BEFORE the admin clicks Reseal — closing the gap that existed
because reboot_kill termination at sealing left no shutdown inventory
for the backend to correlate.

Backend
  * IVulnerabilityCorrelateProducer + AzureQueueVulnerabilityCorrelate
    Producer (fail-soft on agent's hot path).
  * VulnerabilityCorrelateQueueWorker (BackgroundService, 5-attempt
    visibility-timeout retry, poison-queue on exhaustion).
  * VulnerabilityCorrelateHandler ports the full pipeline (correlate +
    store + synth event + re-analyze enqueue + SignalR notify).
  * IVulnerabilityInventoryLoader extracted from
    GetVulnerabilityReportFunction so worker + manual rescan share one
    loader; phase-aware filter via whiteGlovePart parameter +
    includeAllPhases escape-hatch for the rescan UI.
  * EventIngestProcessor cutover: drops three concrete-class deps
    (_vulnerabilityCorrelation, _vulnRepo, _signalRNotification) and
    two now-orphaned private methods. 561 → 388 LOC.

Agent V2
  * WhiteGloveInventoryTrigger subscribes to EspAndHelloHost.WhiteGlove
    Completed and calls AgentAnalyzerManager.RunWhiteGlovePart1Inventory
    Snapshot via injected Action (no concrete-class coupling, trivially
    mockable). Single-fire guarded by Interlocked.Exchange.
  * IWhiteGloveCompletedSource interface keeps EspAndHelloHost internal
    sealed while letting the trigger be unit-tested with a 4-line stub.
  * AgentAnalyzerManager.RunWhiteGlovePart1InventorySnapshot runs ONLY
    SoftwareInventoryAnalyzer (LocalAdmin/IntegrityBypass deliberately
    skipped at WG hand-off — still SYSTEM context).
  * EnrollmentTerminationHandler now also tags Part 2 (WhiteGloveCompleted
    Part2 → wgPart=2); was previously untagged so the new phase filter
    couldn't see it.
  * SoftwareInventoryAnalyzer.AnalyzeAtShutdown gets per-WG-phase atomic
    idempotency via Interlocked.CompareExchange, with reset-on-failure
    so the EnrollmentTerminationHandler stays a working fallback if the
    Trigger crashes.

Cross-phase contamination & out-of-order resilience
  * Inventory loader filters by whiteGlovePart so a Part-2 correlation
    cannot see Part-1 inventory items (and vice versa).
  * Handler checks the existing report on BOTH Part-1 and Part-2 paths:
    Part-2 merges with existing Part-1, Part-1 SKIPS its store when the
    existing report already contains user_enrollment / both findings or
    has scan_summary.phase_contributions.user_enrollment > 0 — protects
    against late-arriving Part-1 retries overwriting the richer Part-2
    or merged report after at-least-once redelivery.
  * VulnerabilityCorrelationService.MergeReports gains NormalizeForMerge
    that recursively unwraps Newtonsoft JObject/JArray/JValue to native
    Dictionary/List/primitive shapes BEFORE the cast-based extract
    helpers run. Without this the normal-order Part-2 merge silently
    dropped Part-1 findings, because GetVulnerabilityReportAsync round-
    trips through JsonConvert and the existing helpers' OfType / "as
    Dictionary" casts produced empty results on JTokens.

Tests (+34 new across Backend & Agent V2.Core)
  * VulnerabilityCorrelateEnvelopeSerializationTests — wire-format pin.
  * VulnerabilityInventoryLoaderTests — phase filtering, includeAllPhases,
    lenient long-parsing.
  * VulnerabilityCorrelateHandlerSkipTests — out-of-order skip helper
    against native, JArray, JObject, both phase tag, phase_contributions
    counter.
  * VulnerabilityCorrelationServiceTests — round-trip safety of
    MergeReports against JsonConvert-deserialized existing reports.
  * WhiteGloveInventoryTriggerTests — fire-once, dedup, dispose-then-late,
    idempotent dispose, action-exception isolation, null-arg validation.
  * SoftwareInventoryAnalyzerIdempotencyTests — Part-1 dedup, Part-2
    dedup, Part-1 ≠ Part-2 independence.

Build green, 0 errors, 0 warnings. Backend 844/844 tests pass, Agent
V2.Core 825/825 tests pass (parallel-flaky tests pass isolated, see
feedback_v2_recoverypath_corrupt_snapshot_flake).

V1 ingest path (/api/agent/ingest) deliberately untouched per
project_v1_vuln_pipeline_gap — V1 is being phased out; fixing it
forward would inflate scope.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Oliver Kieselbach
2026-04-30 15:25:28 +02:00
parent 627784e877
commit dbef6921ae
28 changed files with 2468 additions and 272 deletions
@@ -0,0 +1,103 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
{
/// <summary>
/// Idempotency guards on <see cref="SoftwareInventoryAnalyzer.AnalyzeAtShutdown(int?)"/>.
/// The Part-1 shutdown snapshot is fired by two independent paths
/// (<see cref="WhiteGloveInventoryTrigger"/> and
/// <c>EnrollmentTerminationHandler.RunShutdownAnalyzers</c> at the WhiteGloveSealed exit) —
/// without the per-phase guard the analyzer would emit two shutdown events for the same
/// session, which the backend would translate to two queue messages and two correlation
/// runs. The guard wins single-fire while keeping different phases independent.
/// <para>
/// These tests exercise the real <see cref="SoftwareInventoryAnalyzer"/> end-to-end (it
/// reads the local Uninstall registry hives during <c>CollectAndNormalize</c>). They are
/// intentionally lightweight: assertions count <c>software_inventory_analysis</c> events
/// posted to the fake ingress, never the inventory contents themselves.
/// </para>
/// </summary>
public sealed class SoftwareInventoryAnalyzerIdempotencyTests : IDisposable
{
private readonly TempDirectory _tmp = new();
private readonly AgentLogger _logger;
private readonly FakeSignalIngressSink _ingress = new();
private readonly VirtualClock _clock = new(new DateTime(2026, 4, 30, 8, 45, 0, DateTimeKind.Utc));
private readonly SoftwareInventoryAnalyzer _sut;
public SoftwareInventoryAnalyzerIdempotencyTests()
{
_logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
var post = new InformationalEventPost(_ingress, _clock);
_sut = new SoftwareInventoryAnalyzer(
sessionId: "session-1",
tenantId: "tenant-1",
post: post,
logger: _logger);
}
public void Dispose()
{
try { _tmp.Dispose(); } catch { }
}
[Fact]
public void AnalyzeAtShutdown_Part1_called_twice_emits_only_one_event()
{
_sut.AnalyzeAtShutdown(whiteGlovePart: 1);
_sut.AnalyzeAtShutdown(whiteGlovePart: 1); // second call — must be a no-op
Assert.Equal(1, CountChunkZeroEventsForPart(part: 1));
}
[Fact]
public void AnalyzeAtShutdown_Part2_called_after_Part1_runs_independently()
{
_sut.AnalyzeAtShutdown(whiteGlovePart: 1);
_sut.AnalyzeAtShutdown(whiteGlovePart: 2);
Assert.Equal(1, CountChunkZeroEventsForPart(part: 1));
Assert.Equal(1, CountChunkZeroEventsForPart(part: 2));
}
[Fact]
public void AnalyzeAtShutdown_Part2_called_twice_emits_only_one_event()
{
_sut.AnalyzeAtShutdown(whiteGlovePart: 2);
_sut.AnalyzeAtShutdown(whiteGlovePart: 2);
Assert.Equal(1, CountChunkZeroEventsForPart(part: 2));
}
/// <summary>
/// Counts the chunk-0 software_inventory_analysis events for a given WG part.
/// EnrollmentEvent.Data is forwarded as TypedPayload by InformationalEventPost; the
/// inventory analyzer always emits at least one chunk-0 event per shutdown call, so a
/// chunk-0 count of 1 means the call was non-deduped, 0 means deduped, 2+ means the
/// guard failed.
/// </summary>
private int CountChunkZeroEventsForPart(int part) =>
_ingress.Posted
.Where(p =>
p.Kind == DecisionSignalKind.InformationalEvent
&& p.Payload != null
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
&& et == "software_inventory_analysis"
&& p.TypedPayload is IDictionary<string, object> data
&& data.TryGetValue("triggered_at", out var ta) && ta?.ToString() == "shutdown"
&& data.TryGetValue("chunk_index", out var ci) && Convert.ToInt32(ci) == 0
&& data.TryGetValue("whiteglove_part", out var wp) && Convert.ToInt32(wp) == part)
.Count();
}
}
@@ -0,0 +1,136 @@
#nullable enable
using System;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
/// <summary>
/// Unit tests for <see cref="WhiteGloveInventoryTrigger"/>. Verifies single-fire
/// dedup, action-on-event, and clean unsubscribe-on-dispose. The trigger is the V2
/// agent companion of backend's vulnerability-correlate queue (PR1+PR2): it fires the
/// Part-1 software-inventory snapshot while the OOBE Reseal dialog is still up.
/// </summary>
public sealed class WhiteGloveInventoryTriggerTests : IDisposable
{
private readonly TempDirectory _tmp = new();
private readonly AgentLogger _logger;
public WhiteGloveInventoryTriggerTests()
{
_logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
}
public void Dispose()
{
try { _tmp.Dispose(); } catch { }
}
[Fact]
public void OnWhiteGloveCompleted_invokes_trigger_action_once()
{
var source = new FakeWhiteGloveSource();
var triggerCount = 0;
using var sut = new WhiteGloveInventoryTrigger(
host: source,
onTrigger: () => triggerCount++,
logger: _logger);
source.RaiseWhiteGloveCompleted();
Assert.Equal(1, triggerCount);
}
[Fact]
public void OnWhiteGloveCompleted_dedups_duplicate_event_emissions()
{
var source = new FakeWhiteGloveSource();
var triggerCount = 0;
using var sut = new WhiteGloveInventoryTrigger(
host: source,
onTrigger: () => triggerCount++,
logger: _logger);
source.RaiseWhiteGloveCompleted();
source.RaiseWhiteGloveCompleted();
source.RaiseWhiteGloveCompleted();
Assert.Equal(1, triggerCount);
}
[Fact]
public void Dispose_unsubscribes_so_late_events_are_ignored()
{
var source = new FakeWhiteGloveSource();
var triggerCount = 0;
var sut = new WhiteGloveInventoryTrigger(
host: source,
onTrigger: () => triggerCount++,
logger: _logger);
sut.Dispose();
source.RaiseWhiteGloveCompleted();
Assert.Equal(0, triggerCount);
}
[Fact]
public void Dispose_is_idempotent()
{
var source = new FakeWhiteGloveSource();
using var sut = new WhiteGloveInventoryTrigger(
host: source,
onTrigger: () => { },
logger: _logger);
sut.Dispose();
sut.Dispose(); // must not throw or double-unsubscribe
}
[Fact]
public void Trigger_action_exception_does_not_propagate()
{
var source = new FakeWhiteGloveSource();
using var sut = new WhiteGloveInventoryTrigger(
host: source,
onTrigger: () => throw new InvalidOperationException("boom"),
logger: _logger);
// Must not surface to the source's event-loop — the agent must keep running
// even if the analyzer manager is mid-shutdown when the trigger fires.
var ex = Record.Exception(() => source.RaiseWhiteGloveCompleted());
Assert.Null(ex);
}
[Fact]
public void Constructor_throws_on_null_host()
{
Assert.Throws<ArgumentNullException>(() =>
new WhiteGloveInventoryTrigger(host: null!, onTrigger: () => { }, logger: _logger));
}
[Fact]
public void Constructor_throws_on_null_action()
{
var source = new FakeWhiteGloveSource();
Assert.Throws<ArgumentNullException>(() =>
new WhiteGloveInventoryTrigger(host: source, onTrigger: null!, logger: _logger));
}
[Fact]
public void Constructor_throws_on_null_logger()
{
var source = new FakeWhiteGloveSource();
Assert.Throws<ArgumentNullException>(() =>
new WhiteGloveInventoryTrigger(host: source, onTrigger: () => { }, logger: null!));
}
private sealed class FakeWhiteGloveSource : IWhiteGloveCompletedSource
{
public event EventHandler? WhiteGloveCompleted;
public void RaiseWhiteGloveCompleted() => WhiteGloveCompleted?.Invoke(this, EventArgs.Empty);
}
}
@@ -1,6 +1,7 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Shared.Models;
@@ -29,6 +30,21 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
// Captured at startup for delta detection at shutdown
private List<SoftwareEntry> _startupInventory;
// Idempotency guard — the WhiteGlove-Part-1 shutdown snapshot is fired by TWO
// independent paths: WhiteGloveInventoryTrigger (preferred, fires on Windows
// Event 62407 immediately) and EnrollmentTerminationHandler.RunShutdown(wgPart=1)
// (fallback when the WG-success event never arrives). The two paths can race —
// both check the guard, both pass it, both emit duplicate snapshots. The fix is
// an atomic Interlocked.CompareExchange that wins-or-loses up-front, BEFORE the
// work starts. On unhandled work failure the flag is reset so the second path
// can still serve as the fallback (registry-read errors, transient I/O, etc.)
// — clean failure semantics: a successful emit locks the flag forever, a
// failed attempt frees it.
// Encoded as int (0/1) instead of bool because Interlocked.CompareExchange has
// no bool overload.
private int _part1ShutdownSnapshotEmitted;
private int _part2ShutdownSnapshotEmitted;
public string Name => "SoftwareInventoryAnalyzer";
// Maximum items per event chunk to stay within Table Storage property size limits (~64 KB)
@@ -96,6 +112,29 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
/// </summary>
public void AnalyzeAtShutdown(int? whiteGlovePart)
{
// Per-WG-phase atomic claim. Set the flag BEFORE the work via CompareExchange
// so two concurrent callers (e.g. WhiteGloveInventoryTrigger and
// EnrollmentTerminationHandler firing nearly simultaneously) cannot both pass
// the guard. The loser logs and returns. Non-WG shutdowns (whiteGlovePart=null)
// skip the claim because there is exactly one such call per agent lifetime by
// construction.
if (whiteGlovePart == 1)
{
if (Interlocked.CompareExchange(ref _part1ShutdownSnapshotEmitted, 1, 0) != 0)
{
_logger.Info($"{Name}: AnalyzeAtShutdown(whiteGlovePart=1) skipped — Part-1 snapshot already emitted (likely by WhiteGloveInventoryTrigger)");
return;
}
}
else if (whiteGlovePart == 2)
{
if (Interlocked.CompareExchange(ref _part2ShutdownSnapshotEmitted, 1, 0) != 0)
{
_logger.Info($"{Name}: AnalyzeAtShutdown(whiteGlovePart=2) skipped — Part-2 snapshot already emitted");
return;
}
}
_logger.Info($"{Name}: Running shutdown analysis (delta detection, whiteGlovePart={whiteGlovePart?.ToString() ?? "none"})");
try
{
@@ -106,6 +145,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
catch (Exception ex)
{
_logger.Error($"{Name}: Shutdown analysis failed", ex);
// Reset the flag so the fallback path (e.g. EnrollmentTerminationHandler
// running after a Trigger-side failure) can still emit the snapshot.
// Successful emits keep the flag set forever — duplicate suppression wins.
if (whiteGlovePart == 1) Interlocked.Exchange(ref _part1ShutdownSnapshotEmitted, 0);
else if (whiteGlovePart == 2) Interlocked.Exchange(ref _part2ShutdownSnapshotEmitted, 0);
}
}
@@ -68,8 +68,22 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
private readonly string _stateDirectory;
private ImeLogHost? _imeLogHost;
private EspAndHelloHost? _espAndHelloHost;
private Transport.Telemetry.ITelemetrySpool? _telemetrySpool;
/// <summary>
/// Exposes the WhiteGlove-success event surface from the ESP/Hello coordinator host so
/// downstream subscribers (e.g. <c>WhiteGloveInventoryTrigger</c>) can react to the
/// pre-provisioning success window without reaching into the host's internals. Returns
/// <c>null</c> before <see cref="CreateCollectorHosts"/> has been called.
/// <para>
/// <c>internal</c> because <see cref="EspAndHelloHost"/> itself is <c>internal sealed</c>.
/// AutopilotMonitor.Agent.V2 (the runtime entry-point project) sees it via the
/// <c>InternalsVisibleTo</c> declared on this project's csproj.
/// </para>
/// </summary>
internal EspAndHelloHost? EspAndHelloHost => _espAndHelloHost;
/// <summary>
/// Exposes the IME tracker's package-state list to peripheral consumers such as the
/// <c>FinalStatusBuilder</c> in M4.6.β. Returns <c>null</c> before
@@ -149,7 +163,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
// ----- Kernel hosts (always-on; they produce decision signals) --------------------
hosts.Add(new EspAndHelloHost(
_espAndHelloHost = new EspAndHelloHost(
sessionId: sessionId,
tenantId: tenantId,
logger: logger,
@@ -161,7 +175,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
modernDeploymentBackfillEnabled: collectors.ModernDeploymentBackfillEnabled,
modernDeploymentBackfillLookbackMinutes: collectors.ModernDeploymentBackfillLookbackMinutes,
modernDeploymentHarmlessEventIds: collectors.ModernDeploymentHarmlessEventIds,
stateDirectory: _stateDirectory));
stateDirectory: _stateDirectory);
hosts.Add(_espAndHelloHost);
hosts.Add(new DesktopArrivalHost(logger, ingress, clock));
@@ -8,7 +8,7 @@ using AutopilotMonitor.DecisionCore.Engine;
namespace AutopilotMonitor.Agent.V2.Core.Orchestration
{
internal sealed class EspAndHelloHost : ICollectorHost
internal sealed class EspAndHelloHost : ICollectorHost, IWhiteGloveCompletedSource
{
public string Name => "EspAndHelloTracker";
@@ -16,6 +16,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
private readonly EspAndHelloTrackerAdapter _adapter;
private int _disposed;
/// <summary>
/// Forwarded from the internal <see cref="EspAndHelloTracker.WhiteGloveCompleted"/>
/// event so external components (e.g. <c>WhiteGloveInventoryTrigger</c>) can react
/// to WhiteGlove pre-provisioning success without needing direct access to the
/// (private) tracker. Fires after the agent observes Windows Event 62407
/// ("BootstrapStatus: Exiting page due to White Glove success") — i.e. while the
/// "Continue / Reseal" dialog is shown but BEFORE the admin clicks Reseal and the
/// Sysprep reboot fires. This is the only window in which WhiteGlove Part 1 work
/// (e.g. inventory snapshot for vulnerability correlation) can run.
/// </summary>
public event EventHandler? WhiteGloveCompleted;
public EspAndHelloHost(
string sessionId,
string tenantId,
@@ -46,15 +58,20 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
stateDirectory: stateDirectory,
modernDeploymentHarmlessEventIds: modernDeploymentHarmlessEventIds);
_tracker.WhiteGloveCompleted += OnTrackerWhiteGloveCompleted;
_adapter = new EspAndHelloTrackerAdapter(_tracker, ingress, clock);
}
public void Start() => _tracker.Start();
public void Stop() => _tracker.Stop();
private void OnTrackerWhiteGloveCompleted(object sender, EventArgs e)
=> WhiteGloveCompleted?.Invoke(this, e);
public void Dispose()
{
if (Interlocked.Exchange(ref _disposed, 1) == 1) return;
try { _tracker.WhiteGloveCompleted -= OnTrackerWhiteGloveCompleted; } catch { }
try { _adapter.Dispose(); } catch { }
try { _tracker.Dispose(); } catch { }
}
@@ -0,0 +1,20 @@
#nullable enable
using System;
namespace AutopilotMonitor.Agent.V2.Core.Orchestration
{
/// <summary>
/// Surface for components that observe the Shell-Core WhiteGlove pre-provisioning success
/// signal (Windows Event 62407) and need to forward it to dependents that don't share the
/// host's lifecycle directly. Implemented by <see cref="EspAndHelloHost"/> in production
/// and by tiny test stubs in <see cref="WhiteGloveInventoryTrigger"/> unit tests.
/// </summary>
internal interface IWhiteGloveCompletedSource
{
/// <summary>
/// Fires when the agent observes a successful WhiteGlove sealing — i.e. while the
/// OOBE "Continue / Reseal" dialog is shown and the admin has not yet clicked Reseal.
/// </summary>
event EventHandler? WhiteGloveCompleted;
}
}
@@ -0,0 +1,85 @@
#nullable enable
using System;
using System.Threading;
using AutopilotMonitor.Agent.V2.Core.Logging;
namespace AutopilotMonitor.Agent.V2.Core.Orchestration
{
/// <summary>
/// Subscribes to <see cref="EspAndHelloHost.WhiteGloveCompleted"/> and triggers a single
/// inventory snapshot tagged with <c>whiteGlovePart=1</c> while the OOBE "Continue / Reseal"
/// dialog is shown — i.e. BEFORE the admin clicks Reseal and Sysprep reboots the device.
/// This is the only window in which the agent can stage a Part-1 software-inventory event
/// for downstream vulnerability correlation; once Sysprep starts the agent is killed
/// (<c>previousExitType=reboot_kill</c> on the next start) and any pending Part-1 work is lost
/// from the live spool (it would only replay on the Part-2 agent start, by which point the
/// inventory snapshot has missed its WhiteGlove-Part-1 phase tag).
/// <para>
/// <b>Lifecycle:</b> Constructor subscribes; <see cref="Dispose"/> unsubscribes. The trigger
/// is single-fire — a defensive <see cref="Interlocked.Exchange(ref int, int)"/> guards
/// against duplicate <c>WhiteGloveCompleted</c> emissions (the V2 tracker emits at most
/// once today, but the guard makes the contract explicit).
/// </para>
/// <para>
/// <b>Action injection:</b> The trigger calls a caller-supplied <see cref="Action"/> rather
/// than holding a direct reference to <c>AgentAnalyzerManager</c>. This keeps the trigger
/// trivially testable (no concrete-class mock needed) and avoids a Core→Runtime coupling.
/// In production wiring the action is <c>() =&gt; analyzerManager.RunWhiteGlovePart1InventorySnapshot()</c>.
/// </para>
/// <para>
/// <b>Routing separation:</b> This trigger is intentionally orthogonal to the
/// <see cref="SignalAdapters.EspAndHelloTrackerAdapter"/> that maps the same WG-success
/// event into a <c>WhiteGloveShellCoreSuccess</c> decision signal. Both subscribers see
/// the event independently — single-responsibility, easy to remove.
/// </para>
/// </summary>
internal sealed class WhiteGloveInventoryTrigger : IDisposable
{
private readonly IWhiteGloveCompletedSource _host;
private readonly Action _onTrigger;
private readonly AgentLogger _logger;
private int _fired; // 0 = not yet fired, 1 = fired
private int _disposed; // 0 = live, 1 = disposed
public WhiteGloveInventoryTrigger(
IWhiteGloveCompletedSource host,
Action onTrigger,
AgentLogger logger)
{
_host = host ?? throw new ArgumentNullException(nameof(host));
_onTrigger = onTrigger ?? throw new ArgumentNullException(nameof(onTrigger));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_host.WhiteGloveCompleted += OnWhiteGloveCompleted;
}
private void OnWhiteGloveCompleted(object? sender, EventArgs e)
{
if (Interlocked.Exchange(ref _fired, 1) == 1)
{
_logger.Info("WhiteGloveInventoryTrigger: duplicate WhiteGloveCompleted ignored (already fired)");
return;
}
_logger.Info("WhiteGloveInventoryTrigger: WhiteGloveCompleted observed — running Part-1 inventory snapshot");
try
{
_onTrigger();
}
catch (Exception ex)
{
// Trigger action failures must not propagate into the tracker's event loop.
// Inventory missing is a degraded outcome but the agent must keep running.
_logger.Error("WhiteGloveInventoryTrigger: trigger action threw", ex);
}
}
public void Dispose()
{
if (Interlocked.Exchange(ref _disposed, 1) == 1) return;
try { _host.WhiteGloveCompleted -= OnWhiteGloveCompleted; } catch { }
}
}
}
@@ -148,5 +148,44 @@ namespace AutopilotMonitor.Agent.V2.Core.Runtime
}
}
}
/// <summary>
/// Runs ONLY the <see cref="SoftwareInventoryAnalyzer"/> shutdown snapshot tagged with
/// <c>whiteGlovePart=1</c>. Triggered by <c>WhiteGloveInventoryTrigger</c> when the
/// agent observes the WhiteGlove pre-provisioning success event (Windows Event 62407)
/// — i.e. while the OOBE "Continue / Reseal" dialog is shown but BEFORE the admin
/// clicks Reseal and Sysprep reboots the box.
/// <para>
/// Deliberately does NOT run <see cref="LocalAdminAnalyzer"/> or
/// <see cref="IntegrityBypassAnalyzer"/>: those snapshots are meaningful at a real
/// final agent shutdown. At the WG-Part-1 hand-off the system is still in SYSTEM
/// context with no real user yet — those analyzers would either no-op or report
/// noise. They run normally at the eventual final shutdown via
/// <see cref="RunShutdown"/>.
/// </para>
/// </summary>
public void RunWhiteGlovePart1InventorySnapshot()
{
if (!_initialised) Initialize();
if (_analyzers.Count == 0) return;
foreach (var analyzer in _analyzers)
{
if (analyzer is not SoftwareInventoryAnalyzer softwareAnalyzer) continue;
try
{
_logger.Info("WhiteGlove Part 1 inventory snapshot — running SoftwareInventoryAnalyzer.AnalyzeAtShutdown(whiteGlovePart: 1)");
softwareAnalyzer.AnalyzeAtShutdown(whiteGlovePart: 1);
}
catch (Exception ex)
{
_logger.Error("SoftwareInventoryAnalyzer threw during WhiteGlove Part 1 snapshot", ex);
}
return; // SoftwareInventoryAnalyzer is registered at most once
}
_logger.Warning("WhiteGlove Part 1 inventory snapshot requested but SoftwareInventoryAnalyzer is not registered (disabled by remote config?)");
}
}
}
@@ -263,7 +263,19 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
// WhiteGlove Part 1 exit passes whiteGlovePart=1 so SoftwareInventoryAnalyzer
// takes a baseline snapshot rather than computing a pre/post delta (the user
// sign-in phase has not run yet; the real delta computes on Part-2 completion).
int? wgPart = args.StageName == SessionStage.WhiteGloveSealed.ToString() ? 1 : (int?)null;
// WhiteGlove Part 2 exit passes whiteGlovePart=2 so the backend's vulnerability
// correlation pipeline can filter Part-2 inventory out of the Part-1 set when
// re-correlating, and tag findings with the correct enrollment phase.
// SoftwareInventoryAnalyzer's per-phase idempotency guards make a second
// Part-1 call (when WhiteGloveInventoryTrigger fired earlier) a no-op for the
// inventory analyzer while still running LocalAdmin / IntegrityBypass.
int? wgPart;
if (args.StageName == SessionStage.WhiteGloveSealed.ToString())
wgPart = 1;
else if (args.StageName == SessionStage.WhiteGloveCompletedPart2.ToString())
wgPart = 2;
else
wgPart = null;
_analyzerManager.RunShutdown(wgPart);
}
catch (Exception ex)
@@ -136,6 +136,15 @@ namespace AutopilotMonitor.Agent.V2.Runtime
EnrollmentTerminationHandler terminationHandler = null;
ServerActionDispatcher serverActionDispatcher = null;
// WhiteGlove Part-1 inventory trigger (vulnerability-correlation pipeline,
// companion of backend's vulnerability-correlate queue). Subscribes to the
// Shell-Core WG-success event so the agent can stage a software_inventory_analysis
// event with whiteglove_part=1 BEFORE Sysprep reboots the box on Reseal.
// Constructed AFTER orchestrator.Start so EspAndHelloHost (created inside Start
// at step 14) and analyzerManager (created in the onIngressReady hook at step 13b)
// are both live. Disposed in the same finally block as the rest of the lifecycle.
WhiteGloveInventoryTrigger whiteGloveInventoryTrigger = null;
ConsoleCancelEventHandler cancelHandler = (s, e) =>
{
e.Cancel = true;
@@ -272,6 +281,23 @@ namespace AutopilotMonitor.Agent.V2.Runtime
shutdownComplete,
logger);
// Wire the WhiteGlove Part-1 inventory trigger. componentFactory.EspAndHelloHost
// is set inside CreateCollectorHosts (orchestrator.Start step 14, runs after
// the onIngressReady hook above), so by here both it and analyzerManager are
// live. The Action closes over the local `analyzerManager` variable; null-safe
// in case a race ever inverts the order.
if (componentFactory.EspAndHelloHost != null)
{
whiteGloveInventoryTrigger = new WhiteGloveInventoryTrigger(
host: componentFactory.EspAndHelloHost,
onTrigger: () => analyzerManager?.RunWhiteGlovePart1InventorySnapshot(),
logger: logger);
}
else
{
logger.Warning("WhiteGloveInventoryTrigger not wired — componentFactory.EspAndHelloHost is null after orchestrator.Start.");
}
// WhiteGlove Part-2 resume: EnrollmentOrchestrator.Start already posts
// SessionRecovered, which triggers HandleWhiteGlovePart1To2Bridge —
// that reducer effect emits whiteglove_resumed on the timeline. We
@@ -369,6 +395,12 @@ namespace AutopilotMonitor.Agent.V2.Runtime
}
finally
{
// Detach the WG-Part-1 inventory trigger before TerminationPipeline runs
// so any stragglers fired during shutdown don't reach a half-torn-down
// analyzerManager. Idempotent: Dispose unsubscribes via Interlocked guard.
try { whiteGloveInventoryTrigger?.Dispose(); }
catch (Exception ex) { logger.Warning($"WhiteGloveInventoryTrigger.Dispose threw: {ex.Message}"); }
TerminationPipeline.Run(
orchestrator: orchestrator,
authFailureTracker: auth.AuthFailureTracker,
@@ -1 +1 @@
460
477
@@ -1 +1 @@
1212
1219
@@ -0,0 +1,94 @@
using AutopilotMonitor.Shared.Models;
using Newtonsoft.Json;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// JSON round-trip tests for <see cref="VulnerabilityCorrelateEnvelope"/>. The producer
/// (<c>AzureQueueVulnerabilityCorrelateProducer</c>) serializes with Newtonsoft and the
/// queue worker (<c>VulnerabilityCorrelateQueueWorker</c>) deserializes the same shape —
/// pinning the contract here keeps the queue from silently breaking when fields are added.
/// </summary>
public class VulnerabilityCorrelateEnvelopeSerializationTests
{
private const string TenantId = "a1b2c3d4-e5f6-7890-abcd-ef1234567890";
private const string SessionId = "b2c3d4e5-f6a7-8901-bcde-f12345678901";
private static readonly DateTime EnqueuedAt =
new(2026, 4, 30, 9, 48, 58, DateTimeKind.Utc);
[Theory]
[InlineData(null)]
[InlineData(1)]
[InlineData(2)]
public void Envelope_round_trips_through_newtonsoft_for_all_white_glove_parts(int? whiteGlovePart)
{
var original = new VulnerabilityCorrelateEnvelope
{
EnvelopeVersion = "1",
TenantId = TenantId,
SessionId = SessionId,
WhiteGlovePart = whiteGlovePart,
Reason = "shutdown_inventory",
EnqueuedAt = EnqueuedAt,
};
var json = JsonConvert.SerializeObject(original);
var restored = JsonConvert.DeserializeObject<VulnerabilityCorrelateEnvelope>(json);
Assert.NotNull(restored);
Assert.Equal("1", restored!.EnvelopeVersion);
Assert.Equal(TenantId, restored.TenantId);
Assert.Equal(SessionId, restored.SessionId);
Assert.Equal(whiteGlovePart, restored.WhiteGlovePart);
Assert.Equal("shutdown_inventory", restored.Reason);
Assert.Equal(EnqueuedAt, restored.EnqueuedAt.ToUniversalTime());
}
[Fact]
public void Envelope_default_envelope_version_is_one()
{
var envelope = new VulnerabilityCorrelateEnvelope();
Assert.Equal("1", envelope.EnvelopeVersion);
}
[Fact]
public void Envelope_default_strings_are_empty_not_null()
{
var envelope = new VulnerabilityCorrelateEnvelope();
Assert.Equal(string.Empty, envelope.TenantId);
Assert.Equal(string.Empty, envelope.SessionId);
Assert.Equal(string.Empty, envelope.Reason);
}
[Fact]
public void Envelope_default_white_glove_part_is_null()
{
var envelope = new VulnerabilityCorrelateEnvelope();
Assert.Null(envelope.WhiteGlovePart);
}
[Fact]
public void Envelope_serializes_all_fields_on_wire()
{
// Pin the JSON wire-format so a property rename can't slip through unnoticed.
var envelope = new VulnerabilityCorrelateEnvelope
{
TenantId = TenantId,
SessionId = SessionId,
WhiteGlovePart = 1,
Reason = "whiteglove_part1_inventory",
EnqueuedAt = EnqueuedAt,
};
var json = JsonConvert.SerializeObject(envelope);
Assert.Contains("\"TenantId\"", json);
Assert.Contains("\"SessionId\"", json);
Assert.Contains("\"WhiteGlovePart\"", json);
Assert.Contains("\"Reason\"", json);
Assert.Contains("\"EnqueuedAt\"", json);
Assert.Contains("\"EnvelopeVersion\"", json);
Assert.Contains("whiteglove_part1_inventory", json);
}
}
@@ -0,0 +1,244 @@
using AutopilotMonitor.Functions.Services.Vulnerability;
using Newtonsoft.Json.Linq;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Unit tests for the static <c>ContainsUserEnrollmentFindings</c> helper used by
/// <see cref="VulnerabilityCorrelateHandler"/> to detect out-of-order delivery —
/// specifically a Part-1 queue message arriving AFTER a Part-2 message has already
/// stored the combined / Part-2-only report. In that case the Part-1 handler skips
/// its store to avoid clobbering the richer report.
/// <para>
/// The helper has to handle both shapes <see cref="EnrollmentEvent.Data"/>
/// deserialization can produce: native <see cref="Dictionary{TKey, TValue}"/>
/// values and Newtonsoft <see cref="JObject"/> / <see cref="JArray"/> values that
/// come back when the report is round-tripped through stored JSON.
/// </para>
/// </summary>
public class VulnerabilityCorrelateHandlerSkipTests
{
[Fact]
public void ContainsUserEnrollmentFindings_returns_false_for_empty_report()
{
var report = new Dictionary<string, object>();
Assert.False(VulnerabilityCorrelateHandler.ContainsUserEnrollmentFindings(report));
}
[Fact]
public void ContainsUserEnrollmentFindings_returns_false_when_findings_null()
{
var report = new Dictionary<string, object> { ["findings"] = null! };
Assert.False(VulnerabilityCorrelateHandler.ContainsUserEnrollmentFindings(report));
}
[Fact]
public void ContainsUserEnrollmentFindings_returns_false_for_part1_only_native_report()
{
var report = new Dictionary<string, object>
{
["findings"] = new List<Dictionary<string, object>>
{
new() { ["softwareName"] = "Edge", ["phase"] = "device_setup" },
new() { ["softwareName"] = "Office", ["phase"] = "device_setup" },
}
};
Assert.False(VulnerabilityCorrelateHandler.ContainsUserEnrollmentFindings(report));
}
[Fact]
public void ContainsUserEnrollmentFindings_returns_true_when_native_report_has_user_enrollment_finding()
{
var report = new Dictionary<string, object>
{
["findings"] = new List<Dictionary<string, object>>
{
new() { ["softwareName"] = "Edge", ["phase"] = "device_setup" },
new() { ["softwareName"] = "Office", ["phase"] = "user_enrollment" },
}
};
Assert.True(VulnerabilityCorrelateHandler.ContainsUserEnrollmentFindings(report));
}
[Fact]
public void ContainsUserEnrollmentFindings_returns_true_for_jarray_round_tripped_report()
{
// Simulates the shape we get back from the VulnerabilityReports table after a
// Newtonsoft round-trip of a stored Part-2 report.
var jarr = new JArray
{
JObject.FromObject(new { softwareName = "Edge", phase = "device_setup" }),
JObject.FromObject(new { softwareName = "Office", phase = "user_enrollment" }),
};
var report = new Dictionary<string, object> { ["findings"] = jarr };
Assert.True(VulnerabilityCorrelateHandler.ContainsUserEnrollmentFindings(report));
}
[Fact]
public void ContainsUserEnrollmentFindings_returns_false_for_jarray_with_only_device_setup()
{
var jarr = new JArray
{
JObject.FromObject(new { softwareName = "Edge", phase = "device_setup" }),
};
var report = new Dictionary<string, object> { ["findings"] = jarr };
Assert.False(VulnerabilityCorrelateHandler.ContainsUserEnrollmentFindings(report));
}
[Fact]
public void ContainsUserEnrollmentFindings_returns_false_for_findings_without_phase_tag()
{
// Single-phase (non-WG) reports have no phase tag at all.
var report = new Dictionary<string, object>
{
["findings"] = new List<Dictionary<string, object>>
{
new() { ["softwareName"] = "Edge" }, // no phase
}
};
Assert.False(VulnerabilityCorrelateHandler.ContainsUserEnrollmentFindings(report));
}
[Fact]
public void ContainsUserEnrollmentFindings_handles_loosely_typed_object_list()
{
// Path 3: List<object> (some custom deserializers produce this).
var report = new Dictionary<string, object>
{
["findings"] = new List<object>
{
new Dictionary<string, object> { ["phase"] = "device_setup" },
new Dictionary<string, object> { ["phase"] = "user_enrollment" },
}
};
Assert.True(VulnerabilityCorrelateHandler.ContainsUserEnrollmentFindings(report));
}
// -------------------------------------------------------------------------
// Round-3 — phase=="both" + scan_summary.phase_contributions detection
//
// MergeReports in VulnerabilityCorrelationService collapses identical
// Part-1+Part-2 findings into a single entry tagged phase=="both", so a
// fully-collapsed merged report can have ZERO findings with phase=="user_enrollment"
// and yet still must not be overwritten by a late Part-1 retry. The handler
// also has to consult scan_summary.phase_contributions to handle the symmetric
// shape coming back from Newtonsoft round-trip storage.
// -------------------------------------------------------------------------
[Fact]
public void ContainsUserEnrollmentFindings_returns_true_when_finding_phase_is_both()
{
var report = new Dictionary<string, object>
{
["findings"] = new List<Dictionary<string, object>>
{
new() { ["softwareName"] = "Edge", ["phase"] = "both" },
}
};
Assert.True(VulnerabilityCorrelateHandler.ContainsUserEnrollmentFindings(report));
}
[Fact]
public void ContainsUserEnrollmentFindings_returns_true_when_phase_contributions_user_enrollment_positive()
{
// Even if all findings have phase=="device_setup" (or the findings list is missing),
// a non-zero user_enrollment contribution in scan_summary is conclusive — Part 2
// already participated in this report.
var report = new Dictionary<string, object>
{
["scan_summary"] = new Dictionary<string, object>
{
["phase_contributions"] = new Dictionary<string, object>
{
["device_setup"] = 5,
["user_enrollment"] = 3,
}
},
["findings"] = new List<Dictionary<string, object>>
{
new() { ["softwareName"] = "Edge", ["phase"] = "device_setup" },
}
};
Assert.True(VulnerabilityCorrelateHandler.ContainsUserEnrollmentFindings(report));
}
[Fact]
public void ContainsUserEnrollmentFindings_returns_true_when_phase_contributions_arrives_as_long()
{
// Newtonsoft serializes integers as long by default — must round-trip.
var report = new Dictionary<string, object>
{
["scan_summary"] = new Dictionary<string, object>
{
["phase_contributions"] = new Dictionary<string, object>
{
["user_enrollment"] = 1L,
}
}
};
Assert.True(VulnerabilityCorrelateHandler.ContainsUserEnrollmentFindings(report));
}
[Fact]
public void ContainsUserEnrollmentFindings_returns_false_when_phase_contributions_user_enrollment_zero()
{
var report = new Dictionary<string, object>
{
["scan_summary"] = new Dictionary<string, object>
{
["phase_contributions"] = new Dictionary<string, object>
{
["device_setup"] = 5,
["user_enrollment"] = 0,
}
},
["findings"] = new List<Dictionary<string, object>>
{
new() { ["softwareName"] = "Edge", ["phase"] = "device_setup" },
}
};
Assert.False(VulnerabilityCorrelateHandler.ContainsUserEnrollmentFindings(report));
}
[Fact]
public void ContainsUserEnrollmentFindings_handles_phase_contributions_in_jobject_summary()
{
// Newtonsoft round-trip path: scan_summary deserializes back into a JObject and
// phase_contributions inside as a JObject too (with JValue children).
var summary = JObject.FromObject(new
{
phase_contributions = new
{
device_setup = 1,
user_enrollment = 4,
}
});
var report = new Dictionary<string, object> { ["scan_summary"] = summary };
Assert.True(VulnerabilityCorrelateHandler.ContainsUserEnrollmentFindings(report));
}
[Fact]
public void ContainsUserEnrollmentFindings_returns_true_for_jobject_finding_phase_both()
{
// Fully-collapsed merged report rebuilt from JSON storage — every finding sits
// under phase=="both" and there is no per-phase contributor list left.
var jarr = new JArray
{
JObject.FromObject(new { softwareName = "Edge", phase = "both" }),
};
var report = new Dictionary<string, object> { ["findings"] = jarr };
Assert.True(VulnerabilityCorrelateHandler.ContainsUserEnrollmentFindings(report));
}
}
@@ -1,4 +1,5 @@
using AutopilotMonitor.Functions.Services.Vulnerability;
using Newtonsoft.Json;
using static AutopilotMonitor.Functions.Services.Vulnerability.VulnerabilityCorrelationService;
namespace AutopilotMonitor.Functions.Tests;
@@ -718,4 +719,133 @@ public class VulnerabilityCorrelationServiceTests
Assert.Equal(1, contributions!["device_setup"]);
Assert.Equal(1, contributions!["user_enrollment"]);
}
// -----------------------------------------------------------------------
// MergeReports — Newtonsoft round-trip safety (round-4 fix)
// -----------------------------------------------------------------------
//
// GetVulnerabilityReportAsync deserializes stored reports via
// JsonConvert.DeserializeObject<Dictionary<string, object>>, which produces a
// top-level Dictionary<string, object> but JObject/JArray/JValue children for
// every nested level. Without the NormalizeForMerge step, the cast-based
// helpers in MergeReports (ExtractFindings, ExtractUnmatchedSoftware,
// scan_summary "as Dictionary", vulnerabilities "is Dictionary<string, object>")
// return empty results and silently drop the existing report's findings on
// merge — Part-1 data is lost when Part-2 ran in normal order.
//
// These tests pin the fix by feeding MergeReports with a JObject-shaped
// existingReport (the actual shape from storage) and asserting that the
// existing finding survives the merge.
private static Dictionary<string, object> RoundTripThroughNewtonsoft(Dictionary<string, object> report)
{
var json = JsonConvert.SerializeObject(report);
return JsonConvert.DeserializeObject<Dictionary<string, object>>(json)!;
}
[Fact]
public void MergeReports_PreservesPart1Findings_WhenExistingReportIsNewtonsoftRoundTripped()
{
// Arrange: Part 1 stored, then re-loaded → JObject/JArray children.
var part1Native = MakeReport(new List<Dictionary<string, object>>
{
MakeFinding("Trellix Agent", "5.8.0", "CVE-2024-0213", 8.2, "HIGH")
});
var part1RoundTripped = RoundTripThroughNewtonsoft(part1Native);
var part2 = MakeReport(new List<Dictionary<string, object>>
{
MakeFinding("Edge Update", "1.3.225.7", "CVE-2022-4135", 9.6, "CRITICAL", isKev: true)
});
// Act
var merged = VulnerabilityCorrelationService.MergeReports(part1RoundTripped, part2);
// Assert — both findings must be in the merged report; without the fix only Part-2 survives.
var findings = merged["findings"] as List<Dictionary<string, object>>;
Assert.NotNull(findings);
Assert.Equal(2, findings!.Count);
Assert.Contains(findings, f => f["softwareName"]?.ToString() == "Trellix Agent");
Assert.Contains(findings, f => f["softwareName"]?.ToString() == "Edge Update");
}
[Fact]
public void MergeReports_DeduplicatesSameCve_FromRoundTrippedExistingReport()
{
// Same CVE in both phases must collapse to phase=="both" even when existing came from storage.
var part1Native = MakeReport(new List<Dictionary<string, object>>
{
MakeFinding("Trellix Agent", "5.8.0", "CVE-2024-0213", 8.2, "HIGH")
});
var part1RoundTripped = RoundTripThroughNewtonsoft(part1Native);
var part2 = MakeReport(new List<Dictionary<string, object>>
{
MakeFinding("Trellix Agent", "5.8.0", "CVE-2024-0213", 8.2, "HIGH")
});
var merged = VulnerabilityCorrelationService.MergeReports(part1RoundTripped, part2);
var findings = merged["findings"] as List<Dictionary<string, object>>;
Assert.NotNull(findings);
Assert.Single(findings!);
Assert.Equal("both", findings![0]["phase"]);
}
[Fact]
public void MergeReports_PreservesScanSummaryCounts_FromRoundTrippedExistingReport()
{
// existing scan_summary must contribute its total_software_scanned / matched_to_cpe
// even when re-loaded as a JObject (without the fix the "as Dictionary<...>" cast
// would null-out and existingScanned/existingMatched would be 0).
var part1Native = MakeReport(
new List<Dictionary<string, object>> { MakeFinding("A", "1.0", "CVE-2024-0001", 7.0, "HIGH") },
scanned: 30, matched: 15);
var part1RoundTripped = RoundTripThroughNewtonsoft(part1Native);
var part2 = MakeReport(
new List<Dictionary<string, object>> { MakeFinding("B", "2.0", "CVE-2024-0002", 6.0, "MEDIUM") },
scanned: 12, matched: 6);
var merged = VulnerabilityCorrelationService.MergeReports(part1RoundTripped, part2);
var summary = merged["scan_summary"] as Dictionary<string, object>;
Assert.NotNull(summary);
Assert.Equal(42, Convert.ToInt32(summary!["total_software_scanned"])); // 30 + 12
Assert.Equal(21, Convert.ToInt32(summary!["matched_to_cpe"])); // 15 + 6
}
[Fact]
public void NormalizeForMerge_recursively_unwraps_nested_jtokens()
{
// Direct unit test of the helper — keeps regressions on the recursive walk
// local to this file rather than only catchable through the merge integration.
var native = new Dictionary<string, object>
{
["scan_summary"] = new Dictionary<string, object>
{
["phase_contributions"] = new Dictionary<string, object>
{
["device_setup"] = 3,
}
},
["findings"] = new List<Dictionary<string, object>>
{
new() { ["softwareName"] = "Edge", ["phase"] = "device_setup" }
}
};
var roundTripped = RoundTripThroughNewtonsoft(native);
var normalized = VulnerabilityCorrelationService.NormalizeForMerge(roundTripped);
Assert.IsType<Dictionary<string, object>>(normalized["scan_summary"]);
var summary = (Dictionary<string, object>)normalized["scan_summary"];
Assert.IsType<Dictionary<string, object>>(summary["phase_contributions"]);
Assert.IsType<List<object>>(normalized["findings"]);
var findings = (List<object>)normalized["findings"];
Assert.IsType<Dictionary<string, object>>(findings[0]);
var firstFinding = (Dictionary<string, object>)findings[0];
Assert.Equal("Edge", firstFinding["softwareName"]);
}
}
@@ -0,0 +1,237 @@
using AutopilotMonitor.Functions.Services.Vulnerability;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Unit tests for <see cref="VulnerabilityInventoryLoader"/>. Verifies the shutdown-event
/// filter, chunked-merge, and the JArray vs <c>List&lt;object&gt;</c> dual-representation
/// handling — covers the single source of truth shared by the queue worker and the manual
/// rescan endpoint.
/// </summary>
public class VulnerabilityInventoryLoaderTests
{
private const string TenantId = "tenant-1";
private const string SessionId = "session-1";
[Fact]
public async Task Loader_ignores_baseline_inventory_keeps_shutdown()
{
var events = new List<EnrollmentEvent>
{
BuildInventoryEvent("startup", new[] { ("Edge", "100.0.0.0") }),
BuildInventoryEvent("shutdown", new[] { ("Edge", "100.0.0.0"), ("Cisco AnyConnect", "5.1.5.65") }),
};
var sut = BuildLoader(events);
// Default queue path: no WG part, untagged events only.
var result = await sut.LoadShutdownInventoryAsync(TenantId, SessionId);
Assert.Equal(2, result.Count);
Assert.Contains(result, item => item.GetValueOrDefault("displayName")?.ToString() == "Cisco AnyConnect");
}
[Fact]
public async Task Loader_concatenates_chunked_shutdown_inventory()
{
var events = new List<EnrollmentEvent>
{
BuildInventoryEvent("shutdown", new[] { ("Edge", "100.0.0.0") }, chunkIndex: 0),
BuildInventoryEvent("shutdown", new[] { ("Cisco AnyConnect", "5.1.5.65") }, chunkIndex: 1),
BuildInventoryEvent("shutdown", new[] { ("CrowdStrike", "7.34.0.0") }, chunkIndex: 2),
};
var sut = BuildLoader(events);
var result = await sut.LoadShutdownInventoryAsync(TenantId, SessionId);
Assert.Equal(3, result.Count);
}
[Fact]
public async Task Loader_returns_empty_when_no_shutdown_events()
{
var events = new List<EnrollmentEvent>
{
BuildInventoryEvent("startup", new[] { ("Edge", "100.0.0.0") }),
};
var sut = BuildLoader(events);
var result = await sut.LoadShutdownInventoryAsync(TenantId, SessionId);
Assert.Empty(result);
}
[Fact]
public async Task Loader_returns_empty_when_no_inventory_events_at_all()
{
var events = new List<EnrollmentEvent>
{
new() { EventType = "agent_started", Data = new Dictionary<string, object>() },
new() { EventType = "phase_transition", Data = new Dictionary<string, object>() },
};
var sut = BuildLoader(events);
var result = await sut.LoadShutdownInventoryAsync(TenantId, SessionId);
Assert.Empty(result);
}
[Fact]
public async Task Loader_handles_missing_data_gracefully()
{
var events = new List<EnrollmentEvent>
{
new() { EventType = Constants.EventTypes.SoftwareInventoryAnalysis, Data = null! },
new() { EventType = Constants.EventTypes.SoftwareInventoryAnalysis, Data = new Dictionary<string, object>() },
BuildInventoryEvent("shutdown", new[] { ("Edge", "100.0.0.0") }),
};
var sut = BuildLoader(events);
var result = await sut.LoadShutdownInventoryAsync(TenantId, SessionId);
Assert.Single(result);
}
// -------------------------------------------------------------------------
// Phase filtering — guards against cross-WG-phase contamination
// -------------------------------------------------------------------------
[Fact]
public async Task Loader_with_part1_filter_only_returns_part1_events()
{
var events = new List<EnrollmentEvent>
{
BuildInventoryEvent("shutdown", new[] { ("Edge", "100.0.0.0") }, whiteGlovePart: 1),
BuildInventoryEvent("shutdown", new[] { ("Office", "16.0") }, whiteGlovePart: 2),
BuildInventoryEvent("shutdown", new[] { ("AnyConnect", "5.1") }), // untagged
};
var sut = BuildLoader(events);
var result = await sut.LoadShutdownInventoryAsync(TenantId, SessionId, whiteGlovePart: 1);
Assert.Single(result);
Assert.Equal("Edge", result[0].GetValueOrDefault("displayName"));
}
[Fact]
public async Task Loader_with_part2_filter_only_returns_part2_events()
{
var events = new List<EnrollmentEvent>
{
BuildInventoryEvent("shutdown", new[] { ("Edge", "100.0.0.0") }, whiteGlovePart: 1),
BuildInventoryEvent("shutdown", new[] { ("Office", "16.0") }, whiteGlovePart: 2),
};
var sut = BuildLoader(events);
var result = await sut.LoadShutdownInventoryAsync(TenantId, SessionId, whiteGlovePart: 2);
Assert.Single(result);
Assert.Equal("Office", result[0].GetValueOrDefault("displayName"));
}
[Fact]
public async Task Loader_no_part_filter_returns_only_untagged_events()
{
// Non-WG queue path: caller passes whiteGlovePart=null.
// Must NOT pick up Part-1 events that happen to share the session id
// (defensive — there's no path today that produces both, but the filter
// keeps the contract symmetric with the WG paths).
var events = new List<EnrollmentEvent>
{
BuildInventoryEvent("shutdown", new[] { ("Edge", "100.0.0.0") }, whiteGlovePart: 1),
BuildInventoryEvent("shutdown", new[] { ("AnyConnect", "5.1") }), // untagged
};
var sut = BuildLoader(events);
var result = await sut.LoadShutdownInventoryAsync(TenantId, SessionId);
Assert.Single(result);
Assert.Equal("AnyConnect", result[0].GetValueOrDefault("displayName"));
}
[Fact]
public async Task Loader_includeAllPhases_overrides_filter_for_manual_rescan()
{
var events = new List<EnrollmentEvent>
{
BuildInventoryEvent("shutdown", new[] { ("Edge", "100.0.0.0") }, whiteGlovePart: 1),
BuildInventoryEvent("shutdown", new[] { ("Office", "16.0") }, whiteGlovePart: 2),
BuildInventoryEvent("shutdown", new[] { ("AnyConnect", "5.1") }), // untagged
};
var sut = BuildLoader(events);
var result = await sut.LoadShutdownInventoryAsync(
TenantId, SessionId, whiteGlovePart: null, includeAllPhases: true);
Assert.Equal(3, result.Count);
}
[Theory]
[InlineData(1)] // int
[InlineData(2)]
public async Task Loader_phase_filter_handles_long_serialized_value(int requestedPart)
{
// Newtonsoft default for JSON numeric → long; loader must compare leniently.
var events = new List<EnrollmentEvent>
{
BuildInventoryEvent("shutdown", new[] { ("Edge", "100.0.0.0") }, whiteGlovePart: requestedPart, asLong: true),
};
var sut = BuildLoader(events);
var result = await sut.LoadShutdownInventoryAsync(TenantId, SessionId, whiteGlovePart: requestedPart);
Assert.Single(result);
}
private static VulnerabilityInventoryLoader BuildLoader(List<EnrollmentEvent> events)
{
var sessionRepo = new Mock<ISessionRepository>();
sessionRepo
.Setup(r => r.GetSessionEventsAsync(TenantId, SessionId, It.IsAny<int>()))
.ReturnsAsync(events);
return new VulnerabilityInventoryLoader(
sessionRepo.Object,
NullLogger<VulnerabilityInventoryLoader>.Instance);
}
private static EnrollmentEvent BuildInventoryEvent(
string triggeredAt,
(string DisplayName, string DisplayVersion)[] items,
int chunkIndex = 0,
int? whiteGlovePart = null,
bool asLong = false)
{
var inventory = items
.Select(i => (object)new Dictionary<string, object>
{
["displayName"] = i.DisplayName,
["displayVersion"] = i.DisplayVersion,
})
.ToList();
var data = new Dictionary<string, object>
{
["triggered_at"] = triggeredAt,
["chunk_index"] = chunkIndex,
["inventory"] = inventory,
};
if (whiteGlovePart.HasValue)
{
data["whiteglove_part"] = asLong ? (object)(long)whiteGlovePart.Value : whiteGlovePart.Value;
}
return new EnrollmentEvent
{
TenantId = TenantId,
SessionId = SessionId,
EventType = Constants.EventTypes.SoftwareInventoryAnalysis,
Data = data,
};
}
}
@@ -24,17 +24,20 @@ namespace AutopilotMonitor.Functions.Functions.Vulnerability
private readonly IVulnerabilityRepository _vulnRepo;
private readonly ISessionRepository _sessionRepo;
private readonly VulnerabilityCorrelationService _correlationService;
private readonly IVulnerabilityInventoryLoader _inventoryLoader;
public GetVulnerabilityReportFunction(
ILogger<GetVulnerabilityReportFunction> logger,
IVulnerabilityRepository vulnRepo,
ISessionRepository sessionRepo,
VulnerabilityCorrelationService correlationService)
VulnerabilityCorrelationService correlationService,
IVulnerabilityInventoryLoader inventoryLoader)
{
_logger = logger;
_vulnRepo = vulnRepo;
_sessionRepo = sessionRepo;
_correlationService = correlationService;
_inventoryLoader = inventoryLoader;
}
[Function("GetVulnerabilityReport")]
@@ -69,8 +72,12 @@ namespace AutopilotMonitor.Functions.Functions.Vulnerability
_logger.LogInformation("Vulnerability re-scan requested for session {SessionId} by {User} (query: {Query})",
sessionId, userIdentifier, req.Url.Query);
// Load inventory data from the stored software_inventory_analysis events
var inventoryItems = await LoadInventoryItemsFromEventsAsync(effectiveTenantId, sessionId);
// Load inventory data from the stored software_inventory_analysis events.
// Manual rescan rescans EVERY shutdown inventory regardless of WhiteGlove
// phase tag — the user clicked "Rescan", they expect the full set, not a
// phase-restricted subset. Queue path uses the phase-filtered overload.
var inventoryItems = await _inventoryLoader.LoadShutdownInventoryAsync(
effectiveTenantId, sessionId, includeAllPhases: true);
_logger.LogInformation("Vulnerability re-scan: loaded {Count} inventory items for session {SessionId}", inventoryItems.Count, sessionId);
if (inventoryItems.Count == 0)
{
@@ -81,7 +88,7 @@ namespace AutopilotMonitor.Functions.Functions.Vulnerability
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(60));
var freshReport = await _correlationService.CorrelateAsync(
sessionId, effectiveTenantId, inventoryItems, cts.Token);
sessionId, effectiveTenantId, inventoryItems.ToList(), cts.Token);
_logger.LogInformation("Vulnerability re-scan: CorrelateAsync returned {Result} for session {SessionId}",
freshReport != null ? "report with findings" : "null (no findings)", sessionId);
@@ -134,69 +141,5 @@ namespace AutopilotMonitor.Functions.Functions.Vulnerability
return response;
}
/// <summary>
/// Loads inventory items from the software_inventory_analysis shutdown events stored in the Events table.
/// Merges chunked events and returns the full inventory list for re-correlation.
/// </summary>
private async Task<List<Dictionary<string, object>>> LoadInventoryItemsFromEventsAsync(string tenantId, string sessionId)
{
var items = new List<Dictionary<string, object>>();
// Fetch all session events and filter by EventType in code
var allEvents = await _sessionRepo.GetSessionEventsAsync(tenantId, sessionId);
var inventoryEvents = allEvents
.Where(e => e.EventType == Constants.EventTypes.SoftwareInventoryAnalysis);
foreach (var evt in inventoryEvents)
{
// The Data dictionary contains the event payload
if (evt.Data == null || evt.Data.Count == 0) continue;
try
{
// Only use shutdown events (final inventory, not baseline)
var triggeredAt = evt.Data.ContainsKey("triggered_at") ? evt.Data["triggered_at"]?.ToString() : null;
if (triggeredAt != "shutdown") continue;
if (!evt.Data.ContainsKey("inventory")) continue;
var inventoryData = evt.Data["inventory"];
// DeserializeEventData converts JArrays to List<object> via ConvertJTokenToNative,
// so handle both the native List<object> case and the raw JArray case.
if (inventoryData is IList<object> nativeList)
{
foreach (var entry in nativeList)
{
if (entry is Dictionary<string, object> dict)
{
items.Add(dict);
}
else if (entry is IDictionary<string, object> idict)
{
items.Add(new Dictionary<string, object>(idict));
}
}
}
else if (inventoryData is Newtonsoft.Json.Linq.JArray inventoryArray)
{
foreach (var item in inventoryArray)
{
if (item is Newtonsoft.Json.Linq.JObject obj)
{
items.Add(obj.ToObject<Dictionary<string, object>>() ?? new Dictionary<string, object>());
}
}
}
}
catch (Exception ex)
{
_logger.LogDebug(ex, "Failed to parse inventory event data for re-scan");
}
}
_logger.LogInformation("Loaded {Count} inventory items from Events table for session {SessionId}", items.Count, sessionId);
return items;
}
}
}
@@ -196,6 +196,21 @@ builder.Services.AddSingleton<
builder.Services.AddHostedService<
AutopilotMonitor.Functions.Services.Analyze.AnalyzeOnEnrollmentEndQueueWorker>();
// Vulnerability-correlate fan-out triggered by the shutdown software_inventory_analysis
// event. Replaces the in-function fire-and-forget Task.Run inside EventIngestProcessor.
// Same producer + worker pattern as the analyze queue. Inventory loader is DI-shared with
// the manual rescan endpoint (GetVulnerabilityReportFunction ?rescan=true).
builder.Services.AddSingleton<
AutopilotMonitor.Functions.Services.Vulnerability.IVulnerabilityCorrelateProducer,
AutopilotMonitor.Functions.Services.Vulnerability.AzureQueueVulnerabilityCorrelateProducer>();
builder.Services.AddSingleton<
AutopilotMonitor.Functions.Services.Vulnerability.IVulnerabilityInventoryLoader,
AutopilotMonitor.Functions.Services.Vulnerability.VulnerabilityInventoryLoader>();
builder.Services.AddSingleton<
AutopilotMonitor.Functions.Services.Vulnerability.VulnerabilityCorrelateHandler>();
builder.Services.AddHostedService<
AutopilotMonitor.Functions.Services.Vulnerability.VulnerabilityCorrelateQueueWorker>();
// Programmatic SignalR push for background tasks (rule engine, vulnerability correlation)
builder.Services.AddSingleton<SignalRNotificationService>();
@@ -49,50 +49,44 @@ namespace AutopilotMonitor.Functions.Services
private readonly ISessionRepository _sessionRepo;
private readonly IMetricsRepository _metricsRepo;
private readonly IRuleRepository _ruleRepo;
private readonly IVulnerabilityRepository _vulnRepo;
private readonly TenantConfigurationService _configService;
private readonly AnalyzeRuleService _analyzeRuleService;
private readonly WebhookNotificationService _webhookNotificationService;
private readonly VulnerabilityCorrelationService _vulnerabilityCorrelation;
private readonly AdminConfigurationService _adminConfigService;
private readonly SignalRNotificationService _signalRNotification;
private readonly OpsEventService _opsEventService;
private readonly SlaBreachEvaluationService _slaBreachService;
private readonly TelemetryClient _telemetryClient;
private readonly AutopilotMonitor.Functions.Services.Analyze.IAnalyzeOnEnrollmentEndProducer _analyzeProducer;
private readonly IVulnerabilityCorrelateProducer _vulnProducer;
public EventIngestProcessor(
ILogger<EventIngestProcessor> logger,
ISessionRepository sessionRepo,
IMetricsRepository metricsRepo,
IRuleRepository ruleRepo,
IVulnerabilityRepository vulnRepo,
TenantConfigurationService configService,
AnalyzeRuleService analyzeRuleService,
WebhookNotificationService webhookNotificationService,
VulnerabilityCorrelationService vulnerabilityCorrelation,
AdminConfigurationService adminConfigService,
SignalRNotificationService signalRNotification,
OpsEventService opsEventService,
SlaBreachEvaluationService slaBreachService,
TelemetryClient telemetryClient,
AutopilotMonitor.Functions.Services.Analyze.IAnalyzeOnEnrollmentEndProducer analyzeProducer)
AutopilotMonitor.Functions.Services.Analyze.IAnalyzeOnEnrollmentEndProducer analyzeProducer,
IVulnerabilityCorrelateProducer vulnProducer)
{
_logger = logger;
_sessionRepo = sessionRepo;
_metricsRepo = metricsRepo;
_ruleRepo = ruleRepo;
_vulnRepo = vulnRepo;
_configService = configService;
_analyzeRuleService = analyzeRuleService;
_webhookNotificationService = webhookNotificationService;
_vulnerabilityCorrelation = vulnerabilityCorrelation;
_adminConfigService = adminConfigService;
_signalRNotification = signalRNotification;
_opsEventService = opsEventService;
_slaBreachService = slaBreachService;
_telemetryClient = telemetryClient;
_analyzeProducer = analyzeProducer;
_vulnProducer = vulnProducer;
}
/// <summary>
@@ -234,146 +228,40 @@ namespace AutopilotMonitor.Functions.Services
if (shutdownInventoryDetected)
{
var capturedSessionId = request.SessionId;
var capturedTenantId = request.TenantId;
var capturedPrefix = sessionPrefix;
var allInventoryItems = new List<Dictionary<string, object>>();
var inventoryChunks = storedEvents
// Find the first shutdown chunk to extract the optional WhiteGlove phase tag.
// The handler reloads the full inventory from the Events table itself — this
// is idempotent against queue re-deliveries and means we don't need to capture
// the items here.
var firstShutdownChunk = storedEvents
.Where(e => e.EventType == Shared.Constants.EventTypes.SoftwareInventoryAnalysis &&
e.Data != null &&
e.Data.ContainsKey("triggered_at") &&
e.Data["triggered_at"]?.ToString() == "shutdown" &&
e.Data.ContainsKey("inventory"))
e.Data["triggered_at"]?.ToString() == "shutdown")
.OrderBy(e => Convert.ToInt32(e.Data!.GetValueOrDefault("chunk_index", 0)))
.ToList();
foreach (var chunk in inventoryChunks)
{
if (chunk.Data!["inventory"] is System.Collections.IEnumerable items)
{
foreach (var item in items)
{
if (item is Dictionary<string, object> dict)
allInventoryItems.Add(dict);
}
}
}
.FirstOrDefault();
int? whiteGlovePart = null;
var firstShutdownChunk = inventoryChunks.FirstOrDefault();
if (firstShutdownChunk?.Data != null &&
firstShutdownChunk.Data.TryGetValue("whiteglove_part", out var wgPartObj))
{
whiteGlovePart = Convert.ToInt32(wgPartObj);
}
if (allInventoryItems.Count > 0)
// Hand off to the vulnerability-correlate queue. Replaces the previous
// fire-and-forget Task.Run that could be killed mid-flight by Azure Functions
// scale-in (HTTP 200 returned → worker unloaded → vulnerability report never
// persisted). Producer is fail-soft — a missed enqueue degrades to "no report"
// and the user can manually rescan via the UI.
await _vulnProducer.EnqueueAsync(new VulnerabilityCorrelateEnvelope
{
var capturedWhiteGlovePart = whiteGlovePart;
_ = Task.Run(async () =>
{
try
{
var adminConfig = await _adminConfigService.GetConfigurationAsync();
if (adminConfig?.VulnerabilityCorrelationEnabled != true)
return;
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(60));
var reportData = await _vulnerabilityCorrelation.CorrelateAsync(
capturedSessionId, capturedTenantId, allInventoryItems, cts.Token);
if (reportData != null)
{
var phaseLabel = capturedWhiteGlovePart == 1 ? "device_setup"
: capturedWhiteGlovePart == 2 ? "user_enrollment"
: (string?)null;
if (phaseLabel != null && reportData.ContainsKey("findings")
&& reportData["findings"] is List<Dictionary<string, object>> tagFindings)
{
foreach (var f in tagFindings)
f.TryAdd("phase", phaseLabel);
}
if (capturedWhiteGlovePart == 2)
{
try
{
var existingReport = await _vulnRepo.GetVulnerabilityReportAsync(
capturedTenantId, capturedSessionId);
if (existingReport != null)
{
reportData = VulnerabilityCorrelationService.MergeReports(
existingReport, reportData,
existingPhaseLabel: "device_setup",
newPhaseLabel: "user_enrollment");
_logger.LogInformation(
"{Prefix} WhiteGlove Part 2: merged vulnerability report with Part 1 findings",
capturedPrefix);
}
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"{Prefix} Failed to load Part 1 report for merge (storing Part 2 standalone)",
capturedPrefix);
}
}
await _vulnRepo.StoreVulnerabilityReportAsync(
capturedTenantId, capturedSessionId, reportData);
_logger.LogInformation(
"{Prefix} Vulnerability correlation complete (async, whiteGlovePart={Part})",
capturedPrefix, capturedWhiteGlovePart?.ToString() ?? "none");
// Synthetic `vulnerability_report` event so analyze rules can
// match against `scan_summary.*` fields. The full report stays in
// the VulnerabilityReports table — this event carries only the
// summary slice that rule conditions need.
await EmitVulnerabilityReportEventAsync(
capturedTenantId, capturedSessionId, capturedPrefix, reportData, phaseLabel);
// Re-evaluate analyze rules now that the synth event is in the
// table — vulnerability-driven rules (ANALYZE-ID-003) need this
// synth event as evidence. Enqueued (not awaited) onto the same
// analyze-on-enrollment-end queue as the primary trigger; the
// worker runs RuleEngine with dedup so already-stored results
// from the primary run are not duplicated.
await _analyzeProducer.EnqueueAsync(new AutopilotMonitor.Shared.Models.AnalyzeOnEnrollmentEndEnvelope
{
TenantId = capturedTenantId,
SessionId = capturedSessionId,
Reason = AutopilotMonitor.Functions.Services.Analyze.AnalyzeOnEnrollmentEndHandler.ReasonVulnerabilityCorrelated,
EnqueuedAt = DateTime.UtcNow,
});
var findings = reportData.ContainsKey("findings")
? reportData["findings"] as List<Dictionary<string, object>>
: null;
if (findings != null && findings.Count > 0)
{
_ = _sessionRepo.UpsertCveIndexEntriesAsync(capturedTenantId, capturedSessionId, findings)
.ContinueWith(t => _logger.LogWarning(t.Exception?.InnerException,
"CveIndex update failed (non-fatal)"), TaskContinuationOptions.OnlyOnFaulted);
}
var overallRisk = reportData.ContainsKey("scan_summary")
&& reportData["scan_summary"] is Dictionary<string, object> summary
&& summary.ContainsKey("overall_risk")
? summary["overall_risk"]?.ToString() ?? "unknown"
: "unknown";
await _signalRNotification.NotifyVulnerabilityReportAvailableAsync(
capturedTenantId, capturedSessionId, overallRisk);
}
}
catch (Exception ex)
{
_logger.LogWarning(ex, "{Prefix} Vulnerability correlation failed (async, non-fatal)", capturedPrefix);
}
});
}
TenantId = request.TenantId,
SessionId = request.SessionId,
WhiteGlovePart = whiteGlovePart,
Reason = whiteGlovePart == 1
? VulnerabilityCorrelateHandler.ReasonWhiteGlovePart1Inventory
: VulnerabilityCorrelateHandler.ReasonShutdownInventory,
EnqueuedAt = DateTime.UtcNow,
});
}
_ = _metricsRepo.IncrementPlatformStatAsync("TotalEventsProcessed", processedCount)
@@ -483,67 +371,6 @@ namespace AutopilotMonitor.Functions.Services
};
}
/// <summary>
/// Stores a synthetic <c>vulnerability_report</c> event after a successful scan so analyze
/// rules can fire on the same event-stream surface as agent-emitted events. Carries only
/// <c>scan_summary</c> + <c>reportAvailable</c> + <c>phase</c>; the full per-CVE list lives
/// in the <c>VulnerabilityReports</c> table (read separately by the report API).
/// </summary>
private async Task EmitVulnerabilityReportEventAsync(
string tenantId,
string sessionId,
string sessionPrefix,
Dictionary<string, object> reportData,
string? phaseLabel)
{
try
{
var scanSummary = reportData.TryGetValue("scan_summary", out var summaryObj)
&& summaryObj is Dictionary<string, object> summaryDict
? summaryDict
: new Dictionary<string, object>();
var overallRisk = scanSummary.TryGetValue("overall_risk", out var riskObj)
? riskObj?.ToString()
: null;
var data = new Dictionary<string, object>(StringComparer.Ordinal)
{
["scan_summary"] = scanSummary,
["reportAvailable"] = true,
["phase"] = phaseLabel ?? "single",
};
var evt = new EnrollmentEvent
{
TenantId = tenantId,
SessionId = sessionId,
EventType = "vulnerability_report",
Severity = MapVulnerabilityRiskToSeverity(overallRisk),
Source = "VulnerabilityCorrelationService",
Phase = EnrollmentPhase.Unknown,
Message = $"Vulnerability scan complete (risk={overallRisk ?? "unknown"}).",
Timestamp = DateTime.UtcNow,
Data = data,
};
await _sessionRepo.StoreEventsBatchAsync(new List<EnrollmentEvent> { evt });
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"{Prefix} vulnerability_report synth-event emit failed (non-fatal)", sessionPrefix);
}
}
private static EventSeverity MapVulnerabilityRiskToSeverity(string? overallRisk) =>
(overallRisk ?? string.Empty).ToLowerInvariant() switch
{
"critical" => EventSeverity.Critical,
"high" => EventSeverity.Error,
"medium" => EventSeverity.Warning,
_ => EventSeverity.Info,
};
}
/// <summary>
@@ -0,0 +1,117 @@
using System;
using System.Threading;
using System.Threading.Tasks;
using Azure.Identity;
using Azure.Storage.Queues;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
namespace AutopilotMonitor.Functions.Services.Vulnerability
{
/// <summary>
/// Azure Storage Queue implementation of <see cref="IVulnerabilityCorrelateProducer"/>.
/// Mirrors <c>AzureQueueAnalyzeOnEnrollmentEndProducer</c>: Managed Identity preferred,
/// connection-string fallback, Base64 encoding (matches the consumer worker).
/// <para>
/// <b>Fault tolerance:</b> queue-side exceptions (network, throttling, queue missing) are
/// logged but never rethrown. The agent's HTTP 200 must not be blocked. A missed enqueue
/// means the session has no vulnerability report — the user can still trigger via the
/// "Analyze Now" UI button (<c>GET /sessions/{id}/vulnerability-report?rescan=true</c>).
/// </para>
/// </summary>
public sealed class AzureQueueVulnerabilityCorrelateProducer : IVulnerabilityCorrelateProducer
{
private readonly QueueClient _queueClient;
private readonly ILogger<AzureQueueVulnerabilityCorrelateProducer> _logger;
private int _queueEnsured; // 0 = not yet ensured, 1 = CreateIfNotExistsAsync has run
public AzureQueueVulnerabilityCorrelateProducer(
IConfiguration configuration,
ILogger<AzureQueueVulnerabilityCorrelateProducer> logger)
{
_logger = logger;
var storageAccountName = configuration["AzureStorageAccountName"];
var connectionString = configuration["AzureTableStorageConnectionString"];
// Match the consumer (BackgroundService) Base64 encoding so messages round-trip.
var options = new QueueClientOptions
{
MessageEncoding = QueueMessageEncoding.Base64,
};
if (!string.IsNullOrEmpty(storageAccountName))
{
var queueUri = new Uri(
$"https://{storageAccountName}.queue.core.windows.net/{Constants.QueueNames.VulnerabilityCorrelate}");
_queueClient = new QueueClient(queueUri, new DefaultAzureCredential(), options);
_logger.LogInformation(
"VulnerabilityCorrelate producer initialized with Managed Identity (account: {Account})",
storageAccountName);
}
else if (!string.IsNullOrEmpty(connectionString))
{
_queueClient = new QueueClient(
connectionString, Constants.QueueNames.VulnerabilityCorrelate, options);
_logger.LogInformation("VulnerabilityCorrelate producer initialized with connection string");
}
else
{
throw new InvalidOperationException(
"Queue Storage not configured. Set either 'AzureStorageAccountName' (for Managed Identity) or 'AzureTableStorageConnectionString'.");
}
}
public async Task EnqueueAsync(VulnerabilityCorrelateEnvelope envelope, CancellationToken cancellationToken = default)
{
if (envelope is null) return;
if (string.IsNullOrEmpty(envelope.TenantId) || string.IsNullOrEmpty(envelope.SessionId))
{
_logger.LogWarning(
"VulnerabilityCorrelate enqueue skipped — missing TenantId/SessionId (reason={Reason})",
envelope.Reason);
return;
}
await EnsureQueueExistsAsync(cancellationToken).ConfigureAwait(false);
try
{
var body = JsonConvert.SerializeObject(envelope);
await _queueClient.SendMessageAsync(body, cancellationToken).ConfigureAwait(false);
_logger.LogInformation(
"VulnerabilityCorrelate enqueued (tenant={Tenant} session={Session} whiteGlovePart={Part} reason={Reason})",
envelope.TenantId, envelope.SessionId,
envelope.WhiteGlovePart?.ToString() ?? "none", envelope.Reason);
}
catch (Exception ex)
{
// Fail-soft — agent is on hot path. Manual rescan via the UI remains the fallback.
_logger.LogWarning(
ex,
"VulnerabilityCorrelate enqueue failed (tenant={Tenant} session={Session} reason={Reason}) — manual rescan is the fallback",
envelope.TenantId, envelope.SessionId, envelope.Reason);
}
}
private async Task EnsureQueueExistsAsync(CancellationToken cancellationToken)
{
if (_queueEnsured == 1) return;
try
{
await _queueClient.CreateIfNotExistsAsync(cancellationToken: cancellationToken).ConfigureAwait(false);
Interlocked.Exchange(ref _queueEnsured, 1);
}
catch (Exception ex)
{
// Leave _queueEnsured=0 so a later call retries.
_logger.LogWarning(ex, "VulnerabilityCorrelate queue CreateIfNotExists failed — will retry next enqueue");
}
}
}
}
@@ -0,0 +1,22 @@
using System.Threading;
using System.Threading.Tasks;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Functions.Services.Vulnerability
{
/// <summary>
/// Enqueues <see cref="VulnerabilityCorrelateEnvelope"/> messages onto the
/// <c>vulnerability-correlate</c> queue. Replaces the previous in-function
/// fire-and-forget <c>Task.Run</c> that could be killed by Functions scale-in.
/// <para>
/// Implementations MUST NOT throw on transient send failures — the caller is on
/// the agent's hot HTTP path and must always return 200. A missed enqueue means
/// the session falls back to the manual <c>"Analyze Now"</c> UI path
/// (<c>GET /sessions/{id}/vulnerability-report?rescan=true</c>).
/// </para>
/// </summary>
public interface IVulnerabilityCorrelateProducer
{
Task EnqueueAsync(VulnerabilityCorrelateEnvelope envelope, CancellationToken cancellationToken = default);
}
}
@@ -0,0 +1,44 @@
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
namespace AutopilotMonitor.Functions.Services.Vulnerability
{
/// <summary>
/// Loads merged inventory items from the <c>software_inventory_analysis</c> shutdown
/// events stored in the Events table. Single source of truth for both the queue worker
/// (<c>VulnerabilityCorrelateHandler</c>) and the manual rescan endpoint
/// (<c>GetVulnerabilityReportFunction</c> with <c>?rescan=true</c>) — extracted into a
/// DI-fähig service so callers can be mocked in tests.
/// </summary>
public interface IVulnerabilityInventoryLoader
{
/// <summary>
/// Reads <c>software_inventory_analysis</c> events for the session, filters to
/// shutdown snapshots and (when <paramref name="whiteGlovePart"/> is non-null) to
/// the matching WhiteGlove phase, then returns the flattened inventory list.
/// Chunked snapshots are concatenated. Returns an empty list when nothing matches.
/// <para>
/// <b>Phase filtering:</b>
/// <list type="bullet">
/// <item><c>whiteGlovePart=1</c> — only events with <c>whiteglove_part==1</c>.</item>
/// <item><c>whiteGlovePart=2</c> — only events with <c>whiteglove_part==2</c>.</item>
/// <item><c>whiteGlovePart=null</c> (queue path: non-WG shutdown) — only events
/// WITHOUT a <c>whiteglove_part</c> field, so non-WG correlation does not pick
/// up Part-1 events from a parallel WG flow on the same session id.</item>
/// </list>
/// </para>
/// <para>
/// The manual rescan endpoint passes <c>includeAllPhases=true</c> instead — it
/// rescans every shutdown inventory event regardless of phase tag, matching the
/// pre-PR behaviour the UI depends on.
/// </para>
/// </summary>
Task<IReadOnlyList<Dictionary<string, object>>> LoadShutdownInventoryAsync(
string tenantId,
string sessionId,
int? whiteGlovePart = null,
bool includeAllPhases = false,
CancellationToken cancellationToken = default);
}
}
@@ -0,0 +1,487 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Services.Analyze;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Services.Vulnerability
{
/// <summary>
/// Consumes <see cref="VulnerabilityCorrelateEnvelope"/> messages off the
/// <c>vulnerability-correlate</c> queue and runs the full vulnerability correlation
/// pipeline. Replaces the previous in-function fire-and-forget <c>Task.Run</c> in
/// <c>EventIngestProcessor</c> that could be killed mid-flight by Azure Functions
/// scale-in, leaving the session without a vulnerability report.
/// <para>
/// <b>Pipeline:</b>
/// <list type="number">
/// <item>Reload inventory from <c>software_inventory_analysis</c> events
/// (idempotent against queue re-deliveries — does not depend on caller-side state).</item>
/// <item>Skip when <c>AdminConfiguration.VulnerabilityCorrelationEnabled</c> is off.</item>
/// <item>Run <see cref="VulnerabilityCorrelationService.CorrelateAsync"/> with a 60s timeout.</item>
/// <item>Tag findings with the phase label derived from
/// <see cref="VulnerabilityCorrelateEnvelope.WhiteGlovePart"/>.</item>
/// <item>For Part 2: merge with the existing Part 1 report via
/// <see cref="VulnerabilityCorrelationService.MergeReports"/>.</item>
/// <item>Persist the report, emit the synthetic <c>vulnerability_report</c> event,
/// enqueue a downstream re-analyze
/// (<see cref="AnalyzeOnEnrollmentEndHandler.ReasonVulnerabilityCorrelated"/>),
/// update the CveIndex, notify SignalR.</item>
/// </list>
/// </para>
/// <para>
/// <b>Failure semantics:</b> envelopes with missing TenantId/SessionId are dropped
/// (returning normally so the worker deletes them — retry won't help). Repository or
/// correlation exceptions propagate to the worker so the message is left un-deleted
/// for visibility-timeout retry; after <c>MaxDequeueCount</c> attempts it moves to
/// poison. Side-effect failures (CveIndex update, SignalR notify) are caught locally
/// so they don't trigger a full re-correlation.
/// </para>
/// </summary>
public class VulnerabilityCorrelateHandler
{
public const string ReasonShutdownInventory = "shutdown_inventory";
public const string ReasonWhiteGlovePart1Inventory = "whiteglove_part1_inventory";
private static readonly TimeSpan CorrelationTimeout = TimeSpan.FromSeconds(60);
private readonly AdminConfigurationService _adminConfigService;
private readonly VulnerabilityCorrelationService _vulnerabilityCorrelation;
private readonly IVulnerabilityRepository _vulnRepo;
private readonly ISessionRepository _sessionRepo;
private readonly IVulnerabilityInventoryLoader _inventoryLoader;
private readonly IAnalyzeOnEnrollmentEndProducer _analyzeProducer;
private readonly SignalRNotificationService _signalRNotification;
private readonly ILogger<VulnerabilityCorrelateHandler> _logger;
public VulnerabilityCorrelateHandler(
AdminConfigurationService adminConfigService,
VulnerabilityCorrelationService vulnerabilityCorrelation,
IVulnerabilityRepository vulnRepo,
ISessionRepository sessionRepo,
IVulnerabilityInventoryLoader inventoryLoader,
IAnalyzeOnEnrollmentEndProducer analyzeProducer,
SignalRNotificationService signalRNotification,
ILogger<VulnerabilityCorrelateHandler> logger)
{
_adminConfigService = adminConfigService;
_vulnerabilityCorrelation = vulnerabilityCorrelation;
_vulnRepo = vulnRepo;
_sessionRepo = sessionRepo;
_inventoryLoader = inventoryLoader;
_analyzeProducer = analyzeProducer;
_signalRNotification = signalRNotification;
_logger = logger;
}
public async Task HandleAsync(
VulnerabilityCorrelateEnvelope envelope,
CancellationToken cancellationToken = default)
{
if (envelope is null)
{
_logger.LogWarning("VulnerabilityCorrelate handler: null envelope — dropping");
return;
}
if (string.IsNullOrEmpty(envelope.TenantId) || string.IsNullOrEmpty(envelope.SessionId))
{
_logger.LogWarning(
"VulnerabilityCorrelate handler: dropping envelope with missing TenantId/SessionId (reason={Reason})",
envelope.Reason);
return;
}
var sessionPrefix = $"[Session: {envelope.SessionId.Substring(0, Math.Min(8, envelope.SessionId.Length))}]";
var adminConfig = await _adminConfigService.GetConfigurationAsync().ConfigureAwait(false);
if (adminConfig?.VulnerabilityCorrelationEnabled != true)
{
_logger.LogInformation(
"{Prefix} VulnerabilityCorrelate skipped — VulnerabilityCorrelationEnabled=false",
sessionPrefix);
return;
}
// Filter inventory by the envelope's WhiteGlovePart so a Part-2 correlation
// does not pick up Part-1 inventory items (which would mis-tag findings as
// user_enrollment) and a delayed Part-1 retry does not later overwrite a
// combined Part-1+Part-2 report with phase=device_setup tags.
var inventoryItems = await _inventoryLoader
.LoadShutdownInventoryAsync(
envelope.TenantId, envelope.SessionId,
whiteGlovePart: envelope.WhiteGlovePart,
cancellationToken: cancellationToken)
.ConfigureAwait(false);
if (inventoryItems.Count == 0)
{
_logger.LogWarning(
"{Prefix} VulnerabilityCorrelate: no shutdown inventory items found — dropping (lagMs={Lag})",
sessionPrefix, (long)(DateTime.UtcNow - envelope.EnqueuedAt).TotalMilliseconds);
return;
}
using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
cts.CancelAfter(CorrelationTimeout);
var reportData = await _vulnerabilityCorrelation.CorrelateAsync(
envelope.SessionId, envelope.TenantId,
inventoryItems.ToList(), cts.Token).ConfigureAwait(false);
if (reportData == null)
{
_logger.LogInformation(
"{Prefix} VulnerabilityCorrelate: no findings (whiteGlovePart={Part}, items={Count})",
sessionPrefix, envelope.WhiteGlovePart?.ToString() ?? "none", inventoryItems.Count);
return;
}
var phaseLabel = envelope.WhiteGlovePart == 1 ? "device_setup"
: envelope.WhiteGlovePart == 2 ? "user_enrollment"
: (string?)null;
if (phaseLabel != null
&& reportData.ContainsKey("findings")
&& reportData["findings"] is List<Dictionary<string, object>> tagFindings)
{
foreach (var f in tagFindings)
f.TryAdd("phase", phaseLabel);
}
// Phase-aware report reconciliation. Two scenarios drive the behaviour:
//
// 1. Normal order: Part 1 arrives first, stores standalone. Part 2 arrives,
// loads existing Part-1 report, merges, stores combined. (Default WG flow.)
//
// 2. Out-of-order: Part 2 arrives first (e.g. Part-1 message hit poison and
// retried late), stores Part-2 standalone. Part 1 arrives later — without
// this guard it would overwrite the Part-2 report with a Part-1-only one,
// losing the user_enrollment findings. We detect the case via the
// user_enrollment phase tag on existing findings and SKIP the store.
// Manual rescan ("Analyze Now" with includeAllPhases=true) remains the
// escape hatch to re-correlate everything from raw inventory if the user
// wants the Part-1 findings back.
if (envelope.WhiteGlovePart == 2 || envelope.WhiteGlovePart == 1)
{
try
{
var existingReport = await _vulnRepo.GetVulnerabilityReportAsync(
envelope.TenantId, envelope.SessionId).ConfigureAwait(false);
if (existingReport != null)
{
if (envelope.WhiteGlovePart == 2)
{
reportData = VulnerabilityCorrelationService.MergeReports(
existingReport, reportData,
existingPhaseLabel: "device_setup",
newPhaseLabel: "user_enrollment");
_logger.LogInformation(
"{Prefix} WhiteGlove Part 2: merged vulnerability report with Part 1 findings",
sessionPrefix);
}
else if (envelope.WhiteGlovePart == 1
&& ContainsUserEnrollmentFindings(existingReport))
{
// Out-of-order: Part 2 already correlated and stored. A Part-1
// store now would overwrite the combined / Part-2 report with
// a Part-1-only view. Skip and keep the richer report.
_logger.LogWarning(
"{Prefix} WhiteGlove Part 1: existing report already contains user_enrollment findings — skipping Part-1 store to avoid overwriting Part-2/combined report (out-of-order delivery)",
sessionPrefix);
return;
}
}
}
catch (Exception ex) when (envelope.WhiteGlovePart == 2)
{
// Part-1 read failure is non-fatal for Part 2 — store Part 2 standalone.
_logger.LogWarning(ex,
"{Prefix} Failed to load Part 1 report for merge (storing Part 2 standalone)",
sessionPrefix);
}
// For Part 1 we deliberately let read exceptions propagate so the queue
// worker leaves the message visible for retry — without the existing-report
// check we can't tell whether a Part 2 already won and a Part-1 store is
// safe.
}
// Store + emit synth event in order: report first (canonical source),
// then synth event so analyze rules can reference findings already on disk.
await _vulnRepo.StoreVulnerabilityReportAsync(
envelope.TenantId, envelope.SessionId, reportData).ConfigureAwait(false);
_logger.LogInformation(
"{Prefix} VulnerabilityCorrelate complete (whiteGlovePart={Part}, lagMs={Lag})",
sessionPrefix, envelope.WhiteGlovePart?.ToString() ?? "none",
(long)(DateTime.UtcNow - envelope.EnqueuedAt).TotalMilliseconds);
await EmitVulnerabilityReportEventAsync(
envelope.TenantId, envelope.SessionId, sessionPrefix,
reportData, phaseLabel).ConfigureAwait(false);
// Re-evaluate analyze rules now that the synth event is on disk —
// vulnerability-driven rules (ANALYZE-ID-003) need this synth event as evidence.
// Enqueued onto the analyze-on-enrollment-end queue with a dedicated reason; the
// worker runs RuleEngine with dedup so already-stored results are not duplicated.
await _analyzeProducer.EnqueueAsync(new AnalyzeOnEnrollmentEndEnvelope
{
TenantId = envelope.TenantId,
SessionId = envelope.SessionId,
Reason = AnalyzeOnEnrollmentEndHandler.ReasonVulnerabilityCorrelated,
EnqueuedAt = DateTime.UtcNow,
}, cancellationToken).ConfigureAwait(false);
await SafeUpsertCveIndexAsync(envelope, sessionPrefix, reportData).ConfigureAwait(false);
await SafeNotifySignalRAsync(envelope, sessionPrefix, reportData).ConfigureAwait(false);
}
/// <summary>
/// Stores a synthetic <c>vulnerability_report</c> event after a successful scan so
/// analyze rules can fire on the same event-stream surface as agent-emitted events.
/// Carries only <c>scan_summary</c> + <c>reportAvailable</c> + <c>phase</c>; the full
/// per-CVE list lives in the <c>VulnerabilityReports</c> table (read separately by
/// the report API).
/// </summary>
private async Task EmitVulnerabilityReportEventAsync(
string tenantId,
string sessionId,
string sessionPrefix,
Dictionary<string, object> reportData,
string? phaseLabel)
{
try
{
var scanSummary = reportData.TryGetValue("scan_summary", out var summaryObj)
&& summaryObj is Dictionary<string, object> summaryDict
? summaryDict
: new Dictionary<string, object>();
var overallRisk = scanSummary.TryGetValue("overall_risk", out var riskObj)
? riskObj?.ToString()
: null;
var data = new Dictionary<string, object>(StringComparer.Ordinal)
{
["scan_summary"] = scanSummary,
["reportAvailable"] = true,
["phase"] = phaseLabel ?? "single",
};
var evt = new EnrollmentEvent
{
TenantId = tenantId,
SessionId = sessionId,
EventType = "vulnerability_report",
Severity = MapVulnerabilityRiskToSeverity(overallRisk),
Source = "VulnerabilityCorrelationService",
Phase = EnrollmentPhase.Unknown,
Message = $"Vulnerability scan complete (risk={overallRisk ?? "unknown"}).",
Timestamp = DateTime.UtcNow,
Data = data,
};
await _sessionRepo.StoreEventsBatchAsync(new List<EnrollmentEvent> { evt }).ConfigureAwait(false);
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"{Prefix} vulnerability_report synth-event emit failed (non-fatal)", sessionPrefix);
}
}
private async Task SafeUpsertCveIndexAsync(
VulnerabilityCorrelateEnvelope envelope,
string sessionPrefix,
Dictionary<string, object> reportData)
{
try
{
var findings = reportData.ContainsKey("findings")
? reportData["findings"] as List<Dictionary<string, object>>
: null;
if (findings != null && findings.Count > 0)
{
await _sessionRepo.UpsertCveIndexEntriesAsync(
envelope.TenantId, envelope.SessionId, findings).ConfigureAwait(false);
}
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"{Prefix} CveIndex update failed (non-fatal)", sessionPrefix);
}
}
private async Task SafeNotifySignalRAsync(
VulnerabilityCorrelateEnvelope envelope,
string sessionPrefix,
Dictionary<string, object> reportData)
{
try
{
var overallRisk = reportData.ContainsKey("scan_summary")
&& reportData["scan_summary"] is Dictionary<string, object> summary
&& summary.ContainsKey("overall_risk")
? summary["overall_risk"]?.ToString() ?? "unknown"
: "unknown";
await _signalRNotification.NotifyVulnerabilityReportAvailableAsync(
envelope.TenantId, envelope.SessionId, overallRisk).ConfigureAwait(false);
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"{Prefix} SignalR vulnerability_report notify failed (non-fatal)", sessionPrefix);
}
}
/// <summary>
/// Out-of-order guard for Part-1 store: returns <c>true</c> when the previously
/// stored report has Part-2 content. The Part-1 handler skips its store in that
/// case to avoid overwriting the richer Part-2 / combined report.
/// <para>
/// Detects Part-2 content via TWO signals so neither in-flight shape slips through:
/// </para>
/// <list type="number">
/// <item><b>Aggregated:</b> <c>scan_summary.phase_contributions.user_enrollment &gt; 0</c>
/// — written by <see cref="VulnerabilityCorrelationService.MergeReports"/>
/// whenever Part-2 findings participated in a merge.</item>
/// <item><b>Per-finding:</b> any finding with <c>phase == "user_enrollment"</c>
/// (Part-2-only standalone store, no merge happened) OR
/// <c>phase == "both"</c> (merge collapsed identical Part-1+Part-2 findings into
/// a single combined entry, see <c>AddFindingWithDedup</c>). The
/// <c>"both"</c> case is the one that round-2's user_enrollment-only check
/// missed — a fully-collapsed merged report has zero
/// <c>phase=="user_enrollment"</c> findings yet still must not be overwritten.</item>
/// </list>
/// <para>
/// Defensive against the JArray vs native dual-representation that
/// <c>EnrollmentEvent.Data</c> deserialization produces.
/// </para>
/// </summary>
internal static bool ContainsUserEnrollmentFindings(Dictionary<string, object> existingReport)
{
// Primary: phase_contributions counter from a previous MergeReports run.
if (TryGetUserEnrollmentContribution(existingReport, out var contribution) && contribution > 0)
return true;
// Secondary: scan per-finding phase tags. Accepts "user_enrollment" (Part-2
// standalone, never merged) and "both" (merge collapsed identical findings).
return AnyFindingPhaseIndicatesPart2(existingReport);
}
private static bool TryGetUserEnrollmentContribution(
Dictionary<string, object> report, out int contribution)
{
contribution = 0;
if (!report.TryGetValue("scan_summary", out var summaryObj))
return false;
// Either a typed Dictionary (in-process build) or a JObject (Newtonsoft round-trip).
if (summaryObj is Dictionary<string, object> dict
&& dict.TryGetValue("phase_contributions", out var contribObj))
{
return TryReadUserEnrollmentValue(contribObj, out contribution);
}
if (summaryObj is Newtonsoft.Json.Linq.JObject jobj
&& jobj.TryGetValue("phase_contributions", out var contribTok))
{
return TryReadUserEnrollmentValue(contribTok, out contribution);
}
return false;
}
private static bool TryReadUserEnrollmentValue(object contribContainer, out int value)
{
value = 0;
if (contribContainer is Dictionary<string, object> contrib
&& contrib.TryGetValue("user_enrollment", out var raw))
{
return TryConvertToInt(raw, out value);
}
if (contribContainer is Newtonsoft.Json.Linq.JObject jcontrib
&& jcontrib.TryGetValue("user_enrollment", out var rawTok))
{
return TryConvertToInt(rawTok, out value);
}
return false;
}
private static bool TryConvertToInt(object? raw, out int value)
{
value = 0;
if (raw == null) return false;
switch (raw)
{
case int i: value = i; return true;
case long l: value = (int)l; return true;
case short s: value = s; return true;
case double d: value = (int)d; return true;
case Newtonsoft.Json.Linq.JValue jv when jv.Value != null:
return TryConvertToInt(jv.Value, out value);
case string str when int.TryParse(str, out var p): value = p; return true;
}
return false;
}
private static bool AnyFindingPhaseIndicatesPart2(Dictionary<string, object> report)
{
if (!report.TryGetValue("findings", out var findingsObj) || findingsObj == null)
return false;
// Native .NET path
if (findingsObj is IEnumerable<Dictionary<string, object>> typedFindings)
{
foreach (var f in typedFindings)
if (PhaseIndicatesPart2(f.GetValueOrDefault("phase")?.ToString())) return true;
return false;
}
// JArray path (Newtonsoft round-trip from stored JSON)
if (findingsObj is Newtonsoft.Json.Linq.JArray jarr)
{
foreach (var item in jarr)
{
if (item is Newtonsoft.Json.Linq.JObject jobj
&& jobj.TryGetValue("phase", out var phaseTok)
&& PhaseIndicatesPart2(phaseTok?.ToString()))
return true;
}
return false;
}
// Loose-typed enumerable (e.g. List<object> from custom deserialization)
if (findingsObj is System.Collections.IEnumerable looseEnum)
{
foreach (var item in looseEnum)
{
if (item is IDictionary<string, object> dict
&& dict.TryGetValue("phase", out var phaseObj)
&& PhaseIndicatesPart2(phaseObj?.ToString()))
return true;
}
}
return false;
}
private static bool PhaseIndicatesPart2(string? phase) =>
string.Equals(phase, "user_enrollment", StringComparison.Ordinal)
|| string.Equals(phase, "both", StringComparison.Ordinal);
private static EventSeverity MapVulnerabilityRiskToSeverity(string? overallRisk) =>
(overallRisk ?? string.Empty).ToLowerInvariant() switch
{
"critical" => EventSeverity.Critical,
"high" => EventSeverity.Error,
"medium" => EventSeverity.Warning,
_ => EventSeverity.Info,
};
}
}
@@ -0,0 +1,254 @@
using System;
using System.Threading;
using System.Threading.Tasks;
using Azure.Identity;
using Azure.Storage.Queues;
using Azure.Storage.Queues.Models;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
namespace AutopilotMonitor.Functions.Services.Vulnerability
{
/// <summary>
/// Background worker for the <c>vulnerability-correlate</c> queue. Replaces the previous
/// in-function fire-and-forget <c>Task.Run</c> in <c>EventIngestProcessor</c> that ran
/// vulnerability correlation after a shutdown <c>software_inventory_analysis</c> event —
/// that pattern could be killed mid-flight by Azure Functions scale-in, leaving sessions
/// without a stored report (manual "Analyze Now" rescan was the only recovery).
/// <para>
/// Mirrors <c>AnalyzeOnEnrollmentEndQueueWorker</c> instead of using a Functions
/// <c>QueueTrigger</c>: pure DI <see cref="BackgroundService"/> driven by the existing
/// <c>AzureStorageAccountName</c> (Managed Identity) or <c>AzureTableStorageConnectionString</c>
/// fallback — no bespoke binding glue.
/// </para>
/// <para>
/// <b>Platform parity to QueueTrigger:</b>
/// <list type="bullet">
/// <item>Receive batch of up to <see cref="BatchSize"/> messages with
/// <see cref="VisibilityTimeout"/> per message.</item>
/// <item>Handler success → <c>DeleteMessageAsync</c>.</item>
/// <item>Handler exception → no delete; message becomes visible again after the timeout,
/// <c>DequeueCount</c> increments. Identical to platform-retry semantics.</item>
/// <item>After <see cref="MaxDequeueCount"/> failed attempts the message is moved to
/// <c>vulnerability-correlate-poison</c> and deleted from the main queue.</item>
/// <item>Empty receive → sleep for <see cref="PollInterval"/> and retry.</item>
/// </list>
/// </para>
/// </summary>
public sealed class VulnerabilityCorrelateQueueWorker : BackgroundService
{
/// <summary>Max messages received per poll. Storage Queue caps batch-receive at 32.</summary>
internal const int BatchSize = 32;
/// <summary>
/// Visibility timeout per received message. Correlation involves CVE/KEV lookups
/// against external data and can take several seconds; the inner correlation has its
/// own 60s timeout. 5 minutes leaves plenty of headroom for retries and matches
/// the Analyze + IndexReconcile workers.
/// </summary>
internal static readonly TimeSpan VisibilityTimeout = TimeSpan.FromMinutes(5);
/// <summary>Idle sleep between empty receives.</summary>
internal static readonly TimeSpan PollInterval = TimeSpan.FromSeconds(10);
/// <summary>Cool-down after an unhandled poll-loop exception.</summary>
internal static readonly TimeSpan ErrorBackoff = TimeSpan.FromSeconds(30);
/// <summary>Match the QueueTrigger platform default: 5 attempts, then move to poison.</summary>
internal const int MaxDequeueCount = 5;
private const string PoisonQueueSuffix = "-poison";
private readonly QueueClient _mainQueue;
private readonly QueueClient _poisonQueue;
private readonly VulnerabilityCorrelateHandler _handler;
private readonly ILogger<VulnerabilityCorrelateQueueWorker> _logger;
public VulnerabilityCorrelateQueueWorker(
IConfiguration configuration,
VulnerabilityCorrelateHandler handler,
ILogger<VulnerabilityCorrelateQueueWorker> logger)
{
_handler = handler ?? throw new ArgumentNullException(nameof(handler));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
var options = new QueueClientOptions
{
MessageEncoding = QueueMessageEncoding.Base64,
};
var storageAccountName = configuration["AzureStorageAccountName"];
var connectionString = configuration["AzureTableStorageConnectionString"];
if (!string.IsNullOrEmpty(storageAccountName))
{
var mainUri = new Uri(
$"https://{storageAccountName}.queue.core.windows.net/{Constants.QueueNames.VulnerabilityCorrelate}");
var poisonUri = new Uri(
$"https://{storageAccountName}.queue.core.windows.net/{Constants.QueueNames.VulnerabilityCorrelate}{PoisonQueueSuffix}");
var credential = new DefaultAzureCredential();
_mainQueue = new QueueClient(mainUri, credential, options);
_poisonQueue = new QueueClient(poisonUri, credential, options);
_logger.LogInformation(
"VulnerabilityCorrelateQueueWorker initialized with Managed Identity (account: {Account})",
storageAccountName);
}
else if (!string.IsNullOrEmpty(connectionString))
{
_mainQueue = new QueueClient(
connectionString, Constants.QueueNames.VulnerabilityCorrelate, options);
_poisonQueue = new QueueClient(
connectionString, Constants.QueueNames.VulnerabilityCorrelate + PoisonQueueSuffix, options);
_logger.LogInformation("VulnerabilityCorrelateQueueWorker initialized with connection string");
}
else
{
throw new InvalidOperationException(
"Queue Storage not configured. Set either 'AzureStorageAccountName' (for Managed Identity) or 'AzureTableStorageConnectionString'.");
}
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
await TryCreateQueueAsync(_mainQueue, "main", stoppingToken).ConfigureAwait(false);
await TryCreateQueueAsync(_poisonQueue, "poison", stoppingToken).ConfigureAwait(false);
_logger.LogInformation("VulnerabilityCorrelateQueueWorker: poll loop started (queue {Name})",
_mainQueue.Name);
while (!stoppingToken.IsCancellationRequested)
{
try
{
var batch = await _mainQueue
.ReceiveMessagesAsync(BatchSize, VisibilityTimeout, stoppingToken)
.ConfigureAwait(false);
if (batch?.Value is null || batch.Value.Length == 0)
{
await Task.Delay(PollInterval, stoppingToken).ConfigureAwait(false);
continue;
}
foreach (var msg in batch.Value)
{
if (stoppingToken.IsCancellationRequested) break;
await ProcessOneAsync(msg, stoppingToken).ConfigureAwait(false);
}
}
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
{
break;
}
catch (Exception ex)
{
_logger.LogError(ex,
"VulnerabilityCorrelateQueueWorker: poll-loop error — backing off {Backoff}",
ErrorBackoff);
try { await Task.Delay(ErrorBackoff, stoppingToken).ConfigureAwait(false); }
catch (OperationCanceledException) { break; }
}
}
_logger.LogInformation("VulnerabilityCorrelateQueueWorker: poll loop stopped");
}
private async Task ProcessOneAsync(QueueMessage msg, CancellationToken ct)
{
if (msg.DequeueCount > MaxDequeueCount)
{
await MoveToPoisonAsync(msg, ct).ConfigureAwait(false);
return;
}
VulnerabilityCorrelateEnvelope? envelope;
try
{
envelope = JsonConvert.DeserializeObject<VulnerabilityCorrelateEnvelope>(msg.Body.ToString());
}
catch (JsonException ex)
{
_logger.LogWarning(ex,
"VulnerabilityCorrelateQueueWorker: malformed envelope JSON — dropping (msg {Id})",
msg.MessageId);
await SafeDeleteAsync(msg, ct).ConfigureAwait(false);
return;
}
if (envelope is null)
{
_logger.LogWarning(
"VulnerabilityCorrelateQueueWorker: null envelope after deserialization — dropping (msg {Id})",
msg.MessageId);
await SafeDeleteAsync(msg, ct).ConfigureAwait(false);
return;
}
try
{
await _handler.HandleAsync(envelope, ct).ConfigureAwait(false);
await SafeDeleteAsync(msg, ct).ConfigureAwait(false);
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
throw;
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"VulnerabilityCorrelateQueueWorker: handler failed (msg {Id}, dequeue {N}) — visibility-timeout retry",
msg.MessageId, msg.DequeueCount);
}
}
private async Task MoveToPoisonAsync(QueueMessage msg, CancellationToken ct)
{
try
{
await _poisonQueue.SendMessageAsync(msg.Body.ToString(), ct).ConfigureAwait(false);
await SafeDeleteAsync(msg, ct).ConfigureAwait(false);
_logger.LogWarning(
"VulnerabilityCorrelateQueueWorker: moved message {Id} to poison queue after {N} failed attempts",
msg.MessageId, msg.DequeueCount - 1);
}
catch (Exception ex)
{
_logger.LogError(ex,
"VulnerabilityCorrelateQueueWorker: poison move failed for message {Id} (will retry)",
msg.MessageId);
}
}
private async Task SafeDeleteAsync(QueueMessage msg, CancellationToken ct)
{
try
{
await _mainQueue.DeleteMessageAsync(msg.MessageId, msg.PopReceipt, ct).ConfigureAwait(false);
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"VulnerabilityCorrelateQueueWorker: delete failed for message {Id} (will reappear after visibility-timeout)",
msg.MessageId);
}
}
private async Task TryCreateQueueAsync(QueueClient queue, string label, CancellationToken ct)
{
try
{
await queue.CreateIfNotExistsAsync(cancellationToken: ct).ConfigureAwait(false);
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"VulnerabilityCorrelateQueueWorker: CreateIfNotExists failed for {Label} queue — will continue, send/receive will retry",
label);
}
}
}
}
@@ -1099,6 +1099,18 @@ namespace AutopilotMonitor.Functions.Services.Vulnerability
string existingPhaseLabel = "device_setup",
string newPhaseLabel = "user_enrollment")
{
// Reports loaded via IVulnerabilityRepository.GetVulnerabilityReportAsync are
// deserialized by Newtonsoft into Dictionary<string, object> at the top level
// but with JObject/JArray/JValue children for everything below. The casts and
// OfType<Dictionary<string,object>>() in ExtractFindings, ExtractUnmatchedSoftware,
// the scan_summary read at line ~1176, and the vulnerabilities iteration in
// AddFindingWithDedup all silently produce empty results on JToken inputs —
// making a Part-2 merge against a stored Part-1 report effectively drop all
// Part-1 findings. Normalize both reports once up-front so every helper below
// sees native Dictionary/List shapes.
existingReport = NormalizeForMerge(existingReport);
newReport = NormalizeForMerge(newReport);
var existingFindings = ExtractFindings(existingReport);
var newFindings = ExtractFindings(newReport);
@@ -1287,6 +1299,91 @@ namespace AutopilotMonitor.Functions.Services.Vulnerability
return new List<Dictionary<string, object>>();
}
/// <summary>
/// Recursively converts a report's Newtonsoft <c>JObject</c>/<c>JArray</c>/<c>JValue</c>
/// children into native <c>Dictionary&lt;string, object&gt;</c> / <c>List&lt;object&gt;</c> /
/// primitive shapes, so the cast-based extraction helpers in <see cref="MergeReports"/>
/// see uniform inputs regardless of whether a report came in fresh from
/// <see cref="VulnerabilityCorrelationService"/> (native shapes) or was reloaded from
/// the <c>VulnerabilityReports</c> table via Newtonsoft <c>JsonConvert.DeserializeObject</c>
/// (JToken shapes for nested values).
/// <para>
/// Exposed as <c>internal</c> for direct unit-testing — callers should not invoke
/// this directly, the merge entry-point handles it transparently.
/// </para>
/// </summary>
internal static Dictionary<string, object> NormalizeForMerge(Dictionary<string, object> report)
{
if (report == null) return new Dictionary<string, object>();
var normalized = new Dictionary<string, object>(report.Count);
foreach (var kv in report)
{
var value = NormalizeValue(kv.Value);
if (value != null) normalized[kv.Key] = value;
}
return normalized;
}
private static object? NormalizeValue(object? value)
{
switch (value)
{
case null:
return null;
case Newtonsoft.Json.Linq.JObject jobj:
{
var dict = new Dictionary<string, object>(jobj.Count);
foreach (var prop in jobj.Properties())
{
var inner = NormalizeValue(prop.Value);
if (inner != null) dict[prop.Name] = inner;
}
return dict;
}
case Newtonsoft.Json.Linq.JArray jarr:
{
var list = new List<object>(jarr.Count);
foreach (var item in jarr)
{
var inner = NormalizeValue(item);
if (inner != null) list.Add(inner);
}
return list;
}
case Newtonsoft.Json.Linq.JValue jval:
return jval.Value;
case Dictionary<string, object> dict:
{
var copy = new Dictionary<string, object>(dict.Count);
foreach (var kv in dict)
{
var inner = NormalizeValue(kv.Value);
if (inner != null) copy[kv.Key] = inner;
}
return copy;
}
case System.Collections.IEnumerable enumerable when value is not string:
{
var list = new List<object>();
foreach (var item in enumerable)
{
var inner = NormalizeValue(item);
if (inner != null) list.Add(inner);
}
return list;
}
default:
return value;
}
}
}
// -----------------------------------------------------------------------
@@ -0,0 +1,130 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.DataAccess;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Services.Vulnerability
{
/// <summary>
/// Default <see cref="IVulnerabilityInventoryLoader"/> backed by
/// <see cref="ISessionRepository.GetSessionEventsAsync"/>. Extracted from
/// <c>GetVulnerabilityReportFunction.LoadInventoryItemsFromEventsAsync</c> so the queue
/// worker and the manual rescan endpoint share one loader.
/// </summary>
public sealed class VulnerabilityInventoryLoader : IVulnerabilityInventoryLoader
{
private readonly ISessionRepository _sessionRepo;
private readonly ILogger<VulnerabilityInventoryLoader> _logger;
public VulnerabilityInventoryLoader(
ISessionRepository sessionRepo,
ILogger<VulnerabilityInventoryLoader> logger)
{
_sessionRepo = sessionRepo;
_logger = logger;
}
public async Task<IReadOnlyList<Dictionary<string, object>>> LoadShutdownInventoryAsync(
string tenantId,
string sessionId,
int? whiteGlovePart = null,
bool includeAllPhases = false,
CancellationToken cancellationToken = default)
{
var items = new List<Dictionary<string, object>>();
var allEvents = await _sessionRepo.GetSessionEventsAsync(tenantId, sessionId).ConfigureAwait(false);
var inventoryEvents = allEvents
.Where(e => e.EventType == Constants.EventTypes.SoftwareInventoryAnalysis);
foreach (var evt in inventoryEvents)
{
cancellationToken.ThrowIfCancellationRequested();
if (evt.Data == null || evt.Data.Count == 0) continue;
try
{
// Only use shutdown events (final inventory, not baseline)
var triggeredAt = evt.Data.ContainsKey("triggered_at") ? evt.Data["triggered_at"]?.ToString() : null;
if (triggeredAt != "shutdown") continue;
// Phase filtering — see interface XML doc. The manual rescan endpoint
// passes includeAllPhases=true to keep its legacy "scan everything"
// behaviour; the queue handler always passes a specific phase.
if (!includeAllPhases && !PhaseMatches(evt.Data, whiteGlovePart)) continue;
if (!evt.Data.ContainsKey("inventory")) continue;
var inventoryData = evt.Data["inventory"];
// DeserializeEventData converts JArrays to List<object> via ConvertJTokenToNative,
// so handle both the native List<object> case and the raw JArray case.
if (inventoryData is IList<object> nativeList)
{
foreach (var entry in nativeList)
{
if (entry is Dictionary<string, object> dict)
{
items.Add(dict);
}
else if (entry is IDictionary<string, object> idict)
{
items.Add(new Dictionary<string, object>(idict));
}
}
}
else if (inventoryData is Newtonsoft.Json.Linq.JArray inventoryArray)
{
foreach (var item in inventoryArray)
{
if (item is Newtonsoft.Json.Linq.JObject obj)
{
items.Add(obj.ToObject<Dictionary<string, object>>() ?? new Dictionary<string, object>());
}
}
}
}
catch (Exception ex)
{
_logger.LogDebug(ex, "Failed to parse inventory event data (session {SessionId})", sessionId);
}
}
_logger.LogInformation(
"Loaded {Count} inventory items from Events table for session {SessionId} (whiteGlovePart={Part}, includeAllPhases={All})",
items.Count, sessionId,
whiteGlovePart?.ToString() ?? "none", includeAllPhases);
return items;
}
/// <summary>
/// Checks whether an event's <c>whiteglove_part</c> tag matches the requested phase.
/// Numeric parsing is lenient because <c>EnrollmentEvent.Data</c> values can come
/// back as <see cref="long"/> (Newtonsoft default), <see cref="int"/>, or string
/// depending on serializer round-trip.
/// </summary>
private static bool PhaseMatches(Dictionary<string, object> data, int? whiteGlovePart)
{
var hasTag = data.TryGetValue("whiteglove_part", out var raw) && raw != null;
if (!whiteGlovePart.HasValue) return !hasTag;
if (!hasTag) return false;
int parsed;
switch (raw)
{
case int i: parsed = i; break;
case long l: parsed = (int)l; break;
case short s: parsed = s; break;
case string str when int.TryParse(str, out var p): parsed = p; break;
default:
return false;
}
return parsed == whiteGlovePart.Value;
}
}
}
@@ -463,6 +463,18 @@ namespace AutopilotMonitor.Shared
/// rule results. Manual "Analyze Now" remains the user-side fallback.
/// </summary>
public const string AnalyzeOnEnrollmentEnd = "analyze-on-enrollment-end";
/// <summary>
/// Vulnerability correlation fan-out triggered by the shutdown <c>software_inventory_analysis</c>
/// event. Replaces the in-function fire-and-forget <c>Task.Run</c> in
/// <c>EventIngestProcessor</c> that could be killed by Functions scale-in, leaving the
/// session without a vulnerability report (manual "Analyze Now" was the only recovery).
/// Consumer runs <c>VulnerabilityCorrelationService.CorrelateAsync</c>, stores the report,
/// emits the synthetic <c>vulnerability_report</c> event, and enqueues a downstream
/// re-analyze on <see cref="AnalyzeOnEnrollmentEnd"/> with
/// <c>ReasonVulnerabilityCorrelated</c> so vulnerability-driven analyze rules can fire.
/// </summary>
public const string VulnerabilityCorrelate = "vulnerability-correlate";
}
}
}
@@ -0,0 +1,39 @@
using System;
namespace AutopilotMonitor.Shared.Models
{
/// <summary>
/// Queue-message envelope for the <c>vulnerability-correlate</c> queue. Replaces the
/// previous in-function fire-and-forget <c>Task.Run</c> in <c>EventIngestProcessor</c>
/// that ran the vulnerability correlation after a shutdown software_inventory_analysis
/// event — that <c>Task.Run</c> could be killed mid-flight by Azure Functions scale-in,
/// leaving the session without a vulnerability report until the user clicked "Analyze Now".
/// <para>
/// The handler reloads the inventory from the Events table itself (idempotent against
/// queue re-deliveries) — only <see cref="TenantId"/>, <see cref="SessionId"/>, and
/// <see cref="WhiteGlovePart"/> are required to drive correlation behaviour.
/// <see cref="Reason"/> and <see cref="EnqueuedAt"/> are diagnostic-only.
/// </para>
/// </summary>
public sealed class VulnerabilityCorrelateEnvelope
{
/// <summary>Schema version — bump on breaking envelope changes so consumers can reject or migrate.</summary>
public string EnvelopeVersion { get; set; } = "1";
public string TenantId { get; set; } = string.Empty;
public string SessionId { get; set; } = string.Empty;
/// <summary>
/// WhiteGlove enrollment phase indicator: <c>1</c> = pre-provisioning (device_setup),
/// <c>2</c> = user enrollment, <c>null</c> = single-phase (non-WhiteGlove). Drives
/// per-phase tagging of findings and the Part-1+Part-2 merge in the handler.
/// </summary>
public int? WhiteGlovePart { get; set; }
/// <summary>What caused the enqueue. Diagnostics only — handler does not branch on this.</summary>
public string Reason { get; set; } = string.Empty;
/// <summary>UTC time the producer enqueued the message; useful for measuring queue lag.</summary>
public DateTime EnqueuedAt { get; set; }
}
}