fix(agent): network-link gate before initial config fetch + post-registration recovery

Root cause (edu.kosh audit 2026-08-24, session ba828038): on boot-time relaunches the agent starts before the Wi-Fi link is up. The Phase 4 config fetch had no network gate (98f8114d only guarded Phase 6 registration), burned its 10/30/60s retry budget into a dead link and stranded the whole session on built-in defaults (ConfigVersion=0) — up to ~50% of that tenant's sessions, with no self-heal short of a manual rotate_config action. Not a version regression: rates varied wildly per day within the same agent build; the failed requests never reached the backend. Two complementary layers: - RemoteConfigService gates the initial fetch on the shared NetworkLinkWait poll (60s cap — longer than registration's 15s because the fetch runs first), so the retry budget starts on a live link. rotate_config stays ungated (latency-bound). - Phase 6.5: after registration proves the backend reachable, TryRecoverConfigAsync refetches once and AgentRuntimeConfig.TryRecoverAfterRegistration re-applies the live config (same merge/integrity pipeline as Phase 4). LastFetch* keeps the original failure record; remote_config_fetch_failed reports it with a recovered flag at Info severity, agent_started flags remoteConfigRecovered. Deliberately not re-run in recovery: kill-signal check (telemetry-channel control signals cover post-registration kills) and endpoint migration (next start). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
infra(telemetry): DCR also drops host-emitted successful SignalR dependencies
2026-10-25 05:00:07 +00:00 · 2026-08-24 11:49:37 +02:00 · 2026-08-23 19:15:41 +02:00 · 2026-08-23 18:31:32 +02:00 · 2026-08-23 17:52:02 +02:00
19 changed files with 903 additions and 85 deletions
@@ -0,0 +1,89 @@
+---
+type: Concept
+title: Telemetry Ingest Shaping (App Insights / Log Analytics)
+description: What the backend deliberately keeps out of Application Insights and why - one canonical request record (worker copy, per-item sampling bypass; host HTTP duplicate dropped by a workspace transformation), expected storage outcomes (404/412/409) and successful SignalR REST calls filtered in the worker, and the query rules that follow.
+resource: /src/Backend/AutopilotMonitor.Functions/Telemetry/StorageDependencyFilterProcessor.cs
+tags:
+  - backend
+  - telemetry
+  - application-insights
+  - cost
+timestamp: 2026-08-23
+---
+
+# Telemetry Ingest Shaping
+
+The backend is storage-I/O heavy and runs on the Functions isolated worker, which emits telemetry from TWO processes (host and worker). Left alone, Log Analytics ingestion is dominated by rows nobody reads. This document records what is removed, where, and what that means for queries.
+
+# Schema
+
+## Request telemetry — one canonical record
+
+| Source | Emitted by | Content | Fate |
+|---|---|---|---|
+| Host HTTP request (`Properties.Source` empty, `Url` set) | Functions host process | Name, duration, status — no tenant/user/correlation | **Dropped at ingestion** by the workspace transformation on `AppRequests` (`infra/appinsights-workspace-transforms.bicep`) |
+| Host non-HTTP invocation (`Url` empty) | Functions host process | Timer / queue executions | Kept (the worker middleware never sees them) |
+| Worker request (`Properties.Source == 'WorkerMiddleware'`) | `RequestTelemetryMiddleware` | TenantId, UserId, UserRole, CorrelationId, ClientSource, McpToolName, HttpPath | **Kept, unsampled** — the canonical record |
+
+Why two mechanisms:
+
+* The host request item cannot be suppressed from worker code or from `host.json` (`Host.Results` controls log traces, not the request item; verified 2026-06-09). The only lever is a Log Analytics workspace transformation: `source | where tostring(Properties.Source) == 'WorkerMiddleware' or isempty(Url)`.
+* `host.json` `samplingSettings.excludedTypes: "Request;Event"` only applies to the host process. The worker's `AddApplicationInsightsTelemetryWorkerService` runs its own adaptive sampling; before 2026-08-23 worker request rows carried `ItemCount` 2–5 on weekdays (≈30 % of individual requests missing, `count()` undercounting by the same share — `sum(ItemCount)` matched the host copy to 1.00 per day). Re-configuring the worker sampling chain did not take effect in the isolated worker (see the L4 note in `Program.cs`). The per-item opt-out does: the middleware sets `SamplingPercentage = 100` on its own `RequestTelemetry`, and the SDK sampling processor passes through every item whose sampling percentage is already set.
+
+## Dependency telemetry — keep only signal
+
+`StorageDependencyFilterProcessor` (worker pipeline) drops:
+
+* successful Azure Storage calls (Table/Queue/Blob) in both shapes — the HTTP shape (`Azure table`, target `*.table.core.windows.net`) and the Azure SDK ActivitySource shape (`InProc | Microsoft.Tables`, `InProc | Microsoft.Storage`);
+* storage calls with an **expected outcome**: `404` (point-read miss — EventTypeIndex, BlockedDevices lookups), `412` (ETag precondition), `409` (idempotent insert conflict). HTTP-shape rows carry the code in `ResultCode`; InProc rows only in `Properties["Error"]` (`Status: 404 (`). Before this filter these were 100 % of the remaining "failed" storage rows (~540 MB/week, two rows per call);
+* successful Azure SignalR Service REST calls the worker makes itself (`*.service.signalr.net`).
+
+The bulk of SignalR REST traffic (group add/remove per connection, ~470 MB/week) is emitted by the **host** process for the SignalR output binding (`SDKVersion rdddsc:*`) and never passes the worker pipeline — live-verified after the 2026-08-23 deploy (worker rows: 0, host rows: ~560/h). Those are removed by the second data flow of the same workspace transformation: `AppDependencies | where not(Success == true and Target endswith '.service.signalr.net')`.
+
+Everything else passes: storage 429/5xx/auth/timeouts, failed SignalR calls, all HTTP/Graph/NVD/MSRC dependencies, the worker's own `InProc Invoke` span, and every non-dependency item.
+
+## Measured baseline (7 days before the change, `_BilledSize`, `autopilotmonitor-api-eu`)
+
+| Table | Rows | MB | After |
+|---|---|---|---|
+| AppDependencies | 976k | 1248 | ≈ 240 (SignalR + expected-outcome rows gone) |
+| AppRequests host copy | 274k | 323 | ≈ 0 (only timer/queue rows) |
+| AppRequests worker copy | 189k | 208 | ≈ 300 (now 100 % of requests) |
+| AppTraces | 142k | 137 | unchanged |
+| AppEvents | 6k | 4 | unchanged — not worth touching |
+
+Net ≈ −1.3 GB/week with MORE usable data (every request now has tenant context).
+
+# Examples
+
+Counting requests (exact since 2026-08-23; for older data replace `count()` with `sum(itemCount)`):
+
+```kql
+requests
+| where customDimensions.Source == 'WorkerMiddleware'
+| summarize total=count(), failed=countif(success == false) by name
+```
+
+Real storage trouble only:
+
+```kql
+dependencies
+| where type startswith 'InProc | Microsoft.Tables' or target endswith '.table.core.windows.net'
+| summarize count() by resultCode, tostring(customDimensions.Error)
+```
+
+Post-deploy verification of the per-item sampling bypass (must be `ItemCount == 1` only):
+
+```kql
+AppRequests
+| where TimeGenerated > ago(1h) and tostring(Properties.Source) == 'WorkerMiddleware'
+| summarize rows=count() by ItemCount
+```
+
+# Citations
+
+* `src/Backend/AutopilotMonitor.Functions/Middleware/RequestTelemetryMiddleware.cs` — `SamplingPercentage = 100` before `TrackRequest`.
+* `src/Backend/AutopilotMonitor.Functions/Telemetry/StorageDependencyFilterProcessor.cs` + `StorageDependencyFilterProcessorTests.cs` — the filter contract.
+* `src/Backend/AutopilotMonitor.Functions/Program.cs` — the failed L4 worker-sampling attempt and why it must not be retried.
+* `infra/appinsights-workspace-transforms.bicep` — the AppRequests transformation DCR, apply/verify/rollback commands.
+* `.claude/commands/backend-logs.md`, `session-debug.md` — query conventions.
@@ -45,6 +45,7 @@ https://docs.autopilotmonitor.com (separate repository).

 # Backend

+* [Telemetry Ingest Shaping](backend/telemetry-ingest-shaping.md) - One canonical App Insights request record (worker copy with tenant context, per-item sampling bypass; host HTTP duplicate dropped by a workspace transformation DCR), expected storage outcomes 404/412/409 and successful SignalR REST calls filtered in the worker, and the count()/sum(itemCount) query rules that follow.
 * [Table Schema Sentinel](backend/table-schema-sentinel.md) - Startup no longer issues one CreateTableIfNotExists per table: a SHA-256 over TableNames.All in AdminConfiguration gates the full pass (derived, never hand-maintained), the SessionsIndex backfill runs only after a full pass behind a conditional-insert claim, and daily maintenance repairs out-of-band deletions.
 * [Business Timestamps](backend/business-timestamps.md) - Why the Azure Tables system Timestamp is never authoritative (migrations reset it), and the OccurredUtc + RowKey-decode compensation for AuditLogs/OpsEvents/Events after the 2026-07-18 migration.
 * [Send-Time Frame Separation](backend/send-time-frame-separation.md) - One device-clock send timestamp per upload batch (X-Send-Time-Utc → SentAt on Events) splits ReceivedAt − OccurredUtc into pure spool delay and pure device-vs-server clock offset; the clock_skew condition measures SentAt batches directly (spool-immune), the CMTrace tripwire deliberately stays on its differential math.
@@ -2,6 +2,7 @@

 ## 2026-08-23

+* **New**: `backend/telemetry-ingest-shaping.md` — `RequestTelemetryMiddleware` sets `SamplingPercentage=100` on its own item (worker request rows were adaptively sampled, ItemCount 2–5 on weekdays, despite host.json `excludedTypes`), the redundant host HTTP request copy is dropped by a workspace transformation DCR (`infra/appinsights-workspace-transforms.bicep`, timer/queue rows kept), and `StorageDependencyFilterProcessor` now also drops expected storage outcomes (404/412/409, both InProc and HTTP shapes) and successful SignalR REST calls; the host-emitted SignalR binding calls (bulk of the volume, SDK `rdddsc`) are dropped by a second `AppDependencies` flow in the same DCR. Measured ≈ −1.3 GB/week.
 * **Update**: `backend/table-schema-sentinel.md` — "Measuring it": `StartupTelemetryService` emits `BackendStartupMs` (process start → ApplicationStarted) and `BackendTableInitMs` as App Insights custom metrics with version/commit/fullPass dimensions; the MCP server writes one `{"type":"boot","readyMs"}` stderr line per start (query in `.claude/commands/mcp-telemetry.md` §A2).
 * **New**: `backend/table-schema-sentinel.md` — startup table initialization gated by a schema sentinel (SHA-256 over `TableNames.All` in `AdminConfiguration`/`SchemaSentinel`/`tables`): matching sentinel ⇒ zero `CreateTableIfNotExists` calls on a cold start, missing/stale ⇒ one full pass + sentinel rewrite. SessionsIndex startup backfill only after a full pass, `IsSessionIndexEmptyAsync` no longer treats errors as "empty", conditional-insert claim so one scaled-out instance backfills. Daily maintenance runs `IStorageInitializer.EnsureAllAsync` to repair out-of-band table deletions.
 * **Update**: `backend/verdict-calibration.md` — MCP `get_verdict_calibration` response shaping: per-row trend denominators dropped, explicit nulls, `minSharePct` / `top` with an `omitted` report.
@@ -0,0 +1,78 @@
+// Autopilot-Monitor — Log Analytics workspace transformation (ingestion-time filter)
+//
+// Two ingestion-time filters. (2) Successful Azure SignalR REST dependencies emitted by the
+// HOST process for the SignalR output binding (SDKVersion rdddsc:*) — the worker-side
+// StorageDependencyFilterProcessor never sees those. Failed SignalR calls are kept.
+// (1) Drops the redundant HOST-side HTTP request record from AppRequests so that exactly one
+// canonical request row remains: the worker-side copy emitted by RequestTelemetryMiddleware
+// (Properties.Source == 'WorkerMiddleware', carries TenantId/UserId/CorrelationId and is
+// sampling-exempt per item). Non-HTTP host invocations (timer/queue triggers) have an empty
+// Url and are kept — the worker middleware never sees them.
+//
+// Why ingestion-time: the host request item is emitted by the Functions host process and
+// cannot be suppressed from worker code or host.json (verified 2026-06-09, see Program.cs).
+// Live numbers 2026-08-23 (7d): host copy 274k rows / 323 MB, worker copy 189k rows / 208 MB.
+//
+// Cost note: transformation filtering is free up to 50% of the table's incoming volume; the
+// portion beyond that (here ~11% of AppRequests) is billed at the (cheap) data-processing rate.
+//
+// Apply (two steps; the link is a CLI update = GET+PUT, which keeps retention/SKU intact):
+//   az deployment group create -g rg-autopilotmonitor-prd-gwc \
+//     --template-file infra/appinsights-workspace-transforms.bicep \
+//     --query properties.outputs.dcrId.value -o tsv
+//   az monitor log-analytics workspace update -g rg-autopilotmonitor-prd-gwc -n AutopilotMonitor \
+//     --data-collection-rule <dcrId-from-above>
+//
+// Verify after ~20 min (ingestion latency):
+//   AppRequests | where TimeGenerated > ago(15m) | where AppRoleName == 'autopilotmonitor-api-eu'
+//   | summarize rows=count() by source=tostring(Properties.Source), hasUrl=isnotempty(Url)
+//   → only (WorkerMiddleware, true) and ('', false) buckets remain.
+//
+// Rollback: az monitor log-analytics workspace update -g <rg> -n AutopilotMonitor
+//   --set properties.defaultDataCollectionRuleResourceId=null   (then delete the DCR)
+
+@description('Log Analytics workspace that backs the workspace-based Application Insights resources.')
+param workspaceName string = 'AutopilotMonitor'
+
+param location string = resourceGroup().location
+
+resource workspace 'Microsoft.OperationalInsights/workspaces@2023-09-01' existing = {
+  name: workspaceName
+}
+
+resource transforms 'Microsoft.Insights/dataCollectionRules@2023-03-11' = {
+  name: 'dcr-autopilotmonitor-ws-transforms'
+  location: location
+  kind: 'WorkspaceTransforms'
+  properties: {
+    description: 'AppRequests: keep the enriched worker request record (Properties.Source == WorkerMiddleware) and non-HTTP host invocations (empty Url); drop the host HTTP duplicate.'
+    destinations: {
+      logAnalytics: [
+        {
+          workspaceResourceId: workspace.id
+          name: 'ws'
+        }
+      ]
+    }
+    dataFlows: [
+      {
+        streams: [ 'Microsoft-Table-AppRequests' ]
+        destinations: [ 'ws' ]
+        transformKql: 'source | where tostring(Properties.Source) == \'WorkerMiddleware\' or isempty(Url)'
+      }
+      {
+        // Successful Azure SignalR REST calls (group add/remove per connection) are emitted by
+        // the HOST process for the SignalR output binding (SDKVersion rdddsc:*), so the worker's
+        // StorageDependencyFilterProcessor never sees them. Failed SignalR calls are kept.
+        streams: [ 'Microsoft-Table-AppDependencies' ]
+        destinations: [ 'ws' ]
+        transformKql: 'source | where not(Success == true and Target endswith \'.service.signalr.net\')'
+      }
+    ]
+  }
+}
+
+// A workspace has exactly one transformation DCR, activated by linking it as the workspace's
+// defaultDataCollectionRuleResourceId — deliberately NOT modelled here (see Apply above).
+
+output dcrId string = transforms.id
@@ -128,6 +128,94 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
            Assert.Equal(4, svc.LastFetchAttempts);
        }

+        // ── Network-link gate (edu.kosh audit 2026-08-24) ───────────────────
+
+        [Fact]
+        public async Task FetchConfig_InitialFetch_WaitsForNetworkLink()
+        {
+            // The initial startup fetch must gate on the link-level signal so the retry
+            // budget isn't burned into a NIC that is provably still down (agent relaunched
+            // at boot, Wi-Fi still associating).
+            var svc = NewService(BackendBehaviour.SuccessImmediately);
+
+            await svc.FetchConfigAsync(retryOnTransientErrors: true);
+
+            Assert.Equal(1, svc.NetworkLinkWaits);
+        }
+
+        [Fact]
+        public async Task FetchConfig_RotatePath_SkipsNetworkLinkGate()
+        {
+            // rotate_config runs mid-session on a proven link and must stay latency-bounded.
+            var svc = NewService(BackendBehaviour.SuccessImmediately);
+
+            await svc.FetchConfigAsync(retryOnTransientErrors: false);
+
+            Assert.Equal(0, svc.NetworkLinkWaits);
+        }
+
+        // ── Post-registration recovery ──────────────────────────────────────
+
+        [Fact]
+        public async Task TryRecover_AfterFailedFetch_AppliesLiveConfig_KeepsFailureRecord()
+        {
+            // The core self-heal contract: registration proved the backend reachable, the
+            // recovery swaps in the live config — but the LastFetch* observability keeps
+            // describing the failed startup fetch so the wire event can report both.
+            var svc = NewService(BackendBehaviour.FailAllThenSucceed);
+            await svc.FetchConfigAsync(retryOnTransientErrors: true);
+            Assert.Equal(RemoteConfigFetchOutcome.UsedDefaults, svc.LastFetchOutcome);
+
+            var recovered = await svc.TryRecoverConfigAsync();
+
+            Assert.NotNull(recovered);
+            Assert.Equal(42, recovered.ConfigVersion);
+            Assert.Equal(42, svc.CurrentConfig.ConfigVersion);
+            Assert.True(svc.RecoveredAfterFetchFailure);
+            // Startup-failure record intact:
+            Assert.Equal(RemoteConfigFetchOutcome.UsedDefaults, svc.LastFetchOutcome);
+            Assert.Equal(1 + RemoteConfigService.InitialFetchRetryBackoff.Length, svc.LastFetchAttempts);
+            Assert.Equal(nameof(HttpRequestException), svc.LastFetchFailureType);
+        }
+
+        [Fact]
+        public async Task TryRecover_AfterSuccessfulFetch_IsNoOp()
+        {
+            var svc = NewService(BackendBehaviour.SuccessImmediately);
+            await svc.FetchConfigAsync(retryOnTransientErrors: true);
+
+            var recovered = await svc.TryRecoverConfigAsync();
+
+            Assert.Null(recovered);
+            Assert.False(svc.RecoveredAfterFetchFailure);
+        }
+
+        [Fact]
+        public async Task TryRecover_BeforeAnyFetch_IsNoOp()
+        {
+            var svc = NewService(BackendBehaviour.SuccessImmediately);
+
+            var recovered = await svc.TryRecoverConfigAsync();
+
+            Assert.Null(recovered);
+            Assert.False(svc.RecoveredAfterFetchFailure);
+        }
+
+        [Fact]
+        public async Task TryRecover_WhenBackendStillFailing_KeepsFallback_NeverThrows()
+        {
+            var svc = NewService(BackendBehaviour.AlwaysTransientFailure);
+            var fallback = await svc.FetchConfigAsync(retryOnTransientErrors: true);
+            Assert.Equal(0, fallback.ConfigVersion);
+
+            var recovered = await svc.TryRecoverConfigAsync();
+
+            Assert.Null(recovered);
+            Assert.False(svc.RecoveredAfterFetchFailure);
+            Assert.Equal(0, svc.CurrentConfig.ConfigVersion); // fallback config untouched
+            Assert.Equal(RemoteConfigFetchOutcome.UsedDefaults, svc.LastFetchOutcome);
+        }
+
        // ── Outcome state defaults ──────────────────────────────────────────

        [Fact]
@@ -145,6 +233,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
            SuccessImmediately,
            AlwaysTransientFailure,
            FailTwiceThenSucceed,
+            /// <summary>All 4 initial attempts fail; the NEXT call (the recovery fetch) succeeds.</summary>
+            FailAllThenSucceed,
            AlwaysAuthFailure401,
            AlwaysReturnNull,
        }
@@ -177,6 +267,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
                _behaviour = behaviour;
            }

+            /// <summary>Number of times the network-link gate seam was awaited.</summary>
+            public int NetworkLinkWaits { get; private set; }
+
            protected override Task<AgentConfigResponse> CallBackendAsync()
            {
                _calls++;
@@ -188,6 +281,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
                        if (_calls < 3)
                            throw new HttpRequestException("simulated cold-start timeout");
                        return Task.FromResult(new AgentConfigResponse { ConfigVersion = 42 });
+                    case BackendBehaviour.FailAllThenSucceed:
+                        if (_calls <= 1 + RemoteConfigService.InitialFetchRetryBackoff.Length)
+                            throw new HttpRequestException("simulated dead network link");
+                        return Task.FromResult(new AgentConfigResponse { ConfigVersion = 42 });
                    case BackendBehaviour.AlwaysTransientFailure:
                        throw new HttpRequestException("simulated network failure");
                    case BackendBehaviour.AlwaysAuthFailure401:
@@ -202,6 +299,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
            // Zero-delay so the suite finishes in ms instead of the live 10/30/60 s schedule.
            protected override Task DelayBetweenAttemptsAsync(TimeSpan delay) => Task.CompletedTask;

+            // Record + skip the real link poll so the suite is deterministic on offline CI.
+            protected override Task WaitForNetworkLinkAsync()
+            {
+                NetworkLinkWaits++;
+                return Task.CompletedTask;
+            }
+
            // In-memory + per-instance cache so a prior test's successful fetch can't
            // bleed into a later test that expects UsedDefaults.
            private AgentConfigResponse? _cached;
@@ -182,6 +182,73 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Program
            Assert.Equal("FromCache", data["outcome"]);
        }

+        // ── Post-registration recovery (Phase 6.5, edu.kosh audit 2026-08-24) ─
+
+        [Fact]
+        public void EmitFetchFailedIfAny_AfterRecovery_MarksRecoveredAndDemotesToInfo()
+        {
+            using var rig = new EmitterRig();
+            // Startup fetch failed (network link not up yet), then the post-registration
+            // recovery fetched ConfigVersion 39. The event must keep the original failure
+            // details AND mark the heal.
+            rig.RemoteConfig.SetFakeOutcome(
+                RemoteConfigFetchOutcome.UsedDefaults,
+                configVersion: 39, // CurrentConfig now holds the recovered config
+                attempts: 4,
+                failureType: "HttpRequestException",
+                failureMessage: "simulated dead network link",
+                recovered: true);
+
+            LifecycleEmitters.EmitRemoteConfigFetchFailedIfAny(
+                rig.Post, rig.AgentConfig, rig.RemoteConfig, rig.Logger);
+
+            var data = AssertSingleEmit(rig.Sink, expectedEventType: "remote_config_fetch_failed");
+            Assert.Equal("UsedDefaults", data["outcome"]); // startup failure record intact
+            Assert.Equal(4, data["attempts"]);
+            Assert.Equal(true, data["recovered"]);
+            Assert.Equal(39, data["recoveredConfigVersion"]);
+            Assert.Equal(nameof(EventSeverity.Info), rig.Sink.Posted[0].Payload!["severity"]);
+        }
+
+        [Fact]
+        public void EmitFetchFailedIfAny_WithoutRecovery_StaysWarning()
+        {
+            using var rig = new EmitterRig();
+            rig.RemoteConfig.SetFakeOutcome(
+                RemoteConfigFetchOutcome.UsedDefaults,
+                configVersion: 0,
+                attempts: 4,
+                failureType: "HttpRequestException",
+                failureMessage: "still down");
+
+            LifecycleEmitters.EmitRemoteConfigFetchFailedIfAny(
+                rig.Post, rig.AgentConfig, rig.RemoteConfig, rig.Logger);
+
+            var data = AssertSingleEmit(rig.Sink, expectedEventType: "remote_config_fetch_failed");
+            Assert.False(data.ContainsKey("recovered"));
+            Assert.False(data.ContainsKey("recoveredConfigVersion"));
+            Assert.Equal(nameof(EventSeverity.Warning), rig.Sink.Posted[0].Payload!["severity"]);
+        }
+
+        [Fact]
+        public void EmitAgentStarted_AfterRecovery_FlagsLiveConfig()
+        {
+            using var rig = new EmitterRig();
+            rig.RemoteConfig.SetFakeOutcome(
+                RemoteConfigFetchOutcome.UsedDefaults,
+                configVersion: 39,
+                recovered: true);
+
+            LifecycleEmitters.EmitAgentStarted(
+                rig.Post, rig.AgentConfig, rig.PreviousExit, "2.0.806", rig.RemoteConfig, rig.Logger);
+
+            var data = AssertSingleEmit(rig.Sink, expectedEventType: "agent_started");
+            Assert.Equal(39, data["configVersion"]);
+            Assert.Equal(true, data["remoteConfigFetched"]); // running on a live-fetched config
+            Assert.Equal("UsedDefaults", data["remoteConfigOutcome"]); // startup outcome stays honest
+            Assert.Equal(true, data["remoteConfigRecovered"]);
+        }
+
        // ── Helpers ─────────────────────────────────────────────────────────

        private static Dictionary<string, object> AssertSingleEmit(
@@ -258,8 +325,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Program
                int attempts = 1,
                string? failureType = null,
                string? failureMessage = null,
-                int? authStatusCode = null)
+                int? authStatusCode = null,
+                bool recovered = false)
            {
+                typeof(RemoteConfigService).GetProperty(nameof(RecoveredAfterFetchFailure))?.SetValue(this, recovered);
                // CurrentConfig is set via the protected SetConfig path — easiest:
                // a real FetchConfigAsync replacement that just stashes a canned config.
                typeof(RemoteConfigService).GetMethod("SetConfig",
@@ -42,6 +42,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
            TimeSpan.FromSeconds(60),
        };

+        /// <summary>
+        /// Upper bound for the pre-fetch network-link wait on the initial startup fetch.
+        /// Deliberately longer than the registration gate's 15s: the fetch runs FIRST after a
+        /// boot-time relaunch, when Wi-Fi may still be associating (802.1X, OOBE), and a fetch
+        /// burned into a dead link strands the whole session on built-in defaults — edu.kosh
+        /// audit 2026-08-24: ~50% of sessions lost their tenant config exactly this way.
+        /// </summary>
+        internal static readonly TimeSpan InitialFetchNetworkLinkWaitMax = TimeSpan.FromSeconds(60);
+
        /// <summary>
        /// Gets the current configuration (thread-safe)
        /// </summary>
@@ -76,6 +85,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
        /// <summary>HTTP status for auth failures (401/403), null otherwise.</summary>
        public int? LastFetchAuthStatusCode { get; private set; }

+        /// <summary>
+        /// True once <see cref="TryRecoverConfigAsync"/> replaced a failed startup fetch with a
+        /// live config. The <c>LastFetch*</c> properties intentionally keep describing the failed
+        /// startup fetch so the wire event can report both the original failure and the recovery.
+        /// </summary>
+        public bool RecoveredAfterFetchFailure { get; private set; }
+
        public RemoteConfigService(BackendApiClient apiClient, string tenantId, AgentLogger logger, EmergencyReporter emergencyReporter = null, DistressReporter distressReporter = null, AuthFailureTracker authFailureTracker = null)
        {
            _apiClient = apiClient ?? throw new ArgumentNullException(nameof(apiClient));
@@ -116,6 +132,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
            LastFetchFailureMessage = null;
            LastFetchAuthStatusCode = null;

+            // Initial startup fetch only: give a still-associating NIC a chance to come up
+            // before the first attempt. Without this the retry budget (10/30/60s) is burned
+            // into a provably dead link — the agent starts at boot, Wi-Fi associates minutes
+            // later, and the session runs on defaults although the backend was never the
+            // problem. The rotate_config path (retryOnTransientErrors:false) runs mid-session
+            // on a proven link and must stay latency-bounded, so it skips the gate.
+            if (retryOnTransientErrors)
+            {
+                try { await WaitForNetworkLinkAsync().ConfigureAwait(false); }
+                catch (Exception ex) { _logger.Debug($"Pre-fetch network wait failed ({ex.Message}) — fetching anyway."); }
+            }
+
            var maxAttempts = retryOnTransientErrors ? InitialFetchRetryBackoff.Length + 1 : 1;
            Exception lastException = null;

@@ -211,6 +239,52 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
            return defaults;
        }

+        /// <summary>
+        /// Single-attempt recovery fetch for a session whose startup fetch failed. Called right
+        /// after session registration succeeded — registration proves the backend is reachable,
+        /// so the config that Phase 4 could not get is one cheap call away instead of the whole
+        /// session being stranded on cache/defaults (only a manual <c>rotate_config</c> action
+        /// could rescue it before this existed).
+        /// <para>
+        /// Unlike <see cref="FetchConfigAsync"/> this does NOT reset the <c>LastFetch*</c>
+        /// observability state — those keep describing the failed startup fetch so
+        /// <c>remote_config_fetch_failed</c> can report the original failure AND the recovery.
+        /// Returns the recovered config, or null when there is nothing to recover (startup fetch
+        /// succeeded / never ran) or the recovery attempt itself failed. Never throws.
+        /// </para>
+        /// </summary>
+        public async Task<AgentConfigResponse> TryRecoverConfigAsync()
+        {
+            if (LastFetchOutcome != RemoteConfigFetchOutcome.FromCache &&
+                LastFetchOutcome != RemoteConfigFetchOutcome.UsedDefaults)
+                return null;
+
+            try
+            {
+                _logger.Info("Retrying remote config fetch after successful session registration...");
+                var config = await CallBackendAsync().ConfigureAwait(false);
+                if (config == null)
+                {
+                    _logger.Warning("Config recovery fetch returned empty response — keeping fallback config.");
+                    return null;
+                }
+
+                _logger.Info($"Remote config recovered: ConfigVersion={config.ConfigVersion}, " +
+                             $"GatherRules={config.GatherRules?.Count ?? 0}");
+                LogCollectorSettings(config.Collectors);
+                SetConfig(config);
+                CacheConfig(config);
+                _authFailureTracker?.RecordSuccess();
+                RecoveredAfterFetchFailure = true;
+                return config;
+            }
+            catch (Exception ex)
+            {
+                _logger.Warning($"Config recovery fetch failed ({ex.Message}) — keeping fallback config.");
+                return null;
+            }
+        }
+
        private void SetConfig(AgentConfigResponse config)
        {
            lock (_configLock)
@@ -358,6 +432,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
        protected virtual Task DelayBetweenAttemptsAsync(TimeSpan delay)
            => Task.Delay(delay);

+        /// <summary>
+        /// Awaits the pre-fetch network-link gate (initial startup fetch only). Production
+        /// polls the link-level signal via <see cref="NetworkLinkWait"/> bounded by
+        /// <see cref="InitialFetchNetworkLinkWaitMax"/>; tests subclass to record the call
+        /// and return immediately.
+        /// </summary>
+        protected virtual Task WaitForNetworkLinkAsync()
+            => NetworkLinkWait.WaitAsync(_logger, InitialFetchNetworkLinkWaitMax, "Remote config fetch");
+
        public void Dispose()
        {
            // Nothing to dispose
@@ -0,0 +1,38 @@
+using System;
+using System.Net.NetworkInformation;
+using System.Threading.Tasks;
+using AutopilotMonitor.Agent.V2.Core.Logging;
+
+namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
+{
+    /// <summary>
+    /// Boot-time NIC grace shared by every first-contact backend call (remote-config fetch,
+    /// session registration, emergency-break reporter pattern): polls the cheap link-level
+    /// signal once per second for at most <paramref name="maxWait"/>. It cannot prove backend
+    /// reachability — the callers' retry loops handle that — it only keeps retry budgets from
+    /// being burned into a link that is provably still down (BootTrigger relaunch after a
+    /// mid-enrollment reboot, Wi-Fi still associating; tenant aebdce78 audits 2026-08-23/24).
+    /// Free on the normal path — a live link returns immediately. Probe errors end the wait,
+    /// never the caller's operation.
+    /// </summary>
+    internal static class NetworkLinkWait
+    {
+        internal static async Task WaitAsync(AgentLogger logger, TimeSpan maxWait, string context)
+        {
+            if (NetworkInterface.GetIsNetworkAvailable()) return;
+
+            logger?.Info($"{context}: no network link yet — waiting up to {maxWait.TotalSeconds:F0}s.");
+            var deadline = DateTime.UtcNow + maxWait;
+            while (DateTime.UtcNow < deadline)
+            {
+                await Task.Delay(1000).ConfigureAwait(false);
+                if (NetworkInterface.GetIsNetworkAvailable())
+                {
+                    logger?.Info($"{context}: network link is up.");
+                    return;
+                }
+            }
+            logger?.Warning($"{context}: still no network link after {maxWait.TotalSeconds:F0}s — attempting anyway.");
+        }
+    }
+}
@@ -1,5 +1,4 @@
 using System;
-using System.Net.NetworkInformation;
 using System.Threading.Tasks;
 using AutopilotMonitor.Agent.V2.Core.Configuration;
 using AutopilotMonitor.Agent.V2.Core.Logging;
@@ -165,29 +164,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
            => Task.Delay(((int)Math.Pow(2, attempt)) * 1000);

        /// <summary>
-        /// Boot-time NIC grace (same pattern as the emergency-break reporter): polls the cheap
-        /// link-level signal once per second for at most <see cref="NetworkLinkWaitMax"/>. It
-        /// cannot prove backend reachability — the retry loop handles that — it only keeps the
-        /// first attempts from being burned into a link that is provably still down. Probe
+        /// Boot-time NIC grace before the first registration attempt — shared polling loop in
+        /// <see cref="NetworkLinkWait"/>, bounded by <see cref="NetworkLinkWaitMax"/>. Probe
        /// errors end the wait, never the registration.
        /// </summary>
-        internal static async Task WaitForNetworkLinkAsync(AgentLogger logger)
-        {
-            if (NetworkInterface.GetIsNetworkAvailable()) return;
-
-            logger?.Info($"Session registration: no network link yet — waiting up to {NetworkLinkWaitMax.TotalSeconds:F0}s.");
-            var deadline = DateTime.UtcNow + NetworkLinkWaitMax;
-            while (DateTime.UtcNow < deadline)
-            {
-                await Task.Delay(1000).ConfigureAwait(false);
-                if (NetworkInterface.GetIsNetworkAvailable())
-                {
-                    logger?.Info("Session registration: network link is up.");
-                    return;
-                }
-            }
-            logger?.Warning($"Session registration: still no network link after {NetworkLinkWaitMax.TotalSeconds:F0}s — attempting anyway.");
-        }
+        internal static Task WaitForNetworkLinkAsync(AgentLogger logger)
+            => NetworkLinkWait.WaitAsync(logger, NetworkLinkWaitMax, "Session registration");

        private static SessionRegistration BuildRegistration(
            AgentConfiguration agentConfig,
@@ -274,6 +274,15 @@ namespace AutopilotMonitor.Agent.V2
            {
                return registration.ExitCode;
            }
+
+            // Phase 6.5 (config self-heal): when the Phase 4 fetch ran before the network link
+            // was up, registration has just proven the backend reachable — retry the fetch once
+            // so the session runs on the live tenant config instead of cache/defaults.
+            // See AgentRuntimeConfig.TryRecoverAfterRegistration for what is deliberately
+            // NOT re-run here (kill-signal check, endpoint migration).
+            Runtime.AgentRuntimeConfig.TryRecoverAfterRegistration(
+                bootstrap.AgentConfig, auth, runtimeConfig, GetAgentVersion(), consoleMode, logger);
+
            // Phase 7+8 (orchestrator + shutdown lifecycle + onIngressReady wiring +
            // initial signal posts + analyzer/probes startup + shutdown.Wait + finally)
            // is encapsulated in AgentRuntimeHost. See Runtime/AgentRuntimeHost.cs.
@@ -69,6 +69,63 @@ namespace AutopilotMonitor.Agent.V2.Runtime
                }
            }

+            var configMergeResult = ApplyFetchedConfig(agentConfig, remoteConfig, auth, agentVersion, consoleMode, logger);
+
+            return new RuntimeConfigBundle(remoteConfigService, remoteConfig, configMergeResult, pendingMigrationTarget: null);
+        }
+
+        /// <summary>
+        /// Phase 6.5 — config self-heal. When the Phase 4 fetch ran before the network link was
+        /// up (agent relaunched at boot, Wi-Fi still associating — edu.kosh audit 2026-08-24:
+        /// ~50% of sessions ran on ConfigVersion=0 exactly this way), a successful session
+        /// registration has just proven the backend reachable. Retry the fetch once and apply
+        /// the live tenant config instead of stranding the whole session on cache/defaults.
+        /// <para>
+        /// Deliberately NOT re-run here: the kill-signal check (post-registration kills arrive
+        /// via the telemetry-channel control signals on every upload response) and endpoint
+        /// migration (registration already succeeded against the current base URL; a migration
+        /// signal is honoured at the next agent start). Never throws — recovery is strictly
+        /// best-effort on top of an already-degraded-but-working fallback.
+        /// </para>
+        /// </summary>
+        public static void TryRecoverAfterRegistration(
+            AgentConfiguration agentConfig,
+            BackendAuthBundle auth,
+            RuntimeConfigBundle bundle,
+            string agentVersion,
+            bool consoleMode,
+            AgentLogger logger)
+        {
+            try
+            {
+                var service = bundle?.RemoteConfigService;
+                if (service == null) return;
+
+                var recovered = service.TryRecoverConfigAsync().GetAwaiter().GetResult();
+                if (recovered == null) return;
+
+                var mergeResult = ApplyFetchedConfig(agentConfig, recovered, auth, agentVersion, consoleMode, logger);
+                bundle.ApplyRecovered(recovered, mergeResult);
+            }
+            catch (Exception ex)
+            {
+                logger.Warning($"Post-registration config recovery failed: {ex.Message}");
+            }
+        }
+
+        /// <summary>
+        /// Applies a live-fetched config to the runtime: merge, tracker ceilings, log level,
+        /// integrity SHA + check, bootstrap-config cleanup. Shared by the Phase 4 startup path
+        /// and the post-registration recovery path so both apply identical semantics.
+        /// </summary>
+        private static RemoteConfigMergeResult ApplyFetchedConfig(
+            AgentConfiguration agentConfig,
+            AgentConfigResponse remoteConfig,
+            BackendAuthBundle auth,
+            string agentVersion,
+            bool consoleMode,
+            AgentLogger logger)
+        {
            // Project remote tenant-controlled knobs onto the runtime AgentConfiguration so that
            // downstream consumers (CleanupService, SummaryDialogLauncher, StartupEnvironmentProbes,
            // DiagnosticsPackageService, EnrollmentTerminationHandler, the logger, the watchdog)
@@ -91,7 +148,7 @@ namespace AutopilotMonitor.Agent.V2.Runtime

            TryCleanupBootstrapConfig(agentConfig, agentVersion, logger);

-            return new RuntimeConfigBundle(remoteConfigService, remoteConfig, configMergeResult, pendingMigrationTarget: null);
+            return configMergeResult;
        }

        /// <summary>
@@ -175,10 +232,21 @@ namespace AutopilotMonitor.Agent.V2.Runtime
    internal sealed class RuntimeConfigBundle
    {
        public RemoteConfigService RemoteConfigService { get; }
-        public AgentConfigResponse RemoteConfig { get; }
-        public RemoteConfigMergeResult MergeResult { get; }
+        public AgentConfigResponse RemoteConfig { get; private set; }
+        public RemoteConfigMergeResult MergeResult { get; private set; }
        public string PendingMigrationTarget { get; }

+        /// <summary>
+        /// Swaps in the config recovered by <see cref="AgentRuntimeConfig.TryRecoverAfterRegistration"/>
+        /// so Phase 7+8 (orchestrator, collectors, gather rules, IME patterns, the
+        /// unrestricted-mode audit) run on the live tenant config instead of the fallback.
+        /// </summary>
+        public void ApplyRecovered(AgentConfigResponse recoveredConfig, RemoteConfigMergeResult mergeResult)
+        {
+            RemoteConfig = recoveredConfig;
+            MergeResult = mergeResult;
+        }
+
        public RuntimeConfigBundle(
            RemoteConfigService remoteConfigService,
            AgentConfigResponse remoteConfig,
@@ -52,6 +52,7 @@ namespace AutopilotMonitor.Agent.V2.Runtime
                // prior to this, ConfigVersion=0 was only deducible by inferring from
                // the absence of tenant-controlled knobs in the DataJson.
                var outcome = remoteConfigService?.LastFetchOutcome ?? RemoteConfigFetchOutcome.NotAttempted;
+                var recovered = remoteConfigService?.RecoveredAfterFetchFailure == true;
                var configVersion = remoteConfigService?.CurrentConfig?.ConfigVersion ?? 0;

                var data = new Dictionary<string, object>
@@ -66,8 +67,12 @@ namespace AutopilotMonitor.Agent.V2.Runtime
                    { "diagnosticsUploadMode", agentConfig.DiagnosticsUploadMode ?? "Off" },
                    { "previousExitType", previousExit?.ExitType ?? "unknown" },
                    { "unrestrictedMode", agentConfig.UnrestrictedMode },
+                    // remoteConfigFetched answers "is the agent running on a live-fetched
+                    // config" — true also when the post-registration recovery replaced a
+                    // failed startup fetch (remoteConfigOutcome then still names the startup
+                    // failure; remoteConfigRecovered marks the heal).
                    { "configVersion", configVersion },
-                    { "remoteConfigFetched", outcome == RemoteConfigFetchOutcome.Succeeded },
+                    { "remoteConfigFetched", outcome == RemoteConfigFetchOutcome.Succeeded || recovered },
                    { "remoteConfigOutcome", outcome.ToString() },
                    // WinRT OOBE state sampled at this start. Together with previousExitType
                    // this makes late starts provable: "completed" on a first_run means OOBE
@@ -75,6 +80,9 @@ namespace AutopilotMonitor.Agent.V2.Runtime
                    { "oobeStateAtAgentStart", OobeStateReader.Read() },
                };

+                if (recovered)
+                    data["remoteConfigRecovered"] = true;
+
                if (!string.IsNullOrEmpty(previousExit?.CrashExceptionType))
                    data["previousCrashException"] = previousExit.CrashExceptionType;

@@ -135,12 +143,25 @@ namespace AutopilotMonitor.Agent.V2.Runtime
                    ? $"Remote config fetch failed after {remoteConfigService.LastFetchAttempts} attempt(s); using cached config."
                    : $"Remote config fetch failed after {remoteConfigService.LastFetchAttempts} attempt(s); running with built-in defaults (ConfigVersion=0).";

+                // Post-registration recovery (Phase 6.5) replaced the fallback with a live
+                // fetch before this emitter ran: keep the failure details (the network-late
+                // condition is worth surfacing) but mark the heal and demote to Info — the
+                // session is NOT degraded.
+                var recovered = remoteConfigService.RecoveredAfterFetchFailure;
+                if (recovered)
+                {
+                    var recoveredVersion = remoteConfigService.CurrentConfig?.ConfigVersion ?? 0;
+                    data["recovered"] = true;
+                    data["recoveredConfigVersion"] = recoveredVersion;
+                    msg += $" Recovered after session registration (ConfigVersion={recoveredVersion}).";
+                }
+
                post.Emit(new EnrollmentEvent
                {
                    SessionId = agentConfig.SessionId,
                    TenantId = agentConfig.TenantId,
                    EventType = SharedConstants.EventTypes.RemoteConfigFetchFailed,
-                    Severity = EventSeverity.Warning,
+                    Severity = recovered ? EventSeverity.Info : EventSeverity.Warning,
                    Source = "Agent",
                    Phase = EnrollmentPhase.Unknown,
                    Message = msg,
@@ -0,0 +1,99 @@
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using AutopilotMonitor.Functions.Services;
+using Xunit;
+
+namespace AutopilotMonitor.Functions.Tests;
+
+/// <summary>
+/// <see cref="TableStorageService.BuildCveIndexEntities"/> collapses report findings to one
+/// CveIndex entity per CVE. Production reports routinely list the same CVE under many
+/// findings (one RD CVE across 14 installed RD components), which previously produced
+/// 14 concurrent upserts racing on the same PK/RK.
+/// </summary>
+public class CveIndexEntityMergeTests
+{
+    private const string Tenant = "11111111-1111-1111-1111-111111111111";
+    private const string Session = "22222222-2222-2222-2222-222222222222";
+
+    private static Dictionary<string, object> Finding(string software, string risk, params Dictionary<string, object>[] vulns)
+        => new()
+        {
+            ["softwareName"] = software,
+            ["riskLevel"] = risk,
+            ["vulnerabilities"] = vulns.ToList(),
+        };
+
+    private static Dictionary<string, object> Vuln(string cve, double score, string severity, bool kev = false)
+        => new()
+        {
+            ["cveId"] = cve,
+            ["cvssScore"] = score,
+            ["cvssSeverity"] = severity,
+            ["isKev"] = kev,
+        };
+
+    [Fact]
+    public void SameCveAcrossManyFindings_CollapsesToOneEntity()
+    {
+        var findings = Enumerable.Range(0, 14)
+            .Select(i => Finding($"RD Component {i}", "high", Vuln("CVE-2025-29966", 8.8, "HIGH")))
+            .ToList();
+
+        var entities = TableStorageService.BuildCveIndexEntities(Tenant, Session, findings);
+
+        var e = Assert.Single(entities);
+        Assert.Equal($"{Tenant}_CVE-2025-29966", e.PartitionKey);
+        Assert.Equal(Session, e.RowKey);
+        Assert.Equal("RD Component 0", e.GetString("SoftwareName")); // tie keeps report order
+    }
+
+    [Fact]
+    public void Merge_HighestScoreWins_KevOrs_RiskMaxes()
+    {
+        var findings = new List<Dictionary<string, object>>
+        {
+            Finding("App A", "medium", Vuln("CVE-1", 5.0, "MEDIUM", kev: true)),
+            Finding("App B", "critical", Vuln("CVE-1", 9.1, "CRITICAL")),
+            Finding("App C", "low", Vuln("CVE-1", 3.0, "LOW")),
+        };
+
+        var e = Assert.Single(TableStorageService.BuildCveIndexEntities(Tenant, Session, findings));
+
+        Assert.Equal(9.1, e.GetDouble("CvssScore"));
+        Assert.Equal("CRITICAL", e.GetString("CvssSeverity"));
+        Assert.Equal("App B", e.GetString("SoftwareName"));
+        Assert.True(e.GetBoolean("IsKev"));
+        Assert.Equal("critical", e.GetString("OverallRisk"));
+    }
+
+    [Fact]
+    public void DistinctCves_StayDistinct_AndInvalidEntriesAreSkipped()
+    {
+        var findings = new List<Dictionary<string, object>>
+        {
+            Finding("App A", "high", Vuln("CVE-1", 7.0, "HIGH"), Vuln("CVE-2", 6.0, "MEDIUM")),
+            Finding("App B", "high", new Dictionary<string, object> { ["cvssScore"] = 9.0 }), // no cveId
+            new() { ["softwareName"] = "No vulns" },
+        };
+
+        var entities = TableStorageService.BuildCveIndexEntities(Tenant, Session, findings);
+
+        Assert.Equal(2, entities.Count);
+        Assert.Contains(entities, x => x.GetString("CveId") == "CVE-1");
+        Assert.Contains(entities, x => x.GetString("CveId") == "CVE-2");
+    }
+
+    [Fact]
+    public void CveIdComparison_IsCaseInsensitive()
+    {
+        var findings = new List<Dictionary<string, object>>
+        {
+            Finding("A", "high", Vuln("CVE-2024-1", 7.0, "HIGH")),
+            Finding("B", "high", Vuln("cve-2024-1", 7.0, "HIGH")),
+        };
+
+        Assert.Single(TableStorageService.BuildCveIndexEntities(Tenant, Session, findings));
+    }
+}
@@ -48,11 +48,74 @@ public class StorageDependencyFilterProcessorTests
        Assert.Empty(next.Received);
    }

-    [Fact]
-    public void FailedStorageDependency_IsKept()
+    [Theory]
+    [InlineData("429")]
+    [InlineData("500")]
+    [InlineData("503")]
+    [InlineData("403")]
+    [InlineData("")] // unknown outcome (timeout/network) — keep
+    public void FailedStorageDependency_IsKept(string resultCode)
    {
        var (processor, next) = Build();
        var dep = Dep("myacct.queue.core.windows.net", success: false);
+        dep.ResultCode = resultCode;
+        processor.Process(dep);
+        Assert.Same(dep, Assert.Single(next.Received));
+    }
+
+    // Expected outcomes (point-read miss, ETag precondition, insert conflict) are normal control
+    // flow — live-verified as 100% of the remaining "failed" storage rows. Dropped.
+    [Theory]
+    [InlineData("404")]
+    [InlineData("412")]
+    [InlineData("409")]
+    public void ExpectedStorageOutcome_HttpShape_IsDropped(string resultCode)
+    {
+        var (processor, next) = Build();
+        var dep = Dep("myacct.table.core.windows.net", success: false);
+        dep.ResultCode = resultCode;
+        processor.Process(dep);
+        Assert.Empty(next.Received);
+    }
+
+    [Theory]
+    [InlineData("Azure.RequestFailedException: The specified resource does not exist.\nRequestId:x\nStatus: 404 (Not Found)\nErrorCode: ResourceNotFound")]
+    [InlineData("Azure.RequestFailedException: The update condition specified in the request was not satisfied.\nStatus: 412 (Precondition Failed)\nErrorCode: UpdateConditionNotSatisfied")]
+    [InlineData("Azure.RequestFailedException: The specified entity already exists.\nStatus: 409 (Conflict)\nErrorCode: EntityAlreadyExists")]
+    public void ExpectedStorageOutcome_InProcShape_IsDropped(string error)
+    {
+        // InProc spans have no ResultCode — only the exception text in Properties["Error"].
+        var (processor, next) = Build();
+        var dep = Dep(target: "TableClient.GetEntity", success: false, type: "InProc | Microsoft.Tables");
+        dep.Properties["Error"] = error;
+        processor.Process(dep);
+        Assert.Empty(next.Received);
+    }
+
+    [Fact]
+    public void ThrottledInProcStorageDependency_IsKept()
+    {
+        var (processor, next) = Build();
+        var dep = Dep(target: "TableClient.SubmitTransaction", success: false, type: "InProc | Microsoft.Tables");
+        dep.Properties["Error"] = "Azure.RequestFailedException: Too many requests.\nStatus: 429 (Too Many Requests)\nErrorCode: ServerBusy";
+        processor.Process(dep);
+        Assert.Same(dep, Assert.Single(next.Received));
+    }
+
+    [Fact]
+    public void SuccessfulSignalRRestCall_IsDropped()
+    {
+        var (processor, next) = Build();
+        processor.Process(Dep("autopilotmonitor-eu.service.signalr.net", success: true, data: "POST /api/hubs/autopilotmonitor/groups/t-x/connections/y", type: "HTTP"));
+        Assert.Empty(next.Received);
+    }
+
+    [Fact]
+    public void FailedSignalRRestCall_IsKept()
+    {
+        var (processor, next) = Build();
+        var dep = Dep("autopilotmonitor-eu.service.signalr.net", success: false, type: "HTTP");
+        dep.ResultCode = "500";
        processor.Process(dep);
        Assert.Same(dep, Assert.Single(next.Received));
    }
@@ -1,5 +1,6 @@
 using System.Diagnostics;
 using Microsoft.ApplicationInsights;
+using Microsoft.ApplicationInsights.Channel;
 using Microsoft.ApplicationInsights.DataContracts;
 using Microsoft.Azure.Functions.Worker;
 using Microsoft.Azure.Functions.Worker.Middleware;
@@ -140,6 +141,15 @@ public class RequestTelemetryMiddleware : IFunctionsWorkerMiddleware
            if (caughtException != null)
                requestTelemetry.Properties["ExceptionType"] = caughtException.GetType().Name;

+            // Bypass the worker's adaptive sampling for this item. The SDK sampling processor
+            // passes through any item whose SamplingPercentage is already set, so this is the
+            // per-item opt-out that host.json "excludedTypes" (host process only) cannot provide.
+            // Live-verified 2026-08-23 BEFORE this change: worker request rows carried
+            // ItemCount 2-5 on weekdays (~30% of individual requests missing), so count()-based
+            // queries undercounted. This copy is the single canonical request record — the host
+            // duplicate is dropped by a workspace DCR transformation on AppRequests.
+            ((ISupportSampling)requestTelemetry).SamplingPercentage = 100;
+
            try
            {
                _telemetryClient.TrackRequest(requestTelemetry);
@@ -71,6 +71,10 @@ builder.Services
 // unchanged host duplicate). The host request item is emitted by the HOST process and cannot be
 // dropped from worker code or by host.json "Host.Results" level. Re-attempt only with a mechanism
 // verified against the isolated-worker telemetry pipeline — do not re-add the manual chain.
+// 2026-08-23: solved differently — RequestTelemetryMiddleware sets SamplingPercentage=100 on its
+// own item (per-item sampling bypass), and the host duplicate is removed by a workspace DCR
+// transformation on AppRequests (keeps rows with Properties.Source=='WorkerMiddleware' plus
+// non-HTTP host rows with empty Url, i.e. timer/queue invocations). See docs/backend/telemetry-ingest-shaping.md.

 // Drop successful Azure Storage dependencies (Table/Queue/Blob) from the worker telemetry
 // pipeline to curb AppDependencies ingestion cost — this backend is storage-I/O heavy and those
@@ -245,61 +245,43 @@ namespace AutopilotMonitor.Functions.Services
        /// <summary>
        /// Upserts CVE index entries so sessions can be searched by CVE identifier.
        /// PartitionKey = {tenantId}_{cveId}, RowKey = sessionId
-        /// Uses individual parallel upserts (not batch transactions) because PK differs per CVE.
+        /// One CVE commonly appears under several findings of the same report (e.g. one
+        /// Remote Desktop CVE across 14 installed RD components), so occurrences are merged
+        /// per CVE first — otherwise concurrent upserts race on the same entity and the
+        /// winner is arbitrary. Writes go out with bounded parallelism (PK differs per CVE,
+        /// so batch transactions are not an option). Per-entity failures are counted, not
+        /// fatal — the index is a search aid, the report itself is stored separately.
        /// </summary>
        public async Task UpsertCveIndexEntriesAsync(string tenantId, string sessionId, List<Dictionary<string, object>> findings)
        {
            try
            {
+                var entities = BuildCveIndexEntities(tenantId, sessionId, findings);
+                if (entities.Count == 0) return;
+
                var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.CveIndex);
-                var tasks = new List<Task>();
+                var failed = 0;

-                foreach (var finding in findings)
-                {
-                    var softwareName = finding.TryGetValue("softwareName", out var sn) ? sn?.ToString() ?? "" : "";
-                    var overallRisk = finding.TryGetValue("riskLevel", out var rl) ? rl?.ToString() ?? "" : "";
-
-                    if (!finding.TryGetValue("vulnerabilities", out var vulnsObj) || vulnsObj == null) continue;
-                    if (vulnsObj is not System.Collections.IEnumerable vulnList) continue;
-
-                    foreach (var vulnObj in vulnList)
+                await Parallel.ForEachAsync(
+                    entities,
+                    new ParallelOptions { MaxDegreeOfParallelism = 8 },
+                    async (entity, ct) =>
                    {
-                        Dictionary<string, object>? vuln = null;
-                        if (vulnObj is Dictionary<string, object> vd)
-                            vuln = vd;
-
-                        if (vuln == null) continue;
-
-                        var cveId = vuln.TryGetValue("cveId", out var cid) ? cid?.ToString() : null;
-                        if (string.IsNullOrEmpty(cveId)) continue;
-
-                        var partitionKey = $"{tenantId}_{cveId}";
-                        double cvssScore = 0;
-                        try { if (vuln.TryGetValue("cvssScore", out var cs) && cs != null) cvssScore = Convert.ToDouble(cs); } catch { }
-
-                        var cvssSeverity = vuln.TryGetValue("cvssSeverity", out var csvs) ? csvs?.ToString() ?? "" : "";
-                        bool isKev = false;
-                        try { if (vuln.TryGetValue("isKev", out var ik) && ik is bool ikb) isKev = ikb; } catch { }
-
-                        var entity = new TableEntity(partitionKey, sessionId)
+                        try
                        {
-                            ["SessionId"] = sessionId,
-                            ["TenantId"] = tenantId,
-                            ["CveId"] = cveId,
-                            ["SoftwareName"] = softwareName,
-                            ["CvssScore"] = cvssScore,
-                            ["CvssSeverity"] = cvssSeverity,
-                            ["IsKev"] = isKev,
-                            ["OverallRisk"] = overallRisk,
-                            ["DetectedAt"] = DateTime.UtcNow,
-                        };
+                            await tableClient.UpsertEntityAsync(entity, cancellationToken: ct);
+                        }
+                        catch (Exception ex)
+                        {
+                            Interlocked.Increment(ref failed);
+                            _logger.LogWarning(ex, "Failed to upsert CveIndex entry {CveId} for session {SessionId}",
+                                entity.GetString("CveId"), sessionId);
+                        }
+                    });

-                        tasks.Add(tableClient.UpsertEntityAsync(entity));
-                    }
-                }
-
-                if (tasks.Count > 0)
-                    await Task.WhenAll(tasks);
+                if (failed > 0)
+                    _logger.LogWarning("CveIndex upsert for session {SessionId}: {Failed}/{Total} entries failed",
+                        sessionId, failed, entities.Count);
            }
            catch (Exception ex)
            {
@@ -307,6 +289,78 @@ namespace AutopilotMonitor.Functions.Services
            }
        }

+        /// <summary>
+        /// Collapses report findings to one entity per CVE. Merge rule: the occurrence with the
+        /// highest CVSS score supplies score/severity/software name (ties keep report order),
+        /// IsKev is OR-ed, OverallRisk is the highest risk level seen across occurrences.
+        /// </summary>
+        internal static List<TableEntity> BuildCveIndexEntities(string tenantId, string sessionId, List<Dictionary<string, object>> findings)
+        {
+            var byCve = new Dictionary<string, TableEntity>(StringComparer.OrdinalIgnoreCase);
+            var now = DateTime.UtcNow;
+
+            foreach (var finding in findings)
+            {
+                var softwareName = finding.TryGetValue("softwareName", out var sn) ? sn?.ToString() ?? "" : "";
+                var overallRisk = finding.TryGetValue("riskLevel", out var rl) ? rl?.ToString() ?? "" : "";
+
+                if (!finding.TryGetValue("vulnerabilities", out var vulnsObj) || vulnsObj == null) continue;
+                if (vulnsObj is not System.Collections.IEnumerable vulnList) continue;
+
+                foreach (var vulnObj in vulnList)
+                {
+                    if (vulnObj is not Dictionary<string, object> vuln) continue;
+
+                    var cveId = vuln.TryGetValue("cveId", out var cid) ? cid?.ToString() : null;
+                    if (string.IsNullOrEmpty(cveId)) continue;
+
+                    double cvssScore = 0;
+                    try { if (vuln.TryGetValue("cvssScore", out var cs) && cs != null) cvssScore = Convert.ToDouble(cs); } catch { }
+
+                    var cvssSeverity = vuln.TryGetValue("cvssSeverity", out var csvs) ? csvs?.ToString() ?? "" : "";
+                    var isKev = vuln.TryGetValue("isKev", out var ik) && ik is bool ikb && ikb;
+
+                    if (byCve.TryGetValue(cveId, out var existing))
+                    {
+                        if (cvssScore > existing.GetDouble("CvssScore"))
+                        {
+                            existing["CvssScore"] = cvssScore;
+                            existing["CvssSeverity"] = cvssSeverity;
+                            existing["SoftwareName"] = softwareName;
+                        }
+                        existing["IsKev"] = existing.GetBoolean("IsKev") == true || isKev;
+                        if (RiskRank(overallRisk) > RiskRank(existing.GetString("OverallRisk")))
+                            existing["OverallRisk"] = overallRisk;
+                        continue;
+                    }
+
+                    byCve[cveId] = new TableEntity($"{tenantId}_{cveId}", sessionId)
+                    {
+                        ["SessionId"] = sessionId,
+                        ["TenantId"] = tenantId,
+                        ["CveId"] = cveId,
+                        ["SoftwareName"] = softwareName,
+                        ["CvssScore"] = cvssScore,
+                        ["CvssSeverity"] = cvssSeverity,
+                        ["IsKev"] = isKev,
+                        ["OverallRisk"] = overallRisk,
+                        ["DetectedAt"] = now,
+                    };
+                }
+            }
+
+            return byCve.Values.ToList();
+        }
+
+        private static int RiskRank(string? risk) => risk?.ToLowerInvariant() switch
+        {
+            "critical" => 4,
+            "high" => 3,
+            "medium" => 2,
+            "low" => 1,
+            _ => 0,
+        };
+
        // ===== SEARCH METHODS =====

        /// <summary>Typeahead only inspects sessions started within this many days (recent-window scan bound).</summary>
@@ -17,8 +17,14 @@ namespace AutopilotMonitor.Functions.Telemetry;
 /// - Only <see cref="DependencyTelemetry"/> is considered; requests, traces, exceptions,
 ///   metrics and all NON-storage dependencies (HTTP, Microsoft Graph, SQL, SignalR, ...) pass
 ///   through untouched.
-/// - FAILED storage calls are KEPT (Success == false): they are rare and are real signal for
-///   troubleshooting throttling / transient faults, so the noise reduction does not blind us.
+/// - FAILED storage calls are KEPT (Success == false) EXCEPT the expected-outcome status codes
+///   404 (point-read miss), 412 (ETag precondition) and 409 (insert conflict): those are normal
+///   control flow for this backend (index lookups, optimistic concurrency, idempotent inserts).
+///   Live-verified 2026-08-23: 100% of the remaining "failed" storage rows were 404/412/409
+///   (~540 MB/week, two rows per call: InProc span + HTTP span). Throttling (429), 5xx, auth
+///   failures and timeouts stay — they are the real signal.
+/// - Successful Azure SignalR REST management calls (group add/remove per connection, ~470 MB/week)
+///   are dropped for the same reason; failed SignalR calls are kept.
 ///
 /// Registered in Program.cs via AddApplicationInsightsTelemetryProcessor so it runs inside the
 /// isolated worker's telemetry pipeline, where the app's own Azure SDK dependencies are tracked.
@@ -46,6 +52,12 @@ public sealed class StorageDependencyFilterProcessor : ITelemetryProcessor
        "InProc | Microsoft.Storage",  // Azure.Storage.Queues + Azure.Storage.Blobs
    };

+    // Azure SignalR Service REST endpoint (group membership management per connection).
+    private const string SignalRServiceSuffix = ".service.signalr.net";
+
+    // Expected storage outcomes that are normal control flow, not failures worth billing.
+    private static readonly string[] ExpectedStorageStatusCodes = { "404", "412", "409" };
+
    private readonly ITelemetryProcessor _next;

    public StorageDependencyFilterProcessor(ITelemetryProcessor next) => _next = next;
@@ -68,17 +80,50 @@ public sealed class StorageDependencyFilterProcessor : ITelemetryProcessor
            return false;
        }

-        // Keep failures — rare, high-value diagnostic signal. Only successful storage chatter is noise.
-        if (dependency.Success == false)
-        {
-            return false;
-        }
-
-        return IsStorageEndpoint(dependency.Target)
+        var isStorage = IsStorageEndpoint(dependency.Target)
            || IsStorageEndpoint(dependency.Data)
            || IsStorageInProcType(dependency.Type);
+
+        if (isStorage)
+        {
+            // Successful storage chatter is noise; so are expected 404/412/409 outcomes.
+            // Every other failure (429, 5xx, auth, timeout) is kept.
+            return dependency.Success != false || IsExpectedStorageOutcome(dependency);
+        }
+
+        // SignalR REST management calls: keep only failures.
+        return dependency.Success != false && IsSignalRService(dependency.Target);
    }

+    private static bool IsExpectedStorageOutcome(DependencyTelemetry dependency)
+    {
+        // HTTP-shaped rows ("Azure table") carry the status in ResultCode; the Azure SDK InProc
+        // span has no ResultCode, only the exception text in Properties["Error"]
+        // ("...Status: 404 (Not Found)...").
+        var code = dependency.ResultCode;
+        if (!string.IsNullOrEmpty(code))
+        {
+            return Array.IndexOf(ExpectedStorageStatusCodes, code) >= 0;
+        }
+
+        if (dependency.Properties.TryGetValue("Error", out var error) && !string.IsNullOrEmpty(error))
+        {
+            foreach (var expected in ExpectedStorageStatusCodes)
+            {
+                if (error.Contains("Status: " + expected + " (", System.StringComparison.Ordinal))
+                {
+                    return true;
+                }
+            }
+        }
+
+        return false;
+    }
+
+    private static bool IsSignalRService(string? target)
+        => !string.IsNullOrEmpty(target)
+           && target.EndsWith(SignalRServiceSuffix, System.StringComparison.OrdinalIgnoreCase);
+
    private static bool IsStorageInProcType(string? type)
    {
        if (string.IsNullOrEmpty(type))
Author	SHA1	Message	Date
Oliver Kieselbach	c26867ced8	fix(agent): network-link gate before initial config fetch + post-registration recovery Root cause (edu.kosh audit 2026-08-24, session ba828038): on boot-time relaunches the agent starts before the Wi-Fi link is up. The Phase 4 config fetch had no network gate (`98f8114d` only guarded Phase 6 registration), burned its 10/30/60s retry budget into a dead link and stranded the whole session on built-in defaults (ConfigVersion=0) — up to ~50% of that tenant's sessions, with no self-heal short of a manual rotate_config action. Not a version regression: rates varied wildly per day within the same agent build; the failed requests never reached the backend. Two complementary layers: - RemoteConfigService gates the initial fetch on the shared NetworkLinkWait poll (60s cap — longer than registration's 15s because the fetch runs first), so the retry budget starts on a live link. rotate_config stays ungated (latency-bound). - Phase 6.5: after registration proves the backend reachable, TryRecoverConfigAsync refetches once and AgentRuntimeConfig.TryRecoverAfterRegistration re-applies the live config (same merge/integrity pipeline as Phase 4). LastFetch* keeps the original failure record; remote_config_fetch_failed reports it with a recovered flag at Info severity, agent_started flags remoteConfigRecovered. Deliberately not re-run in recovery: kill-signal check (telemetry-channel control signals cover post-registration kills) and endpoint migration (next start). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-24 11:49:37 +02:00
Oliver Kieselbach	3af94d9dcd	infra(telemetry): DCR also drops host-emitted successful SignalR dependencies Live-verified after `0d5e2572` deploy: the worker filter removes every 404/412/409 and worker SignalR row, but the SignalR output-binding calls are emitted by the host process (SDK rdddsc) and only reachable at ingestion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-23 19:15:41 +02:00
Oliver Kieselbach	0d5e2572aa	perf(telemetry): one canonical request record; drop expected storage outcomes and SignalR success deps - RequestTelemetryMiddleware: SamplingPercentage=100 per item (worker copy was adaptively sampled, ItemCount 2-5 on weekdays, despite host.json excludedTypes) - StorageDependencyFilterProcessor: also drop 404/412/409 storage outcomes (InProc + HTTP shape) and successful *.service.signalr.net REST calls - infra/appinsights-workspace-transforms.bicep: DCR dropping the host HTTP request duplicate at ingestion (timer/queue rows kept) - docs/backend/telemetry-ingest-shaping.md; skills updated Measured baseline 7d: ~1.3 GB/week removed, requests now 100% with tenant context. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-23 18:31:32 +02:00
Oliver Kieselbach	793e907e60	fix(cveindex): merge findings per CVE before upsert; bounded parallelism One CVE commonly appears under many findings of the same report (e.g. one RD CVE across 14 installed RD components). Each occurrence produced its own concurrent upsert on the same PK/RK, so the stored SoftwareName/OverallRisk was race-dependent. Collapse to one entity per CVE (highest CVSS wins, IsKev OR-ed, OverallRisk max) and write with Parallel.ForEachAsync(8) and per-entity fail-soft logging instead of Task.WhenAll + one outer catch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-08-23 17:52:02 +02:00