perf(telemetry): one canonical request record; drop expected storage outcomes and SignalR success deps

- RequestTelemetryMiddleware: SamplingPercentage=100 per item (worker copy was
  adaptively sampled, ItemCount 2-5 on weekdays, despite host.json excludedTypes)
- StorageDependencyFilterProcessor: also drop 404/412/409 storage outcomes
  (InProc + HTTP shape) and successful *.service.signalr.net REST calls
- infra/appinsights-workspace-transforms.bicep: DCR dropping the host HTTP
  request duplicate at ingestion (timer/queue rows kept)
- docs/backend/telemetry-ingest-shaping.md; skills updated

Measured baseline 7d: ~1.3 GB/week removed, requests now 100% with tenant context.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

docs/backend/telemetry-ingest-shaping.md

@@ -0,0 +1,87 @@
+---
+type: Concept
+title: Telemetry Ingest Shaping (App Insights / Log Analytics)
+description: What the backend deliberately keeps out of Application Insights and why - one canonical request record (worker copy, per-item sampling bypass; host HTTP duplicate dropped by a workspace transformation), expected storage outcomes (404/412/409) and successful SignalR REST calls filtered in the worker, and the query rules that follow.
+resource: /src/Backend/AutopilotMonitor.Functions/Telemetry/StorageDependencyFilterProcessor.cs
+tags:
+  - backend
+  - telemetry
+  - application-insights
+  - cost
+timestamp: 2026-08-23
+---
+
+# Telemetry Ingest Shaping
+
+The backend is storage-I/O heavy and runs on the Functions isolated worker, which emits telemetry from TWO processes (host and worker). Left alone, Log Analytics ingestion is dominated by rows nobody reads. This document records what is removed, where, and what that means for queries.
+
+# Schema
+
+## Request telemetry — one canonical record
+
+| Source | Emitted by | Content | Fate |
+|---|---|---|---|
+| Host HTTP request (`Properties.Source` empty, `Url` set) | Functions host process | Name, duration, status — no tenant/user/correlation | **Dropped at ingestion** by the workspace transformation on `AppRequests` (`infra/appinsights-workspace-transforms.bicep`) |
+| Host non-HTTP invocation (`Url` empty) | Functions host process | Timer / queue executions | Kept (the worker middleware never sees them) |
+| Worker request (`Properties.Source == 'WorkerMiddleware'`) | `RequestTelemetryMiddleware` | TenantId, UserId, UserRole, CorrelationId, ClientSource, McpToolName, HttpPath | **Kept, unsampled** — the canonical record |
+
+Why two mechanisms:
+
+* The host request item cannot be suppressed from worker code or from `host.json` (`Host.Results` controls log traces, not the request item; verified 2026-06-09). The only lever is a Log Analytics workspace transformation: `source | where tostring(Properties.Source) == 'WorkerMiddleware' or isempty(Url)`.
+* `host.json` `samplingSettings.excludedTypes: "Request;Event"` only applies to the host process. The worker's `AddApplicationInsightsTelemetryWorkerService` runs its own adaptive sampling; before 2026-08-23 worker request rows carried `ItemCount` 2–5 on weekdays (≈30 % of individual requests missing, `count()` undercounting by the same share — `sum(ItemCount)` matched the host copy to 1.00 per day). Re-configuring the worker sampling chain did not take effect in the isolated worker (see the L4 note in `Program.cs`). The per-item opt-out does: the middleware sets `SamplingPercentage = 100` on its own `RequestTelemetry`, and the SDK sampling processor passes through every item whose sampling percentage is already set.
+
+## Dependency telemetry — keep only signal
+
+`StorageDependencyFilterProcessor` (worker pipeline) drops:
+
+* successful Azure Storage calls (Table/Queue/Blob) in both shapes — the HTTP shape (`Azure table`, target `*.table.core.windows.net`) and the Azure SDK ActivitySource shape (`InProc | Microsoft.Tables`, `InProc | Microsoft.Storage`);
+* storage calls with an **expected outcome**: `404` (point-read miss — EventTypeIndex, BlockedDevices lookups), `412` (ETag precondition), `409` (idempotent insert conflict). HTTP-shape rows carry the code in `ResultCode`; InProc rows only in `Properties["Error"]` (`Status: 404 (`). Before this filter these were 100 % of the remaining "failed" storage rows (~540 MB/week, two rows per call);
+* successful Azure SignalR Service REST calls (`*.service.signalr.net` — group add/remove per connection, ~470 MB/week).
+
+Everything else passes: storage 429/5xx/auth/timeouts, failed SignalR calls, all HTTP/Graph/NVD/MSRC dependencies, the worker's own `InProc Invoke` span, and every non-dependency item.
+
+## Measured baseline (7 days before the change, `_BilledSize`, `autopilotmonitor-api-eu`)
+
+| Table | Rows | MB | After |
+|---|---|---|---|
+| AppDependencies | 976k | 1248 | ≈ 240 (SignalR + expected-outcome rows gone) |
+| AppRequests host copy | 274k | 323 | ≈ 0 (only timer/queue rows) |
+| AppRequests worker copy | 189k | 208 | ≈ 300 (now 100 % of requests) |
+| AppTraces | 142k | 137 | unchanged |
+| AppEvents | 6k | 4 | unchanged — not worth touching |
+
+Net ≈ −1.3 GB/week with MORE usable data (every request now has tenant context).
+
+# Examples
+
+Counting requests (exact since 2026-08-23; for older data replace `count()` with `sum(itemCount)`):
+
+```kql
+requests
+| where customDimensions.Source == 'WorkerMiddleware'
+| summarize total=count(), failed=countif(success == false) by name
+```
+
+Real storage trouble only:
+
+```kql
+dependencies
+| where type startswith 'InProc | Microsoft.Tables' or target endswith '.table.core.windows.net'
+| summarize count() by resultCode, tostring(customDimensions.Error)
+```
+
+Post-deploy verification of the per-item sampling bypass (must be `ItemCount == 1` only):
+
+```kql
+AppRequests
+| where TimeGenerated > ago(1h) and tostring(Properties.Source) == 'WorkerMiddleware'
+| summarize rows=count() by ItemCount
+```
+
+# Citations
+
+* `src/Backend/AutopilotMonitor.Functions/Middleware/RequestTelemetryMiddleware.cs` — `SamplingPercentage = 100` before `TrackRequest`.
+* `src/Backend/AutopilotMonitor.Functions/Telemetry/StorageDependencyFilterProcessor.cs` + `StorageDependencyFilterProcessorTests.cs` — the filter contract.
+* `src/Backend/AutopilotMonitor.Functions/Program.cs` — the failed L4 worker-sampling attempt and why it must not be retried.
+* `infra/appinsights-workspace-transforms.bicep` — the AppRequests transformation DCR, apply/verify/rollback commands.
+* `.claude/commands/backend-logs.md`, `session-debug.md` — query conventions.

docs/index.md

@@ -45,6 +45,7 @@ https://docs.autopilotmonitor.com (separate repository).
 
 # Backend
 
+* [Telemetry Ingest Shaping](backend/telemetry-ingest-shaping.md) - One canonical App Insights request record (worker copy with tenant context, per-item sampling bypass; host HTTP duplicate dropped by a workspace transformation DCR), expected storage outcomes 404/412/409 and successful SignalR REST calls filtered in the worker, and the count()/sum(itemCount) query rules that follow.
 * [Table Schema Sentinel](backend/table-schema-sentinel.md) - Startup no longer issues one CreateTableIfNotExists per table: a SHA-256 over TableNames.All in AdminConfiguration gates the full pass (derived, never hand-maintained), the SessionsIndex backfill runs only after a full pass behind a conditional-insert claim, and daily maintenance repairs out-of-band deletions.
 * [Business Timestamps](backend/business-timestamps.md) - Why the Azure Tables system Timestamp is never authoritative (migrations reset it), and the OccurredUtc + RowKey-decode compensation for AuditLogs/OpsEvents/Events after the 2026-07-18 migration.
 * [Send-Time Frame Separation](backend/send-time-frame-separation.md) - One device-clock send timestamp per upload batch (X-Send-Time-Utc → SentAt on Events) splits ReceivedAt − OccurredUtc into pure spool delay and pure device-vs-server clock offset; the clock_skew condition measures SentAt batches directly (spool-immune), the CMTrace tripwire deliberately stays on its differential math.

docs/log.md

@@ -2,6 +2,7 @@
 
 ## 2026-08-23
 
+* **New**: `backend/telemetry-ingest-shaping.md` — `RequestTelemetryMiddleware` sets `SamplingPercentage=100` on its own item (worker request rows were adaptively sampled, ItemCount 2–5 on weekdays, despite host.json `excludedTypes`), the redundant host HTTP request copy is dropped by a workspace transformation DCR (`infra/appinsights-workspace-transforms.bicep`, timer/queue rows kept), and `StorageDependencyFilterProcessor` now also drops expected storage outcomes (404/412/409, both InProc and HTTP shapes) and successful SignalR REST calls. Measured ≈ −1.3 GB/week.
 * **Update**: `backend/table-schema-sentinel.md` — "Measuring it": `StartupTelemetryService` emits `BackendStartupMs` (process start → ApplicationStarted) and `BackendTableInitMs` as App Insights custom metrics with version/commit/fullPass dimensions; the MCP server writes one `{"type":"boot","readyMs"}` stderr line per start (query in `.claude/commands/mcp-telemetry.md` §A2).
 * **New**: `backend/table-schema-sentinel.md` — startup table initialization gated by a schema sentinel (SHA-256 over `TableNames.All` in `AdminConfiguration`/`SchemaSentinel`/`tables`): matching sentinel ⇒ zero `CreateTableIfNotExists` calls on a cold start, missing/stale ⇒ one full pass + sentinel rewrite. SessionsIndex startup backfill only after a full pass, `IsSessionIndexEmptyAsync` no longer treats errors as "empty", conditional-insert claim so one scaled-out instance backfills. Daily maintenance runs `IStorageInitializer.EnsureAllAsync` to repair out-of-band table deletions.
 * **Update**: `backend/verdict-calibration.md` — MCP `get_verdict_calibration` response shaping: per-row trend denominators dropped, explicit nulls, `minSharePct` / `top` with an `omitted` report.

infra/appinsights-workspace-transforms.bicep

@@ -0,0 +1,67 @@
+// Autopilot-Monitor — Log Analytics workspace transformation (ingestion-time filter)
+//
+// Drops the redundant HOST-side HTTP request record from AppRequests so that exactly one
+// canonical request row remains: the worker-side copy emitted by RequestTelemetryMiddleware
+// (Properties.Source == 'WorkerMiddleware', carries TenantId/UserId/CorrelationId and is
+// sampling-exempt per item). Non-HTTP host invocations (timer/queue triggers) have an empty
+// Url and are kept — the worker middleware never sees them.
+//
+// Why ingestion-time: the host request item is emitted by the Functions host process and
+// cannot be suppressed from worker code or host.json (verified 2026-06-09, see Program.cs).
+// Live numbers 2026-08-23 (7d): host copy 274k rows / 323 MB, worker copy 189k rows / 208 MB.
+//
+// Cost note: transformation filtering is free up to 50% of the table's incoming volume; the
+// portion beyond that (here ~11% of AppRequests) is billed at the (cheap) data-processing rate.
+//
+// Apply (two steps; the link is a CLI update = GET+PUT, which keeps retention/SKU intact):
+//   az deployment group create -g rg-autopilotmonitor-prd-gwc \
+//     --template-file infra/appinsights-workspace-transforms.bicep \
+//     --query properties.outputs.dcrId.value -o tsv
+//   az monitor log-analytics workspace update -g rg-autopilotmonitor-prd-gwc -n AutopilotMonitor \
+//     --data-collection-rule <dcrId-from-above>
+//
+// Verify after ~20 min (ingestion latency):
+//   AppRequests | where TimeGenerated > ago(15m) | where AppRoleName == 'autopilotmonitor-api-eu'
+//   | summarize rows=count() by source=tostring(Properties.Source), hasUrl=isnotempty(Url)
+//   → only (WorkerMiddleware, true) and ('', false) buckets remain.
+//
+// Rollback: az monitor log-analytics workspace update -g <rg> -n AutopilotMonitor
+//   --set properties.defaultDataCollectionRuleResourceId=null   (then delete the DCR)
+
+@description('Log Analytics workspace that backs the workspace-based Application Insights resources.')
+param workspaceName string = 'AutopilotMonitor'
+
+param location string = resourceGroup().location
+
+resource workspace 'Microsoft.OperationalInsights/workspaces@2023-09-01' existing = {
+  name: workspaceName
+}
+
+resource transforms 'Microsoft.Insights/dataCollectionRules@2023-03-11' = {
+  name: 'dcr-autopilotmonitor-ws-transforms'
+  location: location
+  kind: 'WorkspaceTransforms'
+  properties: {
+    description: 'AppRequests: keep the enriched worker request record (Properties.Source == WorkerMiddleware) and non-HTTP host invocations (empty Url); drop the host HTTP duplicate.'
+    destinations: {
+      logAnalytics: [
+        {
+          workspaceResourceId: workspace.id
+          name: 'ws'
+        }
+      ]
+    }
+    dataFlows: [
+      {
+        streams: [ 'Microsoft-Table-AppRequests' ]
+        destinations: [ 'ws' ]
+        transformKql: 'source | where tostring(Properties.Source) == \'WorkerMiddleware\' or isempty(Url)'
+      }
+    ]
+  }
+}
+
+// A workspace has exactly one transformation DCR, activated by linking it as the workspace's
+// defaultDataCollectionRuleResourceId — deliberately NOT modelled here (see Apply above).
+
+output dcrId string = transforms.id

src/Backend/AutopilotMonitor.Functions.Tests/StorageDependencyFilterProcessorTests.cs

@@ -48,11 +48,74 @@ public class StorageDependencyFilterProcessorTests
         Assert.Empty(next.Received);
     }
 
-    [Fact]
-    public void FailedStorageDependency_IsKept()
+    [Theory]
+    [InlineData("429")]
+    [InlineData("500")]
+    [InlineData("503")]
+    [InlineData("403")]
+    [InlineData("")] // unknown outcome (timeout/network) — keep
+    public void FailedStorageDependency_IsKept(string resultCode)
     {
         var (processor, next) = Build();
         var dep = Dep("myacct.queue.core.windows.net", success: false);
+        dep.ResultCode = resultCode;
+        processor.Process(dep);
+        Assert.Same(dep, Assert.Single(next.Received));
+    }
+
+    // Expected outcomes (point-read miss, ETag precondition, insert conflict) are normal control
+    // flow — live-verified as 100% of the remaining "failed" storage rows. Dropped.
+    [Theory]
+    [InlineData("404")]
+    [InlineData("412")]
+    [InlineData("409")]
+    public void ExpectedStorageOutcome_HttpShape_IsDropped(string resultCode)
+    {
+        var (processor, next) = Build();
+        var dep = Dep("myacct.table.core.windows.net", success: false);
+        dep.ResultCode = resultCode;
+        processor.Process(dep);
+        Assert.Empty(next.Received);
+    }
+
+    [Theory]
+    [InlineData("Azure.RequestFailedException: The specified resource does not exist.\nRequestId:x\nStatus: 404 (Not Found)\nErrorCode: ResourceNotFound")]
+    [InlineData("Azure.RequestFailedException: The update condition specified in the request was not satisfied.\nStatus: 412 (Precondition Failed)\nErrorCode: UpdateConditionNotSatisfied")]
+    [InlineData("Azure.RequestFailedException: The specified entity already exists.\nStatus: 409 (Conflict)\nErrorCode: EntityAlreadyExists")]
+    public void ExpectedStorageOutcome_InProcShape_IsDropped(string error)
+    {
+        // InProc spans have no ResultCode — only the exception text in Properties["Error"].
+        var (processor, next) = Build();
+        var dep = Dep(target: "TableClient.GetEntity", success: false, type: "InProc | Microsoft.Tables");
+        dep.Properties["Error"] = error;
+        processor.Process(dep);
+        Assert.Empty(next.Received);
+    }
+
+    [Fact]
+    public void ThrottledInProcStorageDependency_IsKept()
+    {
+        var (processor, next) = Build();
+        var dep = Dep(target: "TableClient.SubmitTransaction", success: false, type: "InProc | Microsoft.Tables");
+        dep.Properties["Error"] = "Azure.RequestFailedException: Too many requests.\nStatus: 429 (Too Many Requests)\nErrorCode: ServerBusy";
+        processor.Process(dep);
+        Assert.Same(dep, Assert.Single(next.Received));
+    }
+
+    [Fact]
+    public void SuccessfulSignalRRestCall_IsDropped()
+    {
+        var (processor, next) = Build();
+        processor.Process(Dep("autopilotmonitor-eu.service.signalr.net", success: true, data: "POST /api/hubs/autopilotmonitor/groups/t-x/connections/y", type: "HTTP"));
+        Assert.Empty(next.Received);
+    }
+
+    [Fact]
+    public void FailedSignalRRestCall_IsKept()
+    {
+        var (processor, next) = Build();
+        var dep = Dep("autopilotmonitor-eu.service.signalr.net", success: false, type: "HTTP");
+        dep.ResultCode = "500";
         processor.Process(dep);
         Assert.Same(dep, Assert.Single(next.Received));
     }

src/Backend/AutopilotMonitor.Functions/Middleware/RequestTelemetryMiddleware.cs

@@ -1,5 +1,6 @@
 using System.Diagnostics;
 using Microsoft.ApplicationInsights;
+using Microsoft.ApplicationInsights.Channel;
 using Microsoft.ApplicationInsights.DataContracts;
 using Microsoft.Azure.Functions.Worker;
 using Microsoft.Azure.Functions.Worker.Middleware;
@@ -140,6 +141,15 @@ public class RequestTelemetryMiddleware : IFunctionsWorkerMiddleware
             if (caughtException != null)
                 requestTelemetry.Properties["ExceptionType"] = caughtException.GetType().Name;
 
+            // Bypass the worker's adaptive sampling for this item. The SDK sampling processor
+            // passes through any item whose SamplingPercentage is already set, so this is the
+            // per-item opt-out that host.json "excludedTypes" (host process only) cannot provide.
+            // Live-verified 2026-08-23 BEFORE this change: worker request rows carried
+            // ItemCount 2-5 on weekdays (~30% of individual requests missing), so count()-based
+            // queries undercounted. This copy is the single canonical request record — the host
+            // duplicate is dropped by a workspace DCR transformation on AppRequests.
+            ((ISupportSampling)requestTelemetry).SamplingPercentage = 100;
+
             try
             {
                 _telemetryClient.TrackRequest(requestTelemetry);

src/Backend/AutopilotMonitor.Functions/Program.cs

@@ -71,6 +71,10 @@ builder.Services
 // unchanged host duplicate). The host request item is emitted by the HOST process and cannot be
 // dropped from worker code or by host.json "Host.Results" level. Re-attempt only with a mechanism
 // verified against the isolated-worker telemetry pipeline — do not re-add the manual chain.
+// 2026-08-23: solved differently — RequestTelemetryMiddleware sets SamplingPercentage=100 on its
+// own item (per-item sampling bypass), and the host duplicate is removed by a workspace DCR
+// transformation on AppRequests (keeps rows with Properties.Source=='WorkerMiddleware' plus
+// non-HTTP host rows with empty Url, i.e. timer/queue invocations). See docs/backend/telemetry-ingest-shaping.md.
 
 // Drop successful Azure Storage dependencies (Table/Queue/Blob) from the worker telemetry
 // pipeline to curb AppDependencies ingestion cost — this backend is storage-I/O heavy and those

src/Backend/AutopilotMonitor.Functions/Telemetry/StorageDependencyFilterProcessor.cs

@@ -17,8 +17,14 @@ namespace AutopilotMonitor.Functions.Telemetry;
 /// - Only <see cref="DependencyTelemetry"/> is considered; requests, traces, exceptions,
 ///   metrics and all NON-storage dependencies (HTTP, Microsoft Graph, SQL, SignalR, ...) pass
 ///   through untouched.
-/// - FAILED storage calls are KEPT (Success == false): they are rare and are real signal for
-///   troubleshooting throttling / transient faults, so the noise reduction does not blind us.
+/// - FAILED storage calls are KEPT (Success == false) EXCEPT the expected-outcome status codes
+///   404 (point-read miss), 412 (ETag precondition) and 409 (insert conflict): those are normal
+///   control flow for this backend (index lookups, optimistic concurrency, idempotent inserts).
+///   Live-verified 2026-08-23: 100% of the remaining "failed" storage rows were 404/412/409
+///   (~540 MB/week, two rows per call: InProc span + HTTP span). Throttling (429), 5xx, auth
+///   failures and timeouts stay — they are the real signal.
+/// - Successful Azure SignalR REST management calls (group add/remove per connection, ~470 MB/week)
+///   are dropped for the same reason; failed SignalR calls are kept.
 ///
 /// Registered in Program.cs via AddApplicationInsightsTelemetryProcessor so it runs inside the
 /// isolated worker's telemetry pipeline, where the app's own Azure SDK dependencies are tracked.
@@ -46,6 +52,12 @@ public sealed class StorageDependencyFilterProcessor : ITelemetryProcessor
         "InProc | Microsoft.Storage",  // Azure.Storage.Queues + Azure.Storage.Blobs
     };
 
+    // Azure SignalR Service REST endpoint (group membership management per connection).
+    private const string SignalRServiceSuffix = ".service.signalr.net";
+
+    // Expected storage outcomes that are normal control flow, not failures worth billing.
+    private static readonly string[] ExpectedStorageStatusCodes = { "404", "412", "409" };
+
     private readonly ITelemetryProcessor _next;
 
     public StorageDependencyFilterProcessor(ITelemetryProcessor next) => _next = next;
@@ -68,17 +80,50 @@ public sealed class StorageDependencyFilterProcessor : ITelemetryProcessor
             return false;
         }
 
-        // Keep failures — rare, high-value diagnostic signal. Only successful storage chatter is noise.
-        if (dependency.Success == false)
-        {
-            return false;
-        }
-
-        return IsStorageEndpoint(dependency.Target)
+        var isStorage = IsStorageEndpoint(dependency.Target)
             || IsStorageEndpoint(dependency.Data)
             || IsStorageInProcType(dependency.Type);
+
+        if (isStorage)
+        {
+            // Successful storage chatter is noise; so are expected 404/412/409 outcomes.
+            // Every other failure (429, 5xx, auth, timeout) is kept.
+            return dependency.Success != false || IsExpectedStorageOutcome(dependency);
+        }
+
+        // SignalR REST management calls: keep only failures.
+        return dependency.Success != false && IsSignalRService(dependency.Target);
     }
 
+    private static bool IsExpectedStorageOutcome(DependencyTelemetry dependency)
+    {
+        // HTTP-shaped rows ("Azure table") carry the status in ResultCode; the Azure SDK InProc
+        // span has no ResultCode, only the exception text in Properties["Error"]
+        // ("...Status: 404 (Not Found)...").
+        var code = dependency.ResultCode;
+        if (!string.IsNullOrEmpty(code))
+        {
+            return Array.IndexOf(ExpectedStorageStatusCodes, code) >= 0;
+        }
+
+        if (dependency.Properties.TryGetValue("Error", out var error) && !string.IsNullOrEmpty(error))
+        {
+            foreach (var expected in ExpectedStorageStatusCodes)
+            {
+                if (error.Contains("Status: " + expected + " (", System.StringComparison.Ordinal))
+                {
+                    return true;
+                }
+            }
+        }
+
+        return false;
+    }
+
+    private static bool IsSignalRService(string? target)
+        => !string.IsNullOrEmpty(target)
+           && target.EndsWith(SignalRServiceSuffix, System.StringComparison.OrdinalIgnoreCase);
+
     private static bool IsStorageInProcType(string? type)
     {
         if (string.IsNullOrEmpty(type))