revert(backend): undo L4 telemetry sampling — ineffective in isolated worker

Deploy-verified on f4d51ea4: the worker-side adaptive-sampling exclusion
(EnableAdaptiveSampling=false + manual Configure<TelemetryConfiguration>
UseAdaptiveSampling(excludedTypes:"Request;Event")) does NOT take effect in the
Functions isolated worker. Live data showed the enriched worker request copy
(Source=WorkerMiddleware) sampled to ~63% while the redundant host request item
(Source="") stayed fully billed — strictly worse (lossy enriched copy, host
duplicate unchanged). host.json "Host.Results": "Error" likewise did not drop
the host request item (emitted by the host process, not governed by that level).

Restores known-good: default worker sampling (enriched copy complete at normal
traffic, burst-capped) + host.json Host.Results back to Information. L1 storage
dependency filter and L2 Function-trace suppression are unaffected and still
verified live. L4 deferred pending a mechanism verified against the isolated-
worker pipeline. H2/H3/O1/O2 from f4d51ea4 verified healthy (0 exceptions).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Oliver Kieselbach
2026-06-09 23:17:17 +02:00
parent f4d51ea499
commit bc1f11d2db
2 changed files with 12 additions and 24 deletions
@@ -8,7 +8,6 @@ using AutopilotMonitor.Functions.Functions.Sessions;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Middleware;
using AutopilotMonitor.Functions.Services;
using Microsoft.ApplicationInsights.Extensibility;
using Microsoft.AspNetCore.Authentication.JwtBearer;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
@@ -53,36 +52,25 @@ builder.Services.Configure<JsonSerializerOptions>(options =>
});
builder.Services
.AddApplicationInsightsTelemetryWorkerService(options =>
{
// The DEFAULT worker-side adaptive sampler has no excluded types: under load it thinned
// the enriched request copies from RequestTelemetryMiddleware (TenantId/CorrelationId —
// the per-call audit record) to ~15%, while the unenriched HOST request items stayed
// unsampled (host.json excludedTypes only governs the host pipeline). Protection was
// inverted. Disabled here and re-added below with Request + Event excluded.
options.EnableAdaptiveSampling = false;
})
.AddApplicationInsightsTelemetryWorkerService()
.ConfigureFunctionsApplicationInsights();
// L4 ATTEMPT REVERTED 2026-06-09 (deploy-verified ineffective): tried to exclude Request;Event
// from worker-side adaptive sampling via EnableAdaptiveSampling=false + a manual
// Configure<TelemetryConfiguration> { DefaultTelemetrySink…UseAdaptiveSampling(excludedTypes) }.
// In the Functions isolated worker that classic-ASP.NET pattern does NOT take effect — live data
// showed the enriched worker request copy (Source=WorkerMiddleware) getting sampled to ~63% while
// the redundant host request item (Source="") stayed fully billed. Net worse (lossy enriched copy,
// unchanged host duplicate). The host request item is emitted by the HOST process and cannot be
// dropped from worker code or by host.json "Host.Results" level. Re-attempt only with a mechanism
// verified against the isolated-worker telemetry pipeline — do not re-add the manual chain.
// Drop successful Azure Storage dependencies (Table/Queue/Blob) from the worker telemetry
// pipeline to curb AppDependencies ingestion cost — this backend is storage-I/O heavy and those
// rows are high-volume, low-value. Failed storage calls and all non-storage dependencies
// (HTTP/Graph/SQL/SignalR) are preserved. See StorageDependencyFilterProcessor for the contract.
builder.Services.AddApplicationInsightsTelemetryProcessor<AutopilotMonitor.Functions.Telemetry.StorageDependencyFilterProcessor>();
// Re-introduce worker-side adaptive sampling with Request + Event excluded: the enriched
// request items stay complete (with host.json "Host.Results": "Error" they are the ONLY
// success-request record), and customEvents are low-volume ops signals worth keeping whole.
// Traces/dependencies/exceptions remain adaptively sampled. The sampler is appended to the
// processor chain AFTER the DI-registered processors above, so the dropped storage chatter
// no longer pressures the sampling rate of everything else.
builder.Services.Configure<TelemetryConfiguration>(config =>
{
var chain = config.DefaultTelemetrySink.TelemetryProcessorChainBuilder;
chain.UseAdaptiveSampling(maxTelemetryItemsPerSecond: 5, excludedTypes: "Request;Event");
chain.Build();
});
// Configure JWT Authentication for Multi-Tenant Azure AD
builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
.AddMicrosoftIdentityWebApi(options =>
@@ -8,7 +8,7 @@
"AutopilotMonitor.Functions.Security": "Information",
"Host": "Error",
"Host.Aggregator": "Error",
"Host.Results": "Error",
"Host.Results": "Information",
"Host.Startup": "Error",
"Microsoft.Azure.WebJobs": "Error",
"Microsoft.Azure.WebJobs.Host": "Error",