diff --git a/docs/backend/app-version-regression-radar.md b/docs/backend/app-version-regression-radar.md new file mode 100644 index 00000000..c240b592 --- /dev/null +++ b/docs/backend/app-version-regression-radar.md @@ -0,0 +1,98 @@ +--- +type: Concept +title: App Version Duration Regression Radar +description: How the radar detects app versions whose median install duration regressed vs. the previous version (median lift ≥2x plus an absolute floor over first-seen-ordered versions), how alert episodes live in the appversionregression tracker keyspace, and which surfaces show them. +resource: src/Backend/AutopilotMonitor.Functions/Helpers/AppVersionRegressionRadar.cs +tags: + - backend + - apps + - regression-radar + - insights +timestamp: 2026-08-13T12:00:00+02:00 +--- + +# App Version Duration Regression Radar + +Detects "version X made this app's installs much slower" — the concrete, quotable insight +("median install duration rose from 11 to 29 min after version X") — from the per-(session, +app) `AppInstallSummaries` rows. Duration only: failure-rate per version is already visible +in the apps dashboard's `versionBreakdown`, and rate regressions belong to Wilson-interval +statistics, not medians. + +# Schema + +## Signal & detection + +Per (tenant, app), the MEASURED install population is: `Status == "Succeeded"`, terminal +state not Skipped/Postponed, `MetricsMath.HasMeasuredDuration` (1s..6h — 0s means the start +was unobserved, above 6h means back-stamped batches), no `AppIdCollision`, non-empty +`AppVersion`. Versions are ordered by FIRST-SEEN install time (min `StartedAt`), never by +string comparison ("9.1" vs "2024.10" would lie). The comparison pair is always the newest +version (the one with the most recent measured install) vs. the version with the latest +first-seen strictly before it. + +MEDIANS, not means (nearest-rank, shared with the apps dashboard via +`AppsAnalyticsHelper.Percentile`) — one back-stamped straggler must not fire a fleet alert. +A regression fires only when ALL hold over the trailing 35-day horizon: + +1. both versions have ≥ 10 measured installs; +2. current median ≥ 2× the previous version's median; +3. absolute increase ≥ 300 s (a 20s→45s app is noise, not an incident). + +## Suppression & limits + +* Parallel ring rollouts can make "previous" a concurrently-deployed version — accepted in + v1; the episode key (app, current version) still caps it at one bell per version. +* Platform kill switch: app setting `AppVersionRegressionRadarDisabled=true` skips the + radar (fail-open — it only notifies, never mutates data). +* One cross-tenant install-summary read per pass (the same query the global apps dashboard + runs), grouped by tenant; per-tenant failures are non-fatal. + +## Alert episodes (tracker keyspace) + +An episode lives as `appversionregression|{app-lower}|{version-lower}` (table-key-sanitized; +raw AppName/CurrentVersion live in columns, the mapper never parses the RowKey) in the +notification-tracker table, next to the `ruleregression|` keyspace: the row IS the dedup +(bell + ops event fire exactly once per episode) and the `versionRegressions[]` payload — +both medians, both sample counts, lift. Numbers refresh on every radar pass while the +episode stays active; `FirstNotifiedAt` never moves. The episode closes (dedup re-arms) when +the alerted version drains under 10 measured installs in the horizon or its recomputed +median falls back under 1.5× the previous version's median; the tracker's 30-day retention +sweep also re-arms, so a month-old still-burning regression rings once more by design. + +## Notification chain (v1: bell + ops event) + +* Tenant bell `app_version_duration_regression` (audience: Admin), href + `/apps/detail?name={app}&days=35` (canonical `appDetailUrl` shape). Message carries the + full numbers so the admin can verify without a portal round-trip. +* Ops event `AppVersionDurationRegression` (Tenant category, Warning) — dual-registered in + the web `OPS_EVENT_TYPES` catalog so operators can wire Telegram/Teams/Slack alert rules. + +## Surfaces + +* `apps/{appName}/analytics` (+ global twin with `?tenantId=`) responds with + `versionRegressions[]` (empty for the cross-tenant aggregate — episodes are per-tenant + rows), and `versionBreakdown[]` rows now carry `measuredInstalls` / + `medianDurationSeconds` / `p95DurationSeconds`. +* Apps detail page: amber "↑ Duration regression" banner above the version cards and a + "Median Install Duration by Version" bar chart. + +# Examples + +A tenant ships Contoso VPN 2.4.0 after 2.3.9. 2.3.9 has 40 measured installs, median 660s; +2.4.0 accumulates 12 measured installs with median 1740s. Lift 2.6×, absolute +1080s → +episode `appversionregression|contoso vpn|2.4.0` opens, one bell + one ops event fire. +Subsequent passes refresh the numbers only. When 2.4.1 replaces 2.4.0 and 2.4.0 drains +below 10 measured installs in the horizon, the episode deletes and the dedup re-arms. + +# Citations + +* `src/Backend/AutopilotMonitor.Functions/Helpers/AppVersionRegressionRadar.cs` — pure core + (gates, first-seen ordering, re-arm). +* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.AppVersionRegression.cs` + — orchestration, kill switch, bell/ops-event emission, wording contract. +* `src/Backend/AutopilotMonitor.Functions/DataAccess/TableStorage/TableHardwareRejectionNotificationTracker.cs` + — episode keyspace, entity round-trip. +* `src/Shared/AutopilotMonitor.Shared/Models/Metrics/AppVersionRegressionAlert.cs` — persisted + alert payload. +* [Rule Regression Radar](rule-regression-radar.md) — the sibling pattern this radar mirrors. diff --git a/docs/index.md b/docs/index.md index 5f6e63a2..927676ea 100644 --- a/docs/index.md +++ b/docs/index.md @@ -39,6 +39,7 @@ https://docs.autopilotmonitor.com (separate repository). * [Device Journeys & First-Time-Right](backend/device-journeys.md) - How a device's terminal enrollment attempts are grouped into journeys (trim+casefold device key with junk-serial exclusion, first-success end, 30-day gap, WhiteGlove = one attempt), the DeviceHistories chain with its tombstone-driven deleted-session pruning, the additive daily FTR aggregates, and the banner/fleet/MCP surfaces. * [Enrollment Time Attribution](backend/time-attribution.md) - How a terminal session's authoritative wall clock (DurationSeconds, WhiteGlove pause excluded) is partitioned into named segments with an explicit unattributed remainder, ESP-blocking apps as positive evidence with "up to" what-if bounds, reboot spans from event gaps, and the breakdown/aggregate tables the sweep maintains. * [Rule Regression Radar](backend/rule-regression-radar.md) - How analyze-rule hit-rate regressions are detected (7-day window ≥2× the 28-day baseline with disjoint Wilson intervals), which suppressions keep the radar quiet (grace period, edit-in-window, kill switch), how alert episodes live in the ruleregression tracker keyspace with re-arm semantics, and the on-fire dimension correlation with its "correlated, not necessarily causal" wording contract. +* [App Version Duration Regression Radar](backend/app-version-regression-radar.md) - How app versions whose median install duration regressed vs. the previous version are detected (first-seen-ordered versions, median lift ≥2× plus a 5-minute absolute floor over measured installs only), how alert episodes live in the appversionregression tracker keyspace with drain/median re-arm semantics, and the bell/ops-event/analytics surfaces. * [Tenant Activation & the Once-Only Welcome Mail](backend/tenant-activation-welcome-mail.md) - How a signup becomes an activated tenant (manual approve or the auto-approve queue), and the write-then-read + conditional-marker arbitration that sends the welcome mail exactly once even though activation and the user-typed notification address race each other. * [W365 Cloud PC Device Validation](backend/cloudpc-device-validation.md) - Why Cloud PCs can never pass the Autopilot serial lookup, and the cert-CN-bound CloudPc fallback validator: Subject CN (= Intune device id) matched against the tenant's Graph virtualEndpoint/cloudPCs inventory — admits exactly the service-provisioned Cloud PCs, nothing else. * [ServerActions & On-Demand Diagnostics](backend/server-actions-on-demand-diagnostics.md) - The queue-on-session, deliver-on-ingest action channel, the TenantAdminOrOperator tier with in-function per-type re-gating, the portal's Collect Logs flow (quick-config orders rotate_config before request_diagnostics), and the hosted-prefix sweep in the offboarding cascade. diff --git a/docs/log.md b/docs/log.md index bf225e70..b1a9a34f 100644 --- a/docs/log.md +++ b/docs/log.md @@ -1,5 +1,9 @@ # Log +## 2026-08-13 (later) + +* **Creation**: Added `backend/app-version-regression-radar.md` — app-version duration regression radar: per (tenant, app) the newest version's median measured install duration (nearest-rank; population Succeeded + non-skip + `HasMeasuredDuration` + no AppId collision + non-empty AppVersion) is compared against the previous version (first-seen ordering, never string sort); fires at median lift ≥2× AND ≥300s absolute with ≥10 measured installs on both sides. Episodes live as `appversionregression|{app}|{version}` tracker rows (one bell `app_version_duration_regression` + one `AppVersionDurationRegression` ops event per episode, refresh keeps `FirstNotifiedAt`, re-arm on drain/median-recovery/30d retention). Surfaces: `versionRegressions[]` + per-version `measuredInstalls`/`medianDurationSeconds`/`p95DurationSeconds` in the apps analytics responses, amber banner + duration-by-version chart on the apps detail page. Kill switch `AppVersionRegressionRadarDisabled`. + ## 2026-08-13 * **Update**: `backend/session-annotations.md` — the `/annotations` portal overview page now carries the standard tenant switcher (GA/Reader: "All tenants" aggregate + per-tenant drill-down via `GET global/session-annotations`; delegated: managed subset), with a tenant column and tenant-qualified session deep links. The global list endpoint now derives `excludeGlobalAdminLane` from `HasGlobalScope`, so delegated-rescue callers on their managed `?tenantId=` path never receive the platform-internal `globaladmin` lane (endpoint-test-pinned). diff --git a/src/Backend/AutopilotMonitor.Functions.Tests/AppVersionRegressionRadarTests.cs b/src/Backend/AutopilotMonitor.Functions.Tests/AppVersionRegressionRadarTests.cs new file mode 100644 index 00000000..cdeaa17d --- /dev/null +++ b/src/Backend/AutopilotMonitor.Functions.Tests/AppVersionRegressionRadarTests.cs @@ -0,0 +1,380 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.Json; +using System.Threading.Tasks; +using AutopilotMonitor.Functions.DataAccess.TableStorage; +using AutopilotMonitor.Functions.Functions.Apps; +using AutopilotMonitor.Functions.Helpers; +using AutopilotMonitor.Functions.Services; +using AutopilotMonitor.Shared.DataAccess; +using AutopilotMonitor.Shared.Models; +using Moq; +using Xunit; + +namespace AutopilotMonitor.Functions.Tests; + +/// +/// Pins the app-version duration regression radar (): +/// detection gates (≥10 measured installs on both sides, median lift ≥2×, ≥300s absolute), +/// first-seen version ordering (never string sort), the measured-population exclusions, +/// re-arm semantics, the tracker keyspace round-trip, the wording contract, and the +/// versionBreakdown duration fields on the analytics wire shape. +/// +public class AppVersionRegressionRadarTests +{ + private const string TenantA = "00000000-0000-0000-0000-0000000000a1"; + private const string App = "Contoso VPN"; + private static readonly DateTime Now = new(2026, 8, 13, 12, 0, 0, DateTimeKind.Utc); + + private static int _sessionCounter; + + private static AppInstallSummary Install( + string version, + int durationSeconds, + DateTime? startedAt = null, + string appName = App, + string status = "Succeeded", + string terminalState = "Installed", + bool collision = false) + => new() + { + TenantId = TenantA, + SessionId = $"s-{++_sessionCounter}", + AppName = appName, + AppVersion = version, + Status = status, + TerminalState = terminalState, + DurationSeconds = durationSeconds, + StartedAt = startedAt ?? Now.AddDays(-1), + AppIdCollision = collision, + }; + + /// n measured installs of one version, all with the same duration, spread over one day. + private static List Batch(string version, int count, int durationSeconds, DateTime firstSeen) + => Enumerable.Range(0, count) + .Select(i => Install(version, durationSeconds, firstSeen.AddMinutes(i))) + .ToList(); + + // ── detection gates ───────────────────────────────────────────────────── + + [Theory] + // currentCount, previousCount, previousMedian, currentMedian, expectFire + [InlineData(10, 10, 300, 900, true)] // lift 3×, +600s — fires + [InlineData(10, 10, 300, 600, true)] // exactly 2× AND exactly +300s — boundary fires + [InlineData(9, 10, 300, 900, false)] // current side below sample floor + [InlineData(10, 9, 300, 900, false)] // previous side below sample floor + [InlineData(10, 10, 200, 499, false)] // lift 2.5× but only +299s — absolute floor blocks + [InlineData(10, 10, 1000, 1900, false)] // +900s but lift 1.9× — lift gate blocks + public void Evaluate_Gates(int currentCount, int previousCount, int previousMedian, int currentMedian, bool expectFire) + { + var rows = Batch("1.0", previousCount, previousMedian, Now.AddDays(-20)); + rows.AddRange(Batch("2.0", currentCount, currentMedian, Now.AddDays(-5))); + + var findings = AppVersionRegressionRadar.Evaluate(rows); + + if (!expectFire) + { + Assert.Empty(findings); + return; + } + + var finding = Assert.Single(findings); + Assert.Equal(App, finding.AppName); + Assert.Equal("2.0", finding.CurrentVersion); + Assert.Equal("1.0", finding.PreviousVersion); + Assert.Equal(currentMedian, finding.CurrentMedianSeconds); + Assert.Equal(previousMedian, finding.PreviousMedianSeconds); + Assert.Equal(currentCount, finding.CurrentMeasuredCount); + Assert.Equal(previousCount, finding.PreviousMeasuredCount); + Assert.Equal(Math.Round((double)currentMedian / previousMedian, 1), finding.Lift); + } + + [Fact] + public void Evaluate_MedianIsNearestRank_HandComputed() + { + // Even n: nearest-rank p50 takes the LOWER middle value (rank = ceil(0.5*4)-1 = 1). + var rows = new List(); + foreach (var d in new[] { 100, 200, 300, 400 }) // median 200 + rows.AddRange(Batch("1.0", 3, d, Now.AddDays(-20))); + foreach (var d in new[] { 500, 600, 700, 800 }) // median 600 + rows.AddRange(Batch("2.0", 3, d, Now.AddDays(-5))); + + var finding = Assert.Single(AppVersionRegressionRadar.Evaluate(rows)); + Assert.Equal(600, finding.CurrentMedianSeconds); + Assert.Equal(200, finding.PreviousMedianSeconds); + Assert.Equal(3.0, finding.Lift); + } + + // ── version ordering ──────────────────────────────────────────────────── + + [Fact] + public void Evaluate_VersionOrder_ByFirstSeen_NeverStringSort() + { + // Ordinally "9.1" > "2024.10", but "2024.10" is the NEWER version by first-seen. + // A string sort would flip current/previous and compare in the wrong direction. + var rows = Batch("9.1", 10, 300, Now.AddDays(-20)); + rows.AddRange(Batch("2024.10", 10, 900, Now.AddDays(-5))); + + var finding = Assert.Single(AppVersionRegressionRadar.Evaluate(rows)); + Assert.Equal("2024.10", finding.CurrentVersion); + Assert.Equal("9.1", finding.PreviousVersion); + } + + [Fact] + public void Evaluate_MultipleVersionsInFlight_PreviousIsLatestFirstSeenBeforeCurrent() + { + var rows = Batch("1.0", 10, 300, Now.AddDays(-30)); // oldest + rows.AddRange(Batch("1.1", 10, 320, Now.AddDays(-18))); // the true predecessor + rows.AddRange(Batch("2.0", 10, 960, Now.AddDays(-4))); // newest, regressed + + var finding = Assert.Single(AppVersionRegressionRadar.Evaluate(rows)); + Assert.Equal("2.0", finding.CurrentVersion); + Assert.Equal("1.1", finding.PreviousVersion); // not 1.0 + Assert.Equal(320, finding.PreviousMedianSeconds); + } + + [Fact] + public void Evaluate_SingleVersion_NeverFires() + { + var rows = Batch("1.0", 25, 900, Now.AddDays(-5)); + Assert.Empty(AppVersionRegressionRadar.Evaluate(rows)); + } + + // ── measured-population exclusions ────────────────────────────────────── + + [Fact] + public void Evaluate_ExcludedRows_DoNotCountTowardSamplesOrMedians() + { + var rows = Batch("1.0", 10, 300, Now.AddDays(-20)); + rows.AddRange(Batch("2.0", 10, 900, Now.AddDays(-5))); + + // None of these may move the samples or medians: + rows.Add(Install("2.0", 0, Now.AddDays(-2))); // unmeasured (start unobserved) + rows.Add(Install("2.0", 21601, Now.AddDays(-2))); // over the 6h plausibility cap + rows.Add(Install("2.0", 50, Now.AddDays(-2), terminalState: "Skipped")); // skip — not an attempt + rows.Add(Install("2.0", 50, Now.AddDays(-2), terminalState: "Postponed")); // skip — not an attempt + rows.Add(Install("2.0", 50, Now.AddDays(-2), status: "Failed")); // failed — no duration signal + rows.Add(Install("2.0", 50, Now.AddDays(-2), collision: true)); // AppId collision — foreign outcomes + rows.Add(Install("", 50, Now.AddDays(-2))); // no version + rows.Add(Install("2.0", 50, Now.AddDays(-2), appName: "")); // no app name + + var finding = Assert.Single(AppVersionRegressionRadar.Evaluate(rows)); + Assert.Equal(10, finding.CurrentMeasuredCount); + Assert.Equal(900, finding.CurrentMedianSeconds); + } + + [Fact] + public void Evaluate_DeterministicOrdering_LiftDesc_ThenAppNameOrdinal() + { + var rows = new List(); + rows.AddRange(Batch("1.0", 10, 300, Now.AddDays(-20)).Select(s => { s.AppName = "App B"; return s; })); + rows.AddRange(Batch("2.0", 10, 900, Now.AddDays(-5)).Select(s => { s.AppName = "App B"; return s; })); // lift 3.0 + rows.AddRange(Batch("1.0", 10, 300, Now.AddDays(-20)).Select(s => { s.AppName = "App A"; return s; })); + rows.AddRange(Batch("2.0", 10, 1500, Now.AddDays(-5)).Select(s => { s.AppName = "App A"; return s; })); // lift 5.0 + + var findings = AppVersionRegressionRadar.Evaluate(rows); + Assert.Equal(2, findings.Count); + Assert.Equal("App A", findings[0].AppName); // higher lift first + Assert.Equal("App B", findings[1].AppName); + } + + // ── re-arm ────────────────────────────────────────────────────────────── + + private static AppVersionRegressionAlert Alert( + string currentVersion = "2.0", string previousVersion = "1.0") + => new() + { + TenantId = TenantA, + AppName = App, + CurrentVersion = currentVersion, + PreviousVersion = previousVersion, + CurrentMedianSeconds = 900, + PreviousMedianSeconds = 300, + CurrentMeasuredCount = 10, + PreviousMeasuredCount = 10, + Lift = 3.0, + FirstNotifiedAt = Now.AddDays(-3), + LastEvaluatedAt = Now.AddDays(-1), + }; + + [Fact] + public void ShouldReArm_WhenCurrentVersionDrainedOutOfHorizon() + { + // Only the previous version still has measured installs — the alerted version is gone. + var rows = Batch("1.0", 10, 300, Now.AddDays(-20)); + Assert.True(AppVersionRegressionRadar.ShouldReArm(rows, Alert())); + } + + [Fact] + public void ShouldReArm_WhenCurrentVersionBelowSampleFloor() + { + var rows = Batch("1.0", 10, 300, Now.AddDays(-20)); + rows.AddRange(Batch("2.0", 9, 900, Now.AddDays(-5))); + Assert.True(AppVersionRegressionRadar.ShouldReArm(rows, Alert())); + } + + [Fact] + public void ShouldReArm_WhenMedianFellBackUnderReArmFactor() + { + // 400 < 1.5 × 300 — the regression recovered (e.g. CDN warmed up). + var rows = Batch("1.0", 10, 300, Now.AddDays(-20)); + rows.AddRange(Batch("2.0", 10, 400, Now.AddDays(-5))); + Assert.True(AppVersionRegressionRadar.ShouldReArm(rows, Alert())); + } + + [Fact] + public void ShouldNotReArm_WhileMedianStaysElevated() + { + // 500 ≥ 1.5 × 300 — still elevated (even though below the 2× fire gate). + var rows = Batch("1.0", 10, 300, Now.AddDays(-20)); + rows.AddRange(Batch("2.0", 10, 500, Now.AddDays(-5))); + Assert.False(AppVersionRegressionRadar.ShouldReArm(rows, Alert())); + } + + [Fact] + public void ShouldNotReArm_WhenPreviousVersionDrained_NoComparisonBasis() + { + // The predecessor left the horizon; the episode is kept until the current + // version itself drains (fires-stopped-style re-arm only). + var rows = Batch("2.0", 10, 900, Now.AddDays(-5)); + Assert.False(AppVersionRegressionRadar.ShouldReArm(rows, Alert())); + } + + // ── tracker keyspace (round-trip + key shape) ─────────────────────────── + + [Fact] + public void AppVersionRegressionRowKey_PrefixedCaseFoldedAndSanitized() + { + Assert.Equal("appversionregression|contoso app_2|1.0_beta_", + TableHardwareRejectionNotificationTracker.BuildAppVersionRegressionRowKey( + " Contoso App/2 ", " 1.0#Beta? ")); + } + + [Fact] + public void AppVersionRegressionRowKey_NoPrefixCollisionWithOtherKeyspaces() + { + var key = TableHardwareRejectionNotificationTracker.BuildAppVersionRegressionRowKey(App, "1.0"); + Assert.StartsWith("appversionregression|", key, StringComparison.Ordinal); + // The prefix range [prefix, "appversionregression}") must not swallow the sibling + // keyspaces ("ruleregression|", "tpmpss|", "{mfr}|{model}"). + Assert.True(string.CompareOrdinal("ruleregression|", "appversionregression}") > 0); + Assert.True(string.CompareOrdinal("tpmpss|", "appversionregression}") > 0); + } + + [Fact] + public void AppVersionRegressionEntity_RoundTripsAllFields() + { + var alert = new AppVersionRegressionAlert + { + TenantId = TenantA, + AppName = App, + CurrentVersion = "2.4.0", + PreviousVersion = "2.3.9", + CurrentMedianSeconds = 1740, + PreviousMedianSeconds = 660, + CurrentMeasuredCount = 12, + PreviousMeasuredCount = 40, + Lift = 2.6, + FirstNotifiedAt = Now, + LastEvaluatedAt = Now.AddHours(2), + }; + + var entity = TableHardwareRejectionNotificationTracker.BuildAppVersionRegressionEntity(TenantA, alert); + Assert.Equal(TenantA.ToLowerInvariant(), entity.PartitionKey); + Assert.Equal("appversionregression|contoso vpn|2.4.0", entity.RowKey); + + var mapped = TableHardwareRejectionNotificationTracker.MapToAppVersionRegressionAlert(entity); + Assert.Equal(TenantA, mapped.TenantId); + Assert.Equal(App, mapped.AppName); + Assert.Equal("2.4.0", mapped.CurrentVersion); + Assert.Equal("2.3.9", mapped.PreviousVersion); + Assert.Equal(1740, mapped.CurrentMedianSeconds); + Assert.Equal(660, mapped.PreviousMedianSeconds); + Assert.Equal(12, mapped.CurrentMeasuredCount); + Assert.Equal(40, mapped.PreviousMeasuredCount); + Assert.Equal(2.6, mapped.Lift); + Assert.Equal(Now, mapped.FirstNotifiedAt); + Assert.Equal(Now.AddHours(2), mapped.LastEvaluatedAt); + } + + // ── wording contract ──────────────────────────────────────────────────── + + [Fact] + public void RegressionMessage_CarriesTheFullNumbers_MinutesOneDecimal() + { + var finding = new AppVersionDurationRegressionFinding + { + TenantId = TenantA, + AppName = App, + CurrentVersion = "2.4.0", + PreviousVersion = "2.3.9", + CurrentMedianSeconds = 1740, + PreviousMedianSeconds = 660, + CurrentMeasuredCount = 12, + PreviousMeasuredCount = 40, + Lift = 2.6, + }; + + Assert.Equal( + "Median install duration rose from 11 to 29 min after version 2.4.0 " + + "(12 measured installs vs 40 on version 2.3.9) — lift 2.6x over the last 35 days.", + MaintenanceService.BuildAppVersionRegressionMessage(finding)); + } + + // ── analytics wire shape (versionBreakdown duration fields) ───────────── + + [Fact] + public async Task Analytics_VersionBreakdown_CarriesMeasuredDurationStats() + { + var sessionRepo = new Mock(); + sessionRepo.Setup(r => r.GetSessionAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync((SessionSummary?)null); + + var rows = new List + { + Install("1.0", 100), Install("1.0", 200), Install("1.0", 300), Install("1.0", 400), + Install("1.0", 0), // unmeasured — excluded from stats + Install("1.0", 50, terminalState: "Skipped"), // skip — excluded + Install("1.0", 50, status: "Failed"), // failed — excluded from durations + }; + + var root = JsonSerializer.SerializeToElement(await AppsAnalyticsHelper.BuildAnalyticsResponseAsync( + rows, sessionRepo.Object, App, days: 30)); + + var version = root.GetProperty("versionBreakdown")[0]; + Assert.Equal("1.0", version.GetProperty("appVersion").GetString()); + Assert.Equal(7, version.GetProperty("installs").GetInt32()); + Assert.Equal(4, version.GetProperty("measuredInstalls").GetInt32()); + // Nearest-rank over [100, 200, 300, 400]: p50 = 200 (lower middle), p95 = 400. + Assert.Equal(200, version.GetProperty("medianDurationSeconds").GetInt32()); + Assert.Equal(400, version.GetProperty("p95DurationSeconds").GetInt32()); + } + + [Fact] + public async Task Analytics_VersionRegressions_SerializedCamelCase_AndPresentWhenEmptyData() + { + var sessionRepo = new Mock(); + sessionRepo.Setup(r => r.GetSessionAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync((SessionSummary?)null); + + var alert = Alert(); + // The Functions host serializes camelCase on the wire; mirror that here so the + // asserted property names match what the web client actually parses. + var wireOptions = new JsonSerializerOptions(JsonSerializerDefaults.Web); + + // Empty-data early return still surfaces the block (episodes can outlive window data). + var emptyRoot = JsonSerializer.SerializeToElement(await AppsAnalyticsHelper.BuildAnalyticsResponseAsync( + new List(), sessionRepo.Object, App, days: 30, + new List { alert }), wireOptions); + Assert.Equal(1, emptyRoot.GetProperty("versionRegressions").GetArrayLength()); + + var root = JsonSerializer.SerializeToElement(await AppsAnalyticsHelper.BuildAnalyticsResponseAsync( + new List { Install("1.0", 100) }, sessionRepo.Object, App, days: 30, + new List { alert }), wireOptions); + + var regression = root.GetProperty("versionRegressions")[0]; + Assert.Equal("2.0", regression.GetProperty("currentVersion").GetString()); + Assert.Equal(900, regression.GetProperty("currentMedianSeconds").GetInt32()); + } +} diff --git a/src/Backend/AutopilotMonitor.Functions.Tests/TenantNotificationAudienceCatalogTests.cs b/src/Backend/AutopilotMonitor.Functions.Tests/TenantNotificationAudienceCatalogTests.cs index 25bdbeb6..f43ed0f9 100644 --- a/src/Backend/AutopilotMonitor.Functions.Tests/TenantNotificationAudienceCatalogTests.cs +++ b/src/Backend/AutopilotMonitor.Functions.Tests/TenantNotificationAudienceCatalogTests.cs @@ -17,6 +17,8 @@ public class TenantNotificationAudienceCatalogTests [InlineData("sla_breach", NotificationAudience.Member)] [InlineData("sla_consecutive_failures", NotificationAudience.Member)] [InlineData("sla_resolved", NotificationAudience.Member)] + [InlineData("rule_frequency_regression", NotificationAudience.Admin)] + [InlineData("app_version_duration_regression", NotificationAudience.Admin)] public void Resolve_KnownTypes_ReturnsExpectedAudience(string type, NotificationAudience expected) { Assert.Equal(expected, TenantNotificationAudienceCatalog.Resolve(type)); diff --git a/src/Backend/AutopilotMonitor.Functions/DataAccess/TableStorage/TableHardwareRejectionNotificationTracker.cs b/src/Backend/AutopilotMonitor.Functions/DataAccess/TableStorage/TableHardwareRejectionNotificationTracker.cs index e6a73915..c413939b 100644 --- a/src/Backend/AutopilotMonitor.Functions/DataAccess/TableStorage/TableHardwareRejectionNotificationTracker.cs +++ b/src/Backend/AutopilotMonitor.Functions/DataAccess/TableStorage/TableHardwareRejectionNotificationTracker.cs @@ -17,6 +17,9 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage /// cannot collide with a hardware key because no real manufacturer is named "tpmpss") /// - F3 rule regression: "ruleregression|{ruleId-lower}" (trimmed; payload-carrying rows /// that are refreshed while the episode is active and deleted on re-arm) + /// - app-version duration regression: "appversionregression|{app-lower}|{version-lower}" + /// (trimmed + table-key-sanitized; same payload-carrying episode semantics — raw + /// AppName/CurrentVersion live in columns, the mapper never parses the RowKey) /// Race-safe via AddEntityAsync: Azure Table Storage returns 409 Conflict if the entity already exists. /// public class TableHardwareRejectionNotificationTracker : IHardwareRejectionNotificationTracker @@ -251,6 +254,107 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage return $"tpmpss|{sn}"; } + public async Task TryRegisterAppVersionRegressionAsync(string tenantId, AppVersionRegressionAlert alert) + { + if (string.IsNullOrWhiteSpace(tenantId) + || string.IsNullOrWhiteSpace(alert?.AppName) + || string.IsNullOrWhiteSpace(alert?.CurrentVersion)) + { + return false; + } + + try + { + await _table.AddEntityAsync(BuildAppVersionRegressionEntity(tenantId, alert!)); + _logger.LogInformation( + "AppVersionRegression tracker registered: tenant={TenantId} app={AppName} version={Version}", + tenantId, alert!.AppName, alert!.CurrentVersion); + return true; + } + catch (RequestFailedException ex) when (ex.Status == 409) + { + // Episode already active — one bell per (app, version) episode. + return false; + } + catch (Exception ex) + { + _logger.LogWarning(ex, + "AppVersionRegression tracker failed: tenant={TenantId} app={AppName} version={Version}", + tenantId, alert!.AppName, alert!.CurrentVersion); + // Fail closed so we do not double-fire if the row was actually written. + return false; + } + } + + public async Task RefreshAppVersionRegressionAsync(string tenantId, AppVersionRegressionAlert alert) + { + if (string.IsNullOrWhiteSpace(tenantId) + || string.IsNullOrWhiteSpace(alert?.AppName) + || string.IsNullOrWhiteSpace(alert?.CurrentVersion)) + { + return; + } + try + { + // Replace-mode upsert with the FULL payload incl. the caller-carried + // FirstNotifiedAt — the retention sweep re-arms on the ORIGINAL age, so a + // refresh must never rejuvenate the row. + await _table.UpsertEntityAsync(BuildAppVersionRegressionEntity(tenantId, alert!), TableUpdateMode.Replace); + } + catch (Exception ex) + { + _logger.LogWarning(ex, + "AppVersionRegression tracker refresh failed: tenant={TenantId} app={AppName} version={Version} (stale numbers remain)", + tenantId, alert!.AppName, alert!.CurrentVersion); + } + } + + public async Task DeleteAppVersionRegressionAsync(string tenantId, string appName, string currentVersion) + { + if (string.IsNullOrWhiteSpace(tenantId) + || string.IsNullOrWhiteSpace(appName) + || string.IsNullOrWhiteSpace(currentVersion)) + { + return; + } + try + { + await _table.DeleteEntityAsync( + tenantId.ToLowerInvariant(), BuildAppVersionRegressionRowKey(appName, currentVersion)); + } + catch (RequestFailedException ex) when (ex.Status == 404) + { + // Already gone (retention sweep or concurrent pass) — idempotent. + } + catch (Exception ex) + { + _logger.LogWarning(ex, + "AppVersionRegression tracker delete failed: tenant={TenantId} app={AppName} version={Version} (episode stays until retention)", + tenantId, appName, currentVersion); + } + } + + public async Task> GetAppVersionRegressionsAsync(string tenantId) + { + var results = new List(); + if (string.IsNullOrWhiteSpace(tenantId)) return results; + try + { + var partitionKey = tenantId.ToLowerInvariant(); + // '}' (0x7D) sorts directly after '|' (0x7C) — prefix range over "appversionregression|…". + var filter = $"PartitionKey eq '{partitionKey}' and RowKey ge '{AppVersionRegressionRowKeyPrefix}' and RowKey lt 'appversionregression}}'"; + await foreach (var entity in _table.QueryAsync(filter: filter)) + { + results.Add(MapToAppVersionRegressionAlert(entity)); + } + } + catch (Exception ex) + { + _logger.LogWarning(ex, "AppVersionRegression tracker list failed for tenant {TenantId}", tenantId); + } + return results; + } + internal const string RuleRegressionRowKeyPrefix = "ruleregression|"; internal static string BuildRuleRegressionRowKey(string ruleId) @@ -317,5 +421,69 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage LastEvaluatedAt = entity.GetDateTimeOffset("LastEvaluatedAt")?.UtcDateTime ?? DateTime.MinValue, }; } + + internal const string AppVersionRegressionRowKeyPrefix = "appversionregression|"; + + /// + /// App names and versions are free-form agent payloads — they can contain characters + /// Azure Table keys reject ('/', '\', '#', '?'). Sanitized for the KEY only (worst case + /// two apps share an episode row — tolerable); the raw values live in columns and the + /// mapper reads them, never the RowKey. + /// + internal static string BuildAppVersionRegressionRowKey(string appName, string version) + { + var app = SanitizeTableKey((appName ?? string.Empty).Trim().ToLowerInvariant()); + var ver = SanitizeTableKey((version ?? string.Empty).Trim().ToLowerInvariant()); + return $"{AppVersionRegressionRowKeyPrefix}{app}|{ver}"; + } + + /// Replace characters not allowed in Azure Table Storage keys (/, \, #, ?). + private static string SanitizeTableKey(string key) + { + return key + .Replace("/", "_") + .Replace("\\", "_") + .Replace("#", "_") + .Replace("?", "_"); + } + + // internal static: entity builder + mapper are pinned by round-trip unit tests + // (table-storage-serialization rule — every property must survive Store→Map). + internal static TableEntity BuildAppVersionRegressionEntity(string tenantId, AppVersionRegressionAlert alert) + { + return new TableEntity( + tenantId.ToLowerInvariant(), BuildAppVersionRegressionRowKey(alert.AppName, alert.CurrentVersion)) + { + ["TenantId"] = tenantId, + ["AppName"] = alert.AppName, + ["CurrentVersion"] = alert.CurrentVersion, + ["PreviousVersion"] = alert.PreviousVersion ?? string.Empty, + ["CurrentMedianSeconds"] = alert.CurrentMedianSeconds, + ["PreviousMedianSeconds"] = alert.PreviousMedianSeconds, + ["CurrentMeasuredCount"] = alert.CurrentMeasuredCount, + ["PreviousMeasuredCount"] = alert.PreviousMeasuredCount, + ["Lift"] = alert.Lift, + ["FirstNotifiedAt"] = alert.FirstNotifiedAt, + ["LastEvaluatedAt"] = alert.LastEvaluatedAt, + }; + } + + internal static AppVersionRegressionAlert MapToAppVersionRegressionAlert(TableEntity entity) + { + return new AppVersionRegressionAlert + { + TenantId = entity.GetString("TenantId") ?? entity.PartitionKey, + AppName = entity.GetString("AppName") ?? string.Empty, + CurrentVersion = entity.GetString("CurrentVersion") ?? string.Empty, + PreviousVersion = entity.GetString("PreviousVersion") ?? string.Empty, + CurrentMedianSeconds = entity.GetInt32("CurrentMedianSeconds") ?? 0, + PreviousMedianSeconds = entity.GetInt32("PreviousMedianSeconds") ?? 0, + CurrentMeasuredCount = entity.GetInt32("CurrentMeasuredCount") ?? 0, + PreviousMeasuredCount = entity.GetInt32("PreviousMeasuredCount") ?? 0, + Lift = entity.GetDouble("Lift") ?? 0, + FirstNotifiedAt = entity.GetDateTimeOffset("FirstNotifiedAt")?.UtcDateTime ?? DateTime.MinValue, + LastEvaluatedAt = entity.GetDateTimeOffset("LastEvaluatedAt")?.UtcDateTime ?? DateTime.MinValue, + }; + } } } diff --git a/src/Backend/AutopilotMonitor.Functions/Functions/Apps/AppsAnalyticsHelper.cs b/src/Backend/AutopilotMonitor.Functions/Functions/Apps/AppsAnalyticsHelper.cs index 4fca336b..b5e46ba5 100644 --- a/src/Backend/AutopilotMonitor.Functions/Functions/Apps/AppsAnalyticsHelper.cs +++ b/src/Backend/AutopilotMonitor.Functions/Functions/Apps/AppsAnalyticsHelper.cs @@ -229,8 +229,13 @@ namespace AutopilotMonitor.Functions.Functions.Apps List allSummaries, ISessionRepository sessionRepo, string appName, - int days) + int days, + IReadOnlyList? versionRegressions = null) { + // Active duration-regression episodes for THIS app (tracker rows; models serialize + // camelCase on the wire like the rule-stats regressions[] block). + var appVersionRegressions = versionRegressions ?? Array.Empty(); + var now = DateTime.UtcNow; var cutoff = now.AddDays(-days); var midpoint = now.AddDays(-days / 2.0); @@ -275,7 +280,10 @@ namespace AutopilotMonitor.Functions.Functions.Apps installerPhaseBreakdown = Array.Empty(), topFailureCodes = Array.Empty(), detectionLiesCount = 0, - deviceModelBreakdown = Array.Empty() + deviceModelBreakdown = Array.Empty(), + // Lockstep with the full response: episodes can outlive the window's data + // (e.g. a shrunk days= selection), so the block is still surfaced here. + versionRegressions = appVersionRegressions }; } @@ -313,12 +321,23 @@ namespace AutopilotMonitor.Functions.Functions.Apps var vFailed = g.Count(s => s.Status == "Failed"); // Same PR0 convention as the top-level rate: skips are not attempts. var vSucceeded = g.Count(s => s.Status == "Succeeded" && !MetricsMath.IsSkipTerminalState(s)); + // Same measured population as the top-level duration stats: succeeded, + // non-skip, and a plausible observed duration (0s = start unobserved, + // >6h = back-stamped batch — both excluded, never averaged in). + var vDurations = g + .Where(s => s.Status == "Succeeded" && !MetricsMath.IsSkipTerminalState(s)) + .Where(MetricsMath.HasMeasuredDuration) + .Select(s => s.DurationSeconds) + .ToList(); return new { appVersion = g.Key, installs = vTotal, failed = vFailed, - failureRate = MetricsMath.TerminalFailureRatePct(vFailed, vSucceeded) + failureRate = MetricsMath.TerminalFailureRatePct(vFailed, vSucceeded), + measuredInstalls = vDurations.Count, + medianDurationSeconds = Percentile(vDurations, 0.50), + p95DurationSeconds = Percentile(vDurations, 0.95) }; }) .OrderByDescending(v => v.installs) @@ -421,7 +440,8 @@ namespace AutopilotMonitor.Functions.Functions.Apps installerPhaseBreakdown, topFailureCodes, detectionLiesCount, - deviceModelBreakdown + deviceModelBreakdown, + versionRegressions = appVersionRegressions }; } @@ -586,7 +606,9 @@ namespace AutopilotMonitor.Functions.Functions.Apps return date.AddDays(-diff); } - private static int Percentile(List values, double percentile) + // Internal so AppVersionRegressionRadar shares the same nearest-rank convention + // (a third percentile implementation would invite subtle median drift). + internal static int Percentile(List values, double percentile) { if (values.Count == 0) return 0; var sorted = values.OrderBy(v => v).ToList(); diff --git a/src/Backend/AutopilotMonitor.Functions/Functions/Apps/GetAppAnalyticsFunction.cs b/src/Backend/AutopilotMonitor.Functions/Functions/Apps/GetAppAnalyticsFunction.cs index bdb4106f..28e17849 100644 --- a/src/Backend/AutopilotMonitor.Functions/Functions/Apps/GetAppAnalyticsFunction.cs +++ b/src/Backend/AutopilotMonitor.Functions/Functions/Apps/GetAppAnalyticsFunction.cs @@ -17,15 +17,18 @@ namespace AutopilotMonitor.Functions.Functions.Apps private readonly ILogger _logger; private readonly IMetricsRepository _metricsRepo; private readonly ISessionRepository _sessionRepo; + private readonly IHardwareRejectionNotificationTracker _notificationTracker; public GetAppAnalyticsFunction( ILogger logger, IMetricsRepository metricsRepo, - ISessionRepository sessionRepo) + ISessionRepository sessionRepo, + IHardwareRejectionNotificationTracker notificationTracker) { _logger = logger; _metricsRepo = metricsRepo; _sessionRepo = sessionRepo; + _notificationTracker = notificationTracker; } [Function("GetAppAnalytics")] @@ -51,8 +54,12 @@ namespace AutopilotMonitor.Functions.Functions.Apps days = parsedDays; var summaries = await AppsAnalyticsHelper.LoadSummariesAsync(_metricsRepo, tenantId, days); + // Active duration-regression episodes for this app (fail-soft: empty on error). + var versionRegressions = (await _notificationTracker.GetAppVersionRegressionsAsync(tenantId)) + .Where(a => string.Equals(a.AppName, decodedAppName, StringComparison.OrdinalIgnoreCase)) + .ToList(); var body = await AppsAnalyticsHelper.BuildAnalyticsResponseAsync( - summaries, _sessionRepo, decodedAppName, days); + summaries, _sessionRepo, decodedAppName, days, versionRegressions); var response = req.CreateResponse(HttpStatusCode.OK); await response.WriteAsJsonAsync(body); diff --git a/src/Backend/AutopilotMonitor.Functions/Functions/Apps/GetGlobalAppAnalyticsFunction.cs b/src/Backend/AutopilotMonitor.Functions/Functions/Apps/GetGlobalAppAnalyticsFunction.cs index d67ec325..7e25b4e7 100644 --- a/src/Backend/AutopilotMonitor.Functions/Functions/Apps/GetGlobalAppAnalyticsFunction.cs +++ b/src/Backend/AutopilotMonitor.Functions/Functions/Apps/GetGlobalAppAnalyticsFunction.cs @@ -1,6 +1,7 @@ using System.Net; using AutopilotMonitor.Functions.Helpers; using AutopilotMonitor.Shared.DataAccess; +using AutopilotMonitor.Shared.Models; using Microsoft.Azure.Functions.Worker; using Microsoft.Azure.Functions.Worker.Http; using Microsoft.Extensions.Logging; @@ -17,15 +18,18 @@ namespace AutopilotMonitor.Functions.Functions.Apps private readonly ILogger _logger; private readonly IMetricsRepository _metricsRepo; private readonly ISessionRepository _sessionRepo; + private readonly IHardwareRejectionNotificationTracker _notificationTracker; public GetGlobalAppAnalyticsFunction( ILogger logger, IMetricsRepository metricsRepo, - ISessionRepository sessionRepo) + ISessionRepository sessionRepo, + IHardwareRejectionNotificationTracker notificationTracker) { _logger = logger; _metricsRepo = metricsRepo; _sessionRepo = sessionRepo; + _notificationTracker = notificationTracker; } [Function("GetGlobalAppAnalytics")] @@ -61,8 +65,15 @@ namespace AutopilotMonitor.Functions.Functions.Apps decodedAppName, userEmail, scopedTenantId ?? "", days); var summaries = await AppsAnalyticsHelper.LoadSummariesAsync(_metricsRepo, scopedTenantId, days); + // Episodes are per-tenant tracker rows; without a tenant scope the aggregated + // cross-tenant view carries none (mirrors the rule-stats global route). + var versionRegressions = string.IsNullOrEmpty(scopedTenantId) + ? new List() + : (await _notificationTracker.GetAppVersionRegressionsAsync(scopedTenantId!)) + .Where(a => string.Equals(a.AppName, decodedAppName, StringComparison.OrdinalIgnoreCase)) + .ToList(); var body = await AppsAnalyticsHelper.BuildAnalyticsResponseAsync( - summaries, _sessionRepo, decodedAppName, days); + summaries, _sessionRepo, decodedAppName, days, versionRegressions); var response = req.CreateResponse(HttpStatusCode.OK); await response.WriteAsJsonAsync(body); diff --git a/src/Backend/AutopilotMonitor.Functions/Helpers/AppVersionRegressionRadar.cs b/src/Backend/AutopilotMonitor.Functions/Helpers/AppVersionRegressionRadar.cs new file mode 100644 index 00000000..ee57d36f --- /dev/null +++ b/src/Backend/AutopilotMonitor.Functions/Helpers/AppVersionRegressionRadar.cs @@ -0,0 +1,208 @@ +using AutopilotMonitor.Functions.Functions.Apps; +using AutopilotMonitor.Shared.Models; + +namespace AutopilotMonitor.Functions.Helpers; + +/// One fired duration regression: an app whose newest version installs much slower than its predecessor (see ). +public sealed class AppVersionDurationRegressionFinding +{ + public string TenantId { get; init; } = string.Empty; + public string AppName { get; init; } = string.Empty; + public string CurrentVersion { get; init; } = string.Empty; + public string PreviousVersion { get; init; } = string.Empty; + + public int CurrentMedianSeconds { get; init; } + public int PreviousMedianSeconds { get; init; } + public int CurrentMeasuredCount { get; init; } + public int PreviousMeasuredCount { get; init; } + + /// Current median ÷ previous median, one decimal. Never null — measured durations are ≥1s, so the previous median cannot be 0. + public double Lift { get; init; } +} + +/// +/// Deterministic, I/O-free core of the app-version duration regression radar. Detects +/// "version X made this app's installs much slower" from the per-(session, app) install +/// summaries: per (tenant, app), the newest version's MEDIAN measured install duration is +/// compared against the previous version's median. Medians (nearest-rank, shared with the +/// apps dashboard via ) rather than means — +/// a single back-stamped straggler must not fire a fleet alert. A regression fires only +/// when ALL hold: +/// +/// both versions have ≥ 10 measured installs inside the horizon; +/// current median ≥ 2× the previous version's median; +/// absolute increase ≥ 300 s — a 20s→45s app is noise, not an incident. +/// +/// Measured population: Succeeded, non-skip terminal state, plausible observed duration +/// (), no AppId collision, non-empty +/// AppVersion. Version ORDER comes from first-seen install time (min StartedAt) — version +/// strings are never sorted lexically ("9.1" vs "2024.10" would lie). "Previous" is the +/// version with the latest first-seen strictly before the current version's first-seen; +/// with parallel ring rollouts that can be a concurrently-deployed version — accepted for +/// v1, the episode key (app, current version) still caps it at one bell per version. +/// +public static class AppVersionRegressionRadar +{ + /// Days of install summaries the radar loads and reasons over (mirrors the rule radar's 7+28 horizon). + public const int HorizonDays = 35; + public const int MinMeasuredInstalls = 10; + public const double MinMedianLift = 2.0; + public const int MinAbsoluteIncreaseSeconds = 300; + + /// An active alert re-arms once the current median falls back under 1.5× the previous version's median. + public const double ReArmLiftFactor = 1.5; + + /// Per-version duration stats inside the horizon (see ). + public sealed class VersionStats + { + public string Version { get; init; } = string.Empty; + public DateTime FirstSeen { get; init; } + public DateTime LastSeen { get; init; } + public int MeasuredCount { get; init; } + public int MedianSeconds { get; init; } + } + + /// + /// Evaluates one tenant's install summaries (loaded for the last + /// days). Returns at most one finding per app — the newest version vs. its predecessor; + /// older version pairs are yesterday's news, not an alert. Deterministic ordering: + /// lift descending, then app name ordinal. + /// + public static List Evaluate(IReadOnlyList tenantSummaries) + { + var findings = new List(); + foreach (var app in MeasuredAppGroups(tenantSummaries)) + { + var stats = ComputeVersionStats(app.Value); + var (current, previous) = SelectComparisonPair(stats); + if (current == null || previous == null) continue; + + if (current.MeasuredCount < MinMeasuredInstalls) continue; + if (previous.MeasuredCount < MinMeasuredInstalls) continue; + if (current.MedianSeconds < MinMedianLift * previous.MedianSeconds) continue; + if (current.MedianSeconds - previous.MedianSeconds < MinAbsoluteIncreaseSeconds) continue; + + findings.Add(new AppVersionDurationRegressionFinding + { + TenantId = app.Value[0].TenantId, + AppName = app.Key, + CurrentVersion = current.Version, + PreviousVersion = previous.Version, + CurrentMedianSeconds = current.MedianSeconds, + PreviousMedianSeconds = previous.MedianSeconds, + CurrentMeasuredCount = current.MeasuredCount, + PreviousMeasuredCount = previous.MeasuredCount, + Lift = Math.Round((double)current.MedianSeconds / previous.MedianSeconds, 1), + }); + } + return findings + .OrderByDescending(f => f.Lift) + .ThenBy(f => f.AppName, StringComparer.Ordinal) + .ToList(); + } + + /// + /// True when an ACTIVE alert may re-arm: the alerted version has drained out of the + /// horizon (fewer than measured installs left — it was + /// superseded or pulled), or its recomputed median fell back under + /// × the previous version's recomputed median. When the + /// previous version itself has drained there is no comparison basis anymore — the + /// episode is kept until the current version drains (fires-stopped-style re-arm only). + /// + public static bool ShouldReArm(IReadOnlyList tenantSummaries, AppVersionRegressionAlert alert) + { + var stats = ComputeVersionStatsForApp(tenantSummaries, alert.AppName); + var current = stats.FirstOrDefault(s => string.Equals(s.Version, alert.CurrentVersion, StringComparison.Ordinal)); + if (current == null || current.MeasuredCount < MinMeasuredInstalls) return true; + + var previous = stats.FirstOrDefault(s => string.Equals(s.Version, alert.PreviousVersion, StringComparison.Ordinal)); + if (previous == null || previous.MeasuredCount < MinMeasuredInstalls) return false; + + return current.MedianSeconds < ReArmLiftFactor * previous.MedianSeconds; + } + + /// Per-version stats for one app (measured rows only), or empty when the app has none. + public static List ComputeVersionStatsForApp(IReadOnlyList tenantSummaries, string appName) + { + foreach (var app in MeasuredAppGroups(tenantSummaries)) + { + if (string.Equals(app.Key, appName, StringComparison.OrdinalIgnoreCase)) + return ComputeVersionStats(app.Value); + } + return new List(); + } + + /// + /// Groups the MEASURED rows by app name: succeeded, non-skip, plausible duration, + /// no AppId collision (a second app's outcomes must not fire this app's alert), + /// non-empty version. Rate/failure signals are deliberately out of scope — this + /// radar alerts on duration only. + /// + internal static Dictionary> MeasuredAppGroups(IReadOnlyList summaries) + { + var groups = new Dictionary>(StringComparer.OrdinalIgnoreCase); + foreach (var summary in summaries) + { + if (string.IsNullOrEmpty(summary.AppName)) continue; + if (string.IsNullOrEmpty(summary.AppVersion)) continue; + if (summary.AppIdCollision) continue; + if (summary.Status != "Succeeded") continue; + if (MetricsMath.IsSkipTerminalState(summary)) continue; + if (!MetricsMath.HasMeasuredDuration(summary)) continue; + + if (!groups.TryGetValue(summary.AppName, out var list)) + { + list = new List(); + groups[summary.AppName] = list; + } + list.Add(summary); + } + return groups; + } + + /// Per-version first/last-seen and nearest-rank median over one app's measured rows. + internal static List ComputeVersionStats(IReadOnlyList measuredAppRows) + { + return measuredAppRows + .GroupBy(s => s.AppVersion, StringComparer.Ordinal) + .Select(g => + { + var durations = g.Select(s => s.DurationSeconds).ToList(); + return new VersionStats + { + Version = g.Key, + FirstSeen = g.Min(s => s.StartedAt), + LastSeen = g.Max(s => s.StartedAt), + MeasuredCount = durations.Count, + MedianSeconds = AppsAnalyticsHelper.Percentile(durations, 0.50), + }; + }) + .ToList(); + } + + /// + /// Current = the version of the most recent measured install (ties broken by later + /// first-seen, then version ordinal — deterministic); previous = the version with the + /// latest first-seen strictly before the current version's first-seen (same tie-break). + /// Either is null when the app has fewer than two versions in the horizon. + /// + internal static (VersionStats? Current, VersionStats? Previous) SelectComparisonPair(IReadOnlyList stats) + { + if (stats.Count < 2) return (null, null); + + var current = stats + .OrderByDescending(s => s.LastSeen) + .ThenByDescending(s => s.FirstSeen) + .ThenByDescending(s => s.Version, StringComparer.Ordinal) + .First(); + + var previous = stats + .Where(s => s.FirstSeen < current.FirstSeen) + .OrderByDescending(s => s.FirstSeen) + .ThenByDescending(s => s.LastSeen) + .ThenByDescending(s => s.Version, StringComparer.Ordinal) + .FirstOrDefault(); + + return (current, previous); + } +} diff --git a/src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.AppVersionRegression.cs b/src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.AppVersionRegression.cs new file mode 100644 index 00000000..b9fa26e1 --- /dev/null +++ b/src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.AppVersionRegression.cs @@ -0,0 +1,212 @@ +using System; +using System.Diagnostics; +using System.Linq; +using System.Threading.Tasks; +using AutopilotMonitor.Functions.Helpers; +using AutopilotMonitor.Shared.Models; +using Microsoft.Extensions.Logging; + +namespace AutopilotMonitor.Functions.Services +{ + /// + /// Partial: app-version duration regression radar (sibling of the F3 rule radar). + /// + public partial class MaintenanceService + { + /// App setting kill switch: set to "true" to skip the radar entirely. Fail-open — the radar only notifies, it never mutates data. + internal const string AppVersionRadarKillSwitchSetting = "AppVersionRegressionRadarDisabled"; + + /// + /// Detects per-tenant apps whose newest version's median install duration regressed + /// (≥2× and ≥5 min over the previous version — ) + /// and reconciles the alert episodes in the notification tracker: + /// new finding → tracker row + tenant bell + AppVersionDurationRegression ops event, + /// exactly once per (app, version) episode; still-regressed → numbers refreshed + /// (FirstNotifiedAt untouched); no longer regressed → re-arm check (version drained, + /// or median back under 1.5×) deletes the row so a future regression rings again. + /// The tracker's 30d retention sweep re-arms long-burning episodes by design. + /// + /// One cross-tenant install-summary read over the trailing + /// days (the same query the global + /// apps dashboard runs), then grouped by tenant — mirrors the rule radar's enumeration. + /// Idempotent per pass via the tracker dedup. Fail-soft per tenant and overall. + /// + /// + private async Task RunAppVersionRegressionRadarAsync() + { + try + { + if (string.Equals(_configuration[AppVersionRadarKillSwitchSetting], "true", StringComparison.OrdinalIgnoreCase)) + { + _logger.LogInformation("App-version regression radar skipped ({Setting}=true)", AppVersionRadarKillSwitchSetting); + return; + } + + var sw = Stopwatch.StartNew(); + var sinceUtc = DateTime.UtcNow.AddDays(-AppVersionRegressionRadar.HorizonDays); + var summaries = await _metricsRepo.GetAppsDashboardSummariesAsync(sinceUtc, tenantId: null); + + var byTenant = summaries + .Where(s => !string.IsNullOrEmpty(s.TenantId)) + .GroupBy(s => s.TenantId, StringComparer.OrdinalIgnoreCase); + + int fired = 0, refreshed = 0, rearmed = 0, tenants = 0; + foreach (var tenantGroup in byTenant) + { + tenants++; + try + { + var (f, r, a) = await EvaluateTenantAppVersionRegressionsAsync(tenantGroup.Key, tenantGroup.ToList()); + fired += f; + refreshed += r; + rearmed += a; + } + catch (Exception tenantEx) + { + _logger.LogWarning(tenantEx, "App-version regression radar failed for tenant {TenantId} (non-fatal)", tenantGroup.Key); + } + } + + sw.Stop(); + _logger.LogInformation( + "App-version regression radar: {Tenants} tenants over {Rows} install summaries — {Fired} fired, {Refreshed} refreshed, {Rearmed} re-armed in {Ms}ms", + tenants, summaries.Count, fired, refreshed, rearmed, sw.ElapsedMilliseconds); + } + catch (Exception ex) + { + _logger.LogWarning(ex, "App-version regression radar failed (non-fatal)"); + } + } + + private async Task<(int Fired, int Refreshed, int Rearmed)> EvaluateTenantAppVersionRegressionsAsync( + string tenantId, System.Collections.Generic.List tenantSummaries) + { + var findings = AppVersionRegressionRadar.Evaluate(tenantSummaries); + var active = await _hardwareRejectionTracker.GetAppVersionRegressionsAsync(tenantId); + var activeByKey = active.ToDictionary( + a => AppVersionEpisodeKey(a.AppName, a.CurrentVersion), StringComparer.OrdinalIgnoreCase); + + int fired = 0, refreshed = 0, rearmed = 0; + var firedKeys = new System.Collections.Generic.HashSet(StringComparer.OrdinalIgnoreCase); + + foreach (var finding in findings) + { + var key = AppVersionEpisodeKey(finding.AppName, finding.CurrentVersion); + firedKeys.Add(key); + + if (activeByKey.TryGetValue(key, out var existing)) + { + // Episode still burning: refresh numbers for the versionRegressions[] block; + // FirstNotifiedAt never moves (retention re-arm counts from the bell). + await _hardwareRejectionTracker.RefreshAppVersionRegressionAsync( + tenantId, BuildAppVersionAlert(finding, existing.FirstNotifiedAt)); + refreshed++; + continue; + } + + var alert = BuildAppVersionAlert(finding, DateTime.UtcNow); + if (!await _hardwareRejectionTracker.TryRegisterAppVersionRegressionAsync(tenantId, alert)) + continue; // a concurrent pass won the episode — its bell suffices + + fired++; + await _tenantNotificationService.CreateNotificationAsync( + tenantId, + type: "app_version_duration_regression", + title: $"Install duration regressed: {finding.AppName}", + message: BuildAppVersionRegressionMessage(finding), + // Canonical appDetailUrl shape (lib/routes.ts) with the radar's horizon window. + href: $"/apps/detail?name={Uri.EscapeDataString(finding.AppName)}&days={AppVersionRegressionRadar.HorizonDays}"); + await _opsEventService.RecordAppVersionDurationRegressionAsync( + tenantId, finding.AppName, finding.CurrentVersion, finding.PreviousVersion, + finding.CurrentMedianSeconds, finding.PreviousMedianSeconds, + finding.CurrentMeasuredCount, finding.PreviousMeasuredCount, finding.Lift); + } + + foreach (var alert in active) + { + if (firedKeys.Contains(AppVersionEpisodeKey(alert.AppName, alert.CurrentVersion))) + continue; + + if (AppVersionRegressionRadar.ShouldReArm(tenantSummaries, alert)) + { + await _hardwareRejectionTracker.DeleteAppVersionRegressionAsync(tenantId, alert.AppName, alert.CurrentVersion); + rearmed++; + } + else + { + // Elevated but below the fire gates (between 1.5× and 2×, or the absolute + // delta shrank): keep the episode, refresh the numbers so the + // versionRegressions[] block stays honest. + var refreshedAlert = TryBuildOngoingAppVersionAlert(tenantSummaries, alert); + if (refreshedAlert != null) + { + await _hardwareRejectionTracker.RefreshAppVersionRegressionAsync(tenantId, refreshedAlert); + refreshed++; + } + } + } + + return (fired, refreshed, rearmed); + } + + private static string AppVersionEpisodeKey(string appName, string version) + => $"{appName}\n{version}"; + + private static AppVersionRegressionAlert BuildAppVersionAlert( + AppVersionDurationRegressionFinding finding, DateTime firstNotifiedAt) + => new() + { + TenantId = finding.TenantId, + AppName = finding.AppName, + CurrentVersion = finding.CurrentVersion, + PreviousVersion = finding.PreviousVersion, + CurrentMedianSeconds = finding.CurrentMedianSeconds, + PreviousMedianSeconds = finding.PreviousMedianSeconds, + CurrentMeasuredCount = finding.CurrentMeasuredCount, + PreviousMeasuredCount = finding.PreviousMeasuredCount, + Lift = finding.Lift, + FirstNotifiedAt = firstNotifiedAt, + LastEvaluatedAt = DateTime.UtcNow, + }; + + /// + /// Refresh payload for a still-open episode below the fire gates: recomputed medians + /// against the SAME version pair, everything identifying carried over. Null when either + /// version has left the horizon — the stale numbers then remain until re-arm/retention. + /// + private static AppVersionRegressionAlert? TryBuildOngoingAppVersionAlert( + System.Collections.Generic.IReadOnlyList tenantSummaries, AppVersionRegressionAlert existing) + { + var stats = AppVersionRegressionRadar.ComputeVersionStatsForApp(tenantSummaries, existing.AppName); + var current = stats.FirstOrDefault(s => string.Equals(s.Version, existing.CurrentVersion, StringComparison.Ordinal)); + var previous = stats.FirstOrDefault(s => string.Equals(s.Version, existing.PreviousVersion, StringComparison.Ordinal)); + if (current == null || previous == null || previous.MedianSeconds <= 0) + return null; + + return new AppVersionRegressionAlert + { + TenantId = existing.TenantId, + AppName = existing.AppName, + CurrentVersion = existing.CurrentVersion, + PreviousVersion = existing.PreviousVersion, + CurrentMedianSeconds = current.MedianSeconds, + PreviousMedianSeconds = previous.MedianSeconds, + CurrentMeasuredCount = current.MeasuredCount, + PreviousMeasuredCount = previous.MeasuredCount, + Lift = Math.Round((double)current.MedianSeconds / previous.MedianSeconds, 1), + FirstNotifiedAt = existing.FirstNotifiedAt, + LastEvaluatedAt = DateTime.UtcNow, + }; + } + + /// Bell message with the full numbers (the admin can verify without a portal round-trip). Minutes, one decimal. + internal static string BuildAppVersionRegressionMessage(AppVersionDurationRegressionFinding finding) + { + var fromMin = Math.Round(finding.PreviousMedianSeconds / 60.0, 1); + var toMin = Math.Round(finding.CurrentMedianSeconds / 60.0, 1); + return $"Median install duration rose from {fromMin} to {toMin} min after version {finding.CurrentVersion} " + + $"({finding.CurrentMeasuredCount} measured installs vs {finding.PreviousMeasuredCount} on version {finding.PreviousVersion}) " + + $"— lift {finding.Lift}x over the last {AppVersionRegressionRadar.HorizonDays} days."; + } + } +} diff --git a/src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.cs b/src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.cs index c6590f6e..a36d02bf 100644 --- a/src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.cs +++ b/src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.cs @@ -148,6 +148,9 @@ namespace AutopilotMonitor.Functions.Services // aggregation so the window rows are fresh. Anchored on YESTERDAY — whole // days only, a partial today would understate the window rate. Fail-soft. await RunRuleRegressionRadarAsync(DateTime.UtcNow.Date.AddDays(-1)); + // App-version duration regression radar: same episode/tracker pattern over the + // install summaries (trailing 35d horizon loaded internally). Fail-soft. + await RunAppVersionRegressionRadarAsync(); // F1 PR2: rolling 30d breakdown backfill + daily attribution aggregates. Owns // its own window (NOT the snapshot-gated catch-up above) so late-terminating // sessions still reach their StartedAt-date's aggregate. Fail-soft internally. @@ -222,6 +225,7 @@ namespace AutopilotMonitor.Functions.Services // Timer-path parity: radar re-runs are idempotent (tracker dedup), so a manual // aggregation also re-evaluates regressions anchored on the aggregated date. await RunRuleRegressionRadarAsync(dateToAggregate); + await RunAppVersionRegressionRadarAsync(); // Timer-path parity: manual maintenance also refreshes the attribution // breakdowns + daily aggregates and the device-history/FTR rollups diff --git a/src/Backend/AutopilotMonitor.Functions/Services/OpsEventService.cs b/src/Backend/AutopilotMonitor.Functions/Services/OpsEventService.cs index a7334954..b4905171 100644 --- a/src/Backend/AutopilotMonitor.Functions/Services/OpsEventService.cs +++ b/src/Backend/AutopilotMonitor.Functions/Services/OpsEventService.cs @@ -542,6 +542,35 @@ namespace AutopilotMonitor.Functions.Services lift, }); + /// + /// App-version duration regression radar: an app's newest version installs with a median + /// duration ≥2× (and ≥5 min absolute) over the previous version's median. Fired ONCE per + /// (app, version) episode (the notification-tracker row is the dedup); the message + /// carries the full numbers so the event is verifiable without a portal round-trip. + /// Dual-registered in the web OPS_EVENT_TYPES catalog + /// (memory: feedback_ops_event_types_dual_register). + /// + public Task RecordAppVersionDurationRegressionAsync( + string tenantId, string appName, string currentVersion, string previousVersion, + int currentMedianSeconds, int previousMedianSeconds, + int currentMeasuredCount, int previousMeasuredCount, double lift) + => WriteAsync(OpsEventCategory.Tenant, "AppVersionDurationRegression", OpsEventSeverity.Warning, + $"App '{appName}' median install duration rose from {Math.Round(previousMedianSeconds / 60.0, 1)} to " + + $"{Math.Round(currentMedianSeconds / 60.0, 1)} min after version {currentVersion} " + + $"({currentMeasuredCount} measured installs vs {previousMeasuredCount} on {previousVersion}) — lift {lift}x", + tenantId, "System.Maintenance", + new + { + appName, + currentVersion, + previousVersion, + currentMedianSeconds, + previousMedianSeconds, + currentMeasuredCount, + previousMeasuredCount, + lift, + }); + // ── Agent ────────────────────────────────────────────────────────────── public Task RecordSessionTimeoutsAsync(string tenantId, int sessionCount, int timeoutHours) diff --git a/src/Backend/AutopilotMonitor.Functions/Services/TenantNotificationAudienceCatalog.cs b/src/Backend/AutopilotMonitor.Functions/Services/TenantNotificationAudienceCatalog.cs index 0e6bdf72..116dcd21 100644 --- a/src/Backend/AutopilotMonitor.Functions/Services/TenantNotificationAudienceCatalog.cs +++ b/src/Backend/AutopilotMonitor.Functions/Services/TenantNotificationAudienceCatalog.cs @@ -33,6 +33,8 @@ public static class TenantNotificationAudienceCatalog // F3 regression radar: rule-tuning is an admin concern (the alert deep-links into the // analyze-rules page, which members cannot act on). ["rule_frequency_regression"] = NotificationAudience.Admin, + // App-version duration regression radar: packaging/rollout is an admin concern. + ["app_version_duration_regression"] = NotificationAudience.Admin, }; public static NotificationAudience Resolve(string? type) diff --git a/src/Shared/AutopilotMonitor.Shared/DataAccess/IHardwareRejectionNotificationTracker.cs b/src/Shared/AutopilotMonitor.Shared/DataAccess/IHardwareRejectionNotificationTracker.cs index 74fc77a7..38038bc3 100644 --- a/src/Shared/AutopilotMonitor.Shared/DataAccess/IHardwareRejectionNotificationTracker.cs +++ b/src/Shared/AutopilotMonitor.Shared/DataAccess/IHardwareRejectionNotificationTracker.cs @@ -8,9 +8,10 @@ namespace AutopilotMonitor.Shared.DataAccess /// /// Notification dedup store (one table, RowKey-prefixed key spaces). Subjects: /// hardware rejections per (tenant, manufacturer, model), TPM PSS incompatibilities per - /// (tenant, serial number) — both lifetime insert-once — and F3 rule-frequency - /// regressions per (tenant, ruleId), whose rows additionally carry the alert payload, - /// are refreshed while the episode stays active, and are deleted when the rate re-arms. + /// (tenant, serial number) — both lifetime insert-once — F3 rule-frequency + /// regressions per (tenant, ruleId), and app-version duration regressions per + /// (tenant, app, version). The regression key spaces additionally carry the alert + /// payload, are refreshed while the episode stays active, and are deleted on re-arm. /// public interface IHardwareRejectionNotificationTracker { @@ -54,6 +55,30 @@ namespace AutopilotMonitor.Shared.DataAccess /// Active regression episodes of one tenant (RowKey-prefix scan; empty on failure — fail-soft reads). Task> GetRuleRegressionsAsync(string tenantId); + // --- App-version duration regressions (RowKey "appversionregression|{app}|{version}") --- + + /// + /// Atomically opens a duration-regression episode for (tenantId, alert.AppName, + /// alert.CurrentVersion). Returns true when this pass owns the episode (caller fires + /// bell + ops event exactly once), false when an episode is already active or on + /// failure (fail-closed — never double-fire). + /// + Task TryRegisterAppVersionRegressionAsync(string tenantId, AppVersionRegressionAlert alert); + + /// + /// Refreshes an active episode's numbers (medians, counts, LastEvaluatedAt) so the + /// versionRegressions[] block stays current. FirstNotifiedAt is carried from the alert + /// unchanged — the retention sweep re-arms on the ORIGINAL notification age. + /// Fail-soft; never fires notifications. + /// + Task RefreshAppVersionRegressionAsync(string tenantId, AppVersionRegressionAlert alert); + + /// Closes an episode (median re-armed or the version drained out of the horizon). 404-tolerant. + Task DeleteAppVersionRegressionAsync(string tenantId, string appName, string currentVersion); + + /// Active app-version regression episodes of one tenant (RowKey-prefix scan; empty on failure — fail-soft reads). + Task> GetAppVersionRegressionsAsync(string tenantId); + /// /// Retention cleanup: deletes tracker rows whose FirstNotifiedAt is older than /// . Every key space lives in one table, so pruning re-arms diff --git a/src/Shared/AutopilotMonitor.Shared/Models/Metrics/AppVersionRegressionAlert.cs b/src/Shared/AutopilotMonitor.Shared/Models/Metrics/AppVersionRegressionAlert.cs new file mode 100644 index 00000000..9cd30762 --- /dev/null +++ b/src/Shared/AutopilotMonitor.Shared/Models/Metrics/AppVersionRegressionAlert.cs @@ -0,0 +1,43 @@ +using System; + +namespace AutopilotMonitor.Shared.Models +{ + /// + /// One ACTIVE app-version duration regression: an app whose newest version's median + /// install duration rose ≥2× (and ≥5 minutes absolute) over the previous version's + /// median, both sides with enough measured installs. Persisted as the + /// appversionregression|{app}|{version} keyspace of the notification tracker + /// table — the row IS the dedup (one bell per episode), and the + /// versionRegressions[] payload on the app-analytics response. Deleted when the + /// episode re-arms (median falls back under 1.5× or the version drains out of the + /// horizon) or by the tracker's 30-day retention sweep. Numbers refresh on every radar + /// pass while the episode stays active; never moves. + /// + public class AppVersionRegressionAlert + { + public string TenantId { get; set; } = string.Empty; + public string AppName { get; set; } = string.Empty; + + /// The regressed (newer) version — episode key together with the app. + public string CurrentVersion { get; set; } = string.Empty; + + /// The comparison version: latest first-seen strictly before the current version's first-seen. + public string PreviousVersion { get; set; } = string.Empty; + + public int CurrentMedianSeconds { get; set; } + public int PreviousMedianSeconds { get; set; } + + /// Measured installs backing each median (gate: both ≥10). + public int CurrentMeasuredCount { get; set; } + public int PreviousMeasuredCount { get; set; } + + /// Current median ÷ previous median (gate: ≥2.0; previous median is never 0 — measured durations are ≥1s). + public double Lift { get; set; } + + /// When the episode first fired (bell + ops event moment). Never moves on refresh; drives the 30d retention re-arm. + public DateTime FirstNotifiedAt { get; set; } + + /// Last radar pass that re-confirmed/refreshed this episode. + public DateTime LastEvaluatedAt { get; set; } + } +} diff --git a/src/Web/autopilot-monitor-web/app/admin/components/OpsAlertRulesSection.tsx b/src/Web/autopilot-monitor-web/app/admin/components/OpsAlertRulesSection.tsx index 53e87de2..8f15a6bc 100644 --- a/src/Web/autopilot-monitor-web/app/admin/components/OpsAlertRulesSection.tsx +++ b/src/Web/autopilot-monitor-web/app/admin/components/OpsAlertRulesSection.tsx @@ -108,6 +108,11 @@ const OPS_EVENT_TYPES: Record = { // (Wilson-separated). Warning-tier, once per episode (tracker-deduped). Dual-register // per memory feedback_ops_event_types_dual_register. "RuleFrequencyRegression", + // App-version duration regression radar: an app's newest version installs with a median + // duration ≥2× (and ≥5 min absolute) over the previous version. Warning-tier, once per + // (app, version) episode (tracker-deduped). Dual-register per memory + // feedback_ops_event_types_dual_register. + "AppVersionDurationRegression", // Auto-activation audit: the tenant-auto-approve worker activated a new signup // (AutoApproveNewTenants enabled). Info-tier. Dual-register per memory // feedback_ops_event_types_dual_register. diff --git a/src/Web/autopilot-monitor-web/app/apps/detail/page.tsx b/src/Web/autopilot-monitor-web/app/apps/detail/page.tsx index d7e92418..69c80598 100644 --- a/src/Web/autopilot-monitor-web/app/apps/detail/page.tsx +++ b/src/Web/autopilot-monitor-web/app/apps/detail/page.tsx @@ -51,6 +51,25 @@ interface VersionRow { installs: number; failed: number; failureRate: number; + /** Succeeded installs with a plausible observed duration — the basis of the median/p95 below. */ + measuredInstalls: number; + medianDurationSeconds: number; + p95DurationSeconds: number; +} + +/** Active duration-regression episode from the radar (tracker row, camelCase on the wire). */ +interface AppVersionRegressionRow { + tenantId: string; + appName: string; + currentVersion: string; + previousVersion: string; + currentMedianSeconds: number; + previousMedianSeconds: number; + currentMeasuredCount: number; + previousMeasuredCount: number; + lift: number; + firstNotifiedAt: string; + lastEvaluatedAt: string; } interface PhaseRow { @@ -102,6 +121,7 @@ interface AnalyticsResponse { topFailureCodes: FailureCodeRow[]; detectionLiesCount: number; deviceModelBreakdown: DeviceModelRow[]; + versionRegressions: AppVersionRegressionRow[]; } interface SessionRow { @@ -527,6 +547,23 @@ function AppDetailContent() { + {/* Duration regression banner: active radar episodes for this app (bell-deduped + server-side; this is the durable badge while the episode burns). */} + {(analytics.versionRegressions ?? []).length > 0 && ( +
+ {analytics.versionRegressions.map((reg) => ( +

+ ↑ Duration regression:{" "} + median install duration rose from {(reg.previousMedianSeconds / 60).toFixed(1)} to{" "} + {(reg.currentMedianSeconds / 60).toFixed(1)} min after version{" "} + {reg.currentVersion}{" "} + ({reg.currentMeasuredCount} measured installs vs {reg.previousMeasuredCount} on{" "} + {reg.previousVersion} — lift {reg.lift}x) +

+ ))} +
+ )} + {/* Charts row 2: version breakdown + phase breakdown. Each card is hidden when empty so the row collapses gracefully: - Version: will appear once new agents emit AppVersion data @@ -547,6 +584,17 @@ function AppDetailContent() { /> )} + {analytics.versionBreakdown.some((v) => v.measuredInstalls > 0) && ( + + v.measuredInstalls > 0) as unknown as Array>} + categoryKey="appVersion" + valueKey="medianDurationSeconds" + valueFormatter={(v) => formatDuration(Number(v))} + barColor={chartColors.primary} + /> + + )} {analytics.installerPhaseBreakdown.length > 0 && (