insights: app-version duration regression radar + per-version duration stats

versionBreakdown on the app analytics responses now carries measuredInstalls,
medianDurationSeconds and p95DurationSeconds per version (measured population:
succeeded, non-skip, plausible duration, no AppId collision). A new pure
AppVersionRegressionRadar detects "median install duration rose from X to Y min
after version Z": versions ordered by first-seen install time (never string
sort), fires at median lift >=2x AND >=300s absolute with >=10 measured
installs on both sides.

Episodes follow the rule-radar pattern in a fourth tracker keyspace
appversionregression|{app}|{version} (409-dedup, refresh keeps FirstNotifiedAt,
re-arm on drain / median back under 1.5x, 30d retention sweep re-arms): one
tenant bell app_version_duration_regression (Admin audience, appDetailUrl href)
plus one AppVersionDurationRegression ops event per episode, dual-registered in
the web OPS_EVENT_TYPES catalog. Runs in the maintenance timer after the rule
radar over one cross-tenant 35d install-summary read; kill switch
AppVersionRegressionRadarDisabled (fail-open). Apps detail page shows an amber
regression banner and a median-duration-by-version chart; OKF concept doc added.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Oliver Kieselbach
2026-08-13 13:39:03 +02:00
parent 842db1603a
commit 321ae63c63
18 changed files with 1281 additions and 12 deletions
@@ -0,0 +1,98 @@
---
type: Concept
title: App Version Duration Regression Radar
description: How the radar detects app versions whose median install duration regressed vs. the previous version (median lift ≥2x plus an absolute floor over first-seen-ordered versions), how alert episodes live in the appversionregression tracker keyspace, and which surfaces show them.
resource: src/Backend/AutopilotMonitor.Functions/Helpers/AppVersionRegressionRadar.cs
tags:
- backend
- apps
- regression-radar
- insights
timestamp: 2026-08-13T12:00:00+02:00
---
# App Version Duration Regression Radar
Detects "version X made this app's installs much slower" — the concrete, quotable insight
("median install duration rose from 11 to 29 min after version X") — from the per-(session,
app) `AppInstallSummaries` rows. Duration only: failure-rate per version is already visible
in the apps dashboard's `versionBreakdown`, and rate regressions belong to Wilson-interval
statistics, not medians.
# Schema
## Signal & detection
Per (tenant, app), the MEASURED install population is: `Status == "Succeeded"`, terminal
state not Skipped/Postponed, `MetricsMath.HasMeasuredDuration` (1s..6h — 0s means the start
was unobserved, above 6h means back-stamped batches), no `AppIdCollision`, non-empty
`AppVersion`. Versions are ordered by FIRST-SEEN install time (min `StartedAt`), never by
string comparison ("9.1" vs "2024.10" would lie). The comparison pair is always the newest
version (the one with the most recent measured install) vs. the version with the latest
first-seen strictly before it.
MEDIANS, not means (nearest-rank, shared with the apps dashboard via
`AppsAnalyticsHelper.Percentile`) — one back-stamped straggler must not fire a fleet alert.
A regression fires only when ALL hold over the trailing 35-day horizon:
1. both versions have ≥ 10 measured installs;
2. current median ≥ 2× the previous version's median;
3. absolute increase ≥ 300 s (a 20s→45s app is noise, not an incident).
## Suppression & limits
* Parallel ring rollouts can make "previous" a concurrently-deployed version — accepted in
v1; the episode key (app, current version) still caps it at one bell per version.
* Platform kill switch: app setting `AppVersionRegressionRadarDisabled=true` skips the
radar (fail-open — it only notifies, never mutates data).
* One cross-tenant install-summary read per pass (the same query the global apps dashboard
runs), grouped by tenant; per-tenant failures are non-fatal.
## Alert episodes (tracker keyspace)
An episode lives as `appversionregression|{app-lower}|{version-lower}` (table-key-sanitized;
raw AppName/CurrentVersion live in columns, the mapper never parses the RowKey) in the
notification-tracker table, next to the `ruleregression|` keyspace: the row IS the dedup
(bell + ops event fire exactly once per episode) and the `versionRegressions[]` payload —
both medians, both sample counts, lift. Numbers refresh on every radar pass while the
episode stays active; `FirstNotifiedAt` never moves. The episode closes (dedup re-arms) when
the alerted version drains under 10 measured installs in the horizon or its recomputed
median falls back under 1.5× the previous version's median; the tracker's 30-day retention
sweep also re-arms, so a month-old still-burning regression rings once more by design.
## Notification chain (v1: bell + ops event)
* Tenant bell `app_version_duration_regression` (audience: Admin), href
`/apps/detail?name={app}&days=35` (canonical `appDetailUrl` shape). Message carries the
full numbers so the admin can verify without a portal round-trip.
* Ops event `AppVersionDurationRegression` (Tenant category, Warning) — dual-registered in
the web `OPS_EVENT_TYPES` catalog so operators can wire Telegram/Teams/Slack alert rules.
## Surfaces
* `apps/{appName}/analytics` (+ global twin with `?tenantId=`) responds with
`versionRegressions[]` (empty for the cross-tenant aggregate — episodes are per-tenant
rows), and `versionBreakdown[]` rows now carry `measuredInstalls` /
`medianDurationSeconds` / `p95DurationSeconds`.
* Apps detail page: amber "↑ Duration regression" banner above the version cards and a
"Median Install Duration by Version" bar chart.
# Examples
A tenant ships Contoso VPN 2.4.0 after 2.3.9. 2.3.9 has 40 measured installs, median 660s;
2.4.0 accumulates 12 measured installs with median 1740s. Lift 2.6×, absolute +1080s →
episode `appversionregression|contoso vpn|2.4.0` opens, one bell + one ops event fire.
Subsequent passes refresh the numbers only. When 2.4.1 replaces 2.4.0 and 2.4.0 drains
below 10 measured installs in the horizon, the episode deletes and the dedup re-arms.
# Citations
* `src/Backend/AutopilotMonitor.Functions/Helpers/AppVersionRegressionRadar.cs` — pure core
(gates, first-seen ordering, re-arm).
* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.AppVersionRegression.cs`
— orchestration, kill switch, bell/ops-event emission, wording contract.
* `src/Backend/AutopilotMonitor.Functions/DataAccess/TableStorage/TableHardwareRejectionNotificationTracker.cs`
— episode keyspace, entity round-trip.
* `src/Shared/AutopilotMonitor.Shared/Models/Metrics/AppVersionRegressionAlert.cs` — persisted
alert payload.
* [Rule Regression Radar](rule-regression-radar.md) — the sibling pattern this radar mirrors.
+1
View File
@@ -39,6 +39,7 @@ https://docs.autopilotmonitor.com (separate repository).
* [Device Journeys & First-Time-Right](backend/device-journeys.md) - How a device's terminal enrollment attempts are grouped into journeys (trim+casefold device key with junk-serial exclusion, first-success end, 30-day gap, WhiteGlove = one attempt), the DeviceHistories chain with its tombstone-driven deleted-session pruning, the additive daily FTR aggregates, and the banner/fleet/MCP surfaces.
* [Enrollment Time Attribution](backend/time-attribution.md) - How a terminal session's authoritative wall clock (DurationSeconds, WhiteGlove pause excluded) is partitioned into named segments with an explicit unattributed remainder, ESP-blocking apps as positive evidence with "up to" what-if bounds, reboot spans from event gaps, and the breakdown/aggregate tables the sweep maintains.
* [Rule Regression Radar](backend/rule-regression-radar.md) - How analyze-rule hit-rate regressions are detected (7-day window ≥2× the 28-day baseline with disjoint Wilson intervals), which suppressions keep the radar quiet (grace period, edit-in-window, kill switch), how alert episodes live in the ruleregression tracker keyspace with re-arm semantics, and the on-fire dimension correlation with its "correlated, not necessarily causal" wording contract.
* [App Version Duration Regression Radar](backend/app-version-regression-radar.md) - How app versions whose median install duration regressed vs. the previous version are detected (first-seen-ordered versions, median lift ≥2× plus a 5-minute absolute floor over measured installs only), how alert episodes live in the appversionregression tracker keyspace with drain/median re-arm semantics, and the bell/ops-event/analytics surfaces.
* [Tenant Activation & the Once-Only Welcome Mail](backend/tenant-activation-welcome-mail.md) - How a signup becomes an activated tenant (manual approve or the auto-approve queue), and the write-then-read + conditional-marker arbitration that sends the welcome mail exactly once even though activation and the user-typed notification address race each other.
* [W365 Cloud PC Device Validation](backend/cloudpc-device-validation.md) - Why Cloud PCs can never pass the Autopilot serial lookup, and the cert-CN-bound CloudPc fallback validator: Subject CN (= Intune device id) matched against the tenant's Graph virtualEndpoint/cloudPCs inventory — admits exactly the service-provisioned Cloud PCs, nothing else.
* [ServerActions & On-Demand Diagnostics](backend/server-actions-on-demand-diagnostics.md) - The queue-on-session, deliver-on-ingest action channel, the TenantAdminOrOperator tier with in-function per-type re-gating, the portal's Collect Logs flow (quick-config orders rotate_config before request_diagnostics), and the hosted-prefix sweep in the offboarding cascade.
+4
View File
@@ -1,5 +1,9 @@
# Log
## 2026-08-13 (later)
* **Creation**: Added `backend/app-version-regression-radar.md` — app-version duration regression radar: per (tenant, app) the newest version's median measured install duration (nearest-rank; population Succeeded + non-skip + `HasMeasuredDuration` + no AppId collision + non-empty AppVersion) is compared against the previous version (first-seen ordering, never string sort); fires at median lift ≥2× AND ≥300s absolute with ≥10 measured installs on both sides. Episodes live as `appversionregression|{app}|{version}` tracker rows (one bell `app_version_duration_regression` + one `AppVersionDurationRegression` ops event per episode, refresh keeps `FirstNotifiedAt`, re-arm on drain/median-recovery/30d retention). Surfaces: `versionRegressions[]` + per-version `measuredInstalls`/`medianDurationSeconds`/`p95DurationSeconds` in the apps analytics responses, amber banner + duration-by-version chart on the apps detail page. Kill switch `AppVersionRegressionRadarDisabled`.
## 2026-08-13
* **Update**: `backend/session-annotations.md` — the `/annotations` portal overview page now carries the standard tenant switcher (GA/Reader: "All tenants" aggregate + per-tenant drill-down via `GET global/session-annotations`; delegated: managed subset), with a tenant column and tenant-qualified session deep links. The global list endpoint now derives `excludeGlobalAdminLane` from `HasGlobalScope`, so delegated-rescue callers on their managed `?tenantId=` path never receive the platform-internal `globaladmin` lane (endpoint-test-pinned).
@@ -0,0 +1,380 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text.Json;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.DataAccess.TableStorage;
using AutopilotMonitor.Functions.Functions.Apps;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Moq;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Pins the app-version duration regression radar (<see cref="AppVersionRegressionRadar"/>):
/// detection gates (≥10 measured installs on both sides, median lift ≥2×, ≥300s absolute),
/// first-seen version ordering (never string sort), the measured-population exclusions,
/// re-arm semantics, the tracker keyspace round-trip, the wording contract, and the
/// versionBreakdown duration fields on the analytics wire shape.
/// </summary>
public class AppVersionRegressionRadarTests
{
private const string TenantA = "00000000-0000-0000-0000-0000000000a1";
private const string App = "Contoso VPN";
private static readonly DateTime Now = new(2026, 8, 13, 12, 0, 0, DateTimeKind.Utc);
private static int _sessionCounter;
private static AppInstallSummary Install(
string version,
int durationSeconds,
DateTime? startedAt = null,
string appName = App,
string status = "Succeeded",
string terminalState = "Installed",
bool collision = false)
=> new()
{
TenantId = TenantA,
SessionId = $"s-{++_sessionCounter}",
AppName = appName,
AppVersion = version,
Status = status,
TerminalState = terminalState,
DurationSeconds = durationSeconds,
StartedAt = startedAt ?? Now.AddDays(-1),
AppIdCollision = collision,
};
/// <summary>n measured installs of one version, all with the same duration, spread over one day.</summary>
private static List<AppInstallSummary> Batch(string version, int count, int durationSeconds, DateTime firstSeen)
=> Enumerable.Range(0, count)
.Select(i => Install(version, durationSeconds, firstSeen.AddMinutes(i)))
.ToList();
// ── detection gates ─────────────────────────────────────────────────────
[Theory]
// currentCount, previousCount, previousMedian, currentMedian, expectFire
[InlineData(10, 10, 300, 900, true)] // lift 3×, +600s — fires
[InlineData(10, 10, 300, 600, true)] // exactly 2× AND exactly +300s — boundary fires
[InlineData(9, 10, 300, 900, false)] // current side below sample floor
[InlineData(10, 9, 300, 900, false)] // previous side below sample floor
[InlineData(10, 10, 200, 499, false)] // lift 2.5× but only +299s — absolute floor blocks
[InlineData(10, 10, 1000, 1900, false)] // +900s but lift 1.9× — lift gate blocks
public void Evaluate_Gates(int currentCount, int previousCount, int previousMedian, int currentMedian, bool expectFire)
{
var rows = Batch("1.0", previousCount, previousMedian, Now.AddDays(-20));
rows.AddRange(Batch("2.0", currentCount, currentMedian, Now.AddDays(-5)));
var findings = AppVersionRegressionRadar.Evaluate(rows);
if (!expectFire)
{
Assert.Empty(findings);
return;
}
var finding = Assert.Single(findings);
Assert.Equal(App, finding.AppName);
Assert.Equal("2.0", finding.CurrentVersion);
Assert.Equal("1.0", finding.PreviousVersion);
Assert.Equal(currentMedian, finding.CurrentMedianSeconds);
Assert.Equal(previousMedian, finding.PreviousMedianSeconds);
Assert.Equal(currentCount, finding.CurrentMeasuredCount);
Assert.Equal(previousCount, finding.PreviousMeasuredCount);
Assert.Equal(Math.Round((double)currentMedian / previousMedian, 1), finding.Lift);
}
[Fact]
public void Evaluate_MedianIsNearestRank_HandComputed()
{
// Even n: nearest-rank p50 takes the LOWER middle value (rank = ceil(0.5*4)-1 = 1).
var rows = new List<AppInstallSummary>();
foreach (var d in new[] { 100, 200, 300, 400 }) // median 200
rows.AddRange(Batch("1.0", 3, d, Now.AddDays(-20)));
foreach (var d in new[] { 500, 600, 700, 800 }) // median 600
rows.AddRange(Batch("2.0", 3, d, Now.AddDays(-5)));
var finding = Assert.Single(AppVersionRegressionRadar.Evaluate(rows));
Assert.Equal(600, finding.CurrentMedianSeconds);
Assert.Equal(200, finding.PreviousMedianSeconds);
Assert.Equal(3.0, finding.Lift);
}
// ── version ordering ────────────────────────────────────────────────────
[Fact]
public void Evaluate_VersionOrder_ByFirstSeen_NeverStringSort()
{
// Ordinally "9.1" > "2024.10", but "2024.10" is the NEWER version by first-seen.
// A string sort would flip current/previous and compare in the wrong direction.
var rows = Batch("9.1", 10, 300, Now.AddDays(-20));
rows.AddRange(Batch("2024.10", 10, 900, Now.AddDays(-5)));
var finding = Assert.Single(AppVersionRegressionRadar.Evaluate(rows));
Assert.Equal("2024.10", finding.CurrentVersion);
Assert.Equal("9.1", finding.PreviousVersion);
}
[Fact]
public void Evaluate_MultipleVersionsInFlight_PreviousIsLatestFirstSeenBeforeCurrent()
{
var rows = Batch("1.0", 10, 300, Now.AddDays(-30)); // oldest
rows.AddRange(Batch("1.1", 10, 320, Now.AddDays(-18))); // the true predecessor
rows.AddRange(Batch("2.0", 10, 960, Now.AddDays(-4))); // newest, regressed
var finding = Assert.Single(AppVersionRegressionRadar.Evaluate(rows));
Assert.Equal("2.0", finding.CurrentVersion);
Assert.Equal("1.1", finding.PreviousVersion); // not 1.0
Assert.Equal(320, finding.PreviousMedianSeconds);
}
[Fact]
public void Evaluate_SingleVersion_NeverFires()
{
var rows = Batch("1.0", 25, 900, Now.AddDays(-5));
Assert.Empty(AppVersionRegressionRadar.Evaluate(rows));
}
// ── measured-population exclusions ──────────────────────────────────────
[Fact]
public void Evaluate_ExcludedRows_DoNotCountTowardSamplesOrMedians()
{
var rows = Batch("1.0", 10, 300, Now.AddDays(-20));
rows.AddRange(Batch("2.0", 10, 900, Now.AddDays(-5)));
// None of these may move the samples or medians:
rows.Add(Install("2.0", 0, Now.AddDays(-2))); // unmeasured (start unobserved)
rows.Add(Install("2.0", 21601, Now.AddDays(-2))); // over the 6h plausibility cap
rows.Add(Install("2.0", 50, Now.AddDays(-2), terminalState: "Skipped")); // skip — not an attempt
rows.Add(Install("2.0", 50, Now.AddDays(-2), terminalState: "Postponed")); // skip — not an attempt
rows.Add(Install("2.0", 50, Now.AddDays(-2), status: "Failed")); // failed — no duration signal
rows.Add(Install("2.0", 50, Now.AddDays(-2), collision: true)); // AppId collision — foreign outcomes
rows.Add(Install("", 50, Now.AddDays(-2))); // no version
rows.Add(Install("2.0", 50, Now.AddDays(-2), appName: "")); // no app name
var finding = Assert.Single(AppVersionRegressionRadar.Evaluate(rows));
Assert.Equal(10, finding.CurrentMeasuredCount);
Assert.Equal(900, finding.CurrentMedianSeconds);
}
[Fact]
public void Evaluate_DeterministicOrdering_LiftDesc_ThenAppNameOrdinal()
{
var rows = new List<AppInstallSummary>();
rows.AddRange(Batch("1.0", 10, 300, Now.AddDays(-20)).Select(s => { s.AppName = "App B"; return s; }));
rows.AddRange(Batch("2.0", 10, 900, Now.AddDays(-5)).Select(s => { s.AppName = "App B"; return s; })); // lift 3.0
rows.AddRange(Batch("1.0", 10, 300, Now.AddDays(-20)).Select(s => { s.AppName = "App A"; return s; }));
rows.AddRange(Batch("2.0", 10, 1500, Now.AddDays(-5)).Select(s => { s.AppName = "App A"; return s; })); // lift 5.0
var findings = AppVersionRegressionRadar.Evaluate(rows);
Assert.Equal(2, findings.Count);
Assert.Equal("App A", findings[0].AppName); // higher lift first
Assert.Equal("App B", findings[1].AppName);
}
// ── re-arm ──────────────────────────────────────────────────────────────
private static AppVersionRegressionAlert Alert(
string currentVersion = "2.0", string previousVersion = "1.0")
=> new()
{
TenantId = TenantA,
AppName = App,
CurrentVersion = currentVersion,
PreviousVersion = previousVersion,
CurrentMedianSeconds = 900,
PreviousMedianSeconds = 300,
CurrentMeasuredCount = 10,
PreviousMeasuredCount = 10,
Lift = 3.0,
FirstNotifiedAt = Now.AddDays(-3),
LastEvaluatedAt = Now.AddDays(-1),
};
[Fact]
public void ShouldReArm_WhenCurrentVersionDrainedOutOfHorizon()
{
// Only the previous version still has measured installs — the alerted version is gone.
var rows = Batch("1.0", 10, 300, Now.AddDays(-20));
Assert.True(AppVersionRegressionRadar.ShouldReArm(rows, Alert()));
}
[Fact]
public void ShouldReArm_WhenCurrentVersionBelowSampleFloor()
{
var rows = Batch("1.0", 10, 300, Now.AddDays(-20));
rows.AddRange(Batch("2.0", 9, 900, Now.AddDays(-5)));
Assert.True(AppVersionRegressionRadar.ShouldReArm(rows, Alert()));
}
[Fact]
public void ShouldReArm_WhenMedianFellBackUnderReArmFactor()
{
// 400 < 1.5 × 300 — the regression recovered (e.g. CDN warmed up).
var rows = Batch("1.0", 10, 300, Now.AddDays(-20));
rows.AddRange(Batch("2.0", 10, 400, Now.AddDays(-5)));
Assert.True(AppVersionRegressionRadar.ShouldReArm(rows, Alert()));
}
[Fact]
public void ShouldNotReArm_WhileMedianStaysElevated()
{
// 500 ≥ 1.5 × 300 — still elevated (even though below the 2× fire gate).
var rows = Batch("1.0", 10, 300, Now.AddDays(-20));
rows.AddRange(Batch("2.0", 10, 500, Now.AddDays(-5)));
Assert.False(AppVersionRegressionRadar.ShouldReArm(rows, Alert()));
}
[Fact]
public void ShouldNotReArm_WhenPreviousVersionDrained_NoComparisonBasis()
{
// The predecessor left the horizon; the episode is kept until the current
// version itself drains (fires-stopped-style re-arm only).
var rows = Batch("2.0", 10, 900, Now.AddDays(-5));
Assert.False(AppVersionRegressionRadar.ShouldReArm(rows, Alert()));
}
// ── tracker keyspace (round-trip + key shape) ───────────────────────────
[Fact]
public void AppVersionRegressionRowKey_PrefixedCaseFoldedAndSanitized()
{
Assert.Equal("appversionregression|contoso app_2|1.0_beta_",
TableHardwareRejectionNotificationTracker.BuildAppVersionRegressionRowKey(
" Contoso App/2 ", " 1.0#Beta? "));
}
[Fact]
public void AppVersionRegressionRowKey_NoPrefixCollisionWithOtherKeyspaces()
{
var key = TableHardwareRejectionNotificationTracker.BuildAppVersionRegressionRowKey(App, "1.0");
Assert.StartsWith("appversionregression|", key, StringComparison.Ordinal);
// The prefix range [prefix, "appversionregression}") must not swallow the sibling
// keyspaces ("ruleregression|", "tpmpss|", "{mfr}|{model}").
Assert.True(string.CompareOrdinal("ruleregression|", "appversionregression}") > 0);
Assert.True(string.CompareOrdinal("tpmpss|", "appversionregression}") > 0);
}
[Fact]
public void AppVersionRegressionEntity_RoundTripsAllFields()
{
var alert = new AppVersionRegressionAlert
{
TenantId = TenantA,
AppName = App,
CurrentVersion = "2.4.0",
PreviousVersion = "2.3.9",
CurrentMedianSeconds = 1740,
PreviousMedianSeconds = 660,
CurrentMeasuredCount = 12,
PreviousMeasuredCount = 40,
Lift = 2.6,
FirstNotifiedAt = Now,
LastEvaluatedAt = Now.AddHours(2),
};
var entity = TableHardwareRejectionNotificationTracker.BuildAppVersionRegressionEntity(TenantA, alert);
Assert.Equal(TenantA.ToLowerInvariant(), entity.PartitionKey);
Assert.Equal("appversionregression|contoso vpn|2.4.0", entity.RowKey);
var mapped = TableHardwareRejectionNotificationTracker.MapToAppVersionRegressionAlert(entity);
Assert.Equal(TenantA, mapped.TenantId);
Assert.Equal(App, mapped.AppName);
Assert.Equal("2.4.0", mapped.CurrentVersion);
Assert.Equal("2.3.9", mapped.PreviousVersion);
Assert.Equal(1740, mapped.CurrentMedianSeconds);
Assert.Equal(660, mapped.PreviousMedianSeconds);
Assert.Equal(12, mapped.CurrentMeasuredCount);
Assert.Equal(40, mapped.PreviousMeasuredCount);
Assert.Equal(2.6, mapped.Lift);
Assert.Equal(Now, mapped.FirstNotifiedAt);
Assert.Equal(Now.AddHours(2), mapped.LastEvaluatedAt);
}
// ── wording contract ────────────────────────────────────────────────────
[Fact]
public void RegressionMessage_CarriesTheFullNumbers_MinutesOneDecimal()
{
var finding = new AppVersionDurationRegressionFinding
{
TenantId = TenantA,
AppName = App,
CurrentVersion = "2.4.0",
PreviousVersion = "2.3.9",
CurrentMedianSeconds = 1740,
PreviousMedianSeconds = 660,
CurrentMeasuredCount = 12,
PreviousMeasuredCount = 40,
Lift = 2.6,
};
Assert.Equal(
"Median install duration rose from 11 to 29 min after version 2.4.0 " +
"(12 measured installs vs 40 on version 2.3.9) — lift 2.6x over the last 35 days.",
MaintenanceService.BuildAppVersionRegressionMessage(finding));
}
// ── analytics wire shape (versionBreakdown duration fields) ─────────────
[Fact]
public async Task Analytics_VersionBreakdown_CarriesMeasuredDurationStats()
{
var sessionRepo = new Mock<ISessionRepository>();
sessionRepo.Setup(r => r.GetSessionAsync(It.IsAny<string>(), It.IsAny<string>()))
.ReturnsAsync((SessionSummary?)null);
var rows = new List<AppInstallSummary>
{
Install("1.0", 100), Install("1.0", 200), Install("1.0", 300), Install("1.0", 400),
Install("1.0", 0), // unmeasured — excluded from stats
Install("1.0", 50, terminalState: "Skipped"), // skip — excluded
Install("1.0", 50, status: "Failed"), // failed — excluded from durations
};
var root = JsonSerializer.SerializeToElement(await AppsAnalyticsHelper.BuildAnalyticsResponseAsync(
rows, sessionRepo.Object, App, days: 30));
var version = root.GetProperty("versionBreakdown")[0];
Assert.Equal("1.0", version.GetProperty("appVersion").GetString());
Assert.Equal(7, version.GetProperty("installs").GetInt32());
Assert.Equal(4, version.GetProperty("measuredInstalls").GetInt32());
// Nearest-rank over [100, 200, 300, 400]: p50 = 200 (lower middle), p95 = 400.
Assert.Equal(200, version.GetProperty("medianDurationSeconds").GetInt32());
Assert.Equal(400, version.GetProperty("p95DurationSeconds").GetInt32());
}
[Fact]
public async Task Analytics_VersionRegressions_SerializedCamelCase_AndPresentWhenEmptyData()
{
var sessionRepo = new Mock<ISessionRepository>();
sessionRepo.Setup(r => r.GetSessionAsync(It.IsAny<string>(), It.IsAny<string>()))
.ReturnsAsync((SessionSummary?)null);
var alert = Alert();
// The Functions host serializes camelCase on the wire; mirror that here so the
// asserted property names match what the web client actually parses.
var wireOptions = new JsonSerializerOptions(JsonSerializerDefaults.Web);
// Empty-data early return still surfaces the block (episodes can outlive window data).
var emptyRoot = JsonSerializer.SerializeToElement(await AppsAnalyticsHelper.BuildAnalyticsResponseAsync(
new List<AppInstallSummary>(), sessionRepo.Object, App, days: 30,
new List<AppVersionRegressionAlert> { alert }), wireOptions);
Assert.Equal(1, emptyRoot.GetProperty("versionRegressions").GetArrayLength());
var root = JsonSerializer.SerializeToElement(await AppsAnalyticsHelper.BuildAnalyticsResponseAsync(
new List<AppInstallSummary> { Install("1.0", 100) }, sessionRepo.Object, App, days: 30,
new List<AppVersionRegressionAlert> { alert }), wireOptions);
var regression = root.GetProperty("versionRegressions")[0];
Assert.Equal("2.0", regression.GetProperty("currentVersion").GetString());
Assert.Equal(900, regression.GetProperty("currentMedianSeconds").GetInt32());
}
}
@@ -17,6 +17,8 @@ public class TenantNotificationAudienceCatalogTests
[InlineData("sla_breach", NotificationAudience.Member)]
[InlineData("sla_consecutive_failures", NotificationAudience.Member)]
[InlineData("sla_resolved", NotificationAudience.Member)]
[InlineData("rule_frequency_regression", NotificationAudience.Admin)]
[InlineData("app_version_duration_regression", NotificationAudience.Admin)]
public void Resolve_KnownTypes_ReturnsExpectedAudience(string type, NotificationAudience expected)
{
Assert.Equal(expected, TenantNotificationAudienceCatalog.Resolve(type));
@@ -17,6 +17,9 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
/// cannot collide with a hardware key because no real manufacturer is named "tpmpss")
/// - F3 rule regression: "ruleregression|{ruleId-lower}" (trimmed; payload-carrying rows
/// that are refreshed while the episode is active and deleted on re-arm)
/// - app-version duration regression: "appversionregression|{app-lower}|{version-lower}"
/// (trimmed + table-key-sanitized; same payload-carrying episode semantics — raw
/// AppName/CurrentVersion live in columns, the mapper never parses the RowKey)
/// Race-safe via AddEntityAsync: Azure Table Storage returns 409 Conflict if the entity already exists.
/// </summary>
public class TableHardwareRejectionNotificationTracker : IHardwareRejectionNotificationTracker
@@ -251,6 +254,107 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
return $"tpmpss|{sn}";
}
public async Task<bool> TryRegisterAppVersionRegressionAsync(string tenantId, AppVersionRegressionAlert alert)
{
if (string.IsNullOrWhiteSpace(tenantId)
|| string.IsNullOrWhiteSpace(alert?.AppName)
|| string.IsNullOrWhiteSpace(alert?.CurrentVersion))
{
return false;
}
try
{
await _table.AddEntityAsync(BuildAppVersionRegressionEntity(tenantId, alert!));
_logger.LogInformation(
"AppVersionRegression tracker registered: tenant={TenantId} app={AppName} version={Version}",
tenantId, alert!.AppName, alert!.CurrentVersion);
return true;
}
catch (RequestFailedException ex) when (ex.Status == 409)
{
// Episode already active — one bell per (app, version) episode.
return false;
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"AppVersionRegression tracker failed: tenant={TenantId} app={AppName} version={Version}",
tenantId, alert!.AppName, alert!.CurrentVersion);
// Fail closed so we do not double-fire if the row was actually written.
return false;
}
}
public async Task RefreshAppVersionRegressionAsync(string tenantId, AppVersionRegressionAlert alert)
{
if (string.IsNullOrWhiteSpace(tenantId)
|| string.IsNullOrWhiteSpace(alert?.AppName)
|| string.IsNullOrWhiteSpace(alert?.CurrentVersion))
{
return;
}
try
{
// Replace-mode upsert with the FULL payload incl. the caller-carried
// FirstNotifiedAt — the retention sweep re-arms on the ORIGINAL age, so a
// refresh must never rejuvenate the row.
await _table.UpsertEntityAsync(BuildAppVersionRegressionEntity(tenantId, alert!), TableUpdateMode.Replace);
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"AppVersionRegression tracker refresh failed: tenant={TenantId} app={AppName} version={Version} (stale numbers remain)",
tenantId, alert!.AppName, alert!.CurrentVersion);
}
}
public async Task DeleteAppVersionRegressionAsync(string tenantId, string appName, string currentVersion)
{
if (string.IsNullOrWhiteSpace(tenantId)
|| string.IsNullOrWhiteSpace(appName)
|| string.IsNullOrWhiteSpace(currentVersion))
{
return;
}
try
{
await _table.DeleteEntityAsync(
tenantId.ToLowerInvariant(), BuildAppVersionRegressionRowKey(appName, currentVersion));
}
catch (RequestFailedException ex) when (ex.Status == 404)
{
// Already gone (retention sweep or concurrent pass) — idempotent.
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"AppVersionRegression tracker delete failed: tenant={TenantId} app={AppName} version={Version} (episode stays until retention)",
tenantId, appName, currentVersion);
}
}
public async Task<List<AppVersionRegressionAlert>> GetAppVersionRegressionsAsync(string tenantId)
{
var results = new List<AppVersionRegressionAlert>();
if (string.IsNullOrWhiteSpace(tenantId)) return results;
try
{
var partitionKey = tenantId.ToLowerInvariant();
// '}' (0x7D) sorts directly after '|' (0x7C) — prefix range over "appversionregression|…".
var filter = $"PartitionKey eq '{partitionKey}' and RowKey ge '{AppVersionRegressionRowKeyPrefix}' and RowKey lt 'appversionregression}}'";
await foreach (var entity in _table.QueryAsync<TableEntity>(filter: filter))
{
results.Add(MapToAppVersionRegressionAlert(entity));
}
}
catch (Exception ex)
{
_logger.LogWarning(ex, "AppVersionRegression tracker list failed for tenant {TenantId}", tenantId);
}
return results;
}
internal const string RuleRegressionRowKeyPrefix = "ruleregression|";
internal static string BuildRuleRegressionRowKey(string ruleId)
@@ -317,5 +421,69 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
LastEvaluatedAt = entity.GetDateTimeOffset("LastEvaluatedAt")?.UtcDateTime ?? DateTime.MinValue,
};
}
internal const string AppVersionRegressionRowKeyPrefix = "appversionregression|";
/// <summary>
/// App names and versions are free-form agent payloads — they can contain characters
/// Azure Table keys reject ('/', '\', '#', '?'). Sanitized for the KEY only (worst case
/// two apps share an episode row — tolerable); the raw values live in columns and the
/// mapper reads them, never the RowKey.
/// </summary>
internal static string BuildAppVersionRegressionRowKey(string appName, string version)
{
var app = SanitizeTableKey((appName ?? string.Empty).Trim().ToLowerInvariant());
var ver = SanitizeTableKey((version ?? string.Empty).Trim().ToLowerInvariant());
return $"{AppVersionRegressionRowKeyPrefix}{app}|{ver}";
}
/// <summary>Replace characters not allowed in Azure Table Storage keys (/, \, #, ?).</summary>
private static string SanitizeTableKey(string key)
{
return key
.Replace("/", "_")
.Replace("\\", "_")
.Replace("#", "_")
.Replace("?", "_");
}
// internal static: entity builder + mapper are pinned by round-trip unit tests
// (table-storage-serialization rule — every property must survive Store→Map).
internal static TableEntity BuildAppVersionRegressionEntity(string tenantId, AppVersionRegressionAlert alert)
{
return new TableEntity(
tenantId.ToLowerInvariant(), BuildAppVersionRegressionRowKey(alert.AppName, alert.CurrentVersion))
{
["TenantId"] = tenantId,
["AppName"] = alert.AppName,
["CurrentVersion"] = alert.CurrentVersion,
["PreviousVersion"] = alert.PreviousVersion ?? string.Empty,
["CurrentMedianSeconds"] = alert.CurrentMedianSeconds,
["PreviousMedianSeconds"] = alert.PreviousMedianSeconds,
["CurrentMeasuredCount"] = alert.CurrentMeasuredCount,
["PreviousMeasuredCount"] = alert.PreviousMeasuredCount,
["Lift"] = alert.Lift,
["FirstNotifiedAt"] = alert.FirstNotifiedAt,
["LastEvaluatedAt"] = alert.LastEvaluatedAt,
};
}
internal static AppVersionRegressionAlert MapToAppVersionRegressionAlert(TableEntity entity)
{
return new AppVersionRegressionAlert
{
TenantId = entity.GetString("TenantId") ?? entity.PartitionKey,
AppName = entity.GetString("AppName") ?? string.Empty,
CurrentVersion = entity.GetString("CurrentVersion") ?? string.Empty,
PreviousVersion = entity.GetString("PreviousVersion") ?? string.Empty,
CurrentMedianSeconds = entity.GetInt32("CurrentMedianSeconds") ?? 0,
PreviousMedianSeconds = entity.GetInt32("PreviousMedianSeconds") ?? 0,
CurrentMeasuredCount = entity.GetInt32("CurrentMeasuredCount") ?? 0,
PreviousMeasuredCount = entity.GetInt32("PreviousMeasuredCount") ?? 0,
Lift = entity.GetDouble("Lift") ?? 0,
FirstNotifiedAt = entity.GetDateTimeOffset("FirstNotifiedAt")?.UtcDateTime ?? DateTime.MinValue,
LastEvaluatedAt = entity.GetDateTimeOffset("LastEvaluatedAt")?.UtcDateTime ?? DateTime.MinValue,
};
}
}
}
@@ -229,8 +229,13 @@ namespace AutopilotMonitor.Functions.Functions.Apps
List<AppInstallSummary> allSummaries,
ISessionRepository sessionRepo,
string appName,
int days)
int days,
IReadOnlyList<AppVersionRegressionAlert>? versionRegressions = null)
{
// Active duration-regression episodes for THIS app (tracker rows; models serialize
// camelCase on the wire like the rule-stats regressions[] block).
var appVersionRegressions = versionRegressions ?? Array.Empty<AppVersionRegressionAlert>();
var now = DateTime.UtcNow;
var cutoff = now.AddDays(-days);
var midpoint = now.AddDays(-days / 2.0);
@@ -275,7 +280,10 @@ namespace AutopilotMonitor.Functions.Functions.Apps
installerPhaseBreakdown = Array.Empty<object>(),
topFailureCodes = Array.Empty<object>(),
detectionLiesCount = 0,
deviceModelBreakdown = Array.Empty<object>()
deviceModelBreakdown = Array.Empty<object>(),
// Lockstep with the full response: episodes can outlive the window's data
// (e.g. a shrunk days= selection), so the block is still surfaced here.
versionRegressions = appVersionRegressions
};
}
@@ -313,12 +321,23 @@ namespace AutopilotMonitor.Functions.Functions.Apps
var vFailed = g.Count(s => s.Status == "Failed");
// Same PR0 convention as the top-level rate: skips are not attempts.
var vSucceeded = g.Count(s => s.Status == "Succeeded" && !MetricsMath.IsSkipTerminalState(s));
// Same measured population as the top-level duration stats: succeeded,
// non-skip, and a plausible observed duration (0s = start unobserved,
// >6h = back-stamped batch — both excluded, never averaged in).
var vDurations = g
.Where(s => s.Status == "Succeeded" && !MetricsMath.IsSkipTerminalState(s))
.Where(MetricsMath.HasMeasuredDuration)
.Select(s => s.DurationSeconds)
.ToList();
return new
{
appVersion = g.Key,
installs = vTotal,
failed = vFailed,
failureRate = MetricsMath.TerminalFailureRatePct(vFailed, vSucceeded)
failureRate = MetricsMath.TerminalFailureRatePct(vFailed, vSucceeded),
measuredInstalls = vDurations.Count,
medianDurationSeconds = Percentile(vDurations, 0.50),
p95DurationSeconds = Percentile(vDurations, 0.95)
};
})
.OrderByDescending(v => v.installs)
@@ -421,7 +440,8 @@ namespace AutopilotMonitor.Functions.Functions.Apps
installerPhaseBreakdown,
topFailureCodes,
detectionLiesCount,
deviceModelBreakdown
deviceModelBreakdown,
versionRegressions = appVersionRegressions
};
}
@@ -586,7 +606,9 @@ namespace AutopilotMonitor.Functions.Functions.Apps
return date.AddDays(-diff);
}
private static int Percentile(List<int> values, double percentile)
// Internal so AppVersionRegressionRadar shares the same nearest-rank convention
// (a third percentile implementation would invite subtle median drift).
internal static int Percentile(List<int> values, double percentile)
{
if (values.Count == 0) return 0;
var sorted = values.OrderBy(v => v).ToList();
@@ -17,15 +17,18 @@ namespace AutopilotMonitor.Functions.Functions.Apps
private readonly ILogger<GetAppAnalyticsFunction> _logger;
private readonly IMetricsRepository _metricsRepo;
private readonly ISessionRepository _sessionRepo;
private readonly IHardwareRejectionNotificationTracker _notificationTracker;
public GetAppAnalyticsFunction(
ILogger<GetAppAnalyticsFunction> logger,
IMetricsRepository metricsRepo,
ISessionRepository sessionRepo)
ISessionRepository sessionRepo,
IHardwareRejectionNotificationTracker notificationTracker)
{
_logger = logger;
_metricsRepo = metricsRepo;
_sessionRepo = sessionRepo;
_notificationTracker = notificationTracker;
}
[Function("GetAppAnalytics")]
@@ -51,8 +54,12 @@ namespace AutopilotMonitor.Functions.Functions.Apps
days = parsedDays;
var summaries = await AppsAnalyticsHelper.LoadSummariesAsync(_metricsRepo, tenantId, days);
// Active duration-regression episodes for this app (fail-soft: empty on error).
var versionRegressions = (await _notificationTracker.GetAppVersionRegressionsAsync(tenantId))
.Where(a => string.Equals(a.AppName, decodedAppName, StringComparison.OrdinalIgnoreCase))
.ToList();
var body = await AppsAnalyticsHelper.BuildAnalyticsResponseAsync(
summaries, _sessionRepo, decodedAppName, days);
summaries, _sessionRepo, decodedAppName, days, versionRegressions);
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(body);
@@ -1,6 +1,7 @@
using System.Net;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.Extensions.Logging;
@@ -17,15 +18,18 @@ namespace AutopilotMonitor.Functions.Functions.Apps
private readonly ILogger<GetGlobalAppAnalyticsFunction> _logger;
private readonly IMetricsRepository _metricsRepo;
private readonly ISessionRepository _sessionRepo;
private readonly IHardwareRejectionNotificationTracker _notificationTracker;
public GetGlobalAppAnalyticsFunction(
ILogger<GetGlobalAppAnalyticsFunction> logger,
IMetricsRepository metricsRepo,
ISessionRepository sessionRepo)
ISessionRepository sessionRepo,
IHardwareRejectionNotificationTracker notificationTracker)
{
_logger = logger;
_metricsRepo = metricsRepo;
_sessionRepo = sessionRepo;
_notificationTracker = notificationTracker;
}
[Function("GetGlobalAppAnalytics")]
@@ -61,8 +65,15 @@ namespace AutopilotMonitor.Functions.Functions.Apps
decodedAppName, userEmail, scopedTenantId ?? "<all>", days);
var summaries = await AppsAnalyticsHelper.LoadSummariesAsync(_metricsRepo, scopedTenantId, days);
// Episodes are per-tenant tracker rows; without a tenant scope the aggregated
// cross-tenant view carries none (mirrors the rule-stats global route).
var versionRegressions = string.IsNullOrEmpty(scopedTenantId)
? new List<AppVersionRegressionAlert>()
: (await _notificationTracker.GetAppVersionRegressionsAsync(scopedTenantId!))
.Where(a => string.Equals(a.AppName, decodedAppName, StringComparison.OrdinalIgnoreCase))
.ToList();
var body = await AppsAnalyticsHelper.BuildAnalyticsResponseAsync(
summaries, _sessionRepo, decodedAppName, days);
summaries, _sessionRepo, decodedAppName, days, versionRegressions);
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(body);
@@ -0,0 +1,208 @@
using AutopilotMonitor.Functions.Functions.Apps;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Functions.Helpers;
/// <summary>One fired duration regression: an app whose newest version installs much slower than its predecessor (see <see cref="AppVersionRegressionRadar"/>).</summary>
public sealed class AppVersionDurationRegressionFinding
{
public string TenantId { get; init; } = string.Empty;
public string AppName { get; init; } = string.Empty;
public string CurrentVersion { get; init; } = string.Empty;
public string PreviousVersion { get; init; } = string.Empty;
public int CurrentMedianSeconds { get; init; }
public int PreviousMedianSeconds { get; init; }
public int CurrentMeasuredCount { get; init; }
public int PreviousMeasuredCount { get; init; }
/// <summary>Current median ÷ previous median, one decimal. Never null — measured durations are ≥1s, so the previous median cannot be 0.</summary>
public double Lift { get; init; }
}
/// <summary>
/// Deterministic, I/O-free core of the app-version duration regression radar. Detects
/// "version X made this app's installs much slower" from the per-(session, app) install
/// summaries: per (tenant, app), the newest version's MEDIAN measured install duration is
/// compared against the previous version's median. Medians (nearest-rank, shared with the
/// apps dashboard via <see cref="AppsAnalyticsHelper.Percentile"/>) rather than means —
/// a single back-stamped straggler must not fire a fleet alert. A regression fires only
/// when ALL hold:
/// <list type="number">
/// <item>both versions have ≥ 10 measured installs inside the horizon;</item>
/// <item>current median ≥ 2× the previous version's median;</item>
/// <item>absolute increase ≥ 300 s — a 20s→45s app is noise, not an incident.</item>
/// </list>
/// Measured population: Succeeded, non-skip terminal state, plausible observed duration
/// (<see cref="MetricsMath.HasMeasuredDuration"/>), no AppId collision, non-empty
/// AppVersion. Version ORDER comes from first-seen install time (min StartedAt) — version
/// strings are never sorted lexically ("9.1" vs "2024.10" would lie). "Previous" is the
/// version with the latest first-seen strictly before the current version's first-seen;
/// with parallel ring rollouts that can be a concurrently-deployed version — accepted for
/// v1, the episode key (app, current version) still caps it at one bell per version.
/// </summary>
public static class AppVersionRegressionRadar
{
/// <summary>Days of install summaries the radar loads and reasons over (mirrors the rule radar's 7+28 horizon).</summary>
public const int HorizonDays = 35;
public const int MinMeasuredInstalls = 10;
public const double MinMedianLift = 2.0;
public const int MinAbsoluteIncreaseSeconds = 300;
/// <summary>An active alert re-arms once the current median falls back under 1.5× the previous version's median.</summary>
public const double ReArmLiftFactor = 1.5;
/// <summary>Per-version duration stats inside the horizon (see <see cref="ComputeVersionStats"/>).</summary>
public sealed class VersionStats
{
public string Version { get; init; } = string.Empty;
public DateTime FirstSeen { get; init; }
public DateTime LastSeen { get; init; }
public int MeasuredCount { get; init; }
public int MedianSeconds { get; init; }
}
/// <summary>
/// Evaluates one tenant's install summaries (loaded for the last <see cref="HorizonDays"/>
/// days). Returns at most one finding per app — the newest version vs. its predecessor;
/// older version pairs are yesterday's news, not an alert. Deterministic ordering:
/// lift descending, then app name ordinal.
/// </summary>
public static List<AppVersionDurationRegressionFinding> Evaluate(IReadOnlyList<AppInstallSummary> tenantSummaries)
{
var findings = new List<AppVersionDurationRegressionFinding>();
foreach (var app in MeasuredAppGroups(tenantSummaries))
{
var stats = ComputeVersionStats(app.Value);
var (current, previous) = SelectComparisonPair(stats);
if (current == null || previous == null) continue;
if (current.MeasuredCount < MinMeasuredInstalls) continue;
if (previous.MeasuredCount < MinMeasuredInstalls) continue;
if (current.MedianSeconds < MinMedianLift * previous.MedianSeconds) continue;
if (current.MedianSeconds - previous.MedianSeconds < MinAbsoluteIncreaseSeconds) continue;
findings.Add(new AppVersionDurationRegressionFinding
{
TenantId = app.Value[0].TenantId,
AppName = app.Key,
CurrentVersion = current.Version,
PreviousVersion = previous.Version,
CurrentMedianSeconds = current.MedianSeconds,
PreviousMedianSeconds = previous.MedianSeconds,
CurrentMeasuredCount = current.MeasuredCount,
PreviousMeasuredCount = previous.MeasuredCount,
Lift = Math.Round((double)current.MedianSeconds / previous.MedianSeconds, 1),
});
}
return findings
.OrderByDescending(f => f.Lift)
.ThenBy(f => f.AppName, StringComparer.Ordinal)
.ToList();
}
/// <summary>
/// True when an ACTIVE alert may re-arm: the alerted version has drained out of the
/// horizon (fewer than <see cref="MinMeasuredInstalls"/> measured installs left — it was
/// superseded or pulled), or its recomputed median fell back under
/// <see cref="ReArmLiftFactor"/>× the previous version's recomputed median. When the
/// previous version itself has drained there is no comparison basis anymore — the
/// episode is kept until the current version drains (fires-stopped-style re-arm only).
/// </summary>
public static bool ShouldReArm(IReadOnlyList<AppInstallSummary> tenantSummaries, AppVersionRegressionAlert alert)
{
var stats = ComputeVersionStatsForApp(tenantSummaries, alert.AppName);
var current = stats.FirstOrDefault(s => string.Equals(s.Version, alert.CurrentVersion, StringComparison.Ordinal));
if (current == null || current.MeasuredCount < MinMeasuredInstalls) return true;
var previous = stats.FirstOrDefault(s => string.Equals(s.Version, alert.PreviousVersion, StringComparison.Ordinal));
if (previous == null || previous.MeasuredCount < MinMeasuredInstalls) return false;
return current.MedianSeconds < ReArmLiftFactor * previous.MedianSeconds;
}
/// <summary>Per-version stats for one app (measured rows only), or empty when the app has none.</summary>
public static List<VersionStats> ComputeVersionStatsForApp(IReadOnlyList<AppInstallSummary> tenantSummaries, string appName)
{
foreach (var app in MeasuredAppGroups(tenantSummaries))
{
if (string.Equals(app.Key, appName, StringComparison.OrdinalIgnoreCase))
return ComputeVersionStats(app.Value);
}
return new List<VersionStats>();
}
/// <summary>
/// Groups the MEASURED rows by app name: succeeded, non-skip, plausible duration,
/// no AppId collision (a second app's outcomes must not fire this app's alert),
/// non-empty version. Rate/failure signals are deliberately out of scope — this
/// radar alerts on duration only.
/// </summary>
internal static Dictionary<string, List<AppInstallSummary>> MeasuredAppGroups(IReadOnlyList<AppInstallSummary> summaries)
{
var groups = new Dictionary<string, List<AppInstallSummary>>(StringComparer.OrdinalIgnoreCase);
foreach (var summary in summaries)
{
if (string.IsNullOrEmpty(summary.AppName)) continue;
if (string.IsNullOrEmpty(summary.AppVersion)) continue;
if (summary.AppIdCollision) continue;
if (summary.Status != "Succeeded") continue;
if (MetricsMath.IsSkipTerminalState(summary)) continue;
if (!MetricsMath.HasMeasuredDuration(summary)) continue;
if (!groups.TryGetValue(summary.AppName, out var list))
{
list = new List<AppInstallSummary>();
groups[summary.AppName] = list;
}
list.Add(summary);
}
return groups;
}
/// <summary>Per-version first/last-seen and nearest-rank median over one app's measured rows.</summary>
internal static List<VersionStats> ComputeVersionStats(IReadOnlyList<AppInstallSummary> measuredAppRows)
{
return measuredAppRows
.GroupBy(s => s.AppVersion, StringComparer.Ordinal)
.Select(g =>
{
var durations = g.Select(s => s.DurationSeconds).ToList();
return new VersionStats
{
Version = g.Key,
FirstSeen = g.Min(s => s.StartedAt),
LastSeen = g.Max(s => s.StartedAt),
MeasuredCount = durations.Count,
MedianSeconds = AppsAnalyticsHelper.Percentile(durations, 0.50),
};
})
.ToList();
}
/// <summary>
/// Current = the version of the most recent measured install (ties broken by later
/// first-seen, then version ordinal — deterministic); previous = the version with the
/// latest first-seen strictly before the current version's first-seen (same tie-break).
/// Either is null when the app has fewer than two versions in the horizon.
/// </summary>
internal static (VersionStats? Current, VersionStats? Previous) SelectComparisonPair(IReadOnlyList<VersionStats> stats)
{
if (stats.Count < 2) return (null, null);
var current = stats
.OrderByDescending(s => s.LastSeen)
.ThenByDescending(s => s.FirstSeen)
.ThenByDescending(s => s.Version, StringComparer.Ordinal)
.First();
var previous = stats
.Where(s => s.FirstSeen < current.FirstSeen)
.OrderByDescending(s => s.FirstSeen)
.ThenByDescending(s => s.LastSeen)
.ThenByDescending(s => s.Version, StringComparer.Ordinal)
.FirstOrDefault();
return (current, previous);
}
}
@@ -0,0 +1,212 @@
using System;
using System.Diagnostics;
using System.Linq;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Services
{
/// <summary>
/// Partial: app-version duration regression radar (sibling of the F3 rule radar).
/// </summary>
public partial class MaintenanceService
{
/// <summary>App setting kill switch: set to "true" to skip the radar entirely. Fail-open — the radar only notifies, it never mutates data.</summary>
internal const string AppVersionRadarKillSwitchSetting = "AppVersionRegressionRadarDisabled";
/// <summary>
/// Detects per-tenant apps whose newest version's median install duration regressed
/// (≥2× and ≥5 min over the previous version — <see cref="AppVersionRegressionRadar"/>)
/// and reconciles the alert episodes in the notification tracker:
/// new finding → tracker row + tenant bell + AppVersionDurationRegression ops event,
/// exactly once per (app, version) episode; still-regressed → numbers refreshed
/// (FirstNotifiedAt untouched); no longer regressed → re-arm check (version drained,
/// or median back under 1.5×) deletes the row so a future regression rings again.
/// The tracker's 30d retention sweep re-arms long-burning episodes by design.
/// <para>
/// One cross-tenant install-summary read over the trailing
/// <see cref="AppVersionRegressionRadar.HorizonDays"/> days (the same query the global
/// apps dashboard runs), then grouped by tenant — mirrors the rule radar's enumeration.
/// Idempotent per pass via the tracker dedup. Fail-soft per tenant and overall.
/// </para>
/// </summary>
private async Task RunAppVersionRegressionRadarAsync()
{
try
{
if (string.Equals(_configuration[AppVersionRadarKillSwitchSetting], "true", StringComparison.OrdinalIgnoreCase))
{
_logger.LogInformation("App-version regression radar skipped ({Setting}=true)", AppVersionRadarKillSwitchSetting);
return;
}
var sw = Stopwatch.StartNew();
var sinceUtc = DateTime.UtcNow.AddDays(-AppVersionRegressionRadar.HorizonDays);
var summaries = await _metricsRepo.GetAppsDashboardSummariesAsync(sinceUtc, tenantId: null);
var byTenant = summaries
.Where(s => !string.IsNullOrEmpty(s.TenantId))
.GroupBy(s => s.TenantId, StringComparer.OrdinalIgnoreCase);
int fired = 0, refreshed = 0, rearmed = 0, tenants = 0;
foreach (var tenantGroup in byTenant)
{
tenants++;
try
{
var (f, r, a) = await EvaluateTenantAppVersionRegressionsAsync(tenantGroup.Key, tenantGroup.ToList());
fired += f;
refreshed += r;
rearmed += a;
}
catch (Exception tenantEx)
{
_logger.LogWarning(tenantEx, "App-version regression radar failed for tenant {TenantId} (non-fatal)", tenantGroup.Key);
}
}
sw.Stop();
_logger.LogInformation(
"App-version regression radar: {Tenants} tenants over {Rows} install summaries — {Fired} fired, {Refreshed} refreshed, {Rearmed} re-armed in {Ms}ms",
tenants, summaries.Count, fired, refreshed, rearmed, sw.ElapsedMilliseconds);
}
catch (Exception ex)
{
_logger.LogWarning(ex, "App-version regression radar failed (non-fatal)");
}
}
private async Task<(int Fired, int Refreshed, int Rearmed)> EvaluateTenantAppVersionRegressionsAsync(
string tenantId, System.Collections.Generic.List<AppInstallSummary> tenantSummaries)
{
var findings = AppVersionRegressionRadar.Evaluate(tenantSummaries);
var active = await _hardwareRejectionTracker.GetAppVersionRegressionsAsync(tenantId);
var activeByKey = active.ToDictionary(
a => AppVersionEpisodeKey(a.AppName, a.CurrentVersion), StringComparer.OrdinalIgnoreCase);
int fired = 0, refreshed = 0, rearmed = 0;
var firedKeys = new System.Collections.Generic.HashSet<string>(StringComparer.OrdinalIgnoreCase);
foreach (var finding in findings)
{
var key = AppVersionEpisodeKey(finding.AppName, finding.CurrentVersion);
firedKeys.Add(key);
if (activeByKey.TryGetValue(key, out var existing))
{
// Episode still burning: refresh numbers for the versionRegressions[] block;
// FirstNotifiedAt never moves (retention re-arm counts from the bell).
await _hardwareRejectionTracker.RefreshAppVersionRegressionAsync(
tenantId, BuildAppVersionAlert(finding, existing.FirstNotifiedAt));
refreshed++;
continue;
}
var alert = BuildAppVersionAlert(finding, DateTime.UtcNow);
if (!await _hardwareRejectionTracker.TryRegisterAppVersionRegressionAsync(tenantId, alert))
continue; // a concurrent pass won the episode — its bell suffices
fired++;
await _tenantNotificationService.CreateNotificationAsync(
tenantId,
type: "app_version_duration_regression",
title: $"Install duration regressed: {finding.AppName}",
message: BuildAppVersionRegressionMessage(finding),
// Canonical appDetailUrl shape (lib/routes.ts) with the radar's horizon window.
href: $"/apps/detail?name={Uri.EscapeDataString(finding.AppName)}&days={AppVersionRegressionRadar.HorizonDays}");
await _opsEventService.RecordAppVersionDurationRegressionAsync(
tenantId, finding.AppName, finding.CurrentVersion, finding.PreviousVersion,
finding.CurrentMedianSeconds, finding.PreviousMedianSeconds,
finding.CurrentMeasuredCount, finding.PreviousMeasuredCount, finding.Lift);
}
foreach (var alert in active)
{
if (firedKeys.Contains(AppVersionEpisodeKey(alert.AppName, alert.CurrentVersion)))
continue;
if (AppVersionRegressionRadar.ShouldReArm(tenantSummaries, alert))
{
await _hardwareRejectionTracker.DeleteAppVersionRegressionAsync(tenantId, alert.AppName, alert.CurrentVersion);
rearmed++;
}
else
{
// Elevated but below the fire gates (between 1.5× and 2×, or the absolute
// delta shrank): keep the episode, refresh the numbers so the
// versionRegressions[] block stays honest.
var refreshedAlert = TryBuildOngoingAppVersionAlert(tenantSummaries, alert);
if (refreshedAlert != null)
{
await _hardwareRejectionTracker.RefreshAppVersionRegressionAsync(tenantId, refreshedAlert);
refreshed++;
}
}
}
return (fired, refreshed, rearmed);
}
private static string AppVersionEpisodeKey(string appName, string version)
=> $"{appName}\n{version}";
private static AppVersionRegressionAlert BuildAppVersionAlert(
AppVersionDurationRegressionFinding finding, DateTime firstNotifiedAt)
=> new()
{
TenantId = finding.TenantId,
AppName = finding.AppName,
CurrentVersion = finding.CurrentVersion,
PreviousVersion = finding.PreviousVersion,
CurrentMedianSeconds = finding.CurrentMedianSeconds,
PreviousMedianSeconds = finding.PreviousMedianSeconds,
CurrentMeasuredCount = finding.CurrentMeasuredCount,
PreviousMeasuredCount = finding.PreviousMeasuredCount,
Lift = finding.Lift,
FirstNotifiedAt = firstNotifiedAt,
LastEvaluatedAt = DateTime.UtcNow,
};
/// <summary>
/// Refresh payload for a still-open episode below the fire gates: recomputed medians
/// against the SAME version pair, everything identifying carried over. Null when either
/// version has left the horizon — the stale numbers then remain until re-arm/retention.
/// </summary>
private static AppVersionRegressionAlert? TryBuildOngoingAppVersionAlert(
System.Collections.Generic.IReadOnlyList<AppInstallSummary> tenantSummaries, AppVersionRegressionAlert existing)
{
var stats = AppVersionRegressionRadar.ComputeVersionStatsForApp(tenantSummaries, existing.AppName);
var current = stats.FirstOrDefault(s => string.Equals(s.Version, existing.CurrentVersion, StringComparison.Ordinal));
var previous = stats.FirstOrDefault(s => string.Equals(s.Version, existing.PreviousVersion, StringComparison.Ordinal));
if (current == null || previous == null || previous.MedianSeconds <= 0)
return null;
return new AppVersionRegressionAlert
{
TenantId = existing.TenantId,
AppName = existing.AppName,
CurrentVersion = existing.CurrentVersion,
PreviousVersion = existing.PreviousVersion,
CurrentMedianSeconds = current.MedianSeconds,
PreviousMedianSeconds = previous.MedianSeconds,
CurrentMeasuredCount = current.MeasuredCount,
PreviousMeasuredCount = previous.MeasuredCount,
Lift = Math.Round((double)current.MedianSeconds / previous.MedianSeconds, 1),
FirstNotifiedAt = existing.FirstNotifiedAt,
LastEvaluatedAt = DateTime.UtcNow,
};
}
/// <summary>Bell message with the full numbers (the admin can verify without a portal round-trip). Minutes, one decimal.</summary>
internal static string BuildAppVersionRegressionMessage(AppVersionDurationRegressionFinding finding)
{
var fromMin = Math.Round(finding.PreviousMedianSeconds / 60.0, 1);
var toMin = Math.Round(finding.CurrentMedianSeconds / 60.0, 1);
return $"Median install duration rose from {fromMin} to {toMin} min after version {finding.CurrentVersion} " +
$"({finding.CurrentMeasuredCount} measured installs vs {finding.PreviousMeasuredCount} on version {finding.PreviousVersion}) " +
$"— lift {finding.Lift}x over the last {AppVersionRegressionRadar.HorizonDays} days.";
}
}
}
@@ -148,6 +148,9 @@ namespace AutopilotMonitor.Functions.Services
// aggregation so the window rows are fresh. Anchored on YESTERDAY — whole
// days only, a partial today would understate the window rate. Fail-soft.
await RunRuleRegressionRadarAsync(DateTime.UtcNow.Date.AddDays(-1));
// App-version duration regression radar: same episode/tracker pattern over the
// install summaries (trailing 35d horizon loaded internally). Fail-soft.
await RunAppVersionRegressionRadarAsync();
// F1 PR2: rolling 30d breakdown backfill + daily attribution aggregates. Owns
// its own window (NOT the snapshot-gated catch-up above) so late-terminating
// sessions still reach their StartedAt-date's aggregate. Fail-soft internally.
@@ -222,6 +225,7 @@ namespace AutopilotMonitor.Functions.Services
// Timer-path parity: radar re-runs are idempotent (tracker dedup), so a manual
// aggregation also re-evaluates regressions anchored on the aggregated date.
await RunRuleRegressionRadarAsync(dateToAggregate);
await RunAppVersionRegressionRadarAsync();
// Timer-path parity: manual maintenance also refreshes the attribution
// breakdowns + daily aggregates and the device-history/FTR rollups
@@ -542,6 +542,35 @@ namespace AutopilotMonitor.Functions.Services
lift,
});
/// <summary>
/// App-version duration regression radar: an app's newest version installs with a median
/// duration ≥2× (and ≥5 min absolute) over the previous version's median. Fired ONCE per
/// (app, version) episode (the notification-tracker row is the dedup); the message
/// carries the full numbers so the event is verifiable without a portal round-trip.
/// Dual-registered in the web OPS_EVENT_TYPES catalog
/// (memory: feedback_ops_event_types_dual_register).
/// </summary>
public Task RecordAppVersionDurationRegressionAsync(
string tenantId, string appName, string currentVersion, string previousVersion,
int currentMedianSeconds, int previousMedianSeconds,
int currentMeasuredCount, int previousMeasuredCount, double lift)
=> WriteAsync(OpsEventCategory.Tenant, "AppVersionDurationRegression", OpsEventSeverity.Warning,
$"App '{appName}' median install duration rose from {Math.Round(previousMedianSeconds / 60.0, 1)} to " +
$"{Math.Round(currentMedianSeconds / 60.0, 1)} min after version {currentVersion} " +
$"({currentMeasuredCount} measured installs vs {previousMeasuredCount} on {previousVersion}) — lift {lift}x",
tenantId, "System.Maintenance",
new
{
appName,
currentVersion,
previousVersion,
currentMedianSeconds,
previousMedianSeconds,
currentMeasuredCount,
previousMeasuredCount,
lift,
});
// ── Agent ──────────────────────────────────────────────────────────────
public Task RecordSessionTimeoutsAsync(string tenantId, int sessionCount, int timeoutHours)
@@ -33,6 +33,8 @@ public static class TenantNotificationAudienceCatalog
// F3 regression radar: rule-tuning is an admin concern (the alert deep-links into the
// analyze-rules page, which members cannot act on).
["rule_frequency_regression"] = NotificationAudience.Admin,
// App-version duration regression radar: packaging/rollout is an admin concern.
["app_version_duration_regression"] = NotificationAudience.Admin,
};
public static NotificationAudience Resolve(string? type)
@@ -8,9 +8,10 @@ namespace AutopilotMonitor.Shared.DataAccess
/// <summary>
/// Notification dedup store (one table, RowKey-prefixed key spaces). Subjects:
/// hardware rejections per (tenant, manufacturer, model), TPM PSS incompatibilities per
/// (tenant, serial number) — both lifetime insert-once — and F3 rule-frequency
/// regressions per (tenant, ruleId), whose rows additionally carry the alert payload,
/// are refreshed while the episode stays active, and are deleted when the rate re-arms.
/// (tenant, serial number) — both lifetime insert-once — F3 rule-frequency
/// regressions per (tenant, ruleId), and app-version duration regressions per
/// (tenant, app, version). The regression key spaces additionally carry the alert
/// payload, are refreshed while the episode stays active, and are deleted on re-arm.
/// </summary>
public interface IHardwareRejectionNotificationTracker
{
@@ -54,6 +55,30 @@ namespace AutopilotMonitor.Shared.DataAccess
/// <summary>Active regression episodes of one tenant (RowKey-prefix scan; empty on failure — fail-soft reads).</summary>
Task<List<RuleRegressionAlert>> GetRuleRegressionsAsync(string tenantId);
// --- App-version duration regressions (RowKey "appversionregression|{app}|{version}") ---
/// <summary>
/// Atomically opens a duration-regression episode for (tenantId, alert.AppName,
/// alert.CurrentVersion). Returns true when this pass owns the episode (caller fires
/// bell + ops event exactly once), false when an episode is already active or on
/// failure (fail-closed — never double-fire).
/// </summary>
Task<bool> TryRegisterAppVersionRegressionAsync(string tenantId, AppVersionRegressionAlert alert);
/// <summary>
/// Refreshes an active episode's numbers (medians, counts, LastEvaluatedAt) so the
/// versionRegressions[] block stays current. FirstNotifiedAt is carried from the alert
/// unchanged — the retention sweep re-arms on the ORIGINAL notification age.
/// Fail-soft; never fires notifications.
/// </summary>
Task RefreshAppVersionRegressionAsync(string tenantId, AppVersionRegressionAlert alert);
/// <summary>Closes an episode (median re-armed or the version drained out of the horizon). 404-tolerant.</summary>
Task DeleteAppVersionRegressionAsync(string tenantId, string appName, string currentVersion);
/// <summary>Active app-version regression episodes of one tenant (RowKey-prefix scan; empty on failure — fail-soft reads).</summary>
Task<List<AppVersionRegressionAlert>> GetAppVersionRegressionsAsync(string tenantId);
/// <summary>
/// Retention cleanup: deletes tracker rows whose FirstNotifiedAt is older than
/// <paramref name="cutoffUtc"/>. Every key space lives in one table, so pruning re-arms
@@ -0,0 +1,43 @@
using System;
namespace AutopilotMonitor.Shared.Models
{
/// <summary>
/// One ACTIVE app-version duration regression: an app whose newest version's median
/// install duration rose ≥2× (and ≥5 minutes absolute) over the previous version's
/// median, both sides with enough measured installs. Persisted as the
/// <c>appversionregression|{app}|{version}</c> keyspace of the notification tracker
/// table — the row IS the dedup (one bell per episode), and the
/// <c>versionRegressions[]</c> payload on the app-analytics response. Deleted when the
/// episode re-arms (median falls back under 1.5× or the version drains out of the
/// horizon) or by the tracker's 30-day retention sweep. Numbers refresh on every radar
/// pass while the episode stays active; <see cref="FirstNotifiedAt"/> never moves.
/// </summary>
public class AppVersionRegressionAlert
{
public string TenantId { get; set; } = string.Empty;
public string AppName { get; set; } = string.Empty;
/// <summary>The regressed (newer) version — episode key together with the app.</summary>
public string CurrentVersion { get; set; } = string.Empty;
/// <summary>The comparison version: latest first-seen strictly before the current version's first-seen.</summary>
public string PreviousVersion { get; set; } = string.Empty;
public int CurrentMedianSeconds { get; set; }
public int PreviousMedianSeconds { get; set; }
/// <summary>Measured installs backing each median (gate: both ≥10).</summary>
public int CurrentMeasuredCount { get; set; }
public int PreviousMeasuredCount { get; set; }
/// <summary>Current median ÷ previous median (gate: ≥2.0; previous median is never 0 — measured durations are ≥1s).</summary>
public double Lift { get; set; }
/// <summary>When the episode first fired (bell + ops event moment). Never moves on refresh; drives the 30d retention re-arm.</summary>
public DateTime FirstNotifiedAt { get; set; }
/// <summary>Last radar pass that re-confirmed/refreshed this episode.</summary>
public DateTime LastEvaluatedAt { get; set; }
}
}
@@ -108,6 +108,11 @@ const OPS_EVENT_TYPES: Record<string, string[]> = {
// (Wilson-separated). Warning-tier, once per episode (tracker-deduped). Dual-register
// per memory feedback_ops_event_types_dual_register.
"RuleFrequencyRegression",
// App-version duration regression radar: an app's newest version installs with a median
// duration ≥2× (and ≥5 min absolute) over the previous version. Warning-tier, once per
// (app, version) episode (tracker-deduped). Dual-register per memory
// feedback_ops_event_types_dual_register.
"AppVersionDurationRegression",
// Auto-activation audit: the tenant-auto-approve worker activated a new signup
// (AutoApproveNewTenants enabled). Info-tier. Dual-register per memory
// feedback_ops_event_types_dual_register.
@@ -51,6 +51,25 @@ interface VersionRow {
installs: number;
failed: number;
failureRate: number;
/** Succeeded installs with a plausible observed duration — the basis of the median/p95 below. */
measuredInstalls: number;
medianDurationSeconds: number;
p95DurationSeconds: number;
}
/** Active duration-regression episode from the radar (tracker row, camelCase on the wire). */
interface AppVersionRegressionRow {
tenantId: string;
appName: string;
currentVersion: string;
previousVersion: string;
currentMedianSeconds: number;
previousMedianSeconds: number;
currentMeasuredCount: number;
previousMeasuredCount: number;
lift: number;
firstNotifiedAt: string;
lastEvaluatedAt: string;
}
interface PhaseRow {
@@ -102,6 +121,7 @@ interface AnalyticsResponse {
topFailureCodes: FailureCodeRow[];
detectionLiesCount: number;
deviceModelBreakdown: DeviceModelRow[];
versionRegressions: AppVersionRegressionRow[];
}
interface SessionRow {
@@ -527,6 +547,23 @@ function AppDetailContent() {
</Card>
</div>
{/* Duration regression banner: active radar episodes for this app (bell-deduped
server-side; this is the durable badge while the episode burns). */}
{(analytics.versionRegressions ?? []).length > 0 && (
<div className="rounded-lg border border-amber-300 bg-amber-50 px-4 py-3 space-y-1">
{analytics.versionRegressions.map((reg) => (
<p key={`${reg.appName}|${reg.currentVersion}`} className="text-sm text-amber-800">
<span className="font-semibold">↑ Duration regression:</span>{" "}
median install duration rose from {(reg.previousMedianSeconds / 60).toFixed(1)} to{" "}
{(reg.currentMedianSeconds / 60).toFixed(1)} min after version{" "}
<span className="font-mono">{reg.currentVersion}</span>{" "}
({reg.currentMeasuredCount} measured installs vs {reg.previousMeasuredCount} on{" "}
<span className="font-mono">{reg.previousVersion}</span> — lift {reg.lift}x)
</p>
))}
</div>
)}
{/* Charts row 2: version breakdown + phase breakdown.
Each card is hidden when empty so the row collapses gracefully:
- Version: will appear once new agents emit AppVersion data
@@ -547,6 +584,17 @@ function AppDetailContent() {
/>
</Card>
)}
{analytics.versionBreakdown.some((v) => v.measuredInstalls > 0) && (
<Card title="Median Install Duration by Version">
<AppBarChart
data={analytics.versionBreakdown.filter((v) => v.measuredInstalls > 0) as unknown as Array<Record<string, unknown>>}
categoryKey="appVersion"
valueKey="medianDurationSeconds"
valueFormatter={(v) => formatDuration(Number(v))}
barColor={chartColors.primary}
/>
</Card>
)}
{analytics.installerPhaseBreakdown.length > 0 && (
<Card title="Installer Phase (failures only)">
<AppBarChart