mirror of
https://github.com/okieselbach/AutopilotMonitor.git
synced 2026-10-24 20:53:06 +00:00
insights: app-version duration regression radar + per-version duration stats
versionBreakdown on the app analytics responses now carries measuredInstalls,
medianDurationSeconds and p95DurationSeconds per version (measured population:
succeeded, non-skip, plausible duration, no AppId collision). A new pure
AppVersionRegressionRadar detects "median install duration rose from X to Y min
after version Z": versions ordered by first-seen install time (never string
sort), fires at median lift >=2x AND >=300s absolute with >=10 measured
installs on both sides.
Episodes follow the rule-radar pattern in a fourth tracker keyspace
appversionregression|{app}|{version} (409-dedup, refresh keeps FirstNotifiedAt,
re-arm on drain / median back under 1.5x, 30d retention sweep re-arms): one
tenant bell app_version_duration_regression (Admin audience, appDetailUrl href)
plus one AppVersionDurationRegression ops event per episode, dual-registered in
the web OPS_EVENT_TYPES catalog. Runs in the maintenance timer after the rule
radar over one cross-tenant 35d install-summary read; kill switch
AppVersionRegressionRadarDisabled (fail-open). Apps detail page shows an amber
regression banner and a median-duration-by-version chart; OKF concept doc added.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,98 @@
|
||||
---
|
||||
type: Concept
|
||||
title: App Version Duration Regression Radar
|
||||
description: How the radar detects app versions whose median install duration regressed vs. the previous version (median lift ≥2x plus an absolute floor over first-seen-ordered versions), how alert episodes live in the appversionregression tracker keyspace, and which surfaces show them.
|
||||
resource: src/Backend/AutopilotMonitor.Functions/Helpers/AppVersionRegressionRadar.cs
|
||||
tags:
|
||||
- backend
|
||||
- apps
|
||||
- regression-radar
|
||||
- insights
|
||||
timestamp: 2026-08-13T12:00:00+02:00
|
||||
---
|
||||
|
||||
# App Version Duration Regression Radar
|
||||
|
||||
Detects "version X made this app's installs much slower" — the concrete, quotable insight
|
||||
("median install duration rose from 11 to 29 min after version X") — from the per-(session,
|
||||
app) `AppInstallSummaries` rows. Duration only: failure-rate per version is already visible
|
||||
in the apps dashboard's `versionBreakdown`, and rate regressions belong to Wilson-interval
|
||||
statistics, not medians.
|
||||
|
||||
# Schema
|
||||
|
||||
## Signal & detection
|
||||
|
||||
Per (tenant, app), the MEASURED install population is: `Status == "Succeeded"`, terminal
|
||||
state not Skipped/Postponed, `MetricsMath.HasMeasuredDuration` (1s..6h — 0s means the start
|
||||
was unobserved, above 6h means back-stamped batches), no `AppIdCollision`, non-empty
|
||||
`AppVersion`. Versions are ordered by FIRST-SEEN install time (min `StartedAt`), never by
|
||||
string comparison ("9.1" vs "2024.10" would lie). The comparison pair is always the newest
|
||||
version (the one with the most recent measured install) vs. the version with the latest
|
||||
first-seen strictly before it.
|
||||
|
||||
MEDIANS, not means (nearest-rank, shared with the apps dashboard via
|
||||
`AppsAnalyticsHelper.Percentile`) — one back-stamped straggler must not fire a fleet alert.
|
||||
A regression fires only when ALL hold over the trailing 35-day horizon:
|
||||
|
||||
1. both versions have ≥ 10 measured installs;
|
||||
2. current median ≥ 2× the previous version's median;
|
||||
3. absolute increase ≥ 300 s (a 20s→45s app is noise, not an incident).
|
||||
|
||||
## Suppression & limits
|
||||
|
||||
* Parallel ring rollouts can make "previous" a concurrently-deployed version — accepted in
|
||||
v1; the episode key (app, current version) still caps it at one bell per version.
|
||||
* Platform kill switch: app setting `AppVersionRegressionRadarDisabled=true` skips the
|
||||
radar (fail-open — it only notifies, never mutates data).
|
||||
* One cross-tenant install-summary read per pass (the same query the global apps dashboard
|
||||
runs), grouped by tenant; per-tenant failures are non-fatal.
|
||||
|
||||
## Alert episodes (tracker keyspace)
|
||||
|
||||
An episode lives as `appversionregression|{app-lower}|{version-lower}` (table-key-sanitized;
|
||||
raw AppName/CurrentVersion live in columns, the mapper never parses the RowKey) in the
|
||||
notification-tracker table, next to the `ruleregression|` keyspace: the row IS the dedup
|
||||
(bell + ops event fire exactly once per episode) and the `versionRegressions[]` payload —
|
||||
both medians, both sample counts, lift. Numbers refresh on every radar pass while the
|
||||
episode stays active; `FirstNotifiedAt` never moves. The episode closes (dedup re-arms) when
|
||||
the alerted version drains under 10 measured installs in the horizon or its recomputed
|
||||
median falls back under 1.5× the previous version's median; the tracker's 30-day retention
|
||||
sweep also re-arms, so a month-old still-burning regression rings once more by design.
|
||||
|
||||
## Notification chain (v1: bell + ops event)
|
||||
|
||||
* Tenant bell `app_version_duration_regression` (audience: Admin), href
|
||||
`/apps/detail?name={app}&days=35` (canonical `appDetailUrl` shape). Message carries the
|
||||
full numbers so the admin can verify without a portal round-trip.
|
||||
* Ops event `AppVersionDurationRegression` (Tenant category, Warning) — dual-registered in
|
||||
the web `OPS_EVENT_TYPES` catalog so operators can wire Telegram/Teams/Slack alert rules.
|
||||
|
||||
## Surfaces
|
||||
|
||||
* `apps/{appName}/analytics` (+ global twin with `?tenantId=`) responds with
|
||||
`versionRegressions[]` (empty for the cross-tenant aggregate — episodes are per-tenant
|
||||
rows), and `versionBreakdown[]` rows now carry `measuredInstalls` /
|
||||
`medianDurationSeconds` / `p95DurationSeconds`.
|
||||
* Apps detail page: amber "↑ Duration regression" banner above the version cards and a
|
||||
"Median Install Duration by Version" bar chart.
|
||||
|
||||
# Examples
|
||||
|
||||
A tenant ships Contoso VPN 2.4.0 after 2.3.9. 2.3.9 has 40 measured installs, median 660s;
|
||||
2.4.0 accumulates 12 measured installs with median 1740s. Lift 2.6×, absolute +1080s →
|
||||
episode `appversionregression|contoso vpn|2.4.0` opens, one bell + one ops event fire.
|
||||
Subsequent passes refresh the numbers only. When 2.4.1 replaces 2.4.0 and 2.4.0 drains
|
||||
below 10 measured installs in the horizon, the episode deletes and the dedup re-arms.
|
||||
|
||||
# Citations
|
||||
|
||||
* `src/Backend/AutopilotMonitor.Functions/Helpers/AppVersionRegressionRadar.cs` — pure core
|
||||
(gates, first-seen ordering, re-arm).
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.AppVersionRegression.cs`
|
||||
— orchestration, kill switch, bell/ops-event emission, wording contract.
|
||||
* `src/Backend/AutopilotMonitor.Functions/DataAccess/TableStorage/TableHardwareRejectionNotificationTracker.cs`
|
||||
— episode keyspace, entity round-trip.
|
||||
* `src/Shared/AutopilotMonitor.Shared/Models/Metrics/AppVersionRegressionAlert.cs` — persisted
|
||||
alert payload.
|
||||
* [Rule Regression Radar](rule-regression-radar.md) — the sibling pattern this radar mirrors.
|
||||
@@ -39,6 +39,7 @@ https://docs.autopilotmonitor.com (separate repository).
|
||||
* [Device Journeys & First-Time-Right](backend/device-journeys.md) - How a device's terminal enrollment attempts are grouped into journeys (trim+casefold device key with junk-serial exclusion, first-success end, 30-day gap, WhiteGlove = one attempt), the DeviceHistories chain with its tombstone-driven deleted-session pruning, the additive daily FTR aggregates, and the banner/fleet/MCP surfaces.
|
||||
* [Enrollment Time Attribution](backend/time-attribution.md) - How a terminal session's authoritative wall clock (DurationSeconds, WhiteGlove pause excluded) is partitioned into named segments with an explicit unattributed remainder, ESP-blocking apps as positive evidence with "up to" what-if bounds, reboot spans from event gaps, and the breakdown/aggregate tables the sweep maintains.
|
||||
* [Rule Regression Radar](backend/rule-regression-radar.md) - How analyze-rule hit-rate regressions are detected (7-day window ≥2× the 28-day baseline with disjoint Wilson intervals), which suppressions keep the radar quiet (grace period, edit-in-window, kill switch), how alert episodes live in the ruleregression tracker keyspace with re-arm semantics, and the on-fire dimension correlation with its "correlated, not necessarily causal" wording contract.
|
||||
* [App Version Duration Regression Radar](backend/app-version-regression-radar.md) - How app versions whose median install duration regressed vs. the previous version are detected (first-seen-ordered versions, median lift ≥2× plus a 5-minute absolute floor over measured installs only), how alert episodes live in the appversionregression tracker keyspace with drain/median re-arm semantics, and the bell/ops-event/analytics surfaces.
|
||||
* [Tenant Activation & the Once-Only Welcome Mail](backend/tenant-activation-welcome-mail.md) - How a signup becomes an activated tenant (manual approve or the auto-approve queue), and the write-then-read + conditional-marker arbitration that sends the welcome mail exactly once even though activation and the user-typed notification address race each other.
|
||||
* [W365 Cloud PC Device Validation](backend/cloudpc-device-validation.md) - Why Cloud PCs can never pass the Autopilot serial lookup, and the cert-CN-bound CloudPc fallback validator: Subject CN (= Intune device id) matched against the tenant's Graph virtualEndpoint/cloudPCs inventory — admits exactly the service-provisioned Cloud PCs, nothing else.
|
||||
* [ServerActions & On-Demand Diagnostics](backend/server-actions-on-demand-diagnostics.md) - The queue-on-session, deliver-on-ingest action channel, the TenantAdminOrOperator tier with in-function per-type re-gating, the portal's Collect Logs flow (quick-config orders rotate_config before request_diagnostics), and the hosted-prefix sweep in the offboarding cascade.
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
# Log
|
||||
|
||||
## 2026-08-13 (later)
|
||||
|
||||
* **Creation**: Added `backend/app-version-regression-radar.md` — app-version duration regression radar: per (tenant, app) the newest version's median measured install duration (nearest-rank; population Succeeded + non-skip + `HasMeasuredDuration` + no AppId collision + non-empty AppVersion) is compared against the previous version (first-seen ordering, never string sort); fires at median lift ≥2× AND ≥300s absolute with ≥10 measured installs on both sides. Episodes live as `appversionregression|{app}|{version}` tracker rows (one bell `app_version_duration_regression` + one `AppVersionDurationRegression` ops event per episode, refresh keeps `FirstNotifiedAt`, re-arm on drain/median-recovery/30d retention). Surfaces: `versionRegressions[]` + per-version `measuredInstalls`/`medianDurationSeconds`/`p95DurationSeconds` in the apps analytics responses, amber banner + duration-by-version chart on the apps detail page. Kill switch `AppVersionRegressionRadarDisabled`.
|
||||
|
||||
## 2026-08-13
|
||||
|
||||
* **Update**: `backend/session-annotations.md` — the `/annotations` portal overview page now carries the standard tenant switcher (GA/Reader: "All tenants" aggregate + per-tenant drill-down via `GET global/session-annotations`; delegated: managed subset), with a tenant column and tenant-qualified session deep links. The global list endpoint now derives `excludeGlobalAdminLane` from `HasGlobalScope`, so delegated-rescue callers on their managed `?tenantId=` path never receive the platform-internal `globaladmin` lane (endpoint-test-pinned).
|
||||
|
||||
@@ -0,0 +1,380 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using System.Text.Json;
|
||||
using System.Threading.Tasks;
|
||||
using AutopilotMonitor.Functions.DataAccess.TableStorage;
|
||||
using AutopilotMonitor.Functions.Functions.Apps;
|
||||
using AutopilotMonitor.Functions.Helpers;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using AutopilotMonitor.Shared.DataAccess;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using Moq;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Pins the app-version duration regression radar (<see cref="AppVersionRegressionRadar"/>):
|
||||
/// detection gates (≥10 measured installs on both sides, median lift ≥2×, ≥300s absolute),
|
||||
/// first-seen version ordering (never string sort), the measured-population exclusions,
|
||||
/// re-arm semantics, the tracker keyspace round-trip, the wording contract, and the
|
||||
/// versionBreakdown duration fields on the analytics wire shape.
|
||||
/// </summary>
|
||||
public class AppVersionRegressionRadarTests
|
||||
{
|
||||
private const string TenantA = "00000000-0000-0000-0000-0000000000a1";
|
||||
private const string App = "Contoso VPN";
|
||||
private static readonly DateTime Now = new(2026, 8, 13, 12, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
private static int _sessionCounter;
|
||||
|
||||
private static AppInstallSummary Install(
|
||||
string version,
|
||||
int durationSeconds,
|
||||
DateTime? startedAt = null,
|
||||
string appName = App,
|
||||
string status = "Succeeded",
|
||||
string terminalState = "Installed",
|
||||
bool collision = false)
|
||||
=> new()
|
||||
{
|
||||
TenantId = TenantA,
|
||||
SessionId = $"s-{++_sessionCounter}",
|
||||
AppName = appName,
|
||||
AppVersion = version,
|
||||
Status = status,
|
||||
TerminalState = terminalState,
|
||||
DurationSeconds = durationSeconds,
|
||||
StartedAt = startedAt ?? Now.AddDays(-1),
|
||||
AppIdCollision = collision,
|
||||
};
|
||||
|
||||
/// <summary>n measured installs of one version, all with the same duration, spread over one day.</summary>
|
||||
private static List<AppInstallSummary> Batch(string version, int count, int durationSeconds, DateTime firstSeen)
|
||||
=> Enumerable.Range(0, count)
|
||||
.Select(i => Install(version, durationSeconds, firstSeen.AddMinutes(i)))
|
||||
.ToList();
|
||||
|
||||
// ── detection gates ─────────────────────────────────────────────────────
|
||||
|
||||
[Theory]
|
||||
// currentCount, previousCount, previousMedian, currentMedian, expectFire
|
||||
[InlineData(10, 10, 300, 900, true)] // lift 3×, +600s — fires
|
||||
[InlineData(10, 10, 300, 600, true)] // exactly 2× AND exactly +300s — boundary fires
|
||||
[InlineData(9, 10, 300, 900, false)] // current side below sample floor
|
||||
[InlineData(10, 9, 300, 900, false)] // previous side below sample floor
|
||||
[InlineData(10, 10, 200, 499, false)] // lift 2.5× but only +299s — absolute floor blocks
|
||||
[InlineData(10, 10, 1000, 1900, false)] // +900s but lift 1.9× — lift gate blocks
|
||||
public void Evaluate_Gates(int currentCount, int previousCount, int previousMedian, int currentMedian, bool expectFire)
|
||||
{
|
||||
var rows = Batch("1.0", previousCount, previousMedian, Now.AddDays(-20));
|
||||
rows.AddRange(Batch("2.0", currentCount, currentMedian, Now.AddDays(-5)));
|
||||
|
||||
var findings = AppVersionRegressionRadar.Evaluate(rows);
|
||||
|
||||
if (!expectFire)
|
||||
{
|
||||
Assert.Empty(findings);
|
||||
return;
|
||||
}
|
||||
|
||||
var finding = Assert.Single(findings);
|
||||
Assert.Equal(App, finding.AppName);
|
||||
Assert.Equal("2.0", finding.CurrentVersion);
|
||||
Assert.Equal("1.0", finding.PreviousVersion);
|
||||
Assert.Equal(currentMedian, finding.CurrentMedianSeconds);
|
||||
Assert.Equal(previousMedian, finding.PreviousMedianSeconds);
|
||||
Assert.Equal(currentCount, finding.CurrentMeasuredCount);
|
||||
Assert.Equal(previousCount, finding.PreviousMeasuredCount);
|
||||
Assert.Equal(Math.Round((double)currentMedian / previousMedian, 1), finding.Lift);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Evaluate_MedianIsNearestRank_HandComputed()
|
||||
{
|
||||
// Even n: nearest-rank p50 takes the LOWER middle value (rank = ceil(0.5*4)-1 = 1).
|
||||
var rows = new List<AppInstallSummary>();
|
||||
foreach (var d in new[] { 100, 200, 300, 400 }) // median 200
|
||||
rows.AddRange(Batch("1.0", 3, d, Now.AddDays(-20)));
|
||||
foreach (var d in new[] { 500, 600, 700, 800 }) // median 600
|
||||
rows.AddRange(Batch("2.0", 3, d, Now.AddDays(-5)));
|
||||
|
||||
var finding = Assert.Single(AppVersionRegressionRadar.Evaluate(rows));
|
||||
Assert.Equal(600, finding.CurrentMedianSeconds);
|
||||
Assert.Equal(200, finding.PreviousMedianSeconds);
|
||||
Assert.Equal(3.0, finding.Lift);
|
||||
}
|
||||
|
||||
// ── version ordering ────────────────────────────────────────────────────
|
||||
|
||||
[Fact]
|
||||
public void Evaluate_VersionOrder_ByFirstSeen_NeverStringSort()
|
||||
{
|
||||
// Ordinally "9.1" > "2024.10", but "2024.10" is the NEWER version by first-seen.
|
||||
// A string sort would flip current/previous and compare in the wrong direction.
|
||||
var rows = Batch("9.1", 10, 300, Now.AddDays(-20));
|
||||
rows.AddRange(Batch("2024.10", 10, 900, Now.AddDays(-5)));
|
||||
|
||||
var finding = Assert.Single(AppVersionRegressionRadar.Evaluate(rows));
|
||||
Assert.Equal("2024.10", finding.CurrentVersion);
|
||||
Assert.Equal("9.1", finding.PreviousVersion);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Evaluate_MultipleVersionsInFlight_PreviousIsLatestFirstSeenBeforeCurrent()
|
||||
{
|
||||
var rows = Batch("1.0", 10, 300, Now.AddDays(-30)); // oldest
|
||||
rows.AddRange(Batch("1.1", 10, 320, Now.AddDays(-18))); // the true predecessor
|
||||
rows.AddRange(Batch("2.0", 10, 960, Now.AddDays(-4))); // newest, regressed
|
||||
|
||||
var finding = Assert.Single(AppVersionRegressionRadar.Evaluate(rows));
|
||||
Assert.Equal("2.0", finding.CurrentVersion);
|
||||
Assert.Equal("1.1", finding.PreviousVersion); // not 1.0
|
||||
Assert.Equal(320, finding.PreviousMedianSeconds);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Evaluate_SingleVersion_NeverFires()
|
||||
{
|
||||
var rows = Batch("1.0", 25, 900, Now.AddDays(-5));
|
||||
Assert.Empty(AppVersionRegressionRadar.Evaluate(rows));
|
||||
}
|
||||
|
||||
// ── measured-population exclusions ──────────────────────────────────────
|
||||
|
||||
[Fact]
|
||||
public void Evaluate_ExcludedRows_DoNotCountTowardSamplesOrMedians()
|
||||
{
|
||||
var rows = Batch("1.0", 10, 300, Now.AddDays(-20));
|
||||
rows.AddRange(Batch("2.0", 10, 900, Now.AddDays(-5)));
|
||||
|
||||
// None of these may move the samples or medians:
|
||||
rows.Add(Install("2.0", 0, Now.AddDays(-2))); // unmeasured (start unobserved)
|
||||
rows.Add(Install("2.0", 21601, Now.AddDays(-2))); // over the 6h plausibility cap
|
||||
rows.Add(Install("2.0", 50, Now.AddDays(-2), terminalState: "Skipped")); // skip — not an attempt
|
||||
rows.Add(Install("2.0", 50, Now.AddDays(-2), terminalState: "Postponed")); // skip — not an attempt
|
||||
rows.Add(Install("2.0", 50, Now.AddDays(-2), status: "Failed")); // failed — no duration signal
|
||||
rows.Add(Install("2.0", 50, Now.AddDays(-2), collision: true)); // AppId collision — foreign outcomes
|
||||
rows.Add(Install("", 50, Now.AddDays(-2))); // no version
|
||||
rows.Add(Install("2.0", 50, Now.AddDays(-2), appName: "")); // no app name
|
||||
|
||||
var finding = Assert.Single(AppVersionRegressionRadar.Evaluate(rows));
|
||||
Assert.Equal(10, finding.CurrentMeasuredCount);
|
||||
Assert.Equal(900, finding.CurrentMedianSeconds);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Evaluate_DeterministicOrdering_LiftDesc_ThenAppNameOrdinal()
|
||||
{
|
||||
var rows = new List<AppInstallSummary>();
|
||||
rows.AddRange(Batch("1.0", 10, 300, Now.AddDays(-20)).Select(s => { s.AppName = "App B"; return s; }));
|
||||
rows.AddRange(Batch("2.0", 10, 900, Now.AddDays(-5)).Select(s => { s.AppName = "App B"; return s; })); // lift 3.0
|
||||
rows.AddRange(Batch("1.0", 10, 300, Now.AddDays(-20)).Select(s => { s.AppName = "App A"; return s; }));
|
||||
rows.AddRange(Batch("2.0", 10, 1500, Now.AddDays(-5)).Select(s => { s.AppName = "App A"; return s; })); // lift 5.0
|
||||
|
||||
var findings = AppVersionRegressionRadar.Evaluate(rows);
|
||||
Assert.Equal(2, findings.Count);
|
||||
Assert.Equal("App A", findings[0].AppName); // higher lift first
|
||||
Assert.Equal("App B", findings[1].AppName);
|
||||
}
|
||||
|
||||
// ── re-arm ──────────────────────────────────────────────────────────────
|
||||
|
||||
private static AppVersionRegressionAlert Alert(
|
||||
string currentVersion = "2.0", string previousVersion = "1.0")
|
||||
=> new()
|
||||
{
|
||||
TenantId = TenantA,
|
||||
AppName = App,
|
||||
CurrentVersion = currentVersion,
|
||||
PreviousVersion = previousVersion,
|
||||
CurrentMedianSeconds = 900,
|
||||
PreviousMedianSeconds = 300,
|
||||
CurrentMeasuredCount = 10,
|
||||
PreviousMeasuredCount = 10,
|
||||
Lift = 3.0,
|
||||
FirstNotifiedAt = Now.AddDays(-3),
|
||||
LastEvaluatedAt = Now.AddDays(-1),
|
||||
};
|
||||
|
||||
[Fact]
|
||||
public void ShouldReArm_WhenCurrentVersionDrainedOutOfHorizon()
|
||||
{
|
||||
// Only the previous version still has measured installs — the alerted version is gone.
|
||||
var rows = Batch("1.0", 10, 300, Now.AddDays(-20));
|
||||
Assert.True(AppVersionRegressionRadar.ShouldReArm(rows, Alert()));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ShouldReArm_WhenCurrentVersionBelowSampleFloor()
|
||||
{
|
||||
var rows = Batch("1.0", 10, 300, Now.AddDays(-20));
|
||||
rows.AddRange(Batch("2.0", 9, 900, Now.AddDays(-5)));
|
||||
Assert.True(AppVersionRegressionRadar.ShouldReArm(rows, Alert()));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ShouldReArm_WhenMedianFellBackUnderReArmFactor()
|
||||
{
|
||||
// 400 < 1.5 × 300 — the regression recovered (e.g. CDN warmed up).
|
||||
var rows = Batch("1.0", 10, 300, Now.AddDays(-20));
|
||||
rows.AddRange(Batch("2.0", 10, 400, Now.AddDays(-5)));
|
||||
Assert.True(AppVersionRegressionRadar.ShouldReArm(rows, Alert()));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ShouldNotReArm_WhileMedianStaysElevated()
|
||||
{
|
||||
// 500 ≥ 1.5 × 300 — still elevated (even though below the 2× fire gate).
|
||||
var rows = Batch("1.0", 10, 300, Now.AddDays(-20));
|
||||
rows.AddRange(Batch("2.0", 10, 500, Now.AddDays(-5)));
|
||||
Assert.False(AppVersionRegressionRadar.ShouldReArm(rows, Alert()));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ShouldNotReArm_WhenPreviousVersionDrained_NoComparisonBasis()
|
||||
{
|
||||
// The predecessor left the horizon; the episode is kept until the current
|
||||
// version itself drains (fires-stopped-style re-arm only).
|
||||
var rows = Batch("2.0", 10, 900, Now.AddDays(-5));
|
||||
Assert.False(AppVersionRegressionRadar.ShouldReArm(rows, Alert()));
|
||||
}
|
||||
|
||||
// ── tracker keyspace (round-trip + key shape) ───────────────────────────
|
||||
|
||||
[Fact]
|
||||
public void AppVersionRegressionRowKey_PrefixedCaseFoldedAndSanitized()
|
||||
{
|
||||
Assert.Equal("appversionregression|contoso app_2|1.0_beta_",
|
||||
TableHardwareRejectionNotificationTracker.BuildAppVersionRegressionRowKey(
|
||||
" Contoso App/2 ", " 1.0#Beta? "));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void AppVersionRegressionRowKey_NoPrefixCollisionWithOtherKeyspaces()
|
||||
{
|
||||
var key = TableHardwareRejectionNotificationTracker.BuildAppVersionRegressionRowKey(App, "1.0");
|
||||
Assert.StartsWith("appversionregression|", key, StringComparison.Ordinal);
|
||||
// The prefix range [prefix, "appversionregression}") must not swallow the sibling
|
||||
// keyspaces ("ruleregression|", "tpmpss|", "{mfr}|{model}").
|
||||
Assert.True(string.CompareOrdinal("ruleregression|", "appversionregression}") > 0);
|
||||
Assert.True(string.CompareOrdinal("tpmpss|", "appversionregression}") > 0);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void AppVersionRegressionEntity_RoundTripsAllFields()
|
||||
{
|
||||
var alert = new AppVersionRegressionAlert
|
||||
{
|
||||
TenantId = TenantA,
|
||||
AppName = App,
|
||||
CurrentVersion = "2.4.0",
|
||||
PreviousVersion = "2.3.9",
|
||||
CurrentMedianSeconds = 1740,
|
||||
PreviousMedianSeconds = 660,
|
||||
CurrentMeasuredCount = 12,
|
||||
PreviousMeasuredCount = 40,
|
||||
Lift = 2.6,
|
||||
FirstNotifiedAt = Now,
|
||||
LastEvaluatedAt = Now.AddHours(2),
|
||||
};
|
||||
|
||||
var entity = TableHardwareRejectionNotificationTracker.BuildAppVersionRegressionEntity(TenantA, alert);
|
||||
Assert.Equal(TenantA.ToLowerInvariant(), entity.PartitionKey);
|
||||
Assert.Equal("appversionregression|contoso vpn|2.4.0", entity.RowKey);
|
||||
|
||||
var mapped = TableHardwareRejectionNotificationTracker.MapToAppVersionRegressionAlert(entity);
|
||||
Assert.Equal(TenantA, mapped.TenantId);
|
||||
Assert.Equal(App, mapped.AppName);
|
||||
Assert.Equal("2.4.0", mapped.CurrentVersion);
|
||||
Assert.Equal("2.3.9", mapped.PreviousVersion);
|
||||
Assert.Equal(1740, mapped.CurrentMedianSeconds);
|
||||
Assert.Equal(660, mapped.PreviousMedianSeconds);
|
||||
Assert.Equal(12, mapped.CurrentMeasuredCount);
|
||||
Assert.Equal(40, mapped.PreviousMeasuredCount);
|
||||
Assert.Equal(2.6, mapped.Lift);
|
||||
Assert.Equal(Now, mapped.FirstNotifiedAt);
|
||||
Assert.Equal(Now.AddHours(2), mapped.LastEvaluatedAt);
|
||||
}
|
||||
|
||||
// ── wording contract ────────────────────────────────────────────────────
|
||||
|
||||
[Fact]
|
||||
public void RegressionMessage_CarriesTheFullNumbers_MinutesOneDecimal()
|
||||
{
|
||||
var finding = new AppVersionDurationRegressionFinding
|
||||
{
|
||||
TenantId = TenantA,
|
||||
AppName = App,
|
||||
CurrentVersion = "2.4.0",
|
||||
PreviousVersion = "2.3.9",
|
||||
CurrentMedianSeconds = 1740,
|
||||
PreviousMedianSeconds = 660,
|
||||
CurrentMeasuredCount = 12,
|
||||
PreviousMeasuredCount = 40,
|
||||
Lift = 2.6,
|
||||
};
|
||||
|
||||
Assert.Equal(
|
||||
"Median install duration rose from 11 to 29 min after version 2.4.0 " +
|
||||
"(12 measured installs vs 40 on version 2.3.9) — lift 2.6x over the last 35 days.",
|
||||
MaintenanceService.BuildAppVersionRegressionMessage(finding));
|
||||
}
|
||||
|
||||
// ── analytics wire shape (versionBreakdown duration fields) ─────────────
|
||||
|
||||
[Fact]
|
||||
public async Task Analytics_VersionBreakdown_CarriesMeasuredDurationStats()
|
||||
{
|
||||
var sessionRepo = new Mock<ISessionRepository>();
|
||||
sessionRepo.Setup(r => r.GetSessionAsync(It.IsAny<string>(), It.IsAny<string>()))
|
||||
.ReturnsAsync((SessionSummary?)null);
|
||||
|
||||
var rows = new List<AppInstallSummary>
|
||||
{
|
||||
Install("1.0", 100), Install("1.0", 200), Install("1.0", 300), Install("1.0", 400),
|
||||
Install("1.0", 0), // unmeasured — excluded from stats
|
||||
Install("1.0", 50, terminalState: "Skipped"), // skip — excluded
|
||||
Install("1.0", 50, status: "Failed"), // failed — excluded from durations
|
||||
};
|
||||
|
||||
var root = JsonSerializer.SerializeToElement(await AppsAnalyticsHelper.BuildAnalyticsResponseAsync(
|
||||
rows, sessionRepo.Object, App, days: 30));
|
||||
|
||||
var version = root.GetProperty("versionBreakdown")[0];
|
||||
Assert.Equal("1.0", version.GetProperty("appVersion").GetString());
|
||||
Assert.Equal(7, version.GetProperty("installs").GetInt32());
|
||||
Assert.Equal(4, version.GetProperty("measuredInstalls").GetInt32());
|
||||
// Nearest-rank over [100, 200, 300, 400]: p50 = 200 (lower middle), p95 = 400.
|
||||
Assert.Equal(200, version.GetProperty("medianDurationSeconds").GetInt32());
|
||||
Assert.Equal(400, version.GetProperty("p95DurationSeconds").GetInt32());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Analytics_VersionRegressions_SerializedCamelCase_AndPresentWhenEmptyData()
|
||||
{
|
||||
var sessionRepo = new Mock<ISessionRepository>();
|
||||
sessionRepo.Setup(r => r.GetSessionAsync(It.IsAny<string>(), It.IsAny<string>()))
|
||||
.ReturnsAsync((SessionSummary?)null);
|
||||
|
||||
var alert = Alert();
|
||||
// The Functions host serializes camelCase on the wire; mirror that here so the
|
||||
// asserted property names match what the web client actually parses.
|
||||
var wireOptions = new JsonSerializerOptions(JsonSerializerDefaults.Web);
|
||||
|
||||
// Empty-data early return still surfaces the block (episodes can outlive window data).
|
||||
var emptyRoot = JsonSerializer.SerializeToElement(await AppsAnalyticsHelper.BuildAnalyticsResponseAsync(
|
||||
new List<AppInstallSummary>(), sessionRepo.Object, App, days: 30,
|
||||
new List<AppVersionRegressionAlert> { alert }), wireOptions);
|
||||
Assert.Equal(1, emptyRoot.GetProperty("versionRegressions").GetArrayLength());
|
||||
|
||||
var root = JsonSerializer.SerializeToElement(await AppsAnalyticsHelper.BuildAnalyticsResponseAsync(
|
||||
new List<AppInstallSummary> { Install("1.0", 100) }, sessionRepo.Object, App, days: 30,
|
||||
new List<AppVersionRegressionAlert> { alert }), wireOptions);
|
||||
|
||||
var regression = root.GetProperty("versionRegressions")[0];
|
||||
Assert.Equal("2.0", regression.GetProperty("currentVersion").GetString());
|
||||
Assert.Equal(900, regression.GetProperty("currentMedianSeconds").GetInt32());
|
||||
}
|
||||
}
|
||||
@@ -17,6 +17,8 @@ public class TenantNotificationAudienceCatalogTests
|
||||
[InlineData("sla_breach", NotificationAudience.Member)]
|
||||
[InlineData("sla_consecutive_failures", NotificationAudience.Member)]
|
||||
[InlineData("sla_resolved", NotificationAudience.Member)]
|
||||
[InlineData("rule_frequency_regression", NotificationAudience.Admin)]
|
||||
[InlineData("app_version_duration_regression", NotificationAudience.Admin)]
|
||||
public void Resolve_KnownTypes_ReturnsExpectedAudience(string type, NotificationAudience expected)
|
||||
{
|
||||
Assert.Equal(expected, TenantNotificationAudienceCatalog.Resolve(type));
|
||||
|
||||
+168
@@ -17,6 +17,9 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
|
||||
/// cannot collide with a hardware key because no real manufacturer is named "tpmpss")
|
||||
/// - F3 rule regression: "ruleregression|{ruleId-lower}" (trimmed; payload-carrying rows
|
||||
/// that are refreshed while the episode is active and deleted on re-arm)
|
||||
/// - app-version duration regression: "appversionregression|{app-lower}|{version-lower}"
|
||||
/// (trimmed + table-key-sanitized; same payload-carrying episode semantics — raw
|
||||
/// AppName/CurrentVersion live in columns, the mapper never parses the RowKey)
|
||||
/// Race-safe via AddEntityAsync: Azure Table Storage returns 409 Conflict if the entity already exists.
|
||||
/// </summary>
|
||||
public class TableHardwareRejectionNotificationTracker : IHardwareRejectionNotificationTracker
|
||||
@@ -251,6 +254,107 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
|
||||
return $"tpmpss|{sn}";
|
||||
}
|
||||
|
||||
public async Task<bool> TryRegisterAppVersionRegressionAsync(string tenantId, AppVersionRegressionAlert alert)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(tenantId)
|
||||
|| string.IsNullOrWhiteSpace(alert?.AppName)
|
||||
|| string.IsNullOrWhiteSpace(alert?.CurrentVersion))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
await _table.AddEntityAsync(BuildAppVersionRegressionEntity(tenantId, alert!));
|
||||
_logger.LogInformation(
|
||||
"AppVersionRegression tracker registered: tenant={TenantId} app={AppName} version={Version}",
|
||||
tenantId, alert!.AppName, alert!.CurrentVersion);
|
||||
return true;
|
||||
}
|
||||
catch (RequestFailedException ex) when (ex.Status == 409)
|
||||
{
|
||||
// Episode already active — one bell per (app, version) episode.
|
||||
return false;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex,
|
||||
"AppVersionRegression tracker failed: tenant={TenantId} app={AppName} version={Version}",
|
||||
tenantId, alert!.AppName, alert!.CurrentVersion);
|
||||
// Fail closed so we do not double-fire if the row was actually written.
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
public async Task RefreshAppVersionRegressionAsync(string tenantId, AppVersionRegressionAlert alert)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(tenantId)
|
||||
|| string.IsNullOrWhiteSpace(alert?.AppName)
|
||||
|| string.IsNullOrWhiteSpace(alert?.CurrentVersion))
|
||||
{
|
||||
return;
|
||||
}
|
||||
try
|
||||
{
|
||||
// Replace-mode upsert with the FULL payload incl. the caller-carried
|
||||
// FirstNotifiedAt — the retention sweep re-arms on the ORIGINAL age, so a
|
||||
// refresh must never rejuvenate the row.
|
||||
await _table.UpsertEntityAsync(BuildAppVersionRegressionEntity(tenantId, alert!), TableUpdateMode.Replace);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex,
|
||||
"AppVersionRegression tracker refresh failed: tenant={TenantId} app={AppName} version={Version} (stale numbers remain)",
|
||||
tenantId, alert!.AppName, alert!.CurrentVersion);
|
||||
}
|
||||
}
|
||||
|
||||
public async Task DeleteAppVersionRegressionAsync(string tenantId, string appName, string currentVersion)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(tenantId)
|
||||
|| string.IsNullOrWhiteSpace(appName)
|
||||
|| string.IsNullOrWhiteSpace(currentVersion))
|
||||
{
|
||||
return;
|
||||
}
|
||||
try
|
||||
{
|
||||
await _table.DeleteEntityAsync(
|
||||
tenantId.ToLowerInvariant(), BuildAppVersionRegressionRowKey(appName, currentVersion));
|
||||
}
|
||||
catch (RequestFailedException ex) when (ex.Status == 404)
|
||||
{
|
||||
// Already gone (retention sweep or concurrent pass) — idempotent.
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex,
|
||||
"AppVersionRegression tracker delete failed: tenant={TenantId} app={AppName} version={Version} (episode stays until retention)",
|
||||
tenantId, appName, currentVersion);
|
||||
}
|
||||
}
|
||||
|
||||
public async Task<List<AppVersionRegressionAlert>> GetAppVersionRegressionsAsync(string tenantId)
|
||||
{
|
||||
var results = new List<AppVersionRegressionAlert>();
|
||||
if (string.IsNullOrWhiteSpace(tenantId)) return results;
|
||||
try
|
||||
{
|
||||
var partitionKey = tenantId.ToLowerInvariant();
|
||||
// '}' (0x7D) sorts directly after '|' (0x7C) — prefix range over "appversionregression|…".
|
||||
var filter = $"PartitionKey eq '{partitionKey}' and RowKey ge '{AppVersionRegressionRowKeyPrefix}' and RowKey lt 'appversionregression}}'";
|
||||
await foreach (var entity in _table.QueryAsync<TableEntity>(filter: filter))
|
||||
{
|
||||
results.Add(MapToAppVersionRegressionAlert(entity));
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "AppVersionRegression tracker list failed for tenant {TenantId}", tenantId);
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
internal const string RuleRegressionRowKeyPrefix = "ruleregression|";
|
||||
|
||||
internal static string BuildRuleRegressionRowKey(string ruleId)
|
||||
@@ -317,5 +421,69 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
|
||||
LastEvaluatedAt = entity.GetDateTimeOffset("LastEvaluatedAt")?.UtcDateTime ?? DateTime.MinValue,
|
||||
};
|
||||
}
|
||||
|
||||
internal const string AppVersionRegressionRowKeyPrefix = "appversionregression|";
|
||||
|
||||
/// <summary>
|
||||
/// App names and versions are free-form agent payloads — they can contain characters
|
||||
/// Azure Table keys reject ('/', '\', '#', '?'). Sanitized for the KEY only (worst case
|
||||
/// two apps share an episode row — tolerable); the raw values live in columns and the
|
||||
/// mapper reads them, never the RowKey.
|
||||
/// </summary>
|
||||
internal static string BuildAppVersionRegressionRowKey(string appName, string version)
|
||||
{
|
||||
var app = SanitizeTableKey((appName ?? string.Empty).Trim().ToLowerInvariant());
|
||||
var ver = SanitizeTableKey((version ?? string.Empty).Trim().ToLowerInvariant());
|
||||
return $"{AppVersionRegressionRowKeyPrefix}{app}|{ver}";
|
||||
}
|
||||
|
||||
/// <summary>Replace characters not allowed in Azure Table Storage keys (/, \, #, ?).</summary>
|
||||
private static string SanitizeTableKey(string key)
|
||||
{
|
||||
return key
|
||||
.Replace("/", "_")
|
||||
.Replace("\\", "_")
|
||||
.Replace("#", "_")
|
||||
.Replace("?", "_");
|
||||
}
|
||||
|
||||
// internal static: entity builder + mapper are pinned by round-trip unit tests
|
||||
// (table-storage-serialization rule — every property must survive Store→Map).
|
||||
internal static TableEntity BuildAppVersionRegressionEntity(string tenantId, AppVersionRegressionAlert alert)
|
||||
{
|
||||
return new TableEntity(
|
||||
tenantId.ToLowerInvariant(), BuildAppVersionRegressionRowKey(alert.AppName, alert.CurrentVersion))
|
||||
{
|
||||
["TenantId"] = tenantId,
|
||||
["AppName"] = alert.AppName,
|
||||
["CurrentVersion"] = alert.CurrentVersion,
|
||||
["PreviousVersion"] = alert.PreviousVersion ?? string.Empty,
|
||||
["CurrentMedianSeconds"] = alert.CurrentMedianSeconds,
|
||||
["PreviousMedianSeconds"] = alert.PreviousMedianSeconds,
|
||||
["CurrentMeasuredCount"] = alert.CurrentMeasuredCount,
|
||||
["PreviousMeasuredCount"] = alert.PreviousMeasuredCount,
|
||||
["Lift"] = alert.Lift,
|
||||
["FirstNotifiedAt"] = alert.FirstNotifiedAt,
|
||||
["LastEvaluatedAt"] = alert.LastEvaluatedAt,
|
||||
};
|
||||
}
|
||||
|
||||
internal static AppVersionRegressionAlert MapToAppVersionRegressionAlert(TableEntity entity)
|
||||
{
|
||||
return new AppVersionRegressionAlert
|
||||
{
|
||||
TenantId = entity.GetString("TenantId") ?? entity.PartitionKey,
|
||||
AppName = entity.GetString("AppName") ?? string.Empty,
|
||||
CurrentVersion = entity.GetString("CurrentVersion") ?? string.Empty,
|
||||
PreviousVersion = entity.GetString("PreviousVersion") ?? string.Empty,
|
||||
CurrentMedianSeconds = entity.GetInt32("CurrentMedianSeconds") ?? 0,
|
||||
PreviousMedianSeconds = entity.GetInt32("PreviousMedianSeconds") ?? 0,
|
||||
CurrentMeasuredCount = entity.GetInt32("CurrentMeasuredCount") ?? 0,
|
||||
PreviousMeasuredCount = entity.GetInt32("PreviousMeasuredCount") ?? 0,
|
||||
Lift = entity.GetDouble("Lift") ?? 0,
|
||||
FirstNotifiedAt = entity.GetDateTimeOffset("FirstNotifiedAt")?.UtcDateTime ?? DateTime.MinValue,
|
||||
LastEvaluatedAt = entity.GetDateTimeOffset("LastEvaluatedAt")?.UtcDateTime ?? DateTime.MinValue,
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -229,8 +229,13 @@ namespace AutopilotMonitor.Functions.Functions.Apps
|
||||
List<AppInstallSummary> allSummaries,
|
||||
ISessionRepository sessionRepo,
|
||||
string appName,
|
||||
int days)
|
||||
int days,
|
||||
IReadOnlyList<AppVersionRegressionAlert>? versionRegressions = null)
|
||||
{
|
||||
// Active duration-regression episodes for THIS app (tracker rows; models serialize
|
||||
// camelCase on the wire like the rule-stats regressions[] block).
|
||||
var appVersionRegressions = versionRegressions ?? Array.Empty<AppVersionRegressionAlert>();
|
||||
|
||||
var now = DateTime.UtcNow;
|
||||
var cutoff = now.AddDays(-days);
|
||||
var midpoint = now.AddDays(-days / 2.0);
|
||||
@@ -275,7 +280,10 @@ namespace AutopilotMonitor.Functions.Functions.Apps
|
||||
installerPhaseBreakdown = Array.Empty<object>(),
|
||||
topFailureCodes = Array.Empty<object>(),
|
||||
detectionLiesCount = 0,
|
||||
deviceModelBreakdown = Array.Empty<object>()
|
||||
deviceModelBreakdown = Array.Empty<object>(),
|
||||
// Lockstep with the full response: episodes can outlive the window's data
|
||||
// (e.g. a shrunk days= selection), so the block is still surfaced here.
|
||||
versionRegressions = appVersionRegressions
|
||||
};
|
||||
}
|
||||
|
||||
@@ -313,12 +321,23 @@ namespace AutopilotMonitor.Functions.Functions.Apps
|
||||
var vFailed = g.Count(s => s.Status == "Failed");
|
||||
// Same PR0 convention as the top-level rate: skips are not attempts.
|
||||
var vSucceeded = g.Count(s => s.Status == "Succeeded" && !MetricsMath.IsSkipTerminalState(s));
|
||||
// Same measured population as the top-level duration stats: succeeded,
|
||||
// non-skip, and a plausible observed duration (0s = start unobserved,
|
||||
// >6h = back-stamped batch — both excluded, never averaged in).
|
||||
var vDurations = g
|
||||
.Where(s => s.Status == "Succeeded" && !MetricsMath.IsSkipTerminalState(s))
|
||||
.Where(MetricsMath.HasMeasuredDuration)
|
||||
.Select(s => s.DurationSeconds)
|
||||
.ToList();
|
||||
return new
|
||||
{
|
||||
appVersion = g.Key,
|
||||
installs = vTotal,
|
||||
failed = vFailed,
|
||||
failureRate = MetricsMath.TerminalFailureRatePct(vFailed, vSucceeded)
|
||||
failureRate = MetricsMath.TerminalFailureRatePct(vFailed, vSucceeded),
|
||||
measuredInstalls = vDurations.Count,
|
||||
medianDurationSeconds = Percentile(vDurations, 0.50),
|
||||
p95DurationSeconds = Percentile(vDurations, 0.95)
|
||||
};
|
||||
})
|
||||
.OrderByDescending(v => v.installs)
|
||||
@@ -421,7 +440,8 @@ namespace AutopilotMonitor.Functions.Functions.Apps
|
||||
installerPhaseBreakdown,
|
||||
topFailureCodes,
|
||||
detectionLiesCount,
|
||||
deviceModelBreakdown
|
||||
deviceModelBreakdown,
|
||||
versionRegressions = appVersionRegressions
|
||||
};
|
||||
}
|
||||
|
||||
@@ -586,7 +606,9 @@ namespace AutopilotMonitor.Functions.Functions.Apps
|
||||
return date.AddDays(-diff);
|
||||
}
|
||||
|
||||
private static int Percentile(List<int> values, double percentile)
|
||||
// Internal so AppVersionRegressionRadar shares the same nearest-rank convention
|
||||
// (a third percentile implementation would invite subtle median drift).
|
||||
internal static int Percentile(List<int> values, double percentile)
|
||||
{
|
||||
if (values.Count == 0) return 0;
|
||||
var sorted = values.OrderBy(v => v).ToList();
|
||||
|
||||
@@ -17,15 +17,18 @@ namespace AutopilotMonitor.Functions.Functions.Apps
|
||||
private readonly ILogger<GetAppAnalyticsFunction> _logger;
|
||||
private readonly IMetricsRepository _metricsRepo;
|
||||
private readonly ISessionRepository _sessionRepo;
|
||||
private readonly IHardwareRejectionNotificationTracker _notificationTracker;
|
||||
|
||||
public GetAppAnalyticsFunction(
|
||||
ILogger<GetAppAnalyticsFunction> logger,
|
||||
IMetricsRepository metricsRepo,
|
||||
ISessionRepository sessionRepo)
|
||||
ISessionRepository sessionRepo,
|
||||
IHardwareRejectionNotificationTracker notificationTracker)
|
||||
{
|
||||
_logger = logger;
|
||||
_metricsRepo = metricsRepo;
|
||||
_sessionRepo = sessionRepo;
|
||||
_notificationTracker = notificationTracker;
|
||||
}
|
||||
|
||||
[Function("GetAppAnalytics")]
|
||||
@@ -51,8 +54,12 @@ namespace AutopilotMonitor.Functions.Functions.Apps
|
||||
days = parsedDays;
|
||||
|
||||
var summaries = await AppsAnalyticsHelper.LoadSummariesAsync(_metricsRepo, tenantId, days);
|
||||
// Active duration-regression episodes for this app (fail-soft: empty on error).
|
||||
var versionRegressions = (await _notificationTracker.GetAppVersionRegressionsAsync(tenantId))
|
||||
.Where(a => string.Equals(a.AppName, decodedAppName, StringComparison.OrdinalIgnoreCase))
|
||||
.ToList();
|
||||
var body = await AppsAnalyticsHelper.BuildAnalyticsResponseAsync(
|
||||
summaries, _sessionRepo, decodedAppName, days);
|
||||
summaries, _sessionRepo, decodedAppName, days, versionRegressions);
|
||||
|
||||
var response = req.CreateResponse(HttpStatusCode.OK);
|
||||
await response.WriteAsJsonAsync(body);
|
||||
|
||||
+13
-2
@@ -1,6 +1,7 @@
|
||||
using System.Net;
|
||||
using AutopilotMonitor.Functions.Helpers;
|
||||
using AutopilotMonitor.Shared.DataAccess;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using Microsoft.Azure.Functions.Worker;
|
||||
using Microsoft.Azure.Functions.Worker.Http;
|
||||
using Microsoft.Extensions.Logging;
|
||||
@@ -17,15 +18,18 @@ namespace AutopilotMonitor.Functions.Functions.Apps
|
||||
private readonly ILogger<GetGlobalAppAnalyticsFunction> _logger;
|
||||
private readonly IMetricsRepository _metricsRepo;
|
||||
private readonly ISessionRepository _sessionRepo;
|
||||
private readonly IHardwareRejectionNotificationTracker _notificationTracker;
|
||||
|
||||
public GetGlobalAppAnalyticsFunction(
|
||||
ILogger<GetGlobalAppAnalyticsFunction> logger,
|
||||
IMetricsRepository metricsRepo,
|
||||
ISessionRepository sessionRepo)
|
||||
ISessionRepository sessionRepo,
|
||||
IHardwareRejectionNotificationTracker notificationTracker)
|
||||
{
|
||||
_logger = logger;
|
||||
_metricsRepo = metricsRepo;
|
||||
_sessionRepo = sessionRepo;
|
||||
_notificationTracker = notificationTracker;
|
||||
}
|
||||
|
||||
[Function("GetGlobalAppAnalytics")]
|
||||
@@ -61,8 +65,15 @@ namespace AutopilotMonitor.Functions.Functions.Apps
|
||||
decodedAppName, userEmail, scopedTenantId ?? "<all>", days);
|
||||
|
||||
var summaries = await AppsAnalyticsHelper.LoadSummariesAsync(_metricsRepo, scopedTenantId, days);
|
||||
// Episodes are per-tenant tracker rows; without a tenant scope the aggregated
|
||||
// cross-tenant view carries none (mirrors the rule-stats global route).
|
||||
var versionRegressions = string.IsNullOrEmpty(scopedTenantId)
|
||||
? new List<AppVersionRegressionAlert>()
|
||||
: (await _notificationTracker.GetAppVersionRegressionsAsync(scopedTenantId!))
|
||||
.Where(a => string.Equals(a.AppName, decodedAppName, StringComparison.OrdinalIgnoreCase))
|
||||
.ToList();
|
||||
var body = await AppsAnalyticsHelper.BuildAnalyticsResponseAsync(
|
||||
summaries, _sessionRepo, decodedAppName, days);
|
||||
summaries, _sessionRepo, decodedAppName, days, versionRegressions);
|
||||
|
||||
var response = req.CreateResponse(HttpStatusCode.OK);
|
||||
await response.WriteAsJsonAsync(body);
|
||||
|
||||
@@ -0,0 +1,208 @@
|
||||
using AutopilotMonitor.Functions.Functions.Apps;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Helpers;
|
||||
|
||||
/// <summary>One fired duration regression: an app whose newest version installs much slower than its predecessor (see <see cref="AppVersionRegressionRadar"/>).</summary>
|
||||
public sealed class AppVersionDurationRegressionFinding
|
||||
{
|
||||
public string TenantId { get; init; } = string.Empty;
|
||||
public string AppName { get; init; } = string.Empty;
|
||||
public string CurrentVersion { get; init; } = string.Empty;
|
||||
public string PreviousVersion { get; init; } = string.Empty;
|
||||
|
||||
public int CurrentMedianSeconds { get; init; }
|
||||
public int PreviousMedianSeconds { get; init; }
|
||||
public int CurrentMeasuredCount { get; init; }
|
||||
public int PreviousMeasuredCount { get; init; }
|
||||
|
||||
/// <summary>Current median ÷ previous median, one decimal. Never null — measured durations are ≥1s, so the previous median cannot be 0.</summary>
|
||||
public double Lift { get; init; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Deterministic, I/O-free core of the app-version duration regression radar. Detects
|
||||
/// "version X made this app's installs much slower" from the per-(session, app) install
|
||||
/// summaries: per (tenant, app), the newest version's MEDIAN measured install duration is
|
||||
/// compared against the previous version's median. Medians (nearest-rank, shared with the
|
||||
/// apps dashboard via <see cref="AppsAnalyticsHelper.Percentile"/>) rather than means —
|
||||
/// a single back-stamped straggler must not fire a fleet alert. A regression fires only
|
||||
/// when ALL hold:
|
||||
/// <list type="number">
|
||||
/// <item>both versions have ≥ 10 measured installs inside the horizon;</item>
|
||||
/// <item>current median ≥ 2× the previous version's median;</item>
|
||||
/// <item>absolute increase ≥ 300 s — a 20s→45s app is noise, not an incident.</item>
|
||||
/// </list>
|
||||
/// Measured population: Succeeded, non-skip terminal state, plausible observed duration
|
||||
/// (<see cref="MetricsMath.HasMeasuredDuration"/>), no AppId collision, non-empty
|
||||
/// AppVersion. Version ORDER comes from first-seen install time (min StartedAt) — version
|
||||
/// strings are never sorted lexically ("9.1" vs "2024.10" would lie). "Previous" is the
|
||||
/// version with the latest first-seen strictly before the current version's first-seen;
|
||||
/// with parallel ring rollouts that can be a concurrently-deployed version — accepted for
|
||||
/// v1, the episode key (app, current version) still caps it at one bell per version.
|
||||
/// </summary>
|
||||
public static class AppVersionRegressionRadar
|
||||
{
|
||||
/// <summary>Days of install summaries the radar loads and reasons over (mirrors the rule radar's 7+28 horizon).</summary>
|
||||
public const int HorizonDays = 35;
|
||||
public const int MinMeasuredInstalls = 10;
|
||||
public const double MinMedianLift = 2.0;
|
||||
public const int MinAbsoluteIncreaseSeconds = 300;
|
||||
|
||||
/// <summary>An active alert re-arms once the current median falls back under 1.5× the previous version's median.</summary>
|
||||
public const double ReArmLiftFactor = 1.5;
|
||||
|
||||
/// <summary>Per-version duration stats inside the horizon (see <see cref="ComputeVersionStats"/>).</summary>
|
||||
public sealed class VersionStats
|
||||
{
|
||||
public string Version { get; init; } = string.Empty;
|
||||
public DateTime FirstSeen { get; init; }
|
||||
public DateTime LastSeen { get; init; }
|
||||
public int MeasuredCount { get; init; }
|
||||
public int MedianSeconds { get; init; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Evaluates one tenant's install summaries (loaded for the last <see cref="HorizonDays"/>
|
||||
/// days). Returns at most one finding per app — the newest version vs. its predecessor;
|
||||
/// older version pairs are yesterday's news, not an alert. Deterministic ordering:
|
||||
/// lift descending, then app name ordinal.
|
||||
/// </summary>
|
||||
public static List<AppVersionDurationRegressionFinding> Evaluate(IReadOnlyList<AppInstallSummary> tenantSummaries)
|
||||
{
|
||||
var findings = new List<AppVersionDurationRegressionFinding>();
|
||||
foreach (var app in MeasuredAppGroups(tenantSummaries))
|
||||
{
|
||||
var stats = ComputeVersionStats(app.Value);
|
||||
var (current, previous) = SelectComparisonPair(stats);
|
||||
if (current == null || previous == null) continue;
|
||||
|
||||
if (current.MeasuredCount < MinMeasuredInstalls) continue;
|
||||
if (previous.MeasuredCount < MinMeasuredInstalls) continue;
|
||||
if (current.MedianSeconds < MinMedianLift * previous.MedianSeconds) continue;
|
||||
if (current.MedianSeconds - previous.MedianSeconds < MinAbsoluteIncreaseSeconds) continue;
|
||||
|
||||
findings.Add(new AppVersionDurationRegressionFinding
|
||||
{
|
||||
TenantId = app.Value[0].TenantId,
|
||||
AppName = app.Key,
|
||||
CurrentVersion = current.Version,
|
||||
PreviousVersion = previous.Version,
|
||||
CurrentMedianSeconds = current.MedianSeconds,
|
||||
PreviousMedianSeconds = previous.MedianSeconds,
|
||||
CurrentMeasuredCount = current.MeasuredCount,
|
||||
PreviousMeasuredCount = previous.MeasuredCount,
|
||||
Lift = Math.Round((double)current.MedianSeconds / previous.MedianSeconds, 1),
|
||||
});
|
||||
}
|
||||
return findings
|
||||
.OrderByDescending(f => f.Lift)
|
||||
.ThenBy(f => f.AppName, StringComparer.Ordinal)
|
||||
.ToList();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// True when an ACTIVE alert may re-arm: the alerted version has drained out of the
|
||||
/// horizon (fewer than <see cref="MinMeasuredInstalls"/> measured installs left — it was
|
||||
/// superseded or pulled), or its recomputed median fell back under
|
||||
/// <see cref="ReArmLiftFactor"/>× the previous version's recomputed median. When the
|
||||
/// previous version itself has drained there is no comparison basis anymore — the
|
||||
/// episode is kept until the current version drains (fires-stopped-style re-arm only).
|
||||
/// </summary>
|
||||
public static bool ShouldReArm(IReadOnlyList<AppInstallSummary> tenantSummaries, AppVersionRegressionAlert alert)
|
||||
{
|
||||
var stats = ComputeVersionStatsForApp(tenantSummaries, alert.AppName);
|
||||
var current = stats.FirstOrDefault(s => string.Equals(s.Version, alert.CurrentVersion, StringComparison.Ordinal));
|
||||
if (current == null || current.MeasuredCount < MinMeasuredInstalls) return true;
|
||||
|
||||
var previous = stats.FirstOrDefault(s => string.Equals(s.Version, alert.PreviousVersion, StringComparison.Ordinal));
|
||||
if (previous == null || previous.MeasuredCount < MinMeasuredInstalls) return false;
|
||||
|
||||
return current.MedianSeconds < ReArmLiftFactor * previous.MedianSeconds;
|
||||
}
|
||||
|
||||
/// <summary>Per-version stats for one app (measured rows only), or empty when the app has none.</summary>
|
||||
public static List<VersionStats> ComputeVersionStatsForApp(IReadOnlyList<AppInstallSummary> tenantSummaries, string appName)
|
||||
{
|
||||
foreach (var app in MeasuredAppGroups(tenantSummaries))
|
||||
{
|
||||
if (string.Equals(app.Key, appName, StringComparison.OrdinalIgnoreCase))
|
||||
return ComputeVersionStats(app.Value);
|
||||
}
|
||||
return new List<VersionStats>();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Groups the MEASURED rows by app name: succeeded, non-skip, plausible duration,
|
||||
/// no AppId collision (a second app's outcomes must not fire this app's alert),
|
||||
/// non-empty version. Rate/failure signals are deliberately out of scope — this
|
||||
/// radar alerts on duration only.
|
||||
/// </summary>
|
||||
internal static Dictionary<string, List<AppInstallSummary>> MeasuredAppGroups(IReadOnlyList<AppInstallSummary> summaries)
|
||||
{
|
||||
var groups = new Dictionary<string, List<AppInstallSummary>>(StringComparer.OrdinalIgnoreCase);
|
||||
foreach (var summary in summaries)
|
||||
{
|
||||
if (string.IsNullOrEmpty(summary.AppName)) continue;
|
||||
if (string.IsNullOrEmpty(summary.AppVersion)) continue;
|
||||
if (summary.AppIdCollision) continue;
|
||||
if (summary.Status != "Succeeded") continue;
|
||||
if (MetricsMath.IsSkipTerminalState(summary)) continue;
|
||||
if (!MetricsMath.HasMeasuredDuration(summary)) continue;
|
||||
|
||||
if (!groups.TryGetValue(summary.AppName, out var list))
|
||||
{
|
||||
list = new List<AppInstallSummary>();
|
||||
groups[summary.AppName] = list;
|
||||
}
|
||||
list.Add(summary);
|
||||
}
|
||||
return groups;
|
||||
}
|
||||
|
||||
/// <summary>Per-version first/last-seen and nearest-rank median over one app's measured rows.</summary>
|
||||
internal static List<VersionStats> ComputeVersionStats(IReadOnlyList<AppInstallSummary> measuredAppRows)
|
||||
{
|
||||
return measuredAppRows
|
||||
.GroupBy(s => s.AppVersion, StringComparer.Ordinal)
|
||||
.Select(g =>
|
||||
{
|
||||
var durations = g.Select(s => s.DurationSeconds).ToList();
|
||||
return new VersionStats
|
||||
{
|
||||
Version = g.Key,
|
||||
FirstSeen = g.Min(s => s.StartedAt),
|
||||
LastSeen = g.Max(s => s.StartedAt),
|
||||
MeasuredCount = durations.Count,
|
||||
MedianSeconds = AppsAnalyticsHelper.Percentile(durations, 0.50),
|
||||
};
|
||||
})
|
||||
.ToList();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Current = the version of the most recent measured install (ties broken by later
|
||||
/// first-seen, then version ordinal — deterministic); previous = the version with the
|
||||
/// latest first-seen strictly before the current version's first-seen (same tie-break).
|
||||
/// Either is null when the app has fewer than two versions in the horizon.
|
||||
/// </summary>
|
||||
internal static (VersionStats? Current, VersionStats? Previous) SelectComparisonPair(IReadOnlyList<VersionStats> stats)
|
||||
{
|
||||
if (stats.Count < 2) return (null, null);
|
||||
|
||||
var current = stats
|
||||
.OrderByDescending(s => s.LastSeen)
|
||||
.ThenByDescending(s => s.FirstSeen)
|
||||
.ThenByDescending(s => s.Version, StringComparer.Ordinal)
|
||||
.First();
|
||||
|
||||
var previous = stats
|
||||
.Where(s => s.FirstSeen < current.FirstSeen)
|
||||
.OrderByDescending(s => s.FirstSeen)
|
||||
.ThenByDescending(s => s.LastSeen)
|
||||
.ThenByDescending(s => s.Version, StringComparer.Ordinal)
|
||||
.FirstOrDefault();
|
||||
|
||||
return (current, previous);
|
||||
}
|
||||
}
|
||||
+212
@@ -0,0 +1,212 @@
|
||||
using System;
|
||||
using System.Diagnostics;
|
||||
using System.Linq;
|
||||
using System.Threading.Tasks;
|
||||
using AutopilotMonitor.Functions.Helpers;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Services
|
||||
{
|
||||
/// <summary>
|
||||
/// Partial: app-version duration regression radar (sibling of the F3 rule radar).
|
||||
/// </summary>
|
||||
public partial class MaintenanceService
|
||||
{
|
||||
/// <summary>App setting kill switch: set to "true" to skip the radar entirely. Fail-open — the radar only notifies, it never mutates data.</summary>
|
||||
internal const string AppVersionRadarKillSwitchSetting = "AppVersionRegressionRadarDisabled";
|
||||
|
||||
/// <summary>
|
||||
/// Detects per-tenant apps whose newest version's median install duration regressed
|
||||
/// (≥2× and ≥5 min over the previous version — <see cref="AppVersionRegressionRadar"/>)
|
||||
/// and reconciles the alert episodes in the notification tracker:
|
||||
/// new finding → tracker row + tenant bell + AppVersionDurationRegression ops event,
|
||||
/// exactly once per (app, version) episode; still-regressed → numbers refreshed
|
||||
/// (FirstNotifiedAt untouched); no longer regressed → re-arm check (version drained,
|
||||
/// or median back under 1.5×) deletes the row so a future regression rings again.
|
||||
/// The tracker's 30d retention sweep re-arms long-burning episodes by design.
|
||||
/// <para>
|
||||
/// One cross-tenant install-summary read over the trailing
|
||||
/// <see cref="AppVersionRegressionRadar.HorizonDays"/> days (the same query the global
|
||||
/// apps dashboard runs), then grouped by tenant — mirrors the rule radar's enumeration.
|
||||
/// Idempotent per pass via the tracker dedup. Fail-soft per tenant and overall.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
private async Task RunAppVersionRegressionRadarAsync()
|
||||
{
|
||||
try
|
||||
{
|
||||
if (string.Equals(_configuration[AppVersionRadarKillSwitchSetting], "true", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
_logger.LogInformation("App-version regression radar skipped ({Setting}=true)", AppVersionRadarKillSwitchSetting);
|
||||
return;
|
||||
}
|
||||
|
||||
var sw = Stopwatch.StartNew();
|
||||
var sinceUtc = DateTime.UtcNow.AddDays(-AppVersionRegressionRadar.HorizonDays);
|
||||
var summaries = await _metricsRepo.GetAppsDashboardSummariesAsync(sinceUtc, tenantId: null);
|
||||
|
||||
var byTenant = summaries
|
||||
.Where(s => !string.IsNullOrEmpty(s.TenantId))
|
||||
.GroupBy(s => s.TenantId, StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
int fired = 0, refreshed = 0, rearmed = 0, tenants = 0;
|
||||
foreach (var tenantGroup in byTenant)
|
||||
{
|
||||
tenants++;
|
||||
try
|
||||
{
|
||||
var (f, r, a) = await EvaluateTenantAppVersionRegressionsAsync(tenantGroup.Key, tenantGroup.ToList());
|
||||
fired += f;
|
||||
refreshed += r;
|
||||
rearmed += a;
|
||||
}
|
||||
catch (Exception tenantEx)
|
||||
{
|
||||
_logger.LogWarning(tenantEx, "App-version regression radar failed for tenant {TenantId} (non-fatal)", tenantGroup.Key);
|
||||
}
|
||||
}
|
||||
|
||||
sw.Stop();
|
||||
_logger.LogInformation(
|
||||
"App-version regression radar: {Tenants} tenants over {Rows} install summaries — {Fired} fired, {Refreshed} refreshed, {Rearmed} re-armed in {Ms}ms",
|
||||
tenants, summaries.Count, fired, refreshed, rearmed, sw.ElapsedMilliseconds);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "App-version regression radar failed (non-fatal)");
|
||||
}
|
||||
}
|
||||
|
||||
private async Task<(int Fired, int Refreshed, int Rearmed)> EvaluateTenantAppVersionRegressionsAsync(
|
||||
string tenantId, System.Collections.Generic.List<AppInstallSummary> tenantSummaries)
|
||||
{
|
||||
var findings = AppVersionRegressionRadar.Evaluate(tenantSummaries);
|
||||
var active = await _hardwareRejectionTracker.GetAppVersionRegressionsAsync(tenantId);
|
||||
var activeByKey = active.ToDictionary(
|
||||
a => AppVersionEpisodeKey(a.AppName, a.CurrentVersion), StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
int fired = 0, refreshed = 0, rearmed = 0;
|
||||
var firedKeys = new System.Collections.Generic.HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
foreach (var finding in findings)
|
||||
{
|
||||
var key = AppVersionEpisodeKey(finding.AppName, finding.CurrentVersion);
|
||||
firedKeys.Add(key);
|
||||
|
||||
if (activeByKey.TryGetValue(key, out var existing))
|
||||
{
|
||||
// Episode still burning: refresh numbers for the versionRegressions[] block;
|
||||
// FirstNotifiedAt never moves (retention re-arm counts from the bell).
|
||||
await _hardwareRejectionTracker.RefreshAppVersionRegressionAsync(
|
||||
tenantId, BuildAppVersionAlert(finding, existing.FirstNotifiedAt));
|
||||
refreshed++;
|
||||
continue;
|
||||
}
|
||||
|
||||
var alert = BuildAppVersionAlert(finding, DateTime.UtcNow);
|
||||
if (!await _hardwareRejectionTracker.TryRegisterAppVersionRegressionAsync(tenantId, alert))
|
||||
continue; // a concurrent pass won the episode — its bell suffices
|
||||
|
||||
fired++;
|
||||
await _tenantNotificationService.CreateNotificationAsync(
|
||||
tenantId,
|
||||
type: "app_version_duration_regression",
|
||||
title: $"Install duration regressed: {finding.AppName}",
|
||||
message: BuildAppVersionRegressionMessage(finding),
|
||||
// Canonical appDetailUrl shape (lib/routes.ts) with the radar's horizon window.
|
||||
href: $"/apps/detail?name={Uri.EscapeDataString(finding.AppName)}&days={AppVersionRegressionRadar.HorizonDays}");
|
||||
await _opsEventService.RecordAppVersionDurationRegressionAsync(
|
||||
tenantId, finding.AppName, finding.CurrentVersion, finding.PreviousVersion,
|
||||
finding.CurrentMedianSeconds, finding.PreviousMedianSeconds,
|
||||
finding.CurrentMeasuredCount, finding.PreviousMeasuredCount, finding.Lift);
|
||||
}
|
||||
|
||||
foreach (var alert in active)
|
||||
{
|
||||
if (firedKeys.Contains(AppVersionEpisodeKey(alert.AppName, alert.CurrentVersion)))
|
||||
continue;
|
||||
|
||||
if (AppVersionRegressionRadar.ShouldReArm(tenantSummaries, alert))
|
||||
{
|
||||
await _hardwareRejectionTracker.DeleteAppVersionRegressionAsync(tenantId, alert.AppName, alert.CurrentVersion);
|
||||
rearmed++;
|
||||
}
|
||||
else
|
||||
{
|
||||
// Elevated but below the fire gates (between 1.5× and 2×, or the absolute
|
||||
// delta shrank): keep the episode, refresh the numbers so the
|
||||
// versionRegressions[] block stays honest.
|
||||
var refreshedAlert = TryBuildOngoingAppVersionAlert(tenantSummaries, alert);
|
||||
if (refreshedAlert != null)
|
||||
{
|
||||
await _hardwareRejectionTracker.RefreshAppVersionRegressionAsync(tenantId, refreshedAlert);
|
||||
refreshed++;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return (fired, refreshed, rearmed);
|
||||
}
|
||||
|
||||
private static string AppVersionEpisodeKey(string appName, string version)
|
||||
=> $"{appName}\n{version}";
|
||||
|
||||
private static AppVersionRegressionAlert BuildAppVersionAlert(
|
||||
AppVersionDurationRegressionFinding finding, DateTime firstNotifiedAt)
|
||||
=> new()
|
||||
{
|
||||
TenantId = finding.TenantId,
|
||||
AppName = finding.AppName,
|
||||
CurrentVersion = finding.CurrentVersion,
|
||||
PreviousVersion = finding.PreviousVersion,
|
||||
CurrentMedianSeconds = finding.CurrentMedianSeconds,
|
||||
PreviousMedianSeconds = finding.PreviousMedianSeconds,
|
||||
CurrentMeasuredCount = finding.CurrentMeasuredCount,
|
||||
PreviousMeasuredCount = finding.PreviousMeasuredCount,
|
||||
Lift = finding.Lift,
|
||||
FirstNotifiedAt = firstNotifiedAt,
|
||||
LastEvaluatedAt = DateTime.UtcNow,
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
/// Refresh payload for a still-open episode below the fire gates: recomputed medians
|
||||
/// against the SAME version pair, everything identifying carried over. Null when either
|
||||
/// version has left the horizon — the stale numbers then remain until re-arm/retention.
|
||||
/// </summary>
|
||||
private static AppVersionRegressionAlert? TryBuildOngoingAppVersionAlert(
|
||||
System.Collections.Generic.IReadOnlyList<AppInstallSummary> tenantSummaries, AppVersionRegressionAlert existing)
|
||||
{
|
||||
var stats = AppVersionRegressionRadar.ComputeVersionStatsForApp(tenantSummaries, existing.AppName);
|
||||
var current = stats.FirstOrDefault(s => string.Equals(s.Version, existing.CurrentVersion, StringComparison.Ordinal));
|
||||
var previous = stats.FirstOrDefault(s => string.Equals(s.Version, existing.PreviousVersion, StringComparison.Ordinal));
|
||||
if (current == null || previous == null || previous.MedianSeconds <= 0)
|
||||
return null;
|
||||
|
||||
return new AppVersionRegressionAlert
|
||||
{
|
||||
TenantId = existing.TenantId,
|
||||
AppName = existing.AppName,
|
||||
CurrentVersion = existing.CurrentVersion,
|
||||
PreviousVersion = existing.PreviousVersion,
|
||||
CurrentMedianSeconds = current.MedianSeconds,
|
||||
PreviousMedianSeconds = previous.MedianSeconds,
|
||||
CurrentMeasuredCount = current.MeasuredCount,
|
||||
PreviousMeasuredCount = previous.MeasuredCount,
|
||||
Lift = Math.Round((double)current.MedianSeconds / previous.MedianSeconds, 1),
|
||||
FirstNotifiedAt = existing.FirstNotifiedAt,
|
||||
LastEvaluatedAt = DateTime.UtcNow,
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>Bell message with the full numbers (the admin can verify without a portal round-trip). Minutes, one decimal.</summary>
|
||||
internal static string BuildAppVersionRegressionMessage(AppVersionDurationRegressionFinding finding)
|
||||
{
|
||||
var fromMin = Math.Round(finding.PreviousMedianSeconds / 60.0, 1);
|
||||
var toMin = Math.Round(finding.CurrentMedianSeconds / 60.0, 1);
|
||||
return $"Median install duration rose from {fromMin} to {toMin} min after version {finding.CurrentVersion} " +
|
||||
$"({finding.CurrentMeasuredCount} measured installs vs {finding.PreviousMeasuredCount} on version {finding.PreviousVersion}) " +
|
||||
$"— lift {finding.Lift}x over the last {AppVersionRegressionRadar.HorizonDays} days.";
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -148,6 +148,9 @@ namespace AutopilotMonitor.Functions.Services
|
||||
// aggregation so the window rows are fresh. Anchored on YESTERDAY — whole
|
||||
// days only, a partial today would understate the window rate. Fail-soft.
|
||||
await RunRuleRegressionRadarAsync(DateTime.UtcNow.Date.AddDays(-1));
|
||||
// App-version duration regression radar: same episode/tracker pattern over the
|
||||
// install summaries (trailing 35d horizon loaded internally). Fail-soft.
|
||||
await RunAppVersionRegressionRadarAsync();
|
||||
// F1 PR2: rolling 30d breakdown backfill + daily attribution aggregates. Owns
|
||||
// its own window (NOT the snapshot-gated catch-up above) so late-terminating
|
||||
// sessions still reach their StartedAt-date's aggregate. Fail-soft internally.
|
||||
@@ -222,6 +225,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
// Timer-path parity: radar re-runs are idempotent (tracker dedup), so a manual
|
||||
// aggregation also re-evaluates regressions anchored on the aggregated date.
|
||||
await RunRuleRegressionRadarAsync(dateToAggregate);
|
||||
await RunAppVersionRegressionRadarAsync();
|
||||
|
||||
// Timer-path parity: manual maintenance also refreshes the attribution
|
||||
// breakdowns + daily aggregates and the device-history/FTR rollups
|
||||
|
||||
@@ -542,6 +542,35 @@ namespace AutopilotMonitor.Functions.Services
|
||||
lift,
|
||||
});
|
||||
|
||||
/// <summary>
|
||||
/// App-version duration regression radar: an app's newest version installs with a median
|
||||
/// duration ≥2× (and ≥5 min absolute) over the previous version's median. Fired ONCE per
|
||||
/// (app, version) episode (the notification-tracker row is the dedup); the message
|
||||
/// carries the full numbers so the event is verifiable without a portal round-trip.
|
||||
/// Dual-registered in the web OPS_EVENT_TYPES catalog
|
||||
/// (memory: feedback_ops_event_types_dual_register).
|
||||
/// </summary>
|
||||
public Task RecordAppVersionDurationRegressionAsync(
|
||||
string tenantId, string appName, string currentVersion, string previousVersion,
|
||||
int currentMedianSeconds, int previousMedianSeconds,
|
||||
int currentMeasuredCount, int previousMeasuredCount, double lift)
|
||||
=> WriteAsync(OpsEventCategory.Tenant, "AppVersionDurationRegression", OpsEventSeverity.Warning,
|
||||
$"App '{appName}' median install duration rose from {Math.Round(previousMedianSeconds / 60.0, 1)} to " +
|
||||
$"{Math.Round(currentMedianSeconds / 60.0, 1)} min after version {currentVersion} " +
|
||||
$"({currentMeasuredCount} measured installs vs {previousMeasuredCount} on {previousVersion}) — lift {lift}x",
|
||||
tenantId, "System.Maintenance",
|
||||
new
|
||||
{
|
||||
appName,
|
||||
currentVersion,
|
||||
previousVersion,
|
||||
currentMedianSeconds,
|
||||
previousMedianSeconds,
|
||||
currentMeasuredCount,
|
||||
previousMeasuredCount,
|
||||
lift,
|
||||
});
|
||||
|
||||
// ── Agent ──────────────────────────────────────────────────────────────
|
||||
|
||||
public Task RecordSessionTimeoutsAsync(string tenantId, int sessionCount, int timeoutHours)
|
||||
|
||||
@@ -33,6 +33,8 @@ public static class TenantNotificationAudienceCatalog
|
||||
// F3 regression radar: rule-tuning is an admin concern (the alert deep-links into the
|
||||
// analyze-rules page, which members cannot act on).
|
||||
["rule_frequency_regression"] = NotificationAudience.Admin,
|
||||
// App-version duration regression radar: packaging/rollout is an admin concern.
|
||||
["app_version_duration_regression"] = NotificationAudience.Admin,
|
||||
};
|
||||
|
||||
public static NotificationAudience Resolve(string? type)
|
||||
|
||||
+28
-3
@@ -8,9 +8,10 @@ namespace AutopilotMonitor.Shared.DataAccess
|
||||
/// <summary>
|
||||
/// Notification dedup store (one table, RowKey-prefixed key spaces). Subjects:
|
||||
/// hardware rejections per (tenant, manufacturer, model), TPM PSS incompatibilities per
|
||||
/// (tenant, serial number) — both lifetime insert-once — and F3 rule-frequency
|
||||
/// regressions per (tenant, ruleId), whose rows additionally carry the alert payload,
|
||||
/// are refreshed while the episode stays active, and are deleted when the rate re-arms.
|
||||
/// (tenant, serial number) — both lifetime insert-once — F3 rule-frequency
|
||||
/// regressions per (tenant, ruleId), and app-version duration regressions per
|
||||
/// (tenant, app, version). The regression key spaces additionally carry the alert
|
||||
/// payload, are refreshed while the episode stays active, and are deleted on re-arm.
|
||||
/// </summary>
|
||||
public interface IHardwareRejectionNotificationTracker
|
||||
{
|
||||
@@ -54,6 +55,30 @@ namespace AutopilotMonitor.Shared.DataAccess
|
||||
/// <summary>Active regression episodes of one tenant (RowKey-prefix scan; empty on failure — fail-soft reads).</summary>
|
||||
Task<List<RuleRegressionAlert>> GetRuleRegressionsAsync(string tenantId);
|
||||
|
||||
// --- App-version duration regressions (RowKey "appversionregression|{app}|{version}") ---
|
||||
|
||||
/// <summary>
|
||||
/// Atomically opens a duration-regression episode for (tenantId, alert.AppName,
|
||||
/// alert.CurrentVersion). Returns true when this pass owns the episode (caller fires
|
||||
/// bell + ops event exactly once), false when an episode is already active or on
|
||||
/// failure (fail-closed — never double-fire).
|
||||
/// </summary>
|
||||
Task<bool> TryRegisterAppVersionRegressionAsync(string tenantId, AppVersionRegressionAlert alert);
|
||||
|
||||
/// <summary>
|
||||
/// Refreshes an active episode's numbers (medians, counts, LastEvaluatedAt) so the
|
||||
/// versionRegressions[] block stays current. FirstNotifiedAt is carried from the alert
|
||||
/// unchanged — the retention sweep re-arms on the ORIGINAL notification age.
|
||||
/// Fail-soft; never fires notifications.
|
||||
/// </summary>
|
||||
Task RefreshAppVersionRegressionAsync(string tenantId, AppVersionRegressionAlert alert);
|
||||
|
||||
/// <summary>Closes an episode (median re-armed or the version drained out of the horizon). 404-tolerant.</summary>
|
||||
Task DeleteAppVersionRegressionAsync(string tenantId, string appName, string currentVersion);
|
||||
|
||||
/// <summary>Active app-version regression episodes of one tenant (RowKey-prefix scan; empty on failure — fail-soft reads).</summary>
|
||||
Task<List<AppVersionRegressionAlert>> GetAppVersionRegressionsAsync(string tenantId);
|
||||
|
||||
/// <summary>
|
||||
/// Retention cleanup: deletes tracker rows whose FirstNotifiedAt is older than
|
||||
/// <paramref name="cutoffUtc"/>. Every key space lives in one table, so pruning re-arms
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
using System;
|
||||
|
||||
namespace AutopilotMonitor.Shared.Models
|
||||
{
|
||||
/// <summary>
|
||||
/// One ACTIVE app-version duration regression: an app whose newest version's median
|
||||
/// install duration rose ≥2× (and ≥5 minutes absolute) over the previous version's
|
||||
/// median, both sides with enough measured installs. Persisted as the
|
||||
/// <c>appversionregression|{app}|{version}</c> keyspace of the notification tracker
|
||||
/// table — the row IS the dedup (one bell per episode), and the
|
||||
/// <c>versionRegressions[]</c> payload on the app-analytics response. Deleted when the
|
||||
/// episode re-arms (median falls back under 1.5× or the version drains out of the
|
||||
/// horizon) or by the tracker's 30-day retention sweep. Numbers refresh on every radar
|
||||
/// pass while the episode stays active; <see cref="FirstNotifiedAt"/> never moves.
|
||||
/// </summary>
|
||||
public class AppVersionRegressionAlert
|
||||
{
|
||||
public string TenantId { get; set; } = string.Empty;
|
||||
public string AppName { get; set; } = string.Empty;
|
||||
|
||||
/// <summary>The regressed (newer) version — episode key together with the app.</summary>
|
||||
public string CurrentVersion { get; set; } = string.Empty;
|
||||
|
||||
/// <summary>The comparison version: latest first-seen strictly before the current version's first-seen.</summary>
|
||||
public string PreviousVersion { get; set; } = string.Empty;
|
||||
|
||||
public int CurrentMedianSeconds { get; set; }
|
||||
public int PreviousMedianSeconds { get; set; }
|
||||
|
||||
/// <summary>Measured installs backing each median (gate: both ≥10).</summary>
|
||||
public int CurrentMeasuredCount { get; set; }
|
||||
public int PreviousMeasuredCount { get; set; }
|
||||
|
||||
/// <summary>Current median ÷ previous median (gate: ≥2.0; previous median is never 0 — measured durations are ≥1s).</summary>
|
||||
public double Lift { get; set; }
|
||||
|
||||
/// <summary>When the episode first fired (bell + ops event moment). Never moves on refresh; drives the 30d retention re-arm.</summary>
|
||||
public DateTime FirstNotifiedAt { get; set; }
|
||||
|
||||
/// <summary>Last radar pass that re-confirmed/refreshed this episode.</summary>
|
||||
public DateTime LastEvaluatedAt { get; set; }
|
||||
}
|
||||
}
|
||||
@@ -108,6 +108,11 @@ const OPS_EVENT_TYPES: Record<string, string[]> = {
|
||||
// (Wilson-separated). Warning-tier, once per episode (tracker-deduped). Dual-register
|
||||
// per memory feedback_ops_event_types_dual_register.
|
||||
"RuleFrequencyRegression",
|
||||
// App-version duration regression radar: an app's newest version installs with a median
|
||||
// duration ≥2× (and ≥5 min absolute) over the previous version. Warning-tier, once per
|
||||
// (app, version) episode (tracker-deduped). Dual-register per memory
|
||||
// feedback_ops_event_types_dual_register.
|
||||
"AppVersionDurationRegression",
|
||||
// Auto-activation audit: the tenant-auto-approve worker activated a new signup
|
||||
// (AutoApproveNewTenants enabled). Info-tier. Dual-register per memory
|
||||
// feedback_ops_event_types_dual_register.
|
||||
|
||||
@@ -51,6 +51,25 @@ interface VersionRow {
|
||||
installs: number;
|
||||
failed: number;
|
||||
failureRate: number;
|
||||
/** Succeeded installs with a plausible observed duration — the basis of the median/p95 below. */
|
||||
measuredInstalls: number;
|
||||
medianDurationSeconds: number;
|
||||
p95DurationSeconds: number;
|
||||
}
|
||||
|
||||
/** Active duration-regression episode from the radar (tracker row, camelCase on the wire). */
|
||||
interface AppVersionRegressionRow {
|
||||
tenantId: string;
|
||||
appName: string;
|
||||
currentVersion: string;
|
||||
previousVersion: string;
|
||||
currentMedianSeconds: number;
|
||||
previousMedianSeconds: number;
|
||||
currentMeasuredCount: number;
|
||||
previousMeasuredCount: number;
|
||||
lift: number;
|
||||
firstNotifiedAt: string;
|
||||
lastEvaluatedAt: string;
|
||||
}
|
||||
|
||||
interface PhaseRow {
|
||||
@@ -102,6 +121,7 @@ interface AnalyticsResponse {
|
||||
topFailureCodes: FailureCodeRow[];
|
||||
detectionLiesCount: number;
|
||||
deviceModelBreakdown: DeviceModelRow[];
|
||||
versionRegressions: AppVersionRegressionRow[];
|
||||
}
|
||||
|
||||
interface SessionRow {
|
||||
@@ -527,6 +547,23 @@ function AppDetailContent() {
|
||||
</Card>
|
||||
</div>
|
||||
|
||||
{/* Duration regression banner: active radar episodes for this app (bell-deduped
|
||||
server-side; this is the durable badge while the episode burns). */}
|
||||
{(analytics.versionRegressions ?? []).length > 0 && (
|
||||
<div className="rounded-lg border border-amber-300 bg-amber-50 px-4 py-3 space-y-1">
|
||||
{analytics.versionRegressions.map((reg) => (
|
||||
<p key={`${reg.appName}|${reg.currentVersion}`} className="text-sm text-amber-800">
|
||||
<span className="font-semibold">↑ Duration regression:</span>{" "}
|
||||
median install duration rose from {(reg.previousMedianSeconds / 60).toFixed(1)} to{" "}
|
||||
{(reg.currentMedianSeconds / 60).toFixed(1)} min after version{" "}
|
||||
<span className="font-mono">{reg.currentVersion}</span>{" "}
|
||||
({reg.currentMeasuredCount} measured installs vs {reg.previousMeasuredCount} on{" "}
|
||||
<span className="font-mono">{reg.previousVersion}</span> — lift {reg.lift}x)
|
||||
</p>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Charts row 2: version breakdown + phase breakdown.
|
||||
Each card is hidden when empty so the row collapses gracefully:
|
||||
- Version: will appear once new agents emit AppVersion data
|
||||
@@ -547,6 +584,17 @@ function AppDetailContent() {
|
||||
/>
|
||||
</Card>
|
||||
)}
|
||||
{analytics.versionBreakdown.some((v) => v.measuredInstalls > 0) && (
|
||||
<Card title="Median Install Duration by Version">
|
||||
<AppBarChart
|
||||
data={analytics.versionBreakdown.filter((v) => v.measuredInstalls > 0) as unknown as Array<Record<string, unknown>>}
|
||||
categoryKey="appVersion"
|
||||
valueKey="medianDurationSeconds"
|
||||
valueFormatter={(v) => formatDuration(Number(v))}
|
||||
barColor={chartColors.primary}
|
||||
/>
|
||||
</Card>
|
||||
)}
|
||||
{analytics.installerPhaseBreakdown.length > 0 && (
|
||||
<Card title="Installer Phase (failures only)">
|
||||
<AppBarChart
|
||||
|
||||
Reference in New Issue
Block a user