fix(backend): CMTrace skew tripwire ignores backlog-replay bursts via per-sample grid conformity

CmTraceTimeSkewRegression fired 7x in its first soak week; every checked case
was a relaunched agent re-tailing the IME log and uploading one burst holding
most of the session's IME samples with a continuum of ages whose median hit the
15-minute grid by chance. The tripwire now also requires >= 80 % of the IME
deltas (relative to the other side's median) to individually sit within 2 min of
some grid multiple; era-mixing regressions still fire. Fraction is carried in the
ops event message and details.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Oliver Kieselbach
2026-08-28 16:42:51 +02:00
parent 0eaa3527f9
commit bf156cf862
5 changed files with 96 additions and 2 deletions
+5 -1
View File
@@ -94,7 +94,11 @@ batch, the counter reconcile's existing event-partition scan additionally collec
rest (`CmTraceSkewTripwire`, hooked in `EventIngestProcessor`). If
`median(Δ_IME) − median(Δ_other)` is a clean 15-minute multiple (residual < 2 min — the
same constants as the anchoring grid guard above) with ≥ 20 samples over ≥ 3 distinct
upload batches per side, a `CmTraceTimeSkewRegression` ops event fires (category Agent,
upload batches per side, and ≥ 80 % of the individual IME deltas (relative to the other
side's median) themselves sit within 2 min of some grid multiple — added after the 2026-08-28
soak, when relaunched agents re-tailing the IME log produced one burst holding most of a
session's IME samples with a continuum of ages whose median hit the grid by chance (7 false
fires in a week, 0 true) — a `CmTraceTimeSkewRegression` ops event fires (category Agent,
warning), carrying all numbers plus the session's `sourceOffsetOrigin` histogram.
Bias-dominated sessions are suppressed (writer-declared offsets cannot be an anchoring
regression); `measuredWriterOffsetMinutes` is never consulted (sticky after era
+4
View File
@@ -1,5 +1,9 @@
# Log
## 2026-08-28
* **Update**: `CmTraceSkewTripwire` - the backend regression tripwire additionally requires >= 80 % of the individual IME deltas (relative to the other side's median) to sit within 2 min of some 15-minute grid multiple before `CmTraceTimeSkewRegression` fires; the ops event and its details now carry that fraction. Motivation: the first soak week produced 7 fires, all backlog replays - a relaunched agent re-tailing the IME log uploads one burst holding most of the session's IME samples with a continuum of ages whose median hit the grid by chance (session d832560a: 508 of 627 samples, 46...733 min, median 91). Era-mixing regressions (every line on the grid) still fire. Doc: `docs/agent/cmtrace-time-resolution.md`.
## 2026-08-27 (2)
* **Update**: `SecurityValidator` / `DeviceAssociationValidator` - Microsoft released Windows Autopilot device association on 2026-08-25 (learn.microsoft.com/autopilot/device-preparation/device-association/overview: TPM-backed UEFI marker, device-targeted device preparation policies, associated devices are marked corporate-owned by Intune with no corporate identifier upload). The DevPrep association lookup therefore leaves shadow mode and becomes the third accepting method of the device-validation gate, evaluated inline after the Autopilot and Corporate Identifier lookups and before the Cloud PC fallback; a match sets `ValidatedBy = DeviceAssociation`, a miss joins the combined 403 detail, a transient failure the 503 Retry-After - the same contract as its siblings, so enabling it can only widen acceptance, never reject a device the other methods would have taken. `ValidateDeviceAssociation` counts toward "at least one validation enabled" and is no longer GA-only anywhere (`UpdateTenantConfigurationFunction` revert, `TenantConfigPatchService.GaOnlyFields`, the Global-Admin gate on the settings toggle); the toggle sits under Device Preparation next to Corporate Identifier and follows the shared consent rule (first serial-based validation carries the consent, any further one is a plain persisted toggle). Known gap carried over from the shadow phase: the `/beta/deviceManagement/tenantAssociatedDevices` endpoint is still undocumented in Graph reference and every production lookup so far answered `matched=False` for Autopilot v1 devices, i.e. the positive match path is unproven in the field - first associated enrollment must be checked for `ValidatedBy = DeviceAssociation` on the session row. Landing hero carries a release pill linking to the customer docs; dashboard tip added. Backend before web; customer docs updated in `autopilotmonitor-docs`.
@@ -119,6 +119,53 @@ public class CmTraceSkewTripwireTests
Assert.Null(CmTraceSkewTripwire.Evaluate(scan));
}
[Fact]
public void Silent_OnBacklogReplayBurstWhoseMedianLandsOnGrid()
{
// Field case 2026-08-28 (session d832560a): the agent died on auth failures, the relaunch
// re-tailed the IME log and uploaded 508 lines in one burst with a continuum of ages
// (46…733 min, quartiles 52 / 92 / 100) — median 91 min, i.e. 6×15 by pure chance.
// The other side was live. The per-sample grid-conformity guard must keep this silent.
var scan = MakeScan(imeDeltaMinutes: 0.1, otherDeltaMinutes: 0.1, imeSamples: 100, imeBatches: 80);
void AddRamp(double from, double to, int count)
{
for (int i = 0; i < count; i++)
scan.ImeDeltaMinutes.Add(from + i * (to - from) / count);
}
AddRamp(46.0, 88.0, 125);
AddRamp(88.0, 100.0, 250);
AddRamp(100.0, 733.0, 125);
var medianIme = scan.ImeDeltaMinutes.OrderBy(v => v).ElementAt(scan.ImeDeltaMinutes.Count / 2);
Assert.InRange(medianIme, 88.0, 92.0); // sanity: the median really does sit near 6×15
Assert.Null(CmTraceSkewTripwire.Evaluate(scan));
Assert.True(CmTraceSkewTripwire.ComputeGridConformantFraction(scan.ImeDeltaMinutes, 0.1) < 0.5);
}
[Fact]
public void Fires_OnTwoWriterErasBothOnGrid()
{
// e9753578 shape: one file mixing PDT (−420) and CEST (+120) lines relative to the agent
// zone. Every line is individually on the grid, so the conformity guard must NOT mask it.
var scan = MakeScan(imeDeltaMinutes: -420.0, otherDeltaMinutes: 0.0, imeSamples: 65);
for (int i = 0; i < 35; i++)
scan.ImeDeltaMinutes.Add(120.0);
var result = CmTraceSkewTripwire.Evaluate(scan);
Assert.NotNull(result);
Assert.Equal(-28, result!.GridSteps);
Assert.Equal(1.0, result.GridConformantFraction, 3);
}
[Fact]
public void GridConformantFraction_CountsZeroAndAnyMultiple()
{
var deltas = new List<double> { 0.5, 15.9, -30.1, 7.5, 37.5 }; // last two are off-grid
Assert.Equal(0.6, CmTraceSkewTripwire.ComputeGridConformantFraction(deltas, 0.0), 3);
Assert.Equal(0, CmTraceSkewTripwire.ComputeGridConformantFraction(new List<double>(), 0.0));
}
// ── Bias suppression predicate ─────────────────────────────────────────
[Fact]
@@ -15,6 +15,12 @@ namespace AutopilotMonitor.Functions.Services
/// mis-conversion shifts only the IME side and lands on the 15-minute offset grid.
/// Replayed old log lines produce large OFF-grid deltas — the grid-residual test is what
/// separates the two (field measurement 2026-08-20: without it, 48 instead of 26 true hits).
/// The median alone is not enough, though: when a dead/relaunched agent re-tails the IME log,
/// one upload burst can carry the majority of the session's IME samples with a CONTINUUM of
/// ages (field 2026-08-28, session d832560a: 508 of 627 samples, 46…733 min) whose median
/// lands on the grid by chance. A genuine zone mis-conversion shifts every line by the same
/// grid multiple, so the grid-conformity test is applied per sample as well: the session
/// only counts when the bulk of the IME deltas individually sit on the grid.
///
/// Goal state: this never fires. Any hit is either a case the per-line anchoring does not
/// cover or a detector bug — both are actionable findings, never noise to tune away.
@@ -30,6 +36,11 @@ namespace AutopilotMonitor.Functions.Services
// ReceivedAt is stamped per upload batch, not per event; a side backed by fewer
// distinct batches yields a batch-composition artifact, not a session property.
internal const int MinDistinctBatchesPerSide = 3;
// Share of IME samples whose own delta (relative to the other side's median) must sit
// within ResidualToleranceMinutes of SOME grid multiple (zero included). A replay
// continuum hits that window only ~2·2/15 ≈ 27 % of the time by chance; a real skew,
// even one mixing two writer eras, hits it for essentially every line.
internal const double MinGridConformantFraction = 0.8;
internal sealed class Result
{
@@ -42,6 +53,7 @@ namespace AutopilotMonitor.Functions.Services
public int OtherSampleCount { get; set; }
public int ImeBatchCount { get; set; }
public int OtherBatchCount { get; set; }
public double GridConformantFraction { get; set; }
}
/// <summary>
@@ -67,6 +79,10 @@ namespace AutopilotMonitor.Functions.Services
if (Math.Abs(steps) < 1 || residual >= ResidualToleranceMinutes)
return null;
var conformant = ComputeGridConformantFraction(scan.ImeDeltaMinutes, medianOther);
if (conformant < MinGridConformantFraction)
return null;
return new Result
{
MedianImeDeltaMinutes = medianIme,
@@ -78,9 +94,31 @@ namespace AutopilotMonitor.Functions.Services
OtherSampleCount = scan.OtherDeltaMinutes.Count,
ImeBatchCount = scan.ImeDistinctBatchCount,
OtherBatchCount = scan.OtherDistinctBatchCount,
GridConformantFraction = conformant,
};
}
/// <summary>
/// Fraction of samples whose delta minus <paramref name="reference"/> lies within
/// <see cref="ResidualToleranceMinutes"/> of any multiple of <see cref="GridMinutes"/>
/// (zero included).
/// </summary>
internal static double ComputeGridConformantFraction(IReadOnlyList<double> deltas, double reference)
{
if (deltas.Count == 0)
return 0;
int hits = 0;
foreach (var delta in deltas)
{
var rel = delta - reference;
var nearest = Math.Round(rel / GridMinutes, MidpointRounding.AwayFromZero) * GridMinutes;
if (Math.Abs(rel - nearest) < ResidualToleranceMinutes)
hits++;
}
return (double)hits / deltas.Count;
}
/// <summary>
/// True when the session's IME events are dominated by writer-declared ("bias")
/// offsets — a bias offset comes verbatim from the log line's own zone marker and
@@ -596,7 +596,7 @@ namespace AutopilotMonitor.Functions.Services
histogram.OrderByDescending(kv => kv.Value).Select(kv => $"{kv.Key} {kv.Value}"));
var message =
$"IME-derived events skewed {result.DiffMinutes:+0.0;-0.0} min vs other sources " +
$"({result.GridSteps}x15min grid, residual {result.ResidualMinutes:0.0} min; " +
$"({result.GridSteps}x15min grid, residual {result.ResidualMinutes:0.0} min, {result.GridConformantFraction:P0} of IME samples on grid; " +
$"median dIME {result.MedianImeDeltaMinutes:0.0} / dOther {result.MedianOtherDeltaMinutes:0.0} min; " +
$"{result.ImeSampleCount}/{result.OtherSampleCount} samples over {result.ImeBatchCount}/{result.OtherBatchCount} batches; " +
$"origins: {(origins.Length > 0 ? origins : "none")})";
@@ -608,6 +608,7 @@ namespace AutopilotMonitor.Functions.Services
diffMinutes = Math.Round(result.DiffMinutes, 2),
gridSteps = result.GridSteps,
residualMinutes = Math.Round(result.ResidualMinutes, 2),
gridConformantFraction = Math.Round(result.GridConformantFraction, 3),
medianImeDeltaMinutes = Math.Round(result.MedianImeDeltaMinutes, 2),
medianOtherDeltaMinutes = Math.Round(result.MedianOtherDeltaMinutes, 2),
imeSampleCount = result.ImeSampleCount,