The post-PR-HS1 single-source design (HS-NEW-RESULT JSON only) had a real timing gap:
IME emits the [HS] new result = JSON line only AFTER its batch send to the Microsoft
service, ~30-90 s after the actual script ran (verified in session c1e714e6: 34-74 s
gap per policy). Short Autopilot enrollments often end inside that window, so health-
script rows were entirely missing from completed sessions even though IME had executed
the scripts.
Two-stage approach (matches original V1 timing for the early signal, while keeping
HS-NEW-RESULT as the canonical truth):
- Re-add HS-COMPLIANCE pattern matching BOTH `pre-remdiation` AND `post-remdiation`
detection compliance lines → HandleHealthScriptDetectionResult emits a script_completed
event immediately. Single regex (`(?<part>pre|post)-remdiation`) covers both phases.
- Re-add HS-RUN-CONTEXT / HS-EXITCODE / HS-STDOUT / HS-STDERR patterns feeding a
single-slot `_pendingHealthScript` accumulator (IME runs health scripts sequentially
within a session — verified). HS-COMPLIANCE merges the slot data onto the early-signal
event so exit / stdout / stderr / runContext are populated from the start, not just
the inferred compliance verdict.
- HS-NEW-RESULT continues to fire later with the full JSON (RemediationStatus,
TargetType, ErrorCode, plus the actual remediation phase's data). UI reducer's
dataCompleteness() scoring keeps the more complete entry — early-signal row gets
enriched in place, no duplicate cards.
Remediation phase itself has no early-signal source in IME logs (only the late JSON
exposes Info.RemediationExitCode and RemediationScriptOutputDetails). A follow-up
based on the agentexecutor invocation line's _RemediationScript.output suffix is
possible once we have a real non-compliant cycle log to validate against.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
UX fallout from session e2929c97 debrief:
- Dedupe: same (policyId, scriptType, scriptPart) re-emissions collapse into one row
keeping the most-complete entry via dataCompleteness() score. Real-world Autopilot
pattern is IME re-emitting the same script across ESP-phase transitions, sometimes
with degraded data on the second emission. Trade-off vs. Codex finding 3 (preserving
distinct runs of scheduled scripts in long sessions) — out of scope for this UI.
- Labels: use Intune Admin Center terminology — all health-script rows now read
"Remediation" with phase shown via separate badge ("detection" / "remediation" /
"post-detection"). The customer-facing word for these policies in Intune is
"Remediation", not "Health Detection".
- Flicker fix: React key included firstSeenIndex which was reassigned on every
reducer run, causing ScriptItemRow to remount on upstream prop changes (live
polling / SignalR) and collapse expanded detail panels. New scriptItemKey() uses
only identity-shaped fields (policyId, scriptType, scriptPart). firstSeenIndex
field removed from ScriptItem entirely.
- Animation polish: removed animate-pulse from Running container — only the inline
spinner SVG conveys motion now. Container is neutral gray. With multiple concurrent
Running cards the full-box pulse made the page feel unstable.
- stderr-as-failure rule (consistent across script types): stderr present →
script_failed regardless of exit code or compliance verdict. Eliminates the
inconsistency where a platform script with stderr showed red but a remediation
detection with stderr (and exit 0 + compliant) showed green for the same data
shape. Implemented in BOTH the Agent adapter and the Web reducer.
Agent V2 only — V1 phasing out.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI ajv validation rejected HS-NEW-RESULT.json because the new healthScriptResult
action wasn't in the schema's enum allowlist. Add it next to the other script*
actions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the line-by-line HS-COMPLIANCE / HS-EXITCODE / HS-STDOUT / HS-STDERR /
HS-RUN-CONTEXT pattern set (only ever captured the pre-detection phase, reset state
mid-cycle) with a single atomic source: HS-NEW-RESULT matches IME's "[HS] new result =
{JSON}" line and emits up to three script_completed events per policy run (pre-detection
/ remediation / post-detection). HS-SCRIPT-START now also fires a live script_started
event so the UI shows a "running" indicator immediately, before the consolidated final
result lands ~30s-3min later.
Phase-aware failure semantics: detection / post-detection scripts use exit code as a
compliance verdict (not a crash signal), so a non-compliant detection that gets
remediated no longer counts as a "failed" event in metrics — the row renders with
amber styling via isNonCompliantReport(). Only the remediation phase + platform
scripts route non-zero exit to script_failed.
Defensive against wire-format drift: regex uses [\s\S]* with end-anchor, JSON parsed
with try/catch + counter, frontend toNumber() accepts both numeric and string fields
(backend serializes ints as strings via Dictionary<string,string>). Reducer dedupes
with timestamp included so repeated runs of the same policy in long sessions don't
collapse into one row.
V2 only — V1 (Agent.Core) phasing out.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fires Teams/Slack alert at backend session registration. Default off so
existing tenants don't get a surprise notification storm. Triggers on
fresh session OR WhiteGlove Part 2 resume; agent restart and
re-registration of terminal sessions are skipped via pre-read of the
session row before StoreSessionAsync.
NotifyOnWhiteGloveResume constant in RegisterSessionFunction lets us
disable the resume signal with a 1-line patch if it lands badly with
end users, without removing the fresh-session signal.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Background
----------
The in-memory SlaNotificationThrottleService kept its 4h cooldown in a process-
local ConcurrentDictionary. On Flex Consumption the Function host recycles
between 2h timer fires, wiping the dictionary, so the throttle was effectively
a no-op — observed in production as tenant b54dc1af receiving an SLA breach
notification every 2 hours for days on end (1 failed of 11-13 sessions, target
95%, success rate 91-92%).
Design
------
One row per tenant in a new SlaTenantStatus table (PartitionKey = tenantId,
RowKey = "status"). Flat schema with {Type}_* property prefixes for the four
breach kinds (SuccessRate, Duration, AppInstall, ConsecutiveFailures); one row,
one Upsert per evaluation cycle. Rationale documented inline: per-type rows
would be over-normalised for the access pattern (always all-types-per-tenant),
and ~50 tenants x 4 types = max 200 rows ever.
Throttle correctness
--------------------
The throttle decision (now - LastNotifiedAt >= cooldown) and the LastNotifiedAt
write happen in the same TryUpsertAsync(status, ifMatchETag) call. Two parallel
inline calls cannot both succeed in sending the same notification — the loser
gets a 412 from Table Storage, refetches, sees the updated watermark, and
either skips or schedules a new claim attempt. Up to 4 retries; on exhausted
retries we drop the notification rather than risk double-fire (at-most-once).
Cooldown is configurable: AdminConfiguration.SlaNotificationCooldownHours
(default 24, range 1-168). UI round-trip wired through the four required
files (AdminConfigContext, AdminConfigSettingsSection, types/adminConfig.ts,
SectionGlobalSettings).
Resolve path
------------
When a previously-active breach is no longer detected, the row transitions to
IsActive=false + ResolvedAt=now and a single sla_resolved notification fires.
Orphan sweep: tenants whose toggles get turned off while a breach is active
fall out of the qualifying loop, so the timer now also scans every status row
and silently clears any IsActive that no longer has a toggle backing it
(no resolved notification — the tenant disabled SLA, they don't care).
Per-type notes
--------------
- SuccessRate breach notifies with the configured warn-threshold, not the SLA
target — a 94% rate against target=99% / threshold=95% reads "below 95%",
not "below 99%".
- AppInstall now actually evaluates (the toggle was wired but evaluation was
missing). Aligned to the same ISO-week window the dashboard uses
(SlaMetricsService.GetIsoWeekKey), so notifications cannot diverge from the
/sla page. Sample-size floor of 5 apps per week, same as the dashboard.
- AppInstall has its own notification template (own title/summary/facts plus
TopFailingApp fact) — the builder used to silently fall through to the
Duration template for everything not SuccessRate.
- Telemetry Period tag derived from breachType: CurrentMonth for SuccessRate
and Duration, CurrentWeek for AppInstall — so AppInsights / ops dashboards
don't group AppInstall under the wrong period.
- AppInstall ops events now carry the actual installer totals (10 evaluated,
4 failed) instead of 0/0.
- ConsecutiveFailures: inline (ingest) path detects breaches, timer path
detects resolve. Both paths write to the same row through the same CAS
retry loop.
Clock injection
---------------
SlaBreachEvaluationService takes an optional Func<DateTime> nowProvider
(default () => DateTime.UtcNow) so the week-window regression test runs
deterministically on every test day instead of silently skipping when today
happens to fall in the first ISO week of the month.
Removed
-------
- SlaNotificationThrottleService and its tests. The in-memory model is gone;
there is no scenario where it would be wanted alongside the persistent row.
Tests
-----
- 21 new lifecycle + mapping + telemetry tests (CAS conflict + retry, CAS
exhausted-retries drops notification, threshold-not-target in notification,
orphan clearing, AppInstall template / week-window / totals, ConsecutiveFailures
inline persistence, multi-type race-probe, period-mapping theory).
- Backend xUnit: 1218 passed.
- Web vitest: 266 passed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Azure Functions' attribute router does not honour literal-vs-parametric
specificity the way ASP.NET attribute routing does. A literal route of
"sessions/stats" was being eaten by the sibling GetSessionFunction whose
template is "sessions/{sessionId}" — the runtime bound /api/sessions/stats
to GetSession with sessionId="stats", which then 404'd because "stats"
fails GUID validation.
Diagnosed live via App Insights:
/api/global/sessions/stats -> 200 OK -> Functions.GetAllSessionStats
/api/sessions/stats -> 404 -> Functions.GetSession (!)
Per-tenant + GA stats routes moved out from under sessions/ for symmetry:
/api/sessions/stats -> /api/stats/sessions (MemberRead)
/api/global/sessions/stats -> /api/global/stats/sessions (GlobalAdminOnly)
EndpointAccessPolicyCatalog, api.ts, and the URL-builder tests follow.
Function-side + api.ts both carry a comment pinning the rationale so a
future "let's tidy these routes" refactor doesn't regress it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A customer device 2026-05-11 failed bootstrap at the WMI runtime spawn:
returnValue=2 (Access Denied -- Defender ASR rule
d1e49aac-8f56-4280-b9ba-993a6d77406c "Block process creations originating
from PSExec and WMI commands"). The throw propagated to --install exit 1
-> bootstrap throw -> next IME run SKIPped via pre-flight, leaving the
device with payload deployed + BootTrigger task registered but no live
runtime until manual intervention.
--install now runs a two-tier fallback: try WMI Win32_Process.Create ->
try schtasks /Run on the BootTrigger task -> defer to BootTrigger on next
reboot. Exit code is 0 whenever deployment + Scheduled Task creation
succeeded; runtime-spawn failure is a WARNING, not fatal. The bootstrap
script's Get-Process probe (Install-AutopilotMonitor.ps1:280) is now the
canonical "no runtime came up" signal. Orchestration extracted to pure
function DecideRuntimeLaunchOutcome, unit-tested with 8 cases pinning
the contract (incl. ASR-rule GUID surfaced in diagnostic only when
returnValue=2).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous commit added the two new EventTypes server-side
(PoisonQueueBacklogHigh / PoisonQueueBacklogCritical) but the admin alert
rules section hardcodes its known-event-types catalog in the component
itself - so the dropdown didn't list them and operators couldn't wire
Telegram to the new alerts.
Adds both strings to the Security category, matching how the SignalR/Cert
event types were registered when those watchers landed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends the SignalR/Cert pattern to async-worker poison queues. The previous
PR made backlogs visible on /health/detailed - this one makes them actively
page the operator via the existing OpsAlertDispatch -> Telegram path.
New partial MaintenanceService.PoisonQueue.cs runs on the 2 h timer (and on
manual maintenance triggers) and emits two new EventTypes:
- PoisonQueueBacklogHigh (Severity=Warning, threshold >= 1)
- PoisonQueueBacklogCritical (Severity=Error, threshold >= 10)
Dedup mirrors the cert watcher: composite key {EventType}|{queueName} built
from today's Security OpsEvents -> one alert per queue per tier per UTC day.
A tier escalation (warning -> critical) fires a new alert because the
EventType changes; a stable backlog re-pings once per UTC day so it stays in
the operator's attention.
Constants + classifier + watch-list moved from HealthCheckService to the new
MaintenanceService partial - the live health card and the timer-driven
Telegram alert now classify identically (same canonical source). Boundary
and JSON-extractor tests follow the move; HealthCheck tests are reduced to
the card-form assertions they actually exercise.
Operator follow-up (manual, not in code): add two OpsAlertRules in the admin
UI - one per new EventType - to wire Telegram. Until then the watcher writes
OpsEvents but no message is sent, matching how SignalR/Cert alerts were
introduced.
Backend: 1199/1199 (1193 + 6 net from test resplit). Build 0/0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Live data showed cross-tenant 7d breakdown of non-terminal sessions:
InProgress 11, Pending 43, Stalled 6 = 60 total. The Pending bucket
dominates for any tenant with regular WhiteGlove provisioning — those
sessions have completed pre-prov and are waiting for the end user to
power on the device, which can take days or weeks. Counting them under
"Currently enrolling" implies live activity that isn't happening.
Active now matches the literal in-flight status. Pending and Stalled
remain visible via the sessions list filters; if we want a richer
breakdown on the card row later, it can be split into separate cards
without further changing the Active semantics.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three async workers (analyze-on-enrollment-end, vulnerability-correlate,
telemetry-index-reconcile) park 5x-failed messages on <name>-poison. Until now
there was no operator-visible signal when this happened: backlogs could grow
silently and sessions stayed un-analyzed / vulnerability reports were lost.
Adds a "Poison Queues" entry to the detailed health-check next to "SignalR
Quota" with the same shape: parallel probe via IPoisonQueueProbe, tri-tier
classifier (0=healthy, >=1 warning, >=10 critical, thresholds App-Setting
configurable via PoisonQueueWarningThreshold / PoisonQueueCriticalThreshold),
GA-only visibility. Per-queue 404 counts as zero because poison queues are
created lazily on first poison-move. Per-queue exceptions surface in details
as "error: <msg>" and force at least warning so a flaky Storage call cannot
mask a real backlog.
Web UI needs no change - the generic checks[] render loop on /health-check
picks up the new card automatically.
Backend: 1193/1193 (1178 + 15 new). Build 0 warnings, 0 errors.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to b76a6cca. Two issues surfaced after deploy:
1. Toggling GA mode off left the cross-tenant numbers stuck on the cards.
Hook now setStats(null) synchronously before refetching on every scope
change, and the card render falls back to "..." whenever stats is null
(covers initial load, post-toggle reset, and fetch errors — previously
an error would silently show 0).
2. "Active Sessions" intent: the user wants in-flight sessions in the
broader sense, which matches the original code. AggregateSessionStats
counts InProgress + Pending (WhiteGlove waiting on user) + Stalled
(>60min no progress, non-terminal, can heal back to InProgress).
Test renamed + comment pins the rationale so a future "tighten to
InProgress only" refactor knows what it's changing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The five dashboard cards (Active Sessions, Success Rate, Avg. Duration,
Total Today, Failed Today) were derived from whatever sessions the client
had paginated into memory. With default pageSize=10 and SignalR-driven
appends, "Total Today" would jump 40 -> 22 -> 20 within a single visit
as the list grew/shrank. "Last 7 days" was a printed label only - the
client compute had no 7-day filter at all.
Now:
- GET /api/sessions/stats?days=7 (MemberRead, JWT tenant)
- GET /api/global/sessions/stats?days=7&tenantId= (GlobalAdminOnly,
optional tenant filter; absent = cross-tenant sum)
- TableStorageService.AggregateSessionStats drains SessionsIndex once
over the window: UTC-midnight boundary for today, success rate over
terminal sessions only (InProgress no longer drags it down), avg
duration over Succeeded with DurationSeconds > 0 only.
- useDashboardStats hook owns the cards: initial fetch + debounced
refetch on SignalR newSession/newevents in the active scope + forced
refetch on reconnect to recover messages missed during an outage.
- Client compute removed from useDashboardFilters. New
submittedTenantIdFilter state isolates typing-in-filter from
submitted-filter so keystrokes never trigger backend round-trips.
Tests: 17 xUnit (aggregation invariants + days parsing) +
7 vitest (URL builder). Suites: backend 1177/1178 (unrelated
RawTablePaginationTests flake), web 266/266.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Skip the default 0->3->7->17 Dictionary resize cascade on hot-path
event/audit-trail allocations by setting initial capacity + Ordinal
comparer. ~3k Gen0 allocs saved per session for an agent that runs
continuously on endpoints. Behaviour unchanged, 1295 tests green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Resolves GHSA-q3j6-qgpj-74h6 (path traversal via percent-encoded dot
segments) and GHSA-v39h-62p7-jpjc (host confusion via percent-encoded
authority delimiters). Transitive via @modelcontextprotocol/sdk -> ajv.
Lockfile-only change, no API surface impact.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
HealthCheckService now runs a fourth check (SignalR Quota) alongside Storage,
Processing, and Agent Binaries. Live read of ConnectionCount (max/1h) and
MessageCount (total since 00:00 UTC) via the same IAzureMonitorMetricsReader
that backs the maintenance-time Ops watcher - no cache; the data is already
3-5min stale from Azure Monitor's side, and a /health/detailed re-check is
expected to give fresh values during incident response.
Reuses MaintenanceService's internal classifier (CalculatePercent,
ClassifySignalRQuotaTier) so the watcher and the dashboard agree on the
80%/95% boundaries down to the floor-vs-ceil rounding.
Tier mapping: None -> healthy, Warning -> warning, Critical -> unhealthy.
Worst-of-both-metrics wins. PerformAllChecksAsync now uses a tri-state
combiner (any unhealthy / any warning / else healthy) so a Warning tier
correctly tints the OverallStatus bar yellow instead of red.
Fail-soft:
- SignalRResourceId unset -> Status="unknown" with "watcher not configured"
(excluded from OverallStatus combiner so dev/local stays healthy)
- Both metric reads null -> Status="unhealthy" pointing at the most likely
cause (Function App MI missing 'Monitoring Reader' on the SignalR resource)
- One read null -> "no data" for that row, normal classification for the other
Frontend unchanged: the existing /health-check Backend Services card loop
renders the new check with Connections, Messages, and Resource as Details.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces V1/V2-hardcoded build + bootstrap with a single parametric
pipeline that scales to V3+ without rewrites. Full architecture in
.claude/plans/v2-cutover.md.
Stable namespace (Bootstrap-side, never renamed): generic Bootstrap
script, agent owns its --install defaults so it works for any line.
Versioned namespace (per-line, SelfUpdater-side): isolated *-v{N}.*
blobs; SelfUpdater never crosses lines or touches stable.
Backend GetAgentConfig dispatches per X-Agent-Version-Major to
AdminConfiguration.GetAgentLine(int) -> LatestAgentV{N}* fields.
Future V3 = add field set + switch arm. TableConfigRepository.Map
falls back to legacy unsuffixed columns for transparent migration.
V1 SelfUpdater now reads version-v1.json (would have been clobbered
by the V2 cutover otherwise). V2 agent owns its own 600s TenantId-wait
default so the Bootstrap calls --install plain.
Tests: 3289 across 6 suites, all green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
session_searched fired on every dashboard mount with empty query, inflating
event volume ~3x and making unique-user counts unusable. Skip empty queries
and emit queryLength as a measurement instead of the now-redundant hasQuery.
signalr_reconnected lacked any latency signal. Capture performance.now() in
onreconnecting and emit downtimeMs in onreconnected so recovery quality
(not just existence) shows up in App Insights.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The two notification-bell React contexts polled GET /api/notifications and
GET /api/global/notifications every 60s regardless of activity, generating
3017+618 = 3635 calls/24h with near-zero hit rate (notifications are emitted
sparsely). SignalR was already used app-wide for live session updates; the
bell was the lone hold-out on a pull model.
Switch to the same push model. Steady-state traffic drops ~86% and the bell
updates sub-second instead of within 60s.
Architecture (Option C+):
- Mount -> 1x HTTP fetch (cold-start hydration via existing GET endpoint)
- SignalR onreconnected -> re-fetch (covers the disconnect window)
- SignalR push -> live append/dismiss
- No periodic polling, no setInterval
Audience-tier security: TenantNotificationAudienceCatalog still drives
visibility (Member vs Admin), but enforcement moves into SignalR group
membership: tenant-{id}-notify-member is joined by every tenant member,
tenant-{id}-notify-admin only by Tenant Admins / Global Admins. The
AddToGroup function fail-closes with 403 on Member-tier joins to the
admin group, so Admin-tier notifications cannot leak via push.
Backend
- ISignalRNotificationService extracted; SignalRNotificationService
implements it. Resolves the long-standing test-mocking blocker
(the real service's constructor validates AzureSignalRConnectionString
and BuildServiceManager(); previously you could not unit-test
consumers). Tenant + Global NotificationService consume the
interface; the three legacy callers (Analyze, Ingest, VulnCorrelate)
still bind to the concrete class -- migrate per test need.
- Six new push methods: SendTenantNotification{,Dismissed,DismissedAll}
and SendGlobalNotification{,Dismissed,DismissedAll}. Tenant variant
routes to the audience-correct group.
- Tenant + Global service push after a successful repo write
(fire-and-forget; never throws into the caller).
- SignalRGroupHelper: parsing handles the two new -notify-{member,admin}
suffixes; new IsTenantNotifyAdminGroup predicate; group-name builders.
- SignalRAddToGroupFunction: explicit 403 when a non-admin attempts
to join the admin-tier group.
Frontend
- TenantNotificationContext: setInterval gone. useSignalR() to
join member-group always, admin-group iff isTenantAdmin || isGlobalAdmin.
Listens for tenantNotification (append + dedupe), tenantNotificationDismissed
(filter), tenantNotificationsDismissedAll (clear). onreconnected
triggers re-fetch.
- GlobalNotificationContext: same pattern against global-admins group.
- app/layout.tsx: SignalRProvider hoisted above the notification
providers so useSignalR() resolves inside them.
Tests
- SignalRGroupHelperTests (8): parsing for all four group formats
incl. new -notify suffixes, IsTenantNotifyAdminGroup predicate.
- TenantNotificationServiceSignalRPushTests (8): audience-correct push
on Create (Admin / Member / Unknown -> Admin fail-closed), no-push
on repo failure, no-push on DismissAll(count=0), no-push on empty
tenantId.
- FakeSignalRNotificationService: in-memory spy implementing the
new interface; replaces the unmockable real service in tests.
- AuthFunctionSideEffectTests + TenantNotificationServiceTests:
constructor signatures updated for the new SignalR dep.
Suites
Backend xUnit: 1161/1161 (was 1146, +15 net).
Web vitest: 259/259 (unchanged; existing notification-API +
gate tests still cover).
Solution build: 0 warnings, 0 errors.
Expected post-deploy reduction (24h):
GET GetTenantNotifications: 3017 -> ~50
GET GetGlobalNotifications: 618 -> ~10
POST AddToGroup: ~246 -> ~350-450 (two joins per tab)
Net bell load: -86%, sub-second live UX.
Disconnect tradeoff (accepted): if SignalR drops without reconnect
(rare -- withAutomaticReconnect retries 1/2/4/8/16/30s ... forever),
the bell shows the snapshot from the last fetch but no live updates
until next reload. The initial-3-retry give-up path of the SignalR
client still gets the user a hydrated bell because the HTTP fetch
runs independently on mount.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds toggles for SignalRConnectionsHigh, SignalRConnectionsCritical,
SignalRMessagesHigh, SignalRMessagesCritical to the Security category of
the OpsAlertRules section, so operators can route them to Telegram /
Teams / Slack via the existing alert-dispatch pipeline.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hourly TimerFunction queries Azure Monitor for SignalR ConnectionCount (max
over last hour) and MessageCount (sum since UTC midnight) and emits Warning
(>=80%) / Error (>=95%) OpsEvents when the free-tier limits (20 connections,
20k messages/day) are approached. Mirror of the cert-expiry pattern - global
TenantId=null events, dedup'd to one emission per EventType per UTC day.
Fail-soft: skips silently when SignalRResourceId app setting is unset, logs
warning on Azure Monitor auth/query errors. Function App's managed identity
needs 'Monitoring Reader' on the SignalR resource.
Also called from the manual maintenance trigger so 'Run Maintenance Now'
exercises every health check.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
77fde56f hooked CheckEmbeddedCertExpiryAsync only into RunAllAsync (the 2h
timer). Manual triggers via POST /api/maintenance/trigger call RunManualAsync,
which is a different code path - so manual runs were silently skipping the
watcher and gave a false sense of "no alert means bucket is healthy". Mirror
the timer-path call inside the !aggregateOnly block so manual ops verification
exercises the watcher just like the scheduled run does.
Aggregate-only manual runs (date catch-up) still skip it, since they're for
backfilling historical metrics and don't need cert checks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Alert Rules section in the admin UI builds its rule grid from a
hardcoded OPS_EVENT_TYPES catalog. Without this update, the four
EventTypes added by f2257a4b would not appear as toggle rows, so the
operator could not subscribe Telegram/Teams/Slack to them despite the
backend already emitting the events.
Added under the existing Security category:
- EmbeddedCertExpiringSoon (set MinSeverity Warning)
- EmbeddedCertExpiringUrgent (set MinSeverity Error)
- EmbeddedCertExpired (set MinSeverity Critical)
- EmbeddedCertBundleEmpty (set MinSeverity Critical)
After deploy: admin opens Settings -> Alerts, toggles the 4 rows on,
sets MinSeverity per row, enables Telegram + Chat ID, saves. From the
next maintenance run any cert-expiry event matching the threshold pings
the configured providers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
77fde56f's per-cert alarm fires for every expiring cert independently. That
is wrong: when a Root or Intermediate has a fresh successor in the bundle
(e.g. intune-intermediate-mdm-2026.pem alongside intune-intermediate-mdm-2028
.pem), the older cert is just a chain-continuity helper for in-flight device
certs. Alarming on it would page the operator to "rotate" something that is
already rotated.
Refactor to bucket-level: per role (Root, Intermediate), pick the cert with
the MAXIMUM NotAfter and classify only that. Older expiring/expired
siblings are silent. Operator only gets paged when the freshest itself
nears expiry, i.e. when there is genuinely no successor yet.
Concrete behaviour change for current bundle (today 2026-05-08):
Roots: freshest is intune-root-2030.pem (1591d) -> silent
Intermediates: freshest is intune-intermediate-mdm-2028.pem (859d) -> silent
The expired intune-intermediate-mdm-2026.pem (-3d) no longer triggers a
spurious Critical because mdm-2028 has it covered.
Tests: 7 new EvaluateBucket cases (empty, single fresh, single soon, old+
fresh successor stays silent, old+soon successor fires on successor, all
expired falls back to freshest-expired, two future successors). Total 38
cert-related tests, full suite 1105/1105.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds proactive monitoring of the embedded Intune CA bundle so the operator
gets a Telegram ping months before agent mTLS would break. Bundle rotation
(re-exporting fresh PEMs from a corporate-enrolled device) is the only fix
and needs lead time - today's outage from 9af3de15 was rooted in exactly
this blind spot.
Hooks into the existing 2h MaintenanceService.RunAllAsync timer, iterates
the embedded roots + intermediates, and emits tiered OpsEvents on the
Security category:
daysUntilExpiry | Severity | EventType
-----------------+----------+----------------------------
<= 7 (or past) | Critical | EmbeddedCertExpired
<= 30 | Error | EmbeddedCertExpiringUrgent
<= 90 | Warning | EmbeddedCertExpiringSoon
else | - | (silent)
Plus a Critical EmbeddedCertBundleEmpty event if the loader returns no
roots at all (server misconfiguration / failed deploy).
Dedup is one event per cert per tier per UTC day, by querying today's
Security OpsEvents and matching on EventType + thumbprint extracted from
the Details JSON. Cold starts (Flex Consumption scale-to-zero) rebuild
the dedup index from the table, so repeated process restarts do not
multiply Telegram pings.
Operator action required to receive Telegram alerts: configure four
OpsAlertRules in admin UI - one per new EventType - with appropriate
MinSeverity. The OpsEvents themselves land in the table regardless,
visible in the Ops dashboard.
Tests: 23 new (tier-classifier boundary + dedup-key extraction + JSON
edge cases). Full suite 1098/1098 green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
intune-root-ca.pem and intune-intermediate-ca.pem are bit-for-bit duplicates
of intune-root-2030.pem and intune-intermediate-mdm-2026.pem (matching SHA1
fingerprints A1:97:D6:... and B7:E4:96:...). The post-9af3de15 loader globs
intune-root-*.pem and intune-intermediate-*.pem and dedupes by thumbprint,
so the duplicates were silently de-duped at load. Remove them now to make
the bundle self-explanatory: one PEM per real CA, name encodes its NotAfter.
Verified by running CertificateValidatorTests (8/8 incl. the real-device-cert
happy-path) which loads the bundle via the same glob path the runtime uses.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Post-deploy of 9af3de15 surfaced 6 chain-validation failures from a single
agent stuck in a 10s retry loop. The current LogWarning only includes the
chain-status string ("unable to get local issuer certificate"), which is
not enough to triage whether the rejected cert is a forge attempt the new
pinning is correctly blocking, or a legit fleet device whose chain we
need to add to the embedded bundle.
Add LeafSubject (GUID-shaped DN for Intune device certs), LeafIssuer (CA
name), and existing Thumbprint to the warning. No PII is leaked - device
certs use opaque GUIDs and the issuer is a public CA DN.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Re-applies 7e21591b after the chain-validation outage and adds the missing
trust anchors. The first attempt embedded only the 2030 root and the 2026
intermediate; current production device certs chain via the 2026 root +
the 2028 intermediate, so chain.Build() returned "unable to get local
issuer certificate" for every legitimate agent and broke enrolment.
What is different in v2:
- Bundle expanded with both Intune roots (exp 2026-08-12 and 2030-09-15)
and both currently-issuing MDM Device CA intermediates (exp 2026-05-05
and 2028-09-15). Real device certs chain via 2028-intermediate -> 2030-
root today; the 2026 generation is kept for any older device certs
still in flight.
- Loader now globs intune-root-*.pem and intune-intermediate-*.pem from
embedded resources and dedupes by thumbprint, so a future Intune root
rotation is a drop-in PEM with no code change. Legacy generic-named
files (intune-root-ca.pem, intune-intermediate-ca.pem) are duplicates
of the new ones and harmlessly de-dup at load.
- Adds a happy-path integration test against a real Intune-issued device
cert (public-key only, copied from a test VM). This is the regression
test that would have caught the v1 outage and gates any future bundle
drift in CI.
- Fixes a pre-existing TZ bug in the expiry check: X509Certificate2's
NotBefore/NotAfter return Local time, were being compared raw against
DateTime.UtcNow. Surfaced by the integration test on a CEST dev box;
unobservable in prod because Linux Flex Consumption runs in UTC.
Trust model is unchanged from v1: X509ChainTrustMode.CustomRootTrust
pinned to the embedded Intune roots, OS trust store ignored, fail-closed
when no roots are loaded. Self-signed-with-Intune-DN bypass remains shut.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pairs with ConsentFlowStarted/Failed in the OpsEvents table so admins can
see whether repeated consent failures (AADSTS, redirect-mismatch, denied)
eventually resolved into a successful grant. New POST endpoint
config/{tenantId}/autopilot-device-validation/consent-success is called
from the post-redirect callback only after consent-status confirms
isConsented=true, so generic status polls do not fire false positives.
Trigger whitelist accepts autopilot, corporate, and device-preparation;
the third value is wired through the type system end-to-end so a future
DevPrep-specific consent flow can use it without a schema bump.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CertificateValidator previously combined AllowUnknownCertificateAuthority
with a DN-substring scan ("Microsoft Intune"/"MDM Device CA" in
Subject/Issuer of any chain element). A self-signed leaf with that DN
satisfied the substring gate while the unknown-root flag masked the
untrusted-root status -> the X.509 trust gate was bypassable.
Switch to X509ChainTrustMode.CustomRootTrust with the embedded Intune
root in CustomTrustStore and the intermediate in ExtraStore. chain.Build()
now requires a signature link to a pinned root regardless of the OS
trust store, which keeps Linux/Flex Consumption working without the
self-signed bypass. Empty roots -> fail-closed.
Adds 7 unit tests including the self-signed-impersonating-Intune
regression test.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Removes the "View at lifecycle anchors" link (and its sessionId prop) from
FailureSnapshotBlock; SessionInfoCard updated to match the new prop shape.
Bumps page-lastmod.generated.ts and the V2 agent buildcounter.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes a tenant-boundary write-side hole and removes two manual cross-tenant
checks in favor of catalog-driven middleware enforcement.
SubmitSessionReport + SubmitDiagFilesReport (the violation):
The functions previously fell back to the JWT tenantId only when the body's
TenantId was empty, otherwise trusting the body verbatim. A Tenant Admin of
Tenant-X could POST a body with tenantId=Y and have the report blob, the
SessionReports row, and the audit-log entry all attributed to Tenant-Y
(integrity / non-repudiation issue, no read-side leak).
New behavior: GA-aware. Non-GAs who send a body tenantId that disagrees with
their JWT get a 403 + warning log. GAs may legitimately submit cross-tenant
reports (the UI sends the session's tenantId), but MUST set the body field
explicitly (no silent fallback that would mis-file under the GA's home tenant).
Diagnostics + Progress middleware elevation (defense-in-depth):
diagnostics/download-url and progress/sessions/{sessionId}/events lifted from
TenantScoping.None (in-function guard) to TenantScoping.QueryParam (catalog +
middleware). Same GA bypass semantics — the in-function check was correct but
copy-paste fragile. Functions now read requestCtx.TargetTenantId, which the
middleware has already validated against the JWT.
Test coverage:
CrossTenantAccessTests gains 10 new InlineData rows across the 5 QueryParam
test methods (no-query / same-tenant / cross-non-GA blocked / cross-GA allowed
/ scoping correctness) for both lifted routes. diagnostics/download-url moved
out of JwtImplicitRoutes_NoCrossTenantCheck since it's no longer JWT-implicit.
Verified: dotnet build clean, 1067/1067 backend tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The tenantId useState initializer used to seed itself from localStorage,
opening a small mount-window where a child component could fetch with a
tampered tenantId before the auth effect overrode it from user.tenantId.
No remaining feature reads from this localStorage key (the only
setTenantId call sites are inside TenantContext itself), so the read +
the corresponding write are dead. Initialize empty, clear any stale LS
value once on mount, and rely solely on the JWT-derived user.tenantId.
Backend MUST still derive tenantId from the JWT claim, not from
query/body params - this change is defense-in-depth, not the primary
guarantee.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Drop 'unsafe-eval' from script-src CSP — Next.js prod build doesn't
need it; closes an XSS escalation vector for any future inline payload.
- Cap error messages embedded in the generated OOBE PowerShell script at
200 chars to prevent a multi-MB upstream message from being inlined
verbatim into a script that runs as SYSTEM during Windows OOBE.
- Add X-Content-Type-Options: nosniff to the /go error response for
consistency with the success response.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Customer reported "From Peers 100% / Peer breakdown LAN 0 | Group 0 | Internet 0"
mid-download. Root cause: OS collector emitted LAN/Group/Internet/LinkLocal
breakdown only on the final do_telemetry event, not on download_progress events
during the download — so UI rendered 0/0/0 until completion. Empirical check of
917 do_telemetry events / 146 sessions / 13 tenants over 30 days confirmed IME
[DO TEL] log path is dead code (current IME version no longer logs it); OS
cmdlet is the sole live source.
Changes:
- AppPackageState (V2): new fields DoBytesFromLinkLocalPeers,
DoBytesFromCacheServer, DoCacheHost; UpdateDoTelemetry signature extended
with optional defaults so callers without the data pass 0/null.
- DeliveryOptimizationCollector: read BytesFromLinkLocalPeers + CacheHost
from cmdlet output; centralised event payload via BuildDoEventData() so
download_progress and do_telemetry are field-symmetric.
- ImeLogTracker.HandleDoTelemetry: parses new fields null-safe (older IME
JSON without them defaults to 0/null, no breakage).
- Persistence: PackageStateData DTO + ToPackageStateData / FromPackageStateData
mappers carry the new fields across agent restart.
- ImeLogTrackerAdapter.EmitDoTelemetry: emits new fields on do_telemetry,
including doCacheHost when MCC was used.
- Backend ingest (V1 + V2 paths) + AppInstallSummary model +
TableStorageService.Metrics Store/Map/preserve-from-existing extended for
the new columns.
- DoAggregator + GeographicApiModels + GetGeographicMetricsFunction surface
MCC + LinkLocal at location and per-session level.
- UI DownloadProgress.tsx: new "From Connected Cache" line with cache% and
hover host; LAN merges LinkLocal in breakdown ("local network"); 3-color
progress bar (Peers green / MCC purple / HTTP blue) computed by subtracting
CacheServer from HTTP to avoid double-counting (MCC serves over HTTP).
- UI sessions page: hover title now includes MCC bytes and LAN+LinkLocal sum.
Robustness: every new field defaults to 0 / null / "" on machines or paths
that don't supply them; no existing session display regresses.
Tests: backend 1058/1058, agent V2 1007/1007, web vitest 259/259, tsc clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rename env var STATE_HMAC_KEY → OAuthStateSigningKey and switch decode
from utf-8 to base64, matching the backend's PaginationTokenSigningKey.
Same naming pattern (PascalCase, "...SigningKey" suffix), same encoding
(base64), same fail-loud validation (≥32 bytes after decode).
Bicep param + secret name renamed accordingly (lives on disk only —
infra/ is gitignored). Operator: rotate the secret value during the next
deploy and update the env var name in the Container App.
The fallback-to-random-bytes branch is preserved so single-replica
deployments without explicit key configuration still get a working,
short-lived signature; that is unchanged from the previous commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
External security review (2026-05-08) flagged 5 issues against the MCP
server's auth surface. All five are addressed in code; one (Bicep param
additions) lives on disk only because infra/ is gitignored — the operator
needs to redeploy with the new mcpPublicUrl + stateHmacKey parameters
documented in the bicep header to put F3 fully into effect.
#1 access-guard cache-key collision (Medium)
Cache was keyed on UPN only. A forged JWT with a victim's UPN would
hit a legitimate user's cached `allowed: true` and bypass the backend
signature check for any MCP-local operation (tools/list, resources/
list etc.) plus burn the user's rate-limit budget.
Fix: cache key is now `${upn}:${sha256(token)}`, so distinct tokens
always require a fresh backend roundtrip. accessCache joins the 5-min
cleanup tick to evict expired entries (cardinality went up).
#2 /oauth/register DoS surface (Medium)
Endpoint is unauthenticated per RFC 7591 and the registry was an
unbounded in-memory Map at 0.5 GiB container budget. Per-IP rate-
limit was rejected (cloud-proxy egress collisions — Zscaler etc.).
Fix: route-level express.json({ limit: '8kb' }) registered before
the global parser; map-size cap at 10_000 entries (~3 MB worst-case)
→ 503 when full; field-level caps on redirect_uris.length and
client_name. TTL/LRU eviction is intentionally NOT enabled —
documented as TODO(oauth-map-ttl) because it interacts with the
manual /mcp reconnect flow.
#3 OAuth base URL trusted X-Forwarded-* headers (Medium)
Discovery, issuer, WWW-Authenticate, and `${baseUrl}/oauth/callback`
were all derived from caller-supplied headers. Container Apps Ingress
appends but does not strip X-Forwarded-Host, so a hostile client could
spoof the issuer in /.well-known/oauth-authorization-server.
Fix: startup warning when MCP_PUBLIC_URL is unset. The bicep file
(gitignored) was also extended with a `mcpPublicUrl` param — operator
must redeploy with that pinned to the Container App FQDN after the
first deploy emits it as an output.
#4 PKCE/state hardening (Medium)
code_challenge was optional and only forwarded conditionally;
code_verifier was never required at /oauth/token; state was bare
base64url-JSON without integrity protection.
Fix: /oauth/authorize now rejects requests without code_challenge
or with code_challenge_method != S256 (the discovery doc already
advertised S256-only). /oauth/token requires code_verifier for
grant_type=authorization_code. state is HMAC-SHA256 signed with
STATE_HMAC_KEY (env, fallback per-instance random), payload includes
iat, /oauth/callback rejects tampered or >10-min-old state in
constant-time compare. Bicep adds a secure `stateHmacKey` param;
unset is acceptable for single-replica deployments.
#5 Structured 5xx leak in error-handler (Low)
ApiError.parsed branch flowed all 5xx detail (error / hint /
exceptionType) through to the MCP client. exceptionType in
particular leaks CLR class names that have no business reaching a
downstream caller.
Fix: status>=500 branch moved before the parsed-detail branch and
applies generic-message sanitization to structured 5xx too. Only
correlationId and errorCode survive — those are operational
handles for an operator to pivot on, not internals.
Tests: src/__tests__/security-guards.test.ts. The pre-existing
"preserves structured 5xx" case is inverted to assert sanitization.
16 new tests cover the cache-key fix, the registry caps, and the
state HMAC roundtrip / tamper / expiry / replay paths. 47/47 green
under `vitest run src/__tests__/security-guards.test.ts`. The
tools-*.test.ts integration suites still require AUTOPILOT_API_TOKEN
to run (pre-existing).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
V1 carried the same two bugs as V2: the per-user profile enumeration filter
only matched S-1-5-21-* SIDs, silently skipping the AAD/MSA cloud SIDs
(S-1-12-1-*) that Win10 1903+ uses for AAD-joined users — so on Win11 AAD
boxes the per-user Uninstall hive of the actually-logged-in user was never
read. Additionally, CollectFromKey logged "{source}\{subKeyPath}" which
mashed the telemetry tag (HKU_<RID>) with the relative path, producing a
non-path string like "HKU_1000\S-1-5-21-...\SOFTWARE\..." in debug logs.
Same patch as V2 commit 6c4f7250: extend SID filter to include S-1-12-1-*,
log rootKey.Name (HKEY_USERS / HKEY_LOCAL_MACHINE / ...) as the path prefix
and report the telemetry tag separately as "(source=...)".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Filter in CollectFromAllUserProfiles now also accepts S-1-12-1-* SIDs (Microsoft Account / Azure AD cloud SIDs on Win10 1903+ AAD-joined devices); previously only S-1-5-21-* matched, so the per-user Uninstall hive of the actually-logged-in AAD user was silently skipped on Win11 AAD-joined machines.
- CollectFromKey log line now uses rootKey.Name (HKEY_LOCAL_MACHINE / HKEY_USERS / ...) as the path prefix and reports the telemetry tag separately as "(source=...)". Removes the misleading "HKU_1000\S-1-5-21-...\SOFTWARE\..." Frankenstein-path that mashed the label with the subkey path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Skill now correctly distinguishes customDimensions (string/bool) from
customMeasurements (number) so future SLA/global_search queries don't
fall into the false-zero trap on numeric props like months.
The violator table on /sla linked to /session/${tenantId}/${sessionId},
which 404s — the canonical route is /sessions/[sessionId] (single
segment, plural). Same convention as /diagnosis, /admin and the
inspector. Restores navigation and makes sla_violator_opened actually
fire.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User typed a tenant filter without clicking Submit, then clicked Next →
fetchSessionsBatch read the live tenantIdFilterRef and built a URL with the
typed filter, but the continuation token in state was issued before the typing
under a different filter scope. Backend recomputed fingerprint, mismatched →
400 filter_mismatch.
Track the submitted filter in a separate ref, updated only via
refetch/refetchWith/initial-fetch/global-mode-toggle. The live ref keeps
driving client-side display via effectiveSessions; backend pagination always
queries under the scope its continuation was issued for.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Oliver Kieselbach