Fresh cross-tenant usage computes took 20-43s server-side but the web app's
default 30s fetch timeout aborted them client-side (HTTP 499), and every
retry stacked another concurrent full compute.
Backend:
- UsageMetricsService: cache 5→15min, single-flight per days-window (retries
join the in-flight compute), the four source scans run concurrently via
Task.WhenAll, statics converted to instance fields (singleton, testable)
- New column-projected queries for the usage compute only:
GetUsageWindowSessionsAsync ($select 13 consumed columns, drops
FailureSnapshotJson & co.) and GetAppInstallRefsAsync ($select
SessionId+AppName, lean SessionAppRef DTO); GetSessionsByDateRangeAsync
is untouched for its other consumers
- Projection equivalence pinned by UsageMetricsProjectionEquivalenceTests
(incl. the WhiteGlove Part-2 duration branch via IsPreProvisioned/
ResumedAt); single-flight/cache/failure-eviction pinned by
UsageMetricsServiceSingleFlightTests
Web (SectionPlatformUsage):
- 180s AbortSignal for this call (backend keeps computing and caches)
- Calculating card with elapsed time and a progress bar sized from the last
fresh-compute duration (localStorage, cache hits excluded, capped at 95%)
- Friendly timeout error explaining the retry will hit the server cache
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fail-closed entitlement layer (FeatureEntitlementCatalog + TenantEntitlementService): edition resolves at read time (PlanTier == 'enterprise' OR active trial; legacy free/pro => Community, no migration).
- Plan/trial endpoints: PATCH config/{tid}/plan (GA: tier + grant/extend/end trial, audited, cache-safe) and POST config/{tid}/trial (tenant-admin self-service 30d, once per tenant, 409 after). Generic PUT preserves plan/trial fields for all callers. Fixes SetPlanTier writing past the config cache without audit/UpdatedBy.
- Rate limits: Enterprise floor 150/min for BOTH user (portal/JWT) and device (agent/cert) paths - floor raises the global default, an explicit per-tenant override still wins; MSP users follow their home tenant.
- Retention cap: Community 90d / Enterprise 365d, clamped in the retention fanout (warn log) + non-GA write validation 7..cap (only on changed values; 0=infinite stays GA-only).
- Delegated admin (MSP): Enterprise gate on the delegated admin's HOME tenant (JWT tid, fail-closed on unknown) - managed target tenants may be any edition. Grant-time target check removed; inert grants are suppressed at resolve time (single choke point in GetScopeAsync, applied after the UPN-keyed cache).
- MCP quota: McpQuotaService + McpQuotaEnforcementMiddleware (X-Client-Source: mcp only, GA exempt) enforce daily/monthly budgets - per-user plan override > tenant edition plan; limits from SectionUsagePlans with catalog fallbacks (community 100/3000, enterprise 1000/20000); 429 with X-MCP-Quota-* headers + Retry-After; fail-open on counter reads, fail-closed on plan resolution. /api/metrics/mcp-usage/me reports effective plan + counters.
- Trial visibility: TrialExpirySweepFunction (daily 03:30 UTC, informational) emits TenantTrialExpired (Warning, 24h look-back) + TenantTrialExpiring (Info, <=3d heads-up); dual-registered in web OPS_EVENT_TYPES.
- Web: EditionBadge + one-time trial CTA in settings header, Plan & Trial card in the GA tenant edit modal (own PATCH save), retention cap hints from entitlements, pure lib/edition.ts (vitest), feature-flags surface edition/trial/entitlements; MCP TENANT_SAFE_FIELDS + trial fields.
Verified: solution build clean; backend 2653/2653 tests; web tsc + 516/516 vitest; MCP server tsc + tests green.
Two classes of read-efficiency fixes on hot/aggregation paths. No response
shape or value changes — DTOs untouched, all 2541 backend tests pass.
P2 — select projections on full-mirror SessionsIndex scans (bandwidth only):
- GetMetricsSummaryAsync: cross-partition status tally now projects
[PartitionKey, Status] instead of the ~40-column mirror row (incl. the
large FailureSnapshotJson) -> ~10-20x less transfer.
- QuickSearchSessionsAsync: typeahead scan projects only the matched fields
(+ RowKey for the SessionId fallback) instead of the full row, for up to
QuickSearchMaxScannedRows rows per request.
P3 — push the existing sinceUtc window floor server-side at 8 AppInstall
call sites across 5 files (previously each dematerialized the whole table /
partition and filtered to the window in memory):
- UsageMetricsService (global + tenant): now.AddDays(-days)
- MaintenanceService.Aggregation (global + tenant, per aggregated day): targetDate
- SlaBreachEvaluationService (current ISO week): now.AddDays(-8) margin
- SlaMetricsService: startDate (matches the in-memory lower bound exactly)
- Get(Global)GeographicLocationSessionsFunction: cutoff
Each floor equals the session-window floor the apps are joined against
(app.StartedAt >= session.StartedAt >= floor), so results are unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The app dashboard and metrics endpoints loaded entire AppInstallSummaries
partitions (tenant scope) or the full table (global scope) and filtered
StartedAt >= cutoff in memory, even for days=30. The device-model join then
issued one serial GetSessionAsync point-read per distinct session.
- Add optional DateTime? sinceUtc to GetAppInstallSummariesByTenantAsync /
GetAllAppInstallSummariesAsync: when set, a server-side StartedAt ge filter
scopes the scan so only in-window rows are deserialized/transferred. Default
null preserves the legacy full scan for callers that need all history.
- AppsAnalyticsHelper.LoadSummariesAsync derives sinceUtc from days; all six
/api/apps/* functions pass it. The four metrics/app + metrics/geographic
functions pass their existing cutoff. The in-memory Where stays as the exact
sub-second trim (the OData filter is second-granular). MCP get_app_install_metrics
benefits automatically (days already flows through; response shape unchanged).
- Replace the serial session point-read loop with a bounded-concurrency
(SemaphoreSlim=10) LoadSessionLookupAsync, shared by the analytics and
sessions builders.
AppInstallSummaries is already bounded by the session-retention cascade
(DeletionManifestBuilder) + tenant offboarding, so no separate table retention
is added. Full Backend suite: 2406 passed / 0 failed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GlobalReader = cross-tenant READ visibility like a Global Admin, but no mutating
operations and no platform settings. Provisioned exactly like a GA: a GlobalAdmins
table row with Role=GlobalReader (empty/legacy Role = GlobalAdmin, back-compat).
Semantics are ADDITIVE — a user who is also their own tenant's Admin keeps edit
rights on that tenant; a pure GlobalReader is read-only everywhere.
Backend:
- GlobalAdmins.Role column + GlobalAdminService.GetGlobalRoleAsync; IsGlobalAdminAsync
(incl. TableAdminRepository) delegates so a GlobalReader row never reads back as admin
- RequestContext.IsGlobalReader + HasGlobalScope
- Split GlobalAdminOnly into GlobalReadOrAdmin (cross-tenant reads) vs GlobalAdminOnly
(writes/settings) + TenantAdminOrGlobalReader for config-read
- PolicyEnforcementMiddleware: extracted testable DecideAsync seam; cross-tenant gate keys
off HasGlobalScope; additive own-tenant role carried so IsTenantAdmin stays correct
- Config secret redaction for readers (RedactedCopyForReader) + save-time placeholder
restore (RestoreRedactedSecretsFrom) so a redacted view can't blank real secrets
- raw table/log tools stay GlobalAdminOnly; SignalR global-admins read group admits HasGlobalScope
- /auth/me emits isGlobalReader (+ preview-gate bypass, no auto-admin); /auth/mcp emits globalRole
MCP: hasGlobalScope (GA or Reader) drives routing + tool catalog; raw secret-bearing tools
(query_table/list_tables/query_backend_logs) gated strict-GA only.
Web: AuthContext isGlobalReader + hasGlobalScope; reader reaches /dashboard + cross-tenant
views; platform Settings/Ops/Backups/Software GA-only (nav-hidden + requireGlobalAdmin route
layouts); mutating controls on mixed data pages disabled via useCanMutatePlatform; rules pages
read-only on foreign tenants; additive own-tenant edit preserved.
Tests: backend 2369, MCP 252, web 469 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fleet Health previously drained up to 200k raw sessions into the browser
(MAX_EAGER_PAGES=200 x 1000) over as many HTTP round-trips, then computed
7 aggregates in useMemo on the main thread. Move that aggregation to the
backend, mirroring the dashboard-stats server-aggregation path.
- MetricsMath.BuildFleetHealthPayload: single pass over the windowed
session list producing stats, daily timeline, top failure reasons, and
model health/slowest/top-failing breakdowns. Semantics ported verbatim
from the client (success rate over ALL sessions; avg duration over
non-in-progress sessions incl. failures), so displayed numbers are
unchanged. Typed FleetHealthMetrics DTO in Shared for a typed contract.
- GET /api/metrics/fleet-health (MemberRead) and
GET /api/global/metrics/fleet-health (GlobalAdminOnly, optional
?tenantId=) reuse the existing ISessionRepository window readers.
- Registered both routes in EndpointAccessPolicyCatalog (fail-closed).
- 9 xUnit tests covering the aggregation semantics.
Frontend swap to these endpoints lands in a follow-up commit. App-metrics
(slowest/failing apps) is already backend-aggregated and untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backend:
- Extract MetricsMath.BuildAppMetricsPayload as the single source for the
app-metrics aggregation (was duplicated verbatim across the tenant and
global functions) and add a Delivery Optimization rollup: per-app peer/MCC
offload bytes + overall deliveryOptimization summary (peers and MCC counted
separately, peerOffloadPercent credits both as not-from-CDN).
- New tenant-scoped GetTenantSoftwareInventoryFunction (GET metrics/software-
inventory, MemberRead, JWT tenant) + fail-closed policy-catalog entry.
MCP server:
- get_app_install_metrics: role-aware routing (global/metrics/app vs
metrics/app), surfaces the DO rollup.
- get_software_inventory: one role-aware tool; non-GA sees only its own tenant
(no tenantId/scope in schema), GA additionally gets tenantId + scope=unmatched.
New exported paginateInventory client-side pager.
Tests: MetricsMathAppPayloadTests (DO rollup/offload/sample-floor) + vitest
paginateInventory. Backend 2202/2202, MCP 220 passed, tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three data-hygiene fixes for the MCP metric tools, all driven by
incomplete/abandoned sessions distorting aggregates:
- get_usage_metrics: clamp each session duration to the shared 7-day
ceiling before avg/median/p95/p99, so stuck non-terminal sessions
(unclamped now-StartedAt wall-clock) no longer push p99 into weeks.
Surface SampleCount + ClampedSessionCount so the skew stays visible.
- get_metrics: tally Pending/Stalled into their own buckets and route
any other status into `other`, so the per-tenant buckets reconcile to
total by construction (closes the silent 214/134 gap). Extracted the
fold into SessionStatusBuckets.
- slowestApps: require >=3 successful installs via the shared
MetricsMath.SelectSlowestApps helper, so a single N=1 (often
unfinished, legacy pre-clamp) install cannot dominate the ranking;
also de-dupes the copy-pasted block across the tenant + global
app-metrics functions.
Adds xUnit regression cover for all three.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
get_platform_metrics: expose maxSessions (->&limit=), echo sessionLimit +
truncated. Backend already honored days+limit; the tool only sent days, so
results were silently capped at the newest 100 sessions.
get_geographic_sessions: backend (tenant+global) now matches structured
country/region/city against actual Geo* fields via FilterSessionsByFields
(case-insensitive, optional narrowing); 400 relaxed to require locationKey OR
country. MCP forwards structured filters and infers groupBy on the raw
locationKey path so country/region keys resolve (previously matched nothing).
locationKey path retained for the web UI.
get_geographic_metrics: blank Region/City for coarser groupings so a country
row is not mislabeled with an arbitrary sample city; keep Loc for map pins.
Tests: +GeographicDrilldownFilterTests (backend), +geo-platform-helpers (MCP).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Customer reported "From Peers 100% / Peer breakdown LAN 0 | Group 0 | Internet 0"
mid-download. Root cause: OS collector emitted LAN/Group/Internet/LinkLocal
breakdown only on the final do_telemetry event, not on download_progress events
during the download — so UI rendered 0/0/0 until completion. Empirical check of
917 do_telemetry events / 146 sessions / 13 tenants over 30 days confirmed IME
[DO TEL] log path is dead code (current IME version no longer logs it); OS
cmdlet is the sole live source.
Changes:
- AppPackageState (V2): new fields DoBytesFromLinkLocalPeers,
DoBytesFromCacheServer, DoCacheHost; UpdateDoTelemetry signature extended
with optional defaults so callers without the data pass 0/null.
- DeliveryOptimizationCollector: read BytesFromLinkLocalPeers + CacheHost
from cmdlet output; centralised event payload via BuildDoEventData() so
download_progress and do_telemetry are field-symmetric.
- ImeLogTracker.HandleDoTelemetry: parses new fields null-safe (older IME
JSON without them defaults to 0/null, no breakage).
- Persistence: PackageStateData DTO + ToPackageStateData / FromPackageStateData
mappers carry the new fields across agent restart.
- ImeLogTrackerAdapter.EmitDoTelemetry: emits new fields on do_telemetry,
including doCacheHost when MCC was used.
- Backend ingest (V1 + V2 paths) + AppInstallSummary model +
TableStorageService.Metrics Store/Map/preserve-from-existing extended for
the new columns.
- DoAggregator + GeographicApiModels + GetGeographicMetricsFunction surface
MCC + LinkLocal at location and per-session level.
- UI DownloadProgress.tsx: new "From Connected Cache" line with cache% and
hover host; LAN merges LinkLocal in breakdown ("local network"); 3-color
progress bar (Peers green / MCC purple / HTTP blue) computed by subtracting
CacheServer from HTTP to avoid double-counting (MCC serves over HTTP).
- UI sessions page: hover title now includes MCC bytes and LAN+LinkLocal sum.
Robustness: every new field defaults to 0 / null / "" on machines or paths
that don't supply them; no existing session display regresses.
Tests: backend 1058/1058, agent V2 1007/1007, web vitest 259/259, tsc clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Re-test post-deploy of #60 surfaced two minor follow-ups:
1. get_ops_events MCP tool: backend already accepts tenantId filter
(since C2), but the MCP tool schema didn't expose tenantId so Zod
silently dropped any caller-passed value. Adds tenantId to the input
schema and forwards it in the query. 2 LOC.
2. /api/global/metrics/summary: the global handler was hardcoded to
pass null (all tenants) and ignored ?tenantId=X. Adds the same
optional filter pattern used elsewhere — null when absent, scoped
when present. The tenant-scoped /api/metrics/summary route is
unchanged (JWT-tid still authoritative).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The full LocationSessionRow shape is ~40 fields per session — for
locations with dozens of sessions the response easily blew past 100KB
and tripped MCP token caps. Most fields aren't needed for the typical
"which sessions enrolled from here" triage flow.
Default is now a 17-field lean projection (sessionId, device basics,
status/duration, geo, app/DO summary). UI keeps the full shape via
?full=1 — single-line addition in lib/api.ts.
Also adds an optional tenantId filter on /api/global/metrics/geographic/
sessions so a GA can scope a single tenant's sessions for a location
without leaving the global endpoint.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Critical: GET /api/metrics/mcp-usage/user/{userId} (TenantAdminOrGA, no
TenantScoping) handed the oid to the repo without checking that returned
records belong to the caller's tenant. A tenant admin from tenant A could
read MCP usage records of any oid — including oids in tenant B.
McpUsageMetricsFunction.GetUserUsage now blocks 403 CrossTenantAccessDenied
when records are tenant-foreign and caller is not a Global Admin. The pure
decision logic lives in UsageCrossTenantGuard for unit-testability (9 tests
covering own/foreign/mixed/GA/empty/null/case-insensitive/empty-tid paths).
Also:
- MCP get_api_usage tool was calling /api/metrics/mcp-usage/{userId}
(without /user/), which is unregistered and returned 404 — corrected.
- list_blocked_devices description honestly states the endpoint is
GA-only on both variants (catalog policy, no change to the gate).
- days clamp [1, 365] added to /metrics/{app, geographic, geographic/sessions}
for parity with /metrics/summary, /metrics/usage, /metrics/platform.
- get_geographic_metrics, get_geographic_sessions zod schemas now constrain
days to int().min(1).max(365) — same pattern as the other tools.
- search/sessions, search/sessions-by-event, search/sessions-by-cve switched
limit handling from Math.Min(lim, 100) to Math.Clamp(lim, 1, 100); the old
form let a negative ?limit=-5 reach the repo.
Audit static analysis confirmed all other tenant-scoped endpoints route
tenantId through TenantHelper.GetTenantId(req) (JWT-only, ignores query),
so ?tenantId=<other> is silently dropped. Live-verified with an Operator
token: 14 cross-tenant query probes returned only the caller's tenant,
13 GlobalAdminOnly routes returned 403.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The three metrics MCP tools previously ignored the time-window: get_metrics never
read the days query param (returned all-time totals), get_usage_metrics didn't
expose days at all and the service was hardcoded to 90, get_platform_metrics
had a fixed 500-most-recent-sessions cap. A "last 7 days" question therefore
returned cross-tenant all-time aggregates (~1300) instead of the actual 7-day
count (~500).
End-to-end fix:
- Plumbing days through MetricsSummaryFunction → IMetricsRepository →
TableStorageService.GetMetricsSummaryAsync, with a server-side RowKey cutoff
on SessionsIndex (reuses the inverted-tick D19 prefix pattern; extracted as
ComputeCutoffRowKeyPrefix and the two pre-existing call sites consolidated).
- UsageMetricsService and PlatformMetricsService now accept int days, with
per-window in-memory caches and a WindowDays field echoed on the response.
- MCP zod schemas accept any integer 1-365 (replacing the [7,30,90] whitelist),
so callers can now ask for 5 or 12 days and get the right answer.
- Function-layer defaults stay at 90 for /usage and /platform routes (no
dashboard regression) and 30 for /summary (Agent API, MCP-only path).
Tests: 22 new xUnit cases across MetricsCutoffRowKeyTests,
UsageMetricsServiceWindowTests, PlatformMetricsServiceWindowTests covering
boundary, clamping, and end-to-end window semantics. Backend 808/808 green,
web vitest 208/208 green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace direct TableStorageService injection with ISessionRepository,
IMetricsRepository, and IMaintenanceRepository in 10 Function files.
Add GetSessionEventsAsync to ISessionRepository interface.
Migration in progress — more functions and services to follow.
https://claude.ai/code/session_01NHWJfPaa8TQHuPzMD7gG63
Oliver Kieselbach