Files
AutopilotMonitor/docs/backend/ops-events-query-surface.md
T
Oliver Kieselbach d763a885e2 Ops alert channels, ops-events server-side filters, and generated MCP tool vocabularies (#140)
* feat(backend,web): Telegram als Channel-Provider + ein einziger Dispatch-Pfad

Notifications hatten zwei Sendewege: WebhookNotificationService.SendToChannelsAsync fuer
tenant-konfigurierte Channels, und TelegramNotificationService direkt aus dem Ops-Alert-Pfad.
Ein neuer Provider haette beide anfassen muessen.

Neu: NotificationChannelDispatcher ist DIE Channel-Sende-API und routet pro Provider —
WebhookProviderType.Telegram zum Bot, alles andere als gerenderten Webhook-POST.
SendToChannelsAsync wurde aus WebhookNotificationService entfernt statt dupliziert, damit es
keinen zweiten Pfad gibt; alle sieben Call-Sites ziehen um (EventIngestProcessor, RegisterSession,
ReportDistress, SlaBreachEvaluation, AnalyzeOnEnrollmentEnd, TestWebhookNotification).
SendOpsAlertAsync/SendNotificationAsync sind virtual (Test-Seam), TelegramNotificationService
bekommt SendAlertWithResultAsync fuer die Test-Endpoints und RenderAlertText als testbaren Kern.

Telegram (=40) ist der Sonderfall: kein Webhook. Der Channel-Url traegt die CHAT-ID, der Bot-Token
gehoert der Plattform (PreviewConfig). Ein Tenant-Admin wuerde durch UNSEREN Bot senden — deshalb
ist das Gate eine Server-Regel, keine versteckte Dropdown-Option:
ValidateTelegramChannelGate sitzt in TenantConfigValidation.ValidateModel und deckt damit BEIDE
Schreibpfade ab (Full-Model-PUT und den MCP-Field-Patch ueber TenantConfigPatchService) — in der
Update-Function allein waere der Patch-Pfad daran vorbeigelaufen. Ein unveraenderter, GA-angelegter
Channel passiert das Gate, damit ein Tenant-Admin unrelated Config weiter speichern kann.
Telegram-Ziele ueberspringen SsrfGuard.ValidateWebhookUrlFormat (eine Chat-ID ist keine URL und
wuerde immer failen) und werden stattdessen als numerische Id oder @username geprueft.

Mitgeliefert, weil der Dispatcher und die Ops-Seite sie brauchen: NotificationChannel.RedactList /
RestoreRedactedList als geteilte Helfer hochgezogen (TenantConfiguration delegiert jetzt dorthin,
statt eine zweite Kopie zu pflegen), AdminConfiguration.OpsNotificationChannelsJson +
GetOpsNotificationChannels() mit Legacy-Slot-Synthese, OpsAlertRule.NotifyChannelIds/IncludePayload,
NotificationAlert.DataJson (nur GenericJsonRenderer serialisiert es, additiv in schemaVersion 1.0).

Web: der Channel-Editor liegt jetzt in components/notifications/ChannelEditor.tsx und wird von
Tenant- und Admin-Seite geteilt (showEventToggles trennt die Faelle). Telegram erscheint im
Provider-Dropdown nur fuer Global Admins (Prop aus useAuth().user?.isGlobalAdmin, Hausmuster aus
SectionAutopilotValidation); ein bereits konfigurierter Telegram-Channel bleibt fuer alle sichtbar,
damit ein Speichern ihn nicht still verwirft.

Tests: TelegramChannelProviderTests (27) — Dispatcher-Routing inkl. "Telegram erreicht NIE den
SSRF-Pfad", Chat-ID-Validierung, GA-Gate auf beiden Schreibpfaden. Mutationsgeprueft: Gate
invertiert => 5 rot, zurueck => 27 gruen. OpsNotificationChannelsTests + OpsAlertPayloadTests
decken die mitgelieferten Modell-Additionen ab. TestNotifications ist der gemeinsame inerte Stub,
der die 17 bestehenden OpsAlertDispatchService-Konstruktionen ersetzt.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(backend,web): Ops-Alerts routen auf benannte Channels statt auf drei feste Slots

OpsAlertRule hatte kein Ziel. Jede matchende Regel ging an alle drei fest verdrahteten Slots in
AdminConfiguration (Telegram-ChatId / Teams-URL / Slack-URL), und nur die ERSTE matchende Regel
wurde ueberhaupt ausgewertet (FirstOrDefault). Ein Conversion-Signal an ein Sales-Team zu binden
hiess also: das Sales-Team bekommt auch jede Security-Regel.

Jetzt: AdminConfiguration.OpsNotificationChannelsJson haelt eine benannte Channel-Liste (dasselbe
NotificationChannel-Modell wie tenant-seitig, nur anderer Scope), und OpsAlertRule.NotifyChannelIds
bindet Regeln daran. Dispatch wertet ALLE matchenden Regeln aus (Where statt FirstOrDefault) und
vereinigt ihre Ziele.

Rueckwaertskompatibel ohne Migrationsjob:
- Leere Channel-Liste => GetOpsNotificationChannels() synthetisiert je konfiguriertem Legacy-Slot
  einen Channel mit stabiler Id (legacy-telegram heisst "Autopilot Monitor IT Engineer") und
  uebernimmt die Legacy-Enabled-Flags — ein bewusst abgeschalteter Provider bleibt abgeschaltet.
- GetAdminConfigurationFunction projiziert DIESELBE Synthese in das Read-Model, damit der Editor
  zeigt, was heute tatsaechlich Alerts empfaengt, und der erste Save exakt das materialisiert.
  Die Alternative waere eine zweite Kopie der Synthese-Regeln im Web gewesen, die vom Dispatcher
  wegdriftet.
- Leere NotifyChannelIds = alle aktivierten Channels. Das ist, was jede heute existierende Regel
  traegt, also verhaelt sich am Deploy-Tag nichts anders.
- Ids, die nicht mehr aufloesen, werden verworfen statt auf Broadcast zurueckzufallen: eine Regel,
  deren einziger Channel geloescht wurde, muss NICHTS erreichen, nicht alles.

Die Legacy-Felder bleiben als Migrationsquelle stehen und werden von der Alerts-Section nicht mehr
geschrieben — aufraeumen erst, wenn produktiv Channels stehen.

Sicherheit: OpsNotificationChannelsJson ist secret-tragend und steht in der Deny-List von
RedactedCopyForReader() (ConfigRedactionTests bewacht Drift). Der Admin-PUT hat bisher keinen
Redacted-Restore — er bekommt jetzt einen fuer die Channel-Liste, damit ein zurueckgespieltes
Reader-View nicht "***REDACTED***" als Chat-ID persistiert, plus die strukturelle Validierung, die
auch die Tenant-Liste passiert.

Neuer Endpoint POST /api/global/config/test-ops-channel (GlobalAdminOnly, im Policy-Catalog
registriert). Kein neues Response-DTO: TestWebhookNotificationResponse hat identische Semantik und
wird geteilt — nur der Doc-Kommentar nennt jetzt beide Routen.

Web: die "Alert Providers"-Karte weicht der Channel-Liste (geteilter ChannelEditor, im Admin-Kontext
mit Telegram und ohne die tenant-spezifischen Event-Toggles). Jede Regel bekommt einen
Channel-Picker; leere Auswahl rendert als "all channels", nicht als Warnung — hier ist das der
gueltige Default. Die Bindungslogik liegt testbar in opsChannelRouting.ts und spiegelt
ResolveTargets: der erste explizite Pick startet vom LEEREN Set, sonst liesse sich eine
Broadcast-Regel nie verengen. handleSaveOpsAlertConfig nimmt jetzt ein Objekt statt elf
positionaler Argumente — bei einem Dutzend Feldern schreibt ein vertauschtes Paar sonst still eine
Webhook-URL in den falschen Provider-Slot.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(backend): TenantTrialStarted + Event-Payload optional pro Regel

Der Trial-Start war stumm. POST config/{tenantId}/trial schrieb nur einen Audit-Entry, und der
GA-Pfad (PATCH plan) feuerte nur beim Downgrade — also war genau der Conversion-Moment, den
Sales/Support interessiert, das einzige Plan-Ereignis ohne Ops-Event.

RecordTenantTrialStartedAsync (Tenant/Info) feuert jetzt aus BEIDEN Schreibpfaden, unterschieden
per selfService-Flag, und jeweils NACH dem persistierten Save (SaveConfigurationAsync ist
fail-loud), damit kein Alert eine Trial ankuendigt, die nicht gespeichert wurde. Der GA-Pfad
haengt an der aufgezeichneten Aenderung, nicht am Request: ein reiner planTier-Edit bleibt still,
ein idempotenter Save mit demselben Datum ebenfalls, und das Beenden einer Trial (explizit null)
ist ein Downgrade und bleibt bei TenantPlanDowngraded.

Payload-Zustellung: WriteAsync reicht die serialisierten Details an DispatchAsync durch — aber ein
Channel bekommt sie nur, wenn eine Regel OpsAlertRule.IncludePayload setzt. Default AUS, bewusst:
ein Ops-Alert ist ein "da ist was, schau nach"-Signal, die meisten Payloads sind im Chat Rauschen,
und manche tragen Felder, die die Baseline nie traegt (Tenant-Domain, Contact-Adresse). Das
Informationsniveau aller bestehenden Regeln bleibt damit unveraendert.

ResolveTargets liefert deshalb zwei Gruppen. Ein Channel, den sowohl eine Payload-Regel als auch
eine schlichte Regel adressiert, landet nur in der Payload-Gruppe: eine Nachricht pro Channel, und
ein unbeteiligtes Geschwister-Match kann ein bewusstes Opt-in nicht aushebeln.

Fuer den angereicherten Alert flacht BuildAlert die Top-Level-Skalare in Facts ab (max 12, Werte
<= 256 Zeichen, Nulls und Verschachteltes raus), damit Card-Formate und der Telegram-Klartext
dieselbe Information tragen wie der GenericJson-Consumer unter "data". EventType wird immer
gesetzt — derselbe Wert steckt schon im "Event"-Fact, also keine zusaetzliche Information, nur ein
stabiler Schluessel zum Verzweigen.

Tests: TenantTrialStartedEventTests (8) — aufgezeichnete Event-Form inkl. Payload-Feldern, und die
Emissionsbedingung des GA-Pfads gegen ApplyPlanChanges gepinnt (greift bei Grant und Verlaengerung,
schweigt bei Trial-Ende, planTier-only und idempotentem Save). Die HTTP-Entry-Points bleiben
ungetestet — Suite-Konvention ist, die reinen Kerne zu testen.

Docs: OKF-Concept backend/ops-notification-channels.md. Registrierung in docs/index.md +
docs/log.md fehlt bewusst — beide Dateien tragen noch fremdes uncommittetes WIP (ops-events
Query-Surface), und ein Teil-Commit darin haette es mitgenommen.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(backend,mcp): ops-events server-side field filters + generated MCP tool vocabularies

The ops-events read could only narrow by category and date window, so answering
"did any AgentEmergencyBreak fire yesterday?" meant pulling a whole category and
filtering it client-side — and for the MCP tool that client is a language model,
so the question cost a category-sized response in model context.

Ops-events filter surface (mirrors the audit-log filter pattern one-for-one):
- OpsEventQueryFilters (Shared) + OpsEventFilterRequest (query -> filters +
  pagination extras) + TableOpsEventRepository.AppendFieldFilters (filters ->
  OData clauses). BOTH filter builders call the appender — the single-category /
  unpaged one and the per-partition fan-out builder that adds the RowKey bound —
  so "no category named" cannot silently return unfiltered rows.
- minSeverity expands to an OR-set, not a range: Table Storage compares strings
  lexicographically and the severity names sort Critical < Error < Info <
  Warning, so `Severity ge 'Warning'` would drop exactly the alert-worthy half.
  The ladder now lives in OpsEventSeverity.Rank, shared with
  OpsAlertDispatchService, so "Warning and above" in a read means what it meant
  when the alert rules fired.
- severity is normalised onto the canonical vocabulary and an unknown value is a
  400: `eq` is case-sensitive, so "?severity=warning" would otherwise return an
  empty page that reads exactly like "no such events happened". eventType stays
  verbatim — stored rows outlive the vocabulary, so a retired type must remain
  searchable in the historical window.
- Every filter value is folded into the continuation fingerprint and echoed on
  nextLink; extras are appended AFTER the pre-existing category/tenantId
  discriminators and skipped when empty, so tokens minted before the filters
  existed still fingerprint identically. The tenantId drill deliberately stays
  in memory (OrdinalIgnoreCase; an OData eq would lose rows on casing).
- MCP get_ops_events gains eventType / severity / minSeverity plus a `days`
  shorthand resolved client-side to a concrete dateFrom (a backend-side "last N
  days" would re-resolve "now" on every page and blow the token fingerprint);
  an explicit bound always wins over the shorthand.

Fixed alongside: the paged cross-category fan-out skipped Platform (Azure
Monitor alerts) because its category list had drifted from OpsEventCategory —
written from day one, invisible to every paged reader, still visible on the
unpaged full-table path. The list is now OpsEventCategory.All plus a reflection
coverage test.

That drift had a structural cause, so the same round closes it for the MCP as a
whole: wire TYPES reached the MCP from day one, but vocabularies did not — types
are erased at runtime and z.enum() needs values, so every tool schema hand-typed
its own copy with nothing to compare against. The copies had drifted:
search_sessions offered 5 of 8 session statuses (no AwaitingUser, no Incomplete
— both real stored statuses), both event readers offered 4 of 6 severities (no
Debug, no Trace), get_ops_events named 6 of 7 ops categories.

- The codegen now emits src/generated/wire-vocabularies.generated.ts (VALUES:
  session statuses, event severities, ops categories/severities/types,
  annotation lanes/verdicts) from the manifest sections listed in
  MCP_VOCABULARIES; SharedManifestParityTests exports the two new ops
  vocabularies. Freshness is pinned twice — byte-equality against a fresh
  codegen run in the MCP suite, and the shared-manifests-in-sync CI job.
- Tool schemas derive their z.enum() from it and descriptions interpolate the
  list instead of spelling it.
- vocabulary-drift.guard.test.ts ratchets what is left: every inline z.enum([..])
  must sit on a baseline naming why it has no backend owner (MCP-local knobs,
  plus rule type / connection type / CVE risk, which have no C# constants class
  yet). A new hand-typed list fails, and so does a stale baseline entry.
- OpsEventTypes (Shared) gives the 77 ops event types a canonical home: the
  OpsEventService write sites use the constants (a raw literal now fails
  OpsEventTypeDualRegisterTests, which moved from regex-scraping to reflection),
  the manifest exports them, and get_resource(name="ops_event_types") serves
  categories + severities + types to the model from the generated copy.

Suites: Backend 5427, MCP 629 + tsc, Web tsc + vitest 965, generator idempotent.

Deploy order: backend BEFORE the MCP redeploy — an MCP that sends filters to an
older backend gets silently unfiltered rows back and presents them as filtered.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: register the ops notification channels doc in the index and log

The concept doc landed with the feature commits; its index entry and log entry
were still uncommitted in the working tree.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-01 16:57:41 +02:00

7.7 KiB

type, title, description, resource, tags, timestamp
type title description resource tags timestamp
Concept Ops-Events Query Surface The server-side filter surface of GET /api/global/ops-events (category, eventType, severity/minSeverity, tenant drill, date window), why every filter is folded into the continuation fingerprint, why minSeverity expands to an OR-set, and why the paged cross-category read has to fan out over the complete category vocabulary. /src/Backend/AutopilotMonitor.Functions/Functions/Admin/GetOpsEventsFunction.cs
backend
table-storage
ops-events
pagination
mcp
2026-09-01T00:00:00+02:00

Problem

OpsEvents is the platform operator's log: consent flows, maintenance runs, security blocks, tenant offboards, agent emergency breaks, SLA and Azure Monitor alerts. It is partitioned by category (PartitionKey), with a reverse-tick RowKey for newest-first ordering.

Until 2026-09-01 the read endpoint could only narrow by category and date window. Every other question — "did any AgentEmergencyBreak fire yesterday?", "show me everything at Error or worse" — had to be answered by pulling the whole category and filtering it on the client. For the MCP tool that client is a language model, so an event-type question cost a category-sized response in model context.

Schema

GET /api/global/ops-events (GA + Global Reader; TenantScoping.None, so a delegated admin cannot reach it at all):

Param Where it is applied Notes
category PartitionKey eq — indexed One partition; omit to fan out over all of them
eventType EventType eq — server-side scan Exact and case-sensitive; Table Storage cannot fold case
severity Severity eq — server-side scan Normalised onto the canonical vocabulary before use
minSeverity OR-set over Severity — server-side scan Threshold: that level and everything above
dateFrom / dateTo RowKey range — indexed Both absent ⇒ backend resolves the last 30 days
tenantId in-memory, post-fetch Drill, not an isolation boundary — see below
pageSize / continuation — pageSize opts into pagination; absent ⇒ full window, no cap

The filter surface lives in two shared pieces so no path can honour a different one: OpsEventFilterRequest (query → OpsEventQueryFilters + pagination extras) and TableOpsEventRepository.AppendFieldFilters (filters → OData clauses), which both filter builders call — the single-category/unpaged builder and the per-partition fan-out builder that adds the RowKey gt bound. This mirrors AuditLogFilterRequest / AuditLogQueryFilters on the audit endpoints one-for-one.

Why minSeverity is an OR-set, not a range

Table Storage compares strings lexicographically, and the severity names sort Critical < Error < Info < Warning — nothing to do with severity order. A Severity ge 'Warning' clause would return Warning only, silently dropping Error and Critical, which is precisely the alert-worthy half. minSeverity therefore expands to (Severity eq 'Warning' or Severity eq 'Error' or Severity eq 'Critical'). The ladder itself lives in OpsEventSeverity.Rank and is shared with OpsAlertDispatchService, so "Warning and above" in a read means exactly what "Warning and above" meant when the alert rules decided to fire.

minSeverity=Info emits no clause at all — it is the floor, so the clause would only cost query length.

Why severity is normalised but eventType is not

Both columns are matched with eq, which is case-sensitive. Severity has a closed vocabulary of four values, so ?severity=warning is normalised to Warning, and anything outside the vocabulary is a 400 — an unnormalised value would return an empty page that reads exactly like "no such events happened".

Event types have a declared vocabulary too (OpsEventTypes, reflected into the shared manifest and served to the model as get_resource(name="ops_event_types")), but the filter deliberately does not validate against it: stored rows outlive the vocabulary, so a type that was retired after a refactor must stay searchable in the historical window. The value is passed verbatim and a typo legitimately returns zero rows — the catalog is there so nobody has to guess.

Continuation fingerprint

Every filter value is appended to the pagination extras, which are both folded into the continuation-token fingerprint and echoed on nextLink. Consequences:

  • a token minted for one filter cannot page a different one (it is rejected with a 400),
  • the follow-up request is self-contained — the resolved date window and the filters travel on the link, so no page re-resolves "now",
  • extras are appended after the pre-existing category / tenantId discriminators and skipped when empty, so tokens minted before the field filters existed still fingerprint identically.

Because severity values are normalised before fingerprinting, ?severity=error and ?severity=Error share one token instead of minting two that each reject the other.

The tenant drill stays in memory

?tenantId= is filtered post-fetch (GetOpsEventsFunction.FilterByTenant) because OpsEvents is partitioned by category, and — deliberately — because the in-memory comparison is OrdinalIgnoreCase while an OData eq would be case-sensitive: stored tenant casing is not guaranteed, so pushing this one down would lose rows. It is drill correctness, not isolation: every caller that reaches this route already has full cross-tenant scope. Pages therefore can report fewer items than pageSize when the drill is narrow.

The category vocabulary must be complete

The paged cross-category read does not issue one cross-partition query (Azure would page it by PartitionKey asc, so the first page would come entirely from the alphabetically first category). It fans out per partition over a fixed list and merge-sorts. That list is therefore load-bearing: a category that is written but not listed is invisible to every paged reader while the unpaged full-table path still returns it. Platform (Azure Monitor alerts relayed through the ops alert webhook) was missing that way from its introduction until 2026-09-01.

The list is now owned by OpsEventCategory.All, and OpsEventCategoryCoverageTests reflects over the constants to fail the build when a new category is not added to it.

Examples

# One event type, last 24h, cross-category
GET /api/global/ops-events?eventType=AgentEmergencyBreak&dateFrom=2026-08-31T12:00:00Z&pageSize=200

# Everything alert-worthy in the maintenance partition this month
GET /api/global/ops-events?category=Maintenance&minSeverity=Error&dateFrom=2026-08-01T00:00:00Z

MCP get_ops_events exposes the same surface plus a days shorthand, which it resolves to a concrete dateFrom client-side — a backend-side "last N days" would re-resolve "now" on every page and blow the token fingerprint. An explicit dateFrom/dateTo always wins over the shorthand.

Citations

  • /src/Backend/AutopilotMonitor.Functions/Functions/Admin/GetOpsEventsFunction.cs
  • /src/Backend/AutopilotMonitor.Functions/Functions/Admin/OpsEventFilterRequest.cs
  • /src/Backend/AutopilotMonitor.Functions/DataAccess/TableStorage/TableOpsEventRepository.cs
  • /src/Shared/AutopilotMonitor.Shared/DataAccess/IOpsEventRepository.cs
  • /src/Shared/AutopilotMonitor.Shared/DataAccess/OpsEventTypes.cs — the declared event-type vocabulary
  • /src/Backend/AutopilotMonitor.Functions.Tests/OpsEventFieldFilterTests.cs
  • /src/Backend/AutopilotMonitor.Functions.Tests/OpsEventCategoryCoverageTests.cs
  • /src/McpServer/autopilot-monitor-mcp/src/tools/admin.ts
  • Business Timestamps — why the date window filters on the RowKey