mirror of
https://github.com/okieselbach/AutopilotMonitor.git
synced 2026-10-24 20:53:06 +00:00
729ee56bdee6600ea832d5fc86665d110e6d290d
54 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Oliver Kieselbach
|
729ee56bde |
security: assume-breach alarm for the Global-Admin-only surface (x29 phase 0)
The 2026-09-04 audit found no bypass of the GA boundary, but a failed attempt left only a Warning trace without identity, and a non-GA tools/call for a GA-only MCP tool never reached the backend at all. This makes the boundary observable without adding a second alarm writer (D-185): Backend - PrivilegedRouteDenied (Security): raised from the policy middleware's deny path for every 403 by an authenticated caller on a GlobalAdminOnly route. Critical for a caller without a platform role, Warning for a Global Reader. Details carry upn, oid, tid, client source, MCP tool name, correlation id, reason and policy. Throttled one hour per caller+path per instance (PrivilegedDenialReporter, IPrivilegedDenialReporter seam). PolicyResult now surfaces the denied tier; the DENIED log line gains oid, tid, clientSource and mcpTool. - GET global/raw/access-probe: no-op GlobalAdminOnly route whose denial IS the signal (AccessProbeResponse DTO, catalog entry, parity test). - Defense in depth: AppInsightsQueryFunction and both TableQueryFunction triggers re-check IsGlobalAdmin in the body (RawGlobalAdminGate); an empty RequestContext fails closed. - Dual-register: OpsEventTypes.All, OpsAlertRulesSection.tsx, regenerated manifests. MCP - ga-tool-probe.ts: before dispatch, a caller without the GlobalAdmin role naming a strictGa tool leaves an unconditional [mcp-security] line (tool, upn, scope) and fires the access probe with X-MCP-Tool-Name; the SDK answer stays "Tool not found". - GA_STRICT_TOOL_NAMES exported and pinned equal to the registered GA-minus-Reader set. - debug-session prompt names query_backend_logs/query_table only for strictGa (Readers were told to call tools they cannot see); tool_call_rejected lines carry scope + upn; config-tool comment corrected against the actual catalog tiers. Tests: reporter (severity, details, throttle, fail-soft), middleware policy surfacing and IsPrivilegedDenial/BuildPrivilegedDenial, Program.cs middleware order source scan, raw gate 403, probe catalog + JWT pins, wire parity; MCP probe unit tests, Global Reader over HTTP on both eras, runtime refusal fires the probe, prompt gate. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
db020789e1 |
chore: private submodule docs/ becomes internal/ (knowledge + decisions + work files + private instructions)
- .gitmodules: docs -> internal, repo renamed to Autopilot-Monitor-internal; the OKF bundle now lives at internal/docs/ - .claude/CLAUDE.md rewritten: where knowledge lives, working-file rules, hard rules; imports the private half from internal/CLAUDE.md (silently absent without submodule access) - .claude/architecture-principles.md moved into the internal bundle (reconciled against the code) - tasks/ (gitignored) retired; working files live in internal/work/ - doc path references docs/<bundle> -> internal/docs/<bundle> in code comments, skills gitlink, codeql paths, build script - rule descriptions no longer cite an internal doc path (customer-visible text); rules/dist regenerated - public tree no longer names private operator tooling - .claude/commands gitlink: skill docs point at the new paths |
||
|
Oliver Kieselbach
|
f833882e07 |
fix(ops): AgentEmergencyBreak ops event severity follows session state at break time
The agent's 48h emergency break is wall-clock, so devices powered on after a shelf period fire it days after the sweep already terminalized the session (2026-09-03 audit: 224 of 269 breaks in three days were such late cleanups from one kiosk tenant, all Warning, burying the real ones). - ReportAgentErrorFunction.ClassifyEmergencyBreak: session already terminal (Succeeded/Failed/Incomplete) => Info + lateCleanup=true; open, unknown or unreadable => Warning (historical semantics). Row read is best-effort. - OpsEventService.RecordAgentEmergencyBreakAsync takes the severity and stamps sessionStatusAtBreak/lateCleanup into message and details. - Timeline event agent_emergency_break stays Warning (classifier rule 3 keys on type). - SessionOwnerBindingObserver: document that the ops-event throttle is per Function instance (IMemoryCache) and deliberately so. - Tests for both branches, the fail-safe path and the pure rule; docs submodule bumped. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
7e82decc01 |
feat(plan): company name in the tenant contact profile, required with the contact address at the Pro trial entry point
CompanyName joins ContactEmail as the tenant contact profile: optional on Community, both required by the self-service trial start (one 409 ContactProfileRequired verdict naming every missing part). GA plan assignment stays soft (admin warning). Feature flags expose companyNameSet as a boolean only; TenantTrialStarted carries companyName. Web: Contact section field, Plan gate and dashboard banner name what is missing, admin panel/report/search know the company. Privacy page lists the new datum. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
d763a885e2 |
Ops alert channels, ops-events server-side filters, and generated MCP tool vocabularies (#140)
* feat(backend,web): Telegram als Channel-Provider + ein einziger Dispatch-Pfad Notifications hatten zwei Sendewege: WebhookNotificationService.SendToChannelsAsync fuer tenant-konfigurierte Channels, und TelegramNotificationService direkt aus dem Ops-Alert-Pfad. Ein neuer Provider haette beide anfassen muessen. Neu: NotificationChannelDispatcher ist DIE Channel-Sende-API und routet pro Provider — WebhookProviderType.Telegram zum Bot, alles andere als gerenderten Webhook-POST. SendToChannelsAsync wurde aus WebhookNotificationService entfernt statt dupliziert, damit es keinen zweiten Pfad gibt; alle sieben Call-Sites ziehen um (EventIngestProcessor, RegisterSession, ReportDistress, SlaBreachEvaluation, AnalyzeOnEnrollmentEnd, TestWebhookNotification). SendOpsAlertAsync/SendNotificationAsync sind virtual (Test-Seam), TelegramNotificationService bekommt SendAlertWithResultAsync fuer die Test-Endpoints und RenderAlertText als testbaren Kern. Telegram (=40) ist der Sonderfall: kein Webhook. Der Channel-Url traegt die CHAT-ID, der Bot-Token gehoert der Plattform (PreviewConfig). Ein Tenant-Admin wuerde durch UNSEREN Bot senden — deshalb ist das Gate eine Server-Regel, keine versteckte Dropdown-Option: ValidateTelegramChannelGate sitzt in TenantConfigValidation.ValidateModel und deckt damit BEIDE Schreibpfade ab (Full-Model-PUT und den MCP-Field-Patch ueber TenantConfigPatchService) — in der Update-Function allein waere der Patch-Pfad daran vorbeigelaufen. Ein unveraenderter, GA-angelegter Channel passiert das Gate, damit ein Tenant-Admin unrelated Config weiter speichern kann. Telegram-Ziele ueberspringen SsrfGuard.ValidateWebhookUrlFormat (eine Chat-ID ist keine URL und wuerde immer failen) und werden stattdessen als numerische Id oder @username geprueft. Mitgeliefert, weil der Dispatcher und die Ops-Seite sie brauchen: NotificationChannel.RedactList / RestoreRedactedList als geteilte Helfer hochgezogen (TenantConfiguration delegiert jetzt dorthin, statt eine zweite Kopie zu pflegen), AdminConfiguration.OpsNotificationChannelsJson + GetOpsNotificationChannels() mit Legacy-Slot-Synthese, OpsAlertRule.NotifyChannelIds/IncludePayload, NotificationAlert.DataJson (nur GenericJsonRenderer serialisiert es, additiv in schemaVersion 1.0). Web: der Channel-Editor liegt jetzt in components/notifications/ChannelEditor.tsx und wird von Tenant- und Admin-Seite geteilt (showEventToggles trennt die Faelle). Telegram erscheint im Provider-Dropdown nur fuer Global Admins (Prop aus useAuth().user?.isGlobalAdmin, Hausmuster aus SectionAutopilotValidation); ein bereits konfigurierter Telegram-Channel bleibt fuer alle sichtbar, damit ein Speichern ihn nicht still verwirft. Tests: TelegramChannelProviderTests (27) — Dispatcher-Routing inkl. "Telegram erreicht NIE den SSRF-Pfad", Chat-ID-Validierung, GA-Gate auf beiden Schreibpfaden. Mutationsgeprueft: Gate invertiert => 5 rot, zurueck => 27 gruen. OpsNotificationChannelsTests + OpsAlertPayloadTests decken die mitgelieferten Modell-Additionen ab. TestNotifications ist der gemeinsame inerte Stub, der die 17 bestehenden OpsAlertDispatchService-Konstruktionen ersetzt. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(backend,web): Ops-Alerts routen auf benannte Channels statt auf drei feste Slots OpsAlertRule hatte kein Ziel. Jede matchende Regel ging an alle drei fest verdrahteten Slots in AdminConfiguration (Telegram-ChatId / Teams-URL / Slack-URL), und nur die ERSTE matchende Regel wurde ueberhaupt ausgewertet (FirstOrDefault). Ein Conversion-Signal an ein Sales-Team zu binden hiess also: das Sales-Team bekommt auch jede Security-Regel. Jetzt: AdminConfiguration.OpsNotificationChannelsJson haelt eine benannte Channel-Liste (dasselbe NotificationChannel-Modell wie tenant-seitig, nur anderer Scope), und OpsAlertRule.NotifyChannelIds bindet Regeln daran. Dispatch wertet ALLE matchenden Regeln aus (Where statt FirstOrDefault) und vereinigt ihre Ziele. Rueckwaertskompatibel ohne Migrationsjob: - Leere Channel-Liste => GetOpsNotificationChannels() synthetisiert je konfiguriertem Legacy-Slot einen Channel mit stabiler Id (legacy-telegram heisst "Autopilot Monitor IT Engineer") und uebernimmt die Legacy-Enabled-Flags — ein bewusst abgeschalteter Provider bleibt abgeschaltet. - GetAdminConfigurationFunction projiziert DIESELBE Synthese in das Read-Model, damit der Editor zeigt, was heute tatsaechlich Alerts empfaengt, und der erste Save exakt das materialisiert. Die Alternative waere eine zweite Kopie der Synthese-Regeln im Web gewesen, die vom Dispatcher wegdriftet. - Leere NotifyChannelIds = alle aktivierten Channels. Das ist, was jede heute existierende Regel traegt, also verhaelt sich am Deploy-Tag nichts anders. - Ids, die nicht mehr aufloesen, werden verworfen statt auf Broadcast zurueckzufallen: eine Regel, deren einziger Channel geloescht wurde, muss NICHTS erreichen, nicht alles. Die Legacy-Felder bleiben als Migrationsquelle stehen und werden von der Alerts-Section nicht mehr geschrieben — aufraeumen erst, wenn produktiv Channels stehen. Sicherheit: OpsNotificationChannelsJson ist secret-tragend und steht in der Deny-List von RedactedCopyForReader() (ConfigRedactionTests bewacht Drift). Der Admin-PUT hat bisher keinen Redacted-Restore — er bekommt jetzt einen fuer die Channel-Liste, damit ein zurueckgespieltes Reader-View nicht "***REDACTED***" als Chat-ID persistiert, plus die strukturelle Validierung, die auch die Tenant-Liste passiert. Neuer Endpoint POST /api/global/config/test-ops-channel (GlobalAdminOnly, im Policy-Catalog registriert). Kein neues Response-DTO: TestWebhookNotificationResponse hat identische Semantik und wird geteilt — nur der Doc-Kommentar nennt jetzt beide Routen. Web: die "Alert Providers"-Karte weicht der Channel-Liste (geteilter ChannelEditor, im Admin-Kontext mit Telegram und ohne die tenant-spezifischen Event-Toggles). Jede Regel bekommt einen Channel-Picker; leere Auswahl rendert als "all channels", nicht als Warnung — hier ist das der gueltige Default. Die Bindungslogik liegt testbar in opsChannelRouting.ts und spiegelt ResolveTargets: der erste explizite Pick startet vom LEEREN Set, sonst liesse sich eine Broadcast-Regel nie verengen. handleSaveOpsAlertConfig nimmt jetzt ein Objekt statt elf positionaler Argumente — bei einem Dutzend Feldern schreibt ein vertauschtes Paar sonst still eine Webhook-URL in den falschen Provider-Slot. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(backend): TenantTrialStarted + Event-Payload optional pro Regel Der Trial-Start war stumm. POST config/{tenantId}/trial schrieb nur einen Audit-Entry, und der GA-Pfad (PATCH plan) feuerte nur beim Downgrade — also war genau der Conversion-Moment, den Sales/Support interessiert, das einzige Plan-Ereignis ohne Ops-Event. RecordTenantTrialStartedAsync (Tenant/Info) feuert jetzt aus BEIDEN Schreibpfaden, unterschieden per selfService-Flag, und jeweils NACH dem persistierten Save (SaveConfigurationAsync ist fail-loud), damit kein Alert eine Trial ankuendigt, die nicht gespeichert wurde. Der GA-Pfad haengt an der aufgezeichneten Aenderung, nicht am Request: ein reiner planTier-Edit bleibt still, ein idempotenter Save mit demselben Datum ebenfalls, und das Beenden einer Trial (explizit null) ist ein Downgrade und bleibt bei TenantPlanDowngraded. Payload-Zustellung: WriteAsync reicht die serialisierten Details an DispatchAsync durch — aber ein Channel bekommt sie nur, wenn eine Regel OpsAlertRule.IncludePayload setzt. Default AUS, bewusst: ein Ops-Alert ist ein "da ist was, schau nach"-Signal, die meisten Payloads sind im Chat Rauschen, und manche tragen Felder, die die Baseline nie traegt (Tenant-Domain, Contact-Adresse). Das Informationsniveau aller bestehenden Regeln bleibt damit unveraendert. ResolveTargets liefert deshalb zwei Gruppen. Ein Channel, den sowohl eine Payload-Regel als auch eine schlichte Regel adressiert, landet nur in der Payload-Gruppe: eine Nachricht pro Channel, und ein unbeteiligtes Geschwister-Match kann ein bewusstes Opt-in nicht aushebeln. Fuer den angereicherten Alert flacht BuildAlert die Top-Level-Skalare in Facts ab (max 12, Werte <= 256 Zeichen, Nulls und Verschachteltes raus), damit Card-Formate und der Telegram-Klartext dieselbe Information tragen wie der GenericJson-Consumer unter "data". EventType wird immer gesetzt — derselbe Wert steckt schon im "Event"-Fact, also keine zusaetzliche Information, nur ein stabiler Schluessel zum Verzweigen. Tests: TenantTrialStartedEventTests (8) — aufgezeichnete Event-Form inkl. Payload-Feldern, und die Emissionsbedingung des GA-Pfads gegen ApplyPlanChanges gepinnt (greift bei Grant und Verlaengerung, schweigt bei Trial-Ende, planTier-only und idempotentem Save). Die HTTP-Entry-Points bleiben ungetestet — Suite-Konvention ist, die reinen Kerne zu testen. Docs: OKF-Concept backend/ops-notification-channels.md. Registrierung in docs/index.md + docs/log.md fehlt bewusst — beide Dateien tragen noch fremdes uncommittetes WIP (ops-events Query-Surface), und ein Teil-Commit darin haette es mitgenommen. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(backend,mcp): ops-events server-side field filters + generated MCP tool vocabularies The ops-events read could only narrow by category and date window, so answering "did any AgentEmergencyBreak fire yesterday?" meant pulling a whole category and filtering it client-side — and for the MCP tool that client is a language model, so the question cost a category-sized response in model context. Ops-events filter surface (mirrors the audit-log filter pattern one-for-one): - OpsEventQueryFilters (Shared) + OpsEventFilterRequest (query -> filters + pagination extras) + TableOpsEventRepository.AppendFieldFilters (filters -> OData clauses). BOTH filter builders call the appender — the single-category / unpaged one and the per-partition fan-out builder that adds the RowKey bound — so "no category named" cannot silently return unfiltered rows. - minSeverity expands to an OR-set, not a range: Table Storage compares strings lexicographically and the severity names sort Critical < Error < Info < Warning, so `Severity ge 'Warning'` would drop exactly the alert-worthy half. The ladder now lives in OpsEventSeverity.Rank, shared with OpsAlertDispatchService, so "Warning and above" in a read means what it meant when the alert rules fired. - severity is normalised onto the canonical vocabulary and an unknown value is a 400: `eq` is case-sensitive, so "?severity=warning" would otherwise return an empty page that reads exactly like "no such events happened". eventType stays verbatim — stored rows outlive the vocabulary, so a retired type must remain searchable in the historical window. - Every filter value is folded into the continuation fingerprint and echoed on nextLink; extras are appended AFTER the pre-existing category/tenantId discriminators and skipped when empty, so tokens minted before the filters existed still fingerprint identically. The tenantId drill deliberately stays in memory (OrdinalIgnoreCase; an OData eq would lose rows on casing). - MCP get_ops_events gains eventType / severity / minSeverity plus a `days` shorthand resolved client-side to a concrete dateFrom (a backend-side "last N days" would re-resolve "now" on every page and blow the token fingerprint); an explicit bound always wins over the shorthand. Fixed alongside: the paged cross-category fan-out skipped Platform (Azure Monitor alerts) because its category list had drifted from OpsEventCategory — written from day one, invisible to every paged reader, still visible on the unpaged full-table path. The list is now OpsEventCategory.All plus a reflection coverage test. That drift had a structural cause, so the same round closes it for the MCP as a whole: wire TYPES reached the MCP from day one, but vocabularies did not — types are erased at runtime and z.enum() needs values, so every tool schema hand-typed its own copy with nothing to compare against. The copies had drifted: search_sessions offered 5 of 8 session statuses (no AwaitingUser, no Incomplete — both real stored statuses), both event readers offered 4 of 6 severities (no Debug, no Trace), get_ops_events named 6 of 7 ops categories. - The codegen now emits src/generated/wire-vocabularies.generated.ts (VALUES: session statuses, event severities, ops categories/severities/types, annotation lanes/verdicts) from the manifest sections listed in MCP_VOCABULARIES; SharedManifestParityTests exports the two new ops vocabularies. Freshness is pinned twice — byte-equality against a fresh codegen run in the MCP suite, and the shared-manifests-in-sync CI job. - Tool schemas derive their z.enum() from it and descriptions interpolate the list instead of spelling it. - vocabulary-drift.guard.test.ts ratchets what is left: every inline z.enum([..]) must sit on a baseline naming why it has no backend owner (MCP-local knobs, plus rule type / connection type / CVE risk, which have no C# constants class yet). A new hand-typed list fails, and so does a stale baseline entry. - OpsEventTypes (Shared) gives the 77 ops event types a canonical home: the OpsEventService write sites use the constants (a raw literal now fails OpsEventTypeDualRegisterTests, which moved from regex-scraping to reflection), the manifest exports them, and get_resource(name="ops_event_types") serves categories + severities + types to the model from the generated copy. Suites: Backend 5427, MCP 629 + tsc, Web tsc + vitest 965, generator idempotent. Deploy order: backend BEFORE the MCP redeploy — an MCP that sends filters to an older backend gets silently unfiltered rows back and presents them as filtered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: register the ops notification channels doc in the index and log The concept doc landed with the feature commits; its index entry and log entry were still uncommitted in the working tree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
5f3ae2d06b |
fix(backend): SessionTenantLookup is claimed first-writer-wins at registration — the sessionId→tenantId row that every Global Admin/Reader cross-tenant session resolve trusts was an unconditional last-writer-wins upsert AFTER the session write keyed by the client-supplied sessionId, so any tenant's device (self-service onboarding) that knew a victim session GUID (portal deep links, Teams/Slack notifications, progress-portal URLs) could re-point the mapping at its own tenant and have every no-tenantId GA read and GA write (globaladmin annotation lane, restore fallback) silently served from a forged session; StoreSessionAsync now claims the row create-only BEFORE any Sessions/SessionsIndex write (409 → read owner: same tenant = re-registration, foreign tenant = SessionTenantConflictException → 409 session_owner_mismatch without disclosing the owner + SessionTenantConflict ops event, nothing written, not fail-soft), the legacy self-heal write is Add-only, ResolveSessionScopeAsync(requireGlobalAdmin) corroborates the resolved tenant against its Sessions row (silent, no operator confirmation), and scripts/Migration/Backfill-SessionTenantLookup.ps1 closed the pre-table window (13509 sessions, 8795 rows added, verify dry-run added=0); no tombstone on deletion by design
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
ce202c6d47 |
fix(ops): SlaBreachNotification ops event prints Duration breaches in minutes — the P95 value carried the % unit of the rate breach types ("Duration 864.0% (target 60.0%)"); the unit now follows SlaBreachType like the customer-facing alert builder
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
7a3516e570 |
feat(observability): IME pattern-drift loop — nothing the log tracker skips or stops matching stays silent: the agent keeps cumulative tracker health counters and a per-pattern match histogram (restart-safe in ImeTrackerState), reports ime_* keys in agent_metrics_snapshot (files tailed, backlog bytes, lines read, matches, oversized/timeouts/budget breaks/held tails/unanchored), raises a one-shot immediate-upload ime_tracker_degraded Warning on the first pass that skipped work, and emits ime_pattern_hits at termination (every enabled pattern, zeros included, IME version); the backend folds the histogram into the permanent per-IME-version ImePatternStats table (built-in IDs only, plausible version only, one partition read + one transactional batch) and evaluates drift without a hard-coded list — baseline = the version with the most reporting sessions (≥100), expected = ≥80 % hit rate there, drift = an expected pattern matched in none of ≥25 sessions on a newer version — raising ImePatternDriftSuspected once per version×pattern (ETag-stamped, dual-registered in the web ops-event list with a new guard test); GET metrics/ime-pattern-health (GlobalReadOrAdmin) + Global Admin → Metrics → IME Pattern Health matrix (drift/silent/low/few cells, open alerts, retired patterns) + session-page "IME Tracker" rows (latest snapshot by max sequence) + MCP get_ime_pattern_health (GA/Reader)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
fc46b4c78e |
feat(security): session-owner binding stage 1 (shadow) + dormant agent session rotation
Agent endpoints authenticated per tenant but authorized session writes on a caller-supplied session id. Sessions rows now carry the creating device identity (cert thumbprint + Intune device id from the CN, or bootstrap code + serial) in primary-only Owner* columns. SessionOwnershipPolicy evaluates register / telemetry / error-report requests against it; SessionOwnerBindingObserver records the outcome on the request row, as a warning and as a throttled SessionOwnerMismatch ops event. Nothing is rejected in this stage. Agent: errorCode parsing on 401/403, one-shot session rotation on session_owner_mismatch without feeding the auth-failure tracker, spool drop, agent_started.sessionRotated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
d3261fb890 |
feat(ops): ops events for Collect Logs usage and diagnostics-upload enablement
CollectLogsQuickConfigEnabled fires when a tenant admin switches on Hosted diagnostics upload through the session Collect Logs quick-config dialog (PUT ?intent=collect-logs -> write source portal-collect-logs); every other enable/disable path emits DiagnosticsUploadEnabled/Disabled (PUT, PATCH, revert, MCP). SessionActionQueued records portal-queued server actions. DiagnosticsUploadConfigChange is the single definition behind the diagnosticsUploadConfigured feature flag and the flip detection. Event types dual-registered in the OpsAlertRules catalog. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
b31256c748 |
fix(activation): capture the welcome-mail address instead of redirecting past it
The activation page held the only address field in the product, and its redirect effect fired on the first poll after auto-approve — roughly 60 seconds after signup. Anyone who had not typed by then lost the field, no notification-email row was ever written, and TrySendWelcomeEmailAsync deferred a mail that nothing would send later. Nine of twenty-one August signups got a welcome mail; the last one went out on the 21st. It stayed hidden because every "not sent" branch logged at Information, and worker application logs never reach Application Insights below Warning. - Activation page holds the redirect until the address is saved or deliberately skipped, and switches to an "Activated" state while it waits. Visitors who arrive at an already-activated tenant are passed straight through as before. - Recipient resolution falls back from the activation-page address to the tenant contact address. The signup admin UPN is deliberately not a fallback: those accounts often have no mailbox and the bounces cost sender reputation. - WelcomeEmailSent / WelcomeEmailSkipped / WelcomeEmailFailed ops events make every outcome visible, dual-registered in OPS_EVENT_TYPES. Three defects in the same path, found while fixing it: - The sent-marker was consumed before the send and never released on refusal, so one provider hiccup suppressed a tenant's welcome mail permanently. It now records delivery, not an attempt. - A missing provider key logged at Debug, hiding the one condition that stops every mail at once. Now Warning. - The Global Admin resend endpoint reported success regardless of what the provider answered. It is the manual fallback for exactly this failure, so it now returns 502 and leaves the marker free when the message is refused. The welcome mail is also no longer fire-and-forget: a discarded Task can be torn down when the invocation ends, and its result is what the ops event reports. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
5e4c5e80be |
feat: verdict calibration drift radar (share / silence / evidence-gap) with ops events
Applies the rule-regression statistics (7d vs 28d, lift >=2, Wilson-separated, small-n gates) to the verdict-calibration rows: per-path share regression, sweep+maxlife silence-share regression (agent liveness signal) and an absolute r6 evidence-gap gate. Episodes live in the notification tracker (verdictcalibration|kind|path|status), fire one VerdictCalibrationDrift ops event (operator-only, no tenant bell), re-arm under 1.5x / 15%, and surface as alerts[] + a Drift pill in the calibration matrix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
445ce4e75b |
feat(backend): 30-day retention grace period on Pro downgrade
Retention was the only enforcement point that destroyed state on a Pro -> Community downgrade: the 12-hourly sweep clamps to the Community cap at read time and would immediately start hard-deleting the 90-365-day band (trial expiry identical, silently). New policy: for 30 days after losing Pro (FeatureEntitlementCatalog.RetentionDowngradeGraceDays) the Pro cap keeps applying. Anchor model: ProDowngradedUtc (backend-only field, patch-denied + revert-protected) stamped/cleared by the plan endpoint's new pure ApplyPlanChanges core on EFFECTIVE-edition transitions (covers explicitly ended trials; a planTier downgrade under an active trial deliberately does not stamp); trial expiry anchors read-time via TrialExpiresUtc. days<=0 (infinite) stays untouched. Ops visibility: TenantPlanDowngraded (PATCH handler, new OpsEventService dep) plus TenantRetentionGraceExpiring/Ended from the daily TrialExpirySweep with the same stateless window mechanics, emitted only when stored retention exceeds the Community cap; all dual-registered in the web OPS_EVENT_TYPES catalog. Fanout now uses its _utcNow seam instead of inline DateTime.UtcNow. OKF doc backend/plan-downgrade-retention-grace.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
ab61839685 |
feat(backend): P6 split - CMTrace skew tripwire + customer clock_skew rule
Operator side: CmTraceSkewTripwire piggybacks on the terminal counter
reconcile's partition scan (zero extra reads) and compares median
ReceivedAt-OccurredUtc deltas of IME-derived vs other events. A clean
15-min-grid divergence (residual < 2 min, >=20 samples / >=3 distinct
batches per side) emits the CmTraceTimeSkewRegression ops event
(Agent/Warning, goal: never fires). Bias-dominated sessions are
suppressed; measuredWriterOffsetMinutes is never consulted (sticky
after era flip-backs). Kill switch: CmTraceSkewTripwireDisabled.
Customer side: new clock_skew analyze condition source (skewMetric
clock_jump | sustained_offset, threshold seconds, gt/gte magnitude)
behind built-in ANALYZE-DEV-008 (300 s = the Kerberos/Entra 5-min skew
boundary). Batch-median frames with spool-spread filter, plateau step
detection (a gradual backlog-drain ramp cannot fake a jump) and
end-state persistence; IME-derived events are excluded so a CMTrace
anchoring regression can never fire the customer rule. Evidence carries
field=skewSummary/value for {{skewSummary}} interpolation.
Full mirroring chain: schema enum + skewMetric property, shared
manifest + combine.js regeneration, MCP authoring guide + semantic
lint, web SOURCES/form/card, ops-event dual-register, docs. P5 is
subsumed by the tripwire. PercentileMath is now the single nearest-rank
implementation (Apps/PlatformMetrics delegate).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
c728a0bd41 |
feat(backend): surface genuine agent binary-integrity mismatches as ops events
IntegrityCheckFailed emergency reports used to land in App Insights only — session e9753578's agent reported a hash mismatch and no product surface showed it, which cost a day of mis-attributed root-causing. They now materialize as an AgentBinaryIntegrityMismatch ops event (Warning, Agent category, dual-registered in the web alert-rule catalog per feedback_ops_event_types_dual_register). With suppression for the KNOWN, accepted release race the operator correctly called out: around every release, agents compare against hash snapshots that do not belong to their own binary — an older agent sees the new release's hash on its next config fetch, and a freshly installed latest agent can receive the PREVIOUS release's hashes from the 5-minute AdminConfiguration cache (e9753578: genuine 1410 exe checked against the cached 1409 hash — proven by the 1409 archive logging "verified OK" with exactly that hash). Both self-heal via the forced self-update and are suppressed at materialization time against the CURRENT config: reported running hash == published exe, or reported version != latest. What remains — an agent claiming the latest version whose binary is NOT the published one — is the real signal: tamper, stale blob, or a build outside the release pipeline. Missing config fails open to recording; visibility beats suppression. Ops event only, deliberately no timeline event: the report is about the binary, not the enrollment, and may precede session registration. Backend suite 4320 green, web tsc clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
e5f955354d |
Hourly SessionSweep interleave: stalled-session sweep every hour instead of 2h
New SessionSweepFunction (cron 0 30 * * * *, offset from the 2h Maintenance timer at minute 0) runs only MarkStalledSessionsAsTimedOutAsync, halving the sweep-induced classification latency; thresholds unchanged. The sweep now returns a SessionSweepResult (counts + catastrophic-failure message) so the new timer emits SessionSweepCompleted heartbeats / SessionSweepFailed errors (dual-registered in the ops-alert web catalog). No lease: the sweep is idempotent, overlap with the 2h chain is a no-op. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
321ae63c63 |
insights: app-version duration regression radar + per-version duration stats
versionBreakdown on the app analytics responses now carries measuredInstalls,
medianDurationSeconds and p95DurationSeconds per version (measured population:
succeeded, non-skip, plausible duration, no AppId collision). A new pure
AppVersionRegressionRadar detects "median install duration rose from X to Y min
after version Z": versions ordered by first-seen install time (never string
sort), fires at median lift >=2x AND >=300s absolute with >=10 measured
installs on both sides.
Episodes follow the rule-radar pattern in a fourth tracker keyspace
appversionregression|{app}|{version} (409-dedup, refresh keeps FirstNotifiedAt,
re-arm on drain / median back under 1.5x, 30d retention sweep re-arms): one
tenant bell app_version_duration_regression (Admin audience, appDetailUrl href)
plus one AppVersionDurationRegression ops event per episode, dual-registered in
the web OPS_EVENT_TYPES catalog. Runs in the maintenance timer after the rule
radar over one cross-tenant 35d install-summary read; kill switch
AppVersionRegressionRadarDisabled (fail-open). Apps detail page shows an amber
regression banner and a median-duration-by-version chart; OKF concept doc added.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
0e2d620ad4 |
feat(auth): self-service app-homing flip via consent flow + GA lever
Re-consent, verify and flip collapse into the existing consent flow: the
consent URL funnels legacy-homed tenants to the primary app registration,
and consent-status/access-check verify admin consent via an app-only token
probe under explicit primary credentials, then auto-flip HomedAppClientId.
- AdminConfiguration.SelfServiceAppHomingEnabled: kill switch (default off,
cache-bypassing fail-closed read); gates funnel + auto-flip + tenant-admin
manual flip. GA path deliberately survives the kill switch.
- POST config/{tenantId}/app-homing (TenantAdminOrGA): tenant admins flip to
primary only, probe-gated; GA gets force + revert. Pure decision helper
EvaluateManualFlip shared by function and tests.
- GraphTokenService.GetAccessTokenForAppAsync: probe seam with explicit app
credentials, cache-safe via app-scoped token keys.
- EntraAppRolesEnabled does not block a flip; it emits a Warning ops event
(AppHomingFlippedWithEntraRoles) reminding to re-assign enterprise-app roles.
- Web: homing badge + "App Registration Homing" panel with confirm dialog in
GA tenant management (typed homedAppClientId hardens the PUT round-trip),
funnel/success banners in tenant settings with switchAuthApp re-login,
admin-config kill-switch round-trip, ops event types dual-registered.
- Browser telemetry (App Insights): pre-init event buffer so module-init auth
events survive; strategic events across the whole flow (app_homing_flipped,
consent_flow_*, consent_verify_failed, auth_app_fallback_*, auth_app_switched,
auth_app_homing_learned, app_homing_manual_flip*).
Backend must deploy before web (additive JSON fields); everything is inert
while EntraId:LegacyClientId is unset. Enable the kill switch in the admin
portal after the dual app-reg config swap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
6bd5728652 |
feat(platform): public availability — tenant activation replaces Private Preview, optional auto-approve
Backend: shared TenantApprovalService (whitelist add + auto-promote shape-guard + welcome email), AdminConfiguration.AutoApproveNewTenants (default off, uncached fail-closed accessor), tenant-auto-approve queue (1-min visibility delay, worker drops on flag-off/approved/suspended/missing config, approves as "System (auto-approve)" + TenantAutoApproved ops event), 30s negative whitelist cache, gate 2 error PendingActivation with mode-neutral message, signup strings and welcome email without preview wording. Web: /preview → /activation (SWA 301, robots, AuthGate), AuthContext accepts both error codes and clears pending state on successful auth/me, activation page polls every 30s and auto-redirects into the portal, dashboard banner reframed to "Actively developed" + feedback call, terms/privacy/get-started updated, feature badges → Beta, admin-config web chain + auto-approve checkbox, OPS_EVENT_TYPES. Deliberately unchanged: PreviewWhitelist/PreviewConfig tables, internal preview/* API routes, preview_signup notification type, Microsoft DevPrep preview mentions. Verified: solution build 0 errors, xUnit 3701/3701, tsc 0 errors, vitest 612/612, ESLint ratchet holds (0/115), next build exports /activation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
a70c5de667 |
feat(plan): rename Enterprise plan to Pro; Pro includes Bootstrap by default and Unrestricted Mode on request
- FeatureEntitlementCatalog: TenantEdition.Pro, canonical tier "pro", legacy stored "enterprise" stays readable as Pro (no data migration); new entitlements BootstrapIncluded + UnrestrictedModeAvailable - TenantEntitlementService: IsBootstrapEnabled (Pro-included OR additive GA flag) and IsUnrestrictedModeActive (Pro AND GA gate AND tenant toggle, read-time - trial expiry/downgrade re-arms guardrails fail-closed) - All bootstrap endpoint gates + SecurityValidator use the effective check; /auth/me and feature-flags expose effective availability values - PlanManagement accepts community|pro; AlreadyPro; Pro trial wording - Web: edition.ts accepts legacy "enterprise" (deploy-order safety), Pro labels/cards + new Pro feature bullets, effective section gating, terms/privacy renamed with Last-updated bump - Tests: catalog matrix, effective-gate payload tests, helper tests incl. trial-expiry re-gate (backend 3692 green, web 612 green) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
7b975d46b6 |
feat(backend): Azure Monitor alert webhook - platform alerts join the ops-event bus
New anonymous POST ops/alert-webhook receives Common Alert Schema notifications from an Action Group webhook receiver and re-emits them as ops events (category Platform, type AzureMonitorAlert), so Azure Monitor alerts flow through the existing OpsEventService -> OpsAlertDispatchService fan-out (Telegram/Teams/Slack + portal ops-events list) like every internally raised operational event. The delivery path is independent of the portal: a SWA outage cannot suppress the notification; the action group's e-mail receiver stays as the backend-independent fallback. Security: action-group webhooks cannot send custom headers, so the shared secret travels as ?secret= and is compared in constant time (FixedTimeEquals) against the OpsAlertWebhookSecret app setting - fail-closed while unset. 64 KB payload cap, defensive essentials-only parsing (only alertRule is mandatory), severity mapped Sev0->Critical, Sev1->Error, Sev2->Warning, Sev3/4->Info with unknown values falling back to Info so schema drift can never escalate. Dual-registered per convention: EndpointAccessPolicyCatalog entry (PublicAnonymous), OPS_EVENT_TYPES in OpsAlertRulesSection (rule wiring) and Platform category in OpsEventsSection (badge + filter). Backend suite 3511 green (incl. new helper tests + catalog completeness), web tsc + vitest 651 green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
1bf37f6761 |
feat(insights): F3 regression radar - Wilson-gated rule hit-rate drift detection (PR6)
Analyze rules only (gather stats stay per-batch-deduped until Q6 is
fixed - never silently included). Per (tenant, rule) the radar compares
the trailing 7-day hit rate (FireCount/SessionsEvaluated from the daily
RuleStats rows) against the prior 28-day baseline and fires only when
the window has >=5 hit sessions and >=20 evaluated sessions, the rate
is >=2x baseline AND the Wilson 95% intervals are disjoint
(MetricsMath.WilsonInterval/RateIncreaseSeparated, hand-computed test
vectors). Suppressed for deleted rules, rules inside the CreatedAt
grace period (<35d), rules edited inside the window, empty baselines,
and the RuleRegressionRadarDisabled app-setting kill switch.
Alert episodes live as ruleregression|{ruleId} rows in the notification
tracker (third keyspace): AddEntity-409 dedup = one bell + one
RuleFrequencyRegression ops event (category Tenant, Warning, full
numbers, dual-registered in the web OPS_EVENT_TYPES) per episode;
numbers refresh while active without moving FirstNotifiedAt; episodes
re-arm under 1.5x baseline, on fire-stop, or via the tracker's 30d
retention prune (spec-intended reminder). On-fire dimension correlation
(osBuild/model/agentVersion/imeVersion; hit set from a RuleResults
range scan intersected with the window sessions) reports both shares
with the "correlated, not necessarily causal" wording contract or
honestly "no clear dimension concentration". Radar runs in the daily
maintenance pass after the rule-stats aggregation, anchored on
yesterday (whole days only).
Surfaces: regressions[] in the rule-stats response (MCP get_rule_stats
inherits it), tenant bell deep-linking to the rule card, and the
analyze-rules page gains a densified 30-day fire sparkline plus a
regression badge while an episode is active. OKF doc
backend/rule-regression-radar.md + index/log.
Backend 3474/3474 (27 new: Wilson vectors, every suppression branch,
window boundary dates, re-arm matrix, dimension gates, tracker
round-trips incl. tri-states, keyspace dedup/prune, wording pins),
web tsc + vitest 627/627, MCP tsc + 455 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
ad3632e9fb |
fix(emergency-break): survivable self-destruct, reliable break report, ops visibility
Incident 2026-07-22: the 48h emergency break left exe + loaded DLLs orphaned forever — a second agent instance held the images while the cleanup script (waiting on the launcher PID only) force-deleted marker + state around the locks AFTER removing the Scheduled Task, destroying the retry path. Agent — cleanup script (LIFE-F7): - wait for ALL agent processes by name (PID-reuse-proof), then force-kill stragglers; single-flight lock file serializes concurrent cleanup scripts - probe-rename (GUID target) BEFORE task removal; any failure renames back and aborts with task + marker + files intact for a next-boot retry; the destructive Remove-Item-on-original fallback is gone - script generation extracted to BuildCleanupScript, pinned by 13 tests; all 4 variants parser-validated + E2E-smoked (happy path, locked tree) Agent — break report delivery: - EmergencyReporter counts DELIVERED reports: reservation rolls back on failure (mark-before-send permanently suppressed a key after its first transport failure); optional in-reservation retries - break path: NIC wait <=15s, 3 attempts x 15s timeout (survives Functions cold starts observed as >5s 503s), 75s overall bound Backend: - backend-materialized events (agent_emergency_break, session_timeout) now upsert the cross-session EventType index — they were invisible to every search-by-eventType surface (portal + MCP) - new AgentEmergencyBreak ops event (Agent/Warning, once per session) so operators see silently-lost agents; dual-registered in OPS_EVENT_TYPES - materialization extracted to a static seam, covered by 4 new tests Agent 1963/1963, Backend 3198/3198, tsc clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
71d2c0e47f |
fix(maintenance): remove the time-window auto-block, it never measured data volume
GetExcessiveDataSendersAsync gated on LastEventAt - StartedAt alone. No event count, no bytes, no rate — the name asserted something the code never checked. What it actually selected was every enrollment session that spanned a night, which for Autopilot is the ordinary case: device unboxed in the evening, ESP runs, machine powered off, user signs in the next morning. The per-device alerts from the previous commit made this visible. All three devices blocked at the time of the analysis were false positives: CND3320949 / 64ed21e8 23 events over 15.5h (1.5 events/hour) 5CG4303CPT / db5751bc 192 events over 16.8h (blocked mid Account Setup 1/5) PF5YANFB / 2069de87 781 events over 31.5h 14 such blocks in 30 days across 9 customer tenants. Over the same window the event-count detector fired 5 times and was right every time (2742, 2714, 2712, 2506 events). Each false block costs a live enrollment 8 hours of telemetry. Repairing it rather than removing it does not pay: once a volume condition is added, "many events AND long span" is a subset of "many events", which is what the count detector already does. Volume is the quantity that costs anything; a 23-event session spanning 15 hours costs nothing. So the detector, its two admin settings, the ExcessiveDataBlocked ops event and the portal section go, and the event-count path is the only automatic block left. MaxSessionWindowHours and MaintenanceBlockDurationHours are deleted rather than deprecated: the config row is written with Replace, so the columns drop out on the next portal save, and nothing reads them meanwhile. A comment on the surviving detector records why there is no second one, so it does not get rebuilt a year from now. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
3b2f5ed46d |
feat(runaway-detection): name the device in the time-window alert, link it in the block list
The two runaway detectors emitted alerts of unequal usefulness. The event-count
path fires per session and carries sessionId + serialNumber, so the Ops Events
detail modal can offer View session / Block / Kill. The time-window path
aggregated a whole tenant into one event with {devicesBlocked, windowHours} —
no device, no session, no shortcuts, on precisely the alert where an operator
wants to act. It now writes one event per blocked device with the same payload
shape; the aggregate count stays in the audit entry, where a sum is the useful
view.
The Active Blocks table showed the serial as plain text although the API already
returns BlockedSessionIds. Maintenance blocks now link the serial to the session
that triggered them; manual whole-device blocks have no session and stay text.
Two defects in the event-count path, found while comparing the detectors:
- No isBlocked guard, so the auto-action re-blocked an already-blocked device and
overwrote BlockedAt/UnblockAt/Reason of the existing block — usually the
time-window block on the same session, whose reason then vanished from the list.
- No status filter, so a session that finished days ago with a high EventCount
still cost its device a 24h block. The fix needed a layer below the obvious one:
the runaway query never selected Status, and SessionStatus.InProgress is
ordinal 0, so a guard on the unpopulated field would have been silently
always-true. ParseStatusForAutoAction is stricter than MapToSessionSummary --
Enum.TryParse maps "0" onto InProgress and ORs comma-separated names into an
unrelated member, both of which would open the gate.
Not addressed, and deliberately: a block is session-scoped and stops the upload,
so whichever detector fires first freezes the other for that session. With
MaxSessionWindowHours=12 the time detector wins in practice. That is correct
behaviour, not a defect; merging the detectors would be its own change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
2d5f87140f |
chore: retarget SignalR quota watcher to Standard S1 limits + CI build hygiene
- SignalR quota watcher/health card: defaults 20/20k (Free tier) -> 1000/1M (Standard S1 x1 unit); settings renamed SignalRFreeTier* -> SignalRConnectionLimit/ SignalRDailyMessageLimit (no Azure overrides were set); all operator-facing texts tier-neutral (connections 429 hard, message overage is billed not throttled) - web: drop obsolete edge runtime from opengraph/twitter/apple-icon image routes (next/og runs on Node since 14; images now statically generated at build) - web: update caniuse-lite browser db (silences 6-months-old Browserslist warning) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
b0105b123a |
feat(maintenance): session-deletion run budget, live watchdog counters, manual trigger
- Watchdog fix: FanoutResult is caller-allocated and mutated live per tenant/session, so 30/60min LongRunning warnings report real progress instead of the always-zero counters (today's warning showed 0/0 while 5 tenants / 54 sessions were already done). - 50min run budget: the retention fanout stops cleanly at the deadline (tenant boundary + per-session check) and reports AbortedByBudget — no more risk of the 60min Flex functionTimeout hard-aborting mid-run. New Warning OpsEvent SessionDeletionMaintenanceBudgetExceeded; Completed gains abortedByBudget. - Blob-lease serialization: dedicated sentinel _lock/session-deletion-maintenance.lock (deletion-manifests container) via SessionDeletionMaintenanceLockStore + reused MaintenanceLeaseHolder; contended runs emit SessionDeletionMaintenanceSkippedLocked and return. - Manual trigger: POST /api/global/session-deletions/maintenance/trigger (GlobalAdminOnly, policy catalog) -> fail-hard queue producer -> SessionDeletionMaintenanceQueueWorker runs the same RunCoreAsync body with triggeredBy = requesting admin; lease probe returns 409 while a run is active. New Started OpsEvent carries triggeredBy. - Web: MaintenanceStatusBanner on the Session Cleanup page derives active / budget-exceeded / failed / idle from the lifecycle ops events and offers a Run-now button (202/409 handling). New event types dual-registered in OPS_EVENT_TYPES. - Tests: 7 new (budget abort immediate/mid-tenant, live counter increments, watchdog live-progress regression, BudgetExceeded + Completed flag, SkippedLocked, Started triggeredBy); suites green (backend 2935, web 557, tsc clean). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
d8e8cd15fd |
feat(entitlements): Community/Enterprise editions with self-service trial
Fail-closed entitlement layer (FeatureEntitlementCatalog + TenantEntitlementService): edition resolves at read time (PlanTier == 'enterprise' OR active trial; legacy free/pro => Community, no migration).
- Plan/trial endpoints: PATCH config/{tid}/plan (GA: tier + grant/extend/end trial, audited, cache-safe) and POST config/{tid}/trial (tenant-admin self-service 30d, once per tenant, 409 after). Generic PUT preserves plan/trial fields for all callers. Fixes SetPlanTier writing past the config cache without audit/UpdatedBy.
- Rate limits: Enterprise floor 150/min for BOTH user (portal/JWT) and device (agent/cert) paths - floor raises the global default, an explicit per-tenant override still wins; MSP users follow their home tenant.
- Retention cap: Community 90d / Enterprise 365d, clamped in the retention fanout (warn log) + non-GA write validation 7..cap (only on changed values; 0=infinite stays GA-only).
- Delegated admin (MSP): Enterprise gate on the delegated admin's HOME tenant (JWT tid, fail-closed on unknown) - managed target tenants may be any edition. Grant-time target check removed; inert grants are suppressed at resolve time (single choke point in GetScopeAsync, applied after the UPN-keyed cache).
- MCP quota: McpQuotaService + McpQuotaEnforcementMiddleware (X-Client-Source: mcp only, GA exempt) enforce daily/monthly budgets - per-user plan override > tenant edition plan; limits from SectionUsagePlans with catalog fallbacks (community 100/3000, enterprise 1000/20000); 429 with X-MCP-Quota-* headers + Retry-After; fail-open on counter reads, fail-closed on plan resolution. /api/metrics/mcp-usage/me reports effective plan + counters.
- Trial visibility: TrialExpirySweepFunction (daily 03:30 UTC, informational) emits TenantTrialExpired (Warning, 24h look-back) + TenantTrialExpiring (Info, <=3d heads-up); dual-registered in web OPS_EVENT_TYPES.
- Web: EditionBadge + one-time trial CTA in settings header, Plan & Trial card in the GA tenant edit modal (own PATCH save), retention cap hints from entitlements, pure lib/edition.ts (vitest), feature-flags surface edition/trial/entitlements; MCP TENANT_SAFE_FIELDS + trial fields.
Verified: solution build clean; backend 2653/2653 tests; web tsc + 516/516 vitest; MCP server tsc + tests green.
|
||
|
Oliver Kieselbach
|
34582c0f0e |
feat(kill-switch): deliver kill on config channel + KillSignalDelivered ops event
Kill/Block rules were only served on the telemetry response, which cannot reach an agent whose upload loop is paused (indefinite block pins the drain until DateTime.MaxValue), whose spool is empty, or whose binary is kill-blind. Config is fetched at every process start (boot via Scheduled Task), making it the reliable control channel - no heartbeat added by design (48h emergency break + 6h max lifetime bound the remaining cases). Backend (effective on deploy): - KillSwitchEvaluator: shared device+version check for both channels; ingest wire response and check order unchanged (device first, short-circuits) - Every served Kill emits ops event KillSignalDelivered (Security/Warning), throttled 24h per tenant+serial+pattern; Blocks stay event-free - GET agent/config (incl. bootstrap wrapper) now returns DeviceBlocked/ DeviceKillSignal/UnblockAt alongside the full config; old binaries ignore the new fields - Web: KillSignalDelivered dual-registered in OPS_EVENT_TYPES Agent V2 (effective with next agent release): - CheckConfigKillSignal guard after the Phase-4 fetch: forced self-destruct (overrides SelfDestructOnComplete), enrollment-complete marker, session cleared, exit before mTLS/registration - Kill honoured ONLY from a live fetch: config disk cache strips the flags on write and read (UnrestrictedMode pattern), guard re-checks LastFetchOutcome Tests: 9 KillSwitchEvaluatorTests + 7 ProgramConfigKillSignalTests; Functions 2557/2557, Agent V2 1578/1578, web tsc clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
a993070840 |
feat(ops): include per-orphan tenant/session breakdown in OrphanEventsCleaned
The OrphanEventsCleaned ops-event only persisted counts; the per-orphan
detail (which tenant/session lost lingering events) was logged via worker
LogInformation, which does not reach App Insights, so it was effectively
lost. Enrich the event details with a capped (top-50 by EventCount) list of
{tenantId, sessionId, eventCount} plus a detailsTruncated flag, reporting the
actual rows deleted. Keeps the Table row under the 32KB property limit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
5ba4fb7dc9 |
feat(backend): retention/cleanup for previously-unbounded append-only tables
Several append-only tables had no purge mechanism and grew unbounded. Add age-based retention to the 2h MaintenanceFunction (and the manual path) via a new CleanupUnboundedTablesAsync orchestrator (per-table try/catch, fixed const windows, mirroring the DistressReports pattern): - GlobalNotifications + TenantNotifications: hybrid retention — dismissed rows pruned 30d after DismissedAt, any row pruned 180d after CreatedAt. Unread (still-actionable) admin warnings survive the dismiss window so they are never silently lost; the 180d catch-all bounds the table. Shared pure predicate NotificationRetentionFilter.BuildPredicate (unit-tested). - HardwareRejectionNotificationTracker: 30d (FirstNotifiedAt). Pruning re-arms the lifetime dedup, so a model rejected again >30d later re-bells (intended). - AuditLogs: 180d (system Timestamp, cross-partition). - UsageMetrics: 180d (PartitionKey date-string range). - BackupJobs: 365d (QueuedAtUtc). PlatformStats intentionally excluded — it is a single upserted row, not append-only. Bootstrap sessions left as-is (feature currently unused). Each repo delete mirrors DeleteUserActivityOlderThanAsync: server-side "X lt cutoff" filter + select PK/RK + delete loop. Deletes are resumable across runs, so a partial run self-heals on the next cycle. Observability: end-of-run MaintenanceLongRunning Warning OpsEvent if RunAllAsync exceeds 10min, dual-registered in OpsAlertRulesSection.tsx, to surface a run climbing toward the 60min functionTimeout before a future hard-abort (which emits nothing). Inert until backend redeploy; first post-deploy run bulk-prunes the backlog. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
c434eae179 |
feat(device-block): auto block/kill on excessive session events
Adds a second, higher threshold to the runaway-session detector that auto-blocks or auto-kills the device (admin-selectable mode), so the existing 2000-event warn keeps observing while the harder action runs unattended at e.g. 2500 events. Both tiers stay independently idempotent per session and the maintenance batch picks them up in a single query pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
9e689dc9d8 |
feat(backend/backup): PR2 — single-row restore + admin UI
POST /api/global/backups/{backupId}/restore-row (GA-only) with preview|commit
modes. Preview scans the ETag-pinned NDJSON blob for the requested (pk, rk),
diffs against the live row, and returns rowSha256 + currentETag for echo.
Commit re-validates everything under a short maintenance lease and writes via
UpdateEntity(IfMatch=etag, Replace) or AddEntity; 412/404/409 → CurrentRowChanged.
Backend:
- RestoreTablePreflightValidator (no I/O) + BackupRestoreInputValidator (manifest
parse, entry check, BlobName canonical, ETag-pin + SHA in one stream read)
- CriticalTableRestoreService with PreviewRowAsync (no lease) + CommitRowAsync
(MaintenanceLeaseHolder, 45 s renewal, deterministic finally release)
- BlobBackupStore.OpenNdjsonReadAsync — GetProperties + OpenRead(IfMatch=etag)
- BackupTerminalException carries a stable Code for HTTP envelope mapping
- OpsEventService.RecordBackupRowRestoredAsync (Warning; payload is
container + blobName only, never a SAS URL)
- 27 new xUnit tests; full backend suite 1979/1979 green
Web:
- /admin/backups list page with newest-first list + "Run backup now" trigger
and async job-status poller
- /admin/backups/{backupId} detail page with per-table breakdown + single-row
restore trigger
- RestoreRowDiffModal with Added/Removed/Changed/Unchanged pills and
auth-table security banner (GlobalAdmins/TenantAdmins/McpUsers)
- api.backups namespace + matching DTO types
- globalNavConfig: ga-backups under Ops
- BackupRowRestored registered in OPS_EVENT_TYPES (dual-register policy)
- 6 new vitest cases; full web suite 390/390 green
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
d4034842ee |
feat(backend/backup): PR1 — critical-table backup feature (daily timer + manual GA trigger)
Adds the full backup pipeline that snapshots 15 critical tables (Admin/Tenant config, rules, identities, audit, preview) into a logically-isolated blob container, daily at 04:00 UTC and on operator demand. Restore endpoints follow in PR2/PR3. Architecture: * Daily TimerTrigger + manual POST /api/global/backups/trigger (GA-only, fail-hard producer). HTTP returns 202 + jobId on enqueue, 500 if the queue write fails (and the BackupJobs row is rolled to Failed). * BackgroundService queue worker (not QueueTrigger — matches the Analyze/ IndexReconcile pattern that avoids the Functions-host connection-binding app-setting). Worker holds the maintenance lease for the whole run; service is intentionally lease-unaware so it stays unit-testable. * BlobBackupStore wraps the new 'critical-table-backups' container with a shared maintenance-lease sentinel at '_lock/maintenance.lock' that serialises every backup or restore op across worker/timer/single-row paths. * CriticalTableBackupService streams each table through CryptoStream(SHA256) into per-table NDJSON blobs, then writes manifest.json LAST as the durability anchor. Per-table failures degrade Outcome to Partial without aborting; per-run budget (50 min) → Skipped status. * BackupJobsRepository persists the six-state machine (Queued/Running/Completed/Failed/Skipped/BlockedTerminal) with ETag-CAS updates. Enum→string mapping rejects unknown values on read so refactors surface loudly instead of silently shifting state. * BackupJobWatchdog (30 min timer) heals stuck rows via a probe lease on the shared sentinel: Queued > 60 min → Failed, Running with stale heartbeat AND free lease → Failed (lease-busy means a live worker — leave alone). * OpsEvents: distinct CriticalTableBackupPartial (Warning) vs Failed (Error) so operators can wire softer Telegram rules for partial captures. * MaintenanceService.MonitoredPoisonQueues + OPS_EVENT_TYPES catalog registered (dual-register memory). Test coverage (15 new tests, 1952/1952 backend xUnit green): * BackupManifestJsonTests: enum-as-string (allowIntegerValues=false rejects numeric values), DictionaryKeyPolicy unset (Azure column names preserved case-faithfully), manifest field round-trip. * BackupJobsRepositoryMappingTests: full entity↔domain round-trip, unknown Kind/State/BackupOutcome rejected, legacy BackupOutcome="Failed" rejected. * CriticalTableBackupServiceTests: happy/partial/empty/budget-exceeded paths exercise the per-table loop with in-memory table + blob fakes. Setup: scripts/Backup/setup-backup-container.ps1 — idempotent az-CLI bootstrap for container + lifecycle rule (90 d delete on prefix). Soft-Delete and Versioning are NOT applied automatically (storage-account-wide blast radius — operator must validate per separate doc note). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
c26b25b81b |
feat(backend,web): offboarding farewell-email capture + dedicated feedback table + reports tabs
- Persist tenant Preview-Notification-Email on OffboardingHistory row at Phase 1 so it
survives Phase 2 wipe; ResendEmailService.SendAsync wired but disarmed by a const-false
arm-flag with a [DRAFT] placeholder template — explicit flip required to ship.
- Move user feedback storage out of PreviewConfig discriminator into a dedicated Feedback
table (PK=InApp / Offboarding) that is NOT in any tenant-offboarding wipe list — feedback
from departing tenants is exactly the signal we want to keep. New IFeedbackRepository +
TableFeedbackRepository; FeedbackFunction migrated; (PreviewConfig, "Feedback")
removed from TenantOffboardingHandler.DiscriminatorTables.
- POST /api/tenants/{tenantId}/offboard/feedback collects a free-form farewell comment
during the drain-barrier countdown (marker-gated, fail-soft OpsEvent emit). Reports >
User Feedback page gains an Offboarding-Feedback sub-tab next to In-App Feedback.
- OffboardingFeedbackReceived OpsEventType registered (info-severity) + catalog entry
in OpsAlertRulesSection so admins can wire Telegram in two clicks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
206388da99 |
feat(backend,web): tenant offboarding async cascade with customs archive
Replace synchronous DeleteAllTenantDataAsync with a queued async cascade
that wipes all 24 tenant-scoped tables safely, archives custom rules,
and survives crashes at every step.
Phase 1 (TenantOffboardFunction, ~200ms sync):
- Write History / ByTenantPointer / Marker rows (OffboardingAudit table)
- Flip TenantConfiguration.Disabled=true (the active auth gate)
- Enqueue with 6 min visibility-delay (cache-drain barrier =
TenantConfigurationService.CacheDuration 5 min + 1 min safety)
- Return 202 with EarliestProcessingAt for the UI countdown
Phase 2 (TenantOffboardingWorker + Handler, async after drain):
- Enumerate sessions, per-session cascade-enqueue, persist
Expectations blob (separate offboarding-state container)
- Drain probe against per-session progress blobs; self re-enqueue
up to 60x at 2 min visibility-delay
- 2.D-archive: GatherRules / AnalyzeRules / ImeLogPatterns moved
to TenantOffboardingCustomsArchive table (3-iter cap + verify)
- 2.D-wipe: SafeWipe over all tenant tables (4 variants: exact-PK,
composite-PK range, discriminator+TenantId, property-only) with
fetch+verify+delete pattern + SafeWipeVerificationException
- 2.E: blob-prefix wipe deletion-manifests/{tenantId}/
- 2.D-pass-2: re-wipe AuditLogs (defense-in-depth)
- 2.G: side-effects FIRST (Marker, Pointer, blob, audit, OpsEvent)
then History=Completed LAST (DrainCompletedAt as resume anchor)
- 2.F-final: delete TenantConfiguration row LAST so self-service
re-onboarding (next /api/auth/me auto-creates default) works
Defense-in-depth:
- OffboardingMarkerCleanupFunction (2h timer): sweeps Completed
markers after 15 min grace, conditional TenantConfiguration sweep
(only when row is still the offboarding tombstone — preserves
user-driven self-service re-onboards)
- Worker poison path: durable Failed state MUST land before queue
moves; transient failure leaves message visible for retry
- SessionDeletionHandler reorder: Audit + SignalR notify BEFORE
CompletedAt write so the drain predicate is correct
Web UI:
- /settings danger zone: typed "OFFBOARD" confirmation dialog,
post-202 drain-barrier banner with minute-based countdown, hint
"you can sign out and close this tab — deletion runs in
background", silent auto-logout when countdown elapses
- /admin/customs-archive (GA only): list runs + inspect entries +
typed "DELETE" confirmation modal for run/entry deletion
- OpsAlertRulesSection: TenantOffboardingFailed event registered
- lib/api.ts: shared CustomsArchive* response types
Tests: 100+ new tests covering drain contract, handler ordering,
worker poison path, ETag-CAS, SafeWipe variants, OffboardingFilters
prefix-bleed safety, MarkerCleanup conditional sweep, Bootstrap
auth-disabled-tenant gate.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
884f29b019 | update offboard function | ||
|
Oliver Kieselbach
|
2211399f23 |
chore(backend,web): remove legacy session-delete path (PR7)
Cascade-delete V2 is the only path now. Drops the per-tenant EnableCascadeDeleteV2 flag, the legacy ETag-CAS-guarded DeleteSessionAsync helper, the AppInstallSummaries cleanup helper (only legacy caller), and the SessionsLegacyDeleted counter chain (fanout → maintenance → OpsEvent). DeleteSessionFunction collapses to a single kill-switch gate; the producer owns 404 / 409 / Poisoned mapping AND the intentional Queued/Preparing+Snapshot → 202 recovery resume — gating those at HTTP would strand crashed admin cascades. Fanout now aborts both the inner loop AND outer loop on producer KillSwitchActive instead of leaking calls into the next tenant. Test coverage: - DeleteSessionCasGuardTests removed (helper gone) - AdminConfigurationSessionDeletionKillSwitchTests trimmed to kill-switch only - DeleteSessionFunctionTests pinned to the kill-switch short-circuit + null - SessionRetentionFanoutServiceTests: legacy path tests dropped, two new abort regressions for producer KillSwitchActive (inner + outer loop) Build: 0 warnings, 0 errors. Backend: 1526 passed / 0 failed. Web: tsc clean, vitest 346 passed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
09bfe7a1bc |
feat(backend,web): admin session-cleanup UI for V2 cascade-delete + codex follow-ups
Operationalises the 7-PR backend cascade-delete pipeline that landed via commits
c8658b9f..921994e8: per-tenant V2 opt-in switch, audit-trail consolidation, and
a Global-Admin-only Session Cleanup page to track in-flight / poisoned / stranded
cascades and trigger restores.
== PR-A: per-tenant V2 enrollment ==
- UpdateTenantConfigurationFunction extends the GA-only gate list with
EnableCascadeDeleteV2 so tenant admins cannot self-opt-in / out of the
snapshot+restore pathway.
- TenantManagementSection.tsx adds a purple-bordered "Enable V2 Cascade Delete"
checkbox in the GA edit modal and a "Cascade V2" badge on the tenant card so
rollout state is visible at a glance.
== PR-B: audit consolidation ==
- SessionDeletionHandler drops the 15-per-cascade deletion_step_completed /
deletion_step_failed / deletion_verification_failed audit emissions. Step
telemetry lives in DeletionProgress + structured logs; tenant admins now see
just deletion_started + deletion_completed + deletion_restored.
- SessionDeletionWorker no longer writes a deletion_poisoned tenant audit;
the signal moved to a new SessionDeletionPoisoned OpsEvent (operator-bound,
Telegram-routable, dual-registered in OpsAlertRulesSection.tsx catalog).
IMaintenanceRepository dependency removed from the worker constructor.
- SessionRestoreService stops emitting the deletion_manifest_downloaded
breadcrumb; deletion_restored is the operator-visible outcome.
== PR-C: Session Cleanup admin page ==
- New /admin/ops/session-cleanup route (gated by app/admin/layout.tsx's
ProtectedRoute requireGlobalAdmin). Sidebar entry under ga-ops alongside
Maintenance. Four URL-state tabs: In-Flight / Poisoned / Stranded Queued /
Activity (links to existing /audit page).
- Backend reads: GetSessionDeletionsListFunction
(GET /api/global/session-deletions?state=&strandedSinceMinutes=) +
GetSessionDeletionManifestFunction
(GET /api/admin/sessions/{id}/deletion-manifest?tenantId=&manifestId=&mode=).
Both registered as GlobalAdminOnly in EndpointAccessPolicyCatalog.
- DeletionPreviewModal reads the persisted snapshot via the stored-manifest
endpoint (NOT a fresh dry-run via the existing /delete/preview, which would
enumerate live-but-partially-deleted data for in-flight cascades). Renders
manifest counts, snapshot SHA, worker progress (CompletedSteps,
VerificationDone, TombstoneStarted, aggregate decrements), and the last
verification failure with residual sample. Download button uses
authenticatedFetch + Blob + Object URL so the Bearer token is sent.
- RestoreConfirmDialog drops the Full/Partial radio (mode is auto-selected
by SessionRestoreService based on cascade state — forcing the wrong mode
would corrupt state). Reason field threads through to deletion_restored
audit details. Dry-run default = true and previews the auto-selected mode.
== Codex review follow-ups (across four rounds) ==
- High: tenantId is explicitly accepted on /delete/preview (query) and
/restore (body); GA cross-tenant resolution prefers the explicit value over
JWT TargetTenantId, falling back to SessionsIndex lookup. Without this, a
GA acting on a session in another tenant got their JWT home tenant.
- High: new stored-manifest endpoint replaces the dry-run preview for
in-flight / poisoned rows; previous fix built a fresh manifest against
already-partially-deleted data and misrepresented the cascade.
- Medium: SessionDeletionPoisoned OpsEvent now carries failureType +
failureMessage + observedResidualCount + a small residual preview, persisted
by the handler into DeletionProgress (LastFailureType, LastFailureMessage,
LastObservedResidualCount, LastResidualSampleJson) before the throw so the
worker can include them.
- Medium: dry-run restore reports the AUTO-SELECTED mode (full/partial) so
the dialog's preview reflects what the real run will do, instead of the
literal string "dryRun".
- Medium: residual count field renamed to observedResidualCount with an
honest XML doc — CascadeVerificationService caps at 50 per table AND
short-circuits after the first failing table, so the number is a lower
bound. UI surfaces "N of 50+ observed rows" when N hits the cap.
- Medium: three-layer cap defense on the OpsEvent residual sample to stay
under the 4096-char TableOpsEventRepository truncation: max 5 entries
(OpsEventResidualSamplePreviewSize), per-field max 96 chars with `…` marker
(OpsEventResidualKeyMaxChars), total preview budget 1200 chars
(OpsEventResidualPreviewBudgetChars) with trailing-entry drop. Full sample
stays in the progress blob for the admin UI.
- Low: estimatedSnapshotBytes field name aligned end-to-end; activity tab
text clarifies audit-action vs OpsEvent distinction.
Tests:
- Backend xUnit 1538/1538 pass (up from 1510 baseline; +28 across the new
Session Cleanup endpoints, OpsEvent enrichment, restore-mode reporting,
cap-defense unit + integration tests).
- Web vitest 347/347 pass.
- TypeScript strict + dotnet build clean.
Deliberately deferred:
- xUnit tests for GetSessionDeletionsListFunction +
GetSessionDeletionManifestFunction (no HttpRequestData mock harness in repo;
would be a separate refactor).
- vitest for DeletionPreviewModal + RestoreConfirmDialog (defer until first
field issue).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
4c676d076a |
fix(backend,web): PR6 codex-review follow-ups (F1 legacy pre-check, F2 mid-run kill-switch, F3 OpsEvent audit shift)
Three Round-1 review findings from PR6:
F1 (High) — Legacy retention path mutated side-tables before the
CAS-guarded DeleteSessionAsync. Plan PR6 step 7 says "legacy must skip
locked sessions", but our LegacyDeleteAsync deleted Events / RuleResults
/ AppInstallSummaries first and only discovered the V2 cascade lock at
the tombstone. Fix: pre-read the Sessions row in LegacyDeleteAsync; if
DeletionState != None, return false WITHOUT touching any side table.
The original CAS-guard stays as a backstop for the T1-T3 race after the
pre-read.
F2 (High) — Kill-switch checked once per run; a flip-ON mid-fanout kept
legacy deletes running. SessionRetentionFanoutService now takes
AdminConfigurationService, calls IsSessionDeletionKillSwitchActiveAsync
(uncached, fail-closed per PR5 F1) at both the per-tenant boundary and
inside the per-session loop. FanoutResult gains AbortedByKillSwitch so
the per-tenant audit + the global OpsEvent surface the early halt.
F3 (Medium) — Global-scope lifecycle audits used LogAuditEntryAsync with
tenantId: null!, which the table-storage helper silently dropped because
the schema requires a non-null PartitionKey. Three event types
(Completed / FanoutSkipped / Failed) moved to OpsEvents (TenantId is
nullable on OpsEventEntry). Two new helpers in OpsEventService —
RecordSessionDeletionMaintenance{Completed,FanoutSkipped}Async — also
dual-registered in the OpsAlertRulesSection.tsx catalog per memory
feedback_ops_event_types_dual_register. Per-tenant audits inside the
fanout (which already had a real tenantId) are unchanged.
Tests: 4 new xUnit cases — F1 skip-without-mutation, F1 row-not-found
skip; F2 mid-tenant abort, F2 between-tenants abort. Existing 17 PR6
tests updated to assert OpsEvents instead of audit calls for the global
lifecycle events. Total: 1504/1504 backend pass, 347/347 web pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
50071c5e42 |
feat(backend,web): dedicated cascade-delete maintenance function (PR6/7)
Plan §5 PR6 / §16 R14: extracts cascade-lifecycle maintenance out of the
2h Maintenance timer into a dedicated 12h SessionDeletionMaintenanceFunction
with independent cadence, kill-switch, and OpsEvent watchdogs. Adds four
work-blocks (Plan §10):
1. Manifest-blob TTL sweep — defence-in-depth on top of Azure Blob
Lifecycle Management. EnumerateOldDeletionManifestsAsync +
DeleteDeletionManifestPairAsync (33-day effective window via the
existing soft-delete tombstone, not permanent deletes).
2. Stale-Preparing GC — reverts DeletionState=Preparing rows older than
1h that have NO progress blob (producer crashed before upload). Skips
when a progress blob exists; producer's own Preparing-resume path
handles those.
3. Stranded-Queued detection — alert-only OpsEvent per row in
DeletionState=Queued for > 30min (no auto-clear; operator inspects).
4. Retention fanout via new SessionRetentionFanoutService — per-tenant
DataRetentionDays sweep, dispatched per EnableCascadeDeleteV2 flag:
V2 → producer.EnqueueAsync, legacy → ISessionRepository.DeleteSessionAsync
(now CAS-guarded since PR5 finding 2). Rate-limited to 100 enqueues
per tenant per run with 50ms throttle.
Operational alerting:
- 30min watchdog → OpsEventType SessionDeletionMaintenanceLongRunning (Warning)
- 60min watchdog → SessionDeletionMaintenanceLongRunningSevere (Error)
- Exception path → SessionDeletionMaintenanceFailed (Error) + re-throw
- Per stranded row → SessionDeletionStrandedQueued (Warning)
All four registered both in OpsEventService.cs and the OpsAlertRulesSection.tsx
catalog so admin can wire Telegram per memory feedback_ops_event_types_dual_register.
Reuses PR5 finding 1's IsSessionDeletionKillSwitchActiveAsync so the
fanout-skip check honors flip-ON across scaled-out instances within seconds.
host.json: functionTimeout=01:00:00 (Flex Consumption max, matches the
60min severe watchdog so operator gets alerted before Azure hard-aborts).
Cron: const "0 0 */12 * * *" — matches the repo's IndexReconcileTimer
convention (no AdminConfiguration field; redeploy to change).
Removed: session retention loop from MaintenanceService.CleanupOldDataAsync
(both RunAllAsync and RunManualAsync paths now call CleanupOldUsageDataAsync
directly). Non-session housekeeping (rule-stats, cert-expiry, SignalR
quota, poison-queue watcher) untouched.
EnableCascadeDeleteV2 + SessionDeletionKillSwitch defaults stay false —
PR6 stop-condition keeps real tenants on the legacy path until §18.4
internal-tenant validation.
Backend xUnit: 1500/1500 pass (1483 + 17 new). Web vitest: 347/347 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
c8902de7c9 |
feat(backend/maintenance): poison-queue backlog watcher with Telegram alerts
Extends the SignalR/Cert pattern to async-worker poison queues. The previous
PR made backlogs visible on /health/detailed - this one makes them actively
page the operator via the existing OpsAlertDispatch -> Telegram path.
New partial MaintenanceService.PoisonQueue.cs runs on the 2 h timer (and on
manual maintenance triggers) and emits two new EventTypes:
- PoisonQueueBacklogHigh (Severity=Warning, threshold >= 1)
- PoisonQueueBacklogCritical (Severity=Error, threshold >= 10)
Dedup mirrors the cert watcher: composite key {EventType}|{queueName} built
from today's Security OpsEvents -> one alert per queue per tier per UTC day.
A tier escalation (warning -> critical) fires a new alert because the
EventType changes; a stable backlog re-pings once per UTC day so it stays in
the operator's attention.
Constants + classifier + watch-list moved from HealthCheckService to the new
MaintenanceService partial - the live health card and the timer-driven
Telegram alert now classify identically (same canonical source). Boundary
and JSON-extractor tests follow the move; HealthCheck tests are reduced to
the card-form assertions they actually exercise.
Operator follow-up (manual, not in code): add two OpsAlertRules in the admin
UI - one per new EventType - to wire Telegram. Until then the watcher writes
OpsEvents but no message is sent, matching how SignalR/Cert alerts were
introduced.
Backend: 1199/1199 (1193 + 6 net from test resplit). Build 0/0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
bc3cefa790 |
feat(backend/observability): SignalR free-quota watcher with tiered OpsEvents
Hourly TimerFunction queries Azure Monitor for SignalR ConnectionCount (max over last hour) and MessageCount (sum since UTC midnight) and emits Warning (>=80%) / Error (>=95%) OpsEvents when the free-tier limits (20 connections, 20k messages/day) are approached. Mirror of the cert-expiry pattern - global TenantId=null events, dedup'd to one emission per EventType per UTC day. Fail-soft: skips silently when SignalRResourceId app setting is unset, logs warning on Azure Monitor auth/query errors. Function App's managed identity needs 'Monitoring Reader' on the SignalR resource. Also called from the manual maintenance trigger so 'Run Maintenance Now' exercises every health check. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
f2257a4b02 |
fix(backend/security): cert-expiry watcher uses bucket-level logic, not per-cert
77fde56f's per-cert alarm fires for every expiring cert independently. That is wrong: when a Root or Intermediate has a fresh successor in the bundle (e.g. intune-intermediate-mdm-2026.pem alongside intune-intermediate-mdm-2028 .pem), the older cert is just a chain-continuity helper for in-flight device certs. Alarming on it would page the operator to "rotate" something that is already rotated. Refactor to bucket-level: per role (Root, Intermediate), pick the cert with the MAXIMUM NotAfter and classify only that. Older expiring/expired siblings are silent. Operator only gets paged when the freshest itself nears expiry, i.e. when there is genuinely no successor yet. Concrete behaviour change for current bundle (today 2026-05-08): Roots: freshest is intune-root-2030.pem (1591d) -> silent Intermediates: freshest is intune-intermediate-mdm-2028.pem (859d) -> silent The expired intune-intermediate-mdm-2026.pem (-3d) no longer triggers a spurious Critical because mdm-2028 has it covered. Tests: 7 new EvaluateBucket cases (empty, single fresh, single soon, old+ fresh successor stays silent, old+soon successor fires on successor, all expired falls back to freshest-expired, two future successors). Total 38 cert-related tests, full suite 1105/1105. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
77fde56f30 |
feat(backend/security): cert-expiry watcher in maintenance with tiered OpsEvents
Adds proactive monitoring of the embedded Intune CA bundle so the operator
gets a Telegram ping months before agent mTLS would break. Bundle rotation
(re-exporting fresh PEMs from a corporate-enrolled device) is the only fix
and needs lead time - today's outage from
|
||
|
Oliver Kieselbach
|
6658303c0d |
feat(ops/consent): emit ConsentFlowSuccess to close the consent-flow lifecycle
Pairs with ConsentFlowStarted/Failed in the OpsEvents table so admins can
see whether repeated consent failures (AADSTS, redirect-mismatch, denied)
eventually resolved into a successful grant. New POST endpoint
config/{tenantId}/autopilot-device-validation/consent-success is called
from the post-redirect callback only after consent-status confirms
isConsented=true, so generic status polls do not fire false positives.
Trigger whitelist accepts autopilot, corporate, and device-preparation;
the third value is wired through the type system end-to-end so a future
DevPrep-specific consent flow can use it without a schema bump.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
4b3db3a1cd | whiteglove completion logic and orphan events handling | ||
|
Oliver Kieselbach
|
a0bfffcbd4 | bulk delete speedup, session runaway alarm, Autopilot validation permissions sentance docs | ||
|
Oliver Kieselbach
|
c415872e46 | feat: SLA tracking dashboard with notifications | ||
|
Oliver Kieselbach
|
0d54619a52 | added IME agent update alert |