The tenant contact address had no server-side validation, so a direct API
caller could store any string in the field service notices are sent to.
UpdateTenantConfiguration now rejects non-addresses, recipient lists (a comma
would silently widen who receives those notices), display-name forms and
control characters (mail-header injection), and normalizes what it stores.
Empty stays valid - having no contact address is a legitimate state.
The seed path had a worse problem than missing validation. It was a
read-modify-write of the whole model through SaveTenantConfigurationAsync,
which upserts with TableUpdateMode.Replace and no ETag: a concurrent portal
save was clobbered by the seeder's stale snapshot of EVERY field, not just
ContactEmail. It also wrote straight through IConfigRepository, bypassing the
TenantConfigurationService cache, so a seeded address stayed invisible for the
full 5-minute TTL.
TrySeedTenantContactEmailAsync now writes ETag-conditionally in Merge mode -
one property, and a 412 means someone else wrote the row so the seed abandons
rather than overwriting the winner. A lost seed is recoverable on the next
maintenance run; a lost admin edit is not.
The same seed logic existed twice (preview notification-email save and the
maintenance backfill). Both now call a single owner on
TenantConfigurationService, which holds the never-overwrite invariant, the
cache eviction and the fail-soft guarantee. Fail-soft lives in the service
rather than the repository so it holds for any implementation.
Tests cover the validator, the service contract, and the repository against a
mocked TableClient: Merge mode, the ETag actually passed, only ContactEmail in
the patch, the 412 race, 404, storage faults. Plus a regression pin that
SaveTenantConfigurationAsync is never called from the seed path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 1 - named notification channels (1..N per tenant):
- NotificationChannel model (provider/url/headers/enabled + per-event
opt-ins incl. single NotifyOnSlaEvents); stored as JSON column
NotificationChannelsJson, cap 10, fail-soft ParseList
- No data migration: GetNotificationChannels() synthesizes one channel
from the legacy single-webhook fields under stable id "legacy";
web materializes on first save and clears legacy fields
- All dispatch sites (ingest success/failure/white-glove/ESP, SLA breach/
resolved/consecutive, enrollment-start, hardware-rejected, test) route
via SendToChannelsAsync with per-channel toggle filtering
- Per-channel secret redaction (structure stays visible to readers) with
restore-by-id; save-time validation (dup ids, provider, https, headers)
- Settings UI: channel card list with add/remove, event pills, per-channel
test button ({channelId} body, legacy-compatible without body)
Phase 2 - analyze-rule channel notifications:
- NotifyDefault (definition, false for all built-ins, test-guarded) +
Notify/NotifyChannelIds tenant override in RuleStates (two-tier mirror
of MarkSessionAsFailed); custom rules fold into the tenant-owned row
- Dispatch in AnalyzeOnEnrollmentEndHandler only: outcome.Results is
new-findings-only (engine dedup) so no re-notification; manual
"Analyze Now" reanalyze never dispatches
- BuildRuleFiredAlert (eventType analyze_rule_fired): device, serial,
rule, severity, category, confidence, explanation + session deep link
- Rule card: notify toggle + channel target pills, bell header badge
2858/2858 Functions tests (19 new), web tsc + vitest clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fail-closed entitlement layer (FeatureEntitlementCatalog + TenantEntitlementService): edition resolves at read time (PlanTier == 'enterprise' OR active trial; legacy free/pro => Community, no migration).
- Plan/trial endpoints: PATCH config/{tid}/plan (GA: tier + grant/extend/end trial, audited, cache-safe) and POST config/{tid}/trial (tenant-admin self-service 30d, once per tenant, 409 after). Generic PUT preserves plan/trial fields for all callers. Fixes SetPlanTier writing past the config cache without audit/UpdatedBy.
- Rate limits: Enterprise floor 150/min for BOTH user (portal/JWT) and device (agent/cert) paths - floor raises the global default, an explicit per-tenant override still wins; MSP users follow their home tenant.
- Retention cap: Community 90d / Enterprise 365d, clamped in the retention fanout (warn log) + non-GA write validation 7..cap (only on changed values; 0=infinite stays GA-only).
- Delegated admin (MSP): Enterprise gate on the delegated admin's HOME tenant (JWT tid, fail-closed on unknown) - managed target tenants may be any edition. Grant-time target check removed; inert grants are suppressed at resolve time (single choke point in GetScopeAsync, applied after the UPN-keyed cache).
- MCP quota: McpQuotaService + McpQuotaEnforcementMiddleware (X-Client-Source: mcp only, GA exempt) enforce daily/monthly budgets - per-user plan override > tenant edition plan; limits from SectionUsagePlans with catalog fallbacks (community 100/3000, enterprise 1000/20000); 429 with X-MCP-Quota-* headers + Retry-After; fail-open on counter reads, fail-closed on plan resolution. /api/metrics/mcp-usage/me reports effective plan + counters.
- Trial visibility: TrialExpirySweepFunction (daily 03:30 UTC, informational) emits TenantTrialExpired (Warning, 24h look-back) + TenantTrialExpiring (Info, <=3d heads-up); dual-registered in web OPS_EVENT_TYPES.
- Web: EditionBadge + one-time trial CTA in settings header, Plan & Trial card in the GA tenant edit modal (own PATCH save), retention cap hints from entitlements, pure lib/edition.ts (vitest), feature-flags surface edition/trial/entitlements; MCP TENANT_SAFE_FIELDS + trial fields.
Verified: solution build clean; backend 2653/2653 tests; web tsc + 516/516 vitest; MCP server tsc + tests green.
Replace the denormalized per-tenant RateLimitRequestsPerMinute mirror (and its
SaveConfigurationAsync sync job) with compute-at-read resolution: effective limit =
per-tenant override ?? global default. Fixes the clobbering bug where a per-tenant
edit to the base field was overwritten on the next global config save.
- Model: drop TenantConfiguration.RateLimitRequestsPerMinute; add nullable
CustomUserRateLimitRequestsPerMinute (device override CustomRateLimit... kept).
- RateLimitResolver centralizes precedence (device + user); both paths call it.
- SecurityValidator reads the global default (AdminConfigurationService threaded
through ValidateSecurityAsync + all device-auth callers); 5-min cached.
- UserRateLimitMiddleware: per-tenant user override for tenant-scoped callers only
(skipped for HasGlobalScope = GlobalAdmin|GlobalReader); GA -> GA budget, others ->
user default. Uses GetConfigurationIfExistsAsync so an unregistered-tenant JWT can
no longer auto-create a config row via read endpoints.
- Validation: reject rate limits < 1 at the write functions; RateLimitService clamps
< 1 to 1 as a safety net (avoids Min()-on-empty / throttle-everything).
- Web: Global Settings exposes User + GlobalAdmin limits; tenant modal writes the
override fields (blank = inherit); report + usage-plans updated.
Legacy RateLimitRequestsPerMinute Table column is ignored on read and dropped on the
next Replace save (self-cleaning; no migration).
Backend 2577 xUnit green, tsc clean, web 506 vitest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GlobalReader = cross-tenant READ visibility like a Global Admin, but no mutating
operations and no platform settings. Provisioned exactly like a GA: a GlobalAdmins
table row with Role=GlobalReader (empty/legacy Role = GlobalAdmin, back-compat).
Semantics are ADDITIVE — a user who is also their own tenant's Admin keeps edit
rights on that tenant; a pure GlobalReader is read-only everywhere.
Backend:
- GlobalAdmins.Role column + GlobalAdminService.GetGlobalRoleAsync; IsGlobalAdminAsync
(incl. TableAdminRepository) delegates so a GlobalReader row never reads back as admin
- RequestContext.IsGlobalReader + HasGlobalScope
- Split GlobalAdminOnly into GlobalReadOrAdmin (cross-tenant reads) vs GlobalAdminOnly
(writes/settings) + TenantAdminOrGlobalReader for config-read
- PolicyEnforcementMiddleware: extracted testable DecideAsync seam; cross-tenant gate keys
off HasGlobalScope; additive own-tenant role carried so IsTenantAdmin stays correct
- Config secret redaction for readers (RedactedCopyForReader) + save-time placeholder
restore (RestoreRedactedSecretsFrom) so a redacted view can't blank real secrets
- raw table/log tools stay GlobalAdminOnly; SignalR global-admins read group admits HasGlobalScope
- /auth/me emits isGlobalReader (+ preview-gate bypass, no auto-admin); /auth/mcp emits globalRole
MCP: hasGlobalScope (GA or Reader) drives routing + tool catalog; raw secret-bearing tools
(query_table/list_tables/query_backend_logs) gated strict-GA only.
Web: AuthContext isGlobalReader + hasGlobalScope; reader reaches /dashboard + cross-tenant
views; platform Settings/Ops/Backups/Software GA-only (nav-hidden + requireGlobalAdmin route
layouts); mutating controls on mixed data pages disabled via useCanMutatePlatform; rules pages
read-only on foreign tenants; additive own-tenant edit preserved.
Tests: backend 2369, MCP 252, web 469 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds WebhookProviderType.GenericJson (20) so a tenant can POST a stable,
channel-agnostic JSON payload to any HTTPS endpoint - ticketing systems,
automation, or an SMTP gateway like Postal - instead of only Teams/Slack.
Backend:
- GenericJsonRenderer emits schemaVersion 1.0 with eventType, severity,
title/summary/themeColor, primaryUrl (first openUrl action), facts[],
sections[], actions[]; registered in WebhookNotificationService.
- NotificationAlert.EventType set on all 8 builders (enrollment_succeeded/
_failed/_started, preprovisioning_*, hardware_rejected, sla_breach/_resolved,
consecutive_failures, test) for consumer-side routing.
- Per-tenant WebhookCustomHeadersJson + GetGenericWebhookHeaders() (generic-only,
restricted-header denylist, fail-soft parse) for API-key auth; dispatcher
gained optional customHeaders applied via HttpRequestMessage; threaded into
all 12 tenant call sites (Ops/admin-config sites unchanged).
- TableConfigRepository persists the field; UpdateTenantConfigurationFunction
validates it (object, string values, RFC-7230 token names, CRLF-injection
guard, 8 KB / 25-header caps).
Web:
- Generic provider option + key->value WebhookHeaderEditor (rendered only for
provider 20); leaving the generic provider clears local headers so a later
switch back cannot revive stale auth headers; context/types wiring sends
headers only when provider===20; report shows Generic JSON provider name.
ConfigVersion unchanged (backend-only, no agent payload). No new
Constants.EventTypes, so no MCP catalog sync. Still fire-and-forget (no SMTP
client / retry by design). New GenericWebhookTests cover renderer, header
parsing/denylist, and save-time validation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Grant tenant member roles via Entra ID app-role assignments on the
Enterprise App (the token's "roles" claim), in addition to the
TenantAdmins table. Per-tenant opt-in via the new GA-only
EntraAppRolesEnabled flag (backend-only, no ConfigVersion bump).
Resolution is table-first and tri-state (NotPresent/Disabled/Enabled):
an enabled table row wins, a disabled row is an explicit deny that
suppresses the claim entirely (a disabled member can never be
re-authorized through their app-role assignment), and only a missing
row falls back to the claim. Only Admin and Operator are claim-mappable
(Admin outranks Operator); Viewer and the platform GlobalAdmin role are
never claim-derived.
All effective-role consumers share one pure decision authority
(EntraAppRoleResolver.Resolve): PolicyEnforcementMiddleware, auth/me,
and feedback/status — the latter previously used a table-only check, so
Entra-only members did not see the feedback bubble.
GA toggle wired through the BootstrapToken pattern: TableConfigRepository
persistence, UpdateTenantConfigurationFunction GA-only protection, and a
checkbox in the admin tenant editor.
Tests: EntraAppRoleResolverTests (incl. disabled-row deny) + an
auto-promote regression guard; 2308/2308 backend green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cascade-delete V2 is the only path now. Drops the per-tenant
EnableCascadeDeleteV2 flag, the legacy ETag-CAS-guarded DeleteSessionAsync
helper, the AppInstallSummaries cleanup helper (only legacy caller),
and the SessionsLegacyDeleted counter chain (fanout → maintenance → OpsEvent).
DeleteSessionFunction collapses to a single kill-switch gate; the producer
owns 404 / 409 / Poisoned mapping AND the intentional Queued/Preparing+Snapshot
→ 202 recovery resume — gating those at HTTP would strand crashed admin
cascades. Fanout now aborts both the inner loop AND outer loop on producer
KillSwitchActive instead of leaking calls into the next tenant.
Test coverage:
- DeleteSessionCasGuardTests removed (helper gone)
- AdminConfigurationSessionDeletionKillSwitchTests trimmed to kill-switch only
- DeleteSessionFunctionTests pinned to the kill-switch short-circuit + null
- SessionRetentionFanoutServiceTests: legacy path tests dropped, two new
abort regressions for producer KillSwitchActive (inner + outer loop)
Build: 0 warnings, 0 errors. Backend: 1526 passed / 0 failed.
Web: tsc clean, vitest 346 passed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Operationalises the 7-PR backend cascade-delete pipeline that landed via commits
c8658b9f..921994e8: per-tenant V2 opt-in switch, audit-trail consolidation, and
a Global-Admin-only Session Cleanup page to track in-flight / poisoned / stranded
cascades and trigger restores.
== PR-A: per-tenant V2 enrollment ==
- UpdateTenantConfigurationFunction extends the GA-only gate list with
EnableCascadeDeleteV2 so tenant admins cannot self-opt-in / out of the
snapshot+restore pathway.
- TenantManagementSection.tsx adds a purple-bordered "Enable V2 Cascade Delete"
checkbox in the GA edit modal and a "Cascade V2" badge on the tenant card so
rollout state is visible at a glance.
== PR-B: audit consolidation ==
- SessionDeletionHandler drops the 15-per-cascade deletion_step_completed /
deletion_step_failed / deletion_verification_failed audit emissions. Step
telemetry lives in DeletionProgress + structured logs; tenant admins now see
just deletion_started + deletion_completed + deletion_restored.
- SessionDeletionWorker no longer writes a deletion_poisoned tenant audit;
the signal moved to a new SessionDeletionPoisoned OpsEvent (operator-bound,
Telegram-routable, dual-registered in OpsAlertRulesSection.tsx catalog).
IMaintenanceRepository dependency removed from the worker constructor.
- SessionRestoreService stops emitting the deletion_manifest_downloaded
breadcrumb; deletion_restored is the operator-visible outcome.
== PR-C: Session Cleanup admin page ==
- New /admin/ops/session-cleanup route (gated by app/admin/layout.tsx's
ProtectedRoute requireGlobalAdmin). Sidebar entry under ga-ops alongside
Maintenance. Four URL-state tabs: In-Flight / Poisoned / Stranded Queued /
Activity (links to existing /audit page).
- Backend reads: GetSessionDeletionsListFunction
(GET /api/global/session-deletions?state=&strandedSinceMinutes=) +
GetSessionDeletionManifestFunction
(GET /api/admin/sessions/{id}/deletion-manifest?tenantId=&manifestId=&mode=).
Both registered as GlobalAdminOnly in EndpointAccessPolicyCatalog.
- DeletionPreviewModal reads the persisted snapshot via the stored-manifest
endpoint (NOT a fresh dry-run via the existing /delete/preview, which would
enumerate live-but-partially-deleted data for in-flight cascades). Renders
manifest counts, snapshot SHA, worker progress (CompletedSteps,
VerificationDone, TombstoneStarted, aggregate decrements), and the last
verification failure with residual sample. Download button uses
authenticatedFetch + Blob + Object URL so the Bearer token is sent.
- RestoreConfirmDialog drops the Full/Partial radio (mode is auto-selected
by SessionRestoreService based on cascade state — forcing the wrong mode
would corrupt state). Reason field threads through to deletion_restored
audit details. Dry-run default = true and previews the auto-selected mode.
== Codex review follow-ups (across four rounds) ==
- High: tenantId is explicitly accepted on /delete/preview (query) and
/restore (body); GA cross-tenant resolution prefers the explicit value over
JWT TargetTenantId, falling back to SessionsIndex lookup. Without this, a
GA acting on a session in another tenant got their JWT home tenant.
- High: new stored-manifest endpoint replaces the dry-run preview for
in-flight / poisoned rows; previous fix built a fresh manifest against
already-partially-deleted data and misrepresented the cascade.
- Medium: SessionDeletionPoisoned OpsEvent now carries failureType +
failureMessage + observedResidualCount + a small residual preview, persisted
by the handler into DeletionProgress (LastFailureType, LastFailureMessage,
LastObservedResidualCount, LastResidualSampleJson) before the throw so the
worker can include them.
- Medium: dry-run restore reports the AUTO-SELECTED mode (full/partial) so
the dialog's preview reflects what the real run will do, instead of the
literal string "dryRun".
- Medium: residual count field renamed to observedResidualCount with an
honest XML doc — CascadeVerificationService caps at 50 per table AND
short-circuits after the first failing table, so the number is a lower
bound. UI surfaces "N of 50+ observed rows" when N hits the cap.
- Medium: three-layer cap defense on the OpsEvent residual sample to stay
under the 4096-char TableOpsEventRepository truncation: max 5 entries
(OpsEventResidualSamplePreviewSize), per-field max 96 chars with `…` marker
(OpsEventResidualKeyMaxChars), total preview budget 1200 chars
(OpsEventResidualPreviewBudgetChars) with trailing-entry drop. Full sample
stays in the progress blob for the admin UI.
- Low: estimatedSnapshotBytes field name aligned end-to-end; activity tab
text clarifies audit-action vs OpsEvent distinction.
Tests:
- Backend xUnit 1538/1538 pass (up from 1510 baseline; +28 across the new
Session Cleanup endpoints, OpsEvent enrichment, restore-mode reporting,
cap-defense unit + integration tests).
- Web vitest 347/347 pass.
- TypeScript strict + dotnet build clean.
Deliberately deferred:
- xUnit tests for GetSessionDeletionsListFunction +
GetSessionDeletionManifestFunction (no HttpRequestData mock harness in repo;
would be a separate refactor).
- vitest for DeletionPreviewModal + RestoreConfirmDialog (defer until first
field issue).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Oliver Kieselbach