Consumes the telemetry-index-reconcile queue (M5.d.2) and fans each
envelope out into 0–3 index-table rows via IIndexTableRepository
(M5.d.1). Closes the producer→consumer loop; once the
EnableIndexDualWrite flag is flipped, primary commits drive
eventual-consistent index upserts asynchronously.
Components:
- Microsoft.Azure.Functions.Worker.Extensions.Storage.Queues 5.5.4
package for the isolated-worker QueueTrigger binding.
- IndexReconcileHandler (plain class — not interface-abstracted per
Plan §M5.d design: DAL abstraction lives at IIndexTableRepository,
not here). Routing:
Signal → SignalsByKind (always, 1 row).
DecisionTransition:
IsTerminal=true → SessionsByTerminal
Taken=true → SessionsByStage
Taken=false + reason → DeadEndsByReason
ClassifierVerdictId!=null → ClassifierVerdictsByIdLevel
Up to 3 rows per DecisionTransition envelope (a Taken+terminal+
classifier transition fans to SessionsByTerminal + SessionsByStage
+ ClassifierVerdictsByIdLevel).
- IndexReconcileFunction thin QueueTrigger wrapper
(Connection = AzureTableStorageConnectionString). Malformed JSON
logged + dropped to avoid poison-queue retry loops; transient
repo errors propagate so the platform retries.
- DI: handler registered as Singleton in Program.cs.
Failure semantics:
- Required-field-missing envelopes (Signal without SessionSignalOrdinal
or SignalKind; DecisionTransition without StepIndex) → log warn,
drop. These won't become valid on retry — avoiding the retry loop.
- Transient Table Storage errors → rethrow. QueueTrigger platform
retries up to 5 times by default, then moves to
telemetry-index-reconcile-poison for manual investigation.
Deferred to M5.d.4:
- SessionsByStage stale-row pruning: when a session moves from
Stage-X→Stage-Y, the old Stage-X row stays. Either query-side
dedupe (per SessionId, max-Ticks RK) or reconcile-timer pruning
handles this. Deliberately not solved in the handler to avoid
a query+delete round-trip per Taken transition.
- End-to-end dual-write verification tests (Azurite harness) are
the M5.d.4 release gate before the flag is flipped.
Tests (+10 in Functions) via capture-only FakeIndexTableRepository:
- Signal happy path (1 SignalsByKind row).
- Signal missing SessionSignalOrdinal / SignalKind → 0 rows (x2).
- DecisionTransition Terminal+Taken+Classifier → 3 rows.
- DecisionTransition Taken non-terminal no-classifier → 1 row
(SessionsByStage only).
- DecisionTransition dead-end with classifier → 2 rows
(DeadEndsByReason + ClassifierVerdictsByIdLevel).
- DecisionTransition dead-end without reason → 0 rows (PK-collision
defense).
- DecisionTransition missing StepIndex → 0 rows (all paths gated).
- Unknown SourceKind → 0 rows.
- Null envelope → 0 rows.
Full Functions suite: 682 green (prev 672 + 10 new). Full solution
build green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the async fan-out producer path from Plan §2.8 / §M5.d so primary
Signals + DecisionTransitions rows can be projected onto their index
tables without blocking the hot ingest loop.
Components:
- Constants.QueueNames.TelemetryIndexReconcile (azure-queue-name-safe,
all lowercase + hyphens).
- IndexReconcileEnvelope (Shared DTO, flat shape with nullable fields
per SourceKind="Signal"/"DecisionTransition"). Avoids polymorphic
Newtonsoft deserialization — see M5.c.3 finding.
- IndexReconcileEnvelopeFactory pure static helper (FromSignal /
FromDecisionTransition / BuildBatch) — state-less projection.
- IIndexReconcileProducer (Shared interface) contract: no-op when
the feature flag is off; never rethrows underlying-queue failures.
- AzureQueueIndexReconcileProducer (Functions) — Base64-encoded JSON
messages, lazy CreateIfNotExistsAsync guarded by Interlocked flag,
Managed-Identity-or-connection-string construction mirroring
BlobStorageService / TableStorageService.
- IngestTelemetryFunction ctor gains IIndexReconcileProducer; after
primary Signals + DecisionTransitions commit it builds envelopes
and enqueues them. Event path is untouched (events don't feed the
index tables per §2.8).
- Program.cs registers the producer as a singleton.
Feature gate (AdminConfiguration.EnableIndexDualWrite, default false):
- New bool field in AdminConfiguration.
- TableConfigRepository Store + Map extension (per
feedback_table_storage_serialization).
- Producer checks config each call via AdminConfigurationService's
5-min memory cache. Flag off → primary behaviour is bit-exact.
Fault tolerance: queue-side exceptions (network, throttling, missing
queue before first CreateIfNotExists) are logged per envelope and
swallowed. The primary row is the source of truth; the 2h reconcile
timer (M5.d.4) is the safety-net recovery path.
Tests (+11 in Functions):
- 5 IndexReconcileEnvelopeFactoryTests: Signal/DecisionTransition
projection, dead-end preservation, batch ordering, null/empty.
- 3 IndexReconcileEnvelopeSerializationTests: Newtonsoft round-trip
(Signal, full DecisionTransition, minimal-envelope tolerance).
- 3 AdminConfigurationEnableIndexDualWriteTests: default-false on
new config + CreateDefault, persists when set.
Deferred to later sub-milestones:
- M5.d.3: queue-triggered consumer that maps envelopes to 0–3 index
rows using M5.d.1 TableIndexRepository.
- M5.d.4: 2h reconcile timer + release-gate dual-write verification
tests, then flip EnableIndexDualWrite=true in prod admin config.
Full Functions suite: 672 green (prev 661 + 11 new). Full solution
build green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extract the post-parse event-processing tail of IngestEventsFunction into a
standalone EventIngestProcessor service so /api/agent/telemetry runs the same
downstream logic as /api/agent/ingest. Deliberate copy-duplication, not a
shared refactor:
- Legacy /api/agent/ingest is production-hot — it serves live traffic from
every deployed V1 agent and is NOT touched by this commit. Zero regression
risk on the existing path.
- EventIngestProcessor is a near-verbatim copy of the legacy pipeline's tail
(~1000 LoC). Duplication is accepted because legacy /api/agent/ingest is
scheduled for decommission post-v11 GA (see tasks/todo.md → Follow-Ups →
Legacy-Agent-Ausmusterung). Until then: bugs fixed in legacy must be ported
here manually.
Copied pipeline steps (all 1:1 with legacy ProcessIngestAsync lines 150-566):
StampServerFields → SanitizeEventTimestamps → StoreEventsBatchAsync →
Index updates fire-and-forget → IME agent version history → ClassifyEvents
→ App install summaries → Geo-location update → UpdateSessionStatusAsync
(all 7 status branches) → IncrementSessionEventCountAsync → Rule engine
analysis fire-and-forget + SignalR + stats → Vulnerability correlation with
WhiteGlove Part-2 merge → IncrementPlatformStatAsync TotalEvents +
SuccessfulEnrollments → RecordGatherRuleStatsAsync → Diagnostics blob
update → GetSession for updatedSession → Session age 4h warning → WhiteGlove
IsPreProvisioned error log → SendWebhookNotificationsAsync (all 3 alert
paths) → SLA breach evaluation → Admin override detection → ServerAction
fetch-and-clear with TrackEvent telemetry → BuildSignalRMessages.
Shared (not duplicated, to avoid drift in pure helpers):
IngestEventsFunction.StampServerFields, SanitizeEventTimestamps,
RecalculateAppDurations (all internal static, state-less, stable). The
internal DTOs EventClassification + AppInstallAggregationState are also
reused — they carry no behaviour, only fields.
IngestTelemetryFunction wiring:
- Return type HttpResponseData → IngestEventsOutput so the V2 endpoint can
emit SignalR messages for the real-time UI push (matches legacy binding).
- PersistItemsAsync partitions the batch: Events go through the processor
(full pipeline + AdminAction + pending ServerActions + SignalR); Signal +
Transition batches skip the processor and read control signals directly
via a lean ReadControlSignalsAsync helper on ISessionRepository — no point
firing the full pipeline for a batch that carries no events.
- EventIngestProcessor registered in Program.cs DI.
Known gap (explicitly NOT in this commit): the V2 endpoint lacks a payload
size guard (legacy reads MaxNdjsonPayloadSizeMB from tenant config in
ParseNdjsonRequest). Different payload shape (JSON array vs NDJSON), deferred
to a dedicated follow-up.
All 596 backend tests green (unchanged count — copy path has no new tests
yet; the V2 endpoint's behaviour is validated by the legacy test coverage
through the shared-helper surface + the TelemetryPayloadParser tests from
M5.b.1). Integration coverage lands with M7 test-VM smoke.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduces a clean abstraction layer to decouple domain services from Azure Table Storage,
enabling future migration to Cosmos DB and Event Streaming without touching business logic.
Architecture:
- 10 repository interfaces in Shared/DataAccess/ (ISessionRepository, IRuleRepository, etc.)
- IDataEventPublisher interface with NullDataEventPublisher for future event streaming
- IStorageInitializer for provider-agnostic initialization
- 5 Table Storage implementations that delegate to existing TableStorageService
- DI extension method AddTableStorageDataAccess() with AddEventStreaming<T>() hook
- Migration plan in tasks/todo.md for incremental adoption
https://claude.ai/code/session_01NHWJfPaa8TQHuPzMD7gG63
Adds a second device validation method alongside existing Autopilot Device
Validation. Corporate Identifier Validation checks devices against Intune
Corporate Device Identifiers (manufacturer + model + serial number) via the
Graph beta searchExistingIdentities endpoint.
Backend: Extract GraphTokenService from AutopilotDeviceValidator for shared
token/consent logic. Add CorporateIdentifierValidator with caching. Update
SecurityValidator with cascading validation (Autopilot first, Corporate
Identifier fallback). Wire new validator through all agent endpoints.
Frontend: Add second toggle in settings gated behind preview mode. Shared
consent flow skips redirect when consent already exists from other toggle.
Section renamed to "Enrollment Device Validation" in preview mode.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Galactic Admin Mode for Audit Logs: banner + cross-tenant endpoint (galactic/audit/logs)
- Audit page pagination (12 entries per page) with Previous/Next controls
- Platform Metrics: moved computation to backend PlatformMetricsService with 5-min cache
- New endpoint GET /api/galactic/metrics/platform replaces N+1 client-side fetches
- IME log tracker enhancements, DO telemetry pattern, and download progress improvements
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tenant Admins can report sessions for analysis via a new button on the
session details page. Creates a ZIP with session JSON, events JSON,
analysis results, timeline TXT, events CSV, optional agent log (5 MB
limit), and optional screenshot — uploaded to central blob storage.
Admin-config page gets a new Session Reports table (Galactic Admin only)
showing all submitted reports with comment, email, and blob reference.
Export utilities (generateUiExport, generateCsvExport) extracted to
shared lib/sessionExportUtils.ts for reuse across admin export and
report modal.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
## Rules as GitHub single source of truth
- Add 75 individual JSON rule files (2 gather, 18 analyze, 55 IME patterns)
under rules/gather/, rules/analyze/, rules/ime-log-patterns/
- Add JSON Schemas for all three rule types with VS Code IntelliSense support
- Add CI combine script (rules/scripts/combine.js) + GitHub Action that
validates schemas with ajv-cli and generates rules/dist/ combined files
- Convert BuiltIn*.cs from hardcoded C# objects to embedded JSON resource loaders
- Add GitHubRuleRepository service to fetch dist/ files from raw.githubusercontent.com
- Add ReseedFromGitHubFunction: POST /api/rules/reseed-from-github?type=all|gather|analyze|ime
## IME Log Patterns
- Extend ImeLogPattern model with Description and IsBuiltIn properties
- Add ImeLogPatterns table + API constants
- Add TableStorageService CRUD methods for IME log patterns
- Add ImeLogPatternService (CRUD, merge, auto-seed from embedded JSON)
- Add ImeLogPatternsFunction: GET/PUT /api/rules/ime-log-patterns (Phase 1: read-only
for tenants, Galactic Admin can enable/disable/edit globally)
- Add IME Log Patterns management page (/ime-log-patterns)
- Wire GetAgentConfigFunction to use ImeLogPatternService instead of static class
## API route consolidation
- All rule endpoints now under /api/rules/* for single certificate exclusion path:
/api/rules/gather, /api/rules/analyze, /api/rules/ime-log-patterns,
/api/rules/reseed-from-github
## Export buttons
- Add per-rule and export-all JSON download buttons to gather-rules,
analyze-rules, and ime-log-patterns pages
## Admin configuration
- Replace old individual reseed buttons with unified "Fetch & Reseed from GitHub"
(calls new GitHub endpoint) plus compact individual-type reseed buttons
- Add CDN cache hint: GitHub raw.githubusercontent.com caches up to 5 minutes
## Preview mode
- ?preview=1 URL param seeds localStorage.previewMode (persists across navigation)
- ?preview=0 clears it; toggle in Navbar settings dropdown (indigo, eye icon)
- IME Log Patterns nav link gated behind galacticAdminMode OR previewMode
- Preview badge shown on nav link when active for normal users
## Tests
- Add xUnit test project (net8.0) with 5 tests for embedded resource loading,
known IME actions validation, and valid categories check
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Galactic Admins can block a device by serial number from the admin-configuration
page. The block is enforced at ingest time (after security validation, before
body decompression) and is configurable in duration (default 12h, max 30 days).
The agent receives a DeviceBlocked signal in the IngestEventsResponse and stops
all uploads for the session. Blocks persist across backend restarts via a new
BlockedDevices table in Azure Table Storage, with an in-memory cache for fast
ingest lookups.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sends a Telegram message to @AutopilotMonitor-PrivatePreview whenever
a new tenant signs up. The bot webhook URL is stored in the new
PreviewConfig table (PartitionKey=TelegramBot, RowKey=config,
WebhookUrl=<url>) — same lifecycle as PreviewWhitelist, easy to remove
after GA. Notification is fire-and-forget and non-fatal.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Oliver Kieselbach