mirror of
https://github.com/okieselbach/AutopilotMonitor.git
synced 2026-10-25 13:10:13 +00:00
88dfd331dde5cbb87c83c0a6f796f4b85a204275
18 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Oliver Kieselbach
|
fc46b4c78e |
feat(security): session-owner binding stage 1 (shadow) + dormant agent session rotation
Agent endpoints authenticated per tenant but authorized session writes on a caller-supplied session id. Sessions rows now carry the creating device identity (cert thumbprint + Intune device id from the CN, or bootstrap code + serial) in primary-only Owner* columns. SessionOwnershipPolicy evaluates register / telemetry / error-report requests against it; SessionOwnerBindingObserver records the outcome on the request row, as a warning and as a throttled SessionOwnerMismatch ops event. Nothing is rejected in this stage. Agent: errorCode parsing on 401/403, one-shot session rotation on session_owner_mismatch without feeding the auth-failure tracker, spool drop, agent_started.sessionRotated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
ea6a375c4b |
feat(security): cert-to-device binding, stage 3 GA-gated shadow
Tenant binding (stage 1) proves the client certificate was issued to this
tenant. It does not prove the certificate belongs to a device the tenant
still has: a cert lifted from a decommissioned machine keeps working.
IntuneDeviceBindingValidator resolves the Intune managedDevice id from the
certificate's Subject CN against the tenant's own managedDevices inventory.
Point-GET on managedDevices/{id} rather than an OData filter, so there is no
filter literal to inject into and a 404 is unambiguous. Cache/retry/transient
contract mirrors CloudPcDeviceValidator.
Shadow only, and visible only to Global Admins (same gate as the DevPrep
association preview) because the open question is a race, not an attack: a
device object can appear after the agent's first call. Every result carries
enrolledDateTime and the log line records enrolledAgeSeconds, so a device
created seconds before the request is distinguishable from a foreign cert.
Grep marker: CERT-DEVICE-BINDING-SHADOW.
The lookup is fire-and-forget like the DevPrep shadow so a cold Graph
round-trip never sits in the agent request path. That rules out stage 1's
request-row dimension (the row is written before the task finishes), hence
the Warning log; widening beyond preview should move to inline + dimension.
Needs DeviceManagementManagedDevices.Read.All, granted per tenant via the
customer script as feature IntuneDeviceBinding. The script's catalog,
ValidateSet and help block stay pinned to GraphFeatureCatalog by
GraphFeatureCatalogSyncTests (extended with the help-block case).
Build 0 warnings, backend 4629/4629, web 874/874, tsc clean, eslint 0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
990641b19e |
feat: P14 - per-upload-batch send time separates spool delay from clock offset
Agent stamps X-Send-Time-Utc (device clock, ISO-8601 round-trip) per upload ATTEMPT; ingest parses it (year>=2000 guard, deliberately no upper clamp) and server-stamps SentAt onto every event of the request. SentAt - OccurredUtc is pure spool delay, ReceivedAt - SentAt is network + device-vs-server clock offset - previously indistinguishable and only separable via median statistics. clock_skew condition: batches carrying SentAt measure the frame directly (spool-immune by construction - no spread cap, IME/clamped exclusions irrelevant, IME-only sessions become measurable; evidence sentAtBatchCount); legacy batches keep the per-event median path unchanged. CmTraceSkewTripwire deliberately untouched: its differential median math cancels clock offset as common mode. Full persistence chain: Store x2 + Map, fields= projection (sentAt), MCP get_session_events field list + rule-authoring-guide. New OKF doc backend/send-time-frame-separation.md. Suites: Functions 4370 (+16), Agent 2407 (+1), MCP vitest 564 + tsc 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
c4280e823c |
fragility audit P5.x + P7: drift guards across SignalR names, RowKeys, config sections, scoped routing
P5.x-A: Constants.SignalRMessages catalog (13 names) + SignalRGroupHelper.HubName;
all backend send sites and web subscriptions typed against the shared-manifest
union (SignalRContext.on/off) — unknown names are tsc errors. Dead eventStream
session-payload handling removed from diagnosis page.
P5.x-B: RowKeyCodec as the single inverted-tick encoder (persisted widths pinned,
UserActivity D20 legacy); DeletionRowDump.Table on tombstone rows with
DeletionTombstoneTables.Resolve replacing the duplicated Contains('_') heuristic
(legacy fallback retained for pre-field manifests).
P7.1: PATCH config/{tenantId}/fields flipped to TenantAdminOrGA (phase 2);
Settings sections save via per-section diff PATCH from sectionFieldMap.ts —
the ~39-field stale round-trip vector is gone; legacy webhook clearing now only
on the notifications save; agent-settings reset no longer discards contact edits.
P7.2/7.3: useGlobalAdminScope is a projection over useAggregatedAdminScope
(resolveConcreteScopeView, test-pinned); scopedApi builders own the tenant/global
endpoint-pair routing for all 11 former hand-rolled call sites; scopedFetch adds
the shared JSON fetch layer.
P7.4: navVisibility extraction + roles x nav-config matrix and nav<->route-guard
consistency test with a ratchet for the two known pre-existing mismatches
(Operator and GlobalReader vs /settings — user decision pending).
P7.5: web build stamp public/version.json (prebuild, GITHUB_SHA), SWA no-store
route, deploy-web verify poll + versions/web.json manifest, Portal Build card.
P7.6: OkAsyncBaselineGuardTests freezes the 44 anonymous-response sites
(shrink-only ratchet).
Verified: dotnet build 0W/0E, Functions.Tests 4123, DecisionCore 519, V2.Core
2206, SummaryDialog 44; tsc 0, ESLint 0, vitest 762/762; mutation proofs for all
new drift nets. Docs: docs/drift-guards.md + versioning.md web row.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
1c5f69a9fe |
w365: CloudPc device validator — cert-CN-bound fallback for Cloud PCs (#127)
* w365: CloudPc device validator — cert-CN-bound fallback for Cloud PCs Windows 365 Cloud PCs are never Autopilot-registered, so the serial lookup (and the Corporate Identifier fallback) structurally reject them and Cloud PC first-connect enrollment could not be monitored. New stage-3 fallback in the device-validation chain: extract the Intune device id from the chain-validated MDM client certificate's Subject CN (field-verified: CN == managedDevice id == cloudPC.managedDeviceId) and require a matching cloudPC object in the claimed tenant (Graph v1.0 virtualEndpoint/cloudPCs, $filter=managedDeviceId eq). Only service-provisioned Cloud PCs have such an object — a regular enrolled non-Autopilot device still ends in the 403, and local W365 markers are never consulted for auth. - Shared: ValidatorType.CloudPc = 5, TenantConfiguration.ValidateCloudPcDevice (default false, tenant-admin writable), GraphAppPermissions.CloudPCReadAll, GraphFeatureCatalog feature W365CloudPcValidation -> CloudPC.Read.All - Backend: CloudPcDeviceValidator (AutopilotDeviceValidator sibling contract: 30/5 min cache, 2 attempts, transient->503; GUID gate before the OData filter; exact-match guard; Graph 403 = definitive "grant missing", not transient), SecurityValidator CN extraction + fallback stage + hard gate, wiring through the 5 agent endpoints, TableConfigRepository persistence - Consent script: W365CloudPcValidation in catalog/ValidateSet/help; NEW GraphFeatureCatalogSyncTests pins C# catalog <-> grant script (previously comment-only lock-step) - Bootstrap dev script v2.3-dev.3: guard 4 (uptime window) exempted while the Cloud-PC relax is active — Cloud PCs run headless for days before first connect; the fresh-profile window is the time anchor there (+ Pester tests) - Web: validation toggle + context plumbing, ValidatedBy "CloudPc" label, privacy-page validation claim generalized (Last updated bumped) - Docs: OKF concept doc backend/cloudpc-device-validation.md + index + log Verified: web tsc clean, ESLint 0 errors (99 pre-existing warnings = ratchet cap, none added). .NET build/tests and Pester run in CI — SDK download and pwsh are unavailable in this sandbox's network policy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RqEFrcAHhWgCvgJgHGzMuv * w365: agent-side IsCloudPc flag + engine expectation "no Device-ESP phase" Companion to the CloudPc validator: the agent now classifies Windows 365 Cloud PCs locally with the bootstrap's field-verified marker AND (HKLM\SOFTWARE\Microsoft\Windows365 key + installed CloudManagedDesktopExtension service, probed via its Services registry key so net48 needs no System.ServiceProcess). SKIP-safe: any error resolves false. Never an auth input - backend admission stays exclusively the cert-CN-bound Graph lookup (ValidatedBy=CloudPc); the two signals are independent and may disagree. Two transport rails: - Session metadata: SessionRegistration.IsCloudPc -> Sessions table (sticky-true OR across re-registrations, IsSelfDeployingProfile pattern) -> SessionsIndex full mirror -> SessionSummary -> new isCloudPc search filter end to end (SearchSessions + QueryRawSessions + pagination fingerprint + all three in-memory predicates; MCP search_sessions / query_raw_sessions params + summary projection; web type, Cloud PC pill, SessionInfoCard enrollment label, CSV export appended column). - Decision engine: isCloudPc rides the EnrollmentFactsObserved payload, recorded set-once for BOTH values in EnrollmentScenarioObservations.CloudPc and annotates the scenario profile reason cloud_pc_first_connect:no_device_esp_expected - the engine-recorded expectation that Device-ESP already ran headless at provisioning and the session starts at Account Setup. Deliberately reason+observation only: no completion arm depends on DeviceSetup, Mode stays AccountSetup/IME- classified (Classic), EspConfig keeps configured-FirstSync semantics. Positive marker surfaces as cloud_pc_marker in the audit-trail census. Updater idempotency: the cloudPcAnnotates gate keeps the generic v1 reason branch from re-claiming the slot on repeat posts (would bump EvidenceOrdinal every post). Additive-nullable observation - no state schema bump (v4 convention), serializer roundtrip pinned. Tests: CloudPcDetector resolver (marker AND, throwing probe), registration body, facts-signal payload shape, updater annotation/idempotency/ self-deploying interplay, reducer observation both values + census fold, serializer roundtrip, index projection superset, sticky-true both directions. Suites: DecisionCore 500, Functions 3941, Agent V2 2154, vitest 632 - all green; tsc web+MCP clean; ESLint 0 errors (cap 99). Docs: cloudpc-device-validation.md follow-up section replaced with the shipped mechanism + log entry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
4a33b96db2 |
feat(ratelimit): unify device+user limits as global default + per-tenant override
Replace the denormalized per-tenant RateLimitRequestsPerMinute mirror (and its SaveConfigurationAsync sync job) with compute-at-read resolution: effective limit = per-tenant override ?? global default. Fixes the clobbering bug where a per-tenant edit to the base field was overwritten on the next global config save. - Model: drop TenantConfiguration.RateLimitRequestsPerMinute; add nullable CustomUserRateLimitRequestsPerMinute (device override CustomRateLimit... kept). - RateLimitResolver centralizes precedence (device + user); both paths call it. - SecurityValidator reads the global default (AdminConfigurationService threaded through ValidateSecurityAsync + all device-auth callers); 5-min cached. - UserRateLimitMiddleware: per-tenant user override for tenant-scoped callers only (skipped for HasGlobalScope = GlobalAdmin|GlobalReader); GA -> GA budget, others -> user default. Uses GetConfigurationIfExistsAsync so an unregistered-tenant JWT can no longer auto-create a config row via read endpoints. - Validation: reject rate limits < 1 at the write functions; RateLimitService clamps < 1 to 1 as a safety net (avoids Min()-on-empty / throttle-everything). - Web: Global Settings exposes User + GlobalAdmin limits; tenant modal writes the override fields (blank = inherit); report + usage-plans updated. Legacy RateLimitRequestsPerMinute Table column is ignored on read and dropped on the next Replace save (self-cleaning; no migration). Backend 2577 xUnit green, tsc clean, web 506 vitest green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
34582c0f0e |
feat(kill-switch): deliver kill on config channel + KillSignalDelivered ops event
Kill/Block rules were only served on the telemetry response, which cannot reach an agent whose upload loop is paused (indefinite block pins the drain until DateTime.MaxValue), whose spool is empty, or whose binary is kill-blind. Config is fetched at every process start (boot via Scheduled Task), making it the reliable control channel - no heartbeat added by design (48h emergency break + 6h max lifetime bound the remaining cases). Backend (effective on deploy): - KillSwitchEvaluator: shared device+version check for both channels; ingest wire response and check order unchanged (device first, short-circuits) - Every served Kill emits ops event KillSignalDelivered (Security/Warning), throttled 24h per tenant+serial+pattern; Blocks stay event-free - GET agent/config (incl. bootstrap wrapper) now returns DeviceBlocked/ DeviceKillSignal/UnblockAt alongside the full config; old binaries ignore the new fields - Web: KillSignalDelivered dual-registered in OPS_EVENT_TYPES Agent V2 (effective with next agent release): - CheckConfigKillSignal guard after the Phase-4 fetch: forced self-destruct (overrides SelfDestructOnComplete), enrollment-complete marker, session cleared, exit before mTLS/registration - Kill honoured ONLY from a live fetch: config disk cache strips the flags on write and read (UnrestrictedMode pattern), guard re-checks LastFetchOutcome Tests: 9 KillSwitchEvaluatorTests + 7 ProgramConfigKillSignalTests; Functions 2557/2557, Agent V2 1578/1578, web tsc clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
0f82577606 |
fix(backend): address review mediums M6-M9 (orphan cleanup, retention starvation, ingest observability, counter idempotency)
M6: DeleteByFilterInBatchesAsync is fail-loud and counts only confirmed batch submits; the orphan cleanup keeps the EventSessionIndex entry when the event delete fails, so surviving rows stay discoverable and retry. M7: GetSessionsOlderThanAsync gains excludeInFlightDeletions - sessions locked in deletion states no longer occupy the capped read head, so a stuck head (Poisoned / stranded Queued) cannot starve the retention fanout tail. Warnings surface a fully-locked backlog and zero-progress runs. M8: ingest logs a per-batch warning with per-kind counts and the first TelemetryItemId when unparseable telemetry items are dropped (agent clears its spool on success, so drops were silent data loss). M9: ReconcileSessionRebootCountAsync -> ReconcileSessionCountersAsync; one partition scan reconciles EventCount and RebootCount at terminal transitions, self-correcting at-least-once retry double-counts. Script counters stay increment-only with the drift documented. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
9f7b070e80 |
chore: remove legacy V1 agent and its backend ingest path
The V1 agent line is retired: no new V1 sessions since 2026-05-09, only orphaned straggler traffic remained on /api/agent/ingest (last seen 2026-06-29, events for long-cleaned-up sessions). Bootstrap/Intune/CI have deployed V2-only since the v2 cutover. Agent: - Delete AutopilotMonitor.Agent, .Agent.Core, .Agent.Core.Tests (projects + sln entries). V2 trio, SummaryDialog, DecisionCore.Tests and V2Isolation.targets stay. Backend: - Relocate the shared statics/DTOs V2 reused from the V1 partials (StampServerFields, SanitizeEventTimestamps, IsMaxLifetimeAgentShutdown, RecalculateAppDurations, EventClassification, AppInstallAggregationState) into EventIngestProcessor; IngestEventsOutput moves next to IngestTelemetryFunction (same namespace, zero ripple). - Delete IngestEventsFunction (route agent/ingest), BootstrapIngestEvents (route bootstrap/ingest), NdjsonParser and their DI registration. EventDataNormalizer stays (used by TelemetryPayloadParser). - Drop both routes from EndpointAccessPolicyCatalog and RequestTelemetryMiddleware.TenantHeaderTrustedPaths. - Retire the V1 update-hash oracle: LatestAgentV1*/LatestBootstrapV1* fields + GetAgentLine case 1 (default arm returns empty hashes, so stragglers gracefully skip the integrity check; ParseAgentMajor keeps its default of 1 on purpose — defaulting to 2 would feed V2 hashes to V1 binaries). TableConfigRepository Store+Map updated in lockstep. - Remove ApiEndpoints.IngestEvents/.BootstrapIngestEvents constants. Tests: NdjsonParserTests deleted; timestamp/critical-path/max-lifetime tests repointed to EventIngestProcessor; the two NDJSON pipeline tests removed (coverage retained by direct-list siblings); catalog/no-store InlineData rows updated. Docs: architecture.md (endpoint tables, ingest flow diagram, solution layout, test matrix), web SectionAgentInternals endpoint tables, build.ps1 help text. Azure: no config change needed — /api/agent is not in the client-cert exclusion paths (agent/ingest was always cert-enforced) and the /api/bootstrap prefix exclusion remains for the surviving bootstrap routes (verified via ARM on autopilotmonitor-api). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
f37ee3e6b1 |
perf(backend): cache Graph app token + stream-parse V2 ingest
Ingest hot path + Graph auth latency. IngestTelemetry is the #1 backend endpoint (~35.6k/day); its p99 tail was dominated not by parsing but by an uncached Graph token: GraphTokenService minted a fresh client-credentials token (500-1900ms POST) on every device-validation cache miss and threw it away. Cache it per tenant. GraphTokenService: - Per-tenant token cache keyed on the validated tenantId; TTL from the token's own expires_in minus a 5min skew margin, clamped to [60s, 55min] (expires_in was previously ignored). Only successful tokens are cached. - Per-tenant SemaphoreSlim stampede gate (bounded wait) so an enrollment wave does not fan a burst of identical token POSTs at AAD. - InvalidateTenant() drops the token + consent-status entries so the consent/Graph-permission fresh-read contract still holds. - Per-tenant invalidation generation closes the write-after-invalidate race: a POST in flight when InvalidateTenant runs tags its entry with the pre-invalidation generation, which readers reject. GraphFeatureDetector: - InvalidateTenant() now also invalidates the underlying token cache, and applies the same generation guard to its own parsed-roles cache. - Cross-layer order is load-bearing: invalidate the token layer FIRST, then bump the detector generation, so a concurrent acquire cannot pair a new detector generation with a stale token read. IngestTelemetryFunction: - Deserialize the request body directly from the buffered MemoryStream via JsonTextReader instead of materializing a full UTF-16 string + ToArray() copy (peak ~4N -> ~1N). Cap still applies before parsing; same parse semantics. Pure allocation hygiene on the hottest endpoint. Tests: token caching, cross-tenant isolation, failure-not-cached, stampede coalescing (barrier-gated), in-flight write-after-invalidate, cross-layer invalidation, and cross-layer ordering -- each verified red without its fix. Parse hygiene: PayloadJson stays a raw string, malformed -> 400. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
f4d51ea499 |
perf(backend): cut App Insights cost + ingest hot-path reads (review follow-through)
Telemetry cost (L4): worker-side adaptive sampling now excludes Request;Event so the enriched RequestTelemetryMiddleware copy is never thinned; host.json Host.Results=Error drops the duplicate unenriched host request item. One complete request row per call instead of two (one unsampled+unenriched, one sampled+enriched). Ingest hot path (H2/H3): BlockedDeviceService caches negative answers (30s revalidation, symmetric with positive entries) so healthy devices no longer pay a BlockedDevices point-read per request; SessionDeletionGuard returns its already-loaded row and IncrementSessionEventCountAsync returns the post-merge snapshot, collapsing ~4 Sessions reads/batch to ~2. Queue-worker correctness (O1/O2): new GetSessionEventsStrictAsync propagates storage failures on the analyze + vulnerability-correlate paths so a transient read fault retries/poisons instead of silently deleting the message after "analyzing" an empty stream. Fail-soft variant kept for UI/metrics surfaces. Review fixes: parse prefetched Status as string (the actual wire format, was int -> optimization never fired); session-aware block over a cached negative entry keeps its session scope instead of widening to whole-device. 2274/2274 backend tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
921994e845 |
fix(backend): cascade-delete codex follow-ups (F1 V2 guard + upsert-revival, F2 partial-restore key-level re-increment, F3 legacy delete sessions-first, F4 restore 400 EntityAlreadyExists)
F1 — V2 ingest writes were landing past the cascade lock. Wired SessionDeletionGuard.EnsureWritableAsync into IngestTelemetryFunction before PersistItemsAsync so Event / Signal / Transition writes are refused with 410 Gone when DeletionState is Preparing / Queued / Running / Poisoned. Switched UpdateSessionImeAgentVersionAsync from UpsertEntityAsync to UpdateEntityAsync(ETag.All, Merge) with 404 swallow so a tombstoned Sessions row can no longer be silently resurrected by an in-flight ingest. F2 — Partial-restore could leave SoftwareInventory at -N drift when the cascade poisoned mid-AGGREGATE-loop: per-key decrement was persisted into AggregateDecrementsApplied BEFORE the underlying decrement, but the gate used CompletedSteps.Contains(aggregate.Order) — which is only set AFTER the whole loop. Switched the gate to AggregateDecrementsApplied as the authoritative ledger and pass a filtered DeletionStep into RunInventoryReIncrementWithIdempotencyAsync so only the keys that were actually decremented are re-incremented. F3 — Legacy direct-delete path deleted Events / RuleResults / AppInstallSummaries before the ETag-CAS-guarded DeleteSessionAsync, leaving a TOCTOU window where a V2 producer could CAS-claim DeletionState=Preparing after the pre-read but before our side- table mutations — landing the producer's manifest snapshot on top of already-half-purged state. Reordered both admin (DeleteSessionFunction) and retention (SessionRetentionFanoutService) paths to Sessions-CAS first; side-tables only after the Sessions row is gone (no producer can claim a missing row). F4 — Restore per-row Add fallback only caught HTTP 409 for duplicate- row conflicts; Azurite and some legacy Azure deployments surface duplicate-row as 400 with ErrorCode=EntityAlreadyExists, breaking idempotent restore-retry after partial-failed restore. Centralised the check via IsAlreadyExistsStatus(rfe) helper accepting 409 OR 400 + ErrorCode=EntityAlreadyExists; non-duplicate 400s still propagate. Tests: - SessionRestoreServiceTests: 2 partial-restore tests refit to the new AggregateDecrementsApplied contract + new explicit Codex-scenario test (only-decremented-keys re-incremented, untouched key skipped). - SessionDeletionPr4cCodexFixesTests F4: harness setup adds full AggregateDecrementsApplied so the per-key idempotency assertion reflects the new gate. - SessionRetentionFanoutServiceTests: pre-read tests replaced by two tests covering "DeleteSessionAsync refuses → no side tables" and "DeleteSessionAsync succeeds → side tables run after". - TableStorageRestoreBatchTests: two new F4 tests — 400 EntityAlreadyExists treated as Skipped, 400 with non-duplicate ErrorCode still propagates. - UpdateSessionImeAgentVersionTests (NEW): asserts UpdateEntityAsync is used (not Upsert), 404 swallowed silently, other storage errors non-fatal. Suites: Backend xUnit 1510, Agent V1 530, Agent V2 1042 — all green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
7b64f98e63 |
v2 single-rail 9: soft admin-action semantics + shutdown-race fixes
Addresses two tasks/todo.md entries from 2026-04-23:
- Soft Admin-Action Semantik + SignalIngress-Race-Fix
- Backend AdminAction-Trigger fix
Backend — authoritative AdminMarkedAction field:
New Session.AdminMarkedAction property (null | "Succeeded" | "Failed")
replaces the fragile heuristic "Status is terminal AND current event is not
a completion marker", which fired falsely on every post-completion event
the agent sent (agent_shutting_down, diagnostics_uploaded,
enrollment_summary_shown). The backend's AdminAction response to /register
and /ingest now reads the explicit field. ISessionRepository +
TableSessionRepository + TableStorageService.Sessions +
Ingest{Events,Telemetry}Function + MarkSession{Succeeded,Failed}Function +
RegisterSessionFunction + EventIngestProcessor all updated to set / clear /
read the new field coherently.
Agent — Program.cs WireTelemetryServerResponse refactor:
Routes TelemetryUploadOrchestrator.ServerResponseReceived through three
distinct semantic paths instead of one terminate_session synthesis:
1. DeviceKillSignal -> terminate_session with forceSelfDestruct=true
(hard admin kill — overrides SelfDestructOnComplete).
2. AdminAction=Succeeded -> single admin_marked_session timeline entry
(informational; agent continues own path).
3. AdminAction=Failed -> EnrollmentTerminationHandler.Handle(outcome=Failed)
(soft shutdown; same analyzers/diagnostics/summary
as a locally-detected failure).
Plus a shutdown-race guard: if shutdownComplete is already set, short-circuit
instead of crashing deep in the dispatcher with an InvalidOperationException
from a disposed SignalIngress. Backed by new HandleAdminActionTests.
Agent — ServerActionDispatcher ingress-disposed race:
Added InvalidOperationException swallow around InformationalEventPost.Emit
for the narrow window between SignalIngress.Stop and
TelemetryUploadOrchestrator.DrainAllAsync completion. Dropped events are
acceptable during teardown; the alternative was a nested Emit failure that
corrupted the outer termination sequence. Backed by new
ServerActionDispatcherTests case.
Tests all green: DecisionCore 133, Agent.V2.Core 586, Functions 727.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
ee07059bc3 |
fix(backend): harden /api/agent/telemetry batch validation (Codex Finding 3)
The ingest endpoint trusted the first item's PartitionKey for the whole
batch and silently defaulted RowKey-driver fields to 0 when the agent
envelope omitted them. Both paths can misroute data on the storage layer.
1. Cross-session guard: every TelemetryItemDto in a batch must share the
same PartitionKey as items[0]. Without this, a mixed-session batch
would be stamped with session-A's identity inside PersistItemsAsync
and silently relocate session-B's signals/transitions into session-A's
primary rows. Mismatch now returns 400 BadRequest with the offending
index logged. Extracted as pure helper FindMismatchingPartitionKey so
the guard is unit-testable without an HTTP harness.
2. RowKey-driver enforcement in TelemetryPayloadParser:
- ParseSignal now rejects envelopes where SessionSignalOrdinal is
absent or null (returns null → caller drops the item). A missing
value would default to 0 and clobber a legitimate ordinal-0 row
on UpsertReplace. Explicit ordinal=0 stays a valid value.
- ParseTransition applies the same defense to StepIndex.
- Distinguishes "key missing" from "key present with value 0" via
JObject.TryGetValue + JTokenType check.
Tests (+11 in Functions):
- IngestTelemetryFunctionTests: empty batch, single-item batch, uniform
batch, mismatch in middle, case-sensitive mismatch.
- TelemetryPayloadParserTests: missing/null SessionSignalOrdinal on
Signal → null; explicit ordinal 0 accepted; missing/null StepIndex on
DecisionTransition → null; explicit StepIndex 0 accepted.
Full Functions suite: 703 green (prev 692 + 11 new).
Does not address Codex Findings 1 (V2 agent has no Signal/Transition
upload path) or 2 (AdminAction=Succeeded mis-mapped to Failed locally)
— those land in follow-up commits on this branch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
c3ee0cad55 |
feat(backend): M5.d.2 index-reconcile queue producer + feature flag
Adds the async fan-out producer path from Plan §2.8 / §M5.d so primary Signals + DecisionTransitions rows can be projected onto their index tables without blocking the hot ingest loop. Components: - Constants.QueueNames.TelemetryIndexReconcile (azure-queue-name-safe, all lowercase + hyphens). - IndexReconcileEnvelope (Shared DTO, flat shape with nullable fields per SourceKind="Signal"/"DecisionTransition"). Avoids polymorphic Newtonsoft deserialization — see M5.c.3 finding. - IndexReconcileEnvelopeFactory pure static helper (FromSignal / FromDecisionTransition / BuildBatch) — state-less projection. - IIndexReconcileProducer (Shared interface) contract: no-op when the feature flag is off; never rethrows underlying-queue failures. - AzureQueueIndexReconcileProducer (Functions) — Base64-encoded JSON messages, lazy CreateIfNotExistsAsync guarded by Interlocked flag, Managed-Identity-or-connection-string construction mirroring BlobStorageService / TableStorageService. - IngestTelemetryFunction ctor gains IIndexReconcileProducer; after primary Signals + DecisionTransitions commit it builds envelopes and enqueues them. Event path is untouched (events don't feed the index tables per §2.8). - Program.cs registers the producer as a singleton. Feature gate (AdminConfiguration.EnableIndexDualWrite, default false): - New bool field in AdminConfiguration. - TableConfigRepository Store + Map extension (per feedback_table_storage_serialization). - Producer checks config each call via AdminConfigurationService's 5-min memory cache. Flag off → primary behaviour is bit-exact. Fault tolerance: queue-side exceptions (network, throttling, missing queue before first CreateIfNotExists) are logged per envelope and swallowed. The primary row is the source of truth; the 2h reconcile timer (M5.d.4) is the safety-net recovery path. Tests (+11 in Functions): - 5 IndexReconcileEnvelopeFactoryTests: Signal/DecisionTransition projection, dead-end preservation, batch ordering, null/empty. - 3 IndexReconcileEnvelopeSerializationTests: Newtonsoft round-trip (Signal, full DecisionTransition, minimal-envelope tolerance). - 3 AdminConfigurationEnableIndexDualWriteTests: default-false on new config + CreateDefault, persists when set. Deferred to later sub-milestones: - M5.d.3: queue-triggered consumer that maps envelopes to 0–3 index rows using M5.d.1 TableIndexRepository. - M5.d.4: 2h reconcile timer + release-gate dual-write verification tests, then flip EnableIndexDualWrite=true in prod admin config. Full Functions suite: 672 green (prev 661 + 11 new). Full solution build green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
0936132a00 |
feat(backend): M5.b.3 payload size-cap guard for /api/agent/telemetry
Defense-in-depth: reject oversized request bodies before buffering them in memory + deserialising. Closes the documented gap from M5.b — the V2 endpoint previously accepted unbounded bodies while legacy /api/agent/ingest has capped NDJSON payloads since day one. - Reuse existing TenantConfiguration.MaxNdjsonPayloadSizeMB (table-only, admin-set, default 5 MB). The property name is historical; scope is shape-agnostic — both NDJSON and JSON-array paths now share the knob. Doc comment updated to reflect that. No new config surface. - New internal static helper IngestTelemetryFunction.ReadBodyWithSizeCapAsync reads the already-gzip-decompressed request stream with a strict byte cap. Short-circuits + returns (exceeded=true, body="") the instant the cap would be exceeded, so a malicious sender can't force a 100 MB allocation. Semantics match legacy NdjsonParser: strict greater-than, equal-to accepted. - Cap exceeded → HTTP 413 Payload Too Large with IngestEventsResponse body (Success=false, Message describes limit). Agent's UploadResult parser classifies 413 as Permanent (not retried), which is correct — the same batch will fail again. The agent handles this by dropping the batch from the spool after the permanent-failure log. 4 new tests: under cap, exactly at cap, over cap (asserting the stream is not fully drained — memory bound), empty stream. 600/600 backend tests green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
d218a6ca25 |
feat(backend): M5.b.2 EventIngestProcessor — V2 endpoint event-pipeline parity
Extract the post-parse event-processing tail of IngestEventsFunction into a standalone EventIngestProcessor service so /api/agent/telemetry runs the same downstream logic as /api/agent/ingest. Deliberate copy-duplication, not a shared refactor: - Legacy /api/agent/ingest is production-hot — it serves live traffic from every deployed V1 agent and is NOT touched by this commit. Zero regression risk on the existing path. - EventIngestProcessor is a near-verbatim copy of the legacy pipeline's tail (~1000 LoC). Duplication is accepted because legacy /api/agent/ingest is scheduled for decommission post-v11 GA (see tasks/todo.md → Follow-Ups → Legacy-Agent-Ausmusterung). Until then: bugs fixed in legacy must be ported here manually. Copied pipeline steps (all 1:1 with legacy ProcessIngestAsync lines 150-566): StampServerFields → SanitizeEventTimestamps → StoreEventsBatchAsync → Index updates fire-and-forget → IME agent version history → ClassifyEvents → App install summaries → Geo-location update → UpdateSessionStatusAsync (all 7 status branches) → IncrementSessionEventCountAsync → Rule engine analysis fire-and-forget + SignalR + stats → Vulnerability correlation with WhiteGlove Part-2 merge → IncrementPlatformStatAsync TotalEvents + SuccessfulEnrollments → RecordGatherRuleStatsAsync → Diagnostics blob update → GetSession for updatedSession → Session age 4h warning → WhiteGlove IsPreProvisioned error log → SendWebhookNotificationsAsync (all 3 alert paths) → SLA breach evaluation → Admin override detection → ServerAction fetch-and-clear with TrackEvent telemetry → BuildSignalRMessages. Shared (not duplicated, to avoid drift in pure helpers): IngestEventsFunction.StampServerFields, SanitizeEventTimestamps, RecalculateAppDurations (all internal static, state-less, stable). The internal DTOs EventClassification + AppInstallAggregationState are also reused — they carry no behaviour, only fields. IngestTelemetryFunction wiring: - Return type HttpResponseData → IngestEventsOutput so the V2 endpoint can emit SignalR messages for the real-time UI push (matches legacy binding). - PersistItemsAsync partitions the batch: Events go through the processor (full pipeline + AdminAction + pending ServerActions + SignalR); Signal + Transition batches skip the processor and read control signals directly via a lean ReadControlSignalsAsync helper on ISessionRepository — no point firing the full pipeline for a batch that carries no events. - EventIngestProcessor registered in Program.cs DI. Known gap (explicitly NOT in this commit): the V2 endpoint lacks a payload size guard (legacy reads MaxNdjsonPayloadSizeMB from tenant config in ParseNdjsonRequest). Different payload shape (JSON array vs NDJSON), deferred to a dedicated follow-up. All 596 backend tests green (unchanged count — copy path has no new tests yet; the V2 endpoint's behaviour is validated by the legacy test coverage through the shared-helper surface + the TelemetryPayloadParser tests from M5.b.1). Integration coverage lands with M7 test-VM smoke. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
f0fc3696f6 |
feat(backend): M5.b IngestTelemetry endpoint — POST /api/agent/telemetry
V2 ingest endpoint consuming a heterogeneous batch of TelemetryItemDtos (Event + Signal + DecisionTransition in one JSON array). gzip request bodies are decoded transparently by the existing UseRequestDecompression middleware — new code only has to read a plain stream. - TelemetryItemDto (Shared): wire-format DTO mirroring the agent's TelemetryItem JSON shape so the backend can deserialise without linking agent assemblies. - IngestTelemetryFunction: security check (cert / rate / hardware whitelist) → device & version kill-switch short-circuits with DeviceBlocked=true body → body parse → tenant-header/body match → partition by Kind → persist via the appropriate repository → 200 with AdminAction + Actions control-signal fields populated from ISessionRepository (matching the /api/agent/ingest parity needed by the M4.6.ε agent-side UploadResult parser). - TelemetryPayloadParser: pure defensive parsing layer (wire → storage record). Malformed PayloadJson on a single item returns null; one poisonous item can't poison the batch. Projects index discriminators (IsTerminal, ClassifierVerdictId, ClassifierHypothesisLevel) off the DecisionTransition payload so the later queue-driven index fan-out reads them straight off the primary row. - EndpointAccessPolicyCatalog: POST agent/telemetry registered under DeviceOrBootstrapAuth (same policy as agent/ingest). Without this the fail-closed catalog rejects the route with 403. Scope boundary — Events get stored via StoreEventsBatchAsync only; the rule engine, app-install aggregation, SignalR notification, and vulnerability correlation remain on the legacy /api/agent/ingest path. Sharing that pipeline is tracked as M5.b.2 (extract EventIngestProcessor so both functions run the same downstream steps) — M7 rollout gates on completing it. Documented inline in PersistItemsAsync. 28 new tests: defensive null-handling per kind, typed-column extraction for Signal (with envelope fallback for missing SessionTraceOrdinal), DecisionTransition dead-end + classifier-verdict nesting, IsTerminal authoritative-list guard (Completed / Failed / WhiteGloveSealed / WhiteGloveCompletedPart2 — mirrors DecisionCore SessionStageExtensions), PartitionKey parser including malformed-shape rejection. The EndpointPolicyCatalogCompletenessTests guard the route registration. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |