Close the regression-coverage gaps flagged in the MCP review (H1, tests-1..7).
The auth core was correct but entirely untested; a regression that inverted a
fail-closed decision or moved next() into the error branch would silently open
the server with no failing test. All new tests run without a live backend or
token (stubbed fetch, hand-built JWTs, ephemeral express server).
- access-guard.test.ts: 401/403/429 decisions, fail-closed paths (fetch throws /
empty body / non-boolean allowed -> denied), allow-path caller context, cache,
F1 forged-token distinct-key. (H1, tests-1, tests-2)
- auth.test.ts: extractTokenClaims + isTokenExpired incl. 60s-skew boundary. (tests-3)
- auth.ts: comment documenting aud is parsed but enforced by the backend
(AuthenticationMiddleware.cs:200 ValidateAudience=true); no behavior change. (tests-4)
- oauth-register.test.ts: /oauth/register caps + cap-503, PKCE-S256 enforcement on
/oauth/authorize, code_verifier requirement on /oauth/token. (tests-5)
- client-routing.test.ts: pickGlobalOrTenantPath GA-gating + runWithCaller isolation. (tests-6)
- tools-admin/raw/sessions: describe.skipIf(no token) so integration suites skip
cleanly unattended instead of throwing in beforeAll. (tests-7)
npm test (no token): 100 passed / 0 failed / 39 skipped (was 3 suites failing).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Migrate all 29 tools, 3 prompts and 2 resources from the deprecated
.tool()/.prompt()/.resource() overloads to registerTool/registerPrompt/
registerResource, each with a human-readable title (Anthropic Directory
pass/fail criterion; drives host permission prompts and the tool picker).
title lives at the config-object top level; READ_ONLY/READ_ONLY_OPEN stay
pure ToolAnnotations, so shared.ts is unchanged.
Also:
- tool-design-2: soften the coercive "ALWAYS pass X" wording in
search_sessions to advisory tone (Directory prompt-injection flag).
- protocol-transport-2: add a body-parse error handler so malformed JSON
to /mcp returns a JSON-RPC -32700 envelope (-32600 for oversized) instead
of Express's default HTML error page; other routes get JSON 400/413.
usability-efficiency-2 was already addressed (search handlers surface
truncated + recallSummary); only the registration wrapper changed there.
Bump 1.3.0 -> 1.4.0. tsc strict green; 52/52 security-guards tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After the MCP search tools moved to the paginated /api/raw/events fan-out
(prev commit), /api/raw/events/search had zero callers anywhere in the repo
(no MCP tool, no web). Remove the now-dead surface end to end:
- Delete SearchEventsFunction (raw/events/search + global/raw/events/search).
- Drop both routes from EndpointAccessPolicyCatalog and the no-store cache
middleware allowlist (+ its test InlineData). EndpointPolicyCatalog
completeness test stays green (route and entry removed together).
- Remove the now-orphaned repo method SearchEventsByTypesAsync (its only
caller was the deleted endpoint) from ISessionRepository + TableSessionRepository
delegation + TableStorageService impl.
- Remove the non-paginated SearchSessionsByEventAsync: its only runtime caller
was SearchEventsByTypesAsync, and it is strictly redundant with the resumable
SearchSessionsByEventPageAsync (used by /api/raw/events + search/sessions-by-event).
Reword the surviving paged method's xmldoc/comment that referenced it.
No capability lost: multi-type cross-session event search still runs via the
MCP fan-out over the paginated single-type endpoint; single-type primitives
remain exposed via query_raw_events + search_sessions_by_event. Full solution
builds clean (0 errors); catalog-completeness + cache-middleware tests 77/77.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
deep_search_events / search_events_semantic advertised "maximum recall /
high confidence in completeness" but silently capped: the index path hit
/raw/events/search with fixed sessionLimit:10 + limit:200 and discarded the
backend result's truncation, while the envelope reported totalEventsScanned
(reads as "scanned all") rather than "stopped at the cap".
Replace the capped, cursor-less /raw/events/search call with a paginated
fan-out over the already-resumable sibling /api/raw/events: one nextLink-
following walk per candidate event type, bounded by a page + wall-clock
budget, merged client-side (single-type walks never duplicate events; only
sessions recur, unioned via Set). The truncated flag is now honest -- true
only when a walk stops at its budget (a real recall gap), never when it
drains. Per-type page errors preserve accumulated events and flag truncated
instead of silently collapsing to the 5-failed-sessions legacy path.
Envelope now carries truncated + recallNote ("narrow the query"); the
misleading totalEventsScanned/eventsScanned fields are renamed eventsFetched
in both tools. Descriptions drop the "maximum recall / exhaustive" claims and
state the topK ranking + bounded-scan reality. Tier-1 keeps a fast single-
page-per-type budget; Tier-3 walks deeper.
MCP-only change: no backend/shared/web code touched. /api/raw/events is
unchanged (already used by query_raw_events); /raw/events/search now has no
caller. Pagination/merge/budget logic extracted to exported fetchEventsViaIndex
with injectable fetcher+clock and covered by a deterministic vitest suite
(7 tests). tsc --noEmit clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a second, higher threshold to the runaway-session detector that
auto-blocks or auto-kills the device (admin-selectable mode), so the
existing 2000-event warn keeps observing while the harder action runs
unattended at e.g. 2500 events. Both tiers stay independently
idempotent per session and the maintenance batch picks them up in a
single query pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codex review (Low/Medium): even after the safe-state-first sunset sequence
landed in 4667033f, a narrow race remained — between a partial GC failure
(global row kept for retry, by design) and the next successful retry cycle,
a surviving per-tenant RuleState{Enabled=true} would resurrect the rule via
the override in AnalyzeRuleService.GetAllRulesForTenantAsync line 142
(rule.Enabled = state.Enabled overwrites the safe-state Enabled=false).
For the ANALYZE-CORR-001 / ANALYZE-APP-009 case the impact is small (only
two known orphan RuleState rows on a single partition, GC is essentially
always one-shot), but as a generic mechanism the window can be material
for future sunsets with many opt-in tenants where cross-partition GC is
more failure-prone (rate limits, transients).
Close the window with a runtime filter in GetAllRulesForTenantAsync:
built-in / community rules whose code definition is no longer in
BuiltInAnalyzeRules.GetAll() are excluded from the merged result regardless
of any surviving tenant override. The global DB row is preserved on
purpose so the next seed cycle still sees the rule in the sunset diff
and retries the GC + tombstone. Semantic becomes immediate instead of
eventual: "rule.json removed from source + deployed" = "rule cannot fire
this request".
Implementation:
- LiveCatalogIds: lazy-cached HashSet<string> on the service instance.
BuiltInAnalyzeRules.GetAll() re-deserialises the embedded JSON on every
call, so caching the projected set materially helps the hot read path.
Source of truth is the embedded resource (immutable per deployed
binary), so the cache never needs invalidation.
- Filter: `(rule.IsBuiltIn || rule.IsCommunity) && !liveCatalog.Contains(...)`
Custom tenant rules (IsBuiltIn=false, IsCommunity=false) are
deliberately exempt — they live entirely on the tenant partition and
aren't part of the built-in catalog.
Tests (+2):
- runtime_filter_hides_sunset_rule_even_with_enabled_tenant_override:
pins the exact scenario Codex flagged. GC fails (rule stays in DB),
tenant has RuleState{Enabled=true}, but the merged result still
excludes the sunset rule.
- runtime_filter_does_not_hide_custom_tenant_rules: defense — custom
rules whose RowKey doesn't match any built-in must survive.
Backend 2084/2084.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When an ExcessiveSessionEvents (or other session-bound) ops alert fires,
the operator only has a sessionId in hand. Looking up tenant + serial to
fill the manual Block form was a friction point during incident response.
- Backend: POST /api/devices/block now accepts optional blockedSessionId
and forwards it to the existing BlockedDeviceService session-aware path;
audit log + warn log carry the session id.
- Web (Device Block form): new "Session ID" shortcut input + Resolve
button fills Tenant + Serial via GET /api/sessions/{id} (GA cross-tenant
lookup already supported); resolved-device confirmation pill; URL
deep-linking via ?sessionId=, ?action=, ?reason= for auto-fill on mount;
blockedSessionId only forwarded when the resolved device still matches
Serial+Tenant so manual edits don't bind to a stale session.
- Web (Ops Events modal): when the event details JSON carries a sessionId,
surface "Block this device" / "Kill this device" deep-link buttons with
an auto-generated reason scoped to the event type.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codex review (Medium): even with the GC-first ordering from b855b2b9, a
race remained — after the orphan-GC deleted tenant RuleState rows but
BEFORE the global tombstone delete completed, a transient Azure failure
on the tombstone would leave the global rule alive while tenant overrides
were already gone. For a future default-ENABLED sunset rule this would
silently flip RuleState{Enabled=false} (explicit tenant opt-out) into
"now using the global default (enabled)" — exactly the regression class
the orphan-GC was meant to prevent.
Fix: a strict three-step sequence with mutual dependencies. Extracted
into a single ProcessSunsetRuleAsync helper so EnsureBuiltInRulesSeededAsync,
ReseedBuiltInRulesAsync, and ReseedFromGitHubFunction.ReseedAnalyzeAsync
all use the same primitive:
1. Safe-state: write the global rule with Enabled=false +
MarkSessionAsFailedDefault=false. Neutralises the default BEFORE
touching tenant state. If a default-enabled sunset rule's
RuleState{Enabled=false} (opt-out) survives every later step
failing, the override now matches the global default — same
effective result, no silent flip.
2. Orphan GC across tenants. Only attempted if safe-state succeeded.
3. Tombstone (DeleteAnalyzeRuleAsync global). Only attempted if GC
reported zero failures.
Returns a SunsetOutcome enum with three explicit failure values
(SkippedOnSafeStateFailure / SkippedOnGcFailure / SkippedOnGlobalDeleteFailure)
so callers and tests can pin which step bailed.
Additionally fixes the same-instance retry gap Codex called out:
EnsureBuiltInRulesSeededAsync only sets _seeded=true when EVERY sunset
rule reached Completed. Any partial outcome leaves the seed flag false
so the next GetAllRulesForTenantAsync call retries without waiting for
a process restart. The Function-App can recycle on its own schedule
(hours on a warm instance) and we don't want sunset cleanup to wait
that long.
Behaviour for the original CORR-001/APP-009 case: both shipped
Enabled=false globally and the tenant overrides were Enabled=true
(opt-IN). Safe-state is a no-op (already Enabled=false), GC removes
the opt-ins, tombstone drops the global row. Same end state as before,
just retry-safe across transient Azure errors.
Tests (10 new sunset cases on top of the previous 6):
- Full-path happy case asserts safe-state PRECEDES GC PRECEDES
tombstone via a shared CallOrder log.
- Safe-state failure path: NO GC + NO tombstone (tenant state preserved).
- Per-step failure paths for SkippedOnGcFailure / SkippedOnGlobalDeleteFailure.
- _seeded flag stays false on partial sunset, flips to true only on
a clean cycle; the "next call retries" scenario is exercised by
reconfiguring the mock between two GetAllRulesForTenantAsync calls.
Backend 2082/2082.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codex review of the sunset orphan-GC introduced in dff5ba65 surfaced three
issues; all three are valid and now fixed:
(Medium) The previous sequence was "delete global rule first, then GC
per-tenant RuleStates". If the GC failed partially (per-row error) or
entirely (cross-partition enumeration error), the global rule was already
gone — the sunset diff went empty on the next seed cycle and the surviving
orphan RuleStates would linger indefinitely. Worst case for a future
default-ENABLED sunset rule: tenants that had explicitly disabled it would
lose the disable, silently re-enabling something they didn't want.
Three coordinated fixes:
1. Storage methods made idempotent: DeleteAnalyzeRuleAsync and
DeleteRuleStateAsync now treat Azure 404 (row already gone) as success
instead of returning false. The 404-on-individual-row case inside the
cross-tenant GC enumeration is also counted as success — a row that
was raced-deleted between the query and our delete is a clean state,
not a failure.
2. Cross-tenant GC return shape changed from `int` to `(int deleted,
int failed)`:
- `deleted` counts fresh deletes AND 404s (idempotent retries)
- `failed > 0` signals at least one per-row delete threw a non-404
- `failed == -1` signals the cross-partition enumeration itself
failed — we genuinely don't know how many orphans are still out
there
Callers must check `failed != 0` before proceeding to the global
delete. The integer-only return shape couldn't distinguish "GC found
nothing" from "GC failed silently".
3. Caller sequence inverted to GC-FIRST in all three sunset code paths:
- AnalyzeRuleService.EnsureBuiltInRulesSeededAsync (on-startup seed)
- AnalyzeRuleService.ReseedBuiltInRulesAsync (explicit reseed)
- ReseedFromGitHubFunction.ReseedAnalyzeAsync (admin UI button)
GC is attempted first. The global rule is deleted ONLY when GC returns
`failed == 0`. On partial GC failure the rule stays in the global
catalog so the next seed/reseed cycle sees it in the sunset diff
again and re-attempts the cleanup. End state is eventually consistent
even across transient Azure errors.
(Low) The GitHub-reseed HTTP response shape was extended from
`{ deleted, written }` to `{ deleted, written, orphanStatesGcd,
sunsetSkipped }` so operators can see the cleanup count + any rules
that got skipped for the retry loop.
Tests:
- 3 new sunset-mechanism tests covering partial GC failure, enumeration
failure (-1 sentinel), and the still-living-rule no-touch invariant.
- Existing ReseedBuiltIn_returns_orphanGc_count_in_tuple updated to the
new helper shape.
- ReseedBuiltIn_skips_global_delete_on_partial_GC_failure pins the
retry-safe semantics in the explicit reseed path.
Coverage: backend 2076/2076 (+3 sunset edge-case tests).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Session 080edee9 follow-up — the customer screenshots showed ANALYZE-CORR-001
("Slow Network") and ANALYZE-APP-009 ("App Download Stalled") still firing
on a clean re-analyze, despite my previous fix shipping them as
enabled:false in the JSON. Investigation via direct DB query:
- AnalyzeRules/global rows for both: Enabled=false ✓ (version-bump landed)
- RuleStates/{tenant} rows for both: Enabled=true ✗ (orphan overrides
from the user toggling them in the portal UI on 2026-03-09)
[AnalyzeRuleService.cs:60](src/Backend/AutopilotMonitor.Functions/Services/AnalyzeRuleService.cs#L60)
overlays per-tenant RuleState on top of the global Enabled, so the orphans
kept resurrecting the rules. No reseed path was cleaning them up.
The right semantic is "sunset = remove from catalog", not "ship as
enabled:false and hope nobody overrides". Three coordinated changes:
1. Both sunset rules deleted from the rules/analyze/ source tree (they
never had a working condition shape — CORR-001 didn't actually check
download speeds, APP-009 matched the V2 termination handler's
post-mortem-synthetic download_progress:failed event).
2. New IRuleRepository.DeleteRuleStatesForRuleIdAcrossTenantsAsync
cross-partition cleanup primitive — scans RuleStates by RowKey, deletes
matching rows across every tenant partition.
3. Both reseed paths now do orphan-state GC:
- AnalyzeRuleService.EnsureBuiltInRulesSeededAsync (on-startup):
detects rules that exist in DB as built-in but aren't in
BuiltInAnalyzeRules.GetAll() anymore → deletes the global row +
fans out the per-tenant RuleState GC.
- AnalyzeRuleService.ReseedBuiltInRulesAsync (delete+reimport): same
diff-based detection on the pre-delete snapshot.
- ReseedFromGitHubFunction.ReseedAnalyzeAsync (admin "Reseed from
GitHub" button — the path the user actually clicks): mirrored
identical logic so every reseed flavor leaves the same DB invariant.
4. ReseedBuiltInRulesAsync return shape extended to
(deleted, written, orphanStatesGcd) so the admin UI can surface the
cleanup count to operators.
Coverage:
- RuleEngineDetectionFailureTests: the two version-bump assertions for
CORR-001/APP-009 are replaced with `DoesNotContain` against the
built-in catalog (sunset semantic).
- AnalyzeRuleServiceSunsetTests (new, 3 tests): pins EnsureSeed orphan-GC
invariant, asserts live-rule states are NOT touched, and pins the
ReseedBuiltIn (deleted, written, orphanStatesGcd) tuple shape.
After deploy + a single GetAllRulesForTenantAsync call (any analyze run
triggers it), the orphan RuleStates for sunset rules are automatically
cleaned. The user's tenant 5ca2b350-… will see CORR-001 + APP-009
disappear from the Rules UI on the next analyze cycle.
Backend 2073/2073.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Session 080edee9 follow-up — the analyze cards the user saw on re-analysis
were neither accurate (Slow Network / App Download Stalled both fired as
false positives) nor specific (Enrollment Failed never named the offending
app; ANALYZE-APP-013 never fired). Five coordinated changes, all
version-bumped so EnsureBuiltInRulesSeededAsync re-imports them on the next
backend boot — the title/severity/version-check filter does not pick up
condition-only diffs:
ANALYZE-APP-013 v1.1.0 — fire on legacy + new agent shape:
Previous rule keyed on `failureType == esp_apps_detection_failure`,
which is only emitted by agents that shipped AFTER 4dd050e0. Historical
sessions (and any agent on the old build) won't match.
Replace with a precondition + condition pair that works for BOTH:
* precondition: esp_failure_settle_started.errorCode == 0x87d1041c
(V2 ProvisioningStatusTracker emits this on every Apps subcategory
failure with that HRESULT, regardless of agent version)
* required: app_install_failed.detectionResult == NotDetected
(ImeLogTracker sets this from the IME's DetectionActionHandler line
across all V2 agents)
* optional: esp_failure_settle_started.errorCode (for {{errorCode}}
interpolation; appName comes via auto-field from the required match)
ANALYZE-ENRL-001 v1.1.0 — restore appName + cross-link to APP-013:
My previous trim dropped the failed_app condition on the theory that the
auto-field path would carry it. That was wrong — auto-fields come from
the FIRST matched condition's evidence, which here is `enrollment_failed`
(no appName). Add back the optional failed_app condition with
dataField=appName so byField resolves {{appName}}. Explanation now leads
with the failed app, lists reason/subcategory/errorCode, and cross-links
to the actionable ANALYZE-APP-013 instead of "see other rule results".
ANALYZE-CORR-001 v1.1.0 ("Slow Network") — version bump for DB sync:
Was already enabled:false in source but the DB still had it enabled (the
seed mechanism only resyncs on title/severity/trigger/version diff, so
the source `enabled` flip never propagated). Bump version → next deploy
resyncs. Also adds a suppressByEvent on the app_failed condition so a
later app_install_completed for the same appId suppresses the match (a
retry that succeeded shouldn't count as failure).
ANALYZE-APP-009 v1.1.0 ("App Download Stalled") — same version bump for
DB sync + documents why a status-aware suppression isn't expressible yet:
The V2 EnrollmentTerminationHandler synthesizes
`download_progress: failed` events for promoted-stuck apps AFTER the
real `download_progress: 100%` event already landed. The engine's
suppressByEvent can't filter on a third event's status field, so
status-aware suppression has to wait for either an engine extension or
a dedicated `download_stalled` event type.
RuleEngineDetectionFailureTests — 6 new tests:
* APP-013 fires on the historical 080edee9 event shape
* APP-013 does NOT fire for non-Apps subcategory HRESULTs
* APP-013 does NOT fire without esp_failure_settle_started (precondition)
* ENRL-001 MatchedConditions carries appName + errorCode + subcategory
+ reason from three distinct event types (interpolator material)
* CORR-001 + APP-009 shipped as disabled
Coverage: 2064/2064 backend, 413/413 web.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codex review hygiene: the new
`AppStateChange_Error_promoted_carries_failureType_and_errorCode` Theory
passes null via [InlineData(EspAppsTimeout, null)] for the no-HRESULT
fallback case, but declared the parameter as non-nullable `string` (the
test project has <Nullable>enable</Nullable>). Switch to `string?` to
match the value flow and silence the xUnit1012 analyzer warning. Behaviour
unchanged — the test still asserts that the no-HRESULT branch omits the
DataJson.errorCode key entirely.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codex review (P3): the V2 EnrollmentTerminationHandler previously cached
only the HRESULT from the last registry-derived ESP failure. A non-Apps
ESP failure carrying its own HRESULT (e.g. DeviceSetup/SecurityPolicies
or AccountSetup/CertificatesAccountSetup with their own status codes)
would then mis-classify still-installing apps as
`esp_apps_install_failure` — the HRESULT describes the failed category,
not the in-flight installs.
Replace `LastEspTerminalErrorCode` (string) with a new
`EspTerminalFailureSnapshot` value carrying errorCode + failedSubcategory
+ category as a single atomic snapshot. Plumb through the host stack
(EspAndHelloTracker -> EspAndHelloHost -> DefaultComponentFactory ->
EnrollmentTerminationHandler) and gate the HRESULT-driven classifier on
`snapshot.IsAppsSubcategory == true`. Non-Apps failures fall through to
the generic `esp_apps_timeout` classification with no HRESULT in the
promoted app's DataJson.
Coverage: 2 new gating tests + the 2 existing HRESULT tests updated to
supply the full snapshot (errorCode + "Apps" subcategory).
1176/1176 V2.Core, 2058/2058 backend.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codex review (P2): ANALYZE-APP-013's `failed_app_name` + `detection_hresult`
optional conditions scanned all app_install_failed events independently of
the required condition. In a session with multiple app failures the
template could interpolate {{appName}} / {{errorCode}} from a different
event than the one that matched `failureType == esp_apps_detection_failure`.
Two coordinated changes:
1. `interpolateRuleTemplate` gains an "auto-field" resolution tier between
the explicit byField path and the bySignal fallback. The backend's
`RuleEngine.AddDataFieldsToEvidence` already injects a whitelist of
identifying fields (appId, appName, errorPatternId, errorCode, exitCode,
status) onto every matched-condition evidence dict — the interpolator
now reads those, pinning to the FIRST matched condition (= the required
condition) so a rule with a single required match resolves all four
tokens against the same source event.
2. ANALYZE-APP-013 is trimmed to a single required condition; the redundant
optional conditions for appName / errorCode are removed because the
auto-field tier already exposes them from the required match. Same
pattern applied to ANALYZE-ENRL-001's `failed_app` condition.
Coverage: 4 new vitest cases (multi-failure correct interpolation, first-
condition-wins shadowing, explicit-field precedence, internal-field leakage
guard). 413/413 web tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codex review (P1): ImeLogTrackerAdapter.BuildAppStatePayload only set
DataJson.failureType for `esp_apps_timeout`. After the HRESULT-aware
classifier landed (4dd050e0), apps promoted with `esp_apps_detection_failure`
or `esp_apps_install_failure` still landed with errorPatternId set but
failureType missing. Downstream consequences:
- ANALYZE-APP-013 (dataField: failureType, value: esp_apps_detection_failure)
never matched.
- EventIngestProcessor.AppInstall set AppInstallSummary.FailureCode to the
HRESULT instead of the canonical failureType.
- InstallProgress.tsx fell back to the generic "Failed" badge.
Extend the gate to cover all three AppFailureTypes constants and stamp
DataJson.failureType from app.ErrorPatternId. Adds a Theory test covering
all three flavours and a regression assertion that an unrelated IME-ERROR-*
pattern does NOT spuriously promote to failureType.
Coverage: 108/108 ImeLogTrackerAdapter tests, 1176/1176 V2.Core.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Session 080edee9 fired only ANALYZE-ENRL-001 ("Enrollment Failed") with a
generic explanation, even though we already had all the data to point at
the specific failed app (Microsoft Apps for Enterprise) and the specific
HRESULT (0x87D1041C - Intune detection rule mismatch). Three coordinated
rule changes close that enrichment gap:
ANALYZE-APP-001 (Win32 App Detection Script Failure):
Switch from a regex on the `message` field (which never matched the
agent's "Install status unconfirmed - ESP timed out" wording) to a
direct check on `dataField: detectionResult, operator: equals,
value: NotDetected`. The rule engine already supports nested DataJson
fields on `event_type` conditions, so this is the right shape and the
rule now fires whenever the IME reported a detection-rule mismatch.
ANALYZE-APP-013 (App Detection Failure During ESP - 0x87D1041C, new):
Matches the canonical `failureType: esp_apps_detection_failure` tag
emitted by the V2 agent's new classifier. Severity=critical, baseConf=90.
Pulls failed appName + HRESULT into the explanation via the existing
`{{token}}` interpolator, and ships a three-step remediation plan:
(1) validate the detection rule against an installed reference device,
(2) investigate post-install async tasks (the install may complete only
after ESP's detection probe runs), (3) cross-check the per-app event's
detectionResult + bytesDownloaded + hresultFromWin32.
ANALYZE-ENRL-001 (Enrollment Failed):
Add optional conditions that capture {reason, failedSubcategory, errorCode,
failedApp.appName} into MatchedConditions so the explanation can resolve
`{{reason}}` / `{{failedSubcategory}}` / `{{errorCode}}` / `{{appName}}`
via the existing `interpolateRuleTemplate` engine. Remediation text now
cross-references ANALYZE-APP-* rules for the actionable root cause.
dist regenerated via `node rules/scripts/combine.js` (25 -> 26 rules).
Backend embeds dist/analyze-rules.json as a resource; next deploy picks up
the new content automatically.
Coverage: 2058/2058 backend tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The InstallProgress component previously showed an orange "Likely stuck"
badge for any app where the agent promoted state Installing -> Error on
terminal ESP-Apps failure, with a tooltip claiming "ESP timed out while
this app was still installing". Session 080edee9 had that badge attached
to Microsoft Apps for Enterprise even though the actual session only ran
27 min (not 180), and the failure was a confirmed Intune detection-rule
mismatch (HRESULT 0x87D1041C), not a timeout.
Match the new agent failureType taxonomy from
`AppFailureTypes.ClassifyEspAppsFailure`:
- `esp_apps_detection_failure` -> red "Detection failed" badge
(tooltip: Intune detection rule did not match after install)
- `esp_apps_install_failure` -> red "Install failed" badge
(tooltip: installer returned an error before completing)
- `esp_apps_timeout` -> orange "Likely stuck" badge unchanged
(tooltip clarified to "no per-app HRESULT was available")
Also surface the ESP-level HRESULT (DataJson.errorCode) as a "ESP HRESULT:
0x87d1041c" pill with the description from the shared error-codes catalog,
distinct from the per-app hresultFromWin32 pill - so the user can see at
a glance which cross-app failure caused the ESP termination.
Coverage: 409/409 vitest, tsc clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Session 080edee9 surfaced a long-standing mis-classification on the V2
EnrollmentTerminationHandler "still installing -> error" promotion path: every
terminal ESP-Apps failure was stamped as `esp_apps_timeout` with a message
claiming "ESP timed out (180 min)" - even when the actual session only ran
~27 min and the ESP terminated explicitly with HRESULT 0x87D1041C
("Application not detected after installation completed successfully"). The
"180 min" was the *configured* SyncFailureTimeoutMinutes ceiling, not the
elapsed time.
Plumb the HRESULT captured by ProvisioningStatusTracker through the host
stack (EspAndHelloTracker.LastEspTerminalErrorCode -> EspAndHelloHost ->
DefaultComponentFactory) into the termination handler, and route through a
new `AppFailureTypes.ClassifyEspAppsFailure` helper that produces:
- `esp_apps_detection_failure` for 0x87D1041C (install ran but Intune
detection rule did not match afterwards - confirmed detection failure)
- `esp_apps_install_failure` for any other HRESULT (installer reported
an error before the app finished installing)
- `esp_apps_timeout` fallback when no HRESULT is available
(genuine "ESP gave up" case - now phrased as "configured at X min",
not "timed out X min", to make clear it's a ceiling not an elapsed time)
The promoted `app_install_failed` event now carries DataJson.errorCode so
downstream rules and the UI can correlate the per-app failure to the
ESP-level HRESULT. AppPackageState.SetErrorContext already accepted an
optional errorCode argument, so this only extends the promotion delegate
signature without changing the per-app state shape.
Coverage: 86/86 Termination tests pass (incl. 2 new HRESULT-classification
cases); 6/6 new AppFailureTypesClassifierTests; 1171/1171 V2.Core.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds getEnrichedOrLookup() helper that returns the backend-provided
{description, confidence, source} sibling when present (set by the
backend ErrorCodeEnricher on event responses), otherwise looks up via
the local catalog. EventTimeline event badges now read the enriched
field first — older API responses without enrichment fall through to
the existing local-catalog path so there is no regression.
InstallProgress + apps/[appName] pages consume UI-model rows (not raw
event.Data) and are not touched — they continue to use the local
catalog which already covers their inputs. 5 new helper tests; 409/409
web tests green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Walks each EnrollmentEvent.Data for known error-code keys (errorCode,
exitCode, hresult, hresultFromWin32, failureCode, code, lastError) and
injects a sibling *Info entry { description, confidence, source } via
the shared ErrorCodeCatalog. Applied in GetSessionEvents (legacy +
paginated), SearchEvents (used by search_events_semantic +
deep_search_events MCP tools) and QueryRawEvents (both session-scoped
and cross-session paths).
Idempotent + read-only on storage — alte gespeicherte Events erhalten
die Anreicherung on-the-fly. MCP-Konsumenten (Claude/Copilot) bekommen
ab jetzt automatisch die kuratierten Descriptions mit Confidence-Level,
nicht nur den rohen HRESULT. 14 enricher tests; 2058/2058 backend tests
green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When the V2 agent process dies via AppDomain.UnhandledException or
TaskScheduler.UnobservedTaskException, synchronously write a compact
MiniDump (data-segs + handle + thread-info, ~5-20 MB) and a CrashRecord
JSON sibling to %ProgramData%\AutopilotMonitor\Crashes\. Next agent
start scans the directory, drains pending records into the timeline as
previous_crash_detected events (Warning, ImmediateUpload), and applies
retention (max 5 dumps, max 7 days). Dumps stay on disk so the existing
customer-opt-in diagnostics-upload can collect them.
Handlers register in Program.Main before SelfUpdater so any downstream
crash is captured; AgentRuntimeHost re-registers once session/tenant
are known so the CrashRecord carries those labels. 6 builder/retention
tests; 1163/1163 V2.Core tests green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lifts the Windows/MSI/Intune HRESULT catalog out of the web app into
src/Shared/AutopilotMonitor.Shared/Resources/error-codes.json so backend
and web load from the same authoritative file. Web syncs the JSON at
prebuild time. Adds ErrorCodeCatalog C# service with normalised-key
lookup (hex case, signed-decimal HRESULT, leading-zero padding) and a
parity test on both sides. No behavior change yet — this is groundwork
for backend read-time enrichment so MCP/API responses can carry
errorCodeInfo (description + confidence + source). 11 C# + 14 web tests
green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a fail-soft GetSystemPowerStatus probe to StartupEnvironmentProbes,
emitting a power_state_check event into the enrollment timeline at agent
start. On battery with <80% charge is Warning, otherwise Info; desktops
(no battery) and probe failures handled explicitly. Gives admins early
visibility into a frequent driver behind power-management-related
enrollment stalls. 5 new builder tests; 1157/1157 V2.Core tests green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Normal (non-Global-Admin) MCP users no longer see any Global-Admin / cross-tenant
hints — reducing both confusion and attack surface / curiosity. The per-request
stateless server already runs inside the access-guard's runWithCaller context, so
the caller role is known at server-creation time.
- index.ts: createMcpServer(ga) reads isGlobalAdmin() and tailors instructions
(non-GA gets a tenant-scoped scope line, no cross-tenant mention).
- tools.ts / sessions.ts / search.ts / admin.ts: thread `ga` into every registrar.
* GA-only tools (list_tenants, list_tables, query_table, query_backend_logs,
get_platform_metrics, get_ops_events, list_session_reports, list_blocked_devices,
get_api_usage) are registered only when ga — absent from tools/list for non-GA.
* Dual-role tools: GA/cross-tenant wording in descriptions and tenantId param
text is gated behind `ga ? ... : ...`; non-GA sees clean tenant-scoped text.
- prompts.ts: compare-agent-versions (uses GA-only get_platform_metrics) hidden for
non-GA; cve-exposure-audit cross-tenant wording scrubbed for non-GA.
- error-handler.ts: 403 message no longer names "Global Admin".
- backend McpUserFunction (/api/auth/mcp): omit isGlobalAdmin from the response when
false, so a normal user gets no hint that a GA tier exists. The MCP guard reads
`=== true` (absent → non-GA), so behavior is unchanged.
Verified: GA catalog 29 tools / non-GA 20 (9 GA-only hidden), zero GA/cross-tenant
wording in any non-GA description, prompts split 3/2. tsc clean; backend targeted
suite 135 green; mcp unit 52 green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
get_usage_metrics always called /api/global/metrics/usage (GlobalAdminOnly),
so a Tenant Admin got 403 even though a MemberRead tenant endpoint
(/api/metrics/usage) exists and returns the same shape scoped to their tenant.
Route via pickGlobalOrTenantPath: GA -> global cross-tenant overview;
Tenant Admin/Member -> /api/metrics/usage (JWT-scoped, tenantId ignored).
Description updated for both roles.
Verified live with a Tenant-Admin token: 200, own-tenant usage only
(tenants.total=1), platformStats correctly omitted (GA-only) - no cross-tenant
leak. tsc clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Second batch of pre-release MCP tool audit fixes:
- mcp: get_session_summary phase labels were desynced from the backend
EnrollmentPhase enum — phases 3-7 were mislabeled (e.g. 7 showed
"Pre-Provisioning" but is "Complete"; 4 showed "Device ESP" but is
"Account Setup"). Replaced with the canonical taxonomy (matches the web
phaseConstants.ts), enrollment-type-aware (phase 3 = Apps (Device) on V1,
App Installation on V2), plus -1=Unknown / 99=Failed.
- backend: get_usage_metrics reported PlatformStats.lastFullCompute /
lastUpdated as 0001-01-01 even though the persisted row was fresh -
UsageMetricsService projected a new PlatformStats and dropped the two
timestamp fields (defaulted to DateTime.MinValue). Now carried through.
- backend: query_table (TableQueryFunction) now returns 400 with the
actionable Azure message for an invalid OData filter / bad query input
(RequestFailedException 400) instead of a generic 500.
Tests: backend targeted suite 31 green; mcp tsc clean + unit suite 52 green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pre-release MCP tool audit fixes:
- backend: search_sessions_by_cve returned 0 in GA cross-tenant mode. The
CveIndex PartitionKey is {tenantId}_{cveId}, so the old cross-tenant PK
range anchored on the cveId matched nothing (PKs start with the tenant
GUID). Now filters on the CveId property (server-side scan, same shape as
ScanCveIndexAsync). Extracted BuildCveIndexSearchFilter + 5 regression tests.
- mcp: get_geographic_sessions gains pageSize/continuation (client-side
offset; the endpoint has no server cursor) to bound response size.
- mcp: get_usage_metrics / get_api_usage descriptions corrected to reflect
Global-Admin-only routing (were implying a tenant-scoped mode that 403s).
- mcp: get_session_summary appInstalls.total no longer smaller than its
breakdown (max(started, succeeded+failed+skipped)) + adds started.
- mcp: session status enum gains Pending + Stalled.
Tests: backend targeted suite 24 green; mcp tsc clean + unit suite 52 green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to opt-in pagination: fields= was a client-side trim in the MCP tool,
so it did not round-trip across pages — following nextLink returned the full
safe-field set again, and the shape differed page 1 vs 2+.
Move the projection server-side (matching query_raw_sessions):
- TenantConfigProjection: keep-list intersection — only the 9 non-sensitive
fields can ever be emitted; requested keys are intersected (fields=teamsWebhookUrl
yields nothing), tenantId always included. ParseFields is case-insensitive.
- TenantConfigPagination: parse fields=; BuildNextLink echoes it so the projection
carries across every page. fields is NOT in the continuation fingerprint (it is a
projection, not a filter — changing it mid-stream keeps the cursor valid).
- GetAllTenantConfigurationsFunction projects via TenantConfigProjection.ProjectAll.
- MCP list_tenants passes fields to the backend (followNextLink params) and drops the
client-side projectTenantFields; extractTenantList stays as defense-in-depth.
Now every page returns exactly the requested columns and the backend sends less over
the wire. Tests: TenantConfigProjection (incl. secret-cannot-be-requested) +
pagination fields-echo; backend 24/24, MCP vitest 52/52.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The MCP list_tenants tool fetched GET /api/config/all, which returns all
tenant configs in one unpaginated array. With ~198 tenants the projected
response exceeded the tool response cap and was offloaded to a file, making
the tool unusable for the model.
Add backward-compatible opt-in pagination to config/all (same dual-mode
pattern as GetSessionReportsFunction):
- No pageSize -> unchanged legacy bare full-config array (web + maintenance
aggregation depend on this shape; zero regression).
- pageSize -> secret-stripped { count, tenants, nextLink } envelope using
the existing ContinuationToken HMAC (scope "config-all", caller-bound),
TenantId-ordered pages. Secrets never leave the backend.
Backend: GetTenantConfigurationsPageAsync (repo + service), TenantConfigPagination
helper, dual-mode GetAllTenantConfigurationsFunction. Route/policy unchanged.
MCP: list_tenants now follows nextLink against config/all (extractTenantList as
defense-in-depth + optional fields= trim); client-side selectTenantPage removed.
Tests: TenantConfigPaginationTests 13/13; MCP security-guards 56/56.
No /api/global/ relocation here - that convention cleanup is deferred.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
MCP server review follow-ups (PR A-D):
- Remove stale agents/query.ts dev script + drop unused @anthropic-ai/sdk dep
- Add 256kb body limit on /mcp; server-level instructions + 3 diagnostic prompts
(investigate-failed-session, cve-exposure-audit, compare-agent-versions)
- New list_tenants tool over /api/config/all with a strict keep-list projection
so tenant secrets (webhook/SAS URLs) never reach the model (+ 6 unit tests)
- New get_vulnerability_summary tool + backend GET /api/{global/}metrics/vulnerability
aggregating the CveIndex (top CVEs, severity breakdown, KEV, affected devices;
truncated flag on scan cap) + 6 aggregator unit tests
- Test hygiene: exclude tests from tsc output, pin vitest to src/ (no dist double-run)
- Bump MCP server version 1.2.0 -> 1.3.0
Backend solution builds clean; MCP 52 unit tests + 6 backend aggregator tests green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Section headers (Vulnerability Report, Analysis Results, Integrity Bypass)
and the page action-bar packed icon + title + counter + severity badges +
action button + chevron into a single non-wrapping flex row. On phone-width
viewports that row was wider than the card, widening <main> past the viewport
and making <body> horizontally scrollable. iOS Safari then mis-sized the
fixed-position mobile drawer/overlay, so the drawer no longer covered the
full screen.
- globals.css: overflow-x: clip on <html> as a safety net (clip, not hidden,
so it does not create a containing block for the fixed drawer/overlay).
- The three section headers + page header now wrap (flex-wrap + gap), with
shrink-0 on icons/action groups and break-all on the mono finding string.
tsc clean; vitest 390/390 pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Nightly VulnerabilityDataSync at 03:00 UTC was hitting Azure Table Storage's
1 MB per-entity ceiling on popular CPEs (browsers, Chromium-mass-release CVEs,
Windows components with hundreds of CPE matches). Property-chunking only
guards the 64 KB per-property bound, not the row total.
New CveCacheBudget filters before the cache write so a row can never grow past
the limit:
- date floor: last 24 months (all severities) + last 60 months for CVSS >= 7.0
(keeps current HIGH/CRITICAL findings for LTS software like Java 8 / .NET FX)
- sort CVSS desc, PublishedDate desc, cap 100 CVEs
- per-CVE: cap AffectedVersions at 50 ranges
- unparseable PublishedDate -> keep (conservative)
Both write paths (VulnerabilityCorrelationService.FetchAndCacheCvesFromNvdAsync
and VulnerabilityDataSyncFunction.RefreshStaleCveCacheAsync) go through the same
helper; the sync path also drops the blind .Take(200) and now logs total/
date-filtered/truncated-ranges/bytes per CPE so we can watch the budget land.
Read path is unchanged — older oversized rows continue to work via the existing
TableStorageChunking reassembly until they expire and get refreshed under the
new budget.
Tests: 1990/1990 green; 10 new CveCacheBudgetTests cover the date floor,
LTS-rescue window, sort+cap, AffectedVersions truncation, and edge cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The seed-at-arming fix (dcf89a83) only patched AttachUserRealmJoinSubtreeWatcher's
internal sweep. Two callers still ran a STANDALONE CheckUserPackages BEFORE the seed
pass: NotifyRealmJoinPresence (when phase crossed RunningThreshold with the SID
already set) and ArmHku (when desktop arrival landed after phase had already crossed
the threshold). The HKU dedup sets were empty at that point, so every pre-existing
user-hive package sub-key — typically RJ packages installed during ESP while
DeploymentPhase was still Blank — surfaced as a spurious started+completed pair
before the seed had any chance to suppress them.
Caught in session ff0cdcbe (V2 2.0.886): three user-scope RJ packages
(generic-agilebits-1password, generic-realmjoin-promote-tray-icon,
generic-vlc-usersettings) all fired started+completed at the same millisecond as
the Blank -> RunningFirstDeployment phase transition, exactly the
empty-dedup-set sweep signature.
Removed both standalone calls — EnsureUserRealmJoinStage ->
AttachUserRealmJoinSubtreeWatcher is now the single, ordered entry point
(seed -> watcher.Start -> CheckUserPackages, all post-seed). Added two regression
tests driving the production sequence with a fake registry that holds pre-existing
user packages; both assert zero RealmJoinPackageStarted/Completed signals. The
fake-registry wiring lands as an internal ctor with three delegate seams
(EnumeratePackageIds/TryReadPackage/KeyExists); the public ctor still binds to
RealmJoinInfo + the real Registry probe.
1152/1152 V2.Core tests green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The script applies a lifecycle policy + container creation against a live
storage account using operator-specific parameters (StorageAccountName,
ResourceGroupName). It is not portable across environments and each
operator keeps their own copy, so it does not belong in the repo.
The file is preserved on disk (git rm --cached, not -f) so the local
operator setup keeps working — only the Git tracking is dropped, plus
the matching .gitignore entry so a future `git add scripts/Backup` is
silently ignored.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pre-existing package sub-keys observed at arming time (typically RJ packages
installed during ESP before DeploymentPhase >= 100) were silently fired as
realmjoin_package_started by the initial CheckMachinePackages sweep. The
phase gate only prevented IME/RJ subtree-write misattribution; it did not
distinguish historical RJ rows from rows that genuinely appear after arming.
Attach now seeds both _*PackagesStarted and _*PackagesCompleted with every
sub-key already present under HKLM\SOFTWARE\RealmJoin\Packages (and the
HKU\<sid>\... mirror), without firing events. The seed runs inside the
state-lock BEFORE RegistryWatcher.Start() so no subtree wake-up can race
past it. Only sub-keys that appear AFTER arming surface as events.
Added 3 unit tests covering the ESP-leftover suppression, the
per-PackageId scope of the suppression (siblings still fire), and the
HKU mirror. Outdated comment on the immediate-arm test corrected.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two findings from the post-commit Codex review of PR2 (single-row restore):
Medium — CriticalTableRestoreService.CommitRowAsync now maps the
MaintenanceLeaseHolder's renewal-loss path to BackupTerminalException
"MaintenanceLeaseLost" (HTTP 409 Conflict), instead of letting the
storage SDK's OperationCanceledException bubble up to the HTTP layer's
generic catch as a 500 InternalError. The catch condition gates on
!ct.IsCancellationRequested so a real client cancel still re-throws.
Low — RestoreInputRow (admin/backups/[backupId]/page.tsx) used to
disable the Preview button on empty pk/rk. The backend contract
(RestoreTablePreflightValidator) explicitly allows empty-string keys
(Azure Tables permits them) and only rejects null. UI now mirrors that.
Added a regression test pinning the empty-keys-allowed contract.
1980/1980 backend tests + 390/390 web tests green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
POST /api/global/backups/{backupId}/restore-row (GA-only) with preview|commit
modes. Preview scans the ETag-pinned NDJSON blob for the requested (pk, rk),
diffs against the live row, and returns rowSha256 + currentETag for echo.
Commit re-validates everything under a short maintenance lease and writes via
UpdateEntity(IfMatch=etag, Replace) or AddEntity; 412/404/409 → CurrentRowChanged.
Backend:
- RestoreTablePreflightValidator (no I/O) + BackupRestoreInputValidator (manifest
parse, entry check, BlobName canonical, ETag-pin + SHA in one stream read)
- CriticalTableRestoreService with PreviewRowAsync (no lease) + CommitRowAsync
(MaintenanceLeaseHolder, 45 s renewal, deterministic finally release)
- BlobBackupStore.OpenNdjsonReadAsync — GetProperties + OpenRead(IfMatch=etag)
- BackupTerminalException carries a stable Code for HTTP envelope mapping
- OpsEventService.RecordBackupRowRestoredAsync (Warning; payload is
container + blobName only, never a SAS URL)
- 27 new xUnit tests; full backend suite 1979/1979 green
Web:
- /admin/backups list page with newest-first list + "Run backup now" trigger
and async job-status poller
- /admin/backups/{backupId} detail page with per-table breakdown + single-row
restore trigger
- RestoreRowDiffModal with Added/Removed/Changed/Unchanged pills and
auth-table security banner (GlobalAdmins/TenantAdmins/McpUsers)
- api.backups namespace + matching DTO types
- globalNavConfig: ga-backups under Ops
- BackupRowRestored registered in OPS_EVENT_TYPES (dual-register policy)
- 6 new vitest cases; full web suite 390/390 green
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codex-review finding: after the previous hotfix the retryable Running→Queued
rollback ran AFTER leaseHolder.DisposeAsync(), i.e. AFTER the maintenance lease
was released. That opened a window of "Running + free lease + stale heartbeat"
in which the watchdog could legitimately flip the job to Failed; the rollback
then re-read the fresh row, took the latest ETag, and unconditionally re-set
Queued — silently undoing the watchdog's decision.
Fix:
* Move the retryable rollback INSIDE the leased section, right next to the
success-CAS that produces the Completed terminal state. Both paths now run
while the lease is still held + being renewed, so the watchdog cannot race.
* Defence-in-depth: re-read the row before writing Queued and only do the
state transition if the row is still Running. If a parallel actor (in
theory: another worker that somehow stole the lease, or the watchdog
acting on a transient probe) shifted the state, the rollback skips +
logs instead of clobbering. Belt-and-braces — under a held lease the
predicate should always be true.
* Outer-block retry-handling shrinks to "log + rethrow"; the throw is what
drives the visibility-timeout retry of the queue message.
Tests: 1952/1952 still green. The race fix is structural (ordering relative
to lease release), not test-locked; the existing in-memory fakes don't model
lease semantics so a regression would have to be caught in a future watchdog
race-condition test.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two Codex-review findings on the post-Wave1 hotfixes, both real:
* Medium — lease-loss cancellation was only caught around RunBackupUnderLeaseAsync.
The same handlerCts token is also used by the CAS Queued→Running, the BackupId
stamp, and the final CAS Completed. If renewal failed during one of those
awaits, the OperationCanceledException bypassed the lease-loss branch and
escaped as a generic poll-loop exception, leaving the job in Running until
the watchdog mopped up. Fix: hoist the lease-loss catch around the WHOLE
leased section; the inner try keeps the retryable-vs-OCE classification for
the service call.
* Medium/Low — the manual queue path marked lease-loss jobs Failed but did not
emit CriticalTableBackupFailed. The timer path did, so alert rules fired
there but not here. Fix: emit the same OpsEvent (with the recovered backupId
+ RequestedBy from the job row) so Telegram routing is symmetric.
While restructuring, two small clean-ups:
* runningJob.RequestedBy and runningJob.StartedAtUtc are captured into outer-
scope variables so the success-path OpsEvent + retry-rollback work even when
the inner try exited via the lease-loss path.
* leaseHolder.DisposeAsync now wraps the whole leased section in a single
finally — the previous version had two finally blocks (one for the inner
service-call try, one for the outer). Single dispose ⇒ no double-release race.
Tests: 1952/1952 still green. Lease-loss code paths are exercised at the unit
level via the existing renewal-failure logic on MaintenanceLeaseHolder + the
service throw classification on CriticalTableBackupService.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three Codex-review findings in PR1, all real:
* HIGH — Maintenance lease expires after 60 s but the per-run budget is 50 min.
Without renewal a parallel worker / timer / restore could enter the same
"exclusive" section after one minute, and the watchdog would see a free
lease while a job is still running. New MaintenanceLeaseHolder spins a
renewal task on a 45 s cadence and cancels the handler's CTS on any
renewal failure; the in-flight backup then bails via OperationCanceledException
and the worker/timer route the job to a clean Failed state without
queue-deleting the message.
* HIGH — Retry-rollback resets State=Queued + BackupId=null but left QueuedAtUtc
on the original wall-clock value. After a retryable service throw the
watchdog's 60 min "Queued without pickup" rule could fire mid retry-cycle
and call the job dead. Fix: stamp QueuedAtUtc=now on the rollback so the
watchdog respects the fresh retry budget.
* MEDIUM — ListBackups returned every top-level prefix including runs that
never reached manifest.json. Calling GetBackupManifest on such an id returned
404 — confusing UI signal for incomplete runs. Fix: ListBackupIdsAsync now
Exists()-checks manifest.json per prefix. New ListAllBackupPrefixesAsync
yields (id, hasManifest) tuples for a future admin UI that wants to
surface incomplete runs as their own status.
Tests: 1952/1952 backend xUnit still green — no regressions, no new test
coverage needed because the hotfix paths are exercised indirectly by the
existing happy/partial/budget tests + the production storage SDK semantics.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the full backup pipeline that snapshots 15 critical tables (Admin/Tenant
config, rules, identities, audit, preview) into a logically-isolated blob
container, daily at 04:00 UTC and on operator demand. Restore endpoints follow
in PR2/PR3.
Architecture:
* Daily TimerTrigger + manual POST /api/global/backups/trigger (GA-only,
fail-hard producer). HTTP returns 202 + jobId on enqueue, 500 if the queue
write fails (and the BackupJobs row is rolled to Failed).
* BackgroundService queue worker (not QueueTrigger — matches the Analyze/
IndexReconcile pattern that avoids the Functions-host connection-binding
app-setting). Worker holds the maintenance lease for the whole run; service
is intentionally lease-unaware so it stays unit-testable.
* BlobBackupStore wraps the new 'critical-table-backups' container with a
shared maintenance-lease sentinel at '_lock/maintenance.lock' that
serialises every backup or restore op across worker/timer/single-row paths.
* CriticalTableBackupService streams each table through CryptoStream(SHA256)
into per-table NDJSON blobs, then writes manifest.json LAST as the
durability anchor. Per-table failures degrade Outcome to Partial without
aborting; per-run budget (50 min) → Skipped status.
* BackupJobsRepository persists the six-state machine
(Queued/Running/Completed/Failed/Skipped/BlockedTerminal) with ETag-CAS
updates. Enum→string mapping rejects unknown values on read so refactors
surface loudly instead of silently shifting state.
* BackupJobWatchdog (30 min timer) heals stuck rows via a probe lease on the
shared sentinel: Queued > 60 min → Failed, Running with stale heartbeat
AND free lease → Failed (lease-busy means a live worker — leave alone).
* OpsEvents: distinct CriticalTableBackupPartial (Warning) vs Failed (Error)
so operators can wire softer Telegram rules for partial captures.
* MaintenanceService.MonitoredPoisonQueues + OPS_EVENT_TYPES catalog
registered (dual-register memory).
Test coverage (15 new tests, 1952/1952 backend xUnit green):
* BackupManifestJsonTests: enum-as-string (allowIntegerValues=false rejects
numeric values), DictionaryKeyPolicy unset (Azure column names preserved
case-faithfully), manifest field round-trip.
* BackupJobsRepositoryMappingTests: full entity↔domain round-trip, unknown
Kind/State/BackupOutcome rejected, legacy BackupOutcome="Failed" rejected.
* CriticalTableBackupServiceTests: happy/partial/empty/budget-exceeded paths
exercise the per-table loop with in-memory table + blob fakes.
Setup: scripts/Backup/setup-backup-container.ps1 — idempotent az-CLI bootstrap
for container + lifecycle rule (90 d delete on prefix). Soft-Delete and
Versioning are NOT applied automatically (storage-account-wide blast radius
— operator must validate per separate doc note).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Foundation for the critical-table backup feature (plan §PR0). Pure lib
additions + refactor, no behavior change, no new endpoints.
* Lift MapEntityToDump / ConvertToPropValue out of DeletionManifestBuilder
into Services/Tables/TableEntityDumpConverter (public static) so the
upcoming backup pipeline can share the exact EDM-typed row-dump shape.
* Add TableStorageService.UpsertRowsByExactKeysInBatchesAsync with
TableTransactionActionType.UpsertReplace — the only safe primitive
for "overwrite a corrupt live row from backup" (the existing
Add-only RestoreRowsByExactKeysInBatchesAsync would skip 409s).
* Declare Constants.TableNames.BackupJobs and add it to TableNames.All
so startup auto-creates the per-job state table for PR1.
Build: 0 errors, 0 new warnings. 1937/1937 backend xUnit tests pass —
the existing DeletionManifestBuilder suite now runs through the lifted
converter and confirms byte identity.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Patches a remotely triggerable DoS in qs.stringify (TypeError on
null/undefined entries in comma-format arrays when encodeValuesOnly
is set). Pulled in transitively via express@5.2.1 / body-parser@2.2.2,
both of which already allow ^6.14.x — lockfile-only bump, no
package.json change needed. Closes Dependabot alert #46.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Oliver Kieselbach