mirror of
https://github.com/okieselbach/AutopilotMonitor.git
synced 2026-10-25 05:00:07 +00:00
Compare commits
3 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 6d1ff8d7f0 | |||
| 5c8b273f37 | |||
| da958b5398 |
+1
-1
Submodule .claude/commands updated: f7d154ec30...3ed910ab50
@@ -0,0 +1,95 @@
|
||||
---
|
||||
type: Concept
|
||||
title: Business Timestamps vs. the Azure Tables System Timestamp
|
||||
description: Why the system Timestamp is never authoritative, how the 2026-07-18 migration exposed that, and the OccurredUtc + RowKey-decode compensation (mappers, RowKey-range date filters, retention, backfill).
|
||||
resource: /src/Backend/AutopilotMonitor.Functions/Helpers/BusinessTimestamp.cs
|
||||
tags:
|
||||
- backend
|
||||
- table-storage
|
||||
- audit-log
|
||||
- ops-events
|
||||
- events
|
||||
- migration
|
||||
timestamp: 2026-07-20T00:00:00+02:00
|
||||
---
|
||||
|
||||
# Problem
|
||||
|
||||
Azure Table Storage's `Timestamp` is a **system property**: the server ignores any supplied
|
||||
value on write and stamps the row's *write* time; every row rewrite (storage-account
|
||||
migration, MERGE) resets it. Three tables nevertheless treated it as the business event
|
||||
time — writes like `["Timestamp"] = evt.Timestamp` were silent no-ops, unnoticed because
|
||||
write time ≈ event time in normal operation.
|
||||
|
||||
The 2026-07-18 EU storage migration rewrote every row, so all migrated rows in
|
||||
**AuditLogs**, **OpsEvents**, and **Events** read as "2026-07-18 ~13:58 UTC": displays were
|
||||
wrong, date-window filters matched the wrong rows, and retention froze (nothing older than
|
||||
the migration moment) with a mass-delete cliff one retention period later.
|
||||
|
||||
# Rule
|
||||
|
||||
**The system Timestamp is never authoritative.** Every table that needs an event time
|
||||
stores it in the custom column `OccurredUtc` (`BusinessTimestamp.OccurredUtcColumn`).
|
||||
Custom columns are copied verbatim by `scripts/Migration/Copy-TableStorage.ps1`, so future
|
||||
migrations preserve chronology. Migration checklist: any table whose reads/filters depend
|
||||
on a timestamp must carry it as a custom column — audit this before every storage move.
|
||||
|
||||
# Recovery sources per table
|
||||
|
||||
The original time survives migrations inside the RowKey:
|
||||
|
||||
| Table | RowKey scheme | Decodable |
|
||||
|---|---|---|
|
||||
| AuditLogs | `!{MaxTicks-ticks:D19}_{guid:N}` (since 2026-05-06) | Yes; legacy bare-GUID rows (pre-2026-05) are lost |
|
||||
| OpsEvents | `{MaxTicks-ticks:D19}` | Yes, all rows |
|
||||
| Events | `{ts:yyyyMMddHHmmssfff}_{seq:D10}` (sanitized agent time) | Yes, all rows (ms precision) |
|
||||
|
||||
# Mechanism
|
||||
|
||||
`BusinessTimestamp` (Functions/Helpers) is the single source of truth: RowKey decoders,
|
||||
the OData clause builders, and `GetUtcDateTime` (datetime columns materialize as
|
||||
`DateTimeOffset` *or* `DateTime` depending on payload shape — always read via this helper).
|
||||
|
||||
1. **Read mappers** resolve `OccurredUtc` → RowKey decode → system Timestamp (last
|
||||
resort; only audit legacy GUID rows end there). Resolvers:
|
||||
`TableStorageService.ResolveAuditTimestamp`, `TableOpsEventRepository.ResolveTimestamp`,
|
||||
`TableStorageService.ResolveEventTimestamp` (also used by
|
||||
`GetEarliestSessionEventTimestampAsync`, which feeds StartedAt/Duration reconciliation —
|
||||
a system-Timestamp read there would rewrite migrated sessions' derived fields).
|
||||
2. **Date-window filters and retention use RowKey ranges**, not property comparisons:
|
||||
OData excludes rows missing a compared property (an `OccurredUtc` filter would drop
|
||||
every not-yet-backfilled row), and RowKey ranges are index-backed. Boundary semantics
|
||||
are tick-exact equivalents of the old `Timestamp ge/le` clauses.
|
||||
**Trap:** audit legacy GUID RowKeys (hex first char) sort *after* all `!`-rows, so
|
||||
every lower bound `RowKey ge '!…'` must be paired with `RowKey lt '"'`
|
||||
(`AuditTimeEncodedUpperBound`) — without the guard, a retention sweep would delete all
|
||||
legacy rows. Pinned by `BusinessTimestampFilterClauseTests`.
|
||||
3. **Writes** in all three tables now set `OccurredUtc`; the no-op `["Timestamp"]`
|
||||
entries were removed.
|
||||
4. **Backfill** (`POST /api/maintenance/backfill-occurred-utc`, GlobalAdminOnly, dry-run
|
||||
by default, `table=audit|ops`, batched via `maxRows`/`continuation`): MERGE that adds
|
||||
only `OccurredUtc` decoded from the RowKey — deterministic and idempotent. **Events
|
||||
rows are deliberately never backfilled** (read-path decode is sufficient; no bulk
|
||||
mutation of enrollment evidence). Note: a MERGE resets the row's system Timestamp —
|
||||
irrelevant once nothing depends on it.
|
||||
5. **Audit retention** runs two passes until ~2027-01: RowKey-range for `!`-rows plus a
|
||||
`RowKey ge '0' and Timestamp lt …` pass that ages legacy GUID rows out via their frozen
|
||||
migration-date system Timestamp (remove after 2027-03-01).
|
||||
|
||||
# Consequences
|
||||
|
||||
* Audit legacy GUID rows have "unknown date" semantics: excluded from date-filtered
|
||||
views, shown with the (wrong but honest) 2026-07-18 fallback in unfiltered views.
|
||||
* Raw endpoints (`/api/raw/*`, `query_table`) intentionally keep showing the raw system
|
||||
Timestamp — they are storage-state inspectors, not business views.
|
||||
* Event timestamps for pre-cutover rows resolve to the RowKey's sanitized agent time
|
||||
(ms precision); `ReceivedAt` (server ingest time, rows since 2026-03-19) and
|
||||
`OriginalTimestamp` (pre-clamp agent time, clamped rows only) remain separate fields.
|
||||
|
||||
# Citations
|
||||
|
||||
* `src/Backend/AutopilotMonitor.Functions/Helpers/BusinessTimestamp.cs` — decoders + clause builders
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.Maintenance.cs` — audit write/mapper/filters/retention
|
||||
* `src/Backend/AutopilotMonitor.Functions/DataAccess/TableStorage/TableOpsEventRepository.cs` — ops write/mapper/filters/retention
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.Sessions.cs` — event writes, `ResolveEventTimestamp`, earliest-event read
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/OccurredUtcBackfillService.cs` + `Functions/Admin/BackfillOccurredUtcFunction.cs` — backfill
|
||||
@@ -23,6 +23,10 @@ https://docs.autopilotmonitor.com (separate repository).
|
||||
|
||||
* [Gather Rule Phase Scoping & Emit-on-Change](rules/gather-rule-phase-scoping.md) - Restricting gather rules to enrollment phases (activePhases / activeFromPhase sticky latch) and emitMode "on_change" result-dedup — the anti-spam pair for interval rules.
|
||||
|
||||
# Backend
|
||||
|
||||
* [Business Timestamps](backend/business-timestamps.md) - Why the Azure Tables system Timestamp is never authoritative (migrations reset it), and the OccurredUtc + RowKey-decode compensation for AuditLogs/OpsEvents/Events after the 2026-07-18 migration.
|
||||
|
||||
# Security & Identity
|
||||
|
||||
* [MCP OAuth Flow](mcp-oauth-flow.md) - Who authenticates where when connecting an AI client to the MCP server; two identities, three parties plus Entra ID.
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
## 2026-07-20
|
||||
|
||||
* **Creation**: Added `backend/business-timestamps.md` (new `backend/` section) — the 2026-07-18 migration reset every row's system Timestamp, exposing that AuditLogs/OpsEvents/Events had (unknowingly) displayed and filtered on it all along; documents the `OccurredUtc` column, the RowKey decoders and RowKey-range date/retention clauses in `BusinessTimestamp`, the GUID-guard trap, the audit/ops backfill endpoint, and the "system Timestamp is never authoritative" migration rule.
|
||||
* **Creation**: Added `rules/gather-rule-phase-scoping.md` (new `rules/` section) — gather rules gain `activePhases` (run only during listed phases), `activeFromPhase` (sticky latch from a phase onwards, never via Failed) and `emitMode: "on_change"` (canonical-hash result dedup with `suppressedPolls` observability), ConfigVersion 34. Includes the prerequisite backend fix: the custom-rule toggle PUT no longer wipes the stored rule definition (partial-merge in `GatherRuleService.UpdateRuleAsync`).
|
||||
|
||||
## 2026-07-18
|
||||
|
||||
+19
@@ -136,6 +136,25 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Security
|
||||
}
|
||||
|
||||
watcher.Start();
|
||||
|
||||
// Re-check after arming: Dispose() may have run between the lock release above and
|
||||
// Start() — it then stopped a not-yet-armed watcher and this thread armed it right
|
||||
// back, leaving a WMI subscription no code path could ever reach again. If disposal
|
||||
// won that race, tear the watcher down here.
|
||||
bool disposedDuringArm;
|
||||
lock (_lock)
|
||||
{
|
||||
disposedDuringArm = _disposed;
|
||||
if (disposedDuringArm) _startWatcher = null;
|
||||
}
|
||||
if (disposedDuringArm)
|
||||
{
|
||||
try { watcher.EventArrived -= OnProcessStartTrace; } catch { }
|
||||
try { watcher.Stop(); } catch { }
|
||||
try { watcher.Dispose(); } catch { }
|
||||
return;
|
||||
}
|
||||
|
||||
_logger.Info($"[ConsoleBypassWatcher] watching Win32_ProcessStartTrace for " +
|
||||
$"{string.Join(", ", WatchedProcessNames)} (interactive-session + bare command line)");
|
||||
}
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
using AutopilotMonitor.Functions.Helpers;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// RowKey → business-time decoders. These are the read-path compensation for rows whose
|
||||
/// system Timestamp was reset by the 2026-07-18 storage migration: the RowKey is the only
|
||||
/// time carrier that survives a row rewrite, so the decode must round-trip exactly and
|
||||
/// must reject every foreign format (falling back instead of mis-dating rows).
|
||||
/// </summary>
|
||||
public class BusinessTimestampDecodeTests
|
||||
{
|
||||
private static readonly DateTime Ts = new DateTime(2026, 6, 15, 10, 30, 45, 123, DateTimeKind.Utc).AddTicks(4567);
|
||||
|
||||
[Fact]
|
||||
public void Audit_rowkey_roundtrips_tick_exact()
|
||||
{
|
||||
var rowKey = TableStorageService.BuildAuditLogRowKey(Ts, Guid.NewGuid());
|
||||
Assert.True(BusinessTimestamp.TryDecodeAuditRowKey(rowKey, out var decoded));
|
||||
Assert.Equal(Ts, decoded);
|
||||
Assert.Equal(DateTimeKind.Utc, decoded.Kind);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(null)]
|
||||
[InlineData("")]
|
||||
[InlineData("!")]
|
||||
[InlineData("!123")] // too short
|
||||
[InlineData("!123456789012345678x_guid")] // non-digit in revtick
|
||||
[InlineData("d6f6dd90f41c4b16b290aac042e5b466")] // legacy bare GUID (:N)
|
||||
[InlineData("0000000000000000000")] // ops-style bare revtick is not an audit key
|
||||
public void Audit_decoder_rejects_foreign_formats(string? rowKey)
|
||||
{
|
||||
Assert.False(BusinessTimestamp.TryDecodeAuditRowKey(rowKey, out _));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Ops_rowkey_roundtrips_tick_exact()
|
||||
{
|
||||
var rowKey = $"{DateTime.MaxValue.Ticks - Ts.Ticks:D19}";
|
||||
Assert.True(BusinessTimestamp.TryDecodeOpsRowKey(rowKey, out var decoded));
|
||||
Assert.Equal(Ts, decoded);
|
||||
Assert.Equal(DateTimeKind.Utc, decoded.Kind);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(null)]
|
||||
[InlineData("")]
|
||||
[InlineData("123456789012345678")] // 18 digits
|
||||
[InlineData("12345678901234567890")] // 20 digits
|
||||
[InlineData("123456789012345678x")] // non-digit
|
||||
[InlineData("!1234567890123456789")] // audit-prefixed
|
||||
public void Ops_decoder_rejects_foreign_formats(string? rowKey)
|
||||
{
|
||||
Assert.False(BusinessTimestamp.TryDecodeOpsRowKey(rowKey, out _));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Event_rowkey_prefix_decodes_to_millisecond_truncated_agent_time()
|
||||
{
|
||||
var rowKey = $"{Ts:yyyyMMddHHmmssfff}_{42L:D10}";
|
||||
Assert.True(BusinessTimestamp.TryDecodeEventRowKeyPrefix(rowKey, out var decoded));
|
||||
// The RowKey format truncates sub-millisecond ticks at write time.
|
||||
var expected = new DateTime(2026, 6, 15, 10, 30, 45, 123, DateTimeKind.Utc);
|
||||
Assert.Equal(expected, decoded);
|
||||
Assert.Equal(DateTimeKind.Utc, decoded.Kind);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(null)]
|
||||
[InlineData("")]
|
||||
[InlineData("20260615103045123")] // no separator/sequence
|
||||
[InlineData("2026061510304512_0000000042")] // 16-digit prefix
|
||||
[InlineData("202606151030451234_0000000042")] // 18-digit prefix
|
||||
[InlineData("2026061510304512x_0000000042")] // non-digit in prefix
|
||||
[InlineData("20261315103045123_0000000042")] // month 13
|
||||
[InlineData("d6f6dd90f41c4b16b290aac042e5b466")] // GUID
|
||||
public void Event_decoder_rejects_foreign_formats(string? rowKey)
|
||||
{
|
||||
Assert.False(BusinessTimestamp.TryDecodeEventRowKeyPrefix(rowKey, out _));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,181 @@
|
||||
using System.Text.RegularExpressions;
|
||||
using AutopilotMonitor.Functions.DataAccess.TableStorage;
|
||||
using AutopilotMonitor.Functions.Helpers;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// RowKey-range date-window / retention clauses. Date filtering moved off the system
|
||||
/// Timestamp (reset by storage migrations) onto the reverse-tick RowKey; these tests pin
|
||||
/// (a) tick-exact inclusive/exclusive boundary semantics identical to the old
|
||||
/// `Timestamp ge/le` clauses, and (b) the GUID-safety invariant: legacy bare-GUID audit
|
||||
/// rows sort AFTER all '!'-rows, so any lower RowKey bound without the `RowKey lt '"'`
|
||||
/// guard would match — and a retention sweep would DELETE — every legacy row.
|
||||
/// </summary>
|
||||
public class BusinessTimestampFilterClauseTests
|
||||
{
|
||||
private static readonly DateTime T = new DateTime(2026, 6, 15, 12, 0, 0, DateTimeKind.Utc).AddTicks(789);
|
||||
private static readonly string Tenant = "11111111-1111-1111-1111-111111111111";
|
||||
|
||||
private static string AuditRk(DateTime ts) => TableStorageService.BuildAuditLogRowKey(ts, Guid.NewGuid());
|
||||
private static string OpsRk(DateTime ts) => $"{DateTime.MaxValue.Ticks - ts.Ticks:D19}";
|
||||
|
||||
/// <summary>Evaluates a clause of the form "RowKey op 'literal' [and RowKey op 'literal']…" against a RowKey.</summary>
|
||||
private static bool Matches(string clause, string rowKey)
|
||||
{
|
||||
var parts = Regex.Matches(clause, @"RowKey (ge|gt|le|lt) '([^']*)'");
|
||||
Assert.True(parts.Count > 0, $"no RowKey comparisons in clause: {clause}");
|
||||
foreach (Match m in parts)
|
||||
{
|
||||
var cmp = string.CompareOrdinal(rowKey, m.Groups[2].Value);
|
||||
var ok = m.Groups[1].Value switch
|
||||
{
|
||||
"ge" => cmp >= 0,
|
||||
"gt" => cmp > 0,
|
||||
"le" => cmp <= 0,
|
||||
"lt" => cmp < 0,
|
||||
_ => throw new InvalidOperationException(),
|
||||
};
|
||||
if (!ok) return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
// ===== Audit boundaries =====
|
||||
|
||||
[Fact]
|
||||
public void Audit_dateFrom_is_inclusive_to_the_tick()
|
||||
{
|
||||
var clause = BusinessTimestamp.AuditDateFromClause(T);
|
||||
Assert.True(Matches(clause, AuditRk(T))); // ts == from → included
|
||||
Assert.True(Matches(clause, AuditRk(T.AddTicks(1)))); // newer → included
|
||||
Assert.False(Matches(clause, AuditRk(T.AddTicks(-1)))); // 1 tick older → excluded
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Audit_dateTo_is_inclusive_to_the_tick()
|
||||
{
|
||||
var clause = BusinessTimestamp.AuditDateToClause(T);
|
||||
Assert.True(Matches(clause, AuditRk(T))); // ts == to → included
|
||||
Assert.True(Matches(clause, AuditRk(T.AddTicks(-1)))); // older → included
|
||||
Assert.False(Matches(clause, AuditRk(T.AddTicks(1)))); // 1 tick newer → excluded
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Audit_retention_is_strictly_older_than_cutoff()
|
||||
{
|
||||
var clause = BusinessTimestamp.AuditRetentionClause(T);
|
||||
Assert.False(Matches(clause, AuditRk(T))); // ts == cutoff → kept
|
||||
Assert.True(Matches(clause, AuditRk(T.AddTicks(-1)))); // 1 tick older → deleted
|
||||
Assert.False(Matches(clause, AuditRk(T.AddTicks(1)))); // newer → kept
|
||||
}
|
||||
|
||||
// ===== Audit GUID safety (the single most dangerous invariant of this change) =====
|
||||
|
||||
[Fact]
|
||||
public void Legacy_guid_rowkeys_sort_after_the_time_encoded_guard()
|
||||
{
|
||||
for (var i = 0; i < 20; i++)
|
||||
{
|
||||
var guidRk = Guid.NewGuid().ToString("N");
|
||||
Assert.True(string.CompareOrdinal(guidRk, BusinessTimestamp.AuditTimeEncodedUpperBound) > 0);
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void No_audit_date_or_retention_clause_matches_legacy_guid_rowkeys()
|
||||
{
|
||||
var clauses = new[]
|
||||
{
|
||||
BusinessTimestamp.AuditDateFromClause(T),
|
||||
BusinessTimestamp.AuditDateToClause(T),
|
||||
BusinessTimestamp.AuditDateToClause(DateTime.MaxValue.AddTicks(-1)), // widest possible window
|
||||
BusinessTimestamp.AuditRetentionClause(T),
|
||||
BusinessTimestamp.AuditRetentionClause(DateTime.MaxValue.AddTicks(-1)), // widest possible sweep
|
||||
};
|
||||
foreach (var clause in clauses)
|
||||
{
|
||||
for (var i = 0; i < 20; i++)
|
||||
{
|
||||
Assert.False(Matches(clause, Guid.NewGuid().ToString("N")),
|
||||
$"clause matched a legacy GUID RowKey: {clause}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Audit_lower_bounded_clauses_carry_the_time_encoded_guard()
|
||||
{
|
||||
Assert.Contains($"RowKey lt '{BusinessTimestamp.AuditTimeEncodedUpperBound}'",
|
||||
BusinessTimestamp.AuditDateToClause(T));
|
||||
Assert.Contains($"RowKey lt '{BusinessTimestamp.AuditTimeEncodedUpperBound}'",
|
||||
BusinessTimestamp.AuditRetentionClause(T));
|
||||
}
|
||||
|
||||
// ===== Ops boundaries =====
|
||||
|
||||
[Fact]
|
||||
public void Ops_dateFrom_is_inclusive_to_the_tick()
|
||||
{
|
||||
var clause = BusinessTimestamp.OpsDateFromClause(T);
|
||||
Assert.True(Matches(clause, OpsRk(T)));
|
||||
Assert.True(Matches(clause, OpsRk(T.AddTicks(1))));
|
||||
Assert.False(Matches(clause, OpsRk(T.AddTicks(-1))));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Ops_dateTo_is_inclusive_to_the_tick()
|
||||
{
|
||||
var clause = BusinessTimestamp.OpsDateToClause(T);
|
||||
Assert.True(Matches(clause, OpsRk(T)));
|
||||
Assert.True(Matches(clause, OpsRk(T.AddTicks(-1))));
|
||||
Assert.False(Matches(clause, OpsRk(T.AddTicks(1))));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Ops_retention_is_strictly_older_than_cutoff()
|
||||
{
|
||||
var clause = BusinessTimestamp.OpsRetentionClause(T);
|
||||
Assert.False(Matches(clause, OpsRk(T)));
|
||||
Assert.True(Matches(clause, OpsRk(T.AddTicks(-1))));
|
||||
Assert.False(Matches(clause, OpsRk(T.AddTicks(1))));
|
||||
}
|
||||
|
||||
// ===== Composition into the filter builders =====
|
||||
|
||||
[Fact]
|
||||
public void Audit_filter_builders_emit_rowkey_windows_not_timestamp_clauses()
|
||||
{
|
||||
var from = new DateTime(2026, 6, 1, 0, 0, 0, DateTimeKind.Utc);
|
||||
var to = new DateTime(2026, 7, 1, 0, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
var plain = TableStorageService.BuildAuditLogFilter(Tenant, from, to);
|
||||
var fanOut = TableStorageService.BuildAuditLogFilterWithRowKeyBound(
|
||||
Tenant, from, to, lastRowKey: "!0123_x", excludeDeletions: false);
|
||||
|
||||
foreach (var f in new[] { plain!, fanOut })
|
||||
{
|
||||
Assert.DoesNotContain("Timestamp ge", f);
|
||||
Assert.DoesNotContain("Timestamp le", f);
|
||||
Assert.Contains(BusinessTimestamp.AuditDateFromClause(from), f);
|
||||
Assert.Contains(BusinessTimestamp.AuditDateToClause(to), f);
|
||||
}
|
||||
Assert.Contains("RowKey gt '!0123_x'", fanOut); // fan-out cursor composes with the window
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Ops_filter_builder_emits_rowkey_window_not_timestamp_clauses()
|
||||
{
|
||||
var from = new DateTime(2026, 6, 1, 0, 0, 0, DateTimeKind.Utc);
|
||||
var to = new DateTime(2026, 7, 1, 0, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
var f = TableOpsEventRepository.BuildFilter("Security", from, to);
|
||||
|
||||
Assert.NotNull(f);
|
||||
Assert.DoesNotContain("Timestamp ge", f);
|
||||
Assert.DoesNotContain("Timestamp le", f);
|
||||
Assert.Contains(BusinessTimestamp.OpsDateFromClause(from), f);
|
||||
Assert.Contains(BusinessTimestamp.OpsDateToClause(to), f);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
using Azure.Data.Tables;
|
||||
using AutopilotMonitor.Functions.DataAccess.TableStorage;
|
||||
using AutopilotMonitor.Functions.Helpers;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Read-mapper resolution order for the three affected tables:
|
||||
/// OccurredUtc column → RowKey decode → system Timestamp. The last hop is the
|
||||
/// migration-corrupted value (a row rewrite resets it), so it must only ever win when
|
||||
/// the row carries no better source (audit legacy bare-GUID rows).
|
||||
/// </summary>
|
||||
public class BusinessTimestampPrecedenceTests
|
||||
{
|
||||
private static readonly DateTime Occurred = new(2026, 5, 10, 8, 0, 0, DateTimeKind.Utc);
|
||||
private static readonly DateTime RowKeyTime = new(2026, 6, 20, 9, 30, 0, DateTimeKind.Utc);
|
||||
private static readonly DateTimeOffset SystemTs = new(2026, 7, 18, 13, 58, 0, TimeSpan.Zero); // migration moment
|
||||
|
||||
private static TableEntity WithSystemTimestamp(TableEntity e)
|
||||
{
|
||||
e["Timestamp"] = SystemTs;
|
||||
return e;
|
||||
}
|
||||
|
||||
// ===== Audit =====
|
||||
|
||||
[Fact]
|
||||
public void Audit_occurredutc_wins_over_rowkey_and_system()
|
||||
{
|
||||
var e = WithSystemTimestamp(new TableEntity("t", TableStorageService.BuildAuditLogRowKey(RowKeyTime, Guid.NewGuid()))
|
||||
{
|
||||
[BusinessTimestamp.OccurredUtcColumn] = new DateTimeOffset(Occurred),
|
||||
});
|
||||
Assert.Equal(Occurred, TableStorageService.ResolveAuditTimestamp(e));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Audit_rowkey_decode_wins_over_system_when_column_missing()
|
||||
{
|
||||
var e = WithSystemTimestamp(new TableEntity("t", TableStorageService.BuildAuditLogRowKey(RowKeyTime, Guid.NewGuid())));
|
||||
Assert.Equal(RowKeyTime, TableStorageService.ResolveAuditTimestamp(e));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Audit_legacy_guid_row_falls_back_to_system_timestamp()
|
||||
{
|
||||
var e = WithSystemTimestamp(new TableEntity("t", Guid.NewGuid().ToString("N")));
|
||||
Assert.Equal(SystemTs.UtcDateTime, TableStorageService.ResolveAuditTimestamp(e));
|
||||
}
|
||||
|
||||
// ===== Ops =====
|
||||
|
||||
[Fact]
|
||||
public void Ops_occurredutc_wins_over_rowkey_and_system()
|
||||
{
|
||||
var e = WithSystemTimestamp(new TableEntity("Security", $"{DateTime.MaxValue.Ticks - RowKeyTime.Ticks:D19}")
|
||||
{
|
||||
[BusinessTimestamp.OccurredUtcColumn] = new DateTimeOffset(Occurred),
|
||||
});
|
||||
Assert.Equal(Occurred, TableOpsEventRepository.ResolveTimestamp(e));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Ops_rowkey_decode_wins_over_system_when_column_missing()
|
||||
{
|
||||
var e = WithSystemTimestamp(new TableEntity("Security", $"{DateTime.MaxValue.Ticks - RowKeyTime.Ticks:D19}"));
|
||||
Assert.Equal(RowKeyTime, TableOpsEventRepository.ResolveTimestamp(e));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Ops_undecodable_rowkey_falls_back_to_system_timestamp()
|
||||
{
|
||||
var e = WithSystemTimestamp(new TableEntity("Security", "not-a-revtick"));
|
||||
Assert.Equal(SystemTs.UtcDateTime, TableOpsEventRepository.ResolveTimestamp(e));
|
||||
}
|
||||
|
||||
// ===== Events =====
|
||||
|
||||
private static string EventRowKey(DateTime ts, long seq = 42) => $"{ts:yyyyMMddHHmmssfff}_{seq:D10}";
|
||||
|
||||
[Fact]
|
||||
public void Event_occurredutc_wins_over_rowkey_and_system()
|
||||
{
|
||||
var e = WithSystemTimestamp(new TableEntity("tenant_session", EventRowKey(RowKeyTime))
|
||||
{
|
||||
[BusinessTimestamp.OccurredUtcColumn] = new DateTimeOffset(Occurred),
|
||||
});
|
||||
Assert.Equal(Occurred, TableStorageService.ResolveEventTimestamp(e));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Event_rowkey_prefix_wins_over_system_when_column_missing()
|
||||
{
|
||||
// The pre-cutover case: no OccurredUtc column, system Timestamp = migration moment.
|
||||
var e = WithSystemTimestamp(new TableEntity("tenant_session", EventRowKey(RowKeyTime)));
|
||||
Assert.Equal(RowKeyTime, TableStorageService.ResolveEventTimestamp(e));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Event_undecodable_rowkey_falls_back_to_system_timestamp()
|
||||
{
|
||||
var e = WithSystemTimestamp(new TableEntity("tenant_session", "weird-rowkey"));
|
||||
Assert.Equal(SystemTs.UtcDateTime, TableStorageService.ResolveEventTimestamp(e));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Event_occurredutc_read_tolerates_datetime_materialization()
|
||||
{
|
||||
// SDK reads surface Edm.DateTime as DateTimeOffset or DateTime depending on payload
|
||||
// shape — the resolver must accept both (dual-reader lesson).
|
||||
var e = new TableEntity("tenant_session", EventRowKey(RowKeyTime))
|
||||
{
|
||||
[BusinessTimestamp.OccurredUtcColumn] = Occurred, // plain DateTime, not DateTimeOffset
|
||||
};
|
||||
Assert.Equal(Occurred, TableStorageService.ResolveEventTimestamp(e));
|
||||
}
|
||||
}
|
||||
+26
@@ -129,4 +129,30 @@ public class GetAllTenantConfigurationsFunctionTests
|
||||
[Fact]
|
||||
public void DelegatedBareArrayView_Empty_Empty()
|
||||
=> Assert.Empty(GetAllTenantConfigurationsFunction.DelegatedBareArrayView(new TenantConfiguration[0]));
|
||||
|
||||
[Fact]
|
||||
public void MissingManagedIds_ReturnsOnlyIdsWithoutConfig_CaseInsensitive()
|
||||
{
|
||||
var configs = new List<TenantConfiguration>
|
||||
{
|
||||
new() { TenantId = TenantB.ToUpperInvariant() }, // case variant still counts as present
|
||||
};
|
||||
|
||||
var missing = GetAllTenantConfigurationsFunction.MissingManagedIds(new[] { TenantB, TenantC }, configs);
|
||||
|
||||
var only = Assert.Single(missing);
|
||||
Assert.Equal(TenantC, only);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void MissingManagedIds_AllPresent_Empty()
|
||||
=> Assert.Empty(GetAllTenantConfigurationsFunction.MissingManagedIds(
|
||||
new[] { TenantB },
|
||||
new List<TenantConfiguration> { new() { TenantId = TenantB } }));
|
||||
|
||||
[Fact]
|
||||
public void MissingConfigCacheKey_LowercasesId_SoCaseVariantsShareOneVerdict()
|
||||
=> Assert.Equal(
|
||||
GetAllTenantConfigurationsFunction.MissingConfigCacheKey(TenantB),
|
||||
GetAllTenantConfigurationsFunction.MissingConfigCacheKey(TenantB.ToUpperInvariant()));
|
||||
}
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
using Azure.Data.Tables;
|
||||
using AutopilotMonitor.Functions.Helpers;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Per-row backfill decision (pure function): rows that already carry OccurredUtc are
|
||||
/// skipped (idempotent re-runs), decodable RowKeys yield the deterministic decoded value,
|
||||
/// anything else is reported undecodable — never guessed.
|
||||
/// </summary>
|
||||
public class OccurredUtcBackfillServiceTests
|
||||
{
|
||||
private static readonly DateTime Ts = new(2026, 6, 20, 9, 30, 0, DateTimeKind.Utc);
|
||||
|
||||
[Fact]
|
||||
public void Audit_row_with_existing_column_is_skipped()
|
||||
{
|
||||
var e = new TableEntity("t", TableStorageService.BuildAuditLogRowKey(Ts, Guid.NewGuid()))
|
||||
{
|
||||
[BusinessTimestamp.OccurredUtcColumn] = new DateTimeOffset(Ts),
|
||||
};
|
||||
Assert.Equal(OccurredUtcBackfillService.RowDecision.AlreadySet,
|
||||
OccurredUtcBackfillService.DecideRow(OccurredUtcBackfillService.TableAudit, e, out _));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Audit_time_encoded_row_yields_decoded_write()
|
||||
{
|
||||
var e = new TableEntity("t", TableStorageService.BuildAuditLogRowKey(Ts, Guid.NewGuid()));
|
||||
var decision = OccurredUtcBackfillService.DecideRow(OccurredUtcBackfillService.TableAudit, e, out var occurred);
|
||||
Assert.Equal(OccurredUtcBackfillService.RowDecision.Write, decision);
|
||||
Assert.Equal(Ts, occurred);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Audit_legacy_guid_row_is_undecodable_not_guessed()
|
||||
{
|
||||
var e = new TableEntity("t", Guid.NewGuid().ToString("N"));
|
||||
Assert.Equal(OccurredUtcBackfillService.RowDecision.Undecodable,
|
||||
OccurredUtcBackfillService.DecideRow(OccurredUtcBackfillService.TableAudit, e, out _));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Ops_row_yields_decoded_write_and_existing_column_is_skipped()
|
||||
{
|
||||
var rowKey = $"{DateTime.MaxValue.Ticks - Ts.Ticks:D19}";
|
||||
|
||||
var bare = new TableEntity("Security", rowKey);
|
||||
var decision = OccurredUtcBackfillService.DecideRow(OccurredUtcBackfillService.TableOps, bare, out var occurred);
|
||||
Assert.Equal(OccurredUtcBackfillService.RowDecision.Write, decision);
|
||||
Assert.Equal(Ts, occurred);
|
||||
|
||||
var backfilled = new TableEntity("Security", rowKey)
|
||||
{
|
||||
[BusinessTimestamp.OccurredUtcColumn] = new DateTimeOffset(Ts),
|
||||
};
|
||||
Assert.Equal(OccurredUtcBackfillService.RowDecision.AlreadySet,
|
||||
OccurredUtcBackfillService.DecideRow(OccurredUtcBackfillService.TableOps, backfilled, out _));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,269 @@
|
||||
using AutopilotMonitor.Functions.Functions.Admin;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using AutopilotMonitor.Shared;
|
||||
using AutopilotMonitor.Shared.DataAccess;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Moq;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Covers the SignalR revoke-enforcement cores of <see cref="DelegatedAdminManagementFunction"/> and
|
||||
/// <see cref="TenantGroupManagementFunction"/> (extracted seams — the HTTP entry points are intentionally
|
||||
/// not exercised, same rationale as <see cref="GetAllBlockedDevicesFunctionTests"/>). Locks in three
|
||||
/// security invariants: (1) a miss (no assignment row) neither audits nor disconnects — a typo must not
|
||||
/// write false customer-visible "access removed" rows while the real grant stays live; (2) a real
|
||||
/// revoke/disable/unassign/group-delete always cuts the affected UPN's live streams (group authorization
|
||||
/// is join-time only); (3) grant validation is fail-closed (GUID tenantId, recognized roles only).
|
||||
/// </summary>
|
||||
public class RevokeEnforcementTests
|
||||
{
|
||||
private const string Upn = "MSP-Admin@Partner.Example";
|
||||
private const string UpnLower = "msp-admin@partner.example";
|
||||
private const string TenantA = "11111111-1111-1111-1111-111111111111";
|
||||
private const string TenantB = "22222222-2222-2222-2222-222222222222";
|
||||
private const string GroupId = "tpl-aaaaaaaa";
|
||||
|
||||
private static DelegatedAdminService BuildService(Mock<IAdminRepository> repo) =>
|
||||
new(
|
||||
repo.Object,
|
||||
new StubTenantEntitlementService(AutopilotMonitor.Functions.Security.TenantEdition.Enterprise),
|
||||
new MemoryCache(new MemoryCacheOptions()),
|
||||
NullLogger<DelegatedAdminService>.Instance);
|
||||
|
||||
private static Mock<IMaintenanceRepository> BuildAuditRepo()
|
||||
{
|
||||
var audit = new Mock<IMaintenanceRepository>();
|
||||
audit.Setup(a => a.LogAuditEntryAsync(
|
||||
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<string>(),
|
||||
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<Dictionary<string, string>?>()))
|
||||
.ReturnsAsync(true);
|
||||
return audit;
|
||||
}
|
||||
|
||||
private static void VerifyNoAudit(Mock<IMaintenanceRepository> audit) =>
|
||||
audit.Verify(a => a.LogAuditEntryAsync(
|
||||
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<string>(),
|
||||
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<Dictionary<string, string>?>()), Times.Never);
|
||||
|
||||
private static (DelegatedAdminManagementFunction Fn, Mock<IAdminRepository> Repo,
|
||||
Mock<IMaintenanceRepository> Audit, FakeSignalRNotificationService SignalR) BuildAdminFn()
|
||||
{
|
||||
var repo = new Mock<IAdminRepository>();
|
||||
var audit = BuildAuditRepo();
|
||||
var signalR = new FakeSignalRNotificationService();
|
||||
var fn = new DelegatedAdminManagementFunction(
|
||||
NullLogger<DelegatedAdminManagementFunction>.Instance, BuildService(repo), audit.Object, signalR);
|
||||
return (fn, repo, audit, signalR);
|
||||
}
|
||||
|
||||
private static (TenantGroupManagementFunction Fn, Mock<IAdminRepository> Repo,
|
||||
Mock<IMaintenanceRepository> Audit, FakeSignalRNotificationService SignalR) BuildGroupFn()
|
||||
{
|
||||
var repo = new Mock<IAdminRepository>();
|
||||
// Loose-mock defaults return a null Task — every repo read the flows touch needs a concrete setup.
|
||||
repo.Setup(r => r.GetTenantGroupAsync(It.IsAny<string>())).ReturnsAsync((TenantGroup?)null);
|
||||
repo.Setup(r => r.GetGroupAssignmentsForUpnAsync(It.IsAny<string>()))
|
||||
.ReturnsAsync(new List<TenantGroupAssignment>());
|
||||
repo.Setup(r => r.GetGroupAssigneesAsync(It.IsAny<string>()))
|
||||
.ReturnsAsync(new List<TenantGroupAssignment>());
|
||||
var audit = BuildAuditRepo();
|
||||
var signalR = new FakeSignalRNotificationService();
|
||||
var fn = new TenantGroupManagementFunction(
|
||||
NullLogger<TenantGroupManagementFunction>.Instance, BuildService(repo), audit.Object, signalR);
|
||||
return (fn, repo, audit, signalR);
|
||||
}
|
||||
|
||||
// --- Delegated grant validation (fail-closed) ---
|
||||
|
||||
[Fact]
|
||||
public void ValidateGrant_NullBody_Rejected()
|
||||
=> Assert.Equal("upn and tenantId are required",
|
||||
DelegatedAdminManagementFunction.ValidateGrantRequest(null, out _));
|
||||
|
||||
[Theory]
|
||||
[InlineData("", TenantA)]
|
||||
[InlineData(" ", TenantA)]
|
||||
[InlineData(Upn, "")]
|
||||
public void ValidateGrant_MissingFields_Rejected(string upn, string tenantId)
|
||||
=> Assert.Equal("upn and tenantId are required",
|
||||
DelegatedAdminManagementFunction.ValidateGrantRequest(
|
||||
new GrantDelegatedAdminRequest { Upn = upn, TenantId = tenantId }, out _));
|
||||
|
||||
[Theory]
|
||||
[InlineData("contoso.com")]
|
||||
[InlineData("not-a-guid")]
|
||||
[InlineData("11111111-1111-1111-1111")]
|
||||
public void ValidateGrant_NonGuidTenantId_Rejected(string tenantId)
|
||||
=> Assert.Equal("a valid tenantId (GUID) is required",
|
||||
DelegatedAdminManagementFunction.ValidateGrantRequest(
|
||||
new GrantDelegatedAdminRequest { Upn = Upn, TenantId = tenantId }, out _));
|
||||
|
||||
[Fact]
|
||||
public void ValidateGrant_UnknownRole_Rejected()
|
||||
{
|
||||
var error = DelegatedAdminManagementFunction.ValidateGrantRequest(
|
||||
new GrantDelegatedAdminRequest { Upn = Upn, TenantId = TenantA, Role = "GlobalAdmin" }, out _);
|
||||
Assert.NotNull(error);
|
||||
Assert.Contains(Constants.DelegatedRoles.DelegatedReader, error);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ValidateGrant_EmptyRole_DefaultsToLeastPrivilegedReader()
|
||||
{
|
||||
Assert.Null(DelegatedAdminManagementFunction.ValidateGrantRequest(
|
||||
new GrantDelegatedAdminRequest { Upn = Upn, TenantId = TenantA }, out var role));
|
||||
Assert.Equal(Constants.DelegatedRoles.DelegatedReader, role);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ValidateGrant_ExplicitDelegatedAdmin_Accepted()
|
||||
{
|
||||
Assert.Null(DelegatedAdminManagementFunction.ValidateGrantRequest(
|
||||
new GrantDelegatedAdminRequest { Upn = Upn, TenantId = TenantA, Role = Constants.DelegatedRoles.DelegatedAdmin },
|
||||
out var role));
|
||||
Assert.Equal(Constants.DelegatedRoles.DelegatedAdmin, role);
|
||||
}
|
||||
|
||||
// --- Revoke: 404-on-missing-row vs. audit + disconnect ---
|
||||
|
||||
[Fact]
|
||||
public async Task RevokeCore_MissingRow_NoAuditNoDisconnect()
|
||||
{
|
||||
var (fn, repo, audit, signalR) = BuildAdminFn();
|
||||
repo.Setup(r => r.RemoveDelegatedAdminAsync(UpnLower, TenantA)).ReturnsAsync(false);
|
||||
|
||||
var removed = await fn.RevokeCoreAsync(Upn, TenantA, "ga@vendor.example");
|
||||
|
||||
Assert.False(removed);
|
||||
VerifyNoAudit(audit);
|
||||
Assert.Empty(signalR.DisconnectedUsers);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RevokeCore_RemovedRow_AuditsAndDisconnectsLowercasedUpn()
|
||||
{
|
||||
var (fn, repo, audit, signalR) = BuildAdminFn();
|
||||
repo.Setup(r => r.RemoveDelegatedAdminAsync(UpnLower, TenantA)).ReturnsAsync(true);
|
||||
|
||||
var removed = await fn.RevokeCoreAsync(Upn, TenantA, "ga@vendor.example");
|
||||
|
||||
Assert.True(removed);
|
||||
// SignalR user ids are lowercased UPNs — a mixed-case route param must still hit the user's connections.
|
||||
Assert.Equal(new[] { UpnLower }, signalR.DisconnectedUsers);
|
||||
audit.Verify(a => a.LogAuditEntryAsync(
|
||||
TenantA, "DELETE", "DelegatedAdmin", UpnLower, "ga@vendor.example", null), Times.Once);
|
||||
}
|
||||
|
||||
// --- Disable is a revocation too; enable never disconnects ---
|
||||
|
||||
[Fact]
|
||||
public async Task SetEnabledCore_MissingRow_NoAuditNoDisconnect()
|
||||
{
|
||||
var (fn, repo, audit, signalR) = BuildAdminFn();
|
||||
repo.Setup(r => r.SetDelegatedAdminEnabledAsync(UpnLower, TenantA, false)).ReturnsAsync(false);
|
||||
|
||||
var ok = await fn.SetEnabledCoreAsync(Upn, TenantA, isEnabled: false, "ga@vendor.example");
|
||||
|
||||
Assert.False(ok);
|
||||
VerifyNoAudit(audit);
|
||||
Assert.Empty(signalR.DisconnectedUsers);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SetEnabledCore_Disable_Disconnects()
|
||||
{
|
||||
var (fn, repo, _, signalR) = BuildAdminFn();
|
||||
repo.Setup(r => r.SetDelegatedAdminEnabledAsync(UpnLower, TenantA, false)).ReturnsAsync(true);
|
||||
|
||||
var ok = await fn.SetEnabledCoreAsync(Upn, TenantA, isEnabled: false, "ga@vendor.example");
|
||||
|
||||
Assert.True(ok);
|
||||
Assert.Equal(new[] { UpnLower }, signalR.DisconnectedUsers);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SetEnabledCore_Enable_DoesNotDisconnect()
|
||||
{
|
||||
var (fn, repo, _, signalR) = BuildAdminFn();
|
||||
repo.Setup(r => r.SetDelegatedAdminEnabledAsync(UpnLower, TenantA, true)).ReturnsAsync(true);
|
||||
|
||||
var ok = await fn.SetEnabledCoreAsync(Upn, TenantA, isEnabled: true, "ga@vendor.example");
|
||||
|
||||
Assert.True(ok);
|
||||
Assert.Empty(signalR.DisconnectedUsers);
|
||||
}
|
||||
|
||||
// --- Group unassign: 404-on-not-assigned vs. disconnect + per-tenant audit ---
|
||||
|
||||
[Fact]
|
||||
public async Task UnassignCore_NotAssigned_NoAuditNoDisconnect()
|
||||
{
|
||||
var (fn, _, audit, signalR) = BuildGroupFn(); // default: no assignments for the UPN
|
||||
|
||||
var unassigned = await fn.UnassignCoreAsync(GroupId, Upn, "ga@vendor.example");
|
||||
|
||||
Assert.False(unassigned);
|
||||
VerifyNoAudit(audit);
|
||||
Assert.Empty(signalR.DisconnectedUsers);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task UnassignCore_Assigned_DisconnectsAndAuditsPerGroupTenant()
|
||||
{
|
||||
var (fn, repo, audit, signalR) = BuildGroupFn();
|
||||
repo.Setup(r => r.GetTenantGroupAsync(GroupId)).ReturnsAsync(new TenantGroup
|
||||
{
|
||||
GroupId = GroupId,
|
||||
Name = "MSP Customers",
|
||||
TenantIds = new List<string> { TenantA, TenantB },
|
||||
AssigneeCount = 1,
|
||||
});
|
||||
repo.Setup(r => r.GetGroupAssignmentsForUpnAsync(UpnLower))
|
||||
.ReturnsAsync(new List<TenantGroupAssignment> { new() { Upn = UpnLower, GroupId = GroupId } });
|
||||
repo.Setup(r => r.UnassignGroupAsync(UpnLower, GroupId)).ReturnsAsync(true);
|
||||
|
||||
var unassigned = await fn.UnassignCoreAsync(GroupId, Upn, "ga@vendor.example");
|
||||
|
||||
Assert.True(unassigned);
|
||||
Assert.Equal(new[] { UpnLower }, signalR.DisconnectedUsers);
|
||||
// Lost access is audited under EACH of the group's tenants (customer-visible trail).
|
||||
audit.Verify(a => a.LogAuditEntryAsync(
|
||||
TenantA, "DELETE", "DelegatedGroupAccess", UpnLower, "ga@vendor.example",
|
||||
It.IsAny<Dictionary<string, string>?>()), Times.Once);
|
||||
audit.Verify(a => a.LogAuditEntryAsync(
|
||||
TenantB, "DELETE", "DelegatedGroupAccess", UpnLower, "ga@vendor.example",
|
||||
It.IsAny<Dictionary<string, string>?>()), Times.Once);
|
||||
}
|
||||
|
||||
// --- Group delete: every (former) assignee's streams are cut ---
|
||||
|
||||
[Fact]
|
||||
public async Task DeleteGroupCore_DisconnectsEveryAssignee_AndAuditsPerTenant()
|
||||
{
|
||||
var (fn, repo, audit, signalR) = BuildGroupFn();
|
||||
var assignees = new List<TenantGroupAssignment>
|
||||
{
|
||||
new() { Upn = "one@partner.example", GroupId = GroupId },
|
||||
new() { Upn = "two@partner.example", GroupId = GroupId },
|
||||
};
|
||||
repo.Setup(r => r.GetTenantGroupAsync(GroupId)).ReturnsAsync(new TenantGroup
|
||||
{
|
||||
GroupId = GroupId,
|
||||
Name = "MSP Customers",
|
||||
TenantIds = new List<string> { TenantA },
|
||||
AssigneeCount = assignees.Count,
|
||||
});
|
||||
repo.Setup(r => r.GetGroupAssigneesAsync(GroupId)).ReturnsAsync(assignees);
|
||||
repo.Setup(r => r.DeleteTenantGroupAsync(GroupId)).ReturnsAsync(true);
|
||||
|
||||
await fn.DeleteGroupCoreAsync(GroupId, "ga@vendor.example");
|
||||
|
||||
Assert.Equal(new[] { "one@partner.example", "two@partner.example" }, signalR.DisconnectedUsers);
|
||||
audit.Verify(a => a.LogAuditEntryAsync(
|
||||
TenantA, "DELETE", "DelegatedGroupAccess", "*", "ga@vendor.example",
|
||||
It.IsAny<Dictionary<string, string>?>()), Times.Once);
|
||||
}
|
||||
}
|
||||
+41
-22
@@ -3,6 +3,7 @@ using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using System.Threading.Tasks;
|
||||
using Azure.Data.Tables;
|
||||
using AutopilotMonitor.Functions.Helpers;
|
||||
using AutopilotMonitor.Functions.Pagination;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using AutopilotMonitor.Shared;
|
||||
@@ -43,7 +44,9 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
|
||||
["UserId"] = Truncate(entry.UserId, 128),
|
||||
["Message"] = Truncate(entry.Message, 512),
|
||||
["Details"] = Truncate(entry.Details, 4096),
|
||||
["Timestamp"] = entry.Timestamp,
|
||||
// "Timestamp" is a reserved system property (supplied values are ignored and
|
||||
// any row rewrite resets it), so the event time needs its own column.
|
||||
[BusinessTimestamp.OccurredUtcColumn] = entry.Timestamp,
|
||||
};
|
||||
|
||||
await _table.UpsertEntityAsync(entity);
|
||||
@@ -168,15 +171,16 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
|
||||
if (!string.IsNullOrEmpty(lastRowKey))
|
||||
clauses.Add($"RowKey gt '{lastRowKey!.Replace("'", "''")}'");
|
||||
if (dateFrom.HasValue)
|
||||
clauses.Add($"Timestamp ge datetime'{ToUtc(dateFrom.Value):o}'");
|
||||
clauses.Add(BusinessTimestamp.OpsDateFromClause(ToUtc(dateFrom.Value)));
|
||||
if (dateTo.HasValue)
|
||||
clauses.Add($"Timestamp le datetime'{ToUtc(dateTo.Value):o}'");
|
||||
clauses.Add(BusinessTimestamp.OpsDateToClause(ToUtc(dateTo.Value)));
|
||||
return string.Join(" and ", clauses);
|
||||
}
|
||||
|
||||
// Filters on the system-managed Timestamp; user-defined "Timestamp" property
|
||||
// mirrors it on insert via SaveOpsEventAsync.
|
||||
private static string? BuildFilter(string? category, DateTime? dateFrom, DateTime? dateTo)
|
||||
// Date windows filter on the RowKey (reverse-tick, index-backed): the system
|
||||
// Timestamp is reset by storage migrations, and an OccurredUtc property filter
|
||||
// would exclude rows written before that column existed.
|
||||
internal static string? BuildFilter(string? category, DateTime? dateFrom, DateTime? dateTo)
|
||||
{
|
||||
var clauses = new List<string>();
|
||||
if (!string.IsNullOrEmpty(category))
|
||||
@@ -185,11 +189,11 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
|
||||
}
|
||||
if (dateFrom.HasValue)
|
||||
{
|
||||
clauses.Add($"Timestamp ge datetime'{ToUtc(dateFrom.Value):o}'");
|
||||
clauses.Add(BusinessTimestamp.OpsDateFromClause(ToUtc(dateFrom.Value)));
|
||||
}
|
||||
if (dateTo.HasValue)
|
||||
{
|
||||
clauses.Add($"Timestamp le datetime'{ToUtc(dateTo.Value):o}'");
|
||||
clauses.Add(BusinessTimestamp.OpsDateToClause(ToUtc(dateTo.Value)));
|
||||
}
|
||||
return clauses.Count == 0 ? null : string.Join(" and ", clauses);
|
||||
}
|
||||
@@ -201,27 +205,30 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
|
||||
{
|
||||
var deleted = 0;
|
||||
|
||||
await foreach (var entity in _table.QueryAsync<TableEntity>())
|
||||
// Server-side RowKey range instead of a full-table scan over the system
|
||||
// Timestamp: every ops RowKey is a reverse-tick, so age is index-derivable,
|
||||
// and the system Timestamp lies after storage migrations (it would freeze
|
||||
// deletions for one retention period, then drop the whole pre-migration
|
||||
// corpus in a single run).
|
||||
var filter = BusinessTimestamp.OpsRetentionClause(cutoff);
|
||||
await foreach (var entity in _table.QueryAsync<TableEntity>(
|
||||
filter: filter, select: new[] { "PartitionKey", "RowKey" }))
|
||||
{
|
||||
var timestamp = entity.GetDateTimeOffset("Timestamp")?.UtcDateTime ?? DateTime.MinValue;
|
||||
if (timestamp < cutoff)
|
||||
try
|
||||
{
|
||||
try
|
||||
{
|
||||
await _table.DeleteEntityAsync(entity.PartitionKey, entity.RowKey);
|
||||
deleted++;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to delete ops event {PK}/{RK}", entity.PartitionKey, entity.RowKey);
|
||||
}
|
||||
await _table.DeleteEntityAsync(entity.PartitionKey, entity.RowKey);
|
||||
deleted++;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to delete ops event {PK}/{RK}", entity.PartitionKey, entity.RowKey);
|
||||
}
|
||||
}
|
||||
|
||||
return deleted;
|
||||
}
|
||||
|
||||
private static OpsEventEntry MapToEntry(TableEntity entity)
|
||||
internal static OpsEventEntry MapToEntry(TableEntity entity)
|
||||
{
|
||||
return new OpsEventEntry
|
||||
{
|
||||
@@ -233,10 +240,22 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
|
||||
UserId = entity.GetString("UserId"),
|
||||
Message = entity.GetString("Message") ?? string.Empty,
|
||||
Details = entity.GetString("Details"),
|
||||
Timestamp = entity.GetDateTimeOffset("Timestamp")?.UtcDateTime ?? DateTime.MinValue,
|
||||
Timestamp = ResolveTimestamp(entity),
|
||||
};
|
||||
}
|
||||
|
||||
// Business time: OccurredUtc column → reverse-tick RowKey decode → system
|
||||
// Timestamp (last resort; a row rewrite — e.g. storage migration — resets it).
|
||||
internal static DateTime ResolveTimestamp(TableEntity entity)
|
||||
{
|
||||
var occurred = BusinessTimestamp.GetUtcDateTime(entity, BusinessTimestamp.OccurredUtcColumn);
|
||||
if (occurred.HasValue)
|
||||
return occurred.Value;
|
||||
if (BusinessTimestamp.TryDecodeOpsRowKey(entity.RowKey, out var decoded))
|
||||
return decoded;
|
||||
return BusinessTimestamp.GetUtcDateTime(entity, "Timestamp") ?? DateTime.MinValue;
|
||||
}
|
||||
|
||||
private static string? Truncate(string? value, int maxLength)
|
||||
{
|
||||
if (value == null) return null;
|
||||
|
||||
+103
@@ -0,0 +1,103 @@
|
||||
using System.Net;
|
||||
using AutopilotMonitor.Functions.Helpers;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using Microsoft.Azure.Functions.Worker;
|
||||
using Microsoft.Azure.Functions.Worker.Http;
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Functions.Admin
|
||||
{
|
||||
/// <summary>
|
||||
/// Global-Admin trigger for the one-time OccurredUtc backfill on AuditLogs / OpsEvents
|
||||
/// (business timestamps lost to the system-Timestamp reset of the 2026-07-18 storage
|
||||
/// migration; values are recovered from the reverse-tick RowKeys). DRY-RUN BY DEFAULT —
|
||||
/// a real run requires the explicit <c>dryRun=false</c> query parameter, so the operator
|
||||
/// always sees counts and samples before anything is written.
|
||||
/// </summary>
|
||||
public class BackfillOccurredUtcFunction
|
||||
{
|
||||
private readonly ILogger<BackfillOccurredUtcFunction> _logger;
|
||||
private readonly OccurredUtcBackfillService _service;
|
||||
|
||||
private const int DefaultMaxRows = 250;
|
||||
private const int MaxMaxRows = 1000;
|
||||
|
||||
public BackfillOccurredUtcFunction(
|
||||
ILogger<BackfillOccurredUtcFunction> logger,
|
||||
OccurredUtcBackfillService service)
|
||||
{
|
||||
_logger = logger;
|
||||
_service = service;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// POST /api/maintenance/backfill-occurred-utc
|
||||
/// Query parameters:
|
||||
/// - table: "audit" | "ops" (required).
|
||||
/// - dryRun: "false" to actually write; anything else (or absent) reports only.
|
||||
/// - maxRows: rows examined per invocation (default 250, max 1000). Follow
|
||||
/// nextContinuation to continue; null nextContinuation = table done.
|
||||
/// - continuation: opaque token from the previous response (URL-encode it).
|
||||
/// </summary>
|
||||
[Function("BackfillOccurredUtc")]
|
||||
public async Task<HttpResponseData> Run(
|
||||
[HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = "maintenance/backfill-occurred-utc")] HttpRequestData req,
|
||||
FunctionContext context)
|
||||
{
|
||||
// Authentication + GlobalAdminOnly authorization enforced by PolicyEnforcementMiddleware
|
||||
var userEmail = TenantHelper.GetUserIdentifier(req);
|
||||
|
||||
var table = (req.Query["table"] ?? string.Empty).Trim().ToLowerInvariant();
|
||||
if (table != OccurredUtcBackfillService.TableAudit && table != OccurredUtcBackfillService.TableOps)
|
||||
{
|
||||
var bad = req.CreateResponse(HttpStatusCode.BadRequest);
|
||||
await bad.WriteAsJsonAsync(new
|
||||
{
|
||||
error = $"table must be '{OccurredUtcBackfillService.TableAudit}' or '{OccurredUtcBackfillService.TableOps}'"
|
||||
});
|
||||
return bad;
|
||||
}
|
||||
|
||||
var dryRun = !string.Equals(req.Query["dryRun"], "false", StringComparison.OrdinalIgnoreCase);
|
||||
var continuation = req.Query["continuation"];
|
||||
|
||||
var maxRows = DefaultMaxRows;
|
||||
if (!string.IsNullOrEmpty(req.Query["maxRows"]))
|
||||
{
|
||||
if (!int.TryParse(req.Query["maxRows"], out maxRows) || maxRows < 1)
|
||||
{
|
||||
var bad = req.CreateResponse(HttpStatusCode.BadRequest);
|
||||
await bad.WriteAsJsonAsync(new { error = $"maxRows must be a positive integer (max {MaxMaxRows})" });
|
||||
return bad;
|
||||
}
|
||||
maxRows = Math.Min(maxRows, MaxMaxRows);
|
||||
}
|
||||
|
||||
_logger.LogInformation(
|
||||
"OccurredUtc backfill requested by {User}: table={Table} dryRun={DryRun} maxRows={MaxRows} hasContinuation={HasContinuation}",
|
||||
userEmail, table, dryRun, maxRows, !string.IsNullOrEmpty(continuation));
|
||||
|
||||
try
|
||||
{
|
||||
var result = await _service.RunAsync(table, dryRun, maxRows, continuation);
|
||||
|
||||
var response = req.CreateResponse(HttpStatusCode.OK);
|
||||
await response.WriteAsJsonAsync(new
|
||||
{
|
||||
success = true,
|
||||
result,
|
||||
triggeredBy = userEmail,
|
||||
triggeredAt = DateTime.UtcNow,
|
||||
});
|
||||
return response;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "OccurredUtc backfill run failed");
|
||||
var error = req.CreateResponse(HttpStatusCode.InternalServerError);
|
||||
await error.WriteAsJsonAsync(new { success = false, message = "Internal server error" });
|
||||
return error;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
+69
-32
@@ -64,19 +64,9 @@ public class DelegatedAdminManagementFunction
|
||||
var currentUpn = context.GetRequestContext().UserPrincipalName;
|
||||
|
||||
var body = await req.ReadFromJsonAsync<GrantDelegatedAdminRequest>();
|
||||
if (body == null || string.IsNullOrWhiteSpace(body.Upn) || string.IsNullOrWhiteSpace(body.TenantId))
|
||||
return await Bad(req, "upn and tenantId are required");
|
||||
|
||||
// Mirror the tenant-group path: a mistyped tenantId must not create a garbage scope entry
|
||||
// that pollutes the allow-list and seeds a per-tenant audit trail keyed on the typo.
|
||||
if (!Guid.TryParse(body.TenantId, out _))
|
||||
return await Bad(req, "a valid tenantId (GUID) is required");
|
||||
|
||||
// Default to the least-privileged role; reject anything we don't recognize (fail-closed) so a typo
|
||||
// can never silently widen access.
|
||||
var role = string.IsNullOrWhiteSpace(body.Role) ? Constants.DelegatedRoles.DelegatedReader : body.Role;
|
||||
if (role != Constants.DelegatedRoles.DelegatedReader && role != Constants.DelegatedRoles.DelegatedAdmin)
|
||||
return await Bad(req, $"role must be '{Constants.DelegatedRoles.DelegatedReader}' or '{Constants.DelegatedRoles.DelegatedAdmin}'");
|
||||
var validationError = ValidateGrantRequest(body, out var role);
|
||||
if (validationError != null)
|
||||
return await Bad(req, validationError);
|
||||
|
||||
// Edition note: TARGET tenants may be any edition — an MSP on Enterprise may manage Community
|
||||
// customers. The Enterprise requirement applies to the delegated admin's HOME tenant and is
|
||||
@@ -84,7 +74,7 @@ public class DelegatedAdminManagementFunction
|
||||
// be checked here: at grant time only the UPN is known, and UPN-domain → tenant mapping is not
|
||||
// reliable (multi-domain tenants). Grants to non-Enterprise-homed admins are simply inert.
|
||||
var entry = await _delegatedAdminService.UpsertAsync(
|
||||
body.Upn, body.TenantId, role,
|
||||
body!.Upn, body.TenantId, role,
|
||||
Constants.DelegatedStatus.Active, Constants.DelegatedSource.OperatorGranted, currentUpn ?? "");
|
||||
|
||||
await _maintenanceRepo.LogAuditEntryAsync(
|
||||
@@ -112,10 +102,7 @@ public class DelegatedAdminManagementFunction
|
||||
string upn, string tenantId, FunctionContext context)
|
||||
{
|
||||
var currentUpn = context.GetRequestContext().UserPrincipalName;
|
||||
// The service reports whether a row was actually deleted — a mistyped upn/tenantId must not
|
||||
// 200 and write a false customer-visible "access removed" audit row while the real grant
|
||||
// stays live (mirrors the tenant-group revoke semantics).
|
||||
var removed = await _delegatedAdminService.RemoveAsync(upn, tenantId);
|
||||
var removed = await RevokeCoreAsync(upn, tenantId, currentUpn);
|
||||
if (!removed)
|
||||
{
|
||||
var notFound = req.CreateResponse(HttpStatusCode.NotFound);
|
||||
@@ -123,20 +110,34 @@ public class DelegatedAdminManagementFunction
|
||||
return notFound;
|
||||
}
|
||||
|
||||
await _maintenanceRepo.LogAuditEntryAsync(
|
||||
tenantId.ToLowerInvariant(), "DELETE", "DelegatedAdmin", upn.ToLowerInvariant(), currentUpn ?? "");
|
||||
|
||||
// Enforcement: cut the revoked caller's already-joined live streams — group authorization is
|
||||
// join-time only, so without this they keep receiving tenant telemetry until the connection drops.
|
||||
await _signalRService.DisconnectUserAsync(upn.ToLowerInvariant());
|
||||
|
||||
_logger.LogInformation("Delegated revoke: {Upn} -> {TenantId} by {By}", upn, tenantId, currentUpn);
|
||||
|
||||
var response = req.CreateResponse(HttpStatusCode.OK);
|
||||
await response.WriteAsJsonAsync(new { message = "Delegated assignment revoked" });
|
||||
return response;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Testable core of the revoke flow (no HTTP plumbing — see GetAllBlockedDevicesFunctionTests for the
|
||||
/// convention). Returns false when no assignment row existed (caller 404s): a mistyped upn/tenantId
|
||||
/// must not 200 and write a false customer-visible "access removed" audit row while the real grant
|
||||
/// stays live (mirrors the tenant-group revoke semantics). On a real removal writes the audit entry
|
||||
/// and cuts the revoked UPN's live streams — group authorization is join-time only, so without the
|
||||
/// kick they keep receiving tenant telemetry until the connection drops.
|
||||
/// </summary>
|
||||
internal async Task<bool> RevokeCoreAsync(string upn, string tenantId, string? currentUpn)
|
||||
{
|
||||
var removed = await _delegatedAdminService.RemoveAsync(upn, tenantId);
|
||||
if (!removed)
|
||||
return false;
|
||||
|
||||
await _maintenanceRepo.LogAuditEntryAsync(
|
||||
tenantId.ToLowerInvariant(), "DELETE", "DelegatedAdmin", upn.ToLowerInvariant(), currentUpn ?? "");
|
||||
|
||||
await _signalRService.DisconnectUserAsync(upn.ToLowerInvariant());
|
||||
|
||||
_logger.LogInformation("Delegated revoke: {Upn} -> {TenantId} by {By}", upn, tenantId, currentUpn);
|
||||
return true;
|
||||
}
|
||||
|
||||
/// <summary>PATCH /api/global/delegated-admins/{upn}/{tenantId}/enable — re-enable a disabled assignment. GlobalAdminOnly.</summary>
|
||||
[Function("EnableDelegatedAdmin")]
|
||||
[Authorize]
|
||||
@@ -157,7 +158,7 @@ public class DelegatedAdminManagementFunction
|
||||
HttpRequestData req, string upn, string tenantId, bool isEnabled, FunctionContext context)
|
||||
{
|
||||
var currentUpn = context.GetRequestContext().UserPrincipalName;
|
||||
var ok = await _delegatedAdminService.SetEnabledAsync(upn, tenantId, isEnabled);
|
||||
var ok = await SetEnabledCoreAsync(upn, tenantId, isEnabled, currentUpn);
|
||||
if (!ok)
|
||||
{
|
||||
var notFound = req.CreateResponse(HttpStatusCode.NotFound);
|
||||
@@ -165,20 +166,56 @@ public class DelegatedAdminManagementFunction
|
||||
return notFound;
|
||||
}
|
||||
|
||||
var response = req.CreateResponse(HttpStatusCode.OK);
|
||||
await response.WriteAsJsonAsync(new { message = isEnabled ? "Delegated assignment enabled" : "Delegated assignment disabled" });
|
||||
return response;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Testable core of enable/disable (no HTTP plumbing). Returns false when no assignment row
|
||||
/// existed (caller 404s — no audit, no disconnect). Disable is a revocation too, so it cuts the
|
||||
/// UPN's already-joined live streams (see <see cref="RevokeCoreAsync"/>); enable never disconnects.
|
||||
/// </summary>
|
||||
internal async Task<bool> SetEnabledCoreAsync(string upn, string tenantId, bool isEnabled, string? currentUpn)
|
||||
{
|
||||
var ok = await _delegatedAdminService.SetEnabledAsync(upn, tenantId, isEnabled);
|
||||
if (!ok)
|
||||
return false;
|
||||
|
||||
await _maintenanceRepo.LogAuditEntryAsync(
|
||||
tenantId.ToLowerInvariant(), "UPDATE", "DelegatedAdmin", upn.ToLowerInvariant(), currentUpn ?? "",
|
||||
new Dictionary<string, string> { { "IsEnabled", isEnabled.ToString() } });
|
||||
|
||||
// Disable is a revocation too — cut already-joined live streams (see RevokeDelegatedAdmin).
|
||||
if (!isEnabled)
|
||||
await _signalRService.DisconnectUserAsync(upn.ToLowerInvariant());
|
||||
|
||||
_logger.LogInformation("Delegated {Action}: {Upn} -> {TenantId} by {By}",
|
||||
isEnabled ? "enable" : "disable", upn, tenantId, currentUpn);
|
||||
return true;
|
||||
}
|
||||
|
||||
var response = req.CreateResponse(HttpStatusCode.OK);
|
||||
await response.WriteAsJsonAsync(new { message = isEnabled ? "Delegated assignment enabled" : "Delegated assignment disabled" });
|
||||
return response;
|
||||
/// <summary>
|
||||
/// Pure grant-request validation (testable seam). Returns the 400 error message, or null when valid —
|
||||
/// then <paramref name="role"/> carries the effective role. A mistyped tenantId must not create a
|
||||
/// garbage scope entry that pollutes the allow-list and seeds a per-tenant audit trail keyed on the
|
||||
/// typo; the role defaults to least-privileged and unknown roles are rejected (fail-closed) so a typo
|
||||
/// can never silently widen access.
|
||||
/// </summary>
|
||||
internal static string? ValidateGrantRequest(GrantDelegatedAdminRequest? body, out string role)
|
||||
{
|
||||
role = Constants.DelegatedRoles.DelegatedReader;
|
||||
|
||||
if (body == null || string.IsNullOrWhiteSpace(body.Upn) || string.IsNullOrWhiteSpace(body.TenantId))
|
||||
return "upn and tenantId are required";
|
||||
|
||||
if (!Guid.TryParse(body.TenantId, out _))
|
||||
return "a valid tenantId (GUID) is required";
|
||||
|
||||
role = string.IsNullOrWhiteSpace(body.Role) ? Constants.DelegatedRoles.DelegatedReader : body.Role!;
|
||||
if (role != Constants.DelegatedRoles.DelegatedReader && role != Constants.DelegatedRoles.DelegatedAdmin)
|
||||
return $"role must be '{Constants.DelegatedRoles.DelegatedReader}' or '{Constants.DelegatedRoles.DelegatedAdmin}'";
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
private static async Task<HttpResponseData> Bad(HttpRequestData req, string error)
|
||||
|
||||
+34
-16
@@ -112,15 +112,26 @@ public class TenantGroupManagementFunction
|
||||
string groupId, FunctionContext context)
|
||||
{
|
||||
var currentUpn = context.GetRequestContext().UserPrincipalName;
|
||||
await DeleteGroupCoreAsync(groupId, currentUpn);
|
||||
|
||||
// Capture tenants + assignees BEFORE the cascade delete removes them, so we can audit the
|
||||
// bulk access removal under each affected tenant and cut every assignee's live streams.
|
||||
var response = req.CreateResponse(HttpStatusCode.OK);
|
||||
await response.WriteAsJsonAsync(new { message = "Group deleted" });
|
||||
return response;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Testable core of the group cascade delete (no HTTP plumbing). Captures tenants + assignees
|
||||
/// BEFORE the delete removes them, so the bulk access removal is audited under each affected
|
||||
/// tenant and EVERY (former) assignee's live streams are cut — group authorization is join-time
|
||||
/// only, so without the kick they keep receiving the managed tenants' telemetry until their
|
||||
/// connection drops.
|
||||
/// </summary>
|
||||
internal async Task DeleteGroupCoreAsync(string groupId, string? currentUpn)
|
||||
{
|
||||
var group = await _delegatedAdminService.GetGroupAsync(groupId);
|
||||
var assignees = await _delegatedAdminService.GetGroupAssigneesAsync(groupId);
|
||||
await _delegatedAdminService.DeleteGroupAsync(groupId);
|
||||
|
||||
// Enforcement: group authorization is join-time only — without the kick, every (former)
|
||||
// assignee keeps receiving the managed tenants' live telemetry until their connection drops.
|
||||
foreach (var assignee in assignees)
|
||||
await _signalRService.DisconnectUserAsync(assignee.Upn);
|
||||
|
||||
@@ -137,10 +148,6 @@ public class TenantGroupManagementFunction
|
||||
}
|
||||
|
||||
_logger.LogInformation("Tenant group deleted: {GroupId} by {By}", groupId, currentUpn);
|
||||
|
||||
var response = req.CreateResponse(HttpStatusCode.OK);
|
||||
await response.WriteAsJsonAsync(new { message = "Group deleted" });
|
||||
return response;
|
||||
}
|
||||
|
||||
/// <summary>POST /api/global/tenant-groups/{groupId}/tenants — add a tenant. GlobalAdminOnly. Body: { "tenantId": "<guid>" }.</summary>
|
||||
@@ -271,17 +278,31 @@ public class TenantGroupManagementFunction
|
||||
string groupId, string upn, FunctionContext context)
|
||||
{
|
||||
var currentUpn = context.GetRequestContext().UserPrincipalName;
|
||||
var unassigned = await UnassignCoreAsync(groupId, upn, currentUpn);
|
||||
if (!unassigned)
|
||||
return await NotFound(req);
|
||||
|
||||
var response = req.CreateResponse(HttpStatusCode.OK);
|
||||
await response.WriteAsJsonAsync(new { message = "Unassigned from group" });
|
||||
return response;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Testable core of the unassign flow (no HTTP plumbing). Returns false when the UPN was not
|
||||
/// actually assigned (caller 404s) — so a mistyped UPN can't 200 and write false "access removed"
|
||||
/// audit rows, and no disconnect fires. On a real unassign cuts the UPN's already-joined live
|
||||
/// streams (join-time-only authz) and audits the lost access under each of the group's tenants.
|
||||
/// </summary>
|
||||
internal async Task<bool> UnassignCoreAsync(string groupId, string upn, string? currentUpn)
|
||||
{
|
||||
var normalizedUpn = upn.ToLowerInvariant();
|
||||
|
||||
// Read the group's tenants (unchanged by unassign) so we can audit the UPN's lost access per tenant.
|
||||
var group = await _delegatedAdminService.GetGroupAsync(groupId);
|
||||
// The service returns false when the UPN was not actually assigned — so a mistyped UPN can't 200 and
|
||||
// write false "access removed" audit rows. Only audit after a real unassign.
|
||||
var unassigned = await _delegatedAdminService.UnassignGroupAsync(normalizedUpn, groupId);
|
||||
if (!unassigned)
|
||||
return await NotFound(req);
|
||||
return false;
|
||||
|
||||
// Enforcement: cut the unassigned caller's already-joined live streams (join-time-only authz).
|
||||
await _signalRService.DisconnectUserAsync(normalizedUpn);
|
||||
|
||||
if (group != null)
|
||||
@@ -296,10 +317,7 @@ public class TenantGroupManagementFunction
|
||||
}
|
||||
|
||||
_logger.LogInformation("Group {GroupId} unassigned from {Upn} by {By}", groupId, normalizedUpn, currentUpn);
|
||||
|
||||
var response = req.CreateResponse(HttpStatusCode.OK);
|
||||
await response.WriteAsJsonAsync(new { message = "Unassigned from group" });
|
||||
return response;
|
||||
return true;
|
||||
}
|
||||
|
||||
/// <summary>Logs one audit entry per tenant (the customer-visible "who can read my tenant" trail).</summary>
|
||||
|
||||
+41
-3
@@ -10,6 +10,7 @@ using AutopilotMonitor.Functions.Services;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using AutopilotMonitor.Shared.Pagination;
|
||||
using Microsoft.Azure.Functions.Worker;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Azure.Functions.Worker.Http;
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
@@ -17,15 +18,23 @@ namespace AutopilotMonitor.Functions.Functions.Config
|
||||
{
|
||||
public class GetAllTenantConfigurationsFunction
|
||||
{
|
||||
// TTL for the "this managed id has NO config row in any casing" verdict a rescue scan proved.
|
||||
// Bounds how long a tenant onboarded moments after that scan stays absent from a delegated
|
||||
// caller's config/all (matches the positive config cache TTL in TenantConfigurationService).
|
||||
private static readonly TimeSpan MissingConfigNegativeCacheDuration = TimeSpan.FromMinutes(5);
|
||||
|
||||
private readonly ILogger<GetAllTenantConfigurationsFunction> _logger;
|
||||
private readonly TenantConfigurationService _configService;
|
||||
private readonly IMemoryCache _cache;
|
||||
|
||||
public GetAllTenantConfigurationsFunction(
|
||||
ILogger<GetAllTenantConfigurationsFunction> logger,
|
||||
TenantConfigurationService configService)
|
||||
TenantConfigurationService configService,
|
||||
IMemoryCache cache)
|
||||
{
|
||||
_logger = logger;
|
||||
_configService = configService;
|
||||
_cache = cache;
|
||||
}
|
||||
|
||||
[Function("GetAllTenantConfigurations")]
|
||||
@@ -73,9 +82,21 @@ namespace AutopilotMonitor.Functions.Functions.Config
|
||||
// (agent paths auto-create rows with the caller-supplied id verbatim) — the same caveat
|
||||
// the delegated sessions fan-out handles case-insensitively. A point-read is exact-case,
|
||||
// so any id it missed gets ONE scan+filter rescue pass (the pre-point-read behavior).
|
||||
// Only stale grants or case-variant rows pay for it; the common path stays read-bounded.
|
||||
// An id a previous scan already proved missing in ANY casing is negative-cached and
|
||||
// skipped for the TTL — otherwise a single stale grant (offboarded / never-onboarded
|
||||
// tenant) would make EVERY config/all call pay a full table scan.
|
||||
if (subset.Count < managedIds.Count)
|
||||
subset = MergeRescuedConfigs(managedIds, subset, await _configService.GetAllConfigurationsAsync());
|
||||
{
|
||||
var unproven = MissingManagedIds(managedIds, subset)
|
||||
.Where(id => !_cache.TryGetValue(MissingConfigCacheKey(id), out _))
|
||||
.ToList();
|
||||
if (unproven.Count > 0)
|
||||
{
|
||||
subset = MergeRescuedConfigs(managedIds, subset, await _configService.GetAllConfigurationsAsync());
|
||||
foreach (var id in MissingManagedIds(unproven, subset))
|
||||
_cache.Set(MissingConfigCacheKey(id), true, MissingConfigNegativeCacheDuration);
|
||||
}
|
||||
}
|
||||
_logger.LogInformation("GetAllTenantConfigurations (delegated subset, {Count} tenants) by {User}",
|
||||
subset.Count, userIdentifier);
|
||||
|
||||
@@ -177,6 +198,23 @@ namespace AutopilotMonitor.Functions.Functions.Config
|
||||
IEnumerable<TenantConfiguration?> reads)
|
||||
=> reads.Where(c => c != null).Select(c => c!).ToList();
|
||||
|
||||
/// <summary>
|
||||
/// The managed ids not present (case-insensitively) in <paramref name="configs"/> — before the
|
||||
/// rescue scan these are the point-read misses, after it the proven-missing ids to negative-cache.
|
||||
/// Pure + testable.
|
||||
/// </summary>
|
||||
internal static List<string> MissingManagedIds(
|
||||
IEnumerable<string> managedIds, IReadOnlyCollection<TenantConfiguration> configs)
|
||||
{
|
||||
var have = new HashSet<string>(configs.Select(c => c.TenantId), StringComparer.OrdinalIgnoreCase);
|
||||
return managedIds.Where(id => !have.Contains(id)).ToList();
|
||||
}
|
||||
|
||||
/// <summary>Negative-cache key for a managed id a rescue scan proved has no config row.
|
||||
/// Lowercased so case variants of the same grant share one verdict.</summary>
|
||||
internal static string MissingConfigCacheKey(string tenantId)
|
||||
=> $"tenant-config-missing:{tenantId.ToLowerInvariant()}";
|
||||
|
||||
/// <summary>
|
||||
/// Case-variant rescue for managed ids the exact-case point-reads missed: from the full config scan,
|
||||
/// add every config whose TenantId matches a managed id case-insensitively and is not already in the
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
using System;
|
||||
using System.Globalization;
|
||||
using Azure.Data.Tables;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Helpers
|
||||
{
|
||||
/// <summary>
|
||||
/// Business-timestamp handling for tables whose rows historically relied on the
|
||||
/// Azure Tables system <c>Timestamp</c> (= row write time). That system property is
|
||||
/// server-managed: supplied values are ignored on write and every row rewrite
|
||||
/// (storage migration, merge) resets it. The authoritative event time therefore
|
||||
/// lives in (a) the explicit <see cref="OccurredUtcColumn"/> custom property on
|
||||
/// rows written after its introduction, and (b) the time encoded in the RowKey
|
||||
/// for older rows. Readers resolve OccurredUtc → RowKey decode → system Timestamp.
|
||||
///
|
||||
/// RowKey schemes decoded here:
|
||||
/// AuditLogs: "!{MaxTicks - ticks:D19}_{guid:N}" (legacy pre-2026-05 rows are bare GUIDs — not decodable)
|
||||
/// OpsEvents: "{MaxTicks - ticks:D19}"
|
||||
/// Events: "{timestamp:yyyyMMddHHmmssfff}_{sequence:D10}"
|
||||
/// </summary>
|
||||
internal static class BusinessTimestamp
|
||||
{
|
||||
/// <summary>Custom column carrying the business event time. Survives migrations verbatim
|
||||
/// (unlike the system Timestamp, which any row rewrite resets).</summary>
|
||||
internal const string OccurredUtcColumn = "OccurredUtc";
|
||||
|
||||
internal const string AuditRowKeyPrefix = "!";
|
||||
|
||||
/// <summary>
|
||||
/// '"' (0x22) is the character immediately after '!' (0x21). `RowKey lt '"'` confines a
|
||||
/// range to time-encoded audit rows: legacy bare-GUID RowKeys start with a hex digit
|
||||
/// (≥ '0', 0x30) and would otherwise match ANY lower bound of the form `RowKey ge '!…'`.
|
||||
/// </summary>
|
||||
internal const string AuditTimeEncodedUpperBound = "\"";
|
||||
|
||||
private const string EventRowKeyTimestampFormat = "yyyyMMddHHmmssfff";
|
||||
|
||||
/// <summary>
|
||||
/// Reads a datetime column as UTC regardless of the SDK's materialized type —
|
||||
/// service reads surface Edm.DateTime as either DateTimeOffset or DateTime
|
||||
/// depending on payload shape (the pre-existing mappers already read both).
|
||||
/// Unspecified kinds are treated as UTC (all stored values are UTC).
|
||||
/// </summary>
|
||||
internal static DateTime? GetUtcDateTime(TableEntity entity, string column)
|
||||
{
|
||||
if (!entity.TryGetValue(column, out var value) || value == null)
|
||||
return null;
|
||||
return value switch
|
||||
{
|
||||
DateTimeOffset dto => dto.UtcDateTime,
|
||||
DateTime dt => dt.Kind == DateTimeKind.Local
|
||||
? dt.ToUniversalTime()
|
||||
: DateTime.SpecifyKind(dt, DateTimeKind.Utc),
|
||||
_ => null,
|
||||
};
|
||||
}
|
||||
|
||||
// ===== RowKey decoders =====
|
||||
|
||||
/// <summary>Decodes an audit RowKey ("!{rev:D19}_{guid:N}") to the original UTC write time.
|
||||
/// Returns false for legacy bare-GUID RowKeys and any other foreign format.</summary>
|
||||
internal static bool TryDecodeAuditRowKey(string? rowKey, out DateTime utc)
|
||||
{
|
||||
utc = default;
|
||||
if (rowKey == null || rowKey.Length < 21 || rowKey[0] != '!')
|
||||
return false;
|
||||
return TryDecodeReverseTicks(rowKey.AsSpan(1, 19), out utc);
|
||||
}
|
||||
|
||||
/// <summary>Decodes an ops-event RowKey (bare "{rev:D19}") to the original UTC write time.</summary>
|
||||
internal static bool TryDecodeOpsRowKey(string? rowKey, out DateTime utc)
|
||||
{
|
||||
utc = default;
|
||||
if (rowKey == null || rowKey.Length != 19)
|
||||
return false;
|
||||
return TryDecodeReverseTicks(rowKey.AsSpan(), out utc);
|
||||
}
|
||||
|
||||
/// <summary>Decodes an enrollment-event RowKey ("{yyyyMMddHHmmssfff}_{seq:D10}") to the
|
||||
/// (sanitized) agent event time, millisecond precision, Kind=Utc.</summary>
|
||||
internal static bool TryDecodeEventRowKeyPrefix(string? rowKey, out DateTime utc)
|
||||
{
|
||||
utc = default;
|
||||
if (rowKey == null)
|
||||
return false;
|
||||
var separator = rowKey.IndexOf('_');
|
||||
if (separator != 17)
|
||||
return false;
|
||||
if (!DateTime.TryParseExact(rowKey.Substring(0, 17), EventRowKeyTimestampFormat,
|
||||
CultureInfo.InvariantCulture,
|
||||
DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal, out utc))
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
private static bool TryDecodeReverseTicks(ReadOnlySpan<char> revTickDigits, out DateTime utc)
|
||||
{
|
||||
utc = default;
|
||||
foreach (var c in revTickDigits)
|
||||
{
|
||||
if (c < '0' || c > '9')
|
||||
return false;
|
||||
}
|
||||
if (!long.TryParse(revTickDigits, NumberStyles.None, CultureInfo.InvariantCulture, out var rev))
|
||||
return false;
|
||||
var ticks = DateTime.MaxValue.Ticks - rev;
|
||||
if (ticks < DateTime.MinValue.Ticks || ticks > DateTime.MaxValue.Ticks)
|
||||
return false;
|
||||
utc = new DateTime(ticks, DateTimeKind.Utc);
|
||||
return true;
|
||||
}
|
||||
|
||||
// ===== OData date-window clauses (RowKey ranges) =====
|
||||
// Date windows filter on the RowKey rather than a property: rows missing a property are
|
||||
// excluded from OData comparisons (so an OccurredUtc filter would drop every not-yet-
|
||||
// backfilled row), and RowKey ranges are index-backed. String order equals numeric order
|
||||
// because the reverse-tick is fixed-width D19. rev(t) = MaxTicks - t.Ticks, so larger
|
||||
// timestamps have lexically smaller RowKeys (newest-first).
|
||||
|
||||
private static long Rev(DateTime utc) => DateTime.MaxValue.Ticks - utc.Ticks;
|
||||
|
||||
/// <summary>Audit rows with business time ≥ <paramref name="fromUtc"/> (inclusive):
|
||||
/// rev ≤ rev(from) < rev(from)+1. The exclusive upper bound also excludes all legacy
|
||||
/// GUID rows (they sort after every '!' row).</summary>
|
||||
internal static string AuditDateFromClause(DateTime fromUtc)
|
||||
=> $"RowKey lt '{AuditRowKeyPrefix}{Rev(fromUtc) + 1:D19}'";
|
||||
|
||||
/// <summary>Audit rows with business time ≤ <paramref name="toUtc"/> (inclusive):
|
||||
/// rev ≥ rev(to). The lower bound alone would match every legacy GUID row, hence the
|
||||
/// mandatory time-encoded guard.</summary>
|
||||
internal static string AuditDateToClause(DateTime toUtc)
|
||||
=> $"RowKey ge '{AuditRowKeyPrefix}{Rev(toUtc):D19}' and RowKey lt '{AuditTimeEncodedUpperBound}'";
|
||||
|
||||
/// <summary>Audit rows strictly older than <paramref name="cutoffUtc"/> (retention):
|
||||
/// rev ≥ rev(cutoff)+1 ⇔ ts < cutoff. MUST keep the time-encoded guard — without it
|
||||
/// this clause matches (and a retention sweep would delete) every legacy GUID row.</summary>
|
||||
internal static string AuditRetentionClause(DateTime cutoffUtc)
|
||||
=> $"RowKey ge '{AuditRowKeyPrefix}{Rev(cutoffUtc) + 1:D19}' and RowKey lt '{AuditTimeEncodedUpperBound}'";
|
||||
|
||||
/// <summary>Ops rows with business time ≥ <paramref name="fromUtc"/> (inclusive). Bare D19
|
||||
/// RowKey (no suffix), so `le` is exact.</summary>
|
||||
internal static string OpsDateFromClause(DateTime fromUtc)
|
||||
=> $"RowKey le '{Rev(fromUtc):D19}'";
|
||||
|
||||
/// <summary>Ops rows with business time ≤ <paramref name="toUtc"/> (inclusive).</summary>
|
||||
internal static string OpsDateToClause(DateTime toUtc)
|
||||
=> $"RowKey ge '{Rev(toUtc):D19}'";
|
||||
|
||||
/// <summary>Ops rows strictly older than <paramref name="cutoffUtc"/> (retention):
|
||||
/// `gt` not `ge` — a row at exactly the cutoff is kept, matching `Timestamp lt` semantics.</summary>
|
||||
internal static string OpsRetentionClause(DateTime cutoffUtc)
|
||||
=> $"RowKey gt '{Rev(cutoffUtc):D19}'";
|
||||
}
|
||||
}
|
||||
@@ -252,6 +252,7 @@ builder.Services.AddHttpClient<GitHubRuleRepository>()
|
||||
.AddPolicyHandler((sp, _) => sp.GetRequiredService<ResiliencePolicies>().ExternalDataApi);
|
||||
builder.Services.AddSingleton<MaintenanceService>();
|
||||
builder.Services.AddSingleton<LegacyReclassificationService>();
|
||||
builder.Services.AddSingleton<OccurredUtcBackfillService>();
|
||||
builder.Services.AddSingleton<
|
||||
AutopilotMonitor.Functions.Services.Monitoring.IAzureMonitorMetricsReader,
|
||||
AutopilotMonitor.Functions.Services.Monitoring.AzureMonitorMetricsReader>();
|
||||
|
||||
@@ -399,6 +399,9 @@ public static class EndpointAccessPolicyCatalog
|
||||
// One-time retro-reconcile of misclassified historical sessions (audit 2026-07-16) —
|
||||
// mutating, dry-run by default, GA only.
|
||||
new("POST", "maintenance/reclassify-legacy", EndpointPolicy.GlobalAdminOnly),
|
||||
// One-time OccurredUtc backfill on AuditLogs/OpsEvents (2026-07-18 migration reset
|
||||
// the system Timestamps) — mutating, dry-run by default, GA only.
|
||||
new("POST", "maintenance/backfill-occurred-utc", EndpointPolicy.GlobalAdminOnly),
|
||||
|
||||
// ── Critical-Table Backup (plan §PR1+§PR2) ─────────────────────────────
|
||||
new("POST", "global/backups/trigger", EndpointPolicy.GlobalAdminOnly),
|
||||
|
||||
@@ -0,0 +1,169 @@
|
||||
using Azure;
|
||||
using Azure.Data.Tables;
|
||||
using AutopilotMonitor.Functions.Helpers;
|
||||
using AutopilotMonitor.Functions.Pagination;
|
||||
using AutopilotMonitor.Shared;
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Services
|
||||
{
|
||||
/// <summary>
|
||||
/// One-time backfill of the <c>OccurredUtc</c> business-timestamp column on
|
||||
/// AuditLogs and OpsEvents rows written before the column existed. The value is
|
||||
/// decoded deterministically from the reverse-tick RowKey, so re-runs are
|
||||
/// idempotent and the write is a pure MERGE that adds exactly one property
|
||||
/// (nothing existing is modified or deleted). Audit legacy bare-GUID rows carry
|
||||
/// no recoverable time and are excluded server-side by the scan filter.
|
||||
/// Readers do not depend on this backfill (they decode the RowKey on the fly);
|
||||
/// its value is future-migration robustness and raw-query readability.
|
||||
/// </summary>
|
||||
public class OccurredUtcBackfillService
|
||||
{
|
||||
public const string TableAudit = "audit";
|
||||
public const string TableOps = "ops";
|
||||
|
||||
private const int MaxSamplesInResponse = 100;
|
||||
private const int TransactionChunkSize = 100;
|
||||
|
||||
private readonly TableStorageService _storage;
|
||||
private readonly ILogger<OccurredUtcBackfillService> _logger;
|
||||
|
||||
public OccurredUtcBackfillService(TableStorageService storage, ILogger<OccurredUtcBackfillService> logger)
|
||||
{
|
||||
_storage = storage;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
public class BackfillResult
|
||||
{
|
||||
public string Table { get; set; } = string.Empty;
|
||||
public bool DryRun { get; set; }
|
||||
public int RowsExamined { get; set; }
|
||||
public int WouldWrite { get; set; }
|
||||
public int Written { get; set; }
|
||||
public int SkippedAlreadySet { get; set; }
|
||||
public int SkippedUndecodable { get; set; }
|
||||
public int Errors { get; set; }
|
||||
public string? NextContinuation { get; set; }
|
||||
public List<BackfillSample> Samples { get; set; } = new();
|
||||
}
|
||||
|
||||
public class BackfillSample
|
||||
{
|
||||
public string PartitionKey { get; set; } = string.Empty;
|
||||
public string RowKey { get; set; } = string.Empty;
|
||||
public DateTime DecodedUtc { get; set; }
|
||||
}
|
||||
|
||||
internal enum RowDecision { AlreadySet, Write, Undecodable }
|
||||
|
||||
/// <summary>Pure per-row decision, extracted for unit testing.</summary>
|
||||
internal static RowDecision DecideRow(string table, TableEntity entity, out DateTime occurredUtc)
|
||||
{
|
||||
occurredUtc = default;
|
||||
if (BusinessTimestamp.GetUtcDateTime(entity, BusinessTimestamp.OccurredUtcColumn).HasValue)
|
||||
return RowDecision.AlreadySet;
|
||||
|
||||
var decoded = table == TableAudit
|
||||
? BusinessTimestamp.TryDecodeAuditRowKey(entity.RowKey, out occurredUtc)
|
||||
: BusinessTimestamp.TryDecodeOpsRowKey(entity.RowKey, out occurredUtc);
|
||||
return decoded ? RowDecision.Write : RowDecision.Undecodable;
|
||||
}
|
||||
|
||||
public async Task<BackfillResult> RunAsync(string table, bool dryRun, int maxRows, string? continuation)
|
||||
{
|
||||
var (tableName, scanFilter) = table switch
|
||||
{
|
||||
// Audit: server-side restriction to time-encoded '!'-rows — legacy bare-GUID
|
||||
// rows (which sort after '"') have nothing to decode and are never touched.
|
||||
TableAudit => (Constants.TableNames.AuditLogs,
|
||||
$"RowKey ge '{BusinessTimestamp.AuditRowKeyPrefix}' and RowKey lt '{BusinessTimestamp.AuditTimeEncodedUpperBound}'"),
|
||||
TableOps => (Constants.TableNames.OpsEvents, (string?)null),
|
||||
_ => throw new ArgumentException($"Unknown table '{table}'", nameof(table)),
|
||||
};
|
||||
|
||||
var tableClient = _storage.GetTableClient(tableName);
|
||||
var result = new BackfillResult { Table = table, DryRun = dryRun };
|
||||
|
||||
var (entities, nextToken) = await AzureTablesPaginator.FetchPageAsync<TableEntity>(
|
||||
client: tableClient,
|
||||
filter: scanFilter,
|
||||
pageSize: maxRows,
|
||||
continuation: continuation,
|
||||
select: new[] { "PartitionKey", "RowKey", BusinessTimestamp.OccurredUtcColumn });
|
||||
result.NextContinuation = nextToken;
|
||||
|
||||
var pending = new List<(string PartitionKey, string RowKey, DateTime OccurredUtc)>();
|
||||
foreach (var entity in entities)
|
||||
{
|
||||
result.RowsExamined++;
|
||||
switch (DecideRow(table, entity, out var occurredUtc))
|
||||
{
|
||||
case RowDecision.AlreadySet:
|
||||
result.SkippedAlreadySet++;
|
||||
break;
|
||||
case RowDecision.Undecodable:
|
||||
result.SkippedUndecodable++;
|
||||
_logger.LogWarning("OccurredUtc backfill: undecodable RowKey {PK}/{RK} in {Table}",
|
||||
entity.PartitionKey, entity.RowKey, tableName);
|
||||
break;
|
||||
case RowDecision.Write:
|
||||
result.WouldWrite++;
|
||||
pending.Add((entity.PartitionKey, entity.RowKey, occurredUtc));
|
||||
if (result.Samples.Count < MaxSamplesInResponse)
|
||||
{
|
||||
result.Samples.Add(new BackfillSample
|
||||
{
|
||||
PartitionKey = entity.PartitionKey,
|
||||
RowKey = entity.RowKey,
|
||||
DecodedUtc = occurredUtc,
|
||||
});
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (!dryRun && pending.Count > 0)
|
||||
result.Written = await WritePendingAsync(tableClient, pending, result);
|
||||
|
||||
_logger.LogInformation(
|
||||
"OccurredUtc backfill {Table}: dryRun={DryRun} examined={Examined} wouldWrite={WouldWrite} written={Written} alreadySet={AlreadySet} undecodable={Undecodable} errors={Errors} more={More}",
|
||||
table, dryRun, result.RowsExamined, result.WouldWrite, result.Written,
|
||||
result.SkippedAlreadySet, result.SkippedUndecodable, result.Errors, result.NextContinuation != null);
|
||||
return result;
|
||||
}
|
||||
|
||||
private async Task<int> WritePendingAsync(TableClient tableClient,
|
||||
List<(string PartitionKey, string RowKey, DateTime OccurredUtc)> pending, BackfillResult result)
|
||||
{
|
||||
var written = 0;
|
||||
foreach (var partitionGroup in pending.GroupBy(p => p.PartitionKey))
|
||||
{
|
||||
foreach (var chunk in partitionGroup.Chunk(TransactionChunkSize))
|
||||
{
|
||||
// Merge entity carries ONLY the new column; unconditional (ETag *) merge is
|
||||
// safe because the value is a pure function of the immutable RowKey.
|
||||
var actions = chunk.Select(p => new TableTransactionAction(
|
||||
TableTransactionActionType.UpdateMerge,
|
||||
new TableEntity(p.PartitionKey, p.RowKey)
|
||||
{
|
||||
[BusinessTimestamp.OccurredUtcColumn] = p.OccurredUtc,
|
||||
},
|
||||
ETag.All)).ToList();
|
||||
try
|
||||
{
|
||||
await tableClient.SubmitTransactionAsync(actions);
|
||||
written += actions.Count;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
result.Errors += actions.Count;
|
||||
_logger.LogError(ex, "OccurredUtc backfill: merge transaction failed for partition {PK} ({Count} rows)",
|
||||
partitionGroup.Key, actions.Count);
|
||||
}
|
||||
}
|
||||
}
|
||||
return written;
|
||||
}
|
||||
}
|
||||
}
|
||||
+63
-30
@@ -1,5 +1,6 @@
|
||||
using Azure;
|
||||
using Azure.Data.Tables;
|
||||
using AutopilotMonitor.Functions.Helpers;
|
||||
using AutopilotMonitor.Functions.Pagination;
|
||||
using AutopilotMonitor.Functions.Security;
|
||||
using AutopilotMonitor.Shared;
|
||||
@@ -55,7 +56,10 @@ namespace AutopilotMonitor.Functions.Services
|
||||
{ "EntityType", entityType },
|
||||
{ "EntityId", entityId },
|
||||
{ "PerformedBy", performedBy },
|
||||
{ "Timestamp", timestamp },
|
||||
// Business event time. "Timestamp" is a reserved system property (server
|
||||
// ignores supplied values and resets it on every row rewrite), so the
|
||||
// event time must live in a regular custom column to survive migrations.
|
||||
{ BusinessTimestamp.OccurredUtcColumn, timestamp },
|
||||
{ "Details", details != null ? JsonConvert.SerializeObject(details) : string.Empty }
|
||||
};
|
||||
|
||||
@@ -71,32 +75,26 @@ namespace AutopilotMonitor.Functions.Services
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Retention cleanup: deletes audit log entries whose Timestamp is older than the cutoff.
|
||||
/// The AuditLogs table is append-only (one row per admin action) and is otherwise only
|
||||
/// wiped on tenant offboarding, so without this it grows unbounded. Filters server-side on
|
||||
/// the per-row "Timestamp" property and selects only PK/RK to keep the scan cheap.
|
||||
/// Retention cleanup: deletes audit log entries whose business time is older than the
|
||||
/// cutoff. The AuditLogs table is append-only (one row per admin action) and is otherwise
|
||||
/// only wiped on tenant offboarding, so without this it grows unbounded.
|
||||
/// Two passes: (1) time-encoded '!'-rows via an index-backed RowKey range — the system
|
||||
/// Timestamp cannot be used because storage migrations reset it (post-migration it would
|
||||
/// delete nothing for a retention period, then the whole pre-migration corpus at once);
|
||||
/// (2) legacy bare-GUID rows (true event time unrecoverable) via their frozen system
|
||||
/// Timestamp — they all age out ~180d after the 2026-07-18 migration.
|
||||
/// TODO: Remove pass 2 after 2027-03-01 (no bare-GUID rows can remain by then).
|
||||
/// </summary>
|
||||
public async Task<int> DeleteAuditLogsOlderThanAsync(DateTime cutoffUtc)
|
||||
{
|
||||
try
|
||||
{
|
||||
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.AuditLogs);
|
||||
var filter = $"Timestamp lt datetime'{cutoffUtc:yyyy-MM-ddTHH:mm:ss}Z'";
|
||||
var query = tableClient.QueryAsync<TableEntity>(filter: filter, select: new[] { "PartitionKey", "RowKey" });
|
||||
|
||||
int deleted = 0;
|
||||
await foreach (var entity in query)
|
||||
{
|
||||
try
|
||||
{
|
||||
await tableClient.DeleteEntityAsync(entity.PartitionKey, entity.RowKey);
|
||||
deleted++;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to delete audit log {PK}/{RK}", entity.PartitionKey, entity.RowKey);
|
||||
}
|
||||
}
|
||||
var deleted = 0;
|
||||
deleted += await DeleteAuditLogsByFilterAsync(tableClient, BusinessTimestamp.AuditRetentionClause(cutoffUtc));
|
||||
// 'RowKey ge '0'' cannot match any '!'-row (0x21 < 0x30), only legacy GUIDs.
|
||||
deleted += await DeleteAuditLogsByFilterAsync(tableClient,
|
||||
$"RowKey ge '0' and Timestamp lt datetime'{cutoffUtc:yyyy-MM-ddTHH:mm:ss}Z'");
|
||||
|
||||
if (deleted > 0)
|
||||
_logger.LogInformation("Deleted {Count} audit log entries older than {Cutoff:yyyy-MM-dd}", deleted, cutoffUtc);
|
||||
@@ -110,6 +108,25 @@ namespace AutopilotMonitor.Functions.Services
|
||||
}
|
||||
}
|
||||
|
||||
private async Task<int> DeleteAuditLogsByFilterAsync(TableClient tableClient, string filter)
|
||||
{
|
||||
var query = tableClient.QueryAsync<TableEntity>(filter: filter, select: new[] { "PartitionKey", "RowKey" });
|
||||
int deleted = 0;
|
||||
await foreach (var entity in query)
|
||||
{
|
||||
try
|
||||
{
|
||||
await tableClient.DeleteEntityAsync(entity.PartitionKey, entity.RowKey);
|
||||
deleted++;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to delete audit log {PK}/{RK}", entity.PartitionKey, entity.RowKey);
|
||||
}
|
||||
}
|
||||
return deleted;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Gets audit log entries for a tenant within an optional UTC date window.
|
||||
/// No row cap — returns the full filtered set, sorted newest-first.
|
||||
@@ -384,9 +401,9 @@ namespace AutopilotMonitor.Functions.Services
|
||||
if (!string.IsNullOrEmpty(lastRowKey))
|
||||
clauses.Add($"RowKey gt '{lastRowKey!.Replace("'", "''")}'");
|
||||
if (dateFrom.HasValue)
|
||||
clauses.Add($"Timestamp ge datetime'{ToUtc(dateFrom.Value):o}'");
|
||||
clauses.Add(BusinessTimestamp.AuditDateFromClause(ToUtc(dateFrom.Value)));
|
||||
if (dateTo.HasValue)
|
||||
clauses.Add($"Timestamp le datetime'{ToUtc(dateTo.Value):o}'");
|
||||
clauses.Add(BusinessTimestamp.AuditDateToClause(ToUtc(dateTo.Value)));
|
||||
return string.Join(" and ", clauses);
|
||||
}
|
||||
|
||||
@@ -398,14 +415,30 @@ namespace AutopilotMonitor.Functions.Services
|
||||
EntityType = entity.GetString("EntityType") ?? string.Empty,
|
||||
EntityId = entity.GetString("EntityId") ?? string.Empty,
|
||||
PerformedBy = entity.GetString("PerformedBy") ?? string.Empty,
|
||||
Timestamp = entity.GetDateTimeOffset("Timestamp")?.UtcDateTime ?? DateTime.UtcNow,
|
||||
Timestamp = ResolveAuditTimestamp(entity),
|
||||
Details = entity.GetString("Details") ?? string.Empty,
|
||||
};
|
||||
|
||||
// The system Timestamp is auto-managed by Azure on insert and reliably
|
||||
// sortable; the user-defined "Timestamp" property is set by LogAuditEntry
|
||||
// for parity with the model. We filter on the system Timestamp since it
|
||||
// is always indexed.
|
||||
/// <summary>
|
||||
/// Business time of an audit row: OccurredUtc column → time decoded from the
|
||||
/// reverse-tick RowKey → system Timestamp. The system Timestamp is only a last
|
||||
/// resort (legacy bare-GUID rows): it is reset by any row rewrite, so for rows
|
||||
/// migrated between storage accounts it holds the migration moment, not the
|
||||
/// event time.
|
||||
/// </summary>
|
||||
internal static DateTime ResolveAuditTimestamp(TableEntity entity)
|
||||
{
|
||||
var occurred = BusinessTimestamp.GetUtcDateTime(entity, BusinessTimestamp.OccurredUtcColumn);
|
||||
if (occurred.HasValue)
|
||||
return occurred.Value;
|
||||
if (BusinessTimestamp.TryDecodeAuditRowKey(entity.RowKey, out var decoded))
|
||||
return decoded;
|
||||
return BusinessTimestamp.GetUtcDateTime(entity, "Timestamp") ?? DateTime.UtcNow;
|
||||
}
|
||||
|
||||
// Date windows filter on the RowKey (reverse-tick, index-backed) rather than any
|
||||
// timestamp property: the system Timestamp is reset by migrations, and an
|
||||
// OccurredUtc property filter would exclude rows that lack the column.
|
||||
internal static string? BuildAuditLogFilter(string? tenantId, DateTime? dateFrom, DateTime? dateTo,
|
||||
bool excludeDeletions = false, AuditLogQueryFilters? filters = null)
|
||||
{
|
||||
@@ -424,11 +457,11 @@ namespace AutopilotMonitor.Functions.Services
|
||||
}
|
||||
if (dateFrom.HasValue)
|
||||
{
|
||||
clauses.Add($"Timestamp ge datetime'{ToUtc(dateFrom.Value):o}'");
|
||||
clauses.Add(BusinessTimestamp.AuditDateFromClause(ToUtc(dateFrom.Value)));
|
||||
}
|
||||
if (dateTo.HasValue)
|
||||
{
|
||||
clauses.Add($"Timestamp le datetime'{ToUtc(dateTo.Value):o}'");
|
||||
clauses.Add(BusinessTimestamp.AuditDateToClause(ToUtc(dateTo.Value)));
|
||||
}
|
||||
return string.Join(" and ", clauses);
|
||||
}
|
||||
|
||||
@@ -681,7 +681,10 @@ namespace AutopilotMonitor.Functions.Services
|
||||
["EventId"] = evt.EventId,
|
||||
["SessionId"] = evt.SessionId,
|
||||
["TenantId"] = evt.TenantId,
|
||||
["Timestamp"] = evt.Timestamp,
|
||||
// "Timestamp" is a reserved system property — supplied values are ignored
|
||||
// and any row rewrite (storage migration) resets it. The sanitized agent
|
||||
// event time therefore lives in its own column.
|
||||
[BusinessTimestamp.OccurredUtcColumn] = EnsureUtc(evt.Timestamp),
|
||||
["EventType"] = evt.EventType ?? string.Empty,
|
||||
["Severity"] = (int)evt.Severity,
|
||||
["Source"] = evt.Source ?? string.Empty,
|
||||
@@ -777,7 +780,8 @@ namespace AutopilotMonitor.Functions.Services
|
||||
["EventId"] = evt.EventId,
|
||||
["SessionId"] = evt.SessionId,
|
||||
["TenantId"] = evt.TenantId,
|
||||
["Timestamp"] = evt.Timestamp,
|
||||
// Reserved-name caveat: see StoreEventAsync.
|
||||
[BusinessTimestamp.OccurredUtcColumn] = EnsureUtc(evt.Timestamp),
|
||||
["EventType"] = evt.EventType ?? string.Empty,
|
||||
["Severity"] = (int)evt.Severity,
|
||||
["Source"] = evt.Source ?? string.Empty,
|
||||
@@ -2673,10 +2677,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
EventId = entity.GetString("EventId") ?? string.Empty,
|
||||
SessionId = entity.GetString("SessionId") ?? string.Empty,
|
||||
TenantId = tenantId ?? partitionKey,
|
||||
Timestamp = DateTime.SpecifyKind(
|
||||
entity.GetDateTimeOffset("Timestamp")?.UtcDateTime
|
||||
?? entity.GetDateTime("Timestamp")
|
||||
?? DateTime.UtcNow, DateTimeKind.Utc),
|
||||
Timestamp = DateTime.SpecifyKind(ResolveEventTimestamp(entity), DateTimeKind.Utc),
|
||||
EventType = entity.GetString("EventType") ?? string.Empty,
|
||||
Severity = (EventSeverity)(entity.GetInt32("Severity") ?? 0),
|
||||
Source = entity.GetString("Source") ?? string.Empty,
|
||||
@@ -2827,7 +2828,10 @@ namespace AutopilotMonitor.Functions.Services
|
||||
/// <summary>
|
||||
/// Returns the earliest event timestamp persisted for a session, if any.
|
||||
/// Events are written with RowKey "{Timestamp}_{Sequence}", so querying the partition
|
||||
/// and taking the first row yields the earliest event.
|
||||
/// and taking the first row yields the earliest event. Resolution must NOT rely on
|
||||
/// the system Timestamp: it is reset by storage migrations, and this value feeds the
|
||||
/// StartedAt/Duration reconciliation — a migration-reset read would rewrite those
|
||||
/// session fields with the migration date.
|
||||
/// </summary>
|
||||
private async Task<DateTime?> GetEarliestSessionEventTimestampAsync(string tenantId, string sessionId)
|
||||
{
|
||||
@@ -2838,13 +2842,12 @@ namespace AutopilotMonitor.Functions.Services
|
||||
var query = tableClient.QueryAsync<TableEntity>(
|
||||
filter: $"PartitionKey eq '{partitionKey}'",
|
||||
maxPerPage: 1,
|
||||
select: new[] { "Timestamp", "RowKey" }
|
||||
select: new[] { "RowKey", BusinessTimestamp.OccurredUtcColumn, "Timestamp" }
|
||||
);
|
||||
|
||||
await foreach (var entity in query)
|
||||
{
|
||||
return entity.GetDateTimeOffset("Timestamp")?.UtcDateTime
|
||||
?? entity.GetDateTime("Timestamp");
|
||||
return ResolveEventTimestampOrNull(entity);
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
@@ -2855,6 +2858,26 @@ namespace AutopilotMonitor.Functions.Services
|
||||
return null;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Business time of an event row: OccurredUtc column → agent time decoded from the
|
||||
/// "{yyyyMMddHHmmssfff}_{seq}" RowKey → system Timestamp (last resort — reset by any
|
||||
/// row rewrite, e.g. a storage migration; for migrated rows it holds the migration
|
||||
/// moment). Event rows are deliberately never backfilled, so the RowKey decode is the
|
||||
/// primary path for all pre-OccurredUtc rows.
|
||||
/// </summary>
|
||||
internal static DateTime ResolveEventTimestamp(TableEntity entity)
|
||||
=> ResolveEventTimestampOrNull(entity) ?? DateTime.UtcNow;
|
||||
|
||||
internal static DateTime? ResolveEventTimestampOrNull(TableEntity entity)
|
||||
{
|
||||
var occurred = BusinessTimestamp.GetUtcDateTime(entity, BusinessTimestamp.OccurredUtcColumn);
|
||||
if (occurred.HasValue)
|
||||
return occurred.Value;
|
||||
if (BusinessTimestamp.TryDecodeEventRowKeyPrefix(entity.RowKey, out var decoded))
|
||||
return decoded;
|
||||
return BusinessTimestamp.GetUtcDateTime(entity, "Timestamp");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Returns the authoritative (total, reboot) event-row counts for a session, or <c>null</c>
|
||||
/// if they could not be determined (storage error). Events are written with UpsertReplace
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
"use client";
|
||||
|
||||
import { Suspense, useCallback, useEffect, useMemo, useRef, useState } from "react";
|
||||
import Link from "next/link";
|
||||
import { useRouter, useSearchParams } from "next/navigation";
|
||||
import { ProtectedRoute } from "../../components/ProtectedRoute";
|
||||
import { GlobalAdminBanner } from "@/components/GlobalAdminBanner";
|
||||
|
||||
@@ -31,6 +31,9 @@ export function SignalRProvider({ children }: { children: React.ReactNode }) {
|
||||
const [joinedGroups, setJoinedGroups] = useState<string[]>([]);
|
||||
const retryCountRef = useRef(0);
|
||||
const retryTimeoutRef = useRef<ReturnType<typeof setTimeout> | null>(null);
|
||||
// Collapses rapid successive accessRevoked pushes (one per revoked grant) into a single
|
||||
// restart — a second start() on a non-Disconnected connection throws and burns the retry budget.
|
||||
const revokeRestartInFlightRef = useRef(false);
|
||||
// Set in onreconnecting, read in onreconnected so we can report downtime as a measurement.
|
||||
const disconnectStartedAtRef = useRef<number | null>(null);
|
||||
const maxRetries = 3;
|
||||
@@ -157,12 +160,20 @@ export function SignalRProvider({ children }: { children: React.ReactNode }) {
|
||||
// the next Connected transition — through the join endpoint, which re-runs authorization
|
||||
// against the fresh scope. Revoked groups 403; still-authorized streams recover automatically.
|
||||
newConnection.on('accessRevoked', async () => {
|
||||
// Dropping an event that arrives mid-restart is safe: the rejoin after start() re-runs
|
||||
// authorization against the CURRENT scope, which already reflects that later revoke.
|
||||
if (revokeRestartInFlightRef.current) return;
|
||||
revokeRestartInFlightRef.current = true;
|
||||
console.warn('[SignalR] accessRevoked received — restarting connection to re-authorize group memberships');
|
||||
trackEvent('signalr_access_revoked');
|
||||
try {
|
||||
await newConnection.stop();
|
||||
} catch { /* proceed to restart regardless */ }
|
||||
startConnection();
|
||||
try {
|
||||
await startConnection();
|
||||
} finally {
|
||||
revokeRestartInFlightRef.current = false;
|
||||
}
|
||||
});
|
||||
|
||||
// Start as soon as the user is authenticated. The previous 2s setTimeout
|
||||
|
||||
Reference in New Issue
Block a user