mirror of
https://github.com/okieselbach/AutopilotMonitor.git
synced 2026-10-24 12:42:38 +00:00
09bfe7a1bc
Operationalises the 7-PR backend cascade-delete pipeline that landed via commits
c8658b9f..921994e8: per-tenant V2 opt-in switch, audit-trail consolidation, and
a Global-Admin-only Session Cleanup page to track in-flight / poisoned / stranded
cascades and trigger restores.
== PR-A: per-tenant V2 enrollment ==
- UpdateTenantConfigurationFunction extends the GA-only gate list with
EnableCascadeDeleteV2 so tenant admins cannot self-opt-in / out of the
snapshot+restore pathway.
- TenantManagementSection.tsx adds a purple-bordered "Enable V2 Cascade Delete"
checkbox in the GA edit modal and a "Cascade V2" badge on the tenant card so
rollout state is visible at a glance.
== PR-B: audit consolidation ==
- SessionDeletionHandler drops the 15-per-cascade deletion_step_completed /
deletion_step_failed / deletion_verification_failed audit emissions. Step
telemetry lives in DeletionProgress + structured logs; tenant admins now see
just deletion_started + deletion_completed + deletion_restored.
- SessionDeletionWorker no longer writes a deletion_poisoned tenant audit;
the signal moved to a new SessionDeletionPoisoned OpsEvent (operator-bound,
Telegram-routable, dual-registered in OpsAlertRulesSection.tsx catalog).
IMaintenanceRepository dependency removed from the worker constructor.
- SessionRestoreService stops emitting the deletion_manifest_downloaded
breadcrumb; deletion_restored is the operator-visible outcome.
== PR-C: Session Cleanup admin page ==
- New /admin/ops/session-cleanup route (gated by app/admin/layout.tsx's
ProtectedRoute requireGlobalAdmin). Sidebar entry under ga-ops alongside
Maintenance. Four URL-state tabs: In-Flight / Poisoned / Stranded Queued /
Activity (links to existing /audit page).
- Backend reads: GetSessionDeletionsListFunction
(GET /api/global/session-deletions?state=&strandedSinceMinutes=) +
GetSessionDeletionManifestFunction
(GET /api/admin/sessions/{id}/deletion-manifest?tenantId=&manifestId=&mode=).
Both registered as GlobalAdminOnly in EndpointAccessPolicyCatalog.
- DeletionPreviewModal reads the persisted snapshot via the stored-manifest
endpoint (NOT a fresh dry-run via the existing /delete/preview, which would
enumerate live-but-partially-deleted data for in-flight cascades). Renders
manifest counts, snapshot SHA, worker progress (CompletedSteps,
VerificationDone, TombstoneStarted, aggregate decrements), and the last
verification failure with residual sample. Download button uses
authenticatedFetch + Blob + Object URL so the Bearer token is sent.
- RestoreConfirmDialog drops the Full/Partial radio (mode is auto-selected
by SessionRestoreService based on cascade state — forcing the wrong mode
would corrupt state). Reason field threads through to deletion_restored
audit details. Dry-run default = true and previews the auto-selected mode.
== Codex review follow-ups (across four rounds) ==
- High: tenantId is explicitly accepted on /delete/preview (query) and
/restore (body); GA cross-tenant resolution prefers the explicit value over
JWT TargetTenantId, falling back to SessionsIndex lookup. Without this, a
GA acting on a session in another tenant got their JWT home tenant.
- High: new stored-manifest endpoint replaces the dry-run preview for
in-flight / poisoned rows; previous fix built a fresh manifest against
already-partially-deleted data and misrepresented the cascade.
- Medium: SessionDeletionPoisoned OpsEvent now carries failureType +
failureMessage + observedResidualCount + a small residual preview, persisted
by the handler into DeletionProgress (LastFailureType, LastFailureMessage,
LastObservedResidualCount, LastResidualSampleJson) before the throw so the
worker can include them.
- Medium: dry-run restore reports the AUTO-SELECTED mode (full/partial) so
the dialog's preview reflects what the real run will do, instead of the
literal string "dryRun".
- Medium: residual count field renamed to observedResidualCount with an
honest XML doc — CascadeVerificationService caps at 50 per table AND
short-circuits after the first failing table, so the number is a lower
bound. UI surfaces "N of 50+ observed rows" when N hits the cap.
- Medium: three-layer cap defense on the OpsEvent residual sample to stay
under the 4096-char TableOpsEventRepository truncation: max 5 entries
(OpsEventResidualSamplePreviewSize), per-field max 96 chars with `…` marker
(OpsEventResidualKeyMaxChars), total preview budget 1200 chars
(OpsEventResidualPreviewBudgetChars) with trailing-entry drop. Full sample
stays in the progress blob for the admin UI.
- Low: estimatedSnapshotBytes field name aligned end-to-end; activity tab
text clarifies audit-action vs OpsEvent distinction.
Tests:
- Backend xUnit 1538/1538 pass (up from 1510 baseline; +28 across the new
Session Cleanup endpoints, OpsEvent enrichment, restore-mode reporting,
cap-defense unit + integration tests).
- Web vitest 347/347 pass.
- TypeScript strict + dotnet build clean.
Deliberately deferred:
- xUnit tests for GetSessionDeletionsListFunction +
GetSessionDeletionManifestFunction (no HttpRequestData mock harness in repo;
would be a separate refactor).
- vitest for DeletionPreviewModal + RestoreConfirmDialog (defer until first
field issue).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
268 lines
13 KiB
C#
268 lines
13 KiB
C#
using System;
|
|
using System.Collections.Generic;
|
|
using System.Diagnostics;
|
|
using System.IO;
|
|
using System.IO.Compression;
|
|
using System.Linq;
|
|
using System.Net;
|
|
using System.Text.Json;
|
|
using System.Threading.Tasks;
|
|
using System.Web;
|
|
using AutopilotMonitor.Functions.Helpers;
|
|
using AutopilotMonitor.Functions.Services.Deletion;
|
|
using AutopilotMonitor.Shared.DataAccess;
|
|
using AutopilotMonitor.Shared.Models.Deletion;
|
|
using Microsoft.Azure.Functions.Worker;
|
|
using Microsoft.Azure.Functions.Worker.Http;
|
|
using Microsoft.Extensions.Logging;
|
|
|
|
namespace AutopilotMonitor.Functions.Functions.Sessions
|
|
{
|
|
/// <summary>
|
|
/// <c>GET /api/admin/sessions/{sessionId}/delete/preview?mode={summary|full|download}</c> —
|
|
/// read-only cascade-delete preview gated on <c>GlobalAdminOnly</c>. Runs the
|
|
/// <see cref="DeletionManifestBuilder"/> against current data, returns the would-be manifest,
|
|
/// and audits <c>deletion_preview</c>. NEVER mutates state, NEVER uploads a blob, NEVER
|
|
/// enqueues — safety of the real delete must come from the producer's own CAS-then-build-then-upload
|
|
/// sequence, not from a prior preview call (the gap between preview and delete is unsafe by
|
|
/// design — preview is a debug tool, not a precondition; Plan §5 PR3 / §18).
|
|
/// </summary>
|
|
public class GetSessionDeletePreviewFunction
|
|
{
|
|
private readonly ILogger<GetSessionDeletePreviewFunction> _logger;
|
|
private readonly DeletionManifestBuilder _builder;
|
|
private readonly SessionDeletionGuard _guard;
|
|
private readonly ISessionRepository _sessionRepo;
|
|
private readonly IMaintenanceRepository _maintenanceRepo;
|
|
|
|
public GetSessionDeletePreviewFunction(
|
|
ILogger<GetSessionDeletePreviewFunction> logger,
|
|
DeletionManifestBuilder builder,
|
|
SessionDeletionGuard guard,
|
|
ISessionRepository sessionRepo,
|
|
IMaintenanceRepository maintenanceRepo)
|
|
{
|
|
_logger = logger;
|
|
_builder = builder;
|
|
_guard = guard;
|
|
_sessionRepo = sessionRepo;
|
|
_maintenanceRepo = maintenanceRepo;
|
|
}
|
|
|
|
[Function("GetSessionDeletePreview")]
|
|
public async Task<HttpResponseData> Run(
|
|
[HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "admin/sessions/{sessionId}/delete/preview")] HttpRequestData req,
|
|
string sessionId)
|
|
{
|
|
if (string.IsNullOrWhiteSpace(sessionId))
|
|
{
|
|
var badRequest = req.CreateResponse(HttpStatusCode.BadRequest);
|
|
await badRequest.WriteAsJsonAsync(new { success = false, message = "sessionId is required" });
|
|
return badRequest;
|
|
}
|
|
|
|
var requestCtx = req.GetRequestContext();
|
|
var actorEmail = TenantHelper.GetUserIdentifier(req);
|
|
var query = HttpUtility.ParseQueryString(req.Url.Query ?? string.Empty);
|
|
var mode = (query["mode"] ?? "summary").ToLowerInvariant();
|
|
if (mode != "summary" && mode != "full" && mode != "download")
|
|
{
|
|
var bad = req.CreateResponse(HttpStatusCode.BadRequest);
|
|
await bad.WriteAsJsonAsync(new { success = false, message = "mode must be one of: summary, full, download" });
|
|
return bad;
|
|
}
|
|
|
|
// Resolve tenantId. Three-tier fallback ordered by Codex F1 review:
|
|
// 1. Explicit ?tenantId=<guid> on GA calls — only path that lets a Global Admin
|
|
// preview a session in a tenant other than their JWT home tenant.
|
|
// 2. JWT TargetTenantId, where the policy middleware mirrors the JWT tid for
|
|
// non-RouteParam routes. Sufficient for tenant admins acting on their own tenant.
|
|
// 3. SessionsIndex lookup keyed by sessionId — the historical fallback for the
|
|
// "global" sentinel; preserved so existing callers without ?tenantId still work.
|
|
var explicitTenantId = query["tenantId"];
|
|
string tenantId;
|
|
if (requestCtx.IsGlobalAdmin
|
|
&& !string.IsNullOrEmpty(explicitTenantId)
|
|
&& Guid.TryParse(explicitTenantId, out _))
|
|
{
|
|
tenantId = explicitTenantId!;
|
|
}
|
|
else
|
|
{
|
|
tenantId = requestCtx.TargetTenantId;
|
|
if (string.IsNullOrEmpty(tenantId) || string.Equals(tenantId, "global", StringComparison.OrdinalIgnoreCase))
|
|
{
|
|
tenantId = await _sessionRepo.FindSessionTenantIdAsync(sessionId) ?? string.Empty;
|
|
}
|
|
}
|
|
if (string.IsNullOrEmpty(tenantId))
|
|
{
|
|
var notFound = req.CreateResponse(HttpStatusCode.NotFound);
|
|
await notFound.WriteAsJsonAsync(new { success = false, message = $"Session {sessionId} not found in any tenant." });
|
|
return notFound;
|
|
}
|
|
|
|
// Guard hint: if a cascade is in flight, surface that to the caller. The preview
|
|
// itself never acquires any lock — this read is pure observation, and a guarded
|
|
// cascade may still race with the preview's enumeration. Operators should treat
|
|
// the response as "snapshot of current state at preview time", not a blocking lock.
|
|
string? inFlightHint = null;
|
|
try
|
|
{
|
|
await _guard.EnsureWritableAsync(tenantId, sessionId, callerContext: "DeletePreview");
|
|
}
|
|
catch (SessionDeletionLockedException locked)
|
|
{
|
|
inFlightHint = $"Cascade in progress (state={locked.CurrentState}, manifestId={locked.ManifestId ?? "(none)"}); preview shows current data anyway.";
|
|
_logger.LogInformation(
|
|
"DeletePreview hint: cascade in flight tenant={TenantId} session={SessionId} state={State} manifestId={ManifestId}",
|
|
tenantId, sessionId, locked.CurrentState, locked.ManifestId);
|
|
}
|
|
|
|
// Build the manifest (read-only — no state changes, no blob writes).
|
|
var actor = new DeletionActor { Type = "admin", Actor = actorEmail };
|
|
var stopwatch = Stopwatch.StartNew();
|
|
DeletionManifest manifest;
|
|
try
|
|
{
|
|
manifest = await _builder.BuildAsync(
|
|
tenantId, sessionId,
|
|
reason: "preview",
|
|
actor: actor,
|
|
retentionContext: new DeletionRetentionContext());
|
|
}
|
|
catch (Exception ex)
|
|
{
|
|
_logger.LogError(ex,
|
|
"DeletePreview failed: tenant={TenantId} session={SessionId} mode={Mode}",
|
|
tenantId, sessionId, mode);
|
|
var error = req.CreateResponse(HttpStatusCode.InternalServerError);
|
|
await error.WriteAsJsonAsync(new { success = false, message = "Failed to build deletion manifest", error = ex.Message });
|
|
return error;
|
|
}
|
|
stopwatch.Stop();
|
|
var builderDurationMs = stopwatch.ElapsedMilliseconds;
|
|
|
|
// Audit (Plan §7 deletion_preview). Counts the same per-mode for observability.
|
|
try
|
|
{
|
|
await _maintenanceRepo.LogAuditEntryAsync(
|
|
tenantId,
|
|
action: "deletion_preview",
|
|
entityType: "Session",
|
|
entityId: sessionId,
|
|
performedBy: actorEmail,
|
|
details: new Dictionary<string, string>(StringComparer.Ordinal)
|
|
{
|
|
["mode"] = mode,
|
|
["builderDurationMs"] = builderDurationMs.ToString(System.Globalization.CultureInfo.InvariantCulture),
|
|
["schemaHash"] = manifest.SchemaHash,
|
|
["totalSteps"] = manifest.Steps.Count.ToString(System.Globalization.CultureInfo.InvariantCulture),
|
|
});
|
|
}
|
|
catch (Exception ex)
|
|
{
|
|
_logger.LogWarning(ex, "DeletePreview audit emission failed (non-fatal). tenant={TenantId} session={SessionId}", tenantId, sessionId);
|
|
}
|
|
|
|
return mode switch
|
|
{
|
|
"full" => await BuildFullResponse(req, manifest, inFlightHint, builderDurationMs),
|
|
"download" => await BuildDownloadResponse(req, manifest, tenantId, sessionId),
|
|
_ => await BuildSummaryResponse(req, manifest, inFlightHint, builderDurationMs),
|
|
};
|
|
}
|
|
|
|
private static async Task<HttpResponseData> BuildSummaryResponse(
|
|
HttpRequestData req, DeletionManifest manifest, string? inFlightHint, long builderDurationMs)
|
|
{
|
|
var sampleKeys = new Dictionary<string, List<object>>(StringComparer.Ordinal);
|
|
long totalRowCount = 0;
|
|
foreach (var step in manifest.Steps)
|
|
{
|
|
totalRowCount += step.RowCount;
|
|
if (step.Rows.Count == 0) continue;
|
|
var key = step.Table ?? step.Step ?? $"order_{step.Order}";
|
|
sampleKeys[key] = step.Rows.Take(5).Select(r => (object)new { pk = r.Pk, rk = r.Rk }).ToList();
|
|
}
|
|
|
|
var estimatedBytes = EstimateSnapshotSizeBytes(manifest);
|
|
var response = req.CreateResponse(HttpStatusCode.OK);
|
|
await response.WriteAsJsonAsync(new
|
|
{
|
|
success = true,
|
|
mode = "summary",
|
|
inFlightHint,
|
|
preflightCounts = manifest.PreflightCounts,
|
|
sampleKeys,
|
|
estimatedRowCount = totalRowCount,
|
|
estimatedSnapshotBytes = estimatedBytes,
|
|
builderDurationMs,
|
|
schemaHash = manifest.SchemaHash,
|
|
manifestId = manifest.ManifestId,
|
|
});
|
|
return response;
|
|
}
|
|
|
|
private static async Task<HttpResponseData> BuildFullResponse(
|
|
HttpRequestData req, DeletionManifest manifest, string? inFlightHint, long builderDurationMs)
|
|
{
|
|
// mode=full returns the RAW manifest JSON — byte-identical to what the producer
|
|
// would upload to the deletion-manifests blob container (modulo gzip). This lets
|
|
// operators diff a dry-run against the actual cascade's snapshot without parsing
|
|
// through an envelope. Side-channel data (in-flight hint, builder duration) is
|
|
// surfaced via response headers so the body stays clean.
|
|
var json = JsonSerializer.SerializeToUtf8Bytes(manifest, DeletionManifestJson.SerializerOptions);
|
|
var response = req.CreateResponse(HttpStatusCode.OK);
|
|
response.Headers.Add("Content-Type", "application/json");
|
|
response.Headers.Add("X-Deletion-Preview-Mode", "full");
|
|
response.Headers.Add("X-Deletion-Preview-Builder-Duration-Ms", builderDurationMs.ToString(System.Globalization.CultureInfo.InvariantCulture));
|
|
response.Headers.Add("X-Deletion-Preview-Manifest-Id", manifest.ManifestId);
|
|
response.Headers.Add("X-Deletion-Preview-Schema-Hash", manifest.SchemaHash);
|
|
if (!string.IsNullOrEmpty(inFlightHint))
|
|
{
|
|
response.Headers.Add("X-Deletion-Preview-InFlight-Hint", inFlightHint);
|
|
}
|
|
await response.Body.WriteAsync(json, 0, json.Length);
|
|
return response;
|
|
}
|
|
|
|
private async Task<HttpResponseData> BuildDownloadResponse(
|
|
HttpRequestData req, DeletionManifest manifest, string tenantId, string sessionId)
|
|
{
|
|
var json = JsonSerializer.SerializeToUtf8Bytes(manifest, DeletionManifestJson.SerializerOptions);
|
|
byte[] gzipped;
|
|
using (var output = new MemoryStream())
|
|
{
|
|
using (var gzip = new GZipStream(output, CompressionLevel.Optimal, leaveOpen: true))
|
|
{
|
|
await gzip.WriteAsync(json, 0, json.Length);
|
|
}
|
|
gzipped = output.ToArray();
|
|
}
|
|
|
|
var fileName = $"{tenantId}_{sessionId}_dryrun_{DateTime.UtcNow:yyyyMMddTHHmmssZ}.snapshot.json.gz";
|
|
var response = req.CreateResponse(HttpStatusCode.OK);
|
|
response.Headers.Add("Content-Type", "application/gzip");
|
|
response.Headers.Add("Content-Disposition", $"attachment; filename=\"{fileName}\"");
|
|
await response.Body.WriteAsync(gzipped, 0, gzipped.Length);
|
|
return response;
|
|
}
|
|
|
|
private static long EstimateSnapshotSizeBytes(DeletionManifest manifest)
|
|
{
|
|
// Cheap estimate — serialize once + report length. Compression ratio of ~0.2 gets
|
|
// applied client-side if the operator wants the on-wire size.
|
|
try
|
|
{
|
|
var bytes = JsonSerializer.SerializeToUtf8Bytes(manifest, DeletionManifestJson.SerializerOptions);
|
|
return bytes.LongLength;
|
|
}
|
|
catch
|
|
{
|
|
return -1;
|
|
}
|
|
}
|
|
}
|
|
}
|