mirror of
https://github.com/okieselbach/AutopilotMonitor.git
synced 2026-10-24 12:42:38 +00:00
feat(backend,web): admin session-cleanup UI for V2 cascade-delete + codex follow-ups
Operationalises the 7-PR backend cascade-delete pipeline that landed via commits
c8658b9f..921994e8: per-tenant V2 opt-in switch, audit-trail consolidation, and
a Global-Admin-only Session Cleanup page to track in-flight / poisoned / stranded
cascades and trigger restores.
== PR-A: per-tenant V2 enrollment ==
- UpdateTenantConfigurationFunction extends the GA-only gate list with
EnableCascadeDeleteV2 so tenant admins cannot self-opt-in / out of the
snapshot+restore pathway.
- TenantManagementSection.tsx adds a purple-bordered "Enable V2 Cascade Delete"
checkbox in the GA edit modal and a "Cascade V2" badge on the tenant card so
rollout state is visible at a glance.
== PR-B: audit consolidation ==
- SessionDeletionHandler drops the 15-per-cascade deletion_step_completed /
deletion_step_failed / deletion_verification_failed audit emissions. Step
telemetry lives in DeletionProgress + structured logs; tenant admins now see
just deletion_started + deletion_completed + deletion_restored.
- SessionDeletionWorker no longer writes a deletion_poisoned tenant audit;
the signal moved to a new SessionDeletionPoisoned OpsEvent (operator-bound,
Telegram-routable, dual-registered in OpsAlertRulesSection.tsx catalog).
IMaintenanceRepository dependency removed from the worker constructor.
- SessionRestoreService stops emitting the deletion_manifest_downloaded
breadcrumb; deletion_restored is the operator-visible outcome.
== PR-C: Session Cleanup admin page ==
- New /admin/ops/session-cleanup route (gated by app/admin/layout.tsx's
ProtectedRoute requireGlobalAdmin). Sidebar entry under ga-ops alongside
Maintenance. Four URL-state tabs: In-Flight / Poisoned / Stranded Queued /
Activity (links to existing /audit page).
- Backend reads: GetSessionDeletionsListFunction
(GET /api/global/session-deletions?state=&strandedSinceMinutes=) +
GetSessionDeletionManifestFunction
(GET /api/admin/sessions/{id}/deletion-manifest?tenantId=&manifestId=&mode=).
Both registered as GlobalAdminOnly in EndpointAccessPolicyCatalog.
- DeletionPreviewModal reads the persisted snapshot via the stored-manifest
endpoint (NOT a fresh dry-run via the existing /delete/preview, which would
enumerate live-but-partially-deleted data for in-flight cascades). Renders
manifest counts, snapshot SHA, worker progress (CompletedSteps,
VerificationDone, TombstoneStarted, aggregate decrements), and the last
verification failure with residual sample. Download button uses
authenticatedFetch + Blob + Object URL so the Bearer token is sent.
- RestoreConfirmDialog drops the Full/Partial radio (mode is auto-selected
by SessionRestoreService based on cascade state — forcing the wrong mode
would corrupt state). Reason field threads through to deletion_restored
audit details. Dry-run default = true and previews the auto-selected mode.
== Codex review follow-ups (across four rounds) ==
- High: tenantId is explicitly accepted on /delete/preview (query) and
/restore (body); GA cross-tenant resolution prefers the explicit value over
JWT TargetTenantId, falling back to SessionsIndex lookup. Without this, a
GA acting on a session in another tenant got their JWT home tenant.
- High: new stored-manifest endpoint replaces the dry-run preview for
in-flight / poisoned rows; previous fix built a fresh manifest against
already-partially-deleted data and misrepresented the cascade.
- Medium: SessionDeletionPoisoned OpsEvent now carries failureType +
failureMessage + observedResidualCount + a small residual preview, persisted
by the handler into DeletionProgress (LastFailureType, LastFailureMessage,
LastObservedResidualCount, LastResidualSampleJson) before the throw so the
worker can include them.
- Medium: dry-run restore reports the AUTO-SELECTED mode (full/partial) so
the dialog's preview reflects what the real run will do, instead of the
literal string "dryRun".
- Medium: residual count field renamed to observedResidualCount with an
honest XML doc — CascadeVerificationService caps at 50 per table AND
short-circuits after the first failing table, so the number is a lower
bound. UI surfaces "N of 50+ observed rows" when N hits the cap.
- Medium: three-layer cap defense on the OpsEvent residual sample to stay
under the 4096-char TableOpsEventRepository truncation: max 5 entries
(OpsEventResidualSamplePreviewSize), per-field max 96 chars with `…` marker
(OpsEventResidualKeyMaxChars), total preview budget 1200 chars
(OpsEventResidualPreviewBudgetChars) with trailing-entry drop. Full sample
stays in the progress blob for the admin UI.
- Low: estimatedSnapshotBytes field name aligned end-to-end; activity tab
text clarifies audit-action vs OpsEvent distinction.
Tests:
- Backend xUnit 1538/1538 pass (up from 1510 baseline; +28 across the new
Session Cleanup endpoints, OpsEvent enrichment, restore-mode reporting,
cap-defense unit + integration tests).
- Web vitest 347/347 pass.
- TypeScript strict + dotnet build clean.
Deliberately deferred:
- xUnit tests for GetSessionDeletionsListFunction +
GetSessionDeletionManifestFunction (no HttpRequestData mock harness in repo;
would be a separate refactor).
- vitest for DeletionPreviewModal + RestoreConfirmDialog (defer until first
field issue).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -61,9 +61,13 @@ public class SessionDeletionHandlerTests
|
||||
Assert.Single(harness.SignalR.SessionDeletedCalls);
|
||||
Assert.Equal((TenantId, SessionId), harness.SignalR.SessionDeletedCalls[0]);
|
||||
|
||||
// Final audit asserts both step-completed entries AND deletion_completed.
|
||||
// PR-B audit consolidation: only the lifecycle endpoint deletion_completed lands in the
|
||||
// tenant audit. Step-level progress lives in DeletionProgress (CompletedSteps); step
|
||||
// success goes to structured logs only.
|
||||
Assert.Contains(harness.AuditCalls, a => a.Action == "deletion_completed");
|
||||
Assert.Contains(harness.AuditCalls, a => a.Action == "deletion_step_completed");
|
||||
Assert.DoesNotContain(harness.AuditCalls, a => a.Action == "deletion_step_completed");
|
||||
Assert.DoesNotContain(harness.AuditCalls, a => a.Action == "deletion_step_failed");
|
||||
Assert.DoesNotContain(harness.AuditCalls, a => a.Action == "deletion_verification_failed");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -223,7 +227,95 @@ public class SessionDeletionHandlerTests
|
||||
Constants.TableNames.Sessions, It.IsAny<IReadOnlyList<(string, string)>>(), It.IsAny<CancellationToken>()),
|
||||
Times.Never);
|
||||
Assert.Empty(harness.SignalR.SessionDeletedCalls);
|
||||
Assert.Contains(harness.AuditCalls, a => a.Action == "deletion_verification_failed");
|
||||
// PR-B audit consolidation: verification failure surfaces via the throw → worker
|
||||
// → SessionDeletionPoisoned OpsEvent path. Tenant audit stays silent — the absence of
|
||||
// deletion_completed is the tenant-visible signal.
|
||||
Assert.DoesNotContain(harness.AuditCalls, a => a.Action == "deletion_verification_failed");
|
||||
Assert.DoesNotContain(harness.AuditCalls, a => a.Action == "deletion_completed");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Verification_failure_persists_LastFailure_into_DeletionProgress_for_OpsEvent_enrichment()
|
||||
{
|
||||
// Codex F4: the SessionDeletionPoisoned OpsEvent used to carry only queue-side metadata
|
||||
// (dequeueCount + manifestId). The handler now persists failureType + a residual sample
|
||||
// into DeletionProgress before throwing, so the worker can include the root cause when
|
||||
// it transitions the cascade to Poisoned. This test pins the persistence-before-throw
|
||||
// ordering and the field projection.
|
||||
var harness = new Harness();
|
||||
harness.SetHappyPath();
|
||||
harness.SetFullSessionManifest();
|
||||
harness.Verifier.Setup(v => v.VerifyAsync(It.IsAny<DeletionManifest>(), It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(new CascadeVerificationResult(
|
||||
isClean: false,
|
||||
residuals: new List<CascadeResidualKey>
|
||||
{
|
||||
new CascadeResidualKey("Events", "ghost-pk", "ghost-rk-1"),
|
||||
new CascadeResidualKey("Signals", "ghost-pk", "ghost-rk-2"),
|
||||
}));
|
||||
|
||||
await Assert.ThrowsAsync<InvalidOperationException>(() => harness.Sut.HandleAsync(harness.Envelope));
|
||||
|
||||
Assert.Equal("verification_residuals", harness.ProgressFake.LastFailureType);
|
||||
Assert.Contains("observed residual row(s)", harness.ProgressFake.LastFailureMessage ?? string.Empty);
|
||||
Assert.False(string.IsNullOrEmpty(harness.ProgressFake.LastResidualSampleJson));
|
||||
// The handler persists whatever the verifier returned — which is the OBSERVED count, not
|
||||
// the true total. CascadeVerificationService caps at 50 per table; here we feed it 2.
|
||||
Assert.Equal(2, harness.ProgressFake.LastObservedResidualCount);
|
||||
// Sample is JSON-encoded array; deserialize and assert the cap-respecting shape.
|
||||
var parsed = System.Text.Json.JsonDocument.Parse(harness.ProgressFake.LastResidualSampleJson!);
|
||||
Assert.Equal(System.Text.Json.JsonValueKind.Array, parsed.RootElement.ValueKind);
|
||||
Assert.Equal(2, parsed.RootElement.GetArrayLength());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Verification_failure_persists_OBSERVED_residual_count_matching_verifier_cap()
|
||||
{
|
||||
// Codex F2 round-3: the verifier (CascadeVerificationService) caps each table at
|
||||
// MaxResidualSampleSize=50 AND short-circuits after the first failing table, so
|
||||
// verification.Residuals.Count is the OBSERVED count, not the true total. We honestly
|
||||
// model that — the handler persists what it actually receives. The UI / OpsEvent layer
|
||||
// surfaces this as a "≥ N" lower bound when N equals the cap.
|
||||
var harness = new Harness();
|
||||
harness.SetHappyPath();
|
||||
harness.SetFullSessionManifest();
|
||||
// Simulate the production behaviour: the verifier already capped at 50 internally and
|
||||
// returned exactly that many entries; the real residual count in the database may be
|
||||
// larger but the handler cannot know that.
|
||||
const int observedAtCap = 50;
|
||||
var residualsAtCap = Enumerable.Range(0, observedAtCap)
|
||||
.Select(i => new CascadeResidualKey("Events", "pk", $"ghost-{i}"))
|
||||
.ToList();
|
||||
harness.Verifier.Setup(v => v.VerifyAsync(It.IsAny<DeletionManifest>(), It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(new CascadeVerificationResult(isClean: false, residuals: residualsAtCap));
|
||||
|
||||
await Assert.ThrowsAsync<InvalidOperationException>(() => harness.Sut.HandleAsync(harness.Envelope));
|
||||
|
||||
Assert.Equal(observedAtCap, harness.ProgressFake.LastObservedResidualCount);
|
||||
// Sample length matches the observed count (both share the same cap today). The fields
|
||||
// are kept separate because a future verifier upgrade may decouple them.
|
||||
var sample = System.Text.Json.JsonDocument.Parse(harness.ProgressFake.LastResidualSampleJson!);
|
||||
Assert.Equal(observedAtCap, sample.RootElement.GetArrayLength());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Step_exception_persists_LastFailure_into_DeletionProgress_for_OpsEvent_enrichment()
|
||||
{
|
||||
// Codex F4: same contract for the step-exception path. Step failures bubble up via the
|
||||
// worker's retry/poison loop, so the durable OpsEvent record must carry failureType +
|
||||
// failureMessage so operators can triage without grovelling through App Insights.
|
||||
var harness = new Harness();
|
||||
harness.SetHappyPath();
|
||||
harness.SetFullSessionManifest();
|
||||
harness.Storage.Setup(s => s.DeleteByExactKeysInBatchesAsync(
|
||||
Constants.TableNames.Events, It.IsAny<IReadOnlyList<(string, string)>>(), It.IsAny<CancellationToken>()))
|
||||
.ThrowsAsync(new RequestFailedException(503, "ServiceUnavailable"));
|
||||
|
||||
await Assert.ThrowsAsync<RequestFailedException>(() => harness.Sut.HandleAsync(harness.Envelope));
|
||||
|
||||
Assert.Equal("step_exception", harness.ProgressFake.LastFailureType);
|
||||
Assert.Contains("RequestFailedException", harness.ProgressFake.LastFailureMessage ?? string.Empty);
|
||||
Assert.Null(harness.ProgressFake.LastResidualSampleJson); // step path has no residuals
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -416,19 +508,24 @@ public class SessionDeletionHandlerTests
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Handler_audits_deletion_step_failed_when_table_delete_throws()
|
||||
public async Task Handler_rethrows_step_failure_without_emitting_tenant_audit()
|
||||
{
|
||||
// PR-B audit consolidation: step failures used to write deletion_step_failed +
|
||||
// deletion_step_completed into the tenant audit. Both now route to structured logs;
|
||||
// the failure surfaces to the worker which routes the poisoned cascade to the
|
||||
// SessionDeletionPoisoned OpsEvent. Tenant-visible signal is the absence of
|
||||
// deletion_completed.
|
||||
var harness = new Harness();
|
||||
harness.SetHappyPath();
|
||||
harness.SetFullSessionManifest();
|
||||
// First delete throws — Handler must audit and rethrow.
|
||||
harness.Storage.Setup(s => s.DeleteByExactKeysInBatchesAsync(
|
||||
Constants.TableNames.Events, It.IsAny<IReadOnlyList<(string, string)>>(), It.IsAny<CancellationToken>()))
|
||||
.ThrowsAsync(new RequestFailedException(503, "ServiceUnavailable"));
|
||||
|
||||
await Assert.ThrowsAsync<RequestFailedException>(() => harness.Sut.HandleAsync(harness.Envelope));
|
||||
|
||||
Assert.Contains(harness.AuditCalls, a => a.Action == "deletion_step_failed");
|
||||
Assert.DoesNotContain(harness.AuditCalls, a => a.Action == "deletion_step_failed");
|
||||
Assert.DoesNotContain(harness.AuditCalls, a => a.Action == "deletion_step_completed");
|
||||
Assert.DoesNotContain(harness.AuditCalls, a => a.Action == "deletion_completed");
|
||||
Assert.Empty(harness.SignalR.SessionDeletedCalls);
|
||||
}
|
||||
@@ -555,6 +652,12 @@ public class SessionDeletionHandlerTests
|
||||
ProgressFake.RestoreReIncrementsApplied = p.RestoreReIncrementsApplied == null
|
||||
? null : new HashSet<string>(p.RestoreReIncrementsApplied, StringComparer.Ordinal);
|
||||
ProgressFake.TombstoneStarted = p.TombstoneStarted;
|
||||
// PR-B Codex F4: propagate the new last-failure fields so tests can verify
|
||||
// the handler persists them before throwing on step / verification failures.
|
||||
ProgressFake.LastFailureType = p.LastFailureType;
|
||||
ProgressFake.LastFailureMessage = p.LastFailureMessage;
|
||||
ProgressFake.LastObservedResidualCount = p.LastObservedResidualCount;
|
||||
ProgressFake.LastResidualSampleJson = p.LastResidualSampleJson;
|
||||
return "\"0xFAKE_ETAG_2\"";
|
||||
});
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@ using System;
|
||||
using System.Collections.Generic;
|
||||
using System.IO;
|
||||
using System.Linq;
|
||||
using System.Net.Http;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using Azure;
|
||||
@@ -650,10 +651,29 @@ public class SessionDeletionPr4cCodexFixesTests
|
||||
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<Dictionary<string, string>?>()))
|
||||
.ReturnsAsync(true);
|
||||
|
||||
// PR-B audit consolidation: SessionDeletionWorker no longer takes IMaintenanceRepository;
|
||||
// poisoned cascades flow through OpsEventService. Build a fire-and-forget instance
|
||||
// backed by a noop repository — F5 tests assert CAS ordering, not the OpsEvent itself.
|
||||
var opsRepoStub = new Mock<IOpsEventRepository>();
|
||||
opsRepoStub.Setup(r => r.SaveOpsEventAsync(It.IsAny<OpsEventEntry>())).Returns(Task.CompletedTask);
|
||||
var alertDispatchStub = new OpsAlertDispatchService(
|
||||
AdminConfig.Object,
|
||||
new TelegramNotificationService(new HttpClient(), Mock.Of<IConfigRepository>(), NullLogger<TelegramNotificationService>.Instance),
|
||||
new AutopilotMonitor.Functions.Services.Notifications.WebhookNotificationService(new HttpClient(), NullLogger<AutopilotMonitor.Functions.Services.Notifications.WebhookNotificationService>.Instance),
|
||||
NullLogger<OpsAlertDispatchService>.Instance);
|
||||
var opsServiceStub = new OpsEventService(opsRepoStub.Object, NullLogger<OpsEventService>.Instance, alertDispatchStub);
|
||||
|
||||
// PR-B Codex F4 follow-up: worker now reads DeletionProgress on the poison path.
|
||||
// F5 tests assert CAS ordering, not the OpsEvent payload, so a noop blob suffices.
|
||||
blobMock.Setup(b => b.DownloadDeletionProgressAsync(
|
||||
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<string>(), It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(((AutopilotMonitor.Shared.Models.Deletion.DeletionProgress, string))(
|
||||
new AutopilotMonitor.Shared.Models.Deletion.DeletionProgress(), "etag-stub"));
|
||||
|
||||
Sut = new SessionDeletionWorker(
|
||||
MainQueue.Object, PoisonQueue.Object,
|
||||
handlerMock.Object, StorageMock.Object,
|
||||
AdminConfig.Object, Maintenance.Object,
|
||||
AdminConfig.Object, blobMock.Object, opsServiceStub,
|
||||
NullLogger<SessionDeletionWorker>.Instance,
|
||||
heartbeatInterval: TimeSpan.FromMilliseconds(200),
|
||||
pollInterval: TimeSpan.FromMilliseconds(50));
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using System.Net.Http;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using Azure;
|
||||
@@ -9,6 +10,7 @@ using Azure.Storage.Queues;
|
||||
using Azure.Storage.Queues.Models;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using AutopilotMonitor.Functions.Services.Deletion;
|
||||
using AutopilotMonitor.Functions.Services.Notifications;
|
||||
using AutopilotMonitor.Shared.DataAccess;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using AutopilotMonitor.Shared.Models.Deletion;
|
||||
@@ -49,7 +51,7 @@ public class SessionDeletionWorkerTests
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Worker_moves_message_to_poison_after_max_dequeue_and_audits()
|
||||
public async Task Worker_moves_message_to_poison_after_max_dequeue_and_emits_ops_event()
|
||||
{
|
||||
var harness = new Harness();
|
||||
var envelope = new SessionDeletionEnvelope
|
||||
@@ -68,9 +70,256 @@ public class SessionDeletionWorkerTests
|
||||
harness.MainQueue.Verify(q => q.DeleteMessageAsync(
|
||||
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<CancellationToken>()),
|
||||
Times.AtLeastOnce);
|
||||
// Audit deletion_poisoned written with the parsed envelope's manifestId.
|
||||
Assert.Contains(harness.AuditCalls, a => a.Action == "deletion_poisoned"
|
||||
&& a.Details != null && a.Details["manifestId"] == ManifestId);
|
||||
// PR-B audit consolidation: deletion_poisoned audit moved to SessionDeletionPoisoned
|
||||
// OpsEvent (operator-bound, Telegram-routable). Tenant-audit row is no longer written.
|
||||
var poisoned = Assert.Single(harness.OpsEventCalls, e => e.EventType == "SessionDeletionPoisoned");
|
||||
Assert.Equal(TenantId, poisoned.TenantId);
|
||||
Assert.Equal(SessionId, poisoned.SessionId);
|
||||
Assert.Equal(ManifestId, poisoned.ManifestId);
|
||||
Assert.DoesNotContain(harness.AuditCalls, a => a.Action == "deletion_poisoned");
|
||||
// Codex F4: poison OpsEvent carries the handler's last-failure breadcrumb from the
|
||||
// DeletionProgress blob (harness seeds a verification-failure scenario by default).
|
||||
Assert.Contains("verification_residuals", poisoned.RawDetails);
|
||||
Assert.Contains("residual row(s)", poisoned.RawDetails);
|
||||
// Codex F2 round-3: OpsEvent carries the verifier's OBSERVED residual count, not a
|
||||
// hypothetical "true" total — CascadeVerificationService short-circuits at the first
|
||||
// failing table and caps each table's sample at MaxResidualSampleSize. The degenerate
|
||||
// residualSampleSize key was dropped because it always equalled observedResidualCount.
|
||||
Assert.Contains("\"observedResidualCount\":2", poisoned.RawDetails);
|
||||
Assert.DoesNotContain("\"residualSampleSize\":", poisoned.RawDetails);
|
||||
Assert.DoesNotContain("\"residualCount\":", poisoned.RawDetails); // old, misleading key
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Worker_OpsEvent_carries_observed_count_at_verifier_cap_for_large_failures()
|
||||
{
|
||||
// Codex F2 round-3: when the real residual mountain is bigger than the verifier cap,
|
||||
// verification.Residuals.Count = MaxResidualSampleSize (50) and that's the only number
|
||||
// we have. The worker forwards exactly that. UI surface treats "==cap" as a "≥" lower
|
||||
// bound; the OpsEvent itself stays honest and just reports what was observed.
|
||||
var harness = new Harness();
|
||||
var envelope = new SessionDeletionEnvelope
|
||||
{
|
||||
TenantId = TenantId, SessionId = SessionId, ManifestId = ManifestId,
|
||||
Reason = "admin_delete", EnqueuedAt = DateTime.UtcNow,
|
||||
};
|
||||
harness.EnqueueMessage(JsonConvert.SerializeObject(envelope), dequeueCount: SessionDeletionWorker.MaxDequeueCount + 1);
|
||||
|
||||
const int capObservation = 50;
|
||||
var sample = string.Join(",", Enumerable.Range(0, capObservation)
|
||||
.Select(i => $"{{\"table\":\"Events\",\"pk\":\"t\",\"rk\":\"r{i}\"}}"));
|
||||
harness.BlobMock.Setup(b => b.DownloadDeletionProgressAsync(
|
||||
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<string>(), It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(((AutopilotMonitor.Shared.Models.Deletion.DeletionProgress, string))(
|
||||
new AutopilotMonitor.Shared.Models.Deletion.DeletionProgress
|
||||
{
|
||||
LastFailureType = "verification_residuals",
|
||||
LastFailureMessage = $"{capObservation} observed residual row(s); refusing to tombstone",
|
||||
LastObservedResidualCount = capObservation,
|
||||
LastResidualSampleJson = $"[{sample}]",
|
||||
},
|
||||
"etag-mock"));
|
||||
|
||||
await harness.RunForAsync(TimeSpan.FromMilliseconds(500));
|
||||
|
||||
var poisoned = Assert.Single(harness.OpsEventCalls, e => e.EventType == "SessionDeletionPoisoned");
|
||||
Assert.Contains($"\"observedResidualCount\":{capObservation}", poisoned.RawDetails);
|
||||
// OpsEvents table truncates Details at 4096 chars (TableOpsEventRepository.cs); the
|
||||
// preview cap (OpsEventResidualSamplePreviewSize=5) keeps us safely under that even
|
||||
// with a 50-entry progress-blob sample.
|
||||
Assert.True(poisoned.RawDetails.Length < 4096,
|
||||
$"OpsEvent Details exceeded 4096 chars ({poisoned.RawDetails.Length}); the table repository will truncate mid-JSON.");
|
||||
// Preview JSON has at most OpsEventResidualSamplePreviewSize entries.
|
||||
var match = System.Text.RegularExpressions.Regex.Match(
|
||||
poisoned.RawDetails,
|
||||
"\"residualSamplePreviewJson\":\"(?<json>(?:[^\"\\\\]|\\\\.)*)\"");
|
||||
Assert.True(match.Success, "Expected residualSamplePreviewJson key in OpsEvent details.");
|
||||
var previewJson = System.Text.RegularExpressions.Regex.Unescape(match.Groups["json"].Value);
|
||||
using var previewDoc = System.Text.Json.JsonDocument.Parse(previewJson);
|
||||
Assert.InRange(previewDoc.RootElement.GetArrayLength(), 1,
|
||||
AutopilotMonitor.Shared.Models.Deletion.DeletionProgressConstants.OpsEventResidualSamplePreviewSize);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Worker_OpsEvent_falls_back_to_sample_length_when_observed_count_missing_pre_followup_blob()
|
||||
{
|
||||
// Back-compat: a progress blob written before LastObservedResidualCount existed (this
|
||||
// field was added in a Codex follow-up to PR-B) still produces a useful OpsEvent — the
|
||||
// worker derives the number from the sample-array length so observers don't see a null
|
||||
// observedResidualCount when reading historical poisoned cascades.
|
||||
var harness = new Harness();
|
||||
var envelope = new SessionDeletionEnvelope
|
||||
{
|
||||
TenantId = TenantId, SessionId = SessionId, ManifestId = ManifestId,
|
||||
Reason = "admin_delete", EnqueuedAt = DateTime.UtcNow,
|
||||
};
|
||||
harness.EnqueueMessage(JsonConvert.SerializeObject(envelope), dequeueCount: SessionDeletionWorker.MaxDequeueCount + 1);
|
||||
|
||||
harness.BlobMock.Setup(b => b.DownloadDeletionProgressAsync(
|
||||
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<string>(), It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(((AutopilotMonitor.Shared.Models.Deletion.DeletionProgress, string))(
|
||||
new AutopilotMonitor.Shared.Models.Deletion.DeletionProgress
|
||||
{
|
||||
LastFailureType = "verification_residuals",
|
||||
LastFailureMessage = "observed residual row(s); refusing to tombstone",
|
||||
LastObservedResidualCount = null, // pre-followup blob — field did not exist
|
||||
LastResidualSampleJson = "[{\"table\":\"Events\",\"pk\":\"t\",\"rk\":\"r1\"},{\"table\":\"Events\",\"pk\":\"t\",\"rk\":\"r2\"},{\"table\":\"Events\",\"pk\":\"t\",\"rk\":\"r3\"}]",
|
||||
},
|
||||
"etag-mock"));
|
||||
|
||||
await harness.RunForAsync(TimeSpan.FromMilliseconds(500));
|
||||
|
||||
var poisoned = Assert.Single(harness.OpsEventCalls, e => e.EventType == "SessionDeletionPoisoned");
|
||||
Assert.Contains("\"observedResidualCount\":3", poisoned.RawDetails);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ShrinkResidualSampleForOpsEvent_caps_array_at_preview_size()
|
||||
{
|
||||
// Unit-level pin on the helper so the cap can't silently drift if someone changes the
|
||||
// constant or the serialization plumbing.
|
||||
var full = string.Join(",", Enumerable.Range(0, 50)
|
||||
.Select(i => $"{{\"table\":\"Events\",\"pk\":\"tenant\",\"rk\":\"row-{i}\"}}"));
|
||||
var shrunk = SessionDeletionWorker.ShrinkResidualSampleForOpsEvent($"[{full}]");
|
||||
Assert.NotNull(shrunk);
|
||||
using var doc = System.Text.Json.JsonDocument.Parse(shrunk!);
|
||||
Assert.Equal(System.Text.Json.JsonValueKind.Array, doc.RootElement.ValueKind);
|
||||
Assert.Equal(
|
||||
AutopilotMonitor.Shared.Models.Deletion.DeletionProgressConstants.OpsEventResidualSamplePreviewSize,
|
||||
doc.RootElement.GetArrayLength());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ShrinkResidualSampleForOpsEvent_returns_null_for_malformed_input()
|
||||
{
|
||||
Assert.Null(SessionDeletionWorker.ShrinkResidualSampleForOpsEvent(null));
|
||||
Assert.Null(SessionDeletionWorker.ShrinkResidualSampleForOpsEvent(""));
|
||||
Assert.Null(SessionDeletionWorker.ShrinkResidualSampleForOpsEvent("not-json"));
|
||||
Assert.Null(SessionDeletionWorker.ShrinkResidualSampleForOpsEvent("{\"not\":\"an array\"}"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ShrinkResidualSampleForOpsEvent_trims_per_field_with_ellipsis_marker()
|
||||
{
|
||||
// Codex follow-up: a single entry with a 200-char composite PK could on its own consume
|
||||
// most of the OpsEvent Details budget. The helper trims each field to
|
||||
// OpsEventResidualKeyMaxChars and appends `…` so the truncation is operator-visible.
|
||||
var maxKeyChars = AutopilotMonitor.Shared.Models.Deletion.DeletionProgressConstants.OpsEventResidualKeyMaxChars;
|
||||
var longPk = new string('p', maxKeyChars * 3);
|
||||
var longRk = new string('r', maxKeyChars * 3);
|
||||
var input = $"[{{\"table\":\"Events\",\"pk\":\"{longPk}\",\"rk\":\"{longRk}\"}}]";
|
||||
|
||||
var shrunk = SessionDeletionWorker.ShrinkResidualSampleForOpsEvent(input);
|
||||
Assert.NotNull(shrunk);
|
||||
|
||||
using var doc = System.Text.Json.JsonDocument.Parse(shrunk!);
|
||||
var first = doc.RootElement[0];
|
||||
var pk = first.GetProperty("pk").GetString()!;
|
||||
var rk = first.GetProperty("rk").GetString()!;
|
||||
Assert.Equal(maxKeyChars, pk.Length);
|
||||
Assert.Equal(maxKeyChars, rk.Length);
|
||||
Assert.EndsWith("…", pk);
|
||||
Assert.EndsWith("…", rk);
|
||||
Assert.StartsWith("ppp", pk);
|
||||
Assert.StartsWith("rrr", rk);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ShrinkResidualSampleForOpsEvent_short_fields_pass_through_unchanged()
|
||||
{
|
||||
// Don't paint truncation markers on values that fit naturally.
|
||||
var input = "[{\"table\":\"Events\",\"pk\":\"short-pk\",\"rk\":\"short-rk\"}]";
|
||||
var shrunk = SessionDeletionWorker.ShrinkResidualSampleForOpsEvent(input);
|
||||
Assert.NotNull(shrunk);
|
||||
using var doc = System.Text.Json.JsonDocument.Parse(shrunk!);
|
||||
var first = doc.RootElement[0];
|
||||
Assert.Equal("short-pk", first.GetProperty("pk").GetString());
|
||||
Assert.Equal("short-rk", first.GetProperty("rk").GetString());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ShrinkResidualSampleForOpsEvent_drops_trailing_entries_when_over_total_budget()
|
||||
{
|
||||
// Even with per-field trimming, max-length entries can collectively exceed the preview
|
||||
// budget (~1200 chars). The helper drops trailing entries one at a time until the JSON
|
||||
// fits — operators always see SOMETHING parseable, the full sample stays in the blob.
|
||||
var maxKeyChars = AutopilotMonitor.Shared.Models.Deletion.DeletionProgressConstants.OpsEventResidualKeyMaxChars;
|
||||
var budget = AutopilotMonitor.Shared.Models.Deletion.DeletionProgressConstants.OpsEventResidualPreviewBudgetChars;
|
||||
var maxEntries = AutopilotMonitor.Shared.Models.Deletion.DeletionProgressConstants.OpsEventResidualSamplePreviewSize;
|
||||
// Each entry takes ~(20 table + 96 pk + 96 rk + 30 JSON shape) ≈ 240 chars; 5 entries
|
||||
// would land at ~1200, right at the budget. Make table/pk/rk all max-length so we land
|
||||
// slightly OVER and force a drop.
|
||||
var longTable = new string('T', maxKeyChars);
|
||||
var longPk = new string('p', maxKeyChars);
|
||||
var longRk = new string('r', maxKeyChars);
|
||||
var entry = $"{{\"table\":\"{longTable}\",\"pk\":\"{longPk}\",\"rk\":\"{longRk}\"}}";
|
||||
var input = "[" + string.Join(",", Enumerable.Range(0, maxEntries).Select(_ => entry)) + "]";
|
||||
|
||||
var shrunk = SessionDeletionWorker.ShrinkResidualSampleForOpsEvent(input);
|
||||
Assert.NotNull(shrunk);
|
||||
Assert.True(shrunk!.Length <= budget,
|
||||
$"Preview exceeded total budget: {shrunk.Length} > {budget}.");
|
||||
|
||||
using var doc = System.Text.Json.JsonDocument.Parse(shrunk!);
|
||||
var kept = doc.RootElement.GetArrayLength();
|
||||
// We must have kept at least one entry (a single entry is small enough to fit), and
|
||||
// dropped at least one (otherwise the budget wasn't actually enforced).
|
||||
Assert.InRange(kept, 1, maxEntries - 1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Worker_OpsEvent_stays_under_4096_chars_with_worst_case_long_keys()
|
||||
{
|
||||
// Codex end-to-end pin: the full pipeline (handler → progress blob → worker shrink →
|
||||
// OpsEvent JSON envelope → table-row truncate) must NEVER produce a Details string > 4096
|
||||
// chars, because TableOpsEventRepository hard-truncates at that boundary and would corrupt
|
||||
// the JSON mid-string. We simulate the worst case: max-length PK + RK on every sample
|
||||
// entry, plus a max-length failureMessage.
|
||||
var harness = new Harness();
|
||||
var envelope = new SessionDeletionEnvelope
|
||||
{
|
||||
TenantId = TenantId, SessionId = SessionId, ManifestId = ManifestId,
|
||||
Reason = "admin_delete", EnqueuedAt = DateTime.UtcNow,
|
||||
};
|
||||
harness.EnqueueMessage(JsonConvert.SerializeObject(envelope), dequeueCount: SessionDeletionWorker.MaxDequeueCount + 1);
|
||||
|
||||
// 50 sample entries × ~200-char composite keys = ~10 KB blob; the worker must defend.
|
||||
const int progressSampleSize = 50;
|
||||
const int hugeKeyChars = 240;
|
||||
var hugeTable = new string('T', hugeKeyChars);
|
||||
var hugePk = new string('p', hugeKeyChars);
|
||||
var hugeRk = new string('r', hugeKeyChars);
|
||||
var sample = string.Join(",", Enumerable.Range(0, progressSampleSize)
|
||||
.Select(_ => $"{{\"table\":\"{hugeTable}\",\"pk\":\"{hugePk}\",\"rk\":\"{hugeRk}\"}}"));
|
||||
// Max-length failure message too.
|
||||
var longMessage = new string('m', 1024);
|
||||
|
||||
harness.BlobMock.Setup(b => b.DownloadDeletionProgressAsync(
|
||||
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<string>(), It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(((AutopilotMonitor.Shared.Models.Deletion.DeletionProgress, string))(
|
||||
new AutopilotMonitor.Shared.Models.Deletion.DeletionProgress
|
||||
{
|
||||
LastFailureType = "verification_residuals",
|
||||
LastFailureMessage = longMessage,
|
||||
LastObservedResidualCount = progressSampleSize,
|
||||
LastResidualSampleJson = $"[{sample}]",
|
||||
},
|
||||
"etag-mock"));
|
||||
|
||||
await harness.RunForAsync(TimeSpan.FromMilliseconds(500));
|
||||
|
||||
var poisoned = Assert.Single(harness.OpsEventCalls, e => e.EventType == "SessionDeletionPoisoned");
|
||||
Assert.True(poisoned.RawDetails.Length < 4096,
|
||||
$"OpsEvent Details exceeded 4096 chars ({poisoned.RawDetails.Length}); TableOpsEventRepository would truncate mid-JSON.");
|
||||
|
||||
// Sanity check that the preview is still parseable as JSON after the trim+budget pass.
|
||||
var match = System.Text.RegularExpressions.Regex.Match(
|
||||
poisoned.RawDetails,
|
||||
"\"residualSamplePreviewJson\":\"(?<json>(?:[^\"\\\\]|\\\\.)*)\"");
|
||||
Assert.True(match.Success, "Expected residualSamplePreviewJson key in OpsEvent details.");
|
||||
var previewJson = System.Text.RegularExpressions.Regex.Unescape(match.Groups["json"].Value);
|
||||
using var previewDoc = System.Text.Json.JsonDocument.Parse(previewJson);
|
||||
Assert.Equal(System.Text.Json.JsonValueKind.Array, previewDoc.RootElement.ValueKind);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -198,7 +447,7 @@ public class SessionDeletionWorkerTests
|
||||
|
||||
harness.PoisonQueue.Verify(q => q.SendMessageAsync(It.IsAny<string>(), It.IsAny<CancellationToken>()),
|
||||
Times.AtLeastOnce);
|
||||
Assert.Contains(harness.AuditCalls, a => a.Action == "deletion_poisoned");
|
||||
Assert.Contains(harness.OpsEventCalls, e => e.EventType == "SessionDeletionPoisoned");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -280,8 +529,11 @@ public class SessionDeletionWorkerTests
|
||||
public Mock<QueueClient> PoisonQueue { get; }
|
||||
public Mock<SessionDeletionHandler> HandlerMock { get; }
|
||||
public Mock<AdminConfigurationService> AdminConfig { get; }
|
||||
public Mock<IMaintenanceRepository> Maintenance { get; }
|
||||
// PR-B audit consolidation: deletion_poisoned tenant audit moved to SessionDeletionPoisoned
|
||||
// OpsEvent. We keep AuditCalls as a passthrough capture (always empty for the worker
|
||||
// post-PR-B) so existing assertions like DoesNotContain still compile.
|
||||
public List<AuditEntry> AuditCalls { get; } = new List<AuditEntry>();
|
||||
public List<CapturedOpsEvent> OpsEventCalls { get; } = new List<CapturedOpsEvent>();
|
||||
public SessionDeletionWorker Sut { get; }
|
||||
|
||||
private readonly Queue<QueueMessage> _pendingMessages = new Queue<QueueMessage>();
|
||||
@@ -362,6 +614,24 @@ public class SessionDeletionWorkerTests
|
||||
var blobMock = new Mock<BlobStorageService>(
|
||||
new Azure.Storage.Blobs.BlobServiceClient("UseDevelopmentStorage=true"),
|
||||
NullLogger<BlobStorageService>.Instance, false);
|
||||
// PR-B Codex F4 follow-up: worker reads DeletionProgress on the poison path so the
|
||||
// SessionDeletionPoisoned OpsEvent can include the handler's last-failure breadcrumb.
|
||||
// Default mock returns a populated progress to exercise the enrichment path.
|
||||
blobMock.Setup(b => b.DownloadDeletionProgressAsync(
|
||||
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<string>(), It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(((AutopilotMonitor.Shared.Models.Deletion.DeletionProgress, string))(
|
||||
new AutopilotMonitor.Shared.Models.Deletion.DeletionProgress
|
||||
{
|
||||
LastFailureType = "verification_residuals",
|
||||
LastFailureMessage = "2 observed residual row(s)",
|
||||
// Codex F2 round-3: the persisted count is the verifier's OBSERVED count
|
||||
// (capped at MaxResidualSampleSize=50 per table + short-circuit at first
|
||||
// failing table). Default mock uses a low number so the field matches the
|
||||
// sample length; the cap-hit test below exercises the lower-bound case.
|
||||
LastObservedResidualCount = 2,
|
||||
LastResidualSampleJson = "[{\"table\":\"Events\",\"pk\":\"t\",\"rk\":\"r\"},{\"table\":\"Signals\",\"pk\":\"t\",\"rk\":\"r2\"}]",
|
||||
},
|
||||
"etag-mock"));
|
||||
var verifierMock = new Mock<CascadeVerificationService>(
|
||||
Mock.Of<ISessionDeletionInventoryReader>(),
|
||||
NullLogger<CascadeVerificationService>.Instance);
|
||||
@@ -384,28 +654,37 @@ public class SessionDeletionWorkerTests
|
||||
AdminConfig.Setup(a => a.IsSessionDeletionKillSwitchActiveAsync())
|
||||
.ReturnsAsync(false);
|
||||
|
||||
Maintenance = new Mock<IMaintenanceRepository>();
|
||||
Maintenance.Setup(m => m.LogAuditEntryAsync(
|
||||
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<string>(),
|
||||
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<Dictionary<string, string>?>()))
|
||||
.Returns<string, string, string, string, string, Dictionary<string, string>?>(
|
||||
(tenantId, action, entityType, entityId, performedBy, details) =>
|
||||
{
|
||||
AuditCalls.Add(new AuditEntry(tenantId, action, entityType, entityId, performedBy, details));
|
||||
return Task.FromResult(true);
|
||||
});
|
||||
// PR-B audit consolidation: build a real OpsEventService backed by a captured-list
|
||||
// mock repository so the worker's RecordSessionDeletionPoisonedAsync call is
|
||||
// observable as an OpsEventEntry on the OpsEventCalls list.
|
||||
var opsRepo = new Mock<IOpsEventRepository>();
|
||||
opsRepo.Setup(r => r.SaveOpsEventAsync(It.IsAny<OpsEventEntry>()))
|
||||
.Returns<OpsEventEntry>(e =>
|
||||
{
|
||||
OpsEventCalls.Add(CapturedOpsEvent.From(e));
|
||||
return Task.CompletedTask;
|
||||
});
|
||||
var alertDispatch = new OpsAlertDispatchService(
|
||||
AdminConfig.Object,
|
||||
new TelegramNotificationService(new HttpClient(), Mock.Of<IConfigRepository>(), NullLogger<TelegramNotificationService>.Instance),
|
||||
new AutopilotMonitor.Functions.Services.Notifications.WebhookNotificationService(new HttpClient(), NullLogger<AutopilotMonitor.Functions.Services.Notifications.WebhookNotificationService>.Instance),
|
||||
NullLogger<OpsAlertDispatchService>.Instance);
|
||||
var opsService = new OpsEventService(opsRepo.Object, NullLogger<OpsEventService>.Instance, alertDispatch);
|
||||
|
||||
Sut = new SessionDeletionWorker(
|
||||
MainQueue.Object, PoisonQueue.Object,
|
||||
HandlerMock.Object, storageMock.Object,
|
||||
AdminConfig.Object, Maintenance.Object,
|
||||
AdminConfig.Object, blobMock.Object, opsService,
|
||||
NullLogger<SessionDeletionWorker>.Instance,
|
||||
heartbeatInterval: heartbeatInterval ?? TimeSpan.FromMilliseconds(200),
|
||||
pollInterval: TimeSpan.FromMilliseconds(50));
|
||||
|
||||
StorageMock = storageMock;
|
||||
BlobMock = blobMock;
|
||||
}
|
||||
|
||||
public Mock<BlobStorageService> BlobMock { get; private set; } = null!;
|
||||
|
||||
public Mock<TableStorageService> StorageMock { get; private set; } = null!;
|
||||
|
||||
public void SetKillSwitch(bool active)
|
||||
@@ -445,4 +724,32 @@ public class SessionDeletionWorkerTests
|
||||
private sealed record AuditEntry(
|
||||
string TenantId, string Action, string EntityType, string EntityId, string PerformedBy,
|
||||
Dictionary<string, string>? Details);
|
||||
|
||||
/// <summary>
|
||||
/// Flattened OpsEventEntry projection for assertions. <see cref="ManifestId"/> + the other
|
||||
/// strongly-typed accessors are pulled from <c>Details</c> JSON so tests don't have to
|
||||
/// re-deserialise.
|
||||
/// </summary>
|
||||
private sealed record CapturedOpsEvent(
|
||||
string EventType, string Severity, string Message,
|
||||
string? TenantId, string? SessionId, string? ManifestId, string RawDetails)
|
||||
{
|
||||
public static CapturedOpsEvent From(OpsEventEntry e)
|
||||
{
|
||||
string? tenantId = e.TenantId;
|
||||
string? sessionId = null;
|
||||
string? manifestId = null;
|
||||
if (!string.IsNullOrEmpty(e.Details))
|
||||
{
|
||||
try
|
||||
{
|
||||
using var doc = System.Text.Json.JsonDocument.Parse(e.Details!);
|
||||
if (doc.RootElement.TryGetProperty("sessionId", out var s)) sessionId = s.GetString();
|
||||
if (doc.RootElement.TryGetProperty("manifestId", out var m)) manifestId = m.GetString();
|
||||
}
|
||||
catch (System.Text.Json.JsonException) { /* tolerate malformed test payloads */ }
|
||||
}
|
||||
return new CapturedOpsEvent(e.EventType, e.Severity, e.Message, tenantId, sessionId, manifestId, e.Details ?? string.Empty);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -31,6 +31,45 @@ public class SessionRestoreServiceTests
|
||||
|
||||
// ============================================================ Full restore (8 facts) ====
|
||||
|
||||
[Fact]
|
||||
public async Task Restore_persists_operator_reason_into_audit_details()
|
||||
{
|
||||
// Codex follow-up: the Session Cleanup admin dialog sends a free-text `reason` so the
|
||||
// operator's intent is recoverable from the tenant audit trail. Pre-fix the backend
|
||||
// accepted the field but the service dropped it on the floor.
|
||||
var harness = new Harness();
|
||||
harness.SetCompletedCascade();
|
||||
|
||||
await harness.Sut.RestoreAsync(
|
||||
TenantId, SessionId, ManifestId,
|
||||
dryRun: false, actor: "ga@example.com",
|
||||
operatorReason: "customer ticket ABC-123 — accidental delete");
|
||||
|
||||
var restored = Assert.Single(harness.AuditCalls, a => a.Action == "deletion_restored");
|
||||
Assert.NotNull(restored.Details);
|
||||
Assert.True(restored.Details!.TryGetValue("reason", out var reasonValue));
|
||||
Assert.Equal("customer ticket ABC-123 — accidental delete", reasonValue);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Restore_omits_reason_detail_when_caller_supplied_none()
|
||||
{
|
||||
// Sister to the above — a null/empty reason must NOT add a stray `reason=""` key, since
|
||||
// downstream consumers (audit-log UI filters, kusto queries) treat the key's presence as
|
||||
// signal that the operator deliberately recorded intent.
|
||||
var harness = new Harness();
|
||||
harness.SetCompletedCascade();
|
||||
|
||||
await harness.Sut.RestoreAsync(
|
||||
TenantId, SessionId, ManifestId,
|
||||
dryRun: false, actor: "ga@example.com",
|
||||
operatorReason: null);
|
||||
|
||||
var restored = Assert.Single(harness.AuditCalls, a => a.Action == "deletion_restored");
|
||||
Assert.NotNull(restored.Details);
|
||||
Assert.False(restored.Details!.ContainsKey("reason"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Restore_full_round_trips_session_after_completed_cascade()
|
||||
{
|
||||
@@ -118,18 +157,20 @@ public class SessionRestoreServiceTests
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Restore_full_audits_manifest_download_before_any_read()
|
||||
public async Task Restore_full_does_not_emit_internal_breadcrumb_audit()
|
||||
{
|
||||
// PR-B audit consolidation: deletion_manifest_downloaded was an internal step-breadcrumb
|
||||
// emitted before any read. It doubled the tenant audit row count without adding signal
|
||||
// — the deletion_restored audit on success (or the absence of it on the reject path)
|
||||
// is the operator-relevant outcome.
|
||||
var harness = new Harness();
|
||||
harness.SetCompletedCascade();
|
||||
// Force a post-download reject path so we can verify the audit fired BEFORE the reject.
|
||||
harness.Blob.Setup(b => b.DownloadDeletionManifestWithShaAsync(TenantId, SessionId, ManifestId, It.IsAny<CancellationToken>()))
|
||||
.ThrowsAsync(new InvalidDataException("corruption"));
|
||||
|
||||
await harness.Sut.RestoreAsync(TenantId, SessionId, ManifestId, dryRun: false, actor: "ga@example.com");
|
||||
|
||||
Assert.Contains(harness.AuditCalls, a => a.Action == "deletion_manifest_downloaded"
|
||||
&& a.Details != null && a.Details["reason"] == "restore_attempt");
|
||||
Assert.DoesNotContain(harness.AuditCalls, a => a.Action == "deletion_manifest_downloaded");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -186,7 +227,10 @@ public class SessionRestoreServiceTests
|
||||
var result = await harness.Sut.RestoreAsync(TenantId, SessionId, ManifestId, dryRun: true, actor: "ga@example.com");
|
||||
|
||||
Assert.Equal(SessionRestoreOutcome.DryRunOk, result.Outcome);
|
||||
Assert.Equal("dryRun", result.Mode);
|
||||
// Codex follow-up: dry-run must report the auto-selected mode (full / partial) so the
|
||||
// admin dialog can preview it. The DryRunOk Outcome is the dry-run signal — Mode is the
|
||||
// operator-visible preview of what the real run would do.
|
||||
Assert.Equal("full", result.Mode);
|
||||
Assert.NotEmpty(result.WouldRestoreByTable);
|
||||
// No live writes.
|
||||
harness.Storage.Verify(s => s.RestoreRowsByExactKeysInBatchesAsync(
|
||||
@@ -199,6 +243,24 @@ public class SessionRestoreServiceTests
|
||||
Assert.DoesNotContain(harness.AuditCalls, a => a.Action == "deletion_restored");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Restore_partial_dryRun_reports_partial_mode_for_poisoned_cascade()
|
||||
{
|
||||
// Codex follow-up symmetry test: dry-run on a Poisoned session must preview "partial",
|
||||
// mirroring the completed-cascade test above that previews "full".
|
||||
var harness = new Harness();
|
||||
harness.SetPoisonedCascade();
|
||||
harness.Progress.CompletedSteps.UnionWith(new[] { 1, 2, 3, 4 });
|
||||
|
||||
var result = await harness.Sut.RestoreAsync(TenantId, SessionId, ManifestId, dryRun: true, actor: "ga@example.com");
|
||||
|
||||
Assert.Equal(SessionRestoreOutcome.DryRunOk, result.Outcome);
|
||||
Assert.Equal("partial", result.Mode);
|
||||
harness.Storage.Verify(s => s.RestoreRowsByExactKeysInBatchesAsync(
|
||||
It.IsAny<string>(), It.IsAny<IReadOnlyList<DeletionRowDump>>(), It.IsAny<RestoreMode>(), It.IsAny<CancellationToken>()),
|
||||
Times.Never);
|
||||
}
|
||||
|
||||
// ============================================================ Partial restore (6 facts) ====
|
||||
|
||||
[Fact]
|
||||
|
||||
+5
-1
@@ -95,7 +95,8 @@ namespace AutopilotMonitor.Functions.Functions.Config
|
||||
config.CustomRateLimitRequestsPerMinute != existingConfig.CustomRateLimitRequestsPerMinute ||
|
||||
config.RateLimitRequestsPerMinute != existingConfig.RateLimitRequestsPerMinute ||
|
||||
config.Disabled != existingConfig.Disabled ||
|
||||
config.ValidateDeviceAssociation != existingConfig.ValidateDeviceAssociation)
|
||||
config.ValidateDeviceAssociation != existingConfig.ValidateDeviceAssociation ||
|
||||
config.EnableCascadeDeleteV2 != existingConfig.EnableCascadeDeleteV2)
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"Tenant Admin {User} attempted to modify GA-only fields for tenant {TenantId}",
|
||||
@@ -116,6 +117,9 @@ namespace AutopilotMonitor.Functions.Functions.Config
|
||||
// RequestContext + TenantConfigurationService). Tracked in
|
||||
// memory/project_devprep_followups.md.
|
||||
config.ValidateDeviceAssociation = existingConfig.ValidateDeviceAssociation;
|
||||
// V2 cascade-delete pipeline activation is a rollout decision; tenant
|
||||
// admins cannot self-opt-in (or out) of the snapshot+restore pathway.
|
||||
config.EnableCascadeDeleteV2 = existingConfig.EnableCascadeDeleteV2;
|
||||
}
|
||||
|
||||
// Safety: if GA gate is off, force UnrestrictedMode to false
|
||||
|
||||
+21
-5
@@ -72,12 +72,28 @@ namespace AutopilotMonitor.Functions.Functions.Sessions
|
||||
return bad;
|
||||
}
|
||||
|
||||
// Resolve tenantId: prefer the JWT-derived target, fall back to SessionsIndex lookup
|
||||
// (Global Admin can access any session by sessionId without knowing the tenantId).
|
||||
var tenantId = requestCtx.TargetTenantId;
|
||||
if (string.IsNullOrEmpty(tenantId) || string.Equals(tenantId, "global", StringComparison.OrdinalIgnoreCase))
|
||||
// Resolve tenantId. Three-tier fallback ordered by Codex F1 review:
|
||||
// 1. Explicit ?tenantId=<guid> on GA calls — only path that lets a Global Admin
|
||||
// preview a session in a tenant other than their JWT home tenant.
|
||||
// 2. JWT TargetTenantId, where the policy middleware mirrors the JWT tid for
|
||||
// non-RouteParam routes. Sufficient for tenant admins acting on their own tenant.
|
||||
// 3. SessionsIndex lookup keyed by sessionId — the historical fallback for the
|
||||
// "global" sentinel; preserved so existing callers without ?tenantId still work.
|
||||
var explicitTenantId = query["tenantId"];
|
||||
string tenantId;
|
||||
if (requestCtx.IsGlobalAdmin
|
||||
&& !string.IsNullOrEmpty(explicitTenantId)
|
||||
&& Guid.TryParse(explicitTenantId, out _))
|
||||
{
|
||||
tenantId = await _sessionRepo.FindSessionTenantIdAsync(sessionId);
|
||||
tenantId = explicitTenantId!;
|
||||
}
|
||||
else
|
||||
{
|
||||
tenantId = requestCtx.TargetTenantId;
|
||||
if (string.IsNullOrEmpty(tenantId) || string.Equals(tenantId, "global", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
tenantId = await _sessionRepo.FindSessionTenantIdAsync(sessionId) ?? string.Empty;
|
||||
}
|
||||
}
|
||||
if (string.IsNullOrEmpty(tenantId))
|
||||
{
|
||||
|
||||
+242
@@ -0,0 +1,242 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.IO;
|
||||
using System.IO.Compression;
|
||||
using System.Linq;
|
||||
using System.Net;
|
||||
using System.Text.Json;
|
||||
using System.Threading.Tasks;
|
||||
using System.Web;
|
||||
using AutopilotMonitor.Functions.Helpers;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using AutopilotMonitor.Shared.Models.Deletion;
|
||||
using Microsoft.Azure.Functions.Worker;
|
||||
using Microsoft.Azure.Functions.Worker.Http;
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Functions.Sessions
|
||||
{
|
||||
/// <summary>
|
||||
/// <c>GET /api/admin/sessions/{sessionId}/deletion-manifest?tenantId=&manifestId=&mode=...</c> —
|
||||
/// reads the persisted cascade-delete snapshot + progress for an in-flight / poisoned cascade.
|
||||
/// Distinct from <see cref="GetSessionDeletePreviewFunction"/>, which builds a fresh dry-run
|
||||
/// manifest from current data: for a running cascade that view is misleading because the live
|
||||
/// data is already partially deleted. The Session Cleanup admin page uses THIS endpoint to
|
||||
/// show the operator the actual snapshot the worker captured at cascade start.
|
||||
/// <para>
|
||||
/// GA-only enforcement comes from <c>EndpointAccessPolicyCatalog</c>. Cross-tenant access is
|
||||
/// the design intent — the caller passes the row's <c>tenantId</c> from the listing endpoint;
|
||||
/// no <c>SessionsIndex</c> fallback because the index row is often the first thing the cascade
|
||||
/// deletes, so we cannot rely on it for a Running/Poisoned manifest.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public class GetSessionDeletionManifestFunction
|
||||
{
|
||||
private readonly ILogger<GetSessionDeletionManifestFunction> _logger;
|
||||
private readonly BlobStorageService _blob;
|
||||
|
||||
public GetSessionDeletionManifestFunction(
|
||||
ILogger<GetSessionDeletionManifestFunction> logger,
|
||||
BlobStorageService blob)
|
||||
{
|
||||
_logger = logger;
|
||||
_blob = blob;
|
||||
}
|
||||
|
||||
[Function("GetSessionDeletionManifest")]
|
||||
public async Task<HttpResponseData> Run(
|
||||
[HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "admin/sessions/{sessionId}/deletion-manifest")]
|
||||
HttpRequestData req,
|
||||
string sessionId)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(sessionId))
|
||||
{
|
||||
return await BadRequest(req, "sessionId is required");
|
||||
}
|
||||
|
||||
var query = HttpUtility.ParseQueryString(req.Url.Query ?? string.Empty);
|
||||
var tenantId = query["tenantId"];
|
||||
var manifestId = query["manifestId"];
|
||||
var mode = (query["mode"] ?? "summary").ToLowerInvariant();
|
||||
if (string.IsNullOrWhiteSpace(tenantId) || !Guid.TryParse(tenantId, out _))
|
||||
{
|
||||
return await BadRequest(req, "Query parameter 'tenantId' must be a GUID.");
|
||||
}
|
||||
if (string.IsNullOrWhiteSpace(manifestId))
|
||||
{
|
||||
return await BadRequest(req, "Query parameter 'manifestId' is required.");
|
||||
}
|
||||
if (mode != "summary" && mode != "full" && mode != "download")
|
||||
{
|
||||
return await BadRequest(req, "mode must be one of: summary, full, download");
|
||||
}
|
||||
|
||||
DeletionManifest manifest;
|
||||
string snapshotSha;
|
||||
try
|
||||
{
|
||||
(manifest, snapshotSha) = await _blob.DownloadDeletionManifestWithShaAsync(
|
||||
tenantId!, sessionId, manifestId!, req.FunctionContext.CancellationToken);
|
||||
}
|
||||
catch (Azure.RequestFailedException rfe) when (rfe.Status == 404)
|
||||
{
|
||||
var notFound = req.CreateResponse(HttpStatusCode.NotFound);
|
||||
await notFound.WriteAsJsonAsync(new
|
||||
{
|
||||
success = false,
|
||||
message = $"Manifest blob not found for tenant={tenantId} session={sessionId} manifestId={manifestId}.",
|
||||
});
|
||||
return notFound;
|
||||
}
|
||||
catch (InvalidDataException ex)
|
||||
{
|
||||
_logger.LogError(ex,
|
||||
"GetSessionDeletionManifest: snapshot integrity failure for tenant={TenantId} session={SessionId} manifestId={ManifestId}",
|
||||
tenantId, sessionId, manifestId);
|
||||
var corrupt = req.CreateResponse(HttpStatusCode.Conflict);
|
||||
await corrupt.WriteAsJsonAsync(new
|
||||
{
|
||||
success = false,
|
||||
message = "Snapshot SHA-256 mismatch — the manifest blob has been tampered with or the producer crashed.",
|
||||
error = ex.Message,
|
||||
});
|
||||
return corrupt;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex,
|
||||
"GetSessionDeletionManifest: unexpected error for tenant={TenantId} session={SessionId} manifestId={ManifestId}",
|
||||
tenantId, sessionId, manifestId);
|
||||
var error = req.CreateResponse(HttpStatusCode.InternalServerError);
|
||||
await error.WriteAsJsonAsync(new { success = false, message = "Internal server error" });
|
||||
return error;
|
||||
}
|
||||
|
||||
// Progress blob — best-effort. A Preparing-phase manifest may exist without a progress
|
||||
// blob (the producer hasn't reached the upload step yet); surface as null and let the
|
||||
// UI render "no progress yet" instead of failing the whole request.
|
||||
DeletionProgress? progress = null;
|
||||
try
|
||||
{
|
||||
var (p, _) = await _blob.DownloadDeletionProgressAsync(
|
||||
tenantId!, sessionId, manifestId!, req.FunctionContext.CancellationToken);
|
||||
progress = p;
|
||||
}
|
||||
catch (Azure.RequestFailedException rfe) when (rfe.Status == 404)
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"GetSessionDeletionManifest: no progress blob yet (likely Preparing phase) for tenant={TenantId} session={SessionId} manifestId={ManifestId}",
|
||||
tenantId, sessionId, manifestId);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex,
|
||||
"GetSessionDeletionManifest: failed to read progress blob — returning manifest only. tenant={TenantId} session={SessionId} manifestId={ManifestId}",
|
||||
tenantId, sessionId, manifestId);
|
||||
}
|
||||
|
||||
return mode switch
|
||||
{
|
||||
"full" => await BuildFullResponse(req, manifest, progress, snapshotSha),
|
||||
"download" => await BuildDownloadResponse(req, manifest, tenantId!, sessionId, manifestId!),
|
||||
_ => await BuildSummaryResponse(req, manifest, progress, snapshotSha),
|
||||
};
|
||||
}
|
||||
|
||||
private static async Task<HttpResponseData> BadRequest(HttpRequestData req, string message)
|
||||
{
|
||||
var bad = req.CreateResponse(HttpStatusCode.BadRequest);
|
||||
await bad.WriteAsJsonAsync(new { success = false, message });
|
||||
return bad;
|
||||
}
|
||||
|
||||
private static async Task<HttpResponseData> BuildSummaryResponse(
|
||||
HttpRequestData req, DeletionManifest manifest, DeletionProgress? progress, string snapshotSha)
|
||||
{
|
||||
long totalRowCount = 0;
|
||||
var sampleKeys = new Dictionary<string, List<object>>(StringComparer.Ordinal);
|
||||
foreach (var step in manifest.Steps)
|
||||
{
|
||||
totalRowCount += step.RowCount;
|
||||
if (step.Rows.Count == 0) continue;
|
||||
var key = step.Table ?? step.Step ?? $"order_{step.Order}";
|
||||
sampleKeys[key] = step.Rows.Take(5).Select(r => (object)new { pk = r.Pk, rk = r.Rk }).ToList();
|
||||
}
|
||||
|
||||
// Serialize once to report bytes — same projection the producer would upload (modulo gzip).
|
||||
var serializedBytes = JsonSerializer.SerializeToUtf8Bytes(manifest, DeletionManifestJson.SerializerOptions);
|
||||
|
||||
var response = req.CreateResponse(HttpStatusCode.OK);
|
||||
await response.WriteAsJsonAsync(new
|
||||
{
|
||||
success = true,
|
||||
mode = "summary",
|
||||
source = "stored",
|
||||
manifestId = manifest.ManifestId,
|
||||
schemaHash = manifest.SchemaHash,
|
||||
snapshotSha256 = snapshotSha,
|
||||
estimatedRowCount = totalRowCount,
|
||||
estimatedSnapshotBytes = serializedBytes.LongLength,
|
||||
preflightCounts = manifest.PreflightCounts,
|
||||
sampleKeys,
|
||||
progress = progress == null ? null : new
|
||||
{
|
||||
progress.SnapshotSha256,
|
||||
completedStepOrders = progress.CompletedSteps,
|
||||
progress.VerificationDone,
|
||||
progress.TombstoneStarted,
|
||||
progress.CompletedAt,
|
||||
aggregateDecrementsApplied = progress.AggregateDecrementsApplied?.Count ?? 0,
|
||||
restoreReIncrementsApplied = progress.RestoreReIncrementsApplied?.Count ?? 0,
|
||||
progress.LastFailureType,
|
||||
progress.LastFailureMessage,
|
||||
// Codex F2 round-3: this is the verifier's OBSERVED count, capped at the
|
||||
// sample size — UI surfaces it as a lower bound when it equals the cap.
|
||||
progress.LastObservedResidualCount,
|
||||
progress.LastResidualSampleJson,
|
||||
},
|
||||
});
|
||||
return response;
|
||||
}
|
||||
|
||||
private static async Task<HttpResponseData> BuildFullResponse(
|
||||
HttpRequestData req, DeletionManifest manifest, DeletionProgress? progress, string snapshotSha)
|
||||
{
|
||||
var response = req.CreateResponse(HttpStatusCode.OK);
|
||||
response.Headers.Add("Content-Type", "application/json");
|
||||
response.Headers.Add("X-Deletion-Manifest-Source", "stored");
|
||||
response.Headers.Add("X-Deletion-Manifest-Snapshot-Sha256", snapshotSha);
|
||||
response.Headers.Add("X-Deletion-Manifest-Has-Progress", (progress != null).ToString());
|
||||
var json = JsonSerializer.SerializeToUtf8Bytes(new
|
||||
{
|
||||
manifest,
|
||||
progress,
|
||||
snapshotSha256 = snapshotSha,
|
||||
}, DeletionManifestJson.SerializerOptions);
|
||||
await response.Body.WriteAsync(json, 0, json.Length);
|
||||
return response;
|
||||
}
|
||||
|
||||
private static async Task<HttpResponseData> BuildDownloadResponse(
|
||||
HttpRequestData req, DeletionManifest manifest, string tenantId, string sessionId, string manifestId)
|
||||
{
|
||||
var json = JsonSerializer.SerializeToUtf8Bytes(manifest, DeletionManifestJson.SerializerOptions);
|
||||
byte[] gzipped;
|
||||
using (var output = new MemoryStream())
|
||||
{
|
||||
using (var gzip = new GZipStream(output, CompressionLevel.Optimal, leaveOpen: true))
|
||||
{
|
||||
await gzip.WriteAsync(json, 0, json.Length);
|
||||
}
|
||||
gzipped = output.ToArray();
|
||||
}
|
||||
|
||||
var fileName = $"{tenantId}_{sessionId}_{manifestId}.snapshot.json.gz";
|
||||
var response = req.CreateResponse(HttpStatusCode.OK);
|
||||
response.Headers.Add("Content-Type", "application/gzip");
|
||||
response.Headers.Add("Content-Disposition", $"attachment; filename=\"{fileName}\"");
|
||||
await response.Body.WriteAsync(gzipped, 0, gzipped.Length);
|
||||
return response;
|
||||
}
|
||||
}
|
||||
}
|
||||
+143
@@ -0,0 +1,143 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Net;
|
||||
using System.Threading.Tasks;
|
||||
using System.Web;
|
||||
using AutopilotMonitor.Functions.Helpers;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using AutopilotMonitor.Shared.Models.Deletion;
|
||||
using Microsoft.Azure.Functions.Worker;
|
||||
using Microsoft.Azure.Functions.Worker.Http;
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Functions.Sessions
|
||||
{
|
||||
/// <summary>
|
||||
/// Cross-tenant read-only listing of sessions currently in a cascade-deletion state.
|
||||
/// Feeds the Global-Admin "Session Cleanup" page (PR-C) — one call per tab:
|
||||
/// <list type="bullet">
|
||||
/// <item><c>?state=Preparing|Queued|Running</c> — In-Flight tab.</item>
|
||||
/// <item><c>?state=Poisoned</c> — Poisoned tab; rows here need an operator restore.</item>
|
||||
/// <item><c>?state=Queued&strandedSinceMinutes=30</c> — Stranded tab; matches the
|
||||
/// <c>SessionDeletionStrandedQueued</c> OpsEvent watchdog (plan §10).</item>
|
||||
/// </list>
|
||||
/// <para>
|
||||
/// Cross-partition scan via <see cref="TableStorageService.GetSessionsByDeletionStateAsync"/>.
|
||||
/// Acceptable here for the same reason the 12h maintenance function uses it: the row count
|
||||
/// for any non-None state is typically < 10 in steady state (poisoned/stranded peak higher
|
||||
/// during incidents). No pagination — page-size pagination would complicate the cross-state
|
||||
/// fan-out the UI needs and the absolute upper bound is bounded by per-tenant retention.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// GA-only enforcement comes from <c>EndpointAccessPolicyCatalog</c> (route registered as
|
||||
/// <c>GlobalAdminOnly</c>); the function body does not re-check.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public class GetSessionDeletionsListFunction
|
||||
{
|
||||
private readonly ILogger<GetSessionDeletionsListFunction> _logger;
|
||||
private readonly TableStorageService _storage;
|
||||
|
||||
public GetSessionDeletionsListFunction(
|
||||
ILogger<GetSessionDeletionsListFunction> logger,
|
||||
TableStorageService storage)
|
||||
{
|
||||
_logger = logger;
|
||||
_storage = storage;
|
||||
}
|
||||
|
||||
[Function("GetSessionDeletionsList")]
|
||||
public async Task<HttpResponseData> Run(
|
||||
[HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "global/session-deletions")]
|
||||
HttpRequestData req)
|
||||
{
|
||||
try
|
||||
{
|
||||
var query = HttpUtility.ParseQueryString(req.Url.Query ?? string.Empty);
|
||||
var state = query["state"] ?? string.Empty;
|
||||
if (!IsAllowedState(state))
|
||||
{
|
||||
var bad = req.CreateResponse(HttpStatusCode.BadRequest);
|
||||
await bad.WriteAsJsonAsync(new
|
||||
{
|
||||
success = false,
|
||||
message = "Query parameter 'state' must be one of: Preparing, Queued, Running, Poisoned.",
|
||||
});
|
||||
return bad;
|
||||
}
|
||||
|
||||
int? strandedSinceMinutes = null;
|
||||
var strandedRaw = query["strandedSinceMinutes"];
|
||||
if (!string.IsNullOrEmpty(strandedRaw))
|
||||
{
|
||||
if (state != SessionDeletionState.Queued)
|
||||
{
|
||||
var bad = req.CreateResponse(HttpStatusCode.BadRequest);
|
||||
await bad.WriteAsJsonAsync(new
|
||||
{
|
||||
success = false,
|
||||
message = "strandedSinceMinutes is only valid with state=Queued.",
|
||||
});
|
||||
return bad;
|
||||
}
|
||||
if (!int.TryParse(strandedRaw, out var minutes) || minutes <= 0 || minutes > 10080)
|
||||
{
|
||||
var bad = req.CreateResponse(HttpStatusCode.BadRequest);
|
||||
await bad.WriteAsJsonAsync(new
|
||||
{
|
||||
success = false,
|
||||
message = "strandedSinceMinutes must be a positive integer up to 10080 (7 days).",
|
||||
});
|
||||
return bad;
|
||||
}
|
||||
strandedSinceMinutes = minutes;
|
||||
}
|
||||
|
||||
var cutoffUtc = strandedSinceMinutes.HasValue
|
||||
? (DateTimeOffset?)DateTimeOffset.UtcNow.AddMinutes(-strandedSinceMinutes.Value)
|
||||
: null;
|
||||
|
||||
var now = DateTimeOffset.UtcNow;
|
||||
var sessions = new List<object>();
|
||||
await foreach (var entity in _storage.GetSessionsByDeletionStateAsync(state, req.FunctionContext.CancellationToken))
|
||||
{
|
||||
var ts = entity.Timestamp ?? now;
|
||||
if (cutoffUtc.HasValue && ts > cutoffUtc.Value) continue;
|
||||
|
||||
var ageMinutes = (int)Math.Floor((now - ts).TotalMinutes);
|
||||
sessions.Add(new
|
||||
{
|
||||
tenantId = entity.PartitionKey,
|
||||
sessionId = entity.RowKey,
|
||||
deletionState = entity.GetString("DeletionState") ?? state,
|
||||
manifestId = entity.GetString("PendingDeletionManifestId") ?? string.Empty,
|
||||
timestamp = ts.UtcDateTime.ToString("o"),
|
||||
ageMinutes,
|
||||
});
|
||||
}
|
||||
|
||||
return await req.OkAsync(new
|
||||
{
|
||||
success = true,
|
||||
state,
|
||||
strandedSinceMinutes,
|
||||
count = sessions.Count,
|
||||
sessions,
|
||||
});
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "GetSessionDeletionsList: unexpected error (state={State})", req.Url.Query);
|
||||
var err = req.CreateResponse(HttpStatusCode.InternalServerError);
|
||||
await err.WriteAsJsonAsync(new { success = false, message = "Internal server error" });
|
||||
return err;
|
||||
}
|
||||
}
|
||||
|
||||
private static bool IsAllowedState(string state) =>
|
||||
state == SessionDeletionState.Preparing
|
||||
|| state == SessionDeletionState.Queued
|
||||
|| state == SessionDeletionState.Running
|
||||
|| state == SessionDeletionState.Poisoned;
|
||||
}
|
||||
}
|
||||
+58
-20
@@ -72,44 +72,74 @@ namespace AutopilotMonitor.Functions.Functions.Sessions
|
||||
new { success = false, message = "manifestId is required in the request body." });
|
||||
}
|
||||
|
||||
// Tenant resolution: prefer JWT TargetTenantId, fall back to SessionsIndex lookup
|
||||
// (Global Admin can target any session by id alone). Mirrors the preview endpoint.
|
||||
// Tenant resolution. Codex F1 review: an explicit body.TenantId from a GA caller is
|
||||
// the only path that lets a Global Admin restore a session in a tenant other than
|
||||
// their JWT home tenant. Order:
|
||||
// 1. Explicit body.TenantId on GA calls (cross-tenant restore — the common case
|
||||
// when restoring from the Session Cleanup admin page).
|
||||
// 2. JWT TargetTenantId (single-tenant operators).
|
||||
// 3. SessionsIndex lookup keyed by sessionId (rare but preserved).
|
||||
// 4. body.TenantId again — last-resort fallback for the "Sessions row gone" path
|
||||
// where neither the index nor the JWT tenant can resolve the target.
|
||||
var requestCtx = req.GetRequestContext();
|
||||
var actorEmail = TenantHelper.GetUserIdentifier(req);
|
||||
var tenantId = requestCtx.TargetTenantId;
|
||||
if (string.IsNullOrEmpty(tenantId) || string.Equals(tenantId, "global", StringComparison.OrdinalIgnoreCase))
|
||||
string tenantId;
|
||||
if (requestCtx.IsGlobalAdmin
|
||||
&& !string.IsNullOrWhiteSpace(body.TenantId)
|
||||
&& Guid.TryParse(body.TenantId, out _))
|
||||
{
|
||||
tenantId = await _sessionRepo.FindSessionTenantIdAsync(sessionId);
|
||||
tenantId = body.TenantId!;
|
||||
}
|
||||
if (string.IsNullOrEmpty(tenantId))
|
||||
else
|
||||
{
|
||||
// Sessions row is gone — but a full restore IS legal in that case. We need a way
|
||||
// to determine the tenantId without the SessionsIndex (which is also gone).
|
||||
// The manifest blob path encodes tenantId, but the operator only gives us
|
||||
// sessionId + manifestId. Require the operator to scope the call by providing
|
||||
// the tenantId in the request body (or via the X-Target-Tenant header).
|
||||
if (string.IsNullOrWhiteSpace(body.TenantId))
|
||||
tenantId = requestCtx.TargetTenantId;
|
||||
if (string.IsNullOrEmpty(tenantId) || string.Equals(tenantId, "global", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return await WriteJsonAsync(req, HttpStatusCode.NotFound, new
|
||||
{
|
||||
success = false,
|
||||
message = $"Session {sessionId} not found in any active SessionsIndex entry. " +
|
||||
"If the cascade completed (Sessions row gone), pass the tenantId explicitly in the request body."
|
||||
});
|
||||
tenantId = await _sessionRepo.FindSessionTenantIdAsync(sessionId) ?? string.Empty;
|
||||
}
|
||||
if (string.IsNullOrEmpty(tenantId))
|
||||
{
|
||||
// Sessions row is gone — but a full restore IS legal in that case. The
|
||||
// manifest blob path encodes tenantId, but the operator only gives us
|
||||
// sessionId + manifestId. Require the operator to scope the call by passing
|
||||
// the tenantId in the request body (or via the X-Target-Tenant header).
|
||||
if (string.IsNullOrWhiteSpace(body.TenantId))
|
||||
{
|
||||
return await WriteJsonAsync(req, HttpStatusCode.NotFound, new
|
||||
{
|
||||
success = false,
|
||||
message = $"Session {sessionId} not found in any active SessionsIndex entry. " +
|
||||
"If the cascade completed (Sessions row gone), pass the tenantId explicitly in the request body."
|
||||
});
|
||||
}
|
||||
tenantId = body.TenantId!;
|
||||
}
|
||||
tenantId = body.TenantId;
|
||||
}
|
||||
|
||||
_logger.LogInformation(
|
||||
"RestoreSession: tenant={TenantId} session={SessionId} manifestId={ManifestId} dryRun={DryRun} actor={Actor}",
|
||||
tenantId, sessionId, body.ManifestId, body.DryRun, actorEmail);
|
||||
|
||||
// Trim + cap operator-supplied reason so it can't bloat the audit-row PartitionKey
|
||||
// payload or be used to smuggle large blobs through. 1024 chars matches the cap the
|
||||
// handler uses for LastFailureMessage in DeletionProgress (same compliance bucket).
|
||||
var reason = body.Reason;
|
||||
if (!string.IsNullOrWhiteSpace(reason))
|
||||
{
|
||||
reason = reason!.Trim();
|
||||
if (reason.Length > 1024) reason = reason.Substring(0, 1024);
|
||||
}
|
||||
else
|
||||
{
|
||||
reason = null;
|
||||
}
|
||||
|
||||
SessionRestoreResult result;
|
||||
try
|
||||
{
|
||||
result = await _restoreService.RestoreAsync(
|
||||
tenantId, sessionId, body.ManifestId,
|
||||
body.DryRun, actorEmail);
|
||||
body.DryRun, actorEmail, reason);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
@@ -175,6 +205,14 @@ namespace AutopilotMonitor.Functions.Functions.Sessions
|
||||
/// should provide tenantId explicitly in this scenario.
|
||||
/// </summary>
|
||||
public string? TenantId { get; set; }
|
||||
/// <summary>
|
||||
/// Optional free-text justification. Persisted into the <c>deletion_restored</c> audit
|
||||
/// row's <c>reason</c> detail so the operator's intent (customer ticket, compliance
|
||||
/// request, …) is recoverable from the tenant audit trail. Codex follow-up to the
|
||||
/// PR-B audit consolidation: previously the UI sent this but the backend dropped it.
|
||||
/// Trimmed + capped at 1024 chars by the function before being passed to the service.
|
||||
/// </summary>
|
||||
public string? Reason { get; set; }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -215,6 +215,8 @@ public static class EndpointAccessPolicyCatalog
|
||||
new("GET", "sessions/{sessionId}/reducer-verification", EndpointPolicy.GlobalAdminOnly),
|
||||
new("GET", "admin/sessions/{sessionId}/delete/preview", EndpointPolicy.GlobalAdminOnly),
|
||||
new("POST", "admin/sessions/{sessionId}/restore", EndpointPolicy.GlobalAdminOnly),
|
||||
new("GET", "admin/sessions/{sessionId}/deletion-manifest", EndpointPolicy.GlobalAdminOnly),
|
||||
new("GET", "global/session-deletions", EndpointPolicy.GlobalAdminOnly),
|
||||
new("GET", "global/raw/sessions", EndpointPolicy.GlobalAdminOnly),
|
||||
new("GET", "global/raw/events", EndpointPolicy.GlobalAdminOnly),
|
||||
new("GET", "global/raw/events/search", EndpointPolicy.GlobalAdminOnly),
|
||||
|
||||
+111
-77
@@ -42,10 +42,9 @@ namespace AutopilotMonitor.Functions.Services.Deletion
|
||||
internal const int ProgressCasMaxAttempts = 10;
|
||||
internal static readonly TimeSpan ProgressCasMaxWallClock = TimeSpan.FromSeconds(60);
|
||||
|
||||
// Cap on the per-table audit-detail `residualKeys` sample — paired with
|
||||
// CascadeVerificationService.MaxResidualSampleSize. The Dictionary<string, string> entry
|
||||
// holds a small JSON array of "{table}/{pk}/{rk}" strings.
|
||||
internal const int AuditResidualSampleSize = 50;
|
||||
// Verification-failure sample cap moved to DeletionProgressConstants so the
|
||||
// log line, the persisted progress-blob field, and the downstream OpsEvent all
|
||||
// use the same projection.
|
||||
|
||||
private readonly TableStorageService _storage;
|
||||
private readonly BlobStorageService _blob;
|
||||
@@ -163,7 +162,18 @@ namespace AutopilotMonitor.Functions.Services.Deletion
|
||||
}
|
||||
catch (Exception ex) when (!(ex is OperationCanceledException))
|
||||
{
|
||||
await AuditStepFailedAsync(tenantId, sessionId, manifestId, step, ex, cancellationToken);
|
||||
LogStepFailed(tenantId, sessionId, manifestId, step, ex);
|
||||
// PR-B Codex F4: persist the failure context into the progress blob BEFORE
|
||||
// rethrowing so the worker's poison-emit path can include the root cause in
|
||||
// the SessionDeletionPoisoned OpsEvent. Best-effort — a persist failure is
|
||||
// logged and swallowed; the original step exception is the one that matters.
|
||||
(progress, etag) = await TryPersistFailureContextAsync(
|
||||
tenantId, sessionId, manifestId, progress, etag,
|
||||
failureType: "step_exception",
|
||||
failureMessage: $"step {step.Order} ({step.Class}{(string.IsNullOrEmpty(step.Table) ? "" : $", {step.Table}")}): {ex.GetType().Name}: {ex.Message}",
|
||||
observedResidualCount: null,
|
||||
residualSampleJson: null,
|
||||
cancellationToken);
|
||||
throw;
|
||||
}
|
||||
|
||||
@@ -175,14 +185,14 @@ namespace AutopilotMonitor.Functions.Services.Deletion
|
||||
isAlreadyApplied: p => p.CompletedSteps.Contains(step.Order),
|
||||
cancellationToken);
|
||||
|
||||
await _maintenanceRepo.LogAuditEntryAsync(
|
||||
tenantId,
|
||||
action: "deletion_step_completed",
|
||||
entityType: "Session",
|
||||
entityId: sessionId,
|
||||
performedBy: "system",
|
||||
details: BuildStepCompletedDetails(manifestId, step, result, aggregateProcessed, sw.ElapsedMilliseconds - stepStart))
|
||||
.ConfigureAwait(false);
|
||||
// PR-B audit consolidation: step-level progress lives in DeletionProgress
|
||||
// (CompletedSteps + AggregateDecrementsApplied + TombstoneStarted) — the admin
|
||||
// Session Cleanup page reads from there. Structured log for App Insights only.
|
||||
_logger.LogInformation(
|
||||
"SessionDeletionHandler step complete: tenant={TenantId} session={SessionId} manifestId={ManifestId} stepOrder={StepOrder} class={Class} table={Table} attempted={Attempted} deletedNow={DeletedNow} alreadyMissing={AlreadyMissing} aggregateProcessed={AggregateProcessed} stepDurationMs={StepDurationMs}",
|
||||
tenantId, sessionId, manifestId, step.Order, step.Class, step.Table ?? string.Empty,
|
||||
result.Attempted, result.DeletedNow, result.AlreadyMissing, aggregateProcessed,
|
||||
sw.ElapsedMilliseconds - stepStart);
|
||||
}
|
||||
|
||||
// (5) Live verification (§1 P4). Outcome dictates whether tombstone runs at all.
|
||||
@@ -191,14 +201,28 @@ namespace AutopilotMonitor.Functions.Services.Deletion
|
||||
var verification = await _verifier.VerifyAsync(manifest, cancellationToken);
|
||||
if (!verification.IsClean)
|
||||
{
|
||||
await _maintenanceRepo.LogAuditEntryAsync(
|
||||
tenantId,
|
||||
action: "deletion_verification_failed",
|
||||
entityType: "Session",
|
||||
entityId: sessionId,
|
||||
performedBy: "system",
|
||||
details: BuildVerificationFailedDetails(manifestId, verification.Residuals))
|
||||
.ConfigureAwait(false);
|
||||
// PR-B audit consolidation: verification residuals are diagnostic detail for
|
||||
// operators on the Session Cleanup page (read from the throw → poison path's
|
||||
// SessionDeletionPoisoned OpsEvent). Tenant admins see deletion_completed
|
||||
// absence as the signal; no tenant-scope audit row needed.
|
||||
LogVerificationFailed(tenantId, sessionId, manifestId, verification.Residuals);
|
||||
|
||||
// PR-B Codex F4: persist failure type + residual sample into the progress blob
|
||||
// so the worker's poison-emit path can include it in the OpsEvent (durable
|
||||
// operator record), not just structured logs which roll off.
|
||||
// Codex F2 round-3: the persisted count is the verifier's OBSERVED count —
|
||||
// CascadeVerificationService stops at MaxResidualSampleSize per table AND at
|
||||
// the first failing table, so the real residual mountain may be larger. We
|
||||
// do not pay the cost of an exhaustive count on the hot path because the
|
||||
// recovery action (poison → operator restore) doesn't need exact magnitudes.
|
||||
var residualSampleJson = BuildResidualSampleJson(verification.Residuals);
|
||||
(progress, etag) = await TryPersistFailureContextAsync(
|
||||
tenantId, sessionId, manifestId, progress, etag,
|
||||
failureType: "verification_residuals",
|
||||
failureMessage: $"{verification.Residuals.Count} observed residual row(s); refusing to tombstone (verifier short-circuits at first failing table)",
|
||||
observedResidualCount: verification.Residuals.Count,
|
||||
residualSampleJson: residualSampleJson,
|
||||
cancellationToken);
|
||||
|
||||
throw new InvalidOperationException(
|
||||
$"Cascade verification found {verification.Residuals.Count} residual row(s) for " +
|
||||
@@ -543,49 +567,70 @@ namespace AutopilotMonitor.Functions.Services.Deletion
|
||||
|
||||
// ============================================================ Audit details builders ====
|
||||
|
||||
private static Dictionary<string, string> BuildStepCompletedDetails(
|
||||
string manifestId, DeletionStep step, DeletionBatchResult tableResult, int aggregateProcessed, long stepDurationMs)
|
||||
private void LogVerificationFailed(
|
||||
string tenantId, string sessionId, string manifestId, IReadOnlyList<CascadeResidualKey> residuals)
|
||||
{
|
||||
var details = new Dictionary<string, string>(StringComparer.Ordinal)
|
||||
{
|
||||
["manifestId"] = manifestId,
|
||||
["stepOrder"] = step.Order.ToString(System.Globalization.CultureInfo.InvariantCulture),
|
||||
["class"] = step.Class,
|
||||
["stepDurationMs"] = stepDurationMs.ToString(System.Globalization.CultureInfo.InvariantCulture),
|
||||
};
|
||||
if (!string.IsNullOrEmpty(step.Table))
|
||||
{
|
||||
details["table"] = step.Table!;
|
||||
}
|
||||
if (!string.IsNullOrEmpty(step.Step))
|
||||
{
|
||||
details["syntheticStep"] = step.Step!;
|
||||
}
|
||||
if (step.Class == DeletionStepClass.Aggregate)
|
||||
{
|
||||
details["decrementsProcessed"] = aggregateProcessed.ToString(System.Globalization.CultureInfo.InvariantCulture);
|
||||
}
|
||||
else
|
||||
{
|
||||
details["attempted"] = tableResult.Attempted.ToString(System.Globalization.CultureInfo.InvariantCulture);
|
||||
details["deletedNow"] = tableResult.DeletedNow.ToString(System.Globalization.CultureInfo.InvariantCulture);
|
||||
details["alreadyMissing"] = tableResult.AlreadyMissing.ToString(System.Globalization.CultureInfo.InvariantCulture);
|
||||
}
|
||||
return details;
|
||||
_logger.LogError(
|
||||
"SessionDeletionHandler verification failed: tenant={TenantId} session={SessionId} manifestId={ManifestId} residualCount={ResidualCount} sampleKeys={SampleKeys}",
|
||||
tenantId, sessionId, manifestId, residuals.Count, BuildResidualSampleJson(residuals));
|
||||
}
|
||||
|
||||
private static Dictionary<string, string> BuildVerificationFailedDetails(
|
||||
string manifestId, IReadOnlyList<CascadeResidualKey> residuals)
|
||||
/// <summary>
|
||||
/// JSON-encoded sample of residual table/pk/rk triples, capped at
|
||||
/// <see cref="DeletionProgressConstants.VerificationResidualSampleSize"/>. Shared by the
|
||||
/// log-only sink (App Insights) and the durable progress-blob sink that feeds the
|
||||
/// <c>SessionDeletionPoisoned</c> OpsEvent.
|
||||
/// </summary>
|
||||
internal static string BuildResidualSampleJson(IReadOnlyList<CascadeResidualKey> residuals)
|
||||
{
|
||||
var sample = residuals
|
||||
.Take(AuditResidualSampleSize)
|
||||
.Take(DeletionProgressConstants.VerificationResidualSampleSize)
|
||||
.Select(r => new { table = r.Table, pk = r.Pk, rk = r.Rk });
|
||||
return new Dictionary<string, string>(StringComparer.Ordinal)
|
||||
return JsonSerializer.Serialize(sample);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Persists <see cref="DeletionProgress.LastFailureType"/> / <see cref="DeletionProgress.LastFailureMessage"/>
|
||||
/// / <see cref="DeletionProgress.LastResidualSampleJson"/> to the progress blob via the
|
||||
/// same ETag-CAS path the other progress mutations use. Best-effort — a CAS failure here
|
||||
/// is logged and swallowed because the caller's next step is to rethrow the underlying
|
||||
/// failure, and burying that exception under "progress write failed" would obscure the
|
||||
/// real cause. Returns the latest (progress, etag) tuple so the caller can keep the
|
||||
/// captured ETag in sync for any further progress writes downstream (the verification
|
||||
/// path does no further writes; the step-exception path does none either before throw).
|
||||
/// Failure messages are clipped to 1024 chars before persistence.
|
||||
/// </summary>
|
||||
private async Task<(DeletionProgress, string)> TryPersistFailureContextAsync(
|
||||
string tenantId, string sessionId, string manifestId,
|
||||
DeletionProgress progress, string etag,
|
||||
string failureType, string failureMessage,
|
||||
int? observedResidualCount, string? residualSampleJson,
|
||||
CancellationToken ct)
|
||||
{
|
||||
var clippedMessage = failureMessage.Length > 1024 ? failureMessage.Substring(0, 1024) : failureMessage;
|
||||
try
|
||||
{
|
||||
["manifestId"] = manifestId,
|
||||
["residualCount"] = residuals.Count.ToString(System.Globalization.CultureInfo.InvariantCulture),
|
||||
["residualKeys"] = JsonSerializer.Serialize(sample),
|
||||
};
|
||||
return await UpdateProgressWithRetryAsync(
|
||||
tenantId, sessionId, manifestId, progress, etag,
|
||||
mutate: p =>
|
||||
{
|
||||
p.LastFailureType = failureType;
|
||||
p.LastFailureMessage = clippedMessage;
|
||||
p.LastObservedResidualCount = observedResidualCount;
|
||||
p.LastResidualSampleJson = residualSampleJson;
|
||||
},
|
||||
isAlreadyApplied: p =>
|
||||
string.Equals(p.LastFailureType, failureType, StringComparison.Ordinal)
|
||||
&& string.Equals(p.LastFailureMessage, clippedMessage, StringComparison.Ordinal),
|
||||
ct);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex,
|
||||
"SessionDeletionHandler: failed to persist failure context (tenant={TenantId} session={SessionId} manifestId={ManifestId} failureType={FailureType}) — proceeding with rethrow",
|
||||
tenantId, sessionId, manifestId, failureType);
|
||||
return (progress, etag);
|
||||
}
|
||||
}
|
||||
|
||||
private static Dictionary<string, string> BuildCompletedDetails(
|
||||
@@ -602,27 +647,16 @@ namespace AutopilotMonitor.Functions.Services.Deletion
|
||||
};
|
||||
}
|
||||
|
||||
private Task AuditStepFailedAsync(
|
||||
string tenantId, string sessionId, string manifestId, DeletionStep step, Exception ex, CancellationToken _)
|
||||
private void LogStepFailed(
|
||||
string tenantId, string sessionId, string manifestId, DeletionStep step, Exception ex)
|
||||
{
|
||||
var details = new Dictionary<string, string>(StringComparer.Ordinal)
|
||||
{
|
||||
["manifestId"] = manifestId,
|
||||
["stepOrder"] = step.Order.ToString(System.Globalization.CultureInfo.InvariantCulture),
|
||||
["class"] = step.Class,
|
||||
["exceptionType"] = ex.GetType().FullName ?? "Unknown",
|
||||
// Trim error message to a reasonable size for the audit row.
|
||||
["error"] = ex.Message.Length > 1024 ? ex.Message.Substring(0, 1024) : ex.Message,
|
||||
};
|
||||
if (!string.IsNullOrEmpty(step.Table)) details["table"] = step.Table!;
|
||||
|
||||
return _maintenanceRepo.LogAuditEntryAsync(
|
||||
tenantId,
|
||||
action: "deletion_step_failed",
|
||||
entityType: "Session",
|
||||
entityId: sessionId,
|
||||
performedBy: "system",
|
||||
details: details);
|
||||
// PR-B audit consolidation: step failures bubble up to the worker → poison path →
|
||||
// SessionDeletionPoisoned OpsEvent. Diagnostic detail goes to App Insights so the
|
||||
// Session Cleanup admin page (or a kusto query) can drill in by manifestId.
|
||||
_logger.LogError(ex,
|
||||
"SessionDeletionHandler step failed: tenant={TenantId} session={SessionId} manifestId={ManifestId} stepOrder={StepOrder} class={Class} table={Table} exceptionType={ExceptionType}",
|
||||
tenantId, sessionId, manifestId, step.Order, step.Class, step.Table ?? string.Empty,
|
||||
ex.GetType().FullName ?? "Unknown");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+205
-26
@@ -1,5 +1,4 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using Azure.Identity;
|
||||
@@ -7,7 +6,6 @@ using Azure.Storage.Queues;
|
||||
using Azure.Storage.Queues.Models;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using AutopilotMonitor.Shared;
|
||||
using AutopilotMonitor.Shared.DataAccess;
|
||||
using AutopilotMonitor.Shared.Models.Deletion;
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
@@ -30,10 +28,14 @@ namespace AutopilotMonitor.Functions.Services.Deletion
|
||||
/// calls <c>UpdateMessageAsync</c> every <see cref="_heartbeatInterval"/> to extend the
|
||||
/// message visibility by <see cref="HeartbeatExtendBy"/>. Prevents queue re-delivery
|
||||
/// from spawning a parallel worker on the same manifest while a step is in flight.</item>
|
||||
/// <item><b>Poison audit</b>: max-dequeue exhaustion writes a <c>deletion_poisoned</c>
|
||||
/// audit entry alongside the poison-queue move so the operator can correlate the
|
||||
/// poisoned cascade with its manifest. The Sessions row's <c>DeletionState</c> is NOT
|
||||
/// auto-cleared — operator action via restore-from-poisoned (§13, PR4b) is required.</item>
|
||||
/// <item><b>Poison OpsEvent</b>: max-dequeue exhaustion records a
|
||||
/// <c>SessionDeletionPoisoned</c> OpsEvent alongside the poison-queue move so the
|
||||
/// operator can correlate the poisoned cascade with its manifest (and wire Telegram
|
||||
/// routing). PR-B audit consolidation: this used to be a per-tenant
|
||||
/// <c>deletion_poisoned</c> audit but tenant admins should only see the lifecycle
|
||||
/// endpoints (started / completed / restored), so the signal moved to OpsEvents.
|
||||
/// The Sessions row's <c>DeletionState</c> is NOT auto-cleared — operator action via
|
||||
/// restore-from-poisoned (§13, PR4b) is required.</item>
|
||||
/// </list>
|
||||
/// <para>
|
||||
/// <b>The worker is NOT registered in <c>Program.cs</c> in PR4</b> — that wires up in PR5
|
||||
@@ -81,7 +83,8 @@ namespace AutopilotMonitor.Functions.Services.Deletion
|
||||
private readonly SessionDeletionHandler _handler;
|
||||
private readonly TableStorageService _storage;
|
||||
private readonly AdminConfigurationService _adminConfig;
|
||||
private readonly IMaintenanceRepository _maintenanceRepo;
|
||||
private readonly BlobStorageService _blob;
|
||||
private readonly OpsEventService _opsEvents;
|
||||
private readonly ILogger<SessionDeletionWorker> _logger;
|
||||
|
||||
public SessionDeletionWorker(
|
||||
@@ -89,13 +92,15 @@ namespace AutopilotMonitor.Functions.Services.Deletion
|
||||
SessionDeletionHandler handler,
|
||||
TableStorageService storage,
|
||||
AdminConfigurationService adminConfig,
|
||||
IMaintenanceRepository maintenanceRepo,
|
||||
BlobStorageService blob,
|
||||
OpsEventService opsEvents,
|
||||
ILogger<SessionDeletionWorker> logger)
|
||||
{
|
||||
_handler = handler ?? throw new ArgumentNullException(nameof(handler));
|
||||
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
|
||||
_adminConfig = adminConfig ?? throw new ArgumentNullException(nameof(adminConfig));
|
||||
_maintenanceRepo = maintenanceRepo ?? throw new ArgumentNullException(nameof(maintenanceRepo));
|
||||
_blob = blob ?? throw new ArgumentNullException(nameof(blob));
|
||||
_opsEvents = opsEvents ?? throw new ArgumentNullException(nameof(opsEvents));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_heartbeatInterval = DefaultHeartbeatInterval;
|
||||
_pollInterval = DefaultPollInterval;
|
||||
@@ -147,7 +152,8 @@ namespace AutopilotMonitor.Functions.Services.Deletion
|
||||
SessionDeletionHandler handler,
|
||||
TableStorageService storage,
|
||||
AdminConfigurationService adminConfig,
|
||||
IMaintenanceRepository maintenanceRepo,
|
||||
BlobStorageService blob,
|
||||
OpsEventService opsEvents,
|
||||
ILogger<SessionDeletionWorker> logger,
|
||||
TimeSpan? heartbeatInterval = null,
|
||||
TimeSpan? pollInterval = null)
|
||||
@@ -157,7 +163,8 @@ namespace AutopilotMonitor.Functions.Services.Deletion
|
||||
_handler = handler ?? throw new ArgumentNullException(nameof(handler));
|
||||
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
|
||||
_adminConfig = adminConfig ?? throw new ArgumentNullException(nameof(adminConfig));
|
||||
_maintenanceRepo = maintenanceRepo ?? throw new ArgumentNullException(nameof(maintenanceRepo));
|
||||
_blob = blob ?? throw new ArgumentNullException(nameof(blob));
|
||||
_opsEvents = opsEvents ?? throw new ArgumentNullException(nameof(opsEvents));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_heartbeatInterval = heartbeatInterval ?? DefaultHeartbeatInterval;
|
||||
_pollInterval = pollInterval ?? DefaultPollInterval;
|
||||
@@ -358,31 +365,203 @@ namespace AutopilotMonitor.Functions.Services.Deletion
|
||||
|
||||
if (envelope != null && !string.IsNullOrEmpty(envelope.TenantId) && !string.IsNullOrEmpty(envelope.SessionId))
|
||||
{
|
||||
// PR-B audit consolidation: poisoned cascades surface as a global-scope OpsEvent
|
||||
// (operator-bound, Telegram-routable) rather than a per-tenant audit. Tenant admins
|
||||
// observe deletion via the lifecycle audits (started / completed / restored).
|
||||
//
|
||||
// Codex F4 follow-up: pull the handler's last-failure breadcrumb out of the
|
||||
// DeletionProgress blob so the OpsEvent carries the root cause (failureType +
|
||||
// residual sample), not just queue-side metadata. Blob read is best-effort —
|
||||
// a missing blob, transient 5xx, or pre-followup progress without LastFailure*
|
||||
// fields still produces a useful OpsEvent, just without root-cause data.
|
||||
string? failureType = null;
|
||||
string? failureMessage = null;
|
||||
int? observedResidualCount = null;
|
||||
string? residualSamplePreviewJson = null;
|
||||
if (!string.IsNullOrEmpty(envelope.ManifestId))
|
||||
{
|
||||
try
|
||||
{
|
||||
var (progress, _) = await _blob.DownloadDeletionProgressAsync(
|
||||
envelope.TenantId, envelope.SessionId, envelope.ManifestId!, ct)
|
||||
.ConfigureAwait(false);
|
||||
failureType = string.IsNullOrEmpty(progress.LastFailureType) ? null : progress.LastFailureType;
|
||||
failureMessage = string.IsNullOrEmpty(progress.LastFailureMessage) ? null : progress.LastFailureMessage;
|
||||
// Codex F2 round-3: this is the verifier's OBSERVED count
|
||||
// (CascadeVerificationService caps at MaxResidualSampleSize per table AND
|
||||
// short-circuits after the first failing table — the real residual count
|
||||
// may be larger). For pre-followup progress blobs lacking the field, fall
|
||||
// back to the sample-array length so the OpsEvent still carries a number.
|
||||
observedResidualCount = progress.LastObservedResidualCount
|
||||
?? ExtractResidualSampleArrayLength(progress.LastResidualSampleJson);
|
||||
// Codex F2 round-2: shrink the residual sample further before embedding
|
||||
// it in the OpsEvent. The OpsEvents table truncates Details at 4096 chars
|
||||
// (TableOpsEventRepository.cs); the full 50-entry sample plus the other
|
||||
// payload fields can blow past that and leave the JSON mid-string-corrupt
|
||||
// — operators end up reading garbled raw text instead of structured data.
|
||||
// The Session Cleanup admin page already shows the full sample by reading
|
||||
// the progress blob directly, so the OpsEvent only needs a preview.
|
||||
residualSamplePreviewJson = ShrinkResidualSampleForOpsEvent(progress.LastResidualSampleJson);
|
||||
}
|
||||
catch (Exception blobEx)
|
||||
{
|
||||
_logger.LogWarning(blobEx,
|
||||
"SessionDeletionWorker: failed to read DeletionProgress for poison enrichment (tenant={Tenant} session={Session} manifestId={Manifest}) — OpsEvent will lack root-cause data",
|
||||
envelope.TenantId, envelope.SessionId, envelope.ManifestId);
|
||||
}
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
await _maintenanceRepo.LogAuditEntryAsync(
|
||||
await _opsEvents.RecordSessionDeletionPoisonedAsync(
|
||||
envelope.TenantId,
|
||||
action: "deletion_poisoned",
|
||||
entityType: "Session",
|
||||
entityId: envelope.SessionId,
|
||||
performedBy: "system",
|
||||
details: new Dictionary<string, string>(StringComparer.Ordinal)
|
||||
{
|
||||
["manifestId"] = envelope.ManifestId ?? string.Empty,
|
||||
["reason"] = envelope.Reason ?? string.Empty,
|
||||
["messageId"] = msg.MessageId,
|
||||
["dequeueCount"] = (msg.DequeueCount - 1).ToString(System.Globalization.CultureInfo.InvariantCulture),
|
||||
}).ConfigureAwait(false);
|
||||
envelope.SessionId,
|
||||
envelope.ManifestId ?? string.Empty,
|
||||
envelope.Reason ?? string.Empty,
|
||||
msg.MessageId,
|
||||
(int)(msg.DequeueCount - 1),
|
||||
failureType,
|
||||
failureMessage,
|
||||
observedResidualCount,
|
||||
residualSamplePreviewJson).ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception auditEx)
|
||||
catch (Exception opsEx)
|
||||
{
|
||||
_logger.LogError(auditEx,
|
||||
"SessionDeletionWorker: deletion_poisoned audit log write failed for tenant={Tenant} session={Session}",
|
||||
_logger.LogError(opsEx,
|
||||
"SessionDeletionWorker: SessionDeletionPoisoned OpsEvent write failed for tenant={Tenant} session={Session}",
|
||||
envelope.TenantId, envelope.SessionId);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Back-compat fallback used only when <see cref="AutopilotMonitor.Shared.Models.Deletion.DeletionProgress.LastObservedResidualCount"/>
|
||||
/// is missing (progress blobs written before the field was introduced). Parses the JSON
|
||||
/// sample array length and uses it as a best-effort substitute for the observed count.
|
||||
/// Returns null on parse errors or missing data.
|
||||
/// </summary>
|
||||
internal static int? ExtractResidualSampleArrayLength(string? residualSampleJson)
|
||||
{
|
||||
if (string.IsNullOrEmpty(residualSampleJson)) return null;
|
||||
try
|
||||
{
|
||||
using var doc = System.Text.Json.JsonDocument.Parse(residualSampleJson!);
|
||||
if (doc.RootElement.ValueKind == System.Text.Json.JsonValueKind.Array)
|
||||
{
|
||||
return doc.RootElement.GetArrayLength();
|
||||
}
|
||||
}
|
||||
catch (System.Text.Json.JsonException)
|
||||
{
|
||||
// Tolerate malformed payloads — the breadcrumb itself still goes through.
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Trims the progress-blob's residual sample for safe embedding in the
|
||||
/// <c>SessionDeletionPoisoned</c> OpsEvent's <c>Details</c> column. Three-layer defense
|
||||
/// against the 4096-char Azure Table truncation discovered by Codex review:
|
||||
/// <list type="number">
|
||||
/// <item>Take at most <see cref="AutopilotMonitor.Shared.Models.Deletion.DeletionProgressConstants.OpsEventResidualSamplePreviewSize"/>
|
||||
/// entries (drop tail, the rest stays in the progress blob for the admin UI).</item>
|
||||
/// <item>Trim each <c>table</c> / <c>pk</c> / <c>rk</c> to
|
||||
/// <see cref="AutopilotMonitor.Shared.Models.Deletion.DeletionProgressConstants.OpsEventResidualKeyMaxChars"/>
|
||||
/// with a trailing <c>…</c> marker — a single composite key can be 200+ chars and
|
||||
/// would otherwise consume most of the budget on its own.</item>
|
||||
/// <item>If the serialized JSON still exceeds
|
||||
/// <see cref="AutopilotMonitor.Shared.Models.Deletion.DeletionProgressConstants.OpsEventResidualPreviewBudgetChars"/>,
|
||||
/// drop trailing entries until it fits. Worst case (all entries too long) returns
|
||||
/// <c>"[]"</c>; operators still get failureType + observedResidualCount and can
|
||||
/// fetch the full sample from the progress blob.</item>
|
||||
/// </list>
|
||||
/// Returns null when there's nothing to embed or the input cannot be parsed.
|
||||
/// </summary>
|
||||
internal static string? ShrinkResidualSampleForOpsEvent(string? fullSampleJson)
|
||||
{
|
||||
if (string.IsNullOrEmpty(fullSampleJson)) return null;
|
||||
|
||||
List<ResidualPreviewEntry> entries;
|
||||
try
|
||||
{
|
||||
using var doc = System.Text.Json.JsonDocument.Parse(fullSampleJson!);
|
||||
if (doc.RootElement.ValueKind != System.Text.Json.JsonValueKind.Array)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
entries = new List<ResidualPreviewEntry>();
|
||||
var max = AutopilotMonitor.Shared.Models.Deletion.DeletionProgressConstants.OpsEventResidualSamplePreviewSize;
|
||||
foreach (var element in doc.RootElement.EnumerateArray())
|
||||
{
|
||||
if (entries.Count >= max) break;
|
||||
if (element.ValueKind != System.Text.Json.JsonValueKind.Object) continue;
|
||||
entries.Add(new ResidualPreviewEntry
|
||||
{
|
||||
Table = TrimResidualKeyField(ReadStringProperty(element, "table")),
|
||||
Pk = TrimResidualKeyField(ReadStringProperty(element, "pk")),
|
||||
Rk = TrimResidualKeyField(ReadStringProperty(element, "rk")),
|
||||
});
|
||||
}
|
||||
}
|
||||
catch (System.Text.Json.JsonException)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
// Total-budget pass: serialize, then drop trailing entries until under the cap.
|
||||
// Worst case (one over-budget entry) ends at "[]" — still valid JSON, and the full
|
||||
// sample remains in the progress blob.
|
||||
var budget = AutopilotMonitor.Shared.Models.Deletion.DeletionProgressConstants.OpsEventResidualPreviewBudgetChars;
|
||||
while (true)
|
||||
{
|
||||
var serialized = SerializeResidualPreview(entries);
|
||||
if (serialized.Length <= budget || entries.Count == 0) return serialized;
|
||||
entries.RemoveAt(entries.Count - 1);
|
||||
}
|
||||
}
|
||||
|
||||
private static string SerializeResidualPreview(List<ResidualPreviewEntry> entries)
|
||||
{
|
||||
using var stream = new System.IO.MemoryStream();
|
||||
using (var writer = new System.Text.Json.Utf8JsonWriter(stream))
|
||||
{
|
||||
writer.WriteStartArray();
|
||||
foreach (var e in entries)
|
||||
{
|
||||
writer.WriteStartObject();
|
||||
writer.WriteString("table", e.Table);
|
||||
writer.WriteString("pk", e.Pk);
|
||||
writer.WriteString("rk", e.Rk);
|
||||
writer.WriteEndObject();
|
||||
}
|
||||
writer.WriteEndArray();
|
||||
}
|
||||
return System.Text.Encoding.UTF8.GetString(stream.ToArray());
|
||||
}
|
||||
|
||||
private static string TrimResidualKeyField(string? raw)
|
||||
{
|
||||
if (string.IsNullOrEmpty(raw)) return string.Empty;
|
||||
var max = AutopilotMonitor.Shared.Models.Deletion.DeletionProgressConstants.OpsEventResidualKeyMaxChars;
|
||||
if (raw!.Length <= max) return raw;
|
||||
// -1 makes room for the trailing ellipsis so the marked-truncated string fits in
|
||||
// the same budget as a max-length untrimmed one.
|
||||
return raw.Substring(0, max - 1) + "…";
|
||||
}
|
||||
|
||||
private static string? ReadStringProperty(System.Text.Json.JsonElement obj, string propertyName)
|
||||
{
|
||||
if (!obj.TryGetProperty(propertyName, out var value)) return null;
|
||||
return value.ValueKind == System.Text.Json.JsonValueKind.String ? value.GetString() : null;
|
||||
}
|
||||
|
||||
private struct ResidualPreviewEntry
|
||||
{
|
||||
public string Table;
|
||||
public string Pk;
|
||||
public string Rk;
|
||||
}
|
||||
|
||||
private async Task SafeDeleteAsync(QueueMessage msg, CancellationToken ct)
|
||||
=> await SafeDeleteAsync(msg.MessageId, msg.PopReceipt, ct).ConfigureAwait(false);
|
||||
|
||||
|
||||
@@ -90,6 +90,7 @@ namespace AutopilotMonitor.Functions.Services.Deletion
|
||||
public virtual async Task<SessionRestoreResult> RestoreAsync(
|
||||
string tenantId, string sessionId, string manifestId,
|
||||
bool dryRun, string actor,
|
||||
string? operatorReason = null,
|
||||
CancellationToken ct = default)
|
||||
{
|
||||
if (string.IsNullOrEmpty(tenantId)) throw new ArgumentException("tenantId is required", nameof(tenantId));
|
||||
@@ -100,8 +101,13 @@ namespace AutopilotMonitor.Functions.Services.Deletion
|
||||
var sw = Stopwatch.StartNew();
|
||||
var result = new SessionRestoreResult();
|
||||
|
||||
// (1) §13.1 — audit BEFORE any read.
|
||||
await AuditManifestDownloadedAsync(tenantId, sessionId, manifestId, actor, ct).ConfigureAwait(false);
|
||||
// (1) PR-B audit consolidation: deletion_manifest_downloaded was an internal-step
|
||||
// breadcrumb that doubled the tenant audit row count without adding signal. The
|
||||
// operator-relevant outcome — deletion_restored — already records the actor +
|
||||
// manifestId once the restore completes (or surfaces failure via the throw path).
|
||||
_logger.LogInformation(
|
||||
"SessionRestoreService: starting restore for tenant={TenantId} session={SessionId} manifestId={ManifestId} actor={Actor}",
|
||||
tenantId, sessionId, manifestId, actor);
|
||||
|
||||
// (2) Download manifest (SHA-verified) + progress. PR4c F6: use the with-Sha variant
|
||||
// and verify the snapshot↔progress binding once both blobs are loaded (plan §3).
|
||||
@@ -207,10 +213,14 @@ namespace AutopilotMonitor.Functions.Services.Deletion
|
||||
}
|
||||
|
||||
// (4) Dry-run path: simulate by counting what each step would restore, no writes.
|
||||
// Codex follow-up: report the AUTO-SELECTED mode (full / partial) here so the admin
|
||||
// UI can show "Mode (auto-selected): partial" before the real run. The Outcome of
|
||||
// DryRunOk is the dry-run signal — using mode="dryRun" hid the operator-visible
|
||||
// mode preview the dialog was built to surface.
|
||||
if (dryRun)
|
||||
{
|
||||
result.Outcome = SessionRestoreOutcome.DryRunOk;
|
||||
result.Mode = "dryRun";
|
||||
result.Mode = mode;
|
||||
foreach (var step in manifest.Steps)
|
||||
{
|
||||
if (step.Class == DeletionStepClass.Aggregate || step.Class == DeletionStepClass.Final) continue;
|
||||
@@ -265,7 +275,7 @@ namespace AutopilotMonitor.Functions.Services.Deletion
|
||||
result.DurationMs = sw.ElapsedMilliseconds;
|
||||
|
||||
// (6) Audit completion.
|
||||
await AuditRestoredAsync(tenantId, sessionId, manifestId, mode, actor, result, ct).ConfigureAwait(false);
|
||||
await AuditRestoredAsync(tenantId, sessionId, manifestId, mode, actor, operatorReason, result, ct).ConfigureAwait(false);
|
||||
|
||||
_logger.LogInformation(
|
||||
"SessionRestoreService completed: tenant={Tenant} session={Session} manifestId={ManifestId} mode={Mode} durationMs={Duration}",
|
||||
@@ -578,40 +588,31 @@ namespace AutopilotMonitor.Functions.Services.Deletion
|
||||
map[key] = map.TryGetValue(key, out var c) ? c + delta : delta;
|
||||
}
|
||||
|
||||
private async Task AuditManifestDownloadedAsync(string tenantId, string sessionId, string manifestId, string actor, CancellationToken ct)
|
||||
{
|
||||
await _maintenanceRepo.LogAuditEntryAsync(
|
||||
tenantId,
|
||||
action: "deletion_manifest_downloaded",
|
||||
entityType: "Session",
|
||||
entityId: sessionId,
|
||||
performedBy: actor,
|
||||
details: new Dictionary<string, string>(StringComparer.Ordinal)
|
||||
{
|
||||
["manifestId"] = manifestId,
|
||||
["reason"] = "restore_attempt",
|
||||
}).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
private async Task AuditRestoredAsync(
|
||||
string tenantId, string sessionId, string manifestId, string mode, string actor,
|
||||
string? operatorReason,
|
||||
SessionRestoreResult result, CancellationToken ct)
|
||||
{
|
||||
var details = new Dictionary<string, string>(StringComparer.Ordinal)
|
||||
{
|
||||
["manifestId"] = manifestId,
|
||||
["mode"] = mode,
|
||||
["rowsRestoredByTable"] = JsonSerializer.Serialize(result.RowsRestoredByTable),
|
||||
["rowsSkippedByTable"] = JsonSerializer.Serialize(result.RowsSkippedByTable),
|
||||
["inventoryReIncrements"] = result.InventoryReIncrements.ToString(System.Globalization.CultureInfo.InvariantCulture),
|
||||
["durationMs"] = result.DurationMs.ToString(System.Globalization.CultureInfo.InvariantCulture),
|
||||
};
|
||||
if (!string.IsNullOrEmpty(operatorReason))
|
||||
{
|
||||
details["reason"] = operatorReason!;
|
||||
}
|
||||
await _maintenanceRepo.LogAuditEntryAsync(
|
||||
tenantId,
|
||||
action: "deletion_restored",
|
||||
entityType: "Session",
|
||||
entityId: sessionId,
|
||||
performedBy: actor,
|
||||
details: new Dictionary<string, string>(StringComparer.Ordinal)
|
||||
{
|
||||
["manifestId"] = manifestId,
|
||||
["mode"] = mode,
|
||||
["rowsRestoredByTable"] = JsonSerializer.Serialize(result.RowsRestoredByTable),
|
||||
["rowsSkippedByTable"] = JsonSerializer.Serialize(result.RowsSkippedByTable),
|
||||
["inventoryReIncrements"] = result.InventoryReIncrements.ToString(System.Globalization.CultureInfo.InvariantCulture),
|
||||
["durationMs"] = result.DurationMs.ToString(System.Globalization.CultureInfo.InvariantCulture),
|
||||
}).ConfigureAwait(false);
|
||||
details: details).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -98,6 +98,54 @@ namespace AutopilotMonitor.Functions.Services
|
||||
$"Session {sessionId} stuck in DeletionState=Queued since {queuedSince:o} (manifestId={manifestId})",
|
||||
tenantId, "System.Maintenance", new { tenantId, sessionId, queuedSince = queuedSince.ToString("o"), manifestId });
|
||||
|
||||
/// <summary>
|
||||
/// Cascade max-dequeue exhaustion (PR-B follow-up): the worker has moved the envelope to
|
||||
/// the poison queue and CAS-transitioned the Sessions row to <see cref="SessionDeletionState.Poisoned"/>.
|
||||
/// Replaces the prior <c>deletion_poisoned</c> tenant audit — tenant admins see only the
|
||||
/// lifecycle endpoints (<c>deletion_started</c>, <c>deletion_completed</c>, <c>deletion_restored</c>),
|
||||
/// while operators get this OpsEvent for the Session Cleanup admin page + Telegram routing.
|
||||
/// <para>
|
||||
/// Codex follow-ups F4 + F2: <paramref name="failureType"/> / <paramref name="failureMessage"/>
|
||||
/// / <paramref name="observedResidualCount"/> / <paramref name="residualSamplePreviewJson"/>
|
||||
/// are populated from <see cref="DeletionProgress"/> fields the handler writes before
|
||||
/// throwing. Worker reads the progress blob in its poison path and passes whatever is
|
||||
/// present (all are nullable so a worker that pre-dates the progress-schema bump still
|
||||
/// emits a useful event, just without root-cause data).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <paramref name="observedResidualCount"/> is the verifier's <b>observed</b> count, not
|
||||
/// the true total: <c>CascadeVerificationService</c> caps at
|
||||
/// <see cref="DeletionProgressConstants.VerificationResidualSampleSize"/> rows per table
|
||||
/// and short-circuits after the first failing table. Operators reading this number should
|
||||
/// treat it as a lower bound, especially when it equals the cap.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <paramref name="residualSamplePreviewJson"/> is a small (≤
|
||||
/// <see cref="DeletionProgressConstants.OpsEventResidualSamplePreviewSize"/>) preview that
|
||||
/// fits under the OpsEvents table's 4096-char Details truncation. The full progress-blob
|
||||
/// sample (up to <see cref="DeletionProgressConstants.VerificationResidualSampleSize"/>
|
||||
/// entries) is available via the Session Cleanup admin page's stored-manifest modal.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public Task RecordSessionDeletionPoisonedAsync(
|
||||
string tenantId, string sessionId, string manifestId, string reason, string messageId, int dequeueCount,
|
||||
string? failureType = null, string? failureMessage = null,
|
||||
int? observedResidualCount = null, string? residualSamplePreviewJson = null)
|
||||
{
|
||||
var cause = !string.IsNullOrEmpty(failureType)
|
||||
? $" — cause: {failureType}{(string.IsNullOrEmpty(failureMessage) ? "" : $" ({failureMessage})")}"
|
||||
: string.Empty;
|
||||
return WriteAsync(OpsEventCategory.Maintenance, "SessionDeletionPoisoned", OpsEventSeverity.Error,
|
||||
$"Session {sessionId} cascade poisoned after {dequeueCount} attempts (manifestId={manifestId}){cause}",
|
||||
tenantId, "System.Maintenance",
|
||||
new
|
||||
{
|
||||
tenantId, sessionId, manifestId, reason, messageId, dequeueCount,
|
||||
failureType, failureMessage, observedResidualCount,
|
||||
residualSamplePreviewJson,
|
||||
});
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Successful end of a <see cref="Functions.Maintenance.SessionDeletionMaintenanceFunction"/>
|
||||
/// run — records the per-block totals so dashboards can fold the cadence into the timeline.
|
||||
|
||||
@@ -59,5 +59,91 @@ namespace AutopilotMonitor.Shared.Models.Deletion
|
||||
/// behaviour for genuine bugs (Sessions row removed outside the cascade).
|
||||
/// </summary>
|
||||
public bool TombstoneStarted { get; set; }
|
||||
|
||||
// ============================================================ PR-B follow-up ====
|
||||
// Codex F4 (Medium): PR-B routed step / verification failures out of the per-tenant
|
||||
// audit (intentional — tenants don't need them), but the SessionDeletionPoisoned
|
||||
// OpsEvent was left with only queue-side data (dequeueCount + manifestId). The actual
|
||||
// root cause was only visible in App Insights structured logs, which is a problem for
|
||||
// long-tail operator forensics (logs roll off, the OpsEvent is the durable record).
|
||||
// These three nullable fields are written by the handler immediately before it throws
|
||||
// and read by the worker when it transitions the cascade to Poisoned, so the
|
||||
// SessionDeletionPoisoned OpsEvent carries the failure context that explains WHY.
|
||||
//
|
||||
// All three are nullable / default-empty so blobs written by pre-PR-B-followup workers
|
||||
// deserialize cleanly. The worker's poison-emit path defends against missing values
|
||||
// with explicit string.IsNullOrEmpty checks before attaching them to the OpsEvent.
|
||||
|
||||
/// <summary>Classification of the most recent failure that triggered a re-queue. Examples:
|
||||
/// <c>"verification_residuals"</c>, <c>"step_exception"</c>. Null until the first failure.</summary>
|
||||
public string? LastFailureType { get; set; }
|
||||
|
||||
/// <summary>Short, human-readable description of the most recent failure (truncated to
|
||||
/// 1024 chars by the writer). Suitable for OpsEvent embedding and operator triage.</summary>
|
||||
public string? LastFailureMessage { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Verification-failure path only: how many residual rows the verifier <b>observed</b>
|
||||
/// before short-circuiting. This is <b>NOT</b> guaranteed to be the true residual count
|
||||
/// — <c>CascadeVerificationService</c> stops both at
|
||||
/// <see cref="DeletionProgressConstants.VerificationResidualSampleSize"/> per table AND
|
||||
/// after the first failing table, so the real count may be higher. Operators treating
|
||||
/// this number as a blast-radius estimate should add "≥" mentally when it equals the cap.
|
||||
/// Null on step-exception failures and on pre-followup progress blobs.
|
||||
/// </summary>
|
||||
public int? LastObservedResidualCount { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Verification-failure path only: JSON-encoded sample of residual <c>{table, pk, rk}</c>
|
||||
/// triples (capped at <see cref="DeletionProgressConstants.VerificationResidualSampleSize"/>).
|
||||
/// Empty/null on step-exception failures. The array length always matches
|
||||
/// <see cref="LastObservedResidualCount"/> for blobs written by the current handler
|
||||
/// because the verifier and the sample share the same cap; the two fields existed
|
||||
/// historically because the design anticipated decoupling them when the verifier becomes
|
||||
/// exhaustive.
|
||||
/// </summary>
|
||||
public string? LastResidualSampleJson { get; set; }
|
||||
}
|
||||
|
||||
/// <summary>Constants relating to <see cref="DeletionProgress"/> serialization caps.</summary>
|
||||
public static class DeletionProgressConstants
|
||||
{
|
||||
/// <summary>
|
||||
/// Cap on the verification residual sample written into the progress blob. Matches the
|
||||
/// in-memory log cap so the two payloads stay aligned. The progress blob has no size
|
||||
/// budget worth worrying about; this is the size operators see when they fetch the
|
||||
/// stored manifest from the admin Session Cleanup page.
|
||||
/// </summary>
|
||||
public const int VerificationResidualSampleSize = 50;
|
||||
|
||||
/// <summary>
|
||||
/// Maximum entries kept in the residual preview the worker embeds in the
|
||||
/// <c>SessionDeletionPoisoned</c> OpsEvent. The OpsEvents table truncates the entire
|
||||
/// <c>Details</c> column at 4096 chars (<c>TableOpsEventRepository.cs</c>); the full
|
||||
/// 50-entry sample plus the other JSON-payload fields can blow past that and leave the
|
||||
/// OpsEvent details mid-string-corrupt, which makes the admin UI fall back to raw text
|
||||
/// rendering. The OpsEvent is a Telegram-routable summary, not the forensic record —
|
||||
/// operators get the full sample from <c>DeletionProgress.LastResidualSampleJson</c>
|
||||
/// via the Session Cleanup page's stored-manifest modal.
|
||||
/// </summary>
|
||||
public const int OpsEventResidualSamplePreviewSize = 5;
|
||||
|
||||
/// <summary>
|
||||
/// Per-field length cap (table / pk / rk) inside the OpsEvent residual preview. Without
|
||||
/// this, an entry with a 200-char composite PK could on its own consume most of the
|
||||
/// 4096-char Details budget. Anything trimmed gets a trailing <c>…</c> marker so the
|
||||
/// truncation is visible to operators reading the OpsEvent.
|
||||
/// </summary>
|
||||
public const int OpsEventResidualKeyMaxChars = 96;
|
||||
|
||||
/// <summary>
|
||||
/// Total character budget for the OpsEvent residual preview JSON (before it gets
|
||||
/// JSON-string-escaped into the outer <c>Details</c> column). Sized so that, after
|
||||
/// escape inflation (~1.5×) + the other Details fields (failureMessage + tenantId +
|
||||
/// manifestId + …, ~1500 chars), the resulting <c>Details</c> string stays comfortably
|
||||
/// under the 4096-char Azure Table column cap. If the per-entry trims still leave the
|
||||
/// JSON over budget, trailing entries are dropped one at a time until it fits.
|
||||
/// </summary>
|
||||
public const int OpsEventResidualPreviewBudgetChars = 1200;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -24,6 +24,10 @@ const OPS_EVENT_TYPES: Record<string, string[]> = {
|
||||
"SessionDeletionMaintenanceFailed",
|
||||
"SessionDeletionMaintenanceFanoutSkipped",
|
||||
"SessionDeletionStrandedQueued",
|
||||
// PR-B audit consolidation: the per-session deletion_poisoned tenant-audit moved here so
|
||||
// tenant admins see only the lifecycle endpoints (deletion_started/completed/restored).
|
||||
// Operators wire Telegram on this event + read it in the Session Cleanup admin page.
|
||||
"SessionDeletionPoisoned",
|
||||
],
|
||||
Security: [
|
||||
"DeviceBlocked",
|
||||
|
||||
@@ -21,6 +21,7 @@ export interface TenantConfiguration {
|
||||
bootstrapTokenEnabled?: boolean;
|
||||
unrestrictedModeEnabled?: boolean;
|
||||
dataRetentionDays: number;
|
||||
enableCascadeDeleteV2?: boolean;
|
||||
sessionTimeoutHours: number;
|
||||
planTier?: string;
|
||||
}
|
||||
@@ -351,6 +352,14 @@ export function TenantManagementSection({
|
||||
Ready
|
||||
</span>
|
||||
)}
|
||||
{tenant.enableCascadeDeleteV2 && (
|
||||
<span
|
||||
className="inline-flex items-center px-2 py-1 rounded-full text-xs font-medium bg-purple-100 text-purple-800"
|
||||
title="V2 cascade-delete pipeline (snapshot + restore) is active for this tenant"
|
||||
>
|
||||
Cascade V2
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
<div className="flex items-center gap-2">
|
||||
<button
|
||||
@@ -570,6 +579,24 @@ export function TenantManagementSection({
|
||||
<span className="text-sm font-medium text-gray-700">Enable Unrestricted Mode</span>
|
||||
</label>
|
||||
|
||||
<div className="p-3 border border-purple-200 bg-purple-50 rounded-lg space-y-1">
|
||||
<label className="flex items-center space-x-2 cursor-pointer">
|
||||
<input
|
||||
type="checkbox"
|
||||
checked={editingTenant.enableCascadeDeleteV2 ?? false}
|
||||
onChange={(e) => setEditingTenant({ ...editingTenant, enableCascadeDeleteV2: e.target.checked })}
|
||||
className="w-4 h-4 text-purple-600 border-gray-300 rounded focus:ring-purple-500"
|
||||
/>
|
||||
<span className="text-sm font-medium text-gray-700">Enable V2 Cascade Delete</span>
|
||||
</label>
|
||||
<p className="text-xs text-gray-600 pl-6">
|
||||
Routes admin-delete and retention through the snapshot+restore pipeline.
|
||||
Activate after validating a delete with the preview endpoint. The global
|
||||
<code className="mx-1 text-xs bg-purple-100 px-1 rounded">SessionDeletionKillSwitch</code>
|
||||
halts both paths regardless of this flag.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
{/* Data Management */}
|
||||
<div>
|
||||
<label className="block text-sm font-medium text-gray-700 mb-1">Data Retention (Days)</label>
|
||||
|
||||
+302
@@ -0,0 +1,302 @@
|
||||
"use client";
|
||||
|
||||
import { useEffect, useState } from "react";
|
||||
import { api } from "@/lib/api";
|
||||
import { authenticatedFetch, TokenExpiredError } from "@/lib/authenticatedFetch";
|
||||
|
||||
interface DeletionPreviewModalProps {
|
||||
tenantId: string;
|
||||
sessionId: string;
|
||||
manifestId: string;
|
||||
onClose: () => void;
|
||||
getAccessToken: (forceRefresh?: boolean) => Promise<string | null>;
|
||||
}
|
||||
|
||||
interface ProgressView {
|
||||
snapshotSha256?: string;
|
||||
completedStepOrders?: number[];
|
||||
verificationDone?: boolean;
|
||||
tombstoneStarted?: boolean;
|
||||
completedAt?: string | null;
|
||||
aggregateDecrementsApplied?: number;
|
||||
restoreReIncrementsApplied?: number;
|
||||
lastFailureType?: string | null;
|
||||
lastFailureMessage?: string | null;
|
||||
/**
|
||||
* Verifier's OBSERVED residual count — capped at 50 per table and short-circuited at the
|
||||
* first failing table. When this equals 50 the real residual mountain is likely larger.
|
||||
*/
|
||||
lastObservedResidualCount?: number | null;
|
||||
/** JSON array of {table, pk, rk}, capped at the residual sample size (50). */
|
||||
lastResidualSampleJson?: string | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Matches the verifier cap in `CascadeVerificationService.MaxResidualSampleSize`. When the
|
||||
* observed count hits this value the operator should treat the number as a lower bound.
|
||||
*/
|
||||
const VERIFIER_OBSERVATION_CAP = 50;
|
||||
|
||||
interface StoredManifestSummary {
|
||||
manifestId?: string;
|
||||
schemaHash?: string;
|
||||
snapshotSha256?: string;
|
||||
estimatedRowCount?: number;
|
||||
estimatedSnapshotBytes?: number;
|
||||
preflightCounts?: Record<string, number>;
|
||||
sampleKeys?: Record<string, Array<{ pk: string; rk: string }>>;
|
||||
progress?: ProgressView | null;
|
||||
source?: string;
|
||||
}
|
||||
|
||||
export function DeletionPreviewModal({
|
||||
tenantId,
|
||||
sessionId,
|
||||
manifestId,
|
||||
onClose,
|
||||
getAccessToken,
|
||||
}: DeletionPreviewModalProps) {
|
||||
const [summary, setSummary] = useState<StoredManifestSummary | null>(null);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
const [loading, setLoading] = useState(true);
|
||||
const [downloading, setDownloading] = useState(false);
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false;
|
||||
(async () => {
|
||||
try {
|
||||
const resp = await authenticatedFetch(
|
||||
api.sessionDeletions.storedManifest(sessionId, tenantId, manifestId, "summary"),
|
||||
getAccessToken,
|
||||
);
|
||||
if (!resp.ok) {
|
||||
const detail = await resp.text().catch(() => "");
|
||||
throw new Error(`Stored-manifest request failed: HTTP ${resp.status}${detail ? ` — ${detail.slice(0, 200)}` : ""}`);
|
||||
}
|
||||
const json = (await resp.json()) as StoredManifestSummary;
|
||||
if (!cancelled) setSummary(json);
|
||||
} catch (err) {
|
||||
if (cancelled) return;
|
||||
if (err instanceof TokenExpiredError) {
|
||||
setError("Session expired; reload the page and try again.");
|
||||
} else {
|
||||
setError(err instanceof Error ? err.message : String(err));
|
||||
}
|
||||
} finally {
|
||||
if (!cancelled) setLoading(false);
|
||||
}
|
||||
})();
|
||||
return () => {
|
||||
cancelled = true;
|
||||
};
|
||||
}, [tenantId, sessionId, manifestId, getAccessToken]);
|
||||
|
||||
const handleDownload = async () => {
|
||||
setDownloading(true);
|
||||
setError(null);
|
||||
try {
|
||||
const resp = await authenticatedFetch(
|
||||
api.sessionDeletions.storedManifest(sessionId, tenantId, manifestId, "download"),
|
||||
getAccessToken,
|
||||
);
|
||||
if (!resp.ok) {
|
||||
throw new Error(`Download failed: HTTP ${resp.status}`);
|
||||
}
|
||||
const blob = await resp.blob();
|
||||
const url = URL.createObjectURL(blob);
|
||||
const a = document.createElement("a");
|
||||
a.href = url;
|
||||
a.download = `${tenantId}_${sessionId}_${manifestId}.snapshot.json.gz`;
|
||||
document.body.appendChild(a);
|
||||
a.click();
|
||||
a.remove();
|
||||
// Browsers keep the Blob alive until the next gc once the object URL is revoked;
|
||||
// a small timeout keeps Firefox happy on slower disks.
|
||||
setTimeout(() => URL.revokeObjectURL(url), 1000);
|
||||
} catch (err) {
|
||||
if (err instanceof TokenExpiredError) {
|
||||
setError("Session expired; reload the page and try again.");
|
||||
} else {
|
||||
setError(err instanceof Error ? err.message : String(err));
|
||||
}
|
||||
} finally {
|
||||
setDownloading(false);
|
||||
}
|
||||
};
|
||||
|
||||
const progress = summary?.progress ?? null;
|
||||
const completedCount = progress?.completedStepOrders?.length ?? 0;
|
||||
const failureSample = (() => {
|
||||
if (!progress?.lastResidualSampleJson) return null;
|
||||
try {
|
||||
const arr = JSON.parse(progress.lastResidualSampleJson) as Array<{ table: string; pk: string; rk: string }>;
|
||||
return Array.isArray(arr) ? arr : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
})();
|
||||
|
||||
return (
|
||||
<div
|
||||
className="fixed inset-0 z-50 flex items-center justify-center bg-black bg-opacity-50 p-4"
|
||||
onClick={onClose}
|
||||
>
|
||||
<div
|
||||
className="bg-white dark:bg-gray-800 rounded-lg shadow-xl max-w-3xl w-full max-h-[80vh] overflow-y-auto"
|
||||
onClick={(e) => e.stopPropagation()}
|
||||
>
|
||||
<div className="sticky top-0 bg-white dark:bg-gray-800 border-b border-gray-200 dark:border-gray-700 px-6 py-4 flex items-start justify-between rounded-t-lg">
|
||||
<div>
|
||||
<h2 className="text-lg font-semibold text-gray-900 dark:text-white">Stored Manifest</h2>
|
||||
<p className="text-xs text-gray-500 dark:text-gray-400 mt-1 font-mono">
|
||||
{tenantId} / {sessionId}
|
||||
</p>
|
||||
<p className="text-xs text-gray-500 dark:text-gray-400 mt-0.5">
|
||||
Reads the persisted snapshot the worker captured at cascade start —
|
||||
<strong className="ml-1">not</strong> a fresh dry-run.
|
||||
</p>
|
||||
</div>
|
||||
<button
|
||||
onClick={onClose}
|
||||
className="text-gray-400 hover:text-gray-600 dark:hover:text-gray-200"
|
||||
aria-label="Close"
|
||||
>
|
||||
<svg className="w-5 h-5" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||
<path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M6 18L18 6M6 6l12 12" />
|
||||
</svg>
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div className="px-6 py-4 space-y-4">
|
||||
{loading && <p className="text-sm text-gray-600 dark:text-gray-400">Loading stored manifest…</p>}
|
||||
{error && (
|
||||
<div className="p-3 bg-red-50 dark:bg-red-900/30 border border-red-200 dark:border-red-700 rounded text-sm text-red-700 dark:text-red-200">
|
||||
{error}
|
||||
</div>
|
||||
)}
|
||||
{!loading && !error && summary && (
|
||||
<>
|
||||
<div className="grid grid-cols-2 gap-3 text-sm">
|
||||
<Field label="Manifest" value={summary.manifestId ?? manifestId} mono />
|
||||
<Field label="Schema hash" value={summary.schemaHash ?? "—"} mono />
|
||||
<Field
|
||||
label="Snapshot SHA-256"
|
||||
value={summary.snapshotSha256 ?? "—"}
|
||||
mono
|
||||
/>
|
||||
<Field
|
||||
label="Snapshot size"
|
||||
value={
|
||||
summary.estimatedSnapshotBytes != null
|
||||
? formatBytes(summary.estimatedSnapshotBytes)
|
||||
: "—"
|
||||
}
|
||||
/>
|
||||
</div>
|
||||
|
||||
{progress && (
|
||||
<div className="border border-purple-200 dark:border-purple-700 bg-purple-50 dark:bg-purple-900/20 rounded p-3 text-sm">
|
||||
<h3 className="text-sm font-semibold text-purple-900 dark:text-purple-100 mb-2">Worker progress</h3>
|
||||
<div className="grid grid-cols-2 gap-2 text-xs">
|
||||
<span className="text-gray-700 dark:text-gray-300">Completed steps: <strong>{completedCount}</strong></span>
|
||||
<span className="text-gray-700 dark:text-gray-300">Verification done: <strong>{progress.verificationDone ? "yes" : "no"}</strong></span>
|
||||
<span className="text-gray-700 dark:text-gray-300">Tombstone started: <strong>{progress.tombstoneStarted ? "yes" : "no"}</strong></span>
|
||||
<span className="text-gray-700 dark:text-gray-300">Aggregate decrements: <strong>{progress.aggregateDecrementsApplied ?? 0}</strong></span>
|
||||
</div>
|
||||
{progress.lastFailureType && (
|
||||
<div className="mt-3 p-2 bg-red-100 dark:bg-red-900/40 rounded text-xs text-red-800 dark:text-red-100">
|
||||
<div><strong>Last failure:</strong> <code>{progress.lastFailureType}</code></div>
|
||||
{progress.lastFailureMessage && (
|
||||
<div className="mt-1 break-words">{progress.lastFailureMessage}</div>
|
||||
)}
|
||||
{failureSample && failureSample.length > 0 && (() => {
|
||||
const sampleSize = failureSample.length;
|
||||
const observed = progress.lastObservedResidualCount ?? sampleSize;
|
||||
const capHit = observed >= VERIFIER_OBSERVATION_CAP;
|
||||
return (
|
||||
<details className="mt-2">
|
||||
<summary className="cursor-pointer text-red-700 dark:text-red-200">
|
||||
Residual sample ({sampleSize} of {observed}{capHit ? "+" : ""} observed row{observed === 1 ? "" : "s"})
|
||||
</summary>
|
||||
<ul className="mt-1 pl-4 list-disc font-mono break-all">
|
||||
{failureSample.slice(0, 20).map((r, i) => (
|
||||
<li key={i}>{r.table}: {r.pk} / {r.rk}</li>
|
||||
))}
|
||||
</ul>
|
||||
{capHit && (
|
||||
<p className="mt-2 text-xs italic text-red-700 dark:text-red-200">
|
||||
The verifier short-circuits at {VERIFIER_OBSERVATION_CAP} rows per
|
||||
table and after the first failing table — the real residual count
|
||||
may be higher. Use the cascade-restore path to recover; do not rely
|
||||
on this number as an exact blast-radius estimate.
|
||||
</p>
|
||||
)}
|
||||
</details>
|
||||
);
|
||||
})()}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{summary.preflightCounts && (
|
||||
<div>
|
||||
<h3 className="text-sm font-semibold text-gray-700 dark:text-gray-200 mb-2">Rows in snapshot (by table)</h3>
|
||||
<div className="border border-gray-200 dark:border-gray-700 rounded overflow-hidden">
|
||||
<table className="w-full text-sm">
|
||||
<thead className="bg-gray-50 dark:bg-gray-900">
|
||||
<tr>
|
||||
<th className="text-left px-3 py-2 text-xs font-medium text-gray-600 dark:text-gray-400 uppercase">Table</th>
|
||||
<th className="text-right px-3 py-2 text-xs font-medium text-gray-600 dark:text-gray-400 uppercase">Count</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody className="divide-y divide-gray-200 dark:divide-gray-700">
|
||||
{Object.entries(summary.preflightCounts).map(([table, count]) => (
|
||||
<tr key={table}>
|
||||
<td className="px-3 py-2 font-mono text-xs text-gray-800 dark:text-gray-200">{table}</td>
|
||||
<td className="px-3 py-2 text-right text-gray-800 dark:text-gray-200">{count}</td>
|
||||
</tr>
|
||||
))}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
|
||||
<div className="sticky bottom-0 bg-gray-50 dark:bg-gray-900 px-6 py-3 border-t border-gray-200 dark:border-gray-700 flex justify-end gap-2 rounded-b-lg">
|
||||
<button
|
||||
onClick={handleDownload}
|
||||
disabled={loading || downloading || !!error}
|
||||
className="px-3 py-2 text-sm border border-gray-300 dark:border-gray-600 rounded text-gray-700 dark:text-gray-200 hover:bg-gray-100 dark:hover:bg-gray-700 disabled:opacity-50"
|
||||
>
|
||||
{downloading ? "Downloading…" : "Download snapshot"}
|
||||
</button>
|
||||
<button
|
||||
onClick={onClose}
|
||||
className="px-3 py-2 text-sm bg-gray-200 dark:bg-gray-700 text-gray-800 dark:text-gray-100 rounded hover:bg-gray-300 dark:hover:bg-gray-600"
|
||||
>
|
||||
Close
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function Field({ label, value, mono }: { label: string; value: string; mono?: boolean }) {
|
||||
return (
|
||||
<div>
|
||||
<p className="text-xs text-gray-500 dark:text-gray-400 uppercase">{label}</p>
|
||||
<p className={`text-sm text-gray-900 dark:text-white ${mono ? "font-mono break-all" : ""}`}>{value}</p>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function formatBytes(bytes: number): string {
|
||||
if (bytes < 1024) return `${bytes} B`;
|
||||
if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KB`;
|
||||
return `${(bytes / 1024 / 1024).toFixed(2)} MB`;
|
||||
}
|
||||
+180
@@ -0,0 +1,180 @@
|
||||
"use client";
|
||||
|
||||
import { useState } from "react";
|
||||
import { api } from "@/lib/api";
|
||||
import { authenticatedFetch, TokenExpiredError } from "@/lib/authenticatedFetch";
|
||||
|
||||
interface RestoreConfirmDialogProps {
|
||||
tenantId: string;
|
||||
sessionId: string;
|
||||
manifestId: string;
|
||||
onClose: () => void;
|
||||
onRestored: () => void;
|
||||
getAccessToken: (forceRefresh?: boolean) => Promise<string | null>;
|
||||
}
|
||||
|
||||
export function RestoreConfirmDialog({
|
||||
tenantId,
|
||||
sessionId,
|
||||
manifestId,
|
||||
onClose,
|
||||
onRestored,
|
||||
getAccessToken,
|
||||
}: RestoreConfirmDialogProps) {
|
||||
const [reason, setReason] = useState("");
|
||||
const [dryRun, setDryRun] = useState(true);
|
||||
const [submitting, setSubmitting] = useState(false);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
const [outcome, setOutcome] = useState<{ outcome: string; mode: string | null; dryRun: boolean } | null>(null);
|
||||
|
||||
const handleSubmit = async () => {
|
||||
setSubmitting(true);
|
||||
setError(null);
|
||||
setOutcome(null);
|
||||
try {
|
||||
const resp = await authenticatedFetch(
|
||||
api.sessionDeletions.restore(sessionId),
|
||||
getAccessToken,
|
||||
{
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
// Codex F1: explicit tenantId is mandatory for cross-tenant GA restores.
|
||||
// Codex follow-up: backend selects the restore mode automatically based on the
|
||||
// cascade state (Full / Partial); operators cannot — and must not — force a Full
|
||||
// restore against a Poisoned session or vice versa. The reason is persisted into
|
||||
// the deletion_restored audit row as `details.reason`.
|
||||
body: JSON.stringify({
|
||||
tenantId,
|
||||
manifestId,
|
||||
reason: reason.trim() || null,
|
||||
dryRun,
|
||||
}),
|
||||
},
|
||||
);
|
||||
const body = await resp.json().catch(() => ({}));
|
||||
if (!resp.ok) {
|
||||
throw new Error(body?.message ?? `Restore failed: HTTP ${resp.status}`);
|
||||
}
|
||||
setOutcome({
|
||||
outcome: body?.outcome ?? "Completed",
|
||||
mode: typeof body?.mode === "string" ? body.mode : null,
|
||||
dryRun,
|
||||
});
|
||||
if (!dryRun) {
|
||||
// Real restore done — let the parent refresh its list.
|
||||
onRestored();
|
||||
}
|
||||
} catch (err) {
|
||||
if (err instanceof TokenExpiredError) {
|
||||
setError("Session expired; reload the page and try again.");
|
||||
} else {
|
||||
setError(err instanceof Error ? err.message : String(err));
|
||||
}
|
||||
} finally {
|
||||
setSubmitting(false);
|
||||
}
|
||||
};
|
||||
|
||||
return (
|
||||
<div
|
||||
className="fixed inset-0 z-50 flex items-center justify-center bg-black bg-opacity-50 p-4"
|
||||
onClick={onClose}
|
||||
>
|
||||
<div
|
||||
className="bg-white dark:bg-gray-800 rounded-lg shadow-xl max-w-lg w-full"
|
||||
onClick={(e) => e.stopPropagation()}
|
||||
>
|
||||
<div className="border-b border-gray-200 dark:border-gray-700 px-6 py-4 flex items-center gap-3">
|
||||
<div className="w-10 h-10 bg-amber-100 dark:bg-amber-900/30 rounded-full flex items-center justify-center">
|
||||
<svg className="w-5 h-5 text-amber-600 dark:text-amber-300" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||
<path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M4 4v6h6M20 20v-6h-6M3 11a8 8 0 0114-5l3 3M21 13a8 8 0 01-14 5l-3-3" />
|
||||
</svg>
|
||||
</div>
|
||||
<div>
|
||||
<h2 className="text-lg font-semibold text-gray-900 dark:text-white">Restore session from snapshot</h2>
|
||||
<p className="text-xs text-gray-500 dark:text-gray-400 mt-0.5 font-mono">{tenantId} / {sessionId}</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="px-6 py-4 space-y-4">
|
||||
<div className="p-3 border border-gray-200 dark:border-gray-700 bg-gray-50 dark:bg-gray-900/40 rounded text-sm text-gray-700 dark:text-gray-200">
|
||||
<p>
|
||||
<strong>Mode is selected automatically</strong> based on the cascade state at execution time:
|
||||
</p>
|
||||
<ul className="mt-1 pl-5 list-disc text-xs text-gray-600 dark:text-gray-400 space-y-0.5">
|
||||
<li><strong>Partial</strong> — Poisoned cascade: replays only the rows the worker had already removed and re-increments inventory counters.</li>
|
||||
<li><strong>Full</strong> — Completed cascade (Sessions row gone): inserts every row from the snapshot and resets <code className="bg-gray-200 dark:bg-gray-700 px-1 rounded">DeletionState=None</code>.</li>
|
||||
</ul>
|
||||
<p className="mt-2 text-xs text-gray-500 dark:text-gray-400">
|
||||
Forcing the wrong mode would corrupt state, so the choice is not exposed. Run with <strong>dry-run</strong> first to confirm the predicted mode.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div>
|
||||
<label className="block text-sm font-medium text-gray-700 dark:text-gray-200 mb-1">
|
||||
Reason (audit trail)
|
||||
</label>
|
||||
<input
|
||||
type="text"
|
||||
value={reason}
|
||||
onChange={(e) => setReason(e.target.value)}
|
||||
placeholder="e.g. customer support — accidental delete"
|
||||
maxLength={200}
|
||||
className="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 rounded text-sm text-gray-900 dark:text-white bg-white dark:bg-gray-900 focus:outline-none focus:ring-2 focus:ring-blue-500"
|
||||
/>
|
||||
<p className="text-xs text-gray-500 dark:text-gray-400 mt-1">
|
||||
Stored on the <code className="bg-gray-100 dark:bg-gray-700 px-1 rounded">deletion_restored</code> audit row.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<label className="flex items-center gap-2 cursor-pointer p-2 border border-amber-200 dark:border-amber-800 bg-amber-50 dark:bg-amber-900/20 rounded">
|
||||
<input
|
||||
type="checkbox"
|
||||
checked={dryRun}
|
||||
onChange={(e) => setDryRun(e.target.checked)}
|
||||
className="w-4 h-4"
|
||||
/>
|
||||
<span className="text-sm text-amber-900 dark:text-amber-100">
|
||||
<strong>Dry run</strong> — verify the snapshot can be replayed without writing anything.
|
||||
Recommended for the first attempt.
|
||||
</span>
|
||||
</label>
|
||||
|
||||
{outcome && (
|
||||
<div className="p-3 bg-green-50 dark:bg-green-900/30 border border-green-200 dark:border-green-700 rounded text-sm text-green-800 dark:text-green-200">
|
||||
Outcome: <strong>{outcome.outcome}</strong>
|
||||
{outcome.mode && (
|
||||
<> · Mode (auto-selected): <strong className="capitalize">{outcome.mode}</strong></>
|
||||
)}
|
||||
{outcome.dryRun ? " · dry-run — no rows written" : ""}
|
||||
</div>
|
||||
)}
|
||||
{error && (
|
||||
<div className="p-3 bg-red-50 dark:bg-red-900/30 border border-red-200 dark:border-red-700 rounded text-sm text-red-700 dark:text-red-200">
|
||||
{error}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
||||
<div className="bg-gray-50 dark:bg-gray-900 px-6 py-3 border-t border-gray-200 dark:border-gray-700 flex justify-end gap-2 rounded-b-lg">
|
||||
<button
|
||||
onClick={onClose}
|
||||
disabled={submitting}
|
||||
className="px-3 py-2 text-sm border border-gray-300 dark:border-gray-600 rounded text-gray-700 dark:text-gray-200 hover:bg-gray-100 dark:hover:bg-gray-700 disabled:opacity-50"
|
||||
>
|
||||
Close
|
||||
</button>
|
||||
<button
|
||||
onClick={handleSubmit}
|
||||
disabled={submitting}
|
||||
className={`px-3 py-2 text-sm text-white rounded disabled:opacity-50 ${
|
||||
dryRun ? "bg-blue-600 hover:bg-blue-700" : "bg-amber-600 hover:bg-amber-700"
|
||||
}`}
|
||||
>
|
||||
{submitting ? "Running…" : dryRun ? "Run dry-run" : "Restore"}
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,506 @@
|
||||
"use client";
|
||||
|
||||
import { useCallback, useEffect, useState } from "react";
|
||||
import Link from "next/link";
|
||||
import { api } from "@/lib/api";
|
||||
import { authenticatedFetch, TokenExpiredError } from "@/lib/authenticatedFetch";
|
||||
import { useAdminConfig } from "../../AdminConfigContext";
|
||||
import { AdminNotifications } from "../../AdminNotifications";
|
||||
import { DeletionPreviewModal } from "./components/DeletionPreviewModal";
|
||||
import { RestoreConfirmDialog } from "./components/RestoreConfirmDialog";
|
||||
|
||||
type CascadeState = "Preparing" | "Queued" | "Running" | "Poisoned";
|
||||
|
||||
interface SessionDeletionRow {
|
||||
tenantId: string;
|
||||
sessionId: string;
|
||||
deletionState: CascadeState | string;
|
||||
manifestId: string;
|
||||
timestamp: string;
|
||||
ageMinutes: number;
|
||||
}
|
||||
|
||||
interface SessionDeletionListResponse {
|
||||
success: boolean;
|
||||
state: string;
|
||||
strandedSinceMinutes: number | null;
|
||||
count: number;
|
||||
sessions: SessionDeletionRow[];
|
||||
}
|
||||
|
||||
type Tab = "in-flight" | "poisoned" | "stranded" | "activity";
|
||||
|
||||
const STRANDED_THRESHOLD_MINUTES = 30;
|
||||
|
||||
const IN_FLIGHT_STATES: CascadeState[] = ["Preparing", "Queued", "Running"];
|
||||
|
||||
export default function SessionCleanupPage() {
|
||||
const { getAccessToken, setError, setSuccessMessage } = useAdminConfig();
|
||||
const [activeTab, setActiveTab] = useState<Tab>("in-flight");
|
||||
|
||||
return (
|
||||
<div className="min-h-screen bg-gray-50 dark:bg-gray-900">
|
||||
<header className="bg-white dark:bg-gray-800 shadow dark:shadow-gray-700">
|
||||
<div className="py-6 px-4 sm:px-6 lg:px-8">
|
||||
<h1 className="text-2xl font-normal text-gray-900 dark:text-white">Session Cleanup</h1>
|
||||
<p className="text-sm text-gray-600 dark:text-gray-400 mt-1">
|
||||
V2 cascade-delete observability — In-Flight, Poisoned, Stranded queue, and lifecycle
|
||||
activity. Operator actions: preview the planned manifest, or restore from snapshot.
|
||||
</p>
|
||||
</div>
|
||||
</header>
|
||||
<main className="max-w-7xl mx-auto px-4 sm:px-6 lg:px-8 py-8 space-y-6">
|
||||
<AdminNotifications />
|
||||
|
||||
{/* Tab bar */}
|
||||
<div className="border-b border-gray-200 dark:border-gray-700">
|
||||
<nav className="-mb-px flex gap-6 overflow-x-auto">
|
||||
<TabButton active={activeTab === "in-flight"} onClick={() => setActiveTab("in-flight")}>
|
||||
In-Flight
|
||||
</TabButton>
|
||||
<TabButton active={activeTab === "poisoned"} onClick={() => setActiveTab("poisoned")}>
|
||||
Poisoned
|
||||
</TabButton>
|
||||
<TabButton active={activeTab === "stranded"} onClick={() => setActiveTab("stranded")}>
|
||||
Stranded Queued
|
||||
</TabButton>
|
||||
<TabButton active={activeTab === "activity"} onClick={() => setActiveTab("activity")}>
|
||||
Activity
|
||||
</TabButton>
|
||||
</nav>
|
||||
</div>
|
||||
|
||||
{activeTab === "in-flight" && (
|
||||
<InFlightTab
|
||||
getAccessToken={getAccessToken}
|
||||
setError={setError}
|
||||
setSuccessMessage={setSuccessMessage}
|
||||
/>
|
||||
)}
|
||||
{activeTab === "poisoned" && (
|
||||
<PoisonedTab
|
||||
getAccessToken={getAccessToken}
|
||||
setError={setError}
|
||||
setSuccessMessage={setSuccessMessage}
|
||||
/>
|
||||
)}
|
||||
{activeTab === "stranded" && (
|
||||
<StrandedTab getAccessToken={getAccessToken} setError={setError} />
|
||||
)}
|
||||
{activeTab === "activity" && <ActivityTab />}
|
||||
</main>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function TabButton({
|
||||
active,
|
||||
onClick,
|
||||
children,
|
||||
}: {
|
||||
active: boolean;
|
||||
onClick: () => void;
|
||||
children: React.ReactNode;
|
||||
}) {
|
||||
return (
|
||||
<button
|
||||
onClick={onClick}
|
||||
className={`py-3 px-1 border-b-2 text-sm font-medium whitespace-nowrap transition-colors ${
|
||||
active
|
||||
? "border-purple-500 text-purple-600 dark:text-purple-300"
|
||||
: "border-transparent text-gray-500 dark:text-gray-400 hover:text-gray-700 dark:hover:text-gray-200 hover:border-gray-300"
|
||||
}`}
|
||||
>
|
||||
{children}
|
||||
</button>
|
||||
);
|
||||
}
|
||||
|
||||
// ── In-Flight tab ────────────────────────────────────────────────────────────
|
||||
|
||||
function InFlightTab({
|
||||
getAccessToken,
|
||||
setError,
|
||||
setSuccessMessage,
|
||||
}: {
|
||||
getAccessToken: (forceRefresh?: boolean) => Promise<string | null>;
|
||||
setError: (error: string | null) => void;
|
||||
setSuccessMessage: (message: string | null) => void;
|
||||
}) {
|
||||
const [rows, setRows] = useState<SessionDeletionRow[]>([]);
|
||||
const [loading, setLoading] = useState(true);
|
||||
const [refreshKey, setRefreshKey] = useState(0);
|
||||
const [previewing, setPreviewing] = useState<SessionDeletionRow | null>(null);
|
||||
|
||||
const fetchAll = useCallback(async () => {
|
||||
setLoading(true);
|
||||
try {
|
||||
const results = await Promise.all(
|
||||
IN_FLIGHT_STATES.map((state) =>
|
||||
authenticatedFetch(api.sessionDeletions.list(state), getAccessToken).then(async (r) => {
|
||||
if (!r.ok) throw new Error(`HTTP ${r.status} for state=${state}`);
|
||||
return (await r.json()) as SessionDeletionListResponse;
|
||||
}),
|
||||
),
|
||||
);
|
||||
const all = results.flatMap((r) => r.sessions);
|
||||
// Sort by age desc (oldest first → likely the one needing attention).
|
||||
all.sort((a, b) => b.ageMinutes - a.ageMinutes);
|
||||
setRows(all);
|
||||
} catch (err) {
|
||||
if (err instanceof TokenExpiredError) {
|
||||
setError("Session expired; reload the page and try again.");
|
||||
} else {
|
||||
setError(err instanceof Error ? err.message : String(err));
|
||||
}
|
||||
} finally {
|
||||
setLoading(false);
|
||||
}
|
||||
}, [getAccessToken, setError]);
|
||||
|
||||
useEffect(() => {
|
||||
void fetchAll();
|
||||
}, [fetchAll, refreshKey]);
|
||||
|
||||
return (
|
||||
<>
|
||||
<SessionList
|
||||
title="Cascade in progress"
|
||||
description="Sessions where the cascade-delete pipeline is currently active. Empty list = no V2 cascade in flight across all tenants."
|
||||
rows={rows}
|
||||
loading={loading}
|
||||
onRefresh={() => setRefreshKey((k) => k + 1)}
|
||||
renderRowActions={(row) => (
|
||||
<button
|
||||
onClick={() => setPreviewing(row)}
|
||||
className="px-3 py-1 text-xs bg-blue-600 text-white rounded hover:bg-blue-700"
|
||||
>
|
||||
Preview
|
||||
</button>
|
||||
)}
|
||||
stateColors={{
|
||||
Preparing: "bg-amber-100 text-amber-800 dark:bg-amber-900/30 dark:text-amber-200",
|
||||
Queued: "bg-blue-100 text-blue-800 dark:bg-blue-900/30 dark:text-blue-200",
|
||||
Running: "bg-purple-100 text-purple-800 dark:bg-purple-900/30 dark:text-purple-200",
|
||||
}}
|
||||
emptyMessage="No active cascades."
|
||||
successMessage={null}
|
||||
/>
|
||||
{previewing && (
|
||||
<DeletionPreviewModal
|
||||
tenantId={previewing.tenantId}
|
||||
sessionId={previewing.sessionId}
|
||||
manifestId={previewing.manifestId}
|
||||
onClose={() => setPreviewing(null)}
|
||||
getAccessToken={getAccessToken}
|
||||
/>
|
||||
)}
|
||||
</>
|
||||
);
|
||||
}
|
||||
|
||||
// ── Poisoned tab ─────────────────────────────────────────────────────────────
|
||||
|
||||
function PoisonedTab({
|
||||
getAccessToken,
|
||||
setError,
|
||||
setSuccessMessage,
|
||||
}: {
|
||||
getAccessToken: (forceRefresh?: boolean) => Promise<string | null>;
|
||||
setError: (error: string | null) => void;
|
||||
setSuccessMessage: (message: string | null) => void;
|
||||
}) {
|
||||
const [rows, setRows] = useState<SessionDeletionRow[]>([]);
|
||||
const [loading, setLoading] = useState(true);
|
||||
const [refreshKey, setRefreshKey] = useState(0);
|
||||
const [previewing, setPreviewing] = useState<SessionDeletionRow | null>(null);
|
||||
const [restoring, setRestoring] = useState<SessionDeletionRow | null>(null);
|
||||
|
||||
const fetchPoisoned = useCallback(async () => {
|
||||
setLoading(true);
|
||||
try {
|
||||
const resp = await authenticatedFetch(api.sessionDeletions.list("Poisoned"), getAccessToken);
|
||||
if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
|
||||
const json = (await resp.json()) as SessionDeletionListResponse;
|
||||
const sorted = [...json.sessions].sort((a, b) => b.ageMinutes - a.ageMinutes);
|
||||
setRows(sorted);
|
||||
} catch (err) {
|
||||
if (err instanceof TokenExpiredError) {
|
||||
setError("Session expired; reload the page and try again.");
|
||||
} else {
|
||||
setError(err instanceof Error ? err.message : String(err));
|
||||
}
|
||||
} finally {
|
||||
setLoading(false);
|
||||
}
|
||||
}, [getAccessToken, setError]);
|
||||
|
||||
useEffect(() => {
|
||||
void fetchPoisoned();
|
||||
}, [fetchPoisoned, refreshKey]);
|
||||
|
||||
return (
|
||||
<>
|
||||
<SessionList
|
||||
title="Poisoned cascades"
|
||||
description="Sessions where the cascade hit max-dequeue or live verification failed. They need an operator restore to recover."
|
||||
rows={rows}
|
||||
loading={loading}
|
||||
onRefresh={() => setRefreshKey((k) => k + 1)}
|
||||
renderRowActions={(row) => (
|
||||
<div className="flex gap-1">
|
||||
<button
|
||||
onClick={() => setPreviewing(row)}
|
||||
className="px-3 py-1 text-xs bg-gray-200 dark:bg-gray-700 text-gray-800 dark:text-gray-100 rounded hover:bg-gray-300 dark:hover:bg-gray-600"
|
||||
>
|
||||
Preview
|
||||
</button>
|
||||
<button
|
||||
onClick={() => setRestoring(row)}
|
||||
className="px-3 py-1 text-xs bg-amber-600 text-white rounded hover:bg-amber-700"
|
||||
>
|
||||
Restore…
|
||||
</button>
|
||||
</div>
|
||||
)}
|
||||
stateColors={{
|
||||
Poisoned: "bg-red-100 text-red-800 dark:bg-red-900/30 dark:text-red-200",
|
||||
}}
|
||||
emptyMessage="No poisoned cascades — V2 is running clean."
|
||||
successMessage={null}
|
||||
/>
|
||||
{previewing && (
|
||||
<DeletionPreviewModal
|
||||
tenantId={previewing.tenantId}
|
||||
sessionId={previewing.sessionId}
|
||||
manifestId={previewing.manifestId}
|
||||
onClose={() => setPreviewing(null)}
|
||||
getAccessToken={getAccessToken}
|
||||
/>
|
||||
)}
|
||||
{restoring && (
|
||||
<RestoreConfirmDialog
|
||||
tenantId={restoring.tenantId}
|
||||
sessionId={restoring.sessionId}
|
||||
manifestId={restoring.manifestId}
|
||||
onClose={() => setRestoring(null)}
|
||||
onRestored={() => {
|
||||
setRestoring(null);
|
||||
setSuccessMessage(`Session ${restoring.sessionId} restore submitted.`);
|
||||
setRefreshKey((k) => k + 1);
|
||||
}}
|
||||
getAccessToken={getAccessToken}
|
||||
/>
|
||||
)}
|
||||
</>
|
||||
);
|
||||
}
|
||||
|
||||
// ── Stranded tab ─────────────────────────────────────────────────────────────
|
||||
|
||||
function StrandedTab({
|
||||
getAccessToken,
|
||||
setError,
|
||||
}: {
|
||||
getAccessToken: (forceRefresh?: boolean) => Promise<string | null>;
|
||||
setError: (error: string | null) => void;
|
||||
}) {
|
||||
const [rows, setRows] = useState<SessionDeletionRow[]>([]);
|
||||
const [loading, setLoading] = useState(true);
|
||||
const [refreshKey, setRefreshKey] = useState(0);
|
||||
const [previewing, setPreviewing] = useState<SessionDeletionRow | null>(null);
|
||||
|
||||
const fetchStranded = useCallback(async () => {
|
||||
setLoading(true);
|
||||
try {
|
||||
const resp = await authenticatedFetch(
|
||||
api.sessionDeletions.list("Queued", STRANDED_THRESHOLD_MINUTES),
|
||||
getAccessToken,
|
||||
);
|
||||
if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
|
||||
const json = (await resp.json()) as SessionDeletionListResponse;
|
||||
const sorted = [...json.sessions].sort((a, b) => b.ageMinutes - a.ageMinutes);
|
||||
setRows(sorted);
|
||||
} catch (err) {
|
||||
if (err instanceof TokenExpiredError) {
|
||||
setError("Session expired; reload the page and try again.");
|
||||
} else {
|
||||
setError(err instanceof Error ? err.message : String(err));
|
||||
}
|
||||
} finally {
|
||||
setLoading(false);
|
||||
}
|
||||
}, [getAccessToken, setError]);
|
||||
|
||||
useEffect(() => {
|
||||
void fetchStranded();
|
||||
}, [fetchStranded, refreshKey]);
|
||||
|
||||
return (
|
||||
<>
|
||||
<SessionList
|
||||
title="Stranded queued"
|
||||
description={`Sessions in DeletionState=Queued for more than ${STRANDED_THRESHOLD_MINUTES} minutes — the worker hasn't picked them up. Matches the SessionDeletionStrandedQueued OpsEvent watchdog.`}
|
||||
rows={rows}
|
||||
loading={loading}
|
||||
onRefresh={() => setRefreshKey((k) => k + 1)}
|
||||
renderRowActions={(row) => (
|
||||
<button
|
||||
onClick={() => setPreviewing(row)}
|
||||
className="px-3 py-1 text-xs bg-blue-600 text-white rounded hover:bg-blue-700"
|
||||
>
|
||||
Preview
|
||||
</button>
|
||||
)}
|
||||
stateColors={{
|
||||
Queued: "bg-yellow-100 text-yellow-800 dark:bg-yellow-900/30 dark:text-yellow-200",
|
||||
}}
|
||||
emptyMessage="No stranded queued cascades."
|
||||
successMessage={null}
|
||||
/>
|
||||
{previewing && (
|
||||
<DeletionPreviewModal
|
||||
tenantId={previewing.tenantId}
|
||||
sessionId={previewing.sessionId}
|
||||
manifestId={previewing.manifestId}
|
||||
onClose={() => setPreviewing(null)}
|
||||
getAccessToken={getAccessToken}
|
||||
/>
|
||||
)}
|
||||
</>
|
||||
);
|
||||
}
|
||||
|
||||
// ── Activity tab ─────────────────────────────────────────────────────────────
|
||||
|
||||
function ActivityTab() {
|
||||
return (
|
||||
<div className="bg-white dark:bg-gray-800 border border-gray-200 dark:border-gray-700 rounded-lg p-6">
|
||||
<h2 className="text-lg font-medium text-gray-900 dark:text-white mb-2">Cascade-Delete Activity</h2>
|
||||
<p className="text-sm text-gray-600 dark:text-gray-400 mb-4">
|
||||
Tenant audit-log action strings for V2 cascade lifecycle:
|
||||
<code className="mx-1 px-1 bg-gray-100 dark:bg-gray-700 rounded text-xs">deletion_started</code>,
|
||||
<code className="mx-1 px-1 bg-gray-100 dark:bg-gray-700 rounded text-xs">deletion_completed</code>,
|
||||
<code className="mx-1 px-1 bg-gray-100 dark:bg-gray-700 rounded text-xs">deletion_restored</code>.
|
||||
Retention-driven cascades land as
|
||||
<code className="mx-1 px-1 bg-gray-100 dark:bg-gray-700 rounded text-xs">SessionDeletionMaintenanceFanout</code>
|
||||
per tenant (aggregate counts) plus the individual <code className="px-1 bg-gray-100 dark:bg-gray-700 rounded text-xs">deletion_started</code>/<code className="px-1 bg-gray-100 dark:bg-gray-700 rounded text-xs">deletion_completed</code> rows.
|
||||
</p>
|
||||
<p className="text-sm text-gray-600 dark:text-gray-400 mb-4">
|
||||
Step-level progress (per-table deletes, verification residuals) lives in DeletionProgress
|
||||
blobs and structured logs — not in the tenant audit trail. Operator-scoped signals are
|
||||
emitted as OpsEvents instead:
|
||||
<code className="mx-1 px-1 bg-gray-100 dark:bg-gray-700 rounded text-xs">SessionDeletionPoisoned</code>
|
||||
(max-dequeue or verification failure — see Poisoned tab),
|
||||
<code className="mx-1 px-1 bg-gray-100 dark:bg-gray-700 rounded text-xs">SessionDeletionStrandedQueued</code>
|
||||
(Stranded tab),
|
||||
<code className="mx-1 px-1 bg-gray-100 dark:bg-gray-700 rounded text-xs">SessionDeletionMaintenanceFanoutSkipped</code>
|
||||
(kill-switch active during a retention sweep). All available on the Ops Events page and
|
||||
wirable for Telegram alerts.
|
||||
</p>
|
||||
<Link
|
||||
href="/audit"
|
||||
className="inline-flex items-center px-4 py-2 text-sm bg-purple-600 text-white rounded hover:bg-purple-700"
|
||||
>
|
||||
Open audit log →
|
||||
</Link>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
// ── Shared list component ────────────────────────────────────────────────────
|
||||
|
||||
function SessionList({
|
||||
title,
|
||||
description,
|
||||
rows,
|
||||
loading,
|
||||
onRefresh,
|
||||
renderRowActions,
|
||||
stateColors,
|
||||
emptyMessage,
|
||||
}: {
|
||||
title: string;
|
||||
description: string;
|
||||
rows: SessionDeletionRow[];
|
||||
loading: boolean;
|
||||
onRefresh: () => void;
|
||||
renderRowActions: (row: SessionDeletionRow) => React.ReactNode;
|
||||
stateColors: Record<string, string>;
|
||||
emptyMessage: string;
|
||||
successMessage: string | null;
|
||||
}) {
|
||||
return (
|
||||
<div className="bg-white dark:bg-gray-800 border border-gray-200 dark:border-gray-700 rounded-lg">
|
||||
<div className="px-6 py-4 border-b border-gray-200 dark:border-gray-700 flex items-start justify-between gap-4">
|
||||
<div>
|
||||
<h2 className="text-lg font-medium text-gray-900 dark:text-white">{title}</h2>
|
||||
<p className="text-sm text-gray-600 dark:text-gray-400 mt-1">{description}</p>
|
||||
</div>
|
||||
<button
|
||||
onClick={onRefresh}
|
||||
disabled={loading}
|
||||
className="px-3 py-1.5 text-sm border border-gray-300 dark:border-gray-600 rounded text-gray-700 dark:text-gray-200 hover:bg-gray-100 dark:hover:bg-gray-700 disabled:opacity-50 whitespace-nowrap"
|
||||
>
|
||||
{loading ? "Loading…" : "Refresh"}
|
||||
</button>
|
||||
</div>
|
||||
|
||||
{loading && (
|
||||
<div className="px-6 py-8 text-center text-sm text-gray-500 dark:text-gray-400">Loading…</div>
|
||||
)}
|
||||
{!loading && rows.length === 0 && (
|
||||
<div className="px-6 py-8 text-center text-sm text-gray-500 dark:text-gray-400">{emptyMessage}</div>
|
||||
)}
|
||||
{!loading && rows.length > 0 && (
|
||||
<div className="overflow-x-auto">
|
||||
<table className="w-full text-sm">
|
||||
<thead className="bg-gray-50 dark:bg-gray-900">
|
||||
<tr>
|
||||
<th className="text-left px-4 py-2 text-xs font-medium text-gray-600 dark:text-gray-400 uppercase">Tenant</th>
|
||||
<th className="text-left px-4 py-2 text-xs font-medium text-gray-600 dark:text-gray-400 uppercase">Session</th>
|
||||
<th className="text-left px-4 py-2 text-xs font-medium text-gray-600 dark:text-gray-400 uppercase">State</th>
|
||||
<th className="text-left px-4 py-2 text-xs font-medium text-gray-600 dark:text-gray-400 uppercase">Manifest</th>
|
||||
<th className="text-right px-4 py-2 text-xs font-medium text-gray-600 dark:text-gray-400 uppercase">Age</th>
|
||||
<th className="text-right px-4 py-2 text-xs font-medium text-gray-600 dark:text-gray-400 uppercase">Actions</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody className="divide-y divide-gray-200 dark:divide-gray-700">
|
||||
{rows.map((row) => (
|
||||
<tr key={`${row.tenantId}-${row.sessionId}`}>
|
||||
<td className="px-4 py-2 font-mono text-xs text-gray-700 dark:text-gray-300">{row.tenantId}</td>
|
||||
<td className="px-4 py-2 font-mono text-xs text-gray-700 dark:text-gray-300">{row.sessionId}</td>
|
||||
<td className="px-4 py-2">
|
||||
<span
|
||||
className={`inline-flex items-center px-2 py-0.5 rounded text-xs font-medium ${
|
||||
stateColors[row.deletionState] ??
|
||||
"bg-gray-100 text-gray-800 dark:bg-gray-700 dark:text-gray-200"
|
||||
}`}
|
||||
>
|
||||
{row.deletionState}
|
||||
</span>
|
||||
</td>
|
||||
<td className="px-4 py-2 font-mono text-xs text-gray-500 dark:text-gray-400 truncate max-w-[12rem]" title={row.manifestId}>
|
||||
{row.manifestId || "—"}
|
||||
</td>
|
||||
<td className="px-4 py-2 text-right text-xs text-gray-600 dark:text-gray-400">
|
||||
{formatAge(row.ageMinutes)}
|
||||
</td>
|
||||
<td className="px-4 py-2 text-right">{renderRowActions(row)}</td>
|
||||
</tr>
|
||||
))}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function formatAge(minutes: number): string {
|
||||
if (minutes < 60) return `${minutes}m`;
|
||||
if (minutes < 60 * 24) return `${Math.floor(minutes / 60)}h ${minutes % 60}m`;
|
||||
const days = Math.floor(minutes / (60 * 24));
|
||||
const hours = Math.floor((minutes - days * 60 * 24) / 60);
|
||||
return `${days}d ${hours}h`;
|
||||
}
|
||||
@@ -311,6 +311,34 @@ export const api = {
|
||||
list: () => `${API_BASE_URL}/api/global/distress-reports`,
|
||||
},
|
||||
|
||||
// ── Session Cascade Deletion (Global-Admin Session Cleanup page) ──────────
|
||||
sessionDeletions: {
|
||||
list: (state: "Preparing" | "Queued" | "Running" | "Poisoned", strandedSinceMinutes?: number) =>
|
||||
`${API_BASE_URL}/api/global/session-deletions${qs({
|
||||
state,
|
||||
strandedSinceMinutes: strandedSinceMinutes?.toString(),
|
||||
})}`,
|
||||
// Reads the persisted manifest + progress blob for an in-flight / poisoned cascade.
|
||||
// This is what the Session Cleanup page uses — NOT the dry-run preview below.
|
||||
storedManifest: (
|
||||
sessionId: string,
|
||||
tenantId: string,
|
||||
manifestId: string,
|
||||
mode: "summary" | "full" | "download" = "summary",
|
||||
) =>
|
||||
`${API_BASE_URL}/api/admin/sessions/${encodeURIComponent(sessionId)}/deletion-manifest${qs({
|
||||
tenantId,
|
||||
manifestId,
|
||||
mode,
|
||||
})}`,
|
||||
// Dry-run builder that enumerates current data — only meaningful BEFORE a cascade starts.
|
||||
// The Session Cleanup page should use storedManifest instead.
|
||||
preview: (sessionId: string, tenantId: string, mode: "summary" | "full" | "download" = "summary") =>
|
||||
`${API_BASE_URL}/api/admin/sessions/${encodeURIComponent(sessionId)}/delete/preview${qs({ tenantId, mode })}`,
|
||||
restore: (sessionId: string) =>
|
||||
`${API_BASE_URL}/api/admin/sessions/${encodeURIComponent(sessionId)}/restore`,
|
||||
},
|
||||
|
||||
// ── Hardware Rejection Insights (tenant-scoped, from distress data) ──────
|
||||
distress: {
|
||||
hardwareRejected: () => `${API_BASE_URL}/api/audit/hardware-rejected`,
|
||||
|
||||
@@ -262,6 +262,7 @@ export const EXPANDABLE_NAV_GROUPS: ExpandableNavGroup[] = [
|
||||
id: "ga-ops", label: "Ops", icon: <WrenchIcon />,
|
||||
items: [
|
||||
{ id: "ga-maintenance", label: "Maintenance", href: "/admin/ops" },
|
||||
{ id: "ga-session-cleanup", label: "Session Cleanup", href: "/admin/ops/session-cleanup" },
|
||||
],
|
||||
},
|
||||
{
|
||||
|
||||
Reference in New Issue
Block a user