mirror of
https://github.com/okieselbach/AutopilotMonitor.git
synced 2026-10-24 20:53:06 +00:00
Oliver Kieselbach
d4034842ee
Adds the full backup pipeline that snapshots 15 critical tables (Admin/Tenant config, rules, identities, audit, preview) into a logically-isolated blob container, daily at 04:00 UTC and on operator demand. Restore endpoints follow in PR2/PR3. Architecture: * Daily TimerTrigger + manual POST /api/global/backups/trigger (GA-only, fail-hard producer). HTTP returns 202 + jobId on enqueue, 500 if the queue write fails (and the BackupJobs row is rolled to Failed). * BackgroundService queue worker (not QueueTrigger — matches the Analyze/ IndexReconcile pattern that avoids the Functions-host connection-binding app-setting). Worker holds the maintenance lease for the whole run; service is intentionally lease-unaware so it stays unit-testable. * BlobBackupStore wraps the new 'critical-table-backups' container with a shared maintenance-lease sentinel at '_lock/maintenance.lock' that serialises every backup or restore op across worker/timer/single-row paths. * CriticalTableBackupService streams each table through CryptoStream(SHA256) into per-table NDJSON blobs, then writes manifest.json LAST as the durability anchor. Per-table failures degrade Outcome to Partial without aborting; per-run budget (50 min) → Skipped status. * BackupJobsRepository persists the six-state machine (Queued/Running/Completed/Failed/Skipped/BlockedTerminal) with ETag-CAS updates. Enum→string mapping rejects unknown values on read so refactors surface loudly instead of silently shifting state. * BackupJobWatchdog (30 min timer) heals stuck rows via a probe lease on the shared sentinel: Queued > 60 min → Failed, Running with stale heartbeat AND free lease → Failed (lease-busy means a live worker — leave alone). * OpsEvents: distinct CriticalTableBackupPartial (Warning) vs Failed (Error) so operators can wire softer Telegram rules for partial captures. * MaintenanceService.MonitoredPoisonQueues + OPS_EVENT_TYPES catalog registered (dual-register memory). Test coverage (15 new tests, 1952/1952 backend xUnit green): * BackupManifestJsonTests: enum-as-string (allowIntegerValues=false rejects numeric values), DictionaryKeyPolicy unset (Azure column names preserved case-faithfully), manifest field round-trip. * BackupJobsRepositoryMappingTests: full entity↔domain round-trip, unknown Kind/State/BackupOutcome rejected, legacy BackupOutcome="Failed" rejected. * CriticalTableBackupServiceTests: happy/partial/empty/budget-exceeded paths exercise the per-table loop with in-memory table + blob fakes. Setup: scripts/Backup/setup-backup-container.ps1 — idempotent az-CLI bootstrap for container + lifecycle rule (90 d delete on prefix). Soft-Delete and Versioning are NOT applied automatically (storage-account-wide blast radius — operator must validate per separate doc note). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
93 lines
3.4 KiB
PowerShell
93 lines
3.4 KiB
PowerShell
# setup-backup-container.ps1
|
|
#
|
|
# Idempotent setup for the critical-table backup feature (plan §PR1).
|
|
#
|
|
# Creates the 'critical-table-backups' container in the backend storage account
|
|
# and applies a lifecycle rule that deletes blobs older than 90 days under the
|
|
# 'critical-table-backups/' prefix. Run once per environment, then again any
|
|
# time the storage account is rebuilt.
|
|
#
|
|
# WARNING (plan §Wave15 Azure-Hinweis): Blob Versioning and Blob Soft-Delete
|
|
# are STORAGE-ACCOUNT settings (not container-scoped). Enabling them affects
|
|
# every container in the account (e.g. 'diagnostics', 'offboarding-state').
|
|
# This script does NOT enable them automatically — operator must validate the
|
|
# blast radius separately and run the corresponding `az storage account
|
|
# blob-service-properties update` command if desired.
|
|
|
|
[CmdletBinding()]
|
|
param(
|
|
[Parameter(Mandatory = $true)] [string] $StorageAccountName,
|
|
[Parameter(Mandatory = $true)] [string] $ResourceGroupName,
|
|
[string] $ContainerName = 'critical-table-backups',
|
|
[int] $RetentionDays = 90
|
|
)
|
|
|
|
$ErrorActionPreference = 'Stop'
|
|
|
|
Write-Host "==> Verifying Azure CLI login"
|
|
az account show --only-show-errors > $null
|
|
|
|
Write-Host "==> Ensuring container '$ContainerName' in account '$StorageAccountName'"
|
|
az storage container create `
|
|
--account-name $StorageAccountName `
|
|
--name $ContainerName `
|
|
--auth-mode login `
|
|
--only-show-errors `
|
|
--fail-on-exist false | Out-Null
|
|
|
|
Write-Host "==> Building lifecycle rule (delete > $RetentionDays days, prefix '$ContainerName/')"
|
|
$policyJson = @"
|
|
{
|
|
"rules": [
|
|
{
|
|
"enabled": true,
|
|
"name": "delete-critical-table-backups",
|
|
"type": "Lifecycle",
|
|
"definition": {
|
|
"actions": {
|
|
"baseBlob": {
|
|
"delete": { "daysAfterModificationGreaterThan": $RetentionDays }
|
|
}
|
|
},
|
|
"filters": {
|
|
"blobTypes": [ "blockBlob" ],
|
|
"prefixMatch": [ "$ContainerName/" ]
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
"@
|
|
|
|
$tmpFile = [System.IO.Path]::GetTempFileName()
|
|
try {
|
|
Set-Content -Path $tmpFile -Value $policyJson -Encoding UTF8
|
|
Write-Host "==> Applying lifecycle policy to storage account '$StorageAccountName'"
|
|
Write-Host " NOTE: this REPLACES the account's lifecycle policy. If other rules"
|
|
Write-Host " exist (e.g. for 'deletion-manifests'), merge them in manually."
|
|
az storage account management-policy create `
|
|
--account-name $StorageAccountName `
|
|
--resource-group $ResourceGroupName `
|
|
--policy "@$tmpFile" `
|
|
--only-show-errors | Out-Null
|
|
}
|
|
finally {
|
|
Remove-Item -Path $tmpFile -ErrorAction SilentlyContinue
|
|
}
|
|
|
|
Write-Host ""
|
|
Write-Host "==> Done."
|
|
Write-Host " Container ready: $ContainerName"
|
|
Write-Host " Lifecycle: delete blobs > $RetentionDays days under '$ContainerName/' prefix"
|
|
Write-Host ""
|
|
Write-Host "Operator follow-ups (NOT applied by this script):"
|
|
Write-Host " * If you also want Blob Versioning + Soft-Delete (defence-in-depth against"
|
|
Write-Host " operator mistakes), run separately and confirm the blast radius first:"
|
|
Write-Host " az storage account blob-service-properties update \\"
|
|
Write-Host " --account-name $StorageAccountName \\"
|
|
Write-Host " --resource-group $ResourceGroupName \\"
|
|
Write-Host " --enable-versioning true \\"
|
|
Write-Host " --enable-delete-retention true \\"
|
|
Write-Host " --delete-retention-days 14"
|
|
Write-Host " Both settings affect ALL containers in the account."
|