Three Codex-review findings in PR1, all real:
* HIGH — Maintenance lease expires after 60 s but the per-run budget is 50 min.
Without renewal a parallel worker / timer / restore could enter the same
"exclusive" section after one minute, and the watchdog would see a free
lease while a job is still running. New MaintenanceLeaseHolder spins a
renewal task on a 45 s cadence and cancels the handler's CTS on any
renewal failure; the in-flight backup then bails via OperationCanceledException
and the worker/timer route the job to a clean Failed state without
queue-deleting the message.
* HIGH — Retry-rollback resets State=Queued + BackupId=null but left QueuedAtUtc
on the original wall-clock value. After a retryable service throw the
watchdog's 60 min "Queued without pickup" rule could fire mid retry-cycle
and call the job dead. Fix: stamp QueuedAtUtc=now on the rollback so the
watchdog respects the fresh retry budget.
* MEDIUM — ListBackups returned every top-level prefix including runs that
never reached manifest.json. Calling GetBackupManifest on such an id returned
404 — confusing UI signal for incomplete runs. Fix: ListBackupIdsAsync now
Exists()-checks manifest.json per prefix. New ListAllBackupPrefixesAsync
yields (id, hasManifest) tuples for a future admin UI that wants to
surface incomplete runs as their own status.
Tests: 1952/1952 backend xUnit still green — no regressions, no new test
coverage needed because the hotfix paths are exercised indirectly by the
existing happy/partial/budget tests + the production storage SDK semantics.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the full backup pipeline that snapshots 15 critical tables (Admin/Tenant
config, rules, identities, audit, preview) into a logically-isolated blob
container, daily at 04:00 UTC and on operator demand. Restore endpoints follow
in PR2/PR3.
Architecture:
* Daily TimerTrigger + manual POST /api/global/backups/trigger (GA-only,
fail-hard producer). HTTP returns 202 + jobId on enqueue, 500 if the queue
write fails (and the BackupJobs row is rolled to Failed).
* BackgroundService queue worker (not QueueTrigger — matches the Analyze/
IndexReconcile pattern that avoids the Functions-host connection-binding
app-setting). Worker holds the maintenance lease for the whole run; service
is intentionally lease-unaware so it stays unit-testable.
* BlobBackupStore wraps the new 'critical-table-backups' container with a
shared maintenance-lease sentinel at '_lock/maintenance.lock' that
serialises every backup or restore op across worker/timer/single-row paths.
* CriticalTableBackupService streams each table through CryptoStream(SHA256)
into per-table NDJSON blobs, then writes manifest.json LAST as the
durability anchor. Per-table failures degrade Outcome to Partial without
aborting; per-run budget (50 min) → Skipped status.
* BackupJobsRepository persists the six-state machine
(Queued/Running/Completed/Failed/Skipped/BlockedTerminal) with ETag-CAS
updates. Enum→string mapping rejects unknown values on read so refactors
surface loudly instead of silently shifting state.
* BackupJobWatchdog (30 min timer) heals stuck rows via a probe lease on the
shared sentinel: Queued > 60 min → Failed, Running with stale heartbeat
AND free lease → Failed (lease-busy means a live worker — leave alone).
* OpsEvents: distinct CriticalTableBackupPartial (Warning) vs Failed (Error)
so operators can wire softer Telegram rules for partial captures.
* MaintenanceService.MonitoredPoisonQueues + OPS_EVENT_TYPES catalog
registered (dual-register memory).
Test coverage (15 new tests, 1952/1952 backend xUnit green):
* BackupManifestJsonTests: enum-as-string (allowIntegerValues=false rejects
numeric values), DictionaryKeyPolicy unset (Azure column names preserved
case-faithfully), manifest field round-trip.
* BackupJobsRepositoryMappingTests: full entity↔domain round-trip, unknown
Kind/State/BackupOutcome rejected, legacy BackupOutcome="Failed" rejected.
* CriticalTableBackupServiceTests: happy/partial/empty/budget-exceeded paths
exercise the per-table loop with in-memory table + blob fakes.
Setup: scripts/Backup/setup-backup-container.ps1 — idempotent az-CLI bootstrap
for container + lifecycle rule (90 d delete on prefix). Soft-Delete and
Versioning are NOT applied automatically (storage-account-wide blast radius
— operator must validate per separate doc note).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Oliver Kieselbach