diff --git a/docs/API.md b/docs/API.md index 578c53e86..ad75ee62c 100644 --- a/docs/API.md +++ b/docs/API.md @@ -484,7 +484,8 @@ Detailed storage usage per node and pool. ### Recovery (formerly Backups / Snapshots) Pulse v6 uses the recovery API to provide a platform-agnostic view of backup and snapshot artifacts. -See `docs/architecture/RECOVERY_CONTRACT.md` for the provider-neutral contract (subjects, points, rollups, and filter semantics). +See `docs/architecture/RECOVERY_CONTRACT.md` for the provider-neutral contract +(subjects, points, rollups, posture, and filter semantics). - `GET /api/recovery/points` - Query params: @@ -494,6 +495,11 @@ See `docs/architecture/RECOVERY_CONTRACT.md` for the provider-neutral contract ( - Normalized filters: `q`, `cluster`, `node`, `namespace`, `scope=workload`, `verification` (`verified` | `unverified` | `unknown`) - `GET /api/recovery/rollups` - Query params: `provider`, `kind`, `mode`, `outcome`, `subjectResourceId`, `rollupId`, `from` (RFC3339), `to` (RFC3339), `page`, `limit` +- `GET /api/recovery/postures` + - Returns server-derived per-resource protection posture and provider evidence quality. + - Query params: repeated `resourceId` values (maximum 200), `state` (`protected` | `attention` | `unprotected` | `unknown`), `page`, `limit` (maximum 200) + - Batch clients must make one bounded request per 200 resource ids, never one request per table row. + - Unknown identity, permission, history, or collection completeness remains `unknown`; clients must not infer a healthier state from raw backup or snapshot artifacts. - `GET /api/recovery/series` - Returns per-day counts for the activity chart. - Query params: same filters as `/api/recovery/points` (except paging), plus `tzOffsetMinutes` (integer; UTC offset minutes for day bucketing) diff --git a/docs/RECOVERY.md b/docs/RECOVERY.md index 542411601..2d90ee95e 100644 --- a/docs/RECOVERY.md +++ b/docs/RECOVERY.md @@ -7,7 +7,7 @@ Pulse v6 includes a **provider-neutral recovery view** that aggregates backup, s Recovery is event-first and answers two questions: 1. **"What happened?"** → The **Recovery events** table shows individual recovery points (artifacts) with timestamps, outcomes, and sizes. -2. **"What is covered?"** -> The **Protection coverage** action opens a rollup review for healthy, stale, failed, warning, running, unknown, and never-succeeded items. +2. **"What can I actually recover?"** → **Protection coverage** shows the canonical posture for each resource: protected, attention, unprotected, or unknown. ## Supported Providers @@ -47,6 +47,25 @@ A rollup groups recovery points for a subject to show: - **Latest point** — when was the most recent successful backup/snapshot? - **Health** — are there recent failures or warnings? +### Protection Posture (A Trust Decision) + +A protection posture combines subject-linked recovery points with the latest +provider collection evidence. It deliberately keeps four operator-facing +states: + +- **Protected** — a qualifying current recovery point is linked to the resource + and complete provider evidence does not invalidate the claim. +- **Attention** — evidence exists, but it is stale, failing, incomplete, or + unverified when verification is expected. +- **Unprotected** — complete evidence confirms that no qualifying protection + exists. +- **Unknown** — identity, permissions, provider history, or collection + completeness cannot support a stronger claim. + +A backup or snapshot artifact may still be shown while posture is unknown. +Artifacts answer what Pulse found; posture answers what Pulse can safely claim. +Snapshot presence alone is never presented as independent recovery. + ## Navigating Recovery ### Recovery Events @@ -64,15 +83,17 @@ Shows individual recovery points. Key columns: ### Protection Coverage -Available from the Recovery header action and compatibility routes when posture or freshness needs review. It shows one row per protected item (or per item + method when multiple backup methods exist). Key columns: +The Proxmox **Backups → Coverage** view shows one row per workload. The default +table stays compact; expanding a row reveals the plain-language posture reason, +provider evidence quality, and individual restore artifacts. | Column | Description | |---|---| | Item | The protected resource (VM name, dataset path, etc.) | | Item Type | Canonical resource category | -| Platform | Which provider created the latest point (PVE, PBS, TrueNAS, Kubernetes) | -| Latest Point | Most recent successful recovery point timestamp | -| Status | Healthy, stale, failed, warning, running, unknown, or never succeeded | +| Posture | Protected, attention, unprotected, or unknown | +| Restore | Most recent successful recovery point timestamp | +| Provider columns | Latest PBS, PVE, or guest-snapshot artifact where available | ### Filtering @@ -89,6 +110,7 @@ Both workspaces support: |---|---|---| | `GET` | `/api/recovery/points` | List individual recovery points | | `GET` | `/api/recovery/rollups` | List subject rollups (protection coverage) | +| `GET` | `/api/recovery/postures` | List canonical per-resource protection postures | | `GET` | `/api/recovery/series` | Time-series data for recovery charts | | `GET` | `/api/recovery/facets` | Available filter facets (providers, kinds, outcomes) | @@ -106,6 +128,13 @@ All recovery endpoints support: | `subject` | Filter by subject reference | | `limit` | Max results (default: 500) | +`/api/recovery/postures` has a deliberately bounded table contract. Supply one +or more repeated `resourceId` parameters for a resource or batch lookup (at +most 200), or omit them for a paged list. It also accepts `state`, `page`, and +`limit`; `state=attention` returns the actionable attention list. The response +includes the posture policy and provider evidence states so clients do not +re-derive trust from raw artifacts. + ## Troubleshooting ### No recovery data showing @@ -129,6 +158,14 @@ Click the row to expand the details drawer, which shows the provider-specific er - **TrueNAS**: Replication target unreachable, dataset locked, insufficient space - **PVE**: Backup storage full, vzdump process error +### Current backups show an unknown posture + +Expand the workload row and inspect the limiting evidence. Pulse uses unknown +when the current poll cannot prove provider-history completeness, permission +scope, or subject identity. Fix the reported collection or access gap and wait +for the next provider poll; Pulse does not promote retained backup artifacts to +protected while that uncertainty remains. + ## See Also - [PBS Integration](PBS.md) — Proxmox Backup Server monitoring diff --git a/docs/release-control/v6/internal/records/operational-trust-protection-posture-2026-07-19.md b/docs/release-control/v6/internal/records/operational-trust-protection-posture-2026-07-19.md new file mode 100644 index 000000000..872592497 --- /dev/null +++ b/docs/release-control/v6/internal/records/operational-trust-protection-posture-2026-07-19.md @@ -0,0 +1,132 @@ +# Operational Trust Protection Posture Record + +Date: 2026-07-19 + +## Scope + +This record closes the Phase 2 protection-posture slice in +`OPERATIONAL_TRUST_IMPLEMENTATION_SPEC.md`. It does not claim completion of +the Patrol attention workbench, availability attachment, governed actions, or +the full Operational Trust specification. + +## User job and evidence + +The least-expert plausible user's job is: “Tell me which workloads I can +actually recover, which need attention, and why.” + +Relevant issue history shows why artifact presence cannot be the answer: + +1. [#1541](https://github.com/rcourtman/Pulse/issues/1541) reports current PBS + backups displayed with an incorrect unverified conclusion. +2. [#81](https://github.com/rcourtman/Pulse/issues/81) records a token that can + list datastores but cannot enumerate backup snapshots, proving that partial + permission must remain visible rather than becoming a healthy empty result. +3. [#1389](https://github.com/rcourtman/Pulse/issues/1389) records VM and + container backups mis-correlated by numeric id, proving that provider-scoped + subject identity is part of protection evidence. +4. [#1592](https://github.com/rcourtman/Pulse/issues/1592) and + [#1437](https://github.com/rcourtman/Pulse/issues/1437) report missing PVE + backups and snapshots, proving that one provider's visible artifacts do not + establish complete cross-provider coverage. +5. [#1056](https://github.com/rcourtman/Pulse/issues/1056) and + [#1162](https://github.com/rcourtman/Pulse/issues/1162) show the operator + cost of treating retained orphan artifacts as current workload protection. + +## Canonical runtime result + +1. `internal/recovery/model/posture.go` defines the provider-neutral posture, + provider-state, policy, query, and provider-observation contracts. The only + customer-facing states are protected, attention, unprotected, and unknown. +2. `internal/recovery/posture.go` derives posture by canonical resource id and + provider scope. Protected requires a current subject-linked backup plus + complete sufficient collection evidence. Attention covers actionable stale, + failed, partial, or expected-but-unverified evidence. Unprotected requires + complete evidence of no qualifying backup. Unknown is mandatory for + identity, permission, history, or collection uncertainty. +3. Backup and snapshot semantics remain distinct. A snapshot alone never + proves independent recovery, provider job success without a subject-linked + point never becomes subject protection, and a limitation from an unrelated + provider cannot erase a confirmed complete PBS recovery. +4. Recovery-point persistence adds provider scope and typed evidence + additively. Existing databases migrate in place, and legacy points receive + explicit unknown-quality evidence rather than invented certainty. +5. Provider observations and materialized postures have indexed, bounded, + retained storage. Requested postures are re-evaluated from current points + and observations at read time so a reassuring stored row cannot age into a + lie. Point, observation, and posture retention share the 90-day boundary. +6. PBS is the first explicit provider adapter. Every poll records complete, + partial, unavailable, or denied collection evidence separately from its + subject-linked recovery points. Collection evidence is persisted before the + point batch and reconciliation, so a large point write, timeout, cached + artifact path, or failed enumeration cannot preserve a healthier claim. +7. `GET /api/recovery/postures` supplies one-resource, bounded batch, paged + list, and `state=attention` reads under `monitoring:read`. Resource batches + are capped at 200, return unknown for requested ids without sufficient + evidence, and include policy plus provider evidence for drill-down. +8. `useProtectionPostures` sorts and deduplicates canonical resource ids, + fetches one batch per 200 resources, and exposes a keyed read model. Proxmox + passes exact unified-resource ids and does not parse table keys or derive + posture from raw PBS, PVE, or guest-snapshot artifacts. +9. The Proxmox Backups coverage table keeps the default monitor compact. Each + workload has one canonical status and latest restore time; the row + drill-down contains the plain-language reason, human-readable provider + evidence quality, and bounded restore artifacts. +10. `docs/API.md` and `docs/RECOVERY.md` document the bounded API, the + artifact-versus-posture distinction, the four states, and unknown-state + troubleshooting. + +## Failure, migration, retention, and performance proof + +The Phase 2 proof covers: + +1. the provider-aware derivation truth table, including stale, failed, + unverified, partial, denied, missing-identity, snapshot-only, complete-empty, + and mixed-provider cases +2. schema migration and legacy evidence backfill +3. read-time re-evaluation over a deliberately corrupted reassuring + materialization +4. indexed query plans and bounded 200-resource API batches +5. recovery-point, provider-observation, and materialized-posture retention +6. complete, partial, unavailable, and denied PBS collection mappings +7. provider-observation persistence before a deliberately failed point write +8. API authorization, validation, pagination, attention filtering, missing + requested resources, and compatibility serialization +9. frontend batch normalization, one-call table integration, state filtering, + evidence drill-down, type-check, and Proxmox regression tests + +## Live product and user-lens proof + +The changed surface was exercised against the managed live runtime at +`/proxmox/backups`, through the deepest row expansion, at desktop and 390px +mobile widths. + +1. Distance to goal is the existing Proxmox Backups route, the Coverage + selector, then one workload expansion for the reason and evidence. +2. The live pass found an inherited presentation defect: the explanation and + provider evidence were initially repeated inside every PBS table cell. That + detail was demoted to the expansion. The resulting workload row is 33px + high and the default table remains scannable. +3. Default-visible elements all support the job: the four-state summary + identifies fleet posture, filters narrow the queue, the table answers per + workload, and the expansion explains the claim. Raw artifacts are retained + only as forensic restore evidence. +4. Provider codes were replaced with product vocabulary in the expansion + (`Proxmox Backup Server`, `Proxmox VE`), and evidence-quality values are + presented as readable labels. +5. The live dataset honestly remained unknown where no complete provider + collection observation had yet been persisted, despite current verified PBS + artifacts. This is the required fail-closed behavior, not a healthy guess. +6. The phone-sized page has no body-level horizontal overflow; wide evidence + tables scroll inside their bounded table wrappers. +7. The durable Playwright guard proves one bounded posture request, compact + default presentation, evidence-on-expansion, and mobile containment. + +The Phase 2 surface is classified as `product`. + +## Boundary carried forward + +Phase 2 supplies canonical protection truth but does not invent a work queue. +Phase 3 must project lifecycle records into typed attention items and consume +this posture as one input. Patrol must not parse recovery metadata, raw +artifacts, provider job payloads, or table presentation state to create its own +protection verdict. diff --git a/docs/release-control/v6/internal/status.json b/docs/release-control/v6/internal/status.json index f6a49c69c..d25624f18 100644 --- a/docs/release-control/v6/internal/status.json +++ b/docs/release-control/v6/internal/status.json @@ -8727,6 +8727,16 @@ "path": "docs/release-control/v6/internal/records/operational-trust-alert-state-boundary-2026-07-18.md", "kind": "file" }, + { + "repo": "pulse", + "path": "docs/release-control/v6/internal/records/operational-trust-lifecycle-evidence-notification-linkage-2026-07-19.md", + "kind": "file" + }, + { + "repo": "pulse", + "path": "docs/release-control/v6/internal/records/operational-trust-protection-posture-2026-07-19.md", + "kind": "file" + }, { "repo": "pulse", "path": "docs/release-control/v6/internal/records/protection-posture-attention-queue-strong-version-2026-06-25.md", @@ -8776,6 +8786,21 @@ "repo": "pulse", "path": "internal/monitoring/proxmox_action_observer.go", "kind": "file" + }, + { + "repo": "pulse", + "path": "internal/recovery/posture.go", + "kind": "file" + }, + { + "repo": "pulse", + "path": "internal/recovery/store/store_posture.go", + "kind": "file" + }, + { + "repo": "pulse", + "path": "tests/integration/tests/90-operational-trust-protection-posture.spec.ts", + "kind": "file" } ] }, diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md index 1e2b935d7..46fd1b9e9 100644 --- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md +++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md @@ -4619,3 +4619,9 @@ header alone is not identity. This lets a Basic-authenticated qualification operator approve or reject an agent action while preserving the agent token's separate reporting and command authority; invalid credentials must never reach the action handler or acquire an operator principal. + +The adjacent recovery handlers under `internal/api/` do not widen this agent +lifecycle boundary. Protection posture is a read-only `monitoring:read` +projection over recovery points and provider collection evidence. It does not +register agents, issue or rotate credentials, interpret observer responses, +or grant backup, restore, command, or remote-configuration authority. diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.md index 2152e2ceb..2d1685e99 100644 --- a/docs/release-control/v6/internal/subsystems/api-contracts.md +++ b/docs/release-control/v6/internal/subsystems/api-contracts.md @@ -7764,3 +7764,20 @@ invalid Basic credentials attach no principal and fail before the handler. This is required for qualification clients and local operators to read, approve, reject, and execute governed actions without weakening the same capability checks for sessions, proxy users, or API tokens. + +### Protection posture transport + +`GET /api/recovery/postures` is the authenticated `monitoring:read` transport +for the canonical storage/recovery posture model. It accepts one or repeated +canonical `resourceId` values, rejects more than 200 unique IDs, and supports a +bounded state-filtered paginated list. Responses carry typed posture data, +provider-specific history and permission limitations, evidence references, +the server evaluation policy, evaluation timestamps, and stable pagination +metadata. A requested resource with insufficient evidence returns an explicit +`unknown` posture rather than disappearing or inheriting a frontend guess. + +Existing recovery point and rollup routes remain additive compatibility +boundaries. Recovery points now expose provider scope and typed evidence, and +rollups expose their already-owned verification intent and last-verification +time. Supported legacy provider, subject, and display aliases remain readable; +they do not create a parallel posture authority. diff --git a/docs/release-control/v6/internal/subsystems/frontend-primitives.md b/docs/release-control/v6/internal/subsystems/frontend-primitives.md index 2ae634d46..8131b28b3 100644 --- a/docs/release-control/v6/internal/subsystems/frontend-primitives.md +++ b/docs/release-control/v6/internal/subsystems/frontend-primitives.md @@ -5048,3 +5048,19 @@ the matching Open or History subtab from server-authored lifecycle state, and removes the query when the dialog closes. Feature pages may summarize action context, but they must not recreate approve, reject, run, progress, or outcome controls outside the shared Actions review. + +### Protection posture presentation boundary + +Platform coverage tables render the storage/recovery-owned four-state +`ProtectionPosture` contract without inventing age, failure, verification, or +coverage states from raw browser payloads. Protected uses the shared success +tone, attention uses warning, unprotected uses danger, and unknown uses muted +presentation. The compact row remains actionable: plain-language rationale and +provider history/permission limitations live one disclosure deeper beside +bounded restore evidence. A posture fetch failure must keep the evidence +inspectable and show unknown, never a locally inferred healthy fallback. + +Protection table controls continue to compose shared filter, table, status, +counter, loading, error, disclosure, and inline-detail primitives. The bounded +batch hook retains fulfilled values during refresh and issues at most one +request per 200 resource rows, rather than placing a request under each row. diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md index ebf571977..e046447fe 100644 --- a/docs/release-control/v6/internal/subsystems/monitoring.md +++ b/docs/release-control/v6/internal/subsystems/monitoring.md @@ -155,6 +155,7 @@ resource health. 54. `internal/monitoring/monitor_backups.go` 55. `internal/monitoring/resource_stale_thresholds.go` 56. `internal/monitoring/recovery_ingest.go` +56a. `internal/monitoring/pbs_protection_observation.go` 57. `internal/monitoring/multi_tenant_monitor.go` 58. `internal/monitoring/proxmox_action_observer.go` @@ -1733,3 +1734,24 @@ delivery status and timestamp alongside guest power state. Downstream Patrol transition detection consumes that status instead of inventing a fixed stale window: a stopped guest can have fresh inventory, while a stale source cannot authoritatively prove either a stopped transition or recovery. + +### PBS protection evidence collection + +Direct PBS backup enumeration emits two separate storage/recovery inputs: +subject-linked recovery points and one typed provider observation for the +polled PBS instance. A complete poll records complete history with sufficient +permissions; a partially successful poll records partial history and the +appropriate partial or unknown permission posture; total transient failure +records unavailable history; total terminal authorization failure records +denied access. Retained backup points survive failed enumeration, but the new +provider observation immediately prevents those cached points from being +presented as current protection truth. + +PBS mapping attaches provider scope and a typed evidence envelope to every +successfully enumerated recovery point. Identity correlation is confirmed only +for direct canonical identity and inferred only for an auditable unique +provider-scoped guest match. Monitoring persists the collection observation +before point reconciliation so completeness and permission failure cannot be +lost behind a successful cached-artifact path. Shared protection semantics stay +in `internal/recovery/`; PBS monitoring owns only this explicit evidence-quality +adapter. diff --git a/docs/release-control/v6/internal/subsystems/registry.json b/docs/release-control/v6/internal/subsystems/registry.json index 47feb547f..60b4b37d7 100644 --- a/docs/release-control/v6/internal/subsystems/registry.json +++ b/docs/release-control/v6/internal/subsystems/registry.json @@ -418,6 +418,14 @@ ], "rationale": "Proxmox backup server table rows are both a storage/recovery backup-health surface and a unified-resource platform-table consumer boundary" }, + { + "path": "frontend-modern/src/features/proxmox/ProxmoxCoverageTable.tsx", + "subsystems": [ + "storage-recovery", + "unified-resources" + ], + "rationale": "Proxmox workload coverage rows are both a storage/recovery protection-posture surface and a unified-resource identity consumer boundary" + }, { "path": "frontend-modern/src/features/proxmox/ProxmoxRecoverableTable.tsx", "subsystems": [ @@ -2538,6 +2546,23 @@ "internal/api/diagnostics_memory_test.go" ] }, + { + "id": "protection-posture-api-contract", + "label": "bounded protection posture API proof", + "match_prefixes": [], + "match_files": [ + "internal/api/recovery_handlers.go", + "internal/api/router_routes_monitoring.go" + ], + "allow_same_subsystem_tests": false, + "test_prefixes": [], + "exact_files": [ + "frontend-modern/src/hooks/__tests__/useProtectionPostures.test.ts", + "frontend-modern/src/types/recovery.ts", + "internal/api/recovery_handlers_test.go", + "internal/api/route_inventory_test.go" + ] + }, { "id": "backend-payload-contracts", "label": "backend API payload proof", @@ -5151,6 +5176,24 @@ "tests/integration/tests/43-platform-mock-runtime.spec.ts" ] }, + { + "id": "pbs-protection-evidence-runtime", + "label": "PBS protection evidence collection proof", + "match_prefixes": [], + "match_files": [ + "internal/monitoring/monitor_backups.go", + "internal/monitoring/pbs_protection_observation.go", + "internal/monitoring/recovery_ingest.go" + ], + "allow_same_subsystem_tests": false, + "test_prefixes": [], + "exact_files": [ + "internal/monitoring/pbs_protection_observation_test.go", + "internal/monitoring/recovery_ingest_test.go", + "internal/recovery/mapper/proxmox/mapper_test.go", + "internal/recovery/posture_test.go" + ] + }, { "id": "monitoring-runtime", "label": "monitoring runtime proof", @@ -6267,8 +6310,10 @@ "frontend-modern/src/features/proxmox/proxmoxBackupSourcePresentation.ts", "frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx", "frontend-modern/src/features/proxmox/proxmoxBackupsTableShared.tsx", + "frontend-modern/src/features/proxmox/ProxmoxCoverageTable.tsx", "frontend-modern/src/features/proxmox/ProxmoxPageSurface.tsx", "frontend-modern/src/features/proxmox/ProxmoxRecoverableTable.tsx", + "frontend-modern/src/hooks/useProtectionPostures.ts", "frontend-modern/src/hooks/useRecoveryPoints.ts", "frontend-modern/src/types/recovery.ts", "frontend-modern/src/utils/recoveryArtifactModePresentation.ts", @@ -6302,7 +6347,9 @@ "internal/recovery/idgen/idgen_test.go", "internal/recovery/manager/manager_test.go", "internal/recovery/mapper/proxmox/mapper_test.go", + "internal/recovery/posture_test.go", "internal/recovery/recovery_test.go", + "internal/recovery/store/store_posture_test.go", "internal/recovery/store/store_queryplan_test.go", "internal/recovery/store/store_rollups_test.go", "internal/recovery/store/store_series_test.go", @@ -6314,6 +6361,7 @@ "label": "recovery product surface proof", "match_prefixes": [], "match_files": [ + "frontend-modern/src/hooks/useProtectionPostures.ts", "frontend-modern/src/hooks/useRecoveryPoints.ts", "frontend-modern/src/types/recovery.ts", "frontend-modern/src/utils/recoveryArtifactModePresentation.ts", @@ -6326,8 +6374,11 @@ "allow_same_subsystem_tests": false, "test_prefixes": [], "exact_files": [ + "frontend-modern/src/api/__tests__/recoveryTransport.test.ts", + "frontend-modern/src/hooks/__tests__/useProtectionPostures.test.ts", "frontend-modern/src/utils/__tests__/frontendResourceTypeBoundaries.test.ts", - "tests/integration/tests/17-proxmox-backups-layout.spec.ts" + "tests/integration/tests/17-proxmox-backups-layout.spec.ts", + "tests/integration/tests/90-operational-trust-protection-posture.spec.ts" ] }, { @@ -6370,16 +6421,20 @@ "frontend-modern/src/features/proxmox/proxmoxBackupSourcePresentation.ts", "frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx", "frontend-modern/src/features/proxmox/proxmoxBackupsTableShared.tsx", + "frontend-modern/src/features/proxmox/ProxmoxCoverageTable.tsx", "frontend-modern/src/features/proxmox/ProxmoxPageSurface.tsx", "frontend-modern/src/features/proxmox/ProxmoxRecoverableTable.tsx" ], "allow_same_subsystem_tests": false, "test_prefixes": [], "exact_files": [ + "frontend-modern/src/features/proxmox/__tests__/proxmoxBackupRecoveryModel.branchcov.test.ts", "frontend-modern/src/features/proxmox/__tests__/proxmoxBackupRecoveryModel.test.ts", "frontend-modern/src/features/proxmox/__tests__/proxmoxBackupSourcePresentation.test.ts", "frontend-modern/src/features/proxmox/__tests__/ProxmoxBackupsTable.test.tsx", - "frontend-modern/src/features/proxmox/__tests__/proxmoxBackupsTableShared.test.tsx" + "frontend-modern/src/features/proxmox/__tests__/proxmoxBackupsTableShared.test.tsx", + "frontend-modern/src/features/proxmox/__tests__/ProxmoxCoverageTable.test.tsx", + "tests/integration/tests/90-operational-trust-protection-posture.spec.ts" ] }, { diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md index 530daa342..c5cfefced 100644 --- a/docs/release-control/v6/internal/subsystems/storage-recovery.md +++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md @@ -53,6 +53,11 @@ state. 24. `frontend-modern/src/features/proxmox/ProxmoxRecoverableTable.tsx` 25. `frontend-modern/src/features/proxmox/proxmoxBackupsTableShared.tsx` 26. `frontend-modern/src/features/proxmox/proxmoxBackupSourcePresentation.ts` +27. `internal/recovery/model/posture.go` +28. `internal/recovery/posture.go` +29. `internal/recovery/store/store_posture.go` +30. `frontend-modern/src/hooks/useProtectionPostures.ts` +31. `frontend-modern/src/features/proxmox/ProxmoxCoverageTable.tsx` ## Shared Boundaries @@ -90,9 +95,10 @@ knowledge persistence. Any future storage action must cross the explicit a recovery-provider read helper or compatibility alias. 1. `frontend-modern/src/features/proxmox/ProxmoxBackupServersTable.tsx` shared with `unified-resources`: Proxmox backup server table rows are both a storage/recovery backup-health surface and a unified-resource platform-table consumer boundary. -2. `frontend-modern/src/features/proxmox/ProxmoxRecoverableTable.tsx` shared with `unified-resources`: Proxmox recoverable workload table rows are both a storage/recovery coverage surface and a unified-resource platform-table consumer boundary. -3. `internal/api/setup_script_render.go` shared with `agent-lifecycle`, `api-contracts`: the generated Proxmox setup-script is a shared boundary across agent lifecycle (forced-command keys, install/uninstall edits), API contracts (rendered token shape and encoded rerun URL), and storage/recovery (backup visibility grants, Pulse-managed temperature SSH keys, and SMART disk-temperature collection). -4. `internal/proxmoxidentity/backup_identity.go` shared with `alerts`, `monitoring`: Proxmox PBS backup subject identity is a shared runtime boundary for monitoring backup freshness, backup-age alert attribution, and recovery-point guest mapping. +2. `frontend-modern/src/features/proxmox/ProxmoxCoverageTable.tsx` shared with `unified-resources`: Proxmox workload coverage rows are both a storage/recovery protection-posture surface and a unified-resource identity consumer boundary. +3. `frontend-modern/src/features/proxmox/ProxmoxRecoverableTable.tsx` shared with `unified-resources`: Proxmox recoverable workload table rows are both a storage/recovery coverage surface and a unified-resource platform-table consumer boundary. +4. `internal/api/setup_script_render.go` shared with `agent-lifecycle`, `api-contracts`: the generated Proxmox setup-script is a shared boundary across agent lifecycle (forced-command keys, install/uninstall edits), API contracts (rendered token shape and encoded rerun URL), and storage/recovery (backup visibility grants, Pulse-managed temperature SSH keys, and SMART disk-temperature collection). +5. `internal/proxmoxidentity/backup_identity.go` shared with `alerts`, `monitoring`: Proxmox PBS backup subject identity is a shared runtime boundary for monitoring backup freshness, backup-age alert attribution, and recovery-point guest mapping. Storage resource projections may show alert indicators from the canonical active-alert read model only while detector evaluation is enabled. They must @@ -2155,6 +2161,33 @@ while storage detail drawers and filter controls must route summary series IDs, source tones, and disk metrics through the shared storage helpers instead of reconstructing them from local table state. +### Canonical protection posture + +`ProtectionPosture` is the storage/recovery-owned subject read model over +recovery points and typed provider collection evidence. Its only public states +are `protected`, `attention`, `unprotected`, and `unknown`. A protected claim +requires a current subject-linked backup supported by complete, sufficiently +authorized provider history, plus verification when the provider declares it +expected. Stale, failed, partial, or unverified evidence is attention; +unprotected requires complete evidence of no qualifying backup; unavailable +identity, history, collection, or permission evidence fails to unknown. +Snapshots remain distinct and never prove independent recovery by themselves. + +The per-tenant recovery store persists provider observations and a materialized +posture index, refreshes it after writes, reconciliation, migration, identity +backfill, purge, and retention, and re-evaluates requested resource batches at +read time so freshness cannot age into a reassuring stale snapshot. Legacy +recovery rows receive additive provider scope and typed evidence during +migration. Provider observations and recovery points share bounded retention. + +`GET /api/recovery/postures` is the bounded read boundary for one resource, +resource-table batches, state-filtered attention lists, policy, evaluation +time, provider limitations, and evidence references. The Proxmox coverage +surface requests canonical resource IDs in batches of at most 200 and never +derives posture from raw PVE/PBS artifacts. Raw backup files, snapshots, tasks, +and provider state remain expandable recovery evidence. Missing or failed +posture reads render unknown rather than falling back to browser inference. + The adjacent shared authentication boundary must remain live when storage or recovery browser reads authenticated with local credentials overlap agent API-token reports. Local credential snapshots inside `checkAuth` use the diff --git a/docs/release-control/v6/internal/subsystems/unified-resources.md b/docs/release-control/v6/internal/subsystems/unified-resources.md index fb24a9a49..a22572f16 100644 --- a/docs/release-control/v6/internal/subsystems/unified-resources.md +++ b/docs/release-control/v6/internal/subsystems/unified-resources.md @@ -574,13 +574,14 @@ container inventory table. 10. `frontend-modern/src/components/Infrastructure/useUnifiedResourceTableState.ts` shared with `performance-and-scalability`: unified resource table state, grouping, and windowing are both a canonical unified-resource consumer surface and a fleet-scale performance hot-path boundary. 11. `frontend-modern/src/components/Infrastructure/useUnifiedResourceTableViewportSync.ts` shared with `performance-and-scalability`: unified resource table viewport sync and selected-row reveal are both a canonical unified-resource consumer surface and a fleet-scale performance hot-path boundary. 12. `frontend-modern/src/features/proxmox/ProxmoxBackupServersTable.tsx` shared with `storage-recovery`: Proxmox backup server table rows are both a storage/recovery backup-health surface and a unified-resource platform-table consumer boundary. -13. `frontend-modern/src/features/proxmox/ProxmoxRecoverableTable.tsx` shared with `storage-recovery`: Proxmox recoverable workload table rows are both a storage/recovery coverage surface and a unified-resource platform-table consumer boundary. -14. `frontend-modern/src/routing/routePreload.ts` shared with `frontend-primitives`, `performance-and-scalability`: the app-shell route preload registry is a canonical frontend shell boundary, an authenticated hot-path performance boundary, and the entry point for the unified-resource Actions workspace. -15. `frontend-modern/src/utils/platformSupportManifest.generated.ts` shared with `frontend-primitives`: the generated platform support projection is both a canonical unified-resource platform union boundary and a shared frontend source/platform vocabulary boundary. +13. `frontend-modern/src/features/proxmox/ProxmoxCoverageTable.tsx` shared with `storage-recovery`: Proxmox workload coverage rows are both a storage/recovery protection-posture surface and a unified-resource identity consumer boundary. +14. `frontend-modern/src/features/proxmox/ProxmoxRecoverableTable.tsx` shared with `storage-recovery`: Proxmox recoverable workload table rows are both a storage/recovery coverage surface and a unified-resource platform-table consumer boundary. +15. `frontend-modern/src/routing/routePreload.ts` shared with `frontend-primitives`, `performance-and-scalability`: the app-shell route preload registry is a canonical frontend shell boundary, an authenticated hot-path performance boundary, and the entry point for the unified-resource Actions workspace. +16. `frontend-modern/src/utils/platformSupportManifest.generated.ts` shared with `frontend-primitives`: the generated platform support projection is both a canonical unified-resource platform union boundary and a shared frontend source/platform vocabulary boundary. It must carry the manifest `surface_kind` distinction so `docker` remains machine-readable as a `runtime-lens` while owning infrastructure sources remain `platform` entries. -16. `frontend-modern/src/utils/sourcePlatforms.ts` shared with `frontend-primitives`: the source platform normalizer is both a canonical unified-resource source adapter boundary and a shared frontend source/platform vocabulary boundary. +17. `frontend-modern/src/utils/sourcePlatforms.ts` shared with `frontend-primitives`: the source platform normalizer is both a canonical unified-resource source adapter boundary and a shared frontend source/platform vocabulary boundary. That shared vocabulary boundary owns the generic `docker` platform label: selectors, badges, and filter options render it as "Docker / Podman" so v5 Docker users can still find the runtime surface while Podman-backed @@ -602,7 +603,7 @@ container inventory table. display/source family; `platformScopes` is the overlap set used when a runtime workload belongs to both Docker and an owning infrastructure platform. -17. `internal/api/resources.go` shared with `api-contracts`: the unified resource endpoint is both a backend payload contract surface and a unified-resource runtime boundary. +18. `internal/api/resources.go` shared with `api-contracts`: the unified resource endpoint is both a backend payload contract surface and a unified-resource runtime boundary. `/api/resources` type filters must accept URL-encoded comma-separated lists from browser query builders exactly like literal comma separators, so Docker / Podman runtime pages do not lose `docker-host` inventory while requesting @@ -1789,6 +1790,17 @@ through the canonical resource model, but unified-resource consumers must not reintroduce removed workload aliases or feature-local resource-type shims just to satisfy one table, drawer, or badge surface. +### Protection posture identity consumer + +`ProxmoxCoverageTable` remains a unified-resource identity consumer while +storage/recovery owns protection truth. Live VM/LXC rows carry the exact +canonical `Resource.id` into one bounded posture batch; the table must not parse +its presentation key, VMID, name, node, or instance to mint a replacement +resource identity. Orphaned backup artifacts have no canonical live resource +ID and therefore render unknown posture while retaining their forensic backup +detail. Unified resources own row identity only; they do not derive backup +freshness, provider completeness, verification, or protection state. + ### APT Product Trust browser projection The durable Actions inbox and resource action history consume the canonical APT diff --git a/frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx b/frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx index 37405957a..4cfa76456 100644 --- a/frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx +++ b/frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx @@ -6,6 +6,7 @@ import { useSearchParams } from '@solidjs/router'; import { FilterBar, type FilterDef, type FilterSelectOption } from '@/components/shared/FilterBar'; import { FilterSegmentedControl } from '@/components/shared/FilterToolbar'; import { useBreakpoint } from '@/hooks/useBreakpoint'; +import { useProtectionPostures } from '@/hooks/useProtectionPostures'; import { apiFetch } from '@/utils/apiClient'; import { PlatformErrorState, @@ -37,7 +38,6 @@ import { import { buildProxmoxBackupRecoveryModel, coverageRowMatchesSearch, - isCoverageAttention, recoverableArtifactMatchesSearch, type RecoverableArtifact, } from './proxmoxBackupRecoveryModel'; @@ -99,6 +99,9 @@ export const ProxmoxBackupsTable: Component<{ const [backups, { refetch }] = createResource(fetchPVEBackups); const [pbsBackups] = createResource(fetchPBSBackups); const { isMobile } = useBreakpoint(); + const protectionPostures = useProtectionPostures(() => + (props.workloads ?? []).map((workload) => workload.id), + ); // Structured scope filters (node, type) live in the URL so the view is // shareable, survives reload, and can be captured by FilterBar saved views. @@ -201,6 +204,7 @@ export const ProxmoxBackupsTable: Component<{ snapshots: snapshots(), tasks: tasks(), nowMs: nowMs(), + protectionPostures: protectionPostures.postureByResourceID(), }), ); @@ -245,9 +249,7 @@ export const ProxmoxBackupsTable: Component<{ const list = recoveryModel().coverageRows.filter((row) => { if (!nodeMatches(row.workload.node)) return false; if (!typeMatches(row.workload.type)) return false; - if (filter === 'attention' && !isCoverageAttention(row.posture)) return false; - if (filter === 'current' && row.posture !== 'current') return false; - if (filter === 'uncovered' && row.posture !== 'uncovered') return false; + if (filter !== 'all' && row.posture !== filter) return false; return coverageRowMatchesSearch(row, term); }); const sortKey = coverageSortKey(); @@ -296,13 +298,18 @@ export const ProxmoxBackupsTable: Component<{ const liveHealthSummary = createMemo(() => { const live = recoveryModel().coverageRows.filter((row) => !row.isOrphaned); return { - current: live.filter((row) => row.posture === 'current').length, - attention: live.filter((row) => isCoverageAttention(row.posture)).length, - uncovered: live.filter((row) => row.posture === 'uncovered').length, + protected: live.filter((row) => row.posture === 'protected').length, + attention: live.filter((row) => row.posture === 'attention').length, + unprotected: live.filter((row) => row.posture === 'unprotected').length, + unknown: live.filter((row) => row.posture === 'unknown').length, }; }); const view = (): BackupView => - selectedView() ?? (liveHealthSummary().attention > 0 ? 'coverage' : 'date'); + selectedView() ?? + (liveHealthSummary().attention + liveHealthSummary().unprotected + liveHealthSummary().unknown > + 0 + ? 'coverage' + : 'date'); const setView = (next: BackupView): void => { setSelectedView(next); }; @@ -462,6 +469,15 @@ export const ProxmoxBackupsTable: Component<{ } >
+ +
+ Protection posture is unavailable. Pulse is showing restore evidence without guessing + whether workloads are protected. +
+ 0}> @@ -472,8 +488,8 @@ export const ProxmoxBackupsTable: Component<{ title="Backup health" tail={ - {liveTotalCount()} targets ·{' '} - {recoveryModel().coverageSummary.recoverableArtifacts} restore points + {liveTotalCount()} targets · {recoveryModel().coverageSummary.recoverableArtifacts}{' '} + restore points 0}> {' · '} {recoveryModel().coverageSummary.withPBS} with{' '} @@ -487,9 +503,9 @@ export const ProxmoxBackupsTable: Component<{ } segments={[ { - key: 'current', - value: liveHealthSummary().current, - label: 'current', + key: 'protected', + value: liveHealthSummary().protected, + label: 'protected', toneClass: 'bg-emerald-500', }, { @@ -500,11 +516,18 @@ export const ProxmoxBackupsTable: Component<{ muted: liveHealthSummary().attention === 0, }, { - key: 'uncovered', - value: liveHealthSummary().uncovered, - label: 'uncovered', + key: 'unprotected', + value: liveHealthSummary().unprotected, + label: 'unprotected', toneClass: 'bg-red-500', - muted: liveHealthSummary().uncovered === 0, + muted: liveHealthSummary().unprotected === 0, + }, + { + key: 'unknown', + value: liveHealthSummary().unknown, + label: 'unknown', + toneClass: 'bg-slate-400', + muted: liveHealthSummary().unknown === 0, }, ]} /> diff --git a/frontend-modern/src/features/proxmox/ProxmoxCoverageTable.tsx b/frontend-modern/src/features/proxmox/ProxmoxCoverageTable.tsx index bf8a7543d..61e4da6bf 100644 --- a/frontend-modern/src/features/proxmox/ProxmoxCoverageTable.tsx +++ b/frontend-modern/src/features/proxmox/ProxmoxCoverageTable.tsx @@ -32,9 +32,10 @@ import { const coveragePostureVariant = ( posture: WorkloadCoverageRow['posture'], ): StatusIndicatorVariant => { - if (posture === 'current') return 'success'; - if (posture === 'uncovered' || posture === 'failed' || posture === 'stale') return 'danger'; - return 'warning'; + if (posture === 'protected') return 'success'; + if (posture === 'unprotected') return 'danger'; + if (posture === 'attention') return 'warning'; + return 'muted'; }; // Colour marks the exception, not the baseline: healthy rows keep neutral text @@ -54,6 +55,17 @@ const taskWordVariant = (label: string): StatusIndicatorVariant => { return 'warning'; }; +const providerLabel = (provider: string): string => { + if (provider === 'proxmox-pbs') return 'Proxmox Backup Server'; + if (provider === 'proxmox-pve') return 'Proxmox VE'; + return provider; +}; + +const evidenceQualityLabel = (value: string): string => { + if (!value) return 'Unknown'; + return value.charAt(0).toUpperCase() + value.slice(1); +}; + // "Workload coverage" table: one row per workload answering "does this have a // backup?" across PBS snapshots / PVE backup files / guest snapshots, each // expanding to its restore evidence. Presentational — the parent owns the @@ -266,10 +278,14 @@ export function ProxmoxCoverageTable(props: { +
+ + {getWorkloadRecoveryPostureLabel(row.posture)}: + {' '} + {row.protectionPosture?.explanation ?? + 'Pulse cannot determine this workload’s protection because no complete provider evidence is linked to it.'} +
+ 0}> +
+
+ Provider evidence +
+
+ + {(provider) => ( +
+ + {providerLabel(provider.provider)} + + + Job {evidenceQualityLabel(provider.jobState)} + + + History {evidenceQualityLabel(provider.historyCompleteness)} + + + Access {evidenceQualityLabel(provider.permissions)} + +
+ )} + +
+
+ 0} fallback={ diff --git a/frontend-modern/src/features/proxmox/__tests__/ProxmoxBackupsTable.test.tsx b/frontend-modern/src/features/proxmox/__tests__/ProxmoxBackupsTable.test.tsx index 35d17b50e..501e8cd30 100644 --- a/frontend-modern/src/features/proxmox/__tests__/ProxmoxBackupsTable.test.tsx +++ b/frontend-modern/src/features/proxmox/__tests__/ProxmoxBackupsTable.test.tsx @@ -25,9 +25,11 @@ const renderInRouter = (component: () => JSX.Element) => )); const apiFetchMock = vi.hoisted(() => vi.fn()); +const apiFetchJSONMock = vi.hoisted(() => vi.fn()); vi.mock('@/utils/apiClient', () => ({ apiFetch: apiFetchMock, + apiFetchJSON: apiFetchJSONMock, })); const jsonResponse = (payload: unknown) => @@ -110,12 +112,40 @@ const pbsPayload = { meta: { totalBackups: 1 }, }; -function mockBackupAPIs() { +function mockBackupAPIs(state: 'protected' | 'attention' = 'protected') { apiFetchMock.mockImplementation((url: string) => { if (url === '/api/backups/pbs') return Promise.resolve(jsonResponse(pbsPayload)); if (url === '/api/backups/pve') return Promise.resolve(jsonResponse(pvePayload)); return Promise.resolve(jsonResponse({})); }); + apiFetchJSONMock.mockResolvedValue({ + data: [ + { + subjectResourceId: 'ct-112', + state, + lastAttemptAt: '2026-05-25T02:00:00Z', + lastSuccessfulPointAt: '2026-05-25T01:34:25Z', + lastVerifiedAt: '2026-05-25T01:34:25Z', + freshness: 'current', + verification: 'verified', + coverage: 'complete', + providerStates: [], + repositoryResourceIds: [], + evidenceIds: ['evidence-1'], + explanation: + state === 'protected' + ? 'A current verified backup is available.' + : 'The latest provider job needs attention.', + evaluatedAt: '2026-05-25T02:05:00Z', + }, + ], + policy: { + freshnessWindowSeconds: 604800, + verificationWindowSeconds: 604800, + requireVerification: true, + }, + meta: { page: 1, limit: 200, total: 1, totalPages: 1 }, + }); } const workloadResource = { @@ -167,11 +197,12 @@ const expectCanonicalPlatformTableShell = (table: HTMLElement): void => { afterEach(() => { cleanup(); apiFetchMock.mockReset(); + apiFetchJSONMock.mockReset(); }); describe('ProxmoxBackupsTable', () => { it('defaults to coverage when protection needs attention and keeps the dated feed one click away', async () => { - mockBackupAPIs(); + mockBackupAPIs('attention'); renderInRouter(() => ( { } expect(apiFetchMock).toHaveBeenCalledWith('/api/backups/pbs'); expect(apiFetchMock).toHaveBeenCalledWith('/api/backups/pve'); + expect(apiFetchJSONMock).toHaveBeenCalledTimes(1); + const postureURL = new URL(apiFetchJSONMock.mock.calls[0][0], 'https://pulse.invalid'); + expect(postureURL.pathname).toBe('/api/recovery/postures'); + expect(postureURL.searchParams.getAll('resourceId')).toEqual(['ct-112']); }); it('offers By date / Coverage views and no legacy sub-tab tree', async () => { @@ -260,10 +295,10 @@ describe('ProxmoxBackupsTable', () => { await screen.findAllByText('pbs-docker'); await fireEvent.click(screen.getByRole('button', { name: /coverage/i })); - // Coverage is the posture view; the workload's recent backup reads - // "Current". + // Coverage is the posture view; the server-owned workload posture reads + // "Protected". expect(screen.getByRole('columnheader', { name: /posture/i })).toBeInTheDocument(); - expect(screen.getAllByText('Current').length).toBeGreaterThan(0); + expect(screen.getAllByText('Protected').length).toBeGreaterThan(0); // Per-source detail is one click down inside the workload's row. await fireEvent.click(screen.getByRole('button', { name: /expand details for pbs-docker/i })); diff --git a/frontend-modern/src/features/proxmox/__tests__/ProxmoxCoverageTable.test.tsx b/frontend-modern/src/features/proxmox/__tests__/ProxmoxCoverageTable.test.tsx index c8929d715..bf1c713ad 100644 --- a/frontend-modern/src/features/proxmox/__tests__/ProxmoxCoverageTable.test.tsx +++ b/frontend-modern/src/features/proxmox/__tests__/ProxmoxCoverageTable.test.tsx @@ -21,8 +21,30 @@ const row = { pbsCount: 1, archiveCount: 0, snapshotCount: 0, - posture: 'current', + posture: 'protected', postureRank: 0, + protectionPosture: { + subjectResourceId: 'resource:vm:100', + state: 'protected', + freshness: 'current', + verification: 'verified', + coverage: 'complete', + providerStates: [ + { + provider: 'proxmox-pbs', + source: 'pbs-backup-enumeration', + scope: 'pbs-main', + jobState: 'success', + historyCompleteness: 'complete', + permissions: 'sufficient', + evidenceIds: ['evidence-provider'], + }, + ], + repositoryResourceIds: [], + evidenceIds: ['evidence-provider'], + explanation: 'A current verified backup is available from complete provider history.', + evaluatedAt: '2026-07-19T00:00:00Z', + }, } as unknown as WorkloadCoverageRow; const headerTexts = () => @@ -69,4 +91,55 @@ describe('ProxmoxCoverageTable column visibility', () => { expect(document.body.textContent).not.toContain('ID 100'); expect(document.body.textContent).not.toContain('Node pve1'); }); + + it('keeps provider evidence in the workload drill-down instead of every table row', () => { + const { unmount } = render(() => ( + } + emptyTitle="" + emptyDescription="" + sortKey={(() => 'posture') as Accessor} + sortDirection={() => 'asc'} + onSort={() => {}} + expandedKeys={new Set()} + onToggleExpand={() => {}} + showPbsColumn={true} + showArchiveColumn={false} + showSnapshotColumn={false} + showTaskColumn={false} + /> + )); + + expect(document.body.textContent).not.toContain('Provider evidence'); + expect(document.body.textContent).not.toContain( + 'A current verified backup is available from complete provider history.', + ); + unmount(); + + render(() => ( + } + emptyTitle="" + emptyDescription="" + sortKey={(() => 'posture') as Accessor} + sortDirection={() => 'asc'} + onSort={() => {}} + expandedKeys={new Set(['w1'])} + onToggleExpand={() => {}} + showPbsColumn={true} + showArchiveColumn={false} + showSnapshotColumn={false} + showTaskColumn={false} + /> + )); + + expect(document.body.textContent).toContain('Provider evidence'); + expect(document.body.textContent).toContain('Proxmox Backup Server'); + expect(document.body.textContent).toContain('History Complete'); + expect(document.body.textContent).toContain('Access Sufficient'); + }); }); diff --git a/frontend-modern/src/features/proxmox/__tests__/proxmoxBackupRecoveryModel.branchcov.test.ts b/frontend-modern/src/features/proxmox/__tests__/proxmoxBackupRecoveryModel.branchcov.test.ts index 9bcb78ea6..fc249722f 100644 --- a/frontend-modern/src/features/proxmox/__tests__/proxmoxBackupRecoveryModel.branchcov.test.ts +++ b/frontend-modern/src/features/proxmox/__tests__/proxmoxBackupRecoveryModel.branchcov.test.ts @@ -1,6 +1,7 @@ import { describe, expect, it } from 'vitest'; -import type { BackupTask, GuestSnapshot, PBSBackup } from '@/types/api'; +import type { BackupTask, PBSBackup } from '@/types/api'; +import type { ProtectionPosture, ProtectionState } from '@/types/recovery'; import type { Resource } from '@/types/resource'; import { @@ -53,18 +54,6 @@ const pbsBackup = (overrides: Partial = {}): PBSBackup => ({ ...overrides, }); -const snap = (overrides: Partial = {}): GuestSnapshot => ({ - id: 'snap-100', - name: 'pre-upgrade', - node: 'node-a', - instance: 'inst-a', - type: 'vm', - vmid: 100, - time: '2026-07-09T00:00:00Z', - vmstate: false, - ...overrides, -}); - const task = (overrides: Partial = {}): BackupTask => ({ id: 'task-100', node: 'node-a', @@ -78,8 +67,25 @@ const task = (overrides: Partial = {}): BackupTask => ({ type ModelInput = Parameters[0]; -const buildModel = (input: Partial): ProxmoxBackupRecoveryModel => - buildProxmoxBackupRecoveryModel({ +const posture = (resourceId: string, state: ProtectionState): ProtectionPosture => ({ + subjectResourceId: resourceId, + state, + freshness: state === 'protected' ? 'current' : 'unknown', + verification: state === 'protected' ? 'verified' : 'unknown', + coverage: state === 'unprotected' ? 'none' : state === 'unknown' ? 'unknown' : 'complete', + providerStates: [], + repositoryResourceIds: [], + evidenceIds: [], + explanation: `Canonical ${state} fixture`, + evaluatedAt: new Date(NOW).toISOString(), +}); + +const buildModel = (input: Partial): ProxmoxBackupRecoveryModel => { + const workloads = input.workloads ?? []; + const protectionPostures = + input.protectionPostures ?? + new Map(workloads.map((resource) => [resource.id, posture(resource.id, 'protected')])); + return buildProxmoxBackupRecoveryModel({ workloads: [], pbsBackups: [], archives: [], @@ -87,7 +93,9 @@ const buildModel = (input: Partial): ProxmoxBackupRecoveryModel => tasks: [], nowMs: NOW, ...input, + protectionPostures, }); +}; // --------------------------------------------------------------------------- // getWorkloadRecoveryPostureLabel — uncovered switch cases @@ -96,97 +104,37 @@ const buildModel = (input: Partial): ProxmoxBackupRecoveryModel => describe('getWorkloadRecoveryPostureLabel uncovered switch cases', () => { it.each<[WorkloadRecoveryPosture, string]>([ - ['aging', 'Aging'], - ['stale', 'Stale'], - ['snapshot-only', 'Snapshot only'], - ['uncovered', 'Uncovered'], - ['unverified', 'Unverified'], + ['protected', 'Protected'], + ['attention', 'Needs attention'], + ['unprotected', 'Unprotected'], + ['unknown', 'Unknown'], ])('renders posture %s as %j', (posture, expected) => { expect(getWorkloadRecoveryPostureLabel(posture)).toBe(expected); }); }); -// --------------------------------------------------------------------------- -// buildPosture — uncovered posture arms (private; exercised via the model). -// Sibling tests cover 'current', 'uncovered', and the 'failed' posture when a -// failed task started AFTER the recovery (startedMs >= createdMs). -// --------------------------------------------------------------------------- - -describe('buildPosture uncovered arms (via buildProxmoxBackupRecoveryModel)', () => { - it('classifies a 14-day-old PBS recovery as "aging"', () => { +describe('canonical protection posture ownership', () => { + it('uses the server posture even when raw artifacts suggest a different answer', () => { + const resource = workload({}); const model = buildModel({ - workloads: [workload({})], - pbsBackups: [pbsBackup({ backupTime: isoDaysAgo(14) })], + workloads: [resource], + pbsBackups: [pbsBackup({ backupTime: isoDaysAgo(1), verified: true })], + protectionPostures: new Map([[resource.id, posture(resource.id, 'attention')]]), }); - expect(model.coverageRows[0].posture).toBe('aging'); - expect(model.coverageRows[0].postureRank).toBe(4); - }); - - it('classifies a 45-day-old PBS recovery as "stale"', () => { - const model = buildModel({ - workloads: [workload({})], - pbsBackups: [pbsBackup({ backupTime: isoDaysAgo(45) })], - }); - expect(model.coverageRows[0].posture).toBe('stale'); - expect(model.coverageRows[0].postureRank).toBe(2); - }); - - it('flags a recent unverified PBS recovery as "unverified"', () => { - const model = buildModel({ - workloads: [workload({})], - pbsBackups: [pbsBackup({ backupTime: isoDaysAgo(1), verified: false })], - }); - expect(model.coverageRows[0].posture).toBe('unverified'); - expect(model.coverageRows[0].postureRank).toBe(2); - }); - - it('flags a guest-snapshot-only workload as "snapshot-only"', () => { - const model = buildModel({ - workloads: [workload({})], - snapshots: [snap({})], - }); - expect(model.coverageRows[0].posture).toBe('snapshot-only'); - expect(model.coverageRows[0].postureRank).toBe(3); - expect(model.coverageRows[0].snapshotCount).toBe(1); - expect(model.coverageRows[0].pbsCount).toBe(0); - }); - - it('treats a failed task with an unparseable start time as the latest failure', () => { - // startedMs is undefined -> the `latestTask.startedMs === undefined` arm of - // the failedTask disjunction fires even though the recovery has a createdMs. - const model = buildModel({ - workloads: [workload({})], - pbsBackups: [pbsBackup({ backupTime: isoDaysAgo(1) })], - tasks: [ - task({ - status: 'failed', - startTime: '', - type: 'vm', - }), - ], - }); - expect(model.coverageRows[0].posture).toBe('failed'); + expect(model.coverageRows[0].posture).toBe('attention'); expect(model.coverageRows[0].postureRank).toBe(0); - expect(model.coverageRows[0].latestTask?.label).toBe('Failed'); - expect(model.coverageRows[0].latestTask?.startedMs).toBeUndefined(); + expect(model.coverageRows[0].protectionPosture?.explanation).toBe( + 'Canonical attention fixture', + ); }); - it('does not flag "failed" when a failed task predates the latest recovery', () => { - // failedTask is false because startedMs < latestRecovery.createdMs, so the - // failedTask conjunction is false and posture falls through to the age band. + it('reports unknown when no canonical posture is available', () => { const model = buildModel({ workloads: [workload({})], - pbsBackups: [pbsBackup({ backupTime: isoDaysAgo(1) })], - tasks: [ - task({ - status: 'failed', - startTime: isoDaysAgo(10), - type: 'vm', - }), - ], + protectionPostures: new Map(), }); - expect(model.coverageRows[0].posture).toBe('current'); - expect(model.coverageRows[0].latestTask?.label).toBe('Failed'); + expect(model.coverageRows[0].posture).toBe('unknown'); + expect(model.coverageRows[0].postureRank).toBe(2); }); }); @@ -324,9 +272,7 @@ describe('resourceBackupType branches (via candidate type)', () => { describe('resourceNode branches (via candidate node)', () => { it('prefers proxmox.nodeName when present', () => { const model = buildModel({ - workloads: [ - workload({ id: 'vm-721', proxmox: { vmid: 721, nodeName: 'named-a' } }), - ], + workloads: [workload({ id: 'vm-721', proxmox: { vmid: 721, nodeName: 'named-a' } })], }); expect(model.coverageRows[0].workload.node).toBe('named-a'); }); @@ -340,9 +286,7 @@ describe('resourceNode branches (via candidate node)', () => { it('falls back to parentName when no proxmox node field is set', () => { const model = buildModel({ - workloads: [ - workload({ id: 'vm-723', parentName: 'parent-x', proxmox: { vmid: 723 } }), - ], + workloads: [workload({ id: 'vm-723', parentName: 'parent-x', proxmox: { vmid: 723 } })], }); expect(model.coverageRows[0].workload.node).toBe('parent-x'); }); @@ -527,9 +471,10 @@ describe('buildProxmoxBackupRecoveryModel summary, sort, and empty edges', () => expect(model.recoverableArtifacts).toHaveLength(0); expect(model.coverageSummary).toEqual({ totalWorkloads: 0, - current: 0, + protected: 0, attention: 0, - uncovered: 0, + unprotected: 0, + unknown: 0, withPBS: 0, recoverableArtifacts: 0, totalBytes: 0, @@ -550,12 +495,17 @@ describe('buildProxmoxBackupRecoveryModel summary, sort, and empty edges', () => expect(model.coverageSummary.totalBytes).toBe(7_000); expect(model.coverageSummary.recoverableArtifacts).toBe(2); expect(model.coverageSummary.withPBS).toBe(2); - expect(model.coverageSummary.current).toBe(2); + expect(model.coverageSummary.protected).toBe(2); expect(model.coverageSummary.attention).toBe(0); }); it('sorts rows by postureRank asc, then latestRecovery createdMs desc', () => { - const model = buildModel({ + const postures = new Map([ + ['vm-830', posture('vm-830', 'unprotected')], + ['vm-840', posture('vm-840', 'protected')], + ['vm-850', posture('vm-850', 'protected')], + ]); + const canonicalModel = buildModel({ workloads: [ workload({ id: 'vm-840', proxmox: { vmid: 840, node: 'a' } }), workload({ id: 'vm-830', proxmox: { vmid: 830, node: 'b' } }), @@ -565,11 +515,14 @@ describe('buildProxmoxBackupRecoveryModel summary, sort, and empty edges', () => pbsBackup({ id: 'pbs-840', vmid: '840', backupTime: isoDaysAgo(1) }), pbsBackup({ id: 'pbs-850', vmid: '850', backupTime: isoDaysAgo(2) }), ], + protectionPostures: postures, }); - // 830 has no recovery -> uncovered (rank 1) sorts first. - // 840 and 850 are current (rank 5); tie broken by newer recovery first. - expect(model.coverageRows.map((row) => row.workload.vmid)).toEqual(['830', '840', '850']); - expect(model.coverageSummary.uncovered).toBe(1); + expect(canonicalModel.coverageRows.map((row) => row.workload.vmid)).toEqual([ + '830', + '840', + '850', + ]); + expect(canonicalModel.coverageSummary.unprotected).toBe(1); }); it('sorts recoverable artifacts by createdMs desc', () => { diff --git a/frontend-modern/src/features/proxmox/__tests__/proxmoxBackupRecoveryModel.test.ts b/frontend-modern/src/features/proxmox/__tests__/proxmoxBackupRecoveryModel.test.ts index aaa8bb82f..09cd66dbb 100644 --- a/frontend-modern/src/features/proxmox/__tests__/proxmoxBackupRecoveryModel.test.ts +++ b/frontend-modern/src/features/proxmox/__tests__/proxmoxBackupRecoveryModel.test.ts @@ -1,6 +1,7 @@ import { describe, expect, it } from 'vitest'; import type { BackupTask, GuestSnapshot, PBSBackup, StorageBackup } from '@/types/api'; +import type { ProtectionPosture, ProtectionState } from '@/types/recovery'; import type { Resource } from '@/types/resource'; import { buildProxmoxBackupRecoveryModel, @@ -83,8 +84,21 @@ const task = (overrides: Partial = {}): BackupTask => ({ ...overrides, }); +const protectionPosture = (resourceId: string, state: ProtectionState): ProtectionPosture => ({ + subjectResourceId: resourceId, + state, + freshness: state === 'protected' ? 'current' : 'unknown', + verification: state === 'protected' ? 'verified' : 'unknown', + coverage: state === 'unprotected' ? 'none' : state === 'unknown' ? 'unknown' : 'complete', + providerStates: [], + repositoryResourceIds: [], + evidenceIds: [], + explanation: `Canonical ${state} fixture`, + evaluatedAt: '2026-05-26T08:00:00Z', +}); + describe('proxmoxBackupRecoveryModel', () => { - it('classifies backup ages with the same thresholds used by coverage posture', () => { + it('classifies backup ages for restore-point presentation', () => { const nowMs = Date.parse('2026-05-26T08:00:00Z'); expect(getRecoveryAgeBand(nowMs - 2 * 24 * 60 * 60 * 1000, nowMs)).toBe('current'); @@ -101,6 +115,7 @@ describe('proxmoxBackupRecoveryModel', () => { snapshots: [snapshot()], tasks: [task()], nowMs: Date.parse('2026-05-26T08:00:00Z'), + protectionPostures: new Map([['vm-112', protectionPosture('vm-112', 'protected')]]), }); expect(model.coverageRows).toHaveLength(1); @@ -112,7 +127,7 @@ describe('proxmoxBackupRecoveryModel', () => { expect(row.archiveCount).toBe(1); expect(row.snapshotCount).toBe(1); expect(row.latestTask?.label).toBe('OK'); - expect(getWorkloadRecoveryPostureLabel(row.posture)).toBe('Current'); + expect(getWorkloadRecoveryPostureLabel(row.posture)).toBe('Protected'); expect(model.recoverableArtifacts.map((artifact) => artifact.sourceLabel)).toEqual([ 'PBS', 'PVE file', @@ -140,7 +155,7 @@ describe('proxmoxBackupRecoveryModel', () => { ); }); - it('surfaces a failed latest backup task as workload attention', () => { + it('uses canonical workload attention while retaining the failed task as evidence', () => { const model = buildProxmoxBackupRecoveryModel({ workloads: [workload({})], pbsBackups: [pbsBackup({ backupTime: '2026-05-25T01:34:25Z' })], @@ -155,27 +170,30 @@ describe('proxmoxBackupRecoveryModel', () => { }), ], nowMs: Date.parse('2026-05-26T08:00:00Z'), + protectionPostures: new Map([['vm-112', protectionPosture('vm-112', 'attention')]]), }); const row = model.coverageRows[0]; - expect(row.posture).toBe('failed'); - expect(getWorkloadRecoveryPostureLabel(row.posture)).toBe('Failed latest task'); + expect(row.posture).toBe('attention'); + expect(getWorkloadRecoveryPostureLabel(row.posture)).toBe('Needs attention'); expect(coverageRowMatchesSearch(row, 'storage unavailable')).toBe(true); }); - it('keeps inventory workloads with no restore point visible as uncovered', () => { + it('keeps an inventory workload with canonical unprotected evidence visible', () => { + const resource = workload({ id: 'vm-200', proxmox: { vmid: 200, node: 'delly' } }); const model = buildProxmoxBackupRecoveryModel({ - workloads: [workload({ id: 'vm-200', proxmox: { vmid: 200, node: 'delly' } })], + workloads: [resource], pbsBackups: [], archives: [], snapshots: [], tasks: [], nowMs: Date.parse('2026-05-26T08:00:00Z'), + protectionPostures: new Map([[resource.id, protectionPosture(resource.id, 'unprotected')]]), }); expect(model.coverageRows).toHaveLength(1); - expect(model.coverageRows[0].posture).toBe('uncovered'); - expect(model.coverageSummary.uncovered).toBe(1); + expect(model.coverageRows[0].posture).toBe('unprotected'); + expect(model.coverageSummary.unprotected).toBe(1); }); it('does not treat a linked Pulse agent facet as recovery evidence or authority', () => { @@ -200,8 +218,8 @@ describe('proxmoxBackupRecoveryModel', () => { }); expect(model.coverageRows).toHaveLength(1); - expect(model.coverageRows[0].posture).toBe('uncovered'); - expect(model.coverageSummary.uncovered).toBe(1); + expect(model.coverageRows[0].posture).toBe('unknown'); + expect(model.coverageSummary.unknown).toBe(1); expect(model.coverageRows[0].pbsCount).toBe(0); expect(model.coverageRows[0].archiveCount).toBe(0); expect(model.coverageRows[0].snapshotCount).toBe(0); diff --git a/frontend-modern/src/features/proxmox/proxmoxBackupRecoveryModel.ts b/frontend-modern/src/features/proxmox/proxmoxBackupRecoveryModel.ts index 8192a3e5a..c315aed5b 100644 --- a/frontend-modern/src/features/proxmox/proxmoxBackupRecoveryModel.ts +++ b/frontend-modern/src/features/proxmox/proxmoxBackupRecoveryModel.ts @@ -1,4 +1,5 @@ import type { BackupTask, GuestSnapshot, PBSBackup, StorageBackup } from '@/types/api'; +import type { ProtectionPosture, ProtectionState } from '@/types/recovery'; import type { Resource } from '@/types/resource'; import { @@ -9,17 +10,11 @@ import { export type RecoverableSourceKind = ProxmoxBackupSourceKind; -export type WorkloadRecoveryPosture = - | 'current' - | 'aging' - | 'stale' - | 'snapshot-only' - | 'failed' - | 'uncovered' - | 'unverified'; +export type WorkloadRecoveryPosture = ProtectionState; export interface WorkloadReference { key: string; + resourceId?: string; type: 'vm' | 'ct' | 'host' | 'unknown'; typeLabel: string; vmid: string; @@ -70,6 +65,7 @@ export interface WorkloadCoverageRow { snapshotCount: number; posture: WorkloadRecoveryPosture; postureRank: number; + protectionPosture?: ProtectionPosture; // True only when a VM/LXC row exists because a backup/task referenced a VMID // with no matching live inventory guest. Host backups can also carry a // `backup:` key, but they are first-class backup targets, not orphaned guests. @@ -81,9 +77,10 @@ export interface ProxmoxBackupRecoveryModel { recoverableArtifacts: RecoverableArtifact[]; coverageSummary: { totalWorkloads: number; - current: number; + protected: number; attention: number; - uncovered: number; + unprotected: number; + unknown: number; withPBS: number; recoverableArtifacts: number; totalBytes: number; @@ -97,6 +94,7 @@ interface BuildModelInput { snapshots: readonly GuestSnapshot[]; tasks: readonly BackupTask[]; nowMs: number; + protectionPostures?: ReadonlyMap; } interface WorkloadCandidate extends WorkloadReference { @@ -242,6 +240,7 @@ function buildCandidateFromResource(resource: Resource): WorkloadCandidate | nul const instance = resourceInstance(resource); return { key: `resource:${resource.id}`, + resourceId: resource.id, type, typeLabel: typeLabel(type), vmid, @@ -380,47 +379,28 @@ function taskDurationSeconds(task: BackupTask): number | undefined { return Math.round((end - start) / 1000); } -function buildPosture(row: WorkloadRowDraft, nowMs: number) { - const failedTask = - row.latestTask?.label === 'Failed' && - (row.latestRecovery?.createdMs === undefined || - row.latestTask.startedMs === undefined || - row.latestTask.startedMs >= row.latestRecovery.createdMs); - if (failedTask) return { posture: 'failed' as const, rank: 0 }; - if (!row.latestRecovery) return { posture: 'uncovered' as const, rank: 1 }; - if (row.latestRecovery.sourceKind === 'pbs' && row.latestRecovery.verified === false) { - return { posture: 'unverified' as const, rank: 2 }; - } - const hasExternalBackup = row.latestPBS !== undefined || row.latestArchive !== undefined; - if (!hasExternalBackup && row.latestSnapshot) - return { posture: 'snapshot-only' as const, rank: 3 }; - const ageBand = getRecoveryAgeBand(row.latestRecovery.createdMs, nowMs); - if (ageBand === 'current') return { posture: 'current' as const, rank: 5 }; - if (ageBand === 'aging') return { posture: 'aging' as const, rank: 4 }; - return { posture: 'stale' as const, rank: 2 }; +function protectionPostureRank(posture: WorkloadRecoveryPosture): number { + if (posture === 'attention') return 0; + if (posture === 'unprotected') return 1; + if (posture === 'unknown') return 2; + return 3; } export function getWorkloadRecoveryPostureLabel(posture: WorkloadRecoveryPosture): string { switch (posture) { - case 'current': - return 'Current'; - case 'aging': - return 'Aging'; - case 'stale': - return 'Stale'; - case 'snapshot-only': - return 'Snapshot only'; - case 'failed': - return 'Failed latest task'; - case 'uncovered': - return 'Uncovered'; - case 'unverified': - return 'Unverified'; + case 'protected': + return 'Protected'; + case 'attention': + return 'Needs attention'; + case 'unprotected': + return 'Unprotected'; + case 'unknown': + return 'Unknown'; } } export function isCoverageAttention(posture: WorkloadRecoveryPosture): boolean { - return posture !== 'current'; + return posture === 'attention'; } export function buildProxmoxBackupRecoveryModel( @@ -569,11 +549,15 @@ export function buildProxmoxBackupRecoveryModel( } const coverageRows = Array.from(rows.values()).map((row) => { - const posture = buildPosture(row, input.nowMs); + const protectionPosture = row.workload.resourceId + ? input.protectionPostures?.get(row.workload.resourceId) + : undefined; + const posture = protectionPosture?.state ?? 'unknown'; return { ...row, - posture: posture.posture, - postureRank: posture.rank, + posture, + postureRank: protectionPostureRank(posture), + protectionPosture, isOrphaned: !row.key.startsWith('resource:') && (row.workload.type === 'vm' || row.workload.type === 'ct'), @@ -592,9 +576,10 @@ export function buildProxmoxBackupRecoveryModel( recoverableArtifacts: artifacts, coverageSummary: { totalWorkloads: coverageRows.length, - current: coverageRows.filter((row) => row.posture === 'current').length, + protected: coverageRows.filter((row) => row.posture === 'protected').length, attention: coverageRows.filter((row) => isCoverageAttention(row.posture)).length, - uncovered: coverageRows.filter((row) => row.posture === 'uncovered').length, + unprotected: coverageRows.filter((row) => row.posture === 'unprotected').length, + unknown: coverageRows.filter((row) => row.posture === 'unknown').length, withPBS: coverageRows.filter((row) => row.pbsCount > 0).length, recoverableArtifacts: artifacts.length, totalBytes, diff --git a/frontend-modern/src/features/proxmox/proxmoxBackupsTableModel.ts b/frontend-modern/src/features/proxmox/proxmoxBackupsTableModel.ts index 8ef5ab74d..b79ed3715 100644 --- a/frontend-modern/src/features/proxmox/proxmoxBackupsTableModel.ts +++ b/frontend-modern/src/features/proxmox/proxmoxBackupsTableModel.ts @@ -11,14 +11,9 @@ import { getWorkloadTypePresentation } from '@/utils/workloadTypePresentation'; export type BackupTabId = 'coverage' | 'recoverable' | 'sources' | 'tasks'; export type SourceDetailTabId = 'pbs' | 'snapshots' | 'archives'; -export type CoverageFilterValue = 'all' | 'attention' | 'current' | 'uncovered'; +export type CoverageFilterValue = 'all' | 'attention' | 'protected' | 'unprotected' | 'unknown'; export type RecoverableFilterValue = - | 'all' - | 'pbs' - | 'archive' - | 'snapshot' - | 'verified' - | 'unverified'; + 'all' | 'pbs' | 'archive' | 'snapshot' | 'verified' | 'unverified'; export type SnapshotFilterValue = 'all' | 'recent' | 'stale' | 'with-ram'; // One guest's aggregated snapshot inventory, the row shape for the Snapshots @@ -159,13 +154,7 @@ export function cmpBool(a: boolean, b: boolean, direction: 'asc' | 'desc'): numb // `asc` (A→Z). Boolean columns default to `desc` so "true" sorts first. export type CoverageSortKey = - | 'posture' - | 'workload' - | 'latest' - | 'pbs' - | 'archive' - | 'snapshot' - | 'task'; + 'posture' | 'workload' | 'latest' | 'pbs' | 'archive' | 'snapshot' | 'task'; export const COVERAGE_SORT_DEFAULT_DIRECTION: Record = { posture: 'asc', workload: 'asc', diff --git a/frontend-modern/src/features/proxmox/proxmoxBackupsTableShared.tsx b/frontend-modern/src/features/proxmox/proxmoxBackupsTableShared.tsx index 2c0a75fcc..d4b6ee678 100644 --- a/frontend-modern/src/features/proxmox/proxmoxBackupsTableShared.tsx +++ b/frontend-modern/src/features/proxmox/proxmoxBackupsTableShared.tsx @@ -116,17 +116,22 @@ export const COVERAGE_FILTERS: FilterOption[] = [ leading: filterChipStatusDot('bg-amber-500'), }, { - value: 'current', - label: 'Current', + value: 'protected', + label: 'Protected', tone: 'success', leading: filterChipStatusDot('bg-emerald-500'), }, { - value: 'uncovered', - label: 'Uncovered', + value: 'unprotected', + label: 'Unprotected', tone: 'danger', leading: filterChipStatusDot('bg-red-500'), }, + { + value: 'unknown', + label: 'Unknown', + leading: filterChipStatusDot('bg-base-content/40'), + }, ]; const recoverableSourceFilterOption = ( diff --git a/frontend-modern/src/hooks/__tests__/useProtectionPostures.test.ts b/frontend-modern/src/hooks/__tests__/useProtectionPostures.test.ts new file mode 100644 index 000000000..00f476dc7 --- /dev/null +++ b/frontend-modern/src/hooks/__tests__/useProtectionPostures.test.ts @@ -0,0 +1,31 @@ +import { describe, expect, it } from 'vitest'; +import { + MAX_PROTECTION_POSTURE_BATCH_SIZE, + buildProtectionPostureBatchURL, + normalizeProtectionPostureResourceIDs, +} from '@/hooks/useProtectionPostures'; + +describe('useProtectionPostures transport', () => { + it('deduplicates and sorts resource IDs for a stable bounded cache key and request', () => { + expect(normalizeProtectionPostureResourceIDs(['vm:b', ' vm:a ', '', 'vm:b'])).toEqual([ + 'vm:a', + 'vm:b', + ]); + + const url = new URL( + buildProtectionPostureBatchURL(['vm:b', 'vm:a', 'vm:b']), + 'https://pulse.invalid', + ); + expect(url.pathname).toBe('/api/recovery/postures'); + expect(url.searchParams.getAll('resourceId')).toEqual(['vm:a', 'vm:b']); + expect(url.searchParams.get('limit')).toBe(String(MAX_PROTECTION_POSTURE_BATCH_SIZE)); + }); + + it('rejects a batch larger than the server contract', () => { + const resourceIDs = Array.from( + { length: MAX_PROTECTION_POSTURE_BATCH_SIZE + 1 }, + (_, index) => `vm:${index}`, + ); + expect(() => buildProtectionPostureBatchURL(resourceIDs)).toThrow(/limited to 200/); + }); +}); diff --git a/frontend-modern/src/hooks/useProtectionPostures.ts b/frontend-modern/src/hooks/useProtectionPostures.ts new file mode 100644 index 000000000..3fde4068b --- /dev/null +++ b/frontend-modern/src/hooks/useProtectionPostures.ts @@ -0,0 +1,115 @@ +import { type Accessor, createMemo } from 'solid-js'; +import { createNonSuspendingQuery } from '@/hooks/createNonSuspendingQuery'; +import type { + ProtectionPosture, + ProtectionPosturePolicy, + ProtectionPosturesResponse, +} from '@/types/recovery'; +import { apiFetchJSON } from '@/utils/apiClient'; + +const PROTECTION_POSTURES_URL = '/api/recovery/postures'; +export const MAX_PROTECTION_POSTURE_BATCH_SIZE = 200; +const REFRESH_MS = 30_000; + +const EMPTY_POLICY: ProtectionPosturePolicy = { + freshnessWindowSeconds: 0, + verificationWindowSeconds: 0, + requireVerification: false, +}; + +const EMPTY_RESPONSE: ProtectionPosturesResponse = { + data: [], + policy: EMPTY_POLICY, + meta: { page: 1, limit: MAX_PROTECTION_POSTURE_BATCH_SIZE, total: 0, totalPages: 0 }, +}; + +export function normalizeProtectionPostureResourceIDs( + resourceIDs: readonly string[] | null | undefined, +): string[] { + return [ + ...new Set( + (resourceIDs ?? []).map((resourceID) => resourceID.trim()).filter((resourceID) => resourceID), + ), + ].sort((left, right) => left.localeCompare(right)); +} + +export function buildProtectionPostureBatchURL(resourceIDs: readonly string[]): string { + const normalized = normalizeProtectionPostureResourceIDs(resourceIDs); + if (normalized.length > MAX_PROTECTION_POSTURE_BATCH_SIZE) { + throw new Error( + `Protection posture batches are limited to ${MAX_PROTECTION_POSTURE_BATCH_SIZE} resource IDs.`, + ); + } + const params = new URLSearchParams(); + for (const resourceID of normalized) { + params.append('resourceId', resourceID); + } + params.set('limit', String(MAX_PROTECTION_POSTURE_BATCH_SIZE)); + return `${PROTECTION_POSTURES_URL}?${params.toString()}`; +} + +async function fetchProtectionPostures(resourceIDs: readonly string[]) { + return apiFetchJSON(buildProtectionPostureBatchURL(resourceIDs)); +} + +/** + * Fetch server-owned protection posture in bounded batches. A fleet may exceed + * one API batch, but request count scales by 200-row pages rather than by row. + */ +async function fetchAllProtectionPostures( + resourceIDs: readonly string[], +): Promise { + const batches: string[][] = []; + for (let offset = 0; offset < resourceIDs.length; offset += MAX_PROTECTION_POSTURE_BATCH_SIZE) { + batches.push(resourceIDs.slice(offset, offset + MAX_PROTECTION_POSTURE_BATCH_SIZE)); + } + const responses = await Promise.all(batches.map(fetchProtectionPostures)); + const data = responses.flatMap((response) => response.data ?? []); + return { + data, + policy: responses[0]?.policy ?? EMPTY_POLICY, + meta: { + page: 1, + limit: MAX_PROTECTION_POSTURE_BATCH_SIZE, + total: data.length, + totalPages: responses.length, + }, + }; +} + +export function useProtectionPostures(resourceIDs: Accessor) { + const source = createMemo(() => { + const normalized = normalizeProtectionPostureResourceIDs(resourceIDs()); + return normalized.length > 0 ? JSON.stringify(normalized) : null; + }); + + const state = createNonSuspendingQuery({ + source, + cacheKey: (key) => `protection-postures:${key}`, + fetcher: (key) => fetchAllProtectionPostures(JSON.parse(key) as string[]), + initialValue: EMPTY_RESPONSE, + pollMs: REFRESH_MS, + }); + + const postures = createMemo(() => state.value().data ?? []); + const postureByResourceID = createMemo>( + () => new Map(postures().map((posture) => [posture.subjectResourceId, posture] as const)), + ); + const policy = createMemo(() => state.value().policy ?? EMPTY_POLICY); + + return { + response: { + get error() { + return state.error(); + }, + get loading() { + return state.loading(); + }, + }, + postures, + postureByResourceID, + policy, + refetch: state.refetch, + resolvedOnce: state.resolvedOnce, + }; +} diff --git a/frontend-modern/src/types/recovery.ts b/frontend-modern/src/types/recovery.ts index e2e562b43..5dd8734ec 100644 --- a/frontend-modern/src/types/recovery.ts +++ b/frontend-modern/src/types/recovery.ts @@ -1,13 +1,10 @@ +import type { EvidenceEnvelope, EvidencePermissions } from '@/types/operationalTrust'; + export type RecoveryPlatform = string; export type RecoveryKind = 'snapshot' | 'backup' | 'other' | (string & {}); export type RecoveryMode = 'snapshot' | 'local' | 'remote' | (string & {}); export type RecoveryOutcome = - | 'success' - | 'warning' - | 'failed' - | 'running' - | 'unknown' - | (string & {}); + 'success' | 'warning' | 'failed' | 'running' | 'unknown' | (string & {}); export interface RecoveryExternalRef { type: string; @@ -59,6 +56,8 @@ export interface RecoveryPoint { itemResourceId?: string; repositoryResourceId?: string; + providerScope?: string; + evidence?: EvidenceEnvelope | null; itemRef?: RecoveryExternalRef | null; subjectRef?: RecoveryExternalRef | null; repositoryRef?: RecoveryExternalRef | null; @@ -146,3 +145,51 @@ export interface RecoveryPointsFacets { hasVerification?: boolean; hasEntityId?: boolean; } + +export type ProtectionState = 'protected' | 'attention' | 'unprotected' | 'unknown'; +export type ProtectionFreshness = 'current' | 'stale' | 'unknown'; +export type ProtectionVerification = 'verified' | 'unverified' | 'stale' | 'unknown'; +export type ProtectionCoverage = 'complete' | 'partial' | 'none' | 'unknown'; +export type ProtectionHistoryCompleteness = 'complete' | 'partial' | 'unavailable' | 'unknown'; + +export interface ProtectionProviderState { + provider: RecoveryPlatform; + source: string; + scope: string; + jobState: RecoveryOutcome; + historyCompleteness: ProtectionHistoryCompleteness; + permissions: EvidencePermissions; + lastAttemptAt?: string | null; + lastSuccessAt?: string | null; + lastVerifiedAt?: string | null; + evidenceIds: string[]; + verificationExpected?: boolean; +} + +export interface ProtectionPosture { + subjectResourceId: string; + state: ProtectionState; + lastAttemptAt?: string | null; + lastSuccessfulPointAt?: string | null; + lastVerifiedAt?: string | null; + freshness: ProtectionFreshness; + verification: ProtectionVerification; + coverage: ProtectionCoverage; + providerStates: ProtectionProviderState[]; + repositoryResourceIds: string[]; + evidenceIds: string[]; + explanation: string; + evaluatedAt: string; +} + +export interface ProtectionPosturePolicy { + freshnessWindowSeconds: number; + verificationWindowSeconds: number; + requireVerification: boolean; +} + +export interface ProtectionPosturesResponse { + data: ProtectionPosture[]; + policy: ProtectionPosturePolicy; + meta: RecoveryResponseMeta; +} diff --git a/internal/api/recovery_handlers.go b/internal/api/recovery_handlers.go index 1b0617390..701a437ac 100644 --- a/internal/api/recovery_handlers.go +++ b/internal/api/recovery_handlers.go @@ -11,6 +11,7 @@ import ( "time" "github.com/rcourtman/pulse-go-rewrite/internal/mock" + "github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust" "github.com/rcourtman/pulse-go-rewrite/internal/recovery" recoverymanager "github.com/rcourtman/pulse-go-rewrite/internal/recovery/manager" kubernetesmapper "github.com/rcourtman/pulse-go-rewrite/internal/recovery/mapper/kubernetes" @@ -63,14 +64,16 @@ type recoveryPointPayload struct { Encrypted *bool `json:"encrypted,omitempty"` Immutable *bool `json:"immutable,omitempty"` - ItemResourceID string `json:"itemResourceId,omitempty"` - SubjectResourceID string `json:"subjectResourceId,omitempty"` - RepositoryResourceID string `json:"repositoryResourceId,omitempty"` - ItemRef *recovery.ExternalRef `json:"itemRef,omitempty"` - SubjectRef *recovery.ExternalRef `json:"subjectRef,omitempty"` - RepositoryRef *recovery.ExternalRef `json:"repositoryRef,omitempty"` - Details map[string]any `json:"details,omitempty"` - Display *recovery.RecoveryPointDisplay `json:"display,omitempty"` + ItemResourceID string `json:"itemResourceId,omitempty"` + SubjectResourceID string `json:"subjectResourceId,omitempty"` + RepositoryResourceID string `json:"repositoryResourceId,omitempty"` + ItemRef *recovery.ExternalRef `json:"itemRef,omitempty"` + SubjectRef *recovery.ExternalRef `json:"subjectRef,omitempty"` + RepositoryRef *recovery.ExternalRef `json:"repositoryRef,omitempty"` + Details map[string]any `json:"details,omitempty"` + ProviderScope string `json:"providerScope,omitempty"` + Evidence *operationaltrust.EvidenceEnvelope `json:"evidence,omitempty"` + Display *recovery.RecoveryPointDisplay `json:"display,omitempty"` } type recoveryRollupPayload struct { @@ -85,6 +88,8 @@ type recoveryRollupPayload struct { LastOutcome recovery.Outcome `json:"lastOutcome"` Platforms []recovery.Provider `json:"platforms,omitempty"` Providers []recovery.Provider `json:"providers,omitempty"` + VerifyIntent recovery.VerifyIntent `json:"verifyIntent,omitempty"` + LastVerifiedAt *time.Time `json:"lastVerifiedAt,omitempty"` } func buildRecoveryPointPayload(point recovery.RecoveryPoint) recoveryPointPayload { @@ -108,6 +113,8 @@ func buildRecoveryPointPayload(point recovery.RecoveryPoint) recoveryPointPayloa SubjectRef: point.SubjectRef, RepositoryRef: point.RepositoryRef, Details: point.Details, + ProviderScope: point.ProviderScope, + Evidence: point.Evidence, Display: point.Display, } } @@ -125,6 +132,8 @@ func buildRecoveryRollupPayload(rollup recovery.ProtectionRollup) recoveryRollup LastOutcome: rollup.LastOutcome, Platforms: rollup.Providers, Providers: rollup.Providers, + VerifyIntent: rollup.VerifyIntent, + LastVerifiedAt: rollup.LastVerifiedAt, } } @@ -699,6 +708,248 @@ func (h *RecoveryHandlers) HandleListRollups(w http.ResponseWriter, r *http.Requ } } +const maxProtectionPostureResourceIDs = 200 + +type protectionPosturesResponse struct { + Data []recovery.ProtectionPosture `json:"data"` + Policy recovery.ProtectionPosturePolicyPayload `json:"policy"` + Meta struct { + Page int `json:"page"` + Limit int `json:"limit"` + Total int `json:"total"` + TotalPages int `json:"totalPages"` + } `json:"meta"` +} + +func parseProtectionPostureResourceIDs(qs url.Values) []string { + values := make([]string, 0, len(qs["resourceId"])+len(qs["resourceIds"])) + values = append(values, qs["resourceId"]...) + values = append(values, qs["resourceIds"]...) + unique := make(map[string]struct{}, len(values)) + for _, value := range values { + for _, part := range strings.Split(value, ",") { + if part = strings.TrimSpace(part); part != "" { + unique[part] = struct{}{} + } + } + } + out := make([]string, 0, len(unique)) + for value := range unique { + out = append(out, value) + } + sort.Strings(out) + return out +} + +func (h *RecoveryHandlers) HandleListProtectionPostures( + w http.ResponseWriter, + r *http.Request, +) { + if r.Method != http.MethodGet { + http.Error(w, "Method not allowed", http.StatusMethodNotAllowed) + return + } + + qs := r.URL.Query() + resourceIDs := parseProtectionPostureResourceIDs(qs) + if len(resourceIDs) > maxProtectionPostureResourceIDs { + writeErrorResponse( + w, + http.StatusBadRequest, + "too_many_resource_ids", + "At most 200 resource IDs may be evaluated in one posture batch.", + map[string]string{"limit": strconv.Itoa(maxProtectionPostureResourceIDs)}, + ) + return + } + state := recovery.ProtectionState(strings.TrimSpace(qs.Get("state"))) + if state != "" && !state.Valid() { + writeErrorResponse( + w, + http.StatusBadRequest, + "invalid_protection_state", + "Protection state must be protected, attention, unprotected, or unknown.", + map[string]string{"state": string(state)}, + ) + return + } + page := parseIntQuery(qs, "page", 1) + limit := parseIntQuery(qs, "limit", 100) + if limit <= 0 { + limit = 100 + } + if limit > maxProtectionPostureResourceIDs { + limit = maxProtectionPostureResourceIDs + } + + query := recovery.ProtectionPostureQuery{ + SubjectResourceIDs: resourceIDs, + State: state, + Page: page, + Limit: limit, + } + var ( + postures []recovery.ProtectionPosture + total int + err error + ) + if mock.IsMockEnabled() { + postures, total = mockProtectionPostures( + mock.CurrentFixtureGraph().RecoveryPoints(), + query, + time.Now().UTC(), + ) + } else { + orgID := GetOrgID(r.Context()) + store, storeErr := h.storeForOrg(orgID) + if storeErr != nil { + http.Error( + w, + sanitizeErrorForClient(storeErr, "Internal server error"), + http.StatusInternalServerError, + ) + return + } + postures, total, err = store.ListProtectionPostures(r.Context(), query) + if err != nil { + http.Error( + w, + sanitizeErrorForClient(err, "Internal server error"), + http.StatusInternalServerError, + ) + return + } + } + + var response protectionPosturesResponse + response.Data = postures + response.Policy = recovery.DefaultProtectionPosturePolicy.Payload() + response.Meta.Page = page + response.Meta.Limit = limit + response.Meta.Total = total + if total == 0 { + response.Meta.TotalPages = 0 + } else { + response.Meta.TotalPages = (total + limit - 1) / limit + } + if err := utils.WriteJSONResponse(w, response); err != nil { + log.Error().Err(err).Msg("Failed to serialize protection posture response") + } +} + +func mockProtectionPostures( + points []recovery.RecoveryPoint, + query recovery.ProtectionPostureQuery, + now time.Time, +) ([]recovery.ProtectionPosture, int) { + pointsByResource := make(map[string][]recovery.RecoveryPoint) + observationsByKey := make(map[string]recovery.ProtectionProviderObservation) + for _, point := range points { + resourceID := strings.TrimSpace(point.SubjectResourceID) + if resourceID == "" { + continue + } + pointsByResource[resourceID] = append(pointsByResource[resourceID], point) + scope := recovery.ProviderScopeForPoint(point) + key := string(point.Provider) + "\x00" + scope + if _, exists := observationsByKey[key]; exists { + continue + } + observation, err := recovery.NewProtectionProviderObservation( + point.Provider, + "mock-complete-recovery-fixture", + scope, + recovery.OutcomeUnknown, + recovery.ProtectionHistoryComplete, + operationaltrust.EvidencePermissionsSufficient, + point.Provider == recovery.ProviderProxmoxPBS, + now, + now, + nil, + ) + if err == nil { + observationsByKey[key] = observation + } + } + observations := make([]recovery.ProtectionProviderObservation, 0, len(observationsByKey)) + for _, observation := range observationsByKey { + observations = append(observations, observation) + } + + resourceIDs := append([]string(nil), query.SubjectResourceIDs...) + if len(resourceIDs) == 0 { + for resourceID := range pointsByResource { + resourceIDs = append(resourceIDs, resourceID) + } + sort.Strings(resourceIDs) + } + postures := make([]recovery.ProtectionPosture, 0, len(resourceIDs)) + for _, resourceID := range resourceIDs { + posture := recovery.BuildProtectionPostureFromPointsAt( + resourceID, + pointsByResource[resourceID], + observations, + recovery.DefaultProtectionPosturePolicy, + now, + ) + if query.State != "" && posture.State != query.State { + continue + } + postures = append(postures, posture) + } + sort.SliceStable(postures, func(i, j int) bool { + iRank := protectionPostureStateRank(postures[i].State) + jRank := protectionPostureStateRank(postures[j].State) + if iRank != jRank { + return iRank < jRank + } + return postures[i].SubjectResourceID < postures[j].SubjectResourceID + }) + total := len(postures) + if len(query.SubjectResourceIDs) > 0 { + return postures, total + } + return paginateProtectionPostures(postures, query.Page, query.Limit), total +} + +func protectionPostureStateRank(state recovery.ProtectionState) int { + switch state { + case recovery.ProtectionStateAttention: + return 0 + case recovery.ProtectionStateUnprotected: + return 1 + case recovery.ProtectionStateUnknown: + return 2 + default: + return 3 + } +} + +func paginateProtectionPostures( + postures []recovery.ProtectionPosture, + page int, + limit int, +) []recovery.ProtectionPosture { + if len(postures) == 0 { + return []recovery.ProtectionPosture{} + } + if page <= 0 { + page = 1 + } + if limit <= 0 { + limit = 100 + } + start := (page - 1) * limit + if start >= len(postures) { + return []recovery.ProtectionPosture{} + } + end := start + limit + if end > len(postures) { + end = len(postures) + } + return postures[start:end] +} + func filterRecoveryPointsForRollups(all []recovery.RecoveryPoint, opts recovery.ListPointsOptions) []recovery.RecoveryPoint { if len(all) == 0 { return nil diff --git a/internal/api/recovery_handlers_test.go b/internal/api/recovery_handlers_test.go index 9ebc03960..a2df5b4d5 100644 --- a/internal/api/recovery_handlers_test.go +++ b/internal/api/recovery_handlers_test.go @@ -8,10 +8,12 @@ import ( "net/http/httptest" "net/url" "path/filepath" + "strconv" "testing" "time" "github.com/rcourtman/pulse-go-rewrite/internal/config" + "github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust" "github.com/rcourtman/pulse-go-rewrite/internal/recovery" recoverymanager "github.com/rcourtman/pulse-go-rewrite/internal/recovery/manager" _ "modernc.org/sqlite" @@ -207,10 +209,13 @@ func TestHandleListRollupsExposeCanonicalPlatformsPayload(t *testing.T) { } func TestBuildRecoveryRollupPayloadExposesCanonicalItemResourceIDField(t *testing.T) { + verifiedAt := time.Date(2026, 7, 19, 6, 0, 0, 0, time.UTC) payload := buildRecoveryRollupPayload(recovery.ProtectionRollup{ RollupID: "res:vm-123", SubjectResourceID: "vm-123", LastOutcome: recovery.Outcome("success"), + VerifyIntent: recovery.VerifyIntentVerified, + LastVerifiedAt: &verifiedAt, }) if payload.ItemResourceID != "vm-123" { @@ -219,6 +224,136 @@ func TestBuildRecoveryRollupPayloadExposesCanonicalItemResourceIDField(t *testin if payload.SubjectResourceID != "vm-123" { t.Fatalf("payload.SubjectResourceID = %q, want %q", payload.SubjectResourceID, "vm-123") } + if payload.VerifyIntent != recovery.VerifyIntentVerified { + t.Fatalf("payload.VerifyIntent = %q, want verified", payload.VerifyIntent) + } + if payload.LastVerifiedAt == nil || !payload.LastVerifiedAt.Equal(verifiedAt) { + t.Fatalf("payload.LastVerifiedAt = %v, want %v", payload.LastVerifiedAt, verifiedAt) + } +} + +func TestHandleListProtectionPosturesReturnsBoundedCanonicalBatch(t *testing.T) { + t.Parallel() + + mtp := config.NewMultiTenantPersistence(t.TempDir()) + manager := recoverymanager.New(mtp) + store, err := manager.StoreForOrg("default") + if err != nil { + t.Fatalf("StoreForOrg(default): %v", err) + } + now := time.Now().UTC().Truncate(time.Millisecond) + completedAt := now.Add(-time.Hour) + verified := true + point := recovery.RecoveryPoint{ + ID: "pbs-backup:vm-123", + Provider: recovery.ProviderProxmoxPBS, + Kind: recovery.KindBackup, + Mode: recovery.ModeRemote, + Outcome: recovery.OutcomeSuccess, + CompletedAt: &completedAt, + Verified: &verified, + SubjectResourceID: "vm-123", + ProviderScope: "pbs-main", + } + evidence, err := recovery.NewRecoveryPointEvidence( + point, + "pbs-backup-inventory", + now, + ) + if err != nil { + t.Fatalf("NewRecoveryPointEvidence() error = %v", err) + } + point.Evidence = evidence + observation, err := recovery.NewProtectionProviderObservation( + recovery.ProviderProxmoxPBS, + "pbs-backup-enumeration", + "pbs-main", + recovery.OutcomeSuccess, + recovery.ProtectionHistoryComplete, + operationaltrust.EvidencePermissionsSufficient, + true, + now, + now, + nil, + ) + if err != nil { + t.Fatalf("NewProtectionProviderObservation() error = %v", err) + } + if err := store.UpsertProtectionProviderObservations( + context.Background(), + []recovery.ProtectionProviderObservation{observation}, + ); err != nil { + t.Fatalf("UpsertProtectionProviderObservations() error = %v", err) + } + if err := store.UpsertPoints(context.Background(), []recovery.RecoveryPoint{point}); err != nil { + t.Fatalf("UpsertPoints() error = %v", err) + } + + handler := NewRecoveryHandlers(manager) + request := httptest.NewRequest( + http.MethodGet, + "/api/recovery/postures?resourceId=vm-123&resourceId=vm-missing", + nil, + ) + response := httptest.NewRecorder() + handler.HandleListProtectionPostures(response, request) + if response.Code != http.StatusOK { + t.Fatalf( + "HandleListProtectionPostures() status = %d, want 200 body=%s", + response.Code, + response.Body.String(), + ) + } + var payload struct { + Data []recovery.ProtectionPosture `json:"data"` + Policy struct { + FreshnessWindowSeconds int64 `json:"freshnessWindowSeconds"` + VerificationWindowSeconds int64 `json:"verificationWindowSeconds"` + } `json:"policy"` + Meta struct { + Total int `json:"total"` + Limit int `json:"limit"` + } `json:"meta"` + } + if err := json.Unmarshal(response.Body.Bytes(), &payload); err != nil { + t.Fatalf("json.Unmarshal() error = %v", err) + } + if payload.Meta.Total != 2 || len(payload.Data) != 2 { + t.Fatalf("postures total=%d len=%d, want 2/2", payload.Meta.Total, len(payload.Data)) + } + byID := make(map[string]recovery.ProtectionPosture) + for _, posture := range payload.Data { + byID[posture.SubjectResourceID] = posture + } + if byID["vm-123"].State != recovery.ProtectionStateProtected { + t.Fatalf("vm-123 posture = %#v, want protected", byID["vm-123"]) + } + if byID["vm-missing"].State != recovery.ProtectionStateUnknown { + t.Fatalf("vm-missing posture = %#v, want unknown", byID["vm-missing"]) + } + if payload.Policy.FreshnessWindowSeconds <= 0 || + payload.Policy.VerificationWindowSeconds <= 0 { + t.Fatalf("policy = %#v, want positive server evaluation windows", payload.Policy) + } +} + +func TestHandleListProtectionPosturesRejectsUnboundedBatch(t *testing.T) { + t.Parallel() + + values := url.Values{} + for i := 0; i <= maxProtectionPostureResourceIDs; i++ { + values.Add("resourceId", "resource-"+strconv.Itoa(i)) + } + request := httptest.NewRequest( + http.MethodGet, + "/api/recovery/postures?"+values.Encode(), + nil, + ) + response := httptest.NewRecorder() + NewRecoveryHandlers(nil).HandleListProtectionPostures(response, request) + if response.Code != http.StatusBadRequest { + t.Fatalf("status = %d, want 400 body=%s", response.Code, response.Body.String()) + } } func TestBuildRecoveryRollupPayloadExposesCanonicalItemRefField(t *testing.T) { diff --git a/internal/api/route_inventory_test.go b/internal/api/route_inventory_test.go index 33027d056..2b65e4c85 100644 --- a/internal/api/route_inventory_test.go +++ b/internal/api/route_inventory_test.go @@ -409,6 +409,7 @@ var allRouteAllowlist = []string{ "/api/recovery/series", "/api/recovery/facets", "/api/recovery/rollups", + "/api/recovery/postures", "/api/resources", "/api/resources/storage-incidents", "/api/resources/storage-summary", diff --git a/internal/api/router_routes_monitoring.go b/internal/api/router_routes_monitoring.go index c5bdff68b..6fd9b4e69 100644 --- a/internal/api/router_routes_monitoring.go +++ b/internal/api/router_routes_monitoring.go @@ -29,6 +29,7 @@ func (r *Router) registerMonitoringResourceRoutes( r.mux.HandleFunc("/api/recovery/series", RequireAuth(r.config, RequireScope(config.ScopeMonitoringRead, r.recoveryHandlers.HandleListSeries))) r.mux.HandleFunc("/api/recovery/facets", RequireAuth(r.config, RequireScope(config.ScopeMonitoringRead, r.recoveryHandlers.HandleListFacets))) r.mux.HandleFunc("/api/recovery/rollups", RequireAuth(r.config, RequireScope(config.ScopeMonitoringRead, r.recoveryHandlers.HandleListRollups))) + r.mux.HandleFunc("/api/recovery/postures", RequireAuth(r.config, RequireScope(config.ScopeMonitoringRead, r.recoveryHandlers.HandleListProtectionPostures))) r.registerReplicationRoutes() r.registerPVEBackupsRoutes() r.registerPBSBackupsRoutes() diff --git a/internal/monitoring/monitor_backups.go b/internal/monitoring/monitor_backups.go index ea8e81a12..0354fdaf6 100644 --- a/internal/monitoring/monitor_backups.go +++ b/internal/monitoring/monitor_backups.go @@ -1582,8 +1582,30 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien Int("count", len(allBackups)). Msg("PBS backups fetched") + protectionObservedAt := time.Now().UTC() + protectionObservation, protectionObservationErr := + buildPBSProtectionProviderObservation( + instanceName, + datastoreCount, + datastoreFetches, + datastoreErrors, + datastoreTerminalFailures, + protectionObservedAt, + ) + if protectionObservationErr != nil { + log.Warn(). + Err(protectionObservationErr). + Str("instance", instanceName). + Msg("Failed to build PBS protection provider observation") + } + // Decide whether to keep existing backups when all queries failed if shouldPreservePBSBackupsWithTerminal(datastoreCount, datastoreFetches, datastoreTerminalFailures) { + if protectionObservationErr == nil { + m.ingestProtectionProviderObservationsAsync( + []recovery.ProtectionProviderObservation{protectionObservation}, + ) + } log.Warn(). Str("instance", instanceName). Int("datastores", datastoreCount). @@ -1600,8 +1622,30 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien // Best-effort ingestion into recovery store (for rollups / unified backups UX). candidates := buildPBSGuestCandidates(m.GetUnifiedReadStateOrSnapshot()) - m.ingestAndReconcileRecoveryPointsAsync( - proxmoxrecoverymapper.FromPBSBackups(allBackups, candidates), + points, evidenceErr := proxmoxrecoverymapper.FromPBSBackupsWithEvidence( + allBackups, + candidates, + protectionObservedAt, + ) + if evidenceErr != nil { + log.Warn(). + Err(evidenceErr). + Str("instance", instanceName). + Msg("Failed to attach PBS recovery evidence; preserving existing points") + if protectionObservationErr == nil { + m.ingestProtectionProviderObservationsAsync( + []recovery.ProtectionProviderObservation{protectionObservation}, + ) + } + return + } + observations := []recovery.ProtectionProviderObservation{} + if protectionObservationErr == nil { + observations = append(observations, protectionObservation) + } + m.ingestAndReconcileRecoveryPointsWithObservationsAsync( + points, + observations, recoveryReconcileScope{ provider: string(recovery.ProviderProxmoxPBS), idPrefix: "pbs-backup:", diff --git a/internal/monitoring/pbs_protection_observation.go b/internal/monitoring/pbs_protection_observation.go new file mode 100644 index 000000000..13f76a781 --- /dev/null +++ b/internal/monitoring/pbs_protection_observation.go @@ -0,0 +1,86 @@ +package monitoring + +import ( + "fmt" + "strings" + "time" + + "github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust" + "github.com/rcourtman/pulse-go-rewrite/internal/recovery" +) + +func buildPBSProtectionProviderObservation( + instanceName string, + datastoreCount int, + datastoreFetches int, + datastoreErrors int, + datastoreTerminalFailures int, + observedAt time.Time, +) (recovery.ProtectionProviderObservation, error) { + instanceName = strings.TrimSpace(instanceName) + if instanceName == "" { + return recovery.ProtectionProviderObservation{}, fmt.Errorf( + "PBS protection observation requires an instance", + ) + } + if observedAt.IsZero() { + return recovery.ProtectionProviderObservation{}, fmt.Errorf( + "PBS protection observation requires an observation time", + ) + } + + jobState := recovery.OutcomeSuccess + historyCompleteness := recovery.ProtectionHistoryComplete + permissions := operationaltrust.EvidencePermissionsSufficient + var reason *operationaltrust.EvidenceReason + + switch { + case datastoreErrors == 0: + // A successful empty enumeration is complete evidence that the + // connection currently exposes no backup history. + case datastoreFetches > 0: + jobState = recovery.OutcomeWarning + historyCompleteness = recovery.ProtectionHistoryPartial + permissions = operationaltrust.EvidencePermissionsUnknown + reason = &operationaltrust.EvidenceReason{ + Code: "pbs_partial_enumeration", + Message: "Some PBS datastore or namespace history could not be enumerated.", + } + if datastoreTerminalFailures > 0 { + permissions = operationaltrust.EvidencePermissionsPartial + reason = &operationaltrust.EvidenceReason{ + Code: "pbs_partial_provider_access", + Message: "PBS authorized only part of the configured backup-history scope.", + } + } + default: + jobState = recovery.OutcomeFailed + historyCompleteness = recovery.ProtectionHistoryUnavailable + permissions = operationaltrust.EvidencePermissionsUnknown + reason = &operationaltrust.EvidenceReason{ + Code: "pbs_collection_unavailable", + Message: "PBS backup history could not be collected; retained points may be stale.", + } + if datastoreCount > 0 && + datastoreTerminalFailures >= datastoreCount { + permissions = operationaltrust.EvidencePermissionsDenied + reason = &operationaltrust.EvidenceReason{ + Code: "pbs_provider_access_denied", + Message: "PBS rejected every configured datastore history request.", + } + } + } + + return recovery.NewProtectionProviderObservation( + recovery.ProviderProxmoxPBS, + "pbs-backup-enumeration", + instanceName, + jobState, + historyCompleteness, + permissions, + true, + observedAt, + observedAt, + reason, + ) +} diff --git a/internal/monitoring/pbs_protection_observation_test.go b/internal/monitoring/pbs_protection_observation_test.go new file mode 100644 index 000000000..73e6253ae --- /dev/null +++ b/internal/monitoring/pbs_protection_observation_test.go @@ -0,0 +1,117 @@ +package monitoring + +import ( + "testing" + "time" + + "github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust" + "github.com/rcourtman/pulse-go-rewrite/internal/recovery" +) + +func TestBuildPBSProtectionProviderObservationMapsEvidenceQuality(t *testing.T) { + t.Parallel() + + now := time.Date(2026, 7, 19, 8, 0, 0, 0, time.UTC) + tests := []struct { + name string + datastores int + fetches int + errors int + terminalFailures int + wantJob recovery.Outcome + wantHistory recovery.ProtectionHistoryCompleteness + wantPermissions operationaltrust.EvidencePermissions + wantReason string + }{ + { + name: "complete enumeration", + datastores: 2, + fetches: 2, + wantJob: recovery.OutcomeSuccess, + wantHistory: recovery.ProtectionHistoryComplete, + wantPermissions: operationaltrust.EvidencePermissionsSufficient, + }, + { + name: "partial transient enumeration", + datastores: 2, + fetches: 1, + errors: 1, + wantJob: recovery.OutcomeWarning, + wantHistory: recovery.ProtectionHistoryPartial, + wantPermissions: operationaltrust.EvidencePermissionsUnknown, + wantReason: "pbs_partial_enumeration", + }, + { + name: "partial provider access", + datastores: 2, + fetches: 1, + errors: 1, + terminalFailures: 1, + wantJob: recovery.OutcomeWarning, + wantHistory: recovery.ProtectionHistoryPartial, + wantPermissions: operationaltrust.EvidencePermissionsPartial, + wantReason: "pbs_partial_provider_access", + }, + { + name: "provider timeout", + datastores: 2, + errors: 2, + wantJob: recovery.OutcomeFailed, + wantHistory: recovery.ProtectionHistoryUnavailable, + wantPermissions: operationaltrust.EvidencePermissionsUnknown, + wantReason: "pbs_collection_unavailable", + }, + { + name: "provider denies every datastore", + datastores: 2, + errors: 2, + terminalFailures: 2, + wantJob: recovery.OutcomeFailed, + wantHistory: recovery.ProtectionHistoryUnavailable, + wantPermissions: operationaltrust.EvidencePermissionsDenied, + wantReason: "pbs_provider_access_denied", + }, + } + + for _, test := range tests { + test := test + t.Run(test.name, func(t *testing.T) { + t.Parallel() + got, err := buildPBSProtectionProviderObservation( + "pbs-main", + test.datastores, + test.fetches, + test.errors, + test.terminalFailures, + now, + ) + if err != nil { + t.Fatalf("buildPBSProtectionProviderObservation() error = %v", err) + } + if got.JobState != test.wantJob { + t.Fatalf("job state = %q, want %q", got.JobState, test.wantJob) + } + if got.HistoryCompleteness != test.wantHistory { + t.Fatalf( + "history completeness = %q, want %q", + got.HistoryCompleteness, + test.wantHistory, + ) + } + if got.Permissions != test.wantPermissions { + t.Fatalf("permissions = %q, want %q", got.Permissions, test.wantPermissions) + } + if test.wantReason == "" { + if got.Evidence.Reason != nil { + t.Fatalf("reason = %#v, want nil", got.Evidence.Reason) + } + } else if got.Evidence.Reason == nil || + got.Evidence.Reason.Code != test.wantReason { + t.Fatalf("reason = %#v, want %q", got.Evidence.Reason, test.wantReason) + } + if err := got.Validate(); err != nil { + t.Fatalf("Validate() error = %v", err) + } + }) + } +} diff --git a/internal/monitoring/recovery_ingest.go b/internal/monitoring/recovery_ingest.go index 5bfd2f386..3a3cb0665 100644 --- a/internal/monitoring/recovery_ingest.go +++ b/internal/monitoring/recovery_ingest.go @@ -24,8 +24,9 @@ type recoveryReconcileScope struct { // recoveryIngestBatch is one poll cycle's worth of recovery points, plus an // optional reconcile scope when the points are a complete enumeration. type recoveryIngestBatch struct { - points []recovery.RecoveryPoint - reconcile *recoveryReconcileScope + points []recovery.RecoveryPoint + observations []recovery.ProtectionProviderObservation + reconcile *recoveryReconcileScope } func (m *Monitor) ingestRecoveryPointsAsync(points []recovery.RecoveryPoint) { @@ -39,8 +40,29 @@ func (m *Monitor) ingestAndReconcileRecoveryPointsAsync(points []recovery.Recove m.enqueueRecoveryIngest(recoveryIngestBatch{points: points, reconcile: &scope}) } +func (m *Monitor) ingestAndReconcileRecoveryPointsWithObservationsAsync( + points []recovery.RecoveryPoint, + observations []recovery.ProtectionProviderObservation, + scope recoveryReconcileScope, +) { + m.enqueueRecoveryIngest(recoveryIngestBatch{ + points: points, + observations: observations, + reconcile: &scope, + }) +} + +func (m *Monitor) ingestProtectionProviderObservationsAsync( + observations []recovery.ProtectionProviderObservation, +) { + m.enqueueRecoveryIngest(recoveryIngestBatch{observations: observations}) +} + func (m *Monitor) enqueueRecoveryIngest(batch recoveryIngestBatch) { - if m == nil || (len(batch.points) == 0 && batch.reconcile == nil) { + if m == nil || + (len(batch.points) == 0 && + len(batch.observations) == 0 && + batch.reconcile == nil) { return } @@ -53,6 +75,7 @@ func (m *Monitor) enqueueRecoveryIngest(batch recoveryIngestBatch) { m.recoveryIngestMu.Unlock() log.Debug(). Int("points", len(batch.points)). + Int("provider_observations", len(batch.observations)). Msg("Queued recovery point ingest behind active batch") return } @@ -88,7 +111,10 @@ func (m *Monitor) runRecoveryPointIngestLoop(batches []recoveryIngestBatch) { } func (m *Monitor) ingestRecoveryPointsBestEffort(ctx context.Context, batch recoveryIngestBatch) { - if m == nil || (len(batch.points) == 0 && batch.reconcile == nil) { + if m == nil || + (len(batch.points) == 0 && + len(batch.observations) == 0 && + batch.reconcile == nil) { return } @@ -110,6 +136,22 @@ func (m *Monitor) ingestRecoveryPointsBestEffort(ctx context.Context, batch reco return } + // Collection-wide evidence is the authority for how much provider history + // Pulse could actually see. Persist it before any point mutation so a large + // point batch, refresh timeout, or reconciliation failure cannot leave + // retained artifacts looking more trustworthy than the poll that produced + // them. + if err := store.UpsertProtectionProviderObservations( + ctx, + batch.observations, + ); err != nil { + log.Warn(). + Err(err). + Str("org_id", orgID). + Int("provider_observations", len(batch.observations)). + Msg("Failed to upsert provider protection observations") + return + } if err := store.UpsertPoints(ctx, batch.points); err != nil { log.Warn().Err(err).Str("org_id", orgID).Int("points", len(batch.points)).Msg("Failed to upsert recovery points from backup polling") // Do not reconcile against a batch that failed to land; deleting on diff --git a/internal/monitoring/recovery_ingest_test.go b/internal/monitoring/recovery_ingest_test.go index b3997b501..c8d54944f 100644 --- a/internal/monitoring/recovery_ingest_test.go +++ b/internal/monitoring/recovery_ingest_test.go @@ -6,6 +6,7 @@ import ( "time" "github.com/rcourtman/pulse-go-rewrite/internal/config" + "github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust" "github.com/rcourtman/pulse-go-rewrite/internal/recovery" recoverymanager "github.com/rcourtman/pulse-go-rewrite/internal/recovery/manager" ) @@ -162,3 +163,107 @@ func TestIngestRecoveryPointsBestEffortSkipsReconcileWhenUpsertFails(t *testing. t.Fatal("reconcile must be skipped when the upsert fails; vzdump-101 was deleted") } } + +func TestIngestRecoveryPointsBestEffortPersistsProviderObservationAndPosture(t *testing.T) { + t.Parallel() + + m, manager := recoveryIngestTestMonitor(t) + now := time.Now().UTC().Truncate(time.Millisecond) + point := recoveryIngestTestPoint("pve-backup:pve1-vzdump-100", "pve1", now) + point.SubjectResourceID = "resource:vm-100" + observation, err := recovery.NewProtectionProviderObservation( + recovery.ProviderProxmoxPVE, + "pve-backup-enumeration", + "pve1", + recovery.OutcomeSuccess, + recovery.ProtectionHistoryComplete, + operationaltrust.EvidencePermissionsSufficient, + false, + now, + now, + nil, + ) + if err != nil { + t.Fatalf("NewProtectionProviderObservation() error = %v", err) + } + + m.ingestRecoveryPointsBestEffort(context.Background(), recoveryIngestBatch{ + points: []recovery.RecoveryPoint{point}, + observations: []recovery.ProtectionProviderObservation{ + observation, + }, + }) + + store, err := manager.StoreForOrg("default") + if err != nil { + t.Fatalf("StoreForOrg(default): %v", err) + } + postures, _, err := store.ListProtectionPostures( + context.Background(), + recovery.ProtectionPostureQuery{ + SubjectResourceIDs: []string{"resource:vm-100"}, + }, + ) + if err != nil { + t.Fatalf("ListProtectionPostures() error = %v", err) + } + if len(postures) != 1 || + postures[0].State != recovery.ProtectionStateProtected { + t.Fatalf("postures = %#v, want one protected posture", postures) + } +} + +func TestIngestRecoveryPointsBestEffortPersistsProviderObservationBeforePointFailure(t *testing.T) { + t.Parallel() + + m, manager := recoveryIngestTestMonitor(t) + store, err := manager.StoreForOrg("default") + if err != nil { + t.Fatalf("StoreForOrg(default): %v", err) + } + now := time.Now().UTC().Truncate(time.Millisecond) + existing := recoveryIngestTestPoint("pve-backup:pve1-vzdump-100", "pve1", now) + existing.SubjectResourceID = "resource:vm-100" + if err := store.UpsertPoints(context.Background(), []recovery.RecoveryPoint{existing}); err != nil { + t.Fatalf("UpsertPoints(existing): %v", err) + } + observation, err := recovery.NewProtectionProviderObservation( + recovery.ProviderProxmoxPVE, + "pve-backup-enumeration", + "pve1", + recovery.OutcomeSuccess, + recovery.ProtectionHistoryComplete, + operationaltrust.EvidencePermissionsSufficient, + false, + now, + now, + nil, + ) + if err != nil { + t.Fatalf("NewProtectionProviderObservation() error = %v", err) + } + + invalidPoint := existing + invalidPoint.ID = "" + m.ingestRecoveryPointsBestEffort(context.Background(), recoveryIngestBatch{ + points: []recovery.RecoveryPoint{invalidPoint}, + observations: []recovery.ProtectionProviderObservation{observation}, + }) + + postures, _, err := store.ListProtectionPostures( + context.Background(), + recovery.ProtectionPostureQuery{ + SubjectResourceIDs: []string{"resource:vm-100"}, + }, + ) + if err != nil { + t.Fatalf("ListProtectionPostures() error = %v", err) + } + if len(postures) != 1 || + postures[0].State != recovery.ProtectionStateProtected { + t.Fatalf( + "postures = %#v, want provider observation to survive point failure", + postures, + ) + } +} diff --git a/internal/recovery/mapper/proxmox/mapper.go b/internal/recovery/mapper/proxmox/mapper.go index 23f31e7e5..26f67aa0e 100644 --- a/internal/recovery/mapper/proxmox/mapper.go +++ b/internal/recovery/mapper/proxmox/mapper.go @@ -461,6 +461,38 @@ func FromPBSBackups(backups []models.PBSBackup, candidatesByKey map[string][]Gue return out } +// FromPBSBackupsWithEvidence is the operational-trust adapter for direct PBS +// inventory. It preserves the supported RecoveryPoint payload while attaching +// explicit provider scope and typed evidence to every successfully enumerated +// backup. Collection-wide completeness and permission state are recorded +// separately as a ProtectionProviderObservation by the polling owner. +func FromPBSBackupsWithEvidence( + backups []models.PBSBackup, + candidatesByKey map[string][]GuestCandidate, + ingestedAt time.Time, +) ([]recovery.RecoveryPoint, error) { + points := FromPBSBackups(backups, candidatesByKey) + if len(points) == 0 { + return []recovery.RecoveryPoint{}, nil + } + if ingestedAt.IsZero() { + return nil, fmt.Errorf("PBS recovery evidence ingestion time is required") + } + for i := range points { + points[i].ProviderScope = recovery.ProviderScopeForPoint(points[i]) + evidence, err := recovery.NewRecoveryPointEvidence( + points[i], + "pbs-backup-inventory", + ingestedAt, + ) + if err != nil { + return nil, fmt.Errorf("map PBS backup %q evidence: %w", points[i].ID, err) + } + points[i].Evidence = evidence + } + return points, nil +} + func sizePtr(v int64) *int64 { if v <= 0 { return nil diff --git a/internal/recovery/mapper/proxmox/mapper_test.go b/internal/recovery/mapper/proxmox/mapper_test.go index 056311ba0..523a24ee5 100644 --- a/internal/recovery/mapper/proxmox/mapper_test.go +++ b/internal/recovery/mapper/proxmox/mapper_test.go @@ -403,3 +403,63 @@ func TestFromPBSBackups_DisambiguatesCandidatesByNamespace(t *testing.T) { t.Fatalf("SubjectRef = %#v, want linked debian-go guest", result[0].SubjectRef) } } + +func TestFromPBSBackupsWithEvidenceAddsProviderScopeAndCorrelation(t *testing.T) { + t.Parallel() + + backupTime := time.Date(2026, 7, 19, 6, 0, 0, 0, time.UTC) + ingestedAt := backupTime.Add(2 * time.Minute) + backups := []models.PBSBackup{ + { + ID: "pbs-main:store-a:vm/100/2026-07-19T06:00:00Z", + VMID: "100", + Instance: "pbs-main", + Datastore: "store-a", + BackupType: "vm", + BackupTime: backupTime, + Verified: true, + }, + } + candidates := map[string][]GuestCandidate{ + "vm:100": { + { + SourceID: "vm-100", + ResourceType: unifiedresources.ResourceTypeVM, + DisplayName: "database", + InstanceName: "pve-main", + NodeName: "pve-a", + VMID: 100, + }, + }, + } + + points, err := FromPBSBackupsWithEvidence(backups, candidates, ingestedAt) + if err != nil { + t.Fatalf("FromPBSBackupsWithEvidence() error = %v", err) + } + if len(points) != 1 { + t.Fatalf("points = %d, want 1", len(points)) + } + point := points[0] + if point.ProviderScope != "pbs-main" { + t.Fatalf("provider scope = %q, want pbs-main", point.ProviderScope) + } + if point.Evidence == nil { + t.Fatal("expected typed PBS evidence") + } + if point.Evidence.ObservedAt != backupTime { + t.Fatalf("observedAt = %v, want %v", point.Evidence.ObservedAt, backupTime) + } + if point.Evidence.Correlation == nil { + t.Fatal("expected auditable canonical guest correlation") + } + if point.Evidence.Correlation.CandidateCount != 1 { + t.Fatalf( + "candidate count = %d, want 1", + point.Evidence.Correlation.CandidateCount, + ) + } + if err := point.Evidence.Validate(); err != nil { + t.Fatalf("evidence Validate() error = %v", err) + } +} diff --git a/internal/recovery/model/posture.go b/internal/recovery/model/posture.go new file mode 100644 index 000000000..bbb6347db --- /dev/null +++ b/internal/recovery/model/posture.go @@ -0,0 +1,441 @@ +package model + +import ( + "errors" + "fmt" + "sort" + "strings" + "time" + + "github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust" +) + +type ProtectionState string + +const ( + ProtectionStateProtected ProtectionState = "protected" + ProtectionStateAttention ProtectionState = "attention" + ProtectionStateUnprotected ProtectionState = "unprotected" + ProtectionStateUnknown ProtectionState = "unknown" +) + +func (state ProtectionState) Valid() bool { + switch state { + case ProtectionStateProtected, + ProtectionStateAttention, + ProtectionStateUnprotected, + ProtectionStateUnknown: + return true + default: + return false + } +} + +type ProtectionFreshness string + +const ( + ProtectionFreshnessCurrent ProtectionFreshness = "current" + ProtectionFreshnessStale ProtectionFreshness = "stale" + ProtectionFreshnessUnknown ProtectionFreshness = "unknown" +) + +func (freshness ProtectionFreshness) Valid() bool { + switch freshness { + case ProtectionFreshnessCurrent, + ProtectionFreshnessStale, + ProtectionFreshnessUnknown: + return true + default: + return false + } +} + +type ProtectionVerification string + +const ( + ProtectionVerificationVerified ProtectionVerification = "verified" + ProtectionVerificationUnverified ProtectionVerification = "unverified" + ProtectionVerificationStale ProtectionVerification = "stale" + ProtectionVerificationUnknown ProtectionVerification = "unknown" +) + +func (verification ProtectionVerification) Valid() bool { + switch verification { + case ProtectionVerificationVerified, + ProtectionVerificationUnverified, + ProtectionVerificationStale, + ProtectionVerificationUnknown: + return true + default: + return false + } +} + +type ProtectionCoverage string + +const ( + ProtectionCoverageComplete ProtectionCoverage = "complete" + ProtectionCoveragePartial ProtectionCoverage = "partial" + ProtectionCoverageNone ProtectionCoverage = "none" + ProtectionCoverageUnknown ProtectionCoverage = "unknown" +) + +func (coverage ProtectionCoverage) Valid() bool { + switch coverage { + case ProtectionCoverageComplete, + ProtectionCoveragePartial, + ProtectionCoverageNone, + ProtectionCoverageUnknown: + return true + default: + return false + } +} + +type ProtectionHistoryCompleteness string + +const ( + ProtectionHistoryComplete ProtectionHistoryCompleteness = "complete" + ProtectionHistoryPartial ProtectionHistoryCompleteness = "partial" + ProtectionHistoryUnavailable ProtectionHistoryCompleteness = "unavailable" + ProtectionHistoryUnknown ProtectionHistoryCompleteness = "unknown" +) + +func (completeness ProtectionHistoryCompleteness) Valid() bool { + switch completeness { + case ProtectionHistoryComplete, + ProtectionHistoryPartial, + ProtectionHistoryUnavailable, + ProtectionHistoryUnknown: + return true + default: + return false + } +} + +type ProtectionProviderState struct { + Provider Provider `json:"provider"` + Source string `json:"source"` + Scope string `json:"scope"` + JobState Outcome `json:"jobState"` + HistoryCompleteness ProtectionHistoryCompleteness `json:"historyCompleteness"` + Permissions operationaltrust.EvidencePermissions `json:"permissions"` + LastAttemptAt *time.Time `json:"lastAttemptAt,omitempty"` + LastSuccessAt *time.Time `json:"lastSuccessAt,omitempty"` + LastVerifiedAt *time.Time `json:"lastVerifiedAt,omitempty"` + EvidenceIDs []string `json:"evidenceIds"` + VerificationExpected bool `json:"verificationExpected,omitempty"` +} + +func (state ProtectionProviderState) Clone() ProtectionProviderState { + clone := state + clone.LastAttemptAt = cloneTime(state.LastAttemptAt) + clone.LastSuccessAt = cloneTime(state.LastSuccessAt) + clone.LastVerifiedAt = cloneTime(state.LastVerifiedAt) + clone.EvidenceIDs = append([]string(nil), state.EvidenceIDs...) + return clone +} + +func (state ProtectionProviderState) Validate() error { + if strings.TrimSpace(string(state.Provider)) == "" { + return errors.New("protection provider is required") + } + if strings.TrimSpace(state.Source) == "" { + return errors.New("protection provider source is required") + } + if strings.TrimSpace(state.Scope) == "" { + return errors.New("protection provider scope is required") + } + if !validOutcome(state.JobState) { + return fmt.Errorf("protection provider job state %q is invalid", state.JobState) + } + if !state.HistoryCompleteness.Valid() { + return fmt.Errorf( + "protection provider history completeness %q is invalid", + state.HistoryCompleteness, + ) + } + if !validEvidencePermissions(state.Permissions) { + return fmt.Errorf("protection provider permissions %q are invalid", state.Permissions) + } + if !sortedUniqueStrings(state.EvidenceIDs) { + return errors.New("protection provider evidence ids must be sorted and unique") + } + return nil +} + +type ProtectionPosture struct { + SubjectResourceID string `json:"subjectResourceId"` + State ProtectionState `json:"state"` + LastAttemptAt *time.Time `json:"lastAttemptAt,omitempty"` + LastSuccessfulPointAt *time.Time `json:"lastSuccessfulPointAt,omitempty"` + LastVerifiedAt *time.Time `json:"lastVerifiedAt,omitempty"` + Freshness ProtectionFreshness `json:"freshness"` + Verification ProtectionVerification `json:"verification"` + Coverage ProtectionCoverage `json:"coverage"` + ProviderStates []ProtectionProviderState `json:"providerStates"` + RepositoryResourceIDs []string `json:"repositoryResourceIds"` + EvidenceIDs []string `json:"evidenceIds"` + Explanation string `json:"explanation"` + EvaluatedAt time.Time `json:"evaluatedAt"` +} + +func (posture ProtectionPosture) Clone() ProtectionPosture { + clone := posture + clone.LastAttemptAt = cloneTime(posture.LastAttemptAt) + clone.LastSuccessfulPointAt = cloneTime(posture.LastSuccessfulPointAt) + clone.LastVerifiedAt = cloneTime(posture.LastVerifiedAt) + clone.ProviderStates = make([]ProtectionProviderState, len(posture.ProviderStates)) + for i := range posture.ProviderStates { + clone.ProviderStates[i] = posture.ProviderStates[i].Clone() + } + clone.RepositoryResourceIDs = append([]string(nil), posture.RepositoryResourceIDs...) + clone.EvidenceIDs = append([]string(nil), posture.EvidenceIDs...) + return clone +} + +func (posture ProtectionPosture) Validate() error { + if strings.TrimSpace(posture.SubjectResourceID) == "" { + return errors.New("protection posture subject resource id is required") + } + if !posture.State.Valid() { + return fmt.Errorf("protection posture state %q is invalid", posture.State) + } + if !posture.Freshness.Valid() { + return fmt.Errorf("protection posture freshness %q is invalid", posture.Freshness) + } + if !posture.Verification.Valid() { + return fmt.Errorf("protection posture verification %q is invalid", posture.Verification) + } + if !posture.Coverage.Valid() { + return fmt.Errorf("protection posture coverage %q is invalid", posture.Coverage) + } + if posture.EvaluatedAt.IsZero() { + return errors.New("protection posture evaluation time is required") + } + if strings.TrimSpace(posture.Explanation) == "" { + return errors.New("protection posture explanation is required") + } + if !sortedUniqueStrings(posture.RepositoryResourceIDs) { + return errors.New("protection repository resource ids must be sorted and unique") + } + if !sortedUniqueStrings(posture.EvidenceIDs) { + return errors.New("protection evidence ids must be sorted and unique") + } + for i := range posture.ProviderStates { + if err := posture.ProviderStates[i].Validate(); err != nil { + return fmt.Errorf("protection provider state %d: %w", i, err) + } + if i > 0 && compareProviderStates( + posture.ProviderStates[i-1], + posture.ProviderStates[i], + ) >= 0 { + return errors.New("protection provider states must be sorted and unique") + } + } + return nil +} + +type ProtectionProviderObservation struct { + ID string `json:"id"` + Provider Provider `json:"provider"` + Source string `json:"source"` + Scope string `json:"scope"` + JobState Outcome `json:"jobState"` + HistoryCompleteness ProtectionHistoryCompleteness `json:"historyCompleteness"` + Permissions operationaltrust.EvidencePermissions `json:"permissions"` + VerificationExpected bool `json:"verificationExpected,omitempty"` + ObservedAt time.Time `json:"observedAt"` + IngestedAt time.Time `json:"ingestedAt"` + Evidence operationaltrust.EvidenceEnvelope `json:"evidence"` +} + +func (observation ProtectionProviderObservation) Clone() ProtectionProviderObservation { + clone := observation + clone.Evidence = observation.Evidence.Clone() + return clone +} + +func (observation ProtectionProviderObservation) Validate() error { + if strings.TrimSpace(observation.ID) == "" { + return errors.New("protection provider observation id is required") + } + if strings.TrimSpace(string(observation.Provider)) == "" { + return errors.New("protection provider observation provider is required") + } + if strings.TrimSpace(observation.Source) == "" { + return errors.New("protection provider observation source is required") + } + if strings.TrimSpace(observation.Scope) == "" { + return errors.New("protection provider observation scope is required") + } + if !validOutcome(observation.JobState) { + return fmt.Errorf( + "protection provider observation job state %q is invalid", + observation.JobState, + ) + } + if !observation.HistoryCompleteness.Valid() { + return fmt.Errorf( + "protection provider observation history completeness %q is invalid", + observation.HistoryCompleteness, + ) + } + if !validEvidencePermissions(observation.Permissions) { + return fmt.Errorf( + "protection provider observation permissions %q are invalid", + observation.Permissions, + ) + } + if observation.ObservedAt.IsZero() || observation.IngestedAt.IsZero() { + return errors.New("protection provider observation times are required") + } + if !observation.Evidence.ObservedAt.Equal(observation.ObservedAt) { + return errors.New("provider observation and evidence observation times must match") + } + if !observation.Evidence.IngestedAt.Equal(observation.IngestedAt) { + return errors.New("provider observation and evidence ingestion times must match") + } + if observation.Evidence.ID != observation.ID { + return errors.New("provider observation id must match its evidence id") + } + if err := observation.Evidence.Validate(); err != nil { + return fmt.Errorf("provider observation evidence: %w", err) + } + return nil +} + +type ProtectionPosturePolicy struct { + FreshnessWindow time.Duration `json:"-"` + VerificationWindow time.Duration `json:"-"` + RequireVerification bool `json:"requireVerification"` +} + +func (policy ProtectionPosturePolicy) Validate() error { + if policy.FreshnessWindow <= 0 { + return errors.New("protection freshness window must be positive") + } + if policy.VerificationWindow <= 0 { + return errors.New("protection verification window must be positive") + } + return nil +} + +type ProtectionPosturePolicyPayload struct { + FreshnessWindowSeconds int64 `json:"freshnessWindowSeconds"` + VerificationWindowSeconds int64 `json:"verificationWindowSeconds"` + RequireVerification bool `json:"requireVerification"` +} + +func (policy ProtectionPosturePolicy) Payload() ProtectionPosturePolicyPayload { + return ProtectionPosturePolicyPayload{ + FreshnessWindowSeconds: int64(policy.FreshnessWindow / time.Second), + VerificationWindowSeconds: int64(policy.VerificationWindow / time.Second), + RequireVerification: policy.RequireVerification, + } +} + +type ProtectionPostureQuery struct { + SubjectResourceIDs []string + State ProtectionState + Page int + Limit int +} + +type ProtectionProviderSummary struct { + Provider Provider + Source string + Scope string + JobState Outcome + HistoryCompleteness ProtectionHistoryCompleteness + Permissions operationaltrust.EvidencePermissions + VerificationExpected bool + LastAttemptAt *time.Time + LastSuccessAt *time.Time + LastVerifiedAt *time.Time + BackupPointCount int + SnapshotPointCount int + RepositoryResourceIDs []string + EvidenceIDs []string +} + +func (summary ProtectionProviderSummary) normalize() ProtectionProviderSummary { + summary.Provider = Provider(strings.TrimSpace(string(summary.Provider))) + summary.Source = strings.TrimSpace(summary.Source) + summary.Scope = strings.TrimSpace(summary.Scope) + summary.RepositoryResourceIDs = normalizeSortedStrings(summary.RepositoryResourceIDs) + summary.EvidenceIDs = normalizeSortedStrings(summary.EvidenceIDs) + if !validOutcome(summary.JobState) { + summary.JobState = OutcomeUnknown + } + if !summary.HistoryCompleteness.Valid() { + summary.HistoryCompleteness = ProtectionHistoryUnknown + } + if !validEvidencePermissions(summary.Permissions) { + summary.Permissions = operationaltrust.EvidencePermissionsUnknown + } + return summary +} + +func cloneTime(value *time.Time) *time.Time { + if value == nil { + return nil + } + cloned := value.UTC() + return &cloned +} + +func validOutcome(outcome Outcome) bool { + switch outcome { + case OutcomeSuccess, OutcomeWarning, OutcomeFailed, OutcomeRunning, OutcomeUnknown: + return true + default: + return false + } +} + +func validEvidencePermissions(value operationaltrust.EvidencePermissions) bool { + switch value { + case operationaltrust.EvidencePermissionsSufficient, + operationaltrust.EvidencePermissionsPartial, + operationaltrust.EvidencePermissionsDenied, + operationaltrust.EvidencePermissionsUnknown: + return true + default: + return false + } +} + +func normalizeSortedStrings(values []string) []string { + unique := make(map[string]struct{}, len(values)) + for _, value := range values { + value = strings.TrimSpace(value) + if value != "" { + unique[value] = struct{}{} + } + } + out := make([]string, 0, len(unique)) + for value := range unique { + out = append(out, value) + } + sort.Strings(out) + return out +} + +func sortedUniqueStrings(values []string) bool { + for i, value := range values { + if strings.TrimSpace(value) == "" { + return false + } + if i > 0 && values[i-1] >= value { + return false + } + } + return true +} + +func compareProviderStates(a, b ProtectionProviderState) int { + aKey := strings.Join([]string{string(a.Provider), a.Scope, a.Source}, "\x00") + bKey := strings.Join([]string{string(b.Provider), b.Scope, b.Source}, "\x00") + return strings.Compare(aKey, bKey) +} diff --git a/internal/recovery/model/types.go b/internal/recovery/model/types.go index 80ac6c8c9..d49c15bf0 100644 --- a/internal/recovery/model/types.go +++ b/internal/recovery/model/types.go @@ -1,6 +1,10 @@ package model -import "time" +import ( + "time" + + "github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust" +) // Provider identifies the system that produced a recovery point. // This is intentionally a string (not an enum) to keep forward-compatibility with new platforms. @@ -83,6 +87,14 @@ type RecoveryPoint struct { // Provider-specific details for drill-down (kept small). Details map[string]any `json:"details,omitempty"` + // ProviderScope is the stable provider-local collection scope used to + // correlate this point with provider history and permission observations. + ProviderScope string `json:"providerScope,omitempty"` + + // Evidence carries the typed provenance for this recovery point. Older + // persisted rows may omit it and are treated as unknown-quality evidence. + Evidence *operationaltrust.EvidenceEnvelope `json:"evidence,omitempty"` + // Display contains normalized, provider-agnostic fields intended for UIs. // These fields are derived at ingest time and/or backfilled in the store. Display *RecoveryPointDisplay `json:"display,omitempty"` diff --git a/internal/recovery/posture.go b/internal/recovery/posture.go new file mode 100644 index 000000000..3d8cb64aa --- /dev/null +++ b/internal/recovery/posture.go @@ -0,0 +1,673 @@ +package recovery + +import ( + "fmt" + "sort" + "strings" + "time" + + "github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust" +) + +const ( + DefaultProtectionFreshnessWindow = 7 * 24 * time.Hour + DefaultProtectionVerificationWindow = 7 * 24 * time.Hour +) + +var DefaultProtectionPosturePolicy = ProtectionPosturePolicy{ + FreshnessWindow: DefaultProtectionFreshnessWindow, + VerificationWindow: DefaultProtectionVerificationWindow, + RequireVerification: false, +} + +func ProviderScopeForPoint(point RecoveryPoint) string { + if scope := strings.TrimSpace(point.ProviderScope); scope != "" { + return scope + } + switch point.Provider { + case ProviderProxmoxPBS: + if point.RepositoryRef != nil { + if scope := strings.TrimSpace(point.RepositoryRef.Namespace); scope != "" { + return scope + } + } + case ProviderProxmoxPVE: + if scope := recoveryDetailString(point, "instance"); scope != "" { + return scope + } + case ProviderKubernetes: + if scope := recoveryDetailString(point, "k8sClusterId"); scope != "" { + return scope + } + case ProviderTrueNAS: + if scope := recoveryDetailString(point, "connectionId"); scope != "" { + return scope + } + } + if point.SubjectRef != nil { + if scope := strings.TrimSpace(point.SubjectRef.Namespace); scope != "" { + return scope + } + } + return "provider-default" +} + +func NewRecoveryPointEvidence( + point RecoveryPoint, + collector string, + ingestedAt time.Time, +) (*operationaltrust.EvidenceEnvelope, error) { + collector = strings.TrimSpace(collector) + if collector == "" { + return nil, fmt.Errorf("recovery evidence collector is required") + } + observedAt := recoveryPointObservedAt(point) + if observedAt.IsZero() { + return nil, fmt.Errorf("recovery point observation time is required") + } + if ingestedAt.IsZero() { + return nil, fmt.Errorf("recovery point ingestion time is required") + } + + scope := ProviderScopeForPoint(point) + source := operationaltrust.EvidenceSource{ + Provider: string(point.Provider), + Collector: collector, + Instance: scope, + } + subject := operationaltrust.EvidenceSubject{} + if resourceID := strings.TrimSpace(point.SubjectResourceID); resourceID != "" { + subject.ResourceID = resourceID + } else { + subject.ProviderRef = SubjectKeyForPoint(point) + subject.ProviderScope = scope + } + evidenceID, err := operationaltrust.NewEvidenceID( + source, + subject, + observedAt, + point.ID, + ) + if err != nil { + return nil, err + } + + envelope := &operationaltrust.EvidenceEnvelope{ + ID: evidenceID, + Source: source, + Subject: subject, + ObservedAt: observedAt, + IngestedAt: ingestedAt.UTC(), + Completeness: operationaltrust.EvidenceComplete, + Confidence: operationaltrust.EvidenceConfirmed, + Permissions: operationaltrust.EvidencePermissionsSufficient, + PayloadRef: &operationaltrust.EvidencePayloadRef{ + Kind: "recovery-point", + ID: strings.TrimSpace(point.ID), + }, + } + if subject.ResourceID != "" && point.Provider == ProviderProxmoxPBS { + index := DeriveIndex(point) + matched := map[string]string{ + "providerScope": scope, + } + if value := strings.TrimSpace(index.ItemType); value != "" { + matched["itemType"] = value + } + if value := strings.TrimSpace(index.EntityIDLabel); value != "" { + matched["entityId"] = value + } + envelope.Confidence = operationaltrust.EvidenceInferred + envelope.Reason = &operationaltrust.EvidenceReason{ + Code: "provider_identity_correlation", + Message: "The PBS subject was linked through a unique provider-scoped guest match.", + } + envelope.Correlation = &operationaltrust.IdentityCorrelation{ + Rule: "provider_scoped_guest_identity", + MatchedFields: matched, + CandidateCount: 1, + } + } + if err := envelope.Validate(); err != nil { + return nil, err + } + return envelope, nil +} + +func NewProtectionProviderObservation( + provider Provider, + source string, + scope string, + jobState Outcome, + historyCompleteness ProtectionHistoryCompleteness, + permissions operationaltrust.EvidencePermissions, + verificationExpected bool, + observedAt time.Time, + ingestedAt time.Time, + reason *operationaltrust.EvidenceReason, +) (ProtectionProviderObservation, error) { + source = strings.TrimSpace(source) + scope = strings.TrimSpace(scope) + evidenceSource := operationaltrust.EvidenceSource{ + Provider: strings.TrimSpace(string(provider)), + Collector: source, + Instance: scope, + } + evidenceSubject := operationaltrust.EvidenceSubject{ + ProviderRef: strings.TrimSpace(string(provider)) + ":" + scope, + ProviderScope: scope, + } + evidenceID, err := operationaltrust.NewEvidenceID( + evidenceSource, + evidenceSubject, + observedAt, + fmt.Sprintf( + "%s:%s:%s:%s", + jobState, + historyCompleteness, + permissions, + scope, + ), + ) + if err != nil { + return ProtectionProviderObservation{}, err + } + + completeness := operationaltrust.EvidenceUnavailable + switch historyCompleteness { + case ProtectionHistoryComplete: + completeness = operationaltrust.EvidenceComplete + case ProtectionHistoryPartial: + completeness = operationaltrust.EvidencePartial + } + confidence := operationaltrust.EvidenceUnknown + if historyCompleteness == ProtectionHistoryComplete && + permissions == operationaltrust.EvidencePermissionsSufficient { + confidence = operationaltrust.EvidenceConfirmed + } + if (completeness != operationaltrust.EvidenceComplete || + confidence != operationaltrust.EvidenceConfirmed || + permissions != operationaltrust.EvidencePermissionsSufficient) && + reason == nil { + reason = &operationaltrust.EvidenceReason{ + Code: "provider_history_limited", + Message: "Provider history could not support a complete protection assertion.", + } + } + + observation := ProtectionProviderObservation{ + ID: evidenceID, + Provider: provider, + Source: source, + Scope: scope, + JobState: jobState, + HistoryCompleteness: historyCompleteness, + Permissions: permissions, + VerificationExpected: verificationExpected, + ObservedAt: observedAt.UTC(), + IngestedAt: ingestedAt.UTC(), + Evidence: operationaltrust.EvidenceEnvelope{ + ID: evidenceID, + Source: evidenceSource, + Subject: evidenceSubject, + ObservedAt: observedAt.UTC(), + IngestedAt: ingestedAt.UTC(), + Completeness: completeness, + Confidence: confidence, + Reason: reason, + Permissions: permissions, + PayloadRef: &operationaltrust.EvidencePayloadRef{ + Kind: "protection-provider-observation", + ID: evidenceID, + }, + }, + } + if err := observation.Validate(); err != nil { + return ProtectionProviderObservation{}, err + } + return observation, nil +} + +func BuildProtectionPostureFromPointsAt( + subjectResourceID string, + points []RecoveryPoint, + observations []ProtectionProviderObservation, + policy ProtectionPosturePolicy, + now time.Time, +) ProtectionPosture { + subjectResourceID = strings.TrimSpace(subjectResourceID) + type aggregate struct { + summary ProtectionProviderSummary + lastEventAt time.Time + } + byProviderScope := make(map[string]*aggregate) + + latestObservations := make(map[string]ProtectionProviderObservation) + for _, observation := range observations { + if err := observation.Validate(); err != nil { + continue + } + key := providerScopeKey(observation.Provider, observation.Scope) + current, exists := latestObservations[key] + if !exists || + observation.ObservedAt.After(current.ObservedAt) || + (observation.ObservedAt.Equal(current.ObservedAt) && observation.ID > current.ID) { + latestObservations[key] = observation.Clone() + } + } + + for _, point := range points { + if strings.TrimSpace(point.SubjectResourceID) != subjectResourceID { + continue + } + scope := ProviderScopeForPoint(point) + key := providerScopeKey(point.Provider, scope) + agg := byProviderScope[key] + if agg == nil { + agg = &aggregate{ + summary: ProtectionProviderSummary{ + Provider: point.Provider, + Source: "legacy-recovery-point", + Scope: scope, + JobState: OutcomeUnknown, + HistoryCompleteness: ProtectionHistoryUnknown, + Permissions: operationaltrust.EvidencePermissionsUnknown, + }, + } + byProviderScope[key] = agg + } + + eventAt := recoveryPointObservedAt(point) + if !eventAt.IsZero() { + if agg.summary.LastAttemptAt == nil || eventAt.After(*agg.summary.LastAttemptAt) { + value := eventAt.UTC() + agg.summary.LastAttemptAt = &value + } + if agg.lastEventAt.IsZero() || eventAt.After(agg.lastEventAt) { + agg.lastEventAt = eventAt + if validRecoveryOutcome(point.Outcome) { + agg.summary.JobState = point.Outcome + } else { + agg.summary.JobState = OutcomeUnknown + } + } + } + + switch point.Kind { + case KindBackup: + agg.summary.BackupPointCount++ + if point.Outcome == OutcomeSuccess && !eventAt.IsZero() { + if agg.summary.LastSuccessAt == nil || eventAt.After(*agg.summary.LastSuccessAt) { + value := eventAt.UTC() + agg.summary.LastSuccessAt = &value + } + } + if point.Outcome == OutcomeSuccess && + point.Verified != nil && + *point.Verified && + !eventAt.IsZero() { + if agg.summary.LastVerifiedAt == nil || eventAt.After(*agg.summary.LastVerifiedAt) { + value := eventAt.UTC() + agg.summary.LastVerifiedAt = &value + } + } + case KindSnapshot: + agg.summary.SnapshotPointCount++ + } + if value := strings.TrimSpace(point.RepositoryResourceID); value != "" { + agg.summary.RepositoryResourceIDs = append( + agg.summary.RepositoryResourceIDs, + value, + ) + } + if point.Evidence != nil { + agg.summary.EvidenceIDs = append(agg.summary.EvidenceIDs, point.Evidence.ID) + if source := strings.TrimSpace(point.Evidence.Source.Collector); source != "" { + agg.summary.Source = source + } + } + } + + for key, agg := range byProviderScope { + observation, ok := latestObservations[key] + if !ok { + continue + } + agg.summary.Source = observation.Source + agg.summary.HistoryCompleteness = observation.HistoryCompleteness + agg.summary.Permissions = observation.Permissions + agg.summary.VerificationExpected = observation.VerificationExpected + agg.summary.EvidenceIDs = append(agg.summary.EvidenceIDs, observation.Evidence.ID) + if !observation.ObservedAt.Before(agg.lastEventAt) && + observation.JobState != OutcomeUnknown { + agg.summary.JobState = observation.JobState + } + } + + summaries := make([]ProtectionProviderSummary, 0, len(byProviderScope)) + for _, agg := range byProviderScope { + agg.summary.RepositoryResourceIDs = sortedUnique(agg.summary.RepositoryResourceIDs) + agg.summary.EvidenceIDs = sortedUnique(agg.summary.EvidenceIDs) + summaries = append(summaries, agg.summary) + } + return DeriveProtectionPostureAt(subjectResourceID, summaries, policy, now) +} + +func DeriveProtectionPostureAt( + subjectResourceID string, + summaries []ProtectionProviderSummary, + policy ProtectionPosturePolicy, + now time.Time, +) ProtectionPosture { + subjectResourceID = strings.TrimSpace(subjectResourceID) + if err := policy.Validate(); err != nil { + policy = DefaultProtectionPosturePolicy + } + if now.IsZero() { + now = time.Now().UTC() + } else { + now = now.UTC() + } + + posture := ProtectionPosture{ + SubjectResourceID: subjectResourceID, + State: ProtectionStateUnknown, + Freshness: ProtectionFreshnessUnknown, + Verification: ProtectionVerificationUnknown, + Coverage: ProtectionCoverageUnknown, + ProviderStates: []ProtectionProviderState{}, + RepositoryResourceIDs: []string{}, + EvidenceIDs: []string{}, + Explanation: "Pulse has no complete provider history linked to this resource, so protection is unknown.", + EvaluatedAt: now, + } + + hasUnknownBlock := false + hasPartial := false + hasCompleteHistory := false + hasQualifyingSuccess := false + hasSupportedQualifyingSuccess := false + hasInvalidatingFailure := false + hasSnapshotsOnly := false + verificationExpected := policy.RequireVerification + supportedVerificationExpected := policy.RequireVerification + var supportedLastSuccessAt *time.Time + var supportedLastVerifiedAt *time.Time + + for _, summary := range summaries { + if strings.TrimSpace(string(summary.Provider)) == "" || + strings.TrimSpace(summary.Scope) == "" { + continue + } + if !validRecoveryOutcome(summary.JobState) { + summary.JobState = OutcomeUnknown + } + if !summary.HistoryCompleteness.Valid() { + summary.HistoryCompleteness = ProtectionHistoryUnknown + } + if !validProtectionPermissions(summary.Permissions) { + summary.Permissions = operationaltrust.EvidencePermissionsUnknown + } + summary.Source = strings.TrimSpace(summary.Source) + if summary.Source == "" { + summary.Source = "legacy-recovery-point" + } + summary.RepositoryResourceIDs = sortedUnique(summary.RepositoryResourceIDs) + summary.EvidenceIDs = sortedUnique(summary.EvidenceIDs) + + state := ProtectionProviderState{ + Provider: summary.Provider, + Source: summary.Source, + Scope: strings.TrimSpace(summary.Scope), + JobState: summary.JobState, + HistoryCompleteness: summary.HistoryCompleteness, + Permissions: summary.Permissions, + LastAttemptAt: cloneProtectionTime(summary.LastAttemptAt), + LastSuccessAt: cloneProtectionTime(summary.LastSuccessAt), + LastVerifiedAt: cloneProtectionTime(summary.LastVerifiedAt), + EvidenceIDs: append([]string(nil), summary.EvidenceIDs...), + VerificationExpected: summary.VerificationExpected, + } + posture.ProviderStates = append(posture.ProviderStates, state) + posture.RepositoryResourceIDs = append( + posture.RepositoryResourceIDs, + summary.RepositoryResourceIDs..., + ) + posture.EvidenceIDs = append(posture.EvidenceIDs, summary.EvidenceIDs...) + + posture.LastAttemptAt = latestProtectionTime(posture.LastAttemptAt, summary.LastAttemptAt) + posture.LastSuccessfulPointAt = latestProtectionTime( + posture.LastSuccessfulPointAt, + summary.LastSuccessAt, + ) + posture.LastVerifiedAt = latestProtectionTime(posture.LastVerifiedAt, summary.LastVerifiedAt) + + switch summary.HistoryCompleteness { + case ProtectionHistoryComplete: + hasCompleteHistory = true + case ProtectionHistoryPartial: + hasPartial = true + case ProtectionHistoryUnavailable, ProtectionHistoryUnknown: + hasUnknownBlock = true + } + switch summary.Permissions { + case operationaltrust.EvidencePermissionsSufficient: + case operationaltrust.EvidencePermissionsPartial: + hasPartial = true + case operationaltrust.EvidencePermissionsDenied, + operationaltrust.EvidencePermissionsUnknown: + hasUnknownBlock = true + } + + if summary.LastSuccessAt != nil { + hasQualifyingSuccess = true + } + if summary.LastSuccessAt != nil && + summary.HistoryCompleteness == ProtectionHistoryComplete && + summary.Permissions == operationaltrust.EvidencePermissionsSufficient { + hasSupportedQualifyingSuccess = true + supportedLastSuccessAt = latestProtectionTime( + supportedLastSuccessAt, + summary.LastSuccessAt, + ) + supportedLastVerifiedAt = latestProtectionTime( + supportedLastVerifiedAt, + summary.LastVerifiedAt, + ) + if summary.VerificationExpected { + supportedVerificationExpected = true + } + } + if summary.SnapshotPointCount > 0 && summary.BackupPointCount == 0 { + hasSnapshotsOnly = true + } + if summary.VerificationExpected { + verificationExpected = true + } + if summary.JobState == OutcomeFailed && + summary.LastAttemptAt != nil && + (summary.LastSuccessAt == nil || + !summary.LastAttemptAt.Before(*summary.LastSuccessAt)) { + hasInvalidatingFailure = true + } + } + + sort.Slice(posture.ProviderStates, func(i, j int) bool { + return providerStateSortKey(posture.ProviderStates[i]) < + providerStateSortKey(posture.ProviderStates[j]) + }) + posture.RepositoryResourceIDs = sortedUnique(posture.RepositoryResourceIDs) + posture.EvidenceIDs = sortedUnique(posture.EvidenceIDs) + + stateLastSuccessAt := posture.LastSuccessfulPointAt + stateLastVerifiedAt := posture.LastVerifiedAt + stateVerificationExpected := verificationExpected + if hasSupportedQualifyingSuccess { + stateLastSuccessAt = supportedLastSuccessAt + stateLastVerifiedAt = supportedLastVerifiedAt + stateVerificationExpected = supportedVerificationExpected + } + if stateLastSuccessAt != nil { + if now.Sub(*stateLastSuccessAt) <= policy.FreshnessWindow { + posture.Freshness = ProtectionFreshnessCurrent + } else { + posture.Freshness = ProtectionFreshnessStale + } + } + if stateLastSuccessAt != nil { + switch { + case stateLastVerifiedAt != nil && + now.Sub(*stateLastVerifiedAt) <= policy.VerificationWindow: + posture.Verification = ProtectionVerificationVerified + case stateLastVerifiedAt != nil: + posture.Verification = ProtectionVerificationStale + case stateVerificationExpected: + posture.Verification = ProtectionVerificationUnverified + } + } + + switch { + case len(posture.ProviderStates) == 0: + posture.Coverage = ProtectionCoverageUnknown + case hasUnknownBlock: + posture.Coverage = ProtectionCoverageUnknown + case hasPartial: + posture.Coverage = ProtectionCoveragePartial + case hasCompleteHistory && !hasQualifyingSuccess: + posture.Coverage = ProtectionCoverageNone + default: + posture.Coverage = ProtectionCoverageComplete + } + + switch { + case len(posture.ProviderStates) == 0: + // Keep the initialized unknown explanation. + case hasSupportedQualifyingSuccess && + posture.Freshness == ProtectionFreshnessCurrent && + !hasInvalidatingFailure && + !hasPartial && + (!stateVerificationExpected || posture.Verification == ProtectionVerificationVerified): + posture.State = ProtectionStateProtected + if posture.Verification == ProtectionVerificationVerified { + posture.Explanation = "A current subject-linked backup is available and has recent verification evidence." + } else { + posture.Explanation = "A current subject-linked backup is available from complete provider history." + } + if hasUnknownBlock { + posture.Explanation += " Another linked provider has unavailable history, but it does not invalidate the confirmed recovery point." + } + case hasSupportedQualifyingSuccess: + posture.State = ProtectionStateAttention + switch { + case hasInvalidatingFailure: + posture.Explanation = "A backup exists, but a newer provider failure needs attention before Pulse can call this resource protected." + case posture.Freshness == ProtectionFreshnessStale: + posture.Explanation = "The strongest subject-linked backup is older than the configured freshness window." + case stateVerificationExpected && posture.Verification != ProtectionVerificationVerified: + posture.Explanation = "A current backup exists, but its verification evidence is missing or stale." + default: + posture.Explanation = "Recovery evidence exists, but provider history or permissions are incomplete." + } + case hasUnknownBlock: + posture.State = ProtectionStateUnknown + posture.Explanation = "Provider history or permissions are unavailable, so Pulse cannot make a stronger protection claim." + case hasQualifyingSuccess || hasPartial: + posture.State = ProtectionStateAttention + posture.Explanation = "Recovery evidence exists, but provider history or permissions are incomplete." + case hasCompleteHistory: + posture.State = ProtectionStateUnprotected + if hasSnapshotsOnly { + posture.Explanation = "Provider history is complete, but only snapshots are present; snapshots alone do not prove independent recovery." + } else { + posture.Explanation = "Provider history is complete, but no qualifying subject-linked backup exists." + } + default: + posture.State = ProtectionStateUnknown + } + + return posture +} + +func recoveryPointObservedAt(point RecoveryPoint) time.Time { + if point.CompletedAt != nil && !point.CompletedAt.IsZero() { + return point.CompletedAt.UTC() + } + if point.StartedAt != nil && !point.StartedAt.IsZero() { + return point.StartedAt.UTC() + } + return time.Time{} +} + +func recoveryDetailString(point RecoveryPoint, key string) string { + if point.Details == nil { + return "" + } + value, _ := point.Details[key].(string) + return strings.TrimSpace(value) +} + +func providerScopeKey(provider Provider, scope string) string { + return strings.TrimSpace(string(provider)) + "\x00" + strings.TrimSpace(scope) +} + +func providerStateSortKey(state ProtectionProviderState) string { + return strings.Join([]string{ + strings.TrimSpace(string(state.Provider)), + strings.TrimSpace(state.Scope), + strings.TrimSpace(state.Source), + }, "\x00") +} + +func sortedUnique(values []string) []string { + set := make(map[string]struct{}, len(values)) + for _, value := range values { + if value = strings.TrimSpace(value); value != "" { + set[value] = struct{}{} + } + } + out := make([]string, 0, len(set)) + for value := range set { + out = append(out, value) + } + sort.Strings(out) + return out +} + +func latestProtectionTime(current, candidate *time.Time) *time.Time { + if candidate == nil { + return cloneProtectionTime(current) + } + if current == nil || candidate.After(*current) { + return cloneProtectionTime(candidate) + } + return cloneProtectionTime(current) +} + +func cloneProtectionTime(value *time.Time) *time.Time { + if value == nil { + return nil + } + clone := value.UTC() + return &clone +} + +func validRecoveryOutcome(value Outcome) bool { + switch value { + case OutcomeSuccess, OutcomeWarning, OutcomeFailed, OutcomeRunning, OutcomeUnknown: + return true + default: + return false + } +} + +func validProtectionPermissions(value operationaltrust.EvidencePermissions) bool { + switch value { + case operationaltrust.EvidencePermissionsSufficient, + operationaltrust.EvidencePermissionsPartial, + operationaltrust.EvidencePermissionsDenied, + operationaltrust.EvidencePermissionsUnknown: + return true + default: + return false + } +} diff --git a/internal/recovery/posture_test.go b/internal/recovery/posture_test.go new file mode 100644 index 000000000..5468ba0b4 --- /dev/null +++ b/internal/recovery/posture_test.go @@ -0,0 +1,321 @@ +package recovery + +import ( + "strings" + "testing" + "time" + + "github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust" +) + +func protectionTestSummary( + now time.Time, + lastSuccess *time.Time, +) ProtectionProviderSummary { + return ProtectionProviderSummary{ + Provider: ProviderProxmoxPBS, + Source: "pbs-backup-enumeration", + Scope: "pbs-main", + JobState: OutcomeSuccess, + HistoryCompleteness: ProtectionHistoryComplete, + Permissions: operationaltrust.EvidencePermissionsSufficient, + VerificationExpected: true, + LastAttemptAt: cloneProtectionTime(lastSuccess), + LastSuccessAt: cloneProtectionTime(lastSuccess), + LastVerifiedAt: cloneProtectionTime(lastSuccess), + BackupPointCount: 1, + RepositoryResourceIDs: []string{ + "repository:pbs-main/store-a", + }, + EvidenceIDs: []string{"evidence:point-1", "evidence:provider-1"}, + } +} + +func TestDeriveProtectionPostureTruthTable(t *testing.T) { + t.Parallel() + + now := time.Date(2026, 7, 19, 8, 0, 0, 0, time.UTC) + recent := now.Add(-2 * time.Hour) + stale := now.Add(-8 * 24 * time.Hour) + policy := DefaultProtectionPosturePolicy + + tests := []struct { + name string + summaries []ProtectionProviderSummary + wantState ProtectionState + wantFresh ProtectionFreshness + wantVerify ProtectionVerification + wantCoverage ProtectionCoverage + wantText string + }{ + { + name: "current verified PBS backup is protected", + summaries: []ProtectionProviderSummary{protectionTestSummary(now, &recent)}, + wantState: ProtectionStateProtected, + wantFresh: ProtectionFreshnessCurrent, + wantVerify: ProtectionVerificationVerified, + wantCoverage: ProtectionCoverageComplete, + wantText: "current subject-linked backup", + }, + { + name: "stale success needs attention", + summaries: []ProtectionProviderSummary{ + protectionTestSummary(now, &stale), + }, + wantState: ProtectionStateAttention, + wantFresh: ProtectionFreshnessStale, + wantVerify: ProtectionVerificationStale, + wantCoverage: ProtectionCoverageComplete, + wantText: "older than", + }, + { + name: "missing expected verification needs attention", + summaries: func() []ProtectionProviderSummary { + summary := protectionTestSummary(now, &recent) + summary.LastVerifiedAt = nil + return []ProtectionProviderSummary{summary} + }(), + wantState: ProtectionStateAttention, + wantFresh: ProtectionFreshnessCurrent, + wantVerify: ProtectionVerificationUnverified, + wantCoverage: ProtectionCoverageComplete, + wantText: "verification evidence", + }, + { + name: "newer provider failure invalidates protected claim", + summaries: func() []ProtectionProviderSummary { + summary := protectionTestSummary(now, &recent) + failedAt := now.Add(-time.Hour) + summary.JobState = OutcomeFailed + summary.LastAttemptAt = &failedAt + return []ProtectionProviderSummary{summary} + }(), + wantState: ProtectionStateAttention, + wantFresh: ProtectionFreshnessCurrent, + wantVerify: ProtectionVerificationVerified, + wantCoverage: ProtectionCoverageComplete, + wantText: "newer provider failure", + }, + { + name: "complete snapshot-only history is unprotected", + summaries: []ProtectionProviderSummary{ + { + Provider: ProviderProxmoxPVE, + Source: "pve-snapshot-enumeration", + Scope: "pve-main", + JobState: OutcomeSuccess, + HistoryCompleteness: ProtectionHistoryComplete, + Permissions: operationaltrust.EvidencePermissionsSufficient, + SnapshotPointCount: 3, + EvidenceIDs: []string{"evidence:snapshot"}, + }, + }, + wantState: ProtectionStateUnprotected, + wantFresh: ProtectionFreshnessUnknown, + wantVerify: ProtectionVerificationUnknown, + wantCoverage: ProtectionCoverageNone, + wantText: "snapshots alone", + }, + { + name: "partial provider history is attention when a backup exists", + summaries: func() []ProtectionProviderSummary { + summary := protectionTestSummary(now, &recent) + summary.HistoryCompleteness = ProtectionHistoryPartial + summary.Permissions = operationaltrust.EvidencePermissionsPartial + return []ProtectionProviderSummary{summary} + }(), + wantState: ProtectionStateAttention, + wantFresh: ProtectionFreshnessCurrent, + wantVerify: ProtectionVerificationVerified, + wantCoverage: ProtectionCoveragePartial, + wantText: "incomplete", + }, + { + name: "permission denied is unknown", + summaries: func() []ProtectionProviderSummary { + summary := protectionTestSummary(now, &recent) + summary.HistoryCompleteness = ProtectionHistoryUnavailable + summary.Permissions = operationaltrust.EvidencePermissionsDenied + return []ProtectionProviderSummary{summary} + }(), + wantState: ProtectionStateUnknown, + wantFresh: ProtectionFreshnessCurrent, + wantVerify: ProtectionVerificationVerified, + wantCoverage: ProtectionCoverageUnknown, + wantText: "permissions are unavailable", + }, + { + name: "confirmed PBS recovery is not invalidated by an unknown legacy provider", + summaries: func() []ProtectionProviderSummary { + confirmed := protectionTestSummary(now, &recent) + legacy := ProtectionProviderSummary{ + Provider: ProviderProxmoxPVE, + Source: "legacy-recovery-point", + Scope: "pve-main", + JobState: OutcomeSuccess, + HistoryCompleteness: ProtectionHistoryUnknown, + Permissions: operationaltrust.EvidencePermissionsUnknown, + LastAttemptAt: &recent, + LastSuccessAt: &recent, + BackupPointCount: 1, + EvidenceIDs: []string{"evidence:legacy-pve"}, + } + return []ProtectionProviderSummary{confirmed, legacy} + }(), + wantState: ProtectionStateProtected, + wantFresh: ProtectionFreshnessCurrent, + wantVerify: ProtectionVerificationVerified, + wantCoverage: ProtectionCoverageUnknown, + wantText: "does not invalidate", + }, + { + name: "no provider evidence is unknown", + summaries: nil, + wantState: ProtectionStateUnknown, + wantFresh: ProtectionFreshnessUnknown, + wantVerify: ProtectionVerificationUnknown, + wantCoverage: ProtectionCoverageUnknown, + wantText: "no complete provider history", + }, + } + + for _, test := range tests { + test := test + t.Run(test.name, func(t *testing.T) { + t.Parallel() + got := DeriveProtectionPostureAt( + "resource:vm-100", + test.summaries, + policy, + now, + ) + if got.State != test.wantState { + t.Fatalf("state = %q, want %q; posture=%#v", got.State, test.wantState, got) + } + if got.Freshness != test.wantFresh { + t.Fatalf("freshness = %q, want %q", got.Freshness, test.wantFresh) + } + if got.Verification != test.wantVerify { + t.Fatalf("verification = %q, want %q", got.Verification, test.wantVerify) + } + if got.Coverage != test.wantCoverage { + t.Fatalf("coverage = %q, want %q", got.Coverage, test.wantCoverage) + } + if !strings.Contains(got.Explanation, test.wantText) { + t.Fatalf("explanation = %q, want substring %q", got.Explanation, test.wantText) + } + if err := got.Validate(); err != nil { + t.Fatalf("Validate() error = %v", err) + } + }) + } +} + +func TestBuildProtectionPostureFromPointsUsesLatestProviderObservation(t *testing.T) { + t.Parallel() + + now := time.Date(2026, 7, 19, 8, 0, 0, 0, time.UTC) + completedAt := now.Add(-time.Hour) + verified := true + point := RecoveryPoint{ + ID: "pbs-backup:vm-100-2026-07-19", + Provider: ProviderProxmoxPBS, + Kind: KindBackup, + Mode: ModeRemote, + Outcome: OutcomeSuccess, + CompletedAt: &completedAt, + Verified: &verified, + SubjectResourceID: "resource:vm-100", + ProviderScope: "pbs-main", + } + envelope, err := NewRecoveryPointEvidence(point, "pbs-backup-inventory", now) + if err != nil { + t.Fatalf("NewRecoveryPointEvidence() error = %v", err) + } + point.Evidence = envelope + + older, err := NewProtectionProviderObservation( + ProviderProxmoxPBS, + "pbs-backup-enumeration", + "pbs-main", + OutcomeFailed, + ProtectionHistoryUnavailable, + operationaltrust.EvidencePermissionsUnknown, + true, + now.Add(-2*time.Hour), + now.Add(-2*time.Hour), + &operationaltrust.EvidenceReason{Code: "pbs_timeout"}, + ) + if err != nil { + t.Fatalf("older observation error = %v", err) + } + current, err := NewProtectionProviderObservation( + ProviderProxmoxPBS, + "pbs-backup-enumeration", + "pbs-main", + OutcomeSuccess, + ProtectionHistoryComplete, + operationaltrust.EvidencePermissionsSufficient, + true, + now, + now, + nil, + ) + if err != nil { + t.Fatalf("current observation error = %v", err) + } + + got := BuildProtectionPostureFromPointsAt( + "resource:vm-100", + []RecoveryPoint{point}, + []ProtectionProviderObservation{current, older}, + DefaultProtectionPosturePolicy, + now, + ) + if got.State != ProtectionStateProtected { + t.Fatalf("state = %q, want protected; posture=%#v", got.State, got) + } + if len(got.ProviderStates) != 1 { + t.Fatalf("provider states = %d, want 1", len(got.ProviderStates)) + } + if got.ProviderStates[0].HistoryCompleteness != ProtectionHistoryComplete { + t.Fatalf( + "history completeness = %q, want complete", + got.ProviderStates[0].HistoryCompleteness, + ) + } + if len(got.EvidenceIDs) != 2 { + t.Fatalf("evidence ids = %#v, want point and latest provider evidence", got.EvidenceIDs) + } +} + +func TestNewProtectionProviderObservationRequiresTypedLimitation(t *testing.T) { + t.Parallel() + + now := time.Date(2026, 7, 19, 8, 0, 0, 0, time.UTC) + observation, err := NewProtectionProviderObservation( + ProviderProxmoxPBS, + "pbs-backup-enumeration", + "pbs-main", + OutcomeFailed, + ProtectionHistoryUnavailable, + operationaltrust.EvidencePermissionsDenied, + true, + now, + now, + &operationaltrust.EvidenceReason{ + Code: "pbs_access_denied", + Message: "PBS did not authorize backup history enumeration.", + }, + ) + if err != nil { + t.Fatalf("NewProtectionProviderObservation() error = %v", err) + } + if observation.Evidence.Reason == nil || + observation.Evidence.Reason.Code != "pbs_access_denied" { + t.Fatalf("reason = %#v, want pbs_access_denied", observation.Evidence.Reason) + } + if observation.Evidence.Permissions != operationaltrust.EvidencePermissionsDenied { + t.Fatalf("permissions = %q, want denied", observation.Evidence.Permissions) + } +} diff --git a/internal/recovery/store/store.go b/internal/recovery/store/store.go index 6af4b29c7..5fcfe69f1 100644 --- a/internal/recovery/store/store.go +++ b/internal/recovery/store/store.go @@ -13,6 +13,7 @@ import ( "sync" "time" + "github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust" "github.com/rcourtman/pulse-go-rewrite/internal/recovery" "github.com/rs/zerolog/log" @@ -362,6 +363,9 @@ func Open(dbPath string) (*Store, error) { // Best-effort backfill for PBS guest rows that predate canonical historical continuity. // This should never block startup; missing backfills only affect protected-item accuracy. _ = store.BackfillHistoricalProxmoxPBSGuestIdentity(context.Background()) + // Best-effort backfill for provider scopes and the materialized canonical + // protection posture read model. + _ = store.BackfillProtectionMetadata(context.Background()) if err := hardenSQLiteArtifacts(dbPath); err != nil { _ = db.Close() return nil, fmt.Errorf("failed to secure recovery db files: %w", err) @@ -394,6 +398,35 @@ func (s *Store) PurgeStalePVEPBSBackups(ctx context.Context) error { ctx = context.Background() } + rows, err := s.db.QueryContext(ctx, ` + SELECT DISTINCT subject_key + FROM recovery_points + WHERE id LIKE 'pve-backup:%' + AND json_extract(details_json, '$.isPBS') = 1 + AND subject_key IS NOT NULL AND TRIM(subject_key) != '' + `) + if err != nil { + return err + } + affectedSubjectKeys := make(map[string]struct{}) + for rows.Next() { + var key string + if err := rows.Scan(&key); err != nil { + _ = rows.Close() + return err + } + if key = strings.TrimSpace(key); key != "" { + affectedSubjectKeys[key] = struct{}{} + } + } + if err := rows.Err(); err != nil { + _ = rows.Close() + return err + } + if err := rows.Close(); err != nil { + return err + } + res, err := s.db.ExecContext(ctx, ` DELETE FROM recovery_points WHERE id LIKE 'pve-backup:%' @@ -408,6 +441,15 @@ func (s *Store) PurgeStalePVEPBSBackups(ctx context.Context) error { return err } if deleted > 0 { + if err := s.RefreshProtectionPostures( + ctx, + sortedStringSet(affectedSubjectKeys), + ); err != nil { + return fmt.Errorf( + "refresh protection postures after stale PVE/PBS purge: %w", + err, + ) + } log.Info(). Int64("deleted", deleted). Msg("Purged stale PVE-sourced PBS backup entries - PBS direct is now authoritative") @@ -447,7 +489,7 @@ func (s *Store) ReconcileInstancePoints(ctx context.Context, provider, idPrefix, // repository_ref_json.namespace. The id prefix keeps the classes apart, so // matching either field never crosses providers. rows, err := s.db.QueryContext(ctx, ` - SELECT id FROM recovery_points + SELECT id, subject_key FROM recovery_points WHERE provider = ? AND id LIKE ? || '%' AND ( @@ -461,13 +503,18 @@ func (s *Store) ReconcileInstancePoints(ctx context.Context, provider, idPrefix, defer rows.Close() var stale []string + affectedSubjectKeys := make(map[string]struct{}) for rows.Next() { var id string - if err := rows.Scan(&id); err != nil { + var subjectKey sql.NullString + if err := rows.Scan(&id, &subjectKey); err != nil { return 0, err } if _, ok := keep[id]; !ok { stale = append(stale, id) + if key := strings.TrimSpace(subjectKey.String); key != "" { + affectedSubjectKeys[key] = struct{}{} + } } } if err := rows.Err(); err != nil { @@ -512,6 +559,15 @@ func (s *Store) ReconcileInstancePoints(ctx context.Context, provider, idPrefix, if err = tx.Commit(); err != nil { return 0, err } + if err := s.RefreshProtectionPostures( + ctx, + sortedStringSet(affectedSubjectKeys), + ); err != nil { + return deleted, fmt.Errorf( + "refresh protection postures after recovery point reconciliation: %w", + err, + ) + } if deleted > 0 { log.Info(). @@ -543,6 +599,9 @@ func (s *Store) initSchema() error { verified INTEGER, encrypted INTEGER, immutable INTEGER, + provider_scope TEXT, + evidence_id TEXT, + evidence_json TEXT, subject_key TEXT, repository_key TEXT, subject_resource_id TEXT, @@ -575,6 +634,37 @@ func (s *Store) initSchema() error { CREATE INDEX IF NOT EXISTS idx_recovery_points_subject_key_completed ON recovery_points(subject_key, completed_at_ms); + + CREATE TABLE IF NOT EXISTS protection_provider_observations ( + id TEXT PRIMARY KEY, + provider TEXT NOT NULL, + source TEXT NOT NULL, + scope TEXT NOT NULL, + job_state TEXT NOT NULL, + history_completeness TEXT NOT NULL, + permissions TEXT NOT NULL, + verification_expected INTEGER NOT NULL DEFAULT 0, + observed_at_ms INTEGER NOT NULL, + ingested_at_ms INTEGER NOT NULL, + evidence_json TEXT NOT NULL, + created_at_ms INTEGER NOT NULL, + updated_at_ms INTEGER NOT NULL + ); + + CREATE INDEX IF NOT EXISTS idx_protection_provider_observations_scope + ON protection_provider_observations(provider, scope, observed_at_ms DESC); + + CREATE TABLE IF NOT EXISTS protection_postures ( + subject_key TEXT PRIMARY KEY, + subject_resource_id TEXT NOT NULL UNIQUE, + state TEXT NOT NULL, + posture_json TEXT NOT NULL, + evaluated_at_ms INTEGER NOT NULL, + updated_at_ms INTEGER NOT NULL + ); + + CREATE INDEX IF NOT EXISTS idx_protection_postures_state + ON protection_postures(state, evaluated_at_ms DESC, subject_resource_id); ` if _, err := s.db.Exec(schema); err != nil { return err @@ -587,6 +677,15 @@ func (s *Store) initSchema() error { if err := s.ensureColumn("recovery_points", "repository_key", "TEXT"); err != nil { return err } + if err := s.ensureColumn("recovery_points", "provider_scope", "TEXT"); err != nil { + return err + } + if err := s.ensureColumn("recovery_points", "evidence_id", "TEXT"); err != nil { + return err + } + if err := s.ensureColumn("recovery_points", "evidence_json", "TEXT"); err != nil { + return err + } // Migration: normalized index columns for efficient filtering/searching and consistent UI display. for _, col := range []struct { @@ -623,6 +722,12 @@ func (s *Store) initSchema() error { CREATE INDEX IF NOT EXISTS idx_recovery_points_namespace_completed ON recovery_points(namespace_label, completed_at_ms); + + CREATE INDEX IF NOT EXISTS idx_recovery_points_subject_provider_scope + ON recovery_points(subject_key, provider, provider_scope, completed_at_ms); + + CREATE INDEX IF NOT EXISTS idx_recovery_points_provider_scope + ON recovery_points(provider, provider_scope, completed_at_ms); ` if _, err := s.db.Exec(postMigrationIndexes); err != nil { return err @@ -975,6 +1080,7 @@ func (s *Store) UpsertPoints(ctx context.Context, points []recovery.RecoveryPoin id, provider, kind, mode, outcome, started_at_ms, completed_at_ms, size_bytes, verified, encrypted, immutable, + provider_scope, evidence_id, evidence_json, subject_key, repository_key, subject_resource_id, repository_resource_id, subject_ref_json, repository_ref_json, details_json, @@ -982,7 +1088,7 @@ func (s *Store) UpsertPoints(ctx context.Context, points []recovery.RecoveryPoin cluster_label, node_host_label, namespace_label, entity_id_label, repository_label, details_summary, created_at_ms, updated_at_ms - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT(id) DO UPDATE SET provider=excluded.provider, kind=excluded.kind, @@ -994,6 +1100,9 @@ func (s *Store) UpsertPoints(ctx context.Context, points []recovery.RecoveryPoin verified=excluded.verified, encrypted=excluded.encrypted, immutable=excluded.immutable, + provider_scope=excluded.provider_scope, + evidence_id=excluded.evidence_id, + evidence_json=excluded.evidence_json, subject_key=excluded.subject_key, repository_key=excluded.repository_key, subject_resource_id=excluded.subject_resource_id, @@ -1018,6 +1127,7 @@ func (s *Store) UpsertPoints(ctx context.Context, points []recovery.RecoveryPoin } defer stmt.Close() + affectedSubjectKeys := make(map[string]struct{}, len(points)) for _, p := range points { adoptHistoricalProxmoxPBSGuestIdentity(&p, proxmoxPBSIdentities, proxmoxPBSLooseIdentities) @@ -1045,6 +1155,10 @@ func (s *Store) UpsertPoints(ctx context.Context, points []recovery.RecoveryPoin if err != nil { return err } + evidence, err := marshalJSON(p.Evidence) + if err != nil { + return err + } var size any if p.SizeBytes != nil { @@ -1054,7 +1168,15 @@ func (s *Store) UpsertPoints(ctx context.Context, points []recovery.RecoveryPoin subjectRID := strings.TrimSpace(p.SubjectResourceID) repoRID := strings.TrimSpace(p.RepositoryResourceID) subjectKey := recovery.SubjectKeyForPoint(p) + if strings.TrimSpace(subjectKey) != "" { + affectedSubjectKeys[strings.TrimSpace(subjectKey)] = struct{}{} + } repoKey := recovery.RepositoryKey(p.Provider, repoRID, p.RepositoryRef) + providerScope := recovery.ProviderScopeForPoint(p) + evidenceID := "" + if p.Evidence != nil { + evidenceID = strings.TrimSpace(p.Evidence.ID) + } idx := recovery.DeriveIndex(p) if _, err := stmt.ExecContext( @@ -1070,6 +1192,9 @@ func (s *Store) UpsertPoints(ctx context.Context, points []recovery.RecoveryPoin boolPtrToDB(p.Verified), boolPtrToDB(p.Encrypted), boolPtrToDB(p.Immutable), + providerScope, + evidenceID, + nullStringToAny(evidence), strings.TrimSpace(subjectKey), strings.TrimSpace(repoKey), subjectRID, @@ -1110,6 +1235,12 @@ func (s *Store) UpsertPoints(ctx context.Context, points []recovery.RecoveryPoin if err := tx.Commit(); err != nil { return err } + if err := s.RefreshProtectionPostures( + ctx, + sortedStringSet(affectedSubjectKeys), + ); err != nil { + return fmt.Errorf("refresh protection postures after recovery point upsert: %w", err) + } return nil } @@ -1137,7 +1268,8 @@ func (s *Store) BackfillHistoricalProxmoxPBSGuestIdentity(ctx context.Context) e id, provider, kind, mode, outcome, subject_resource_id, repository_resource_id, subject_ref_json, repository_ref_json, details_json, - subject_label, item_type, namespace_label, entity_id_label + subject_label, item_type, namespace_label, entity_id_label, + subject_key FROM recovery_points WHERE provider = 'proxmox-pbs' LIMIT `+fmt.Sprint(maxBackfillRows)+` @@ -1162,6 +1294,7 @@ func (s *Store) BackfillHistoricalProxmoxPBSGuestIdentity(ctx context.Context) e itemType sql.NullString namespaceLabel sql.NullString entityIDLabel sql.NullString + subjectKey sql.NullString } items := make([]item, 0, 256) @@ -1182,6 +1315,7 @@ func (s *Store) BackfillHistoricalProxmoxPBSGuestIdentity(ctx context.Context) e &r.itemType, &r.namespaceLabel, &r.entityIDLabel, + &r.subjectKey, ); err != nil { return err } @@ -1223,6 +1357,7 @@ func (s *Store) BackfillHistoricalProxmoxPBSGuestIdentity(ctx context.Context) e } defer stmt.Close() + affectedSubjectKeys := make(map[string]struct{}) for _, item := range items { key := recovery.ProxmoxPBSGuestContinuityKey( item.subjectLabel.String, @@ -1289,9 +1424,10 @@ func (s *Store) BackfillHistoricalProxmoxPBSGuestIdentity(ctx context.Context) e return err } + nextSubjectKey := recovery.SubjectKeyForPoint(p) if _, err := stmt.ExecContext( ctx, - recovery.SubjectKeyForPoint(p), + nextSubjectKey, p.SubjectResourceID, nullStringToAny(subjectRefJSON), strings.TrimSpace(idx.SubjectLabel), @@ -1308,11 +1444,26 @@ func (s *Store) BackfillHistoricalProxmoxPBSGuestIdentity(ctx context.Context) e ); err != nil { return err } + if key := strings.TrimSpace(item.subjectKey.String); key != "" { + affectedSubjectKeys[key] = struct{}{} + } + if key := strings.TrimSpace(nextSubjectKey); key != "" { + affectedSubjectKeys[key] = struct{}{} + } } if err := tx.Commit(); err != nil { return err } + if err := s.RefreshProtectionPostures( + ctx, + sortedStringSet(affectedSubjectKeys), + ); err != nil { + return fmt.Errorf( + "refresh protection postures after PBS identity backfill: %w", + err, + ) + } return nil } @@ -1334,11 +1485,58 @@ func (s *Store) maybePrune(ctx context.Context) { s.lastPrune = now cutoffMs := now.Add(-s.retention).UnixMilli() - // Best-effort cleanup; never block writes if pruning fails. - _, _ = s.db.ExecContext(ctx, ` + // Best-effort cleanup; never block writes if pruning fails. Capture every + // subject whose posture can change before deleting either source points or + // the provider evidence used to qualify their history. + affectedSubjectKeys := make(map[string]struct{}) + rows, err := s.db.QueryContext(ctx, ` + SELECT DISTINCT subject_key + FROM recovery_points + WHERE COALESCE(completed_at_ms, started_at_ms, updated_at_ms) < ? + AND subject_key IS NOT NULL AND TRIM(subject_key) != '' + UNION + SELECT DISTINCT rp.subject_key + FROM recovery_points rp + JOIN protection_provider_observations observation + ON observation.provider = rp.provider + AND observation.scope = rp.provider_scope + WHERE observation.observed_at_ms < ? + AND rp.subject_key IS NOT NULL AND TRIM(rp.subject_key) != '' + `, cutoffMs, cutoffMs) + if err == nil { + for rows.Next() { + var key string + if scanErr := rows.Scan(&key); scanErr != nil { + err = scanErr + break + } + if key = strings.TrimSpace(key); key != "" { + affectedSubjectKeys[key] = struct{}{} + } + } + if rowsErr := rows.Err(); err == nil { + err = rowsErr + } + if closeErr := rows.Close(); err == nil { + err = closeErr + } + } + if err != nil { + return + } + if _, err = s.db.ExecContext(ctx, ` DELETE FROM recovery_points WHERE COALESCE(completed_at_ms, started_at_ms, updated_at_ms) < ? - `, cutoffMs) + `, cutoffMs); err != nil { + return + } + if _, err = s.db.ExecContext(ctx, ` + DELETE FROM protection_provider_observations + WHERE observed_at_ms < ? + `, cutoffMs); err != nil { + return + } + _ = s.RefreshProtectionPostures(ctx, sortedStringSet(affectedSubjectKeys)) } func nullStringToAny(s sql.NullString) any { @@ -1464,6 +1662,7 @@ func (s *Store) ListPoints(ctx context.Context, opts recovery.ListPointsOptions) id, provider, kind, mode, outcome, started_at_ms, completed_at_ms, size_bytes, verified, encrypted, immutable, + provider_scope, evidence_json, subject_resource_id, repository_resource_id, subject_ref_json, repository_ref_json, details_json , subject_label, subject_type, item_type, is_workload, @@ -1489,6 +1688,7 @@ func (s *Store) ListPoints(ctx context.Context, opts recovery.ListPointsOptions) var startedMs, completedMs sql.NullInt64 var sizeBytes sql.NullInt64 var verified, encrypted, immutable sql.NullInt64 + var providerScope, evidenceRaw sql.NullString var subjectRID, repoRID sql.NullString var subjectRefRaw, repoRefRaw, detailsRaw sql.NullString var subjectLabel, subjectType sql.NullString @@ -1509,6 +1709,8 @@ func (s *Store) ListPoints(ctx context.Context, opts recovery.ListPointsOptions) &verified, &encrypted, &immutable, + &providerScope, + &evidenceRaw, &subjectRID, &repoRID, &subjectRefRaw, @@ -1543,6 +1745,12 @@ func (s *Store) ListPoints(ctx context.Context, opts recovery.ListPointsOptions) p.Verified = dbToBoolPtr(verified) p.Encrypted = dbToBoolPtr(encrypted) p.Immutable = dbToBoolPtr(immutable) + p.ProviderScope = strings.TrimSpace(providerScope.String) + var evidence operationaltrust.EvidenceEnvelope + _ = decodeRecoveryJSONField(p.ID, "evidence_json", evidenceRaw, &evidence) + if strings.TrimSpace(evidence.ID) != "" { + p.Evidence = &evidence + } if subjectRID.Valid { p.SubjectResourceID = subjectRID.String diff --git a/internal/recovery/store/store_posture.go b/internal/recovery/store/store_posture.go new file mode 100644 index 000000000..62f249b1f --- /dev/null +++ b/internal/recovery/store/store_posture.go @@ -0,0 +1,769 @@ +package store + +import ( + "context" + "database/sql" + "encoding/json" + "fmt" + "sort" + "strings" + "time" + + "github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust" + "github.com/rcourtman/pulse-go-rewrite/internal/recovery" +) + +const protectionRefreshChunkSize = 200 + +func sortedStringSet(values map[string]struct{}) []string { + out := make([]string, 0, len(values)) + for value := range values { + value = strings.TrimSpace(value) + if value != "" { + out = append(out, value) + } + } + sort.Strings(out) + return out +} + +func (s *Store) BackfillProtectionMetadata(ctx context.Context) error { + if err := s.ensureInitialized(); err != nil { + return err + } + if ctx == nil { + ctx = context.Background() + } + + const maxBackfillRows = 5000 + rows, err := s.db.QueryContext(ctx, ` + SELECT id, provider, kind, mode, outcome, + started_at_ms, completed_at_ms, updated_at_ms, + subject_resource_id, subject_ref_json, + repository_resource_id, repository_ref_json, details_json, + evidence_id, evidence_json + FROM recovery_points + WHERE provider_scope IS NULL OR TRIM(provider_scope) = '' + OR ( + (evidence_json IS NULL OR TRIM(evidence_json) = '') + AND COALESCE(completed_at_ms, started_at_ms, 0) > 0 + ) + LIMIT `+fmt.Sprint(maxBackfillRows)) + if err != nil { + return err + } + + type backfillRow struct { + id string + provider string + kind string + mode string + outcome string + startedMs sql.NullInt64 + completedMs sql.NullInt64 + updatedAtMs int64 + subjectRID sql.NullString + subjectRefRaw sql.NullString + repoRID sql.NullString + repoRefRaw sql.NullString + detailsRaw sql.NullString + evidenceID sql.NullString + evidenceRaw sql.NullString + } + items := make([]backfillRow, 0, 256) + for rows.Next() { + var item backfillRow + if err := rows.Scan( + &item.id, + &item.provider, + &item.kind, + &item.mode, + &item.outcome, + &item.startedMs, + &item.completedMs, + &item.updatedAtMs, + &item.subjectRID, + &item.subjectRefRaw, + &item.repoRID, + &item.repoRefRaw, + &item.detailsRaw, + &item.evidenceID, + &item.evidenceRaw, + ); err != nil { + _ = rows.Close() + return err + } + items = append(items, item) + } + if err := rows.Err(); err != nil { + _ = rows.Close() + return err + } + if err := rows.Close(); err != nil { + return err + } + + if len(items) > 0 { + tx, err := s.db.BeginTx(ctx, nil) + if err != nil { + return err + } + defer func() { _ = tx.Rollback() }() + stmt, err := tx.PrepareContext( + ctx, + `UPDATE recovery_points + SET provider_scope = ?, + evidence_id = CASE + WHEN evidence_id IS NULL OR TRIM(evidence_id) = '' THEN ? + ELSE evidence_id + END, + evidence_json = CASE + WHEN evidence_json IS NULL OR TRIM(evidence_json) = '' THEN ? + ELSE evidence_json + END + WHERE id = ?`, + ) + if err != nil { + return err + } + defer stmt.Close() + for _, item := range items { + point := recovery.RecoveryPoint{ + ID: strings.TrimSpace(item.id), + Provider: recovery.Provider(strings.TrimSpace(item.provider)), + Kind: recovery.Kind(strings.TrimSpace(item.kind)), + Mode: recovery.Mode(strings.TrimSpace(item.mode)), + Outcome: recovery.Outcome(strings.TrimSpace(item.outcome)), + StartedAt: millisToTimePtr(item.startedMs), + CompletedAt: millisToTimePtr(item.completedMs), + SubjectResourceID: strings.TrimSpace(item.subjectRID.String), + RepositoryResourceID: strings.TrimSpace(item.repoRID.String), + } + _ = unmarshalJSON(item.subjectRefRaw, &point.SubjectRef) + _ = unmarshalJSON(item.repoRefRaw, &point.RepositoryRef) + _ = unmarshalJSON(item.detailsRaw, &point.Details) + var evidenceID any + var evidenceJSON any + if strings.TrimSpace(item.evidenceRaw.String) == "" { + ingestedAt := time.UnixMilli(item.updatedAtMs).UTC() + evidence, evidenceErr := recovery.NewRecoveryPointEvidence( + point, + "recovery-point-migration", + ingestedAt, + ) + if evidenceErr == nil { + encoded, marshalErr := json.Marshal(evidence) + if marshalErr != nil { + return marshalErr + } + evidenceID = evidence.ID + evidenceJSON = string(encoded) + } + } + if _, err := stmt.ExecContext( + ctx, + recovery.ProviderScopeForPoint(point), + evidenceID, + evidenceJSON, + point.ID, + ); err != nil { + return err + } + } + if err := tx.Commit(); err != nil { + return err + } + } + + keyRows, err := s.db.QueryContext(ctx, ` + SELECT DISTINCT subject_key + FROM recovery_points + WHERE subject_key IS NOT NULL AND TRIM(subject_key) != '' + ORDER BY subject_key + LIMIT `+fmt.Sprint(maxBackfillRows)) + if err != nil { + return err + } + keys := make([]string, 0, 512) + for keyRows.Next() { + var key string + if err := keyRows.Scan(&key); err != nil { + _ = keyRows.Close() + return err + } + if key = strings.TrimSpace(key); key != "" { + keys = append(keys, key) + } + } + if err := keyRows.Err(); err != nil { + _ = keyRows.Close() + return err + } + if err := keyRows.Close(); err != nil { + return err + } + return s.RefreshProtectionPostures(ctx, keys) +} + +func (s *Store) UpsertProtectionProviderObservations( + ctx context.Context, + observations []recovery.ProtectionProviderObservation, +) error { + if err := s.ensureInitialized(); err != nil { + return err + } + if len(observations) == 0 { + return nil + } + if ctx == nil { + ctx = context.Background() + } + s.maybePrune(ctx) + + nowMs := time.Now().UTC().UnixMilli() + tx, err := s.db.BeginTx(ctx, nil) + if err != nil { + return err + } + defer func() { _ = tx.Rollback() }() + stmt, err := tx.PrepareContext(ctx, ` + INSERT INTO protection_provider_observations ( + id, provider, source, scope, job_state, + history_completeness, permissions, verification_expected, + observed_at_ms, ingested_at_ms, evidence_json, + created_at_ms, updated_at_ms + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(id) DO UPDATE SET + provider=excluded.provider, + source=excluded.source, + scope=excluded.scope, + job_state=excluded.job_state, + history_completeness=excluded.history_completeness, + permissions=excluded.permissions, + verification_expected=excluded.verification_expected, + observed_at_ms=excluded.observed_at_ms, + ingested_at_ms=excluded.ingested_at_ms, + evidence_json=excluded.evidence_json, + updated_at_ms=excluded.updated_at_ms + `) + if err != nil { + return err + } + defer stmt.Close() + + type providerScope struct { + provider recovery.Provider + scope string + } + affectedScopes := make(map[providerScope]struct{}, len(observations)) + for _, observation := range observations { + if err := observation.Validate(); err != nil { + return err + } + evidenceJSON, err := json.Marshal(observation.Evidence) + if err != nil { + return err + } + verificationExpected := 0 + if observation.VerificationExpected { + verificationExpected = 1 + } + if _, err := stmt.ExecContext( + ctx, + observation.ID, + string(observation.Provider), + observation.Source, + observation.Scope, + string(observation.JobState), + string(observation.HistoryCompleteness), + string(observation.Permissions), + verificationExpected, + observation.ObservedAt.UTC().UnixMilli(), + observation.IngestedAt.UTC().UnixMilli(), + string(evidenceJSON), + nowMs, + nowMs, + ); err != nil { + return err + } + affectedScopes[providerScope{ + provider: observation.Provider, + scope: strings.TrimSpace(observation.Scope), + }] = struct{}{} + } + if err := tx.Commit(); err != nil { + return err + } + + affectedKeys := make(map[string]struct{}) + for scope := range affectedScopes { + rows, err := s.db.QueryContext(ctx, ` + SELECT DISTINCT subject_key + FROM recovery_points + WHERE provider = ? AND provider_scope = ? + AND subject_key IS NOT NULL AND TRIM(subject_key) != '' + `, string(scope.provider), scope.scope) + if err != nil { + return err + } + for rows.Next() { + var key string + if err := rows.Scan(&key); err != nil { + _ = rows.Close() + return err + } + if key = strings.TrimSpace(key); key != "" { + affectedKeys[key] = struct{}{} + } + } + if err := rows.Err(); err != nil { + _ = rows.Close() + return err + } + if err := rows.Close(); err != nil { + return err + } + } + return s.RefreshProtectionPostures(ctx, sortedStringSet(affectedKeys)) +} + +func (s *Store) RefreshProtectionPostures(ctx context.Context, subjectKeys []string) error { + if err := s.ensureInitialized(); err != nil { + return err + } + subjectKeys = normalizeProtectionStrings(subjectKeys) + if len(subjectKeys) == 0 { + return nil + } + if ctx == nil { + ctx = context.Background() + } + + observations, err := s.listLatestProtectionProviderObservations(ctx) + if err != nil { + return err + } + for start := 0; start < len(subjectKeys); start += protectionRefreshChunkSize { + end := start + protectionRefreshChunkSize + if end > len(subjectKeys) { + end = len(subjectKeys) + } + chunk := subjectKeys[start:end] + pointsByKey, err := s.loadProtectionPointsForKeys(ctx, chunk) + if err != nil { + return err + } + if err := s.writeProtectionPostures( + ctx, + chunk, + pointsByKey, + observations, + ); err != nil { + return err + } + } + return nil +} + +func (s *Store) loadProtectionPointsForKeys( + ctx context.Context, + subjectKeys []string, +) (map[string][]recovery.RecoveryPoint, error) { + placeholders := strings.TrimSuffix(strings.Repeat("?,", len(subjectKeys)), ",") + args := make([]any, len(subjectKeys)) + for i := range subjectKeys { + args[i] = subjectKeys[i] + } + rows, err := s.db.QueryContext(ctx, ` + SELECT + subject_key, id, provider, kind, mode, outcome, + started_at_ms, completed_at_ms, verified, + subject_resource_id, repository_resource_id, + subject_ref_json, repository_ref_json, details_json, + provider_scope, evidence_json + FROM recovery_points + WHERE subject_key IN (`+placeholders+`) + ORDER BY subject_key, + COALESCE(completed_at_ms, started_at_ms, updated_at_ms) DESC, + updated_at_ms DESC, + id DESC + `, args...) + if err != nil { + return nil, err + } + defer rows.Close() + + out := make(map[string][]recovery.RecoveryPoint, len(subjectKeys)) + for rows.Next() { + var ( + subjectKey string + point recovery.RecoveryPoint + provider, kind string + mode, outcome string + startedMs, completedMs sql.NullInt64 + verified sql.NullInt64 + subjectRID, repoRID sql.NullString + subjectRefRaw sql.NullString + repoRefRaw, detailsRaw sql.NullString + providerScope sql.NullString + evidenceRaw sql.NullString + ) + if err := rows.Scan( + &subjectKey, + &point.ID, + &provider, + &kind, + &mode, + &outcome, + &startedMs, + &completedMs, + &verified, + &subjectRID, + &repoRID, + &subjectRefRaw, + &repoRefRaw, + &detailsRaw, + &providerScope, + &evidenceRaw, + ); err != nil { + return nil, err + } + point.Provider = recovery.Provider(provider) + point.Kind = recovery.Kind(kind) + point.Mode = recovery.Mode(mode) + point.Outcome = recovery.Outcome(outcome) + point.StartedAt = millisToTimePtr(startedMs) + point.CompletedAt = millisToTimePtr(completedMs) + point.Verified = dbToBoolPtr(verified) + point.SubjectResourceID = strings.TrimSpace(subjectRID.String) + point.RepositoryResourceID = strings.TrimSpace(repoRID.String) + point.ProviderScope = strings.TrimSpace(providerScope.String) + _ = unmarshalJSON(subjectRefRaw, &point.SubjectRef) + _ = unmarshalJSON(repoRefRaw, &point.RepositoryRef) + _ = unmarshalJSON(detailsRaw, &point.Details) + var evidence operationaltrust.EvidenceEnvelope + _ = unmarshalJSON(evidenceRaw, &evidence) + if strings.TrimSpace(evidence.ID) != "" { + point.Evidence = &evidence + } + subjectKey = strings.TrimSpace(subjectKey) + out[subjectKey] = append(out[subjectKey], point) + } + return out, rows.Err() +} + +func (s *Store) listLatestProtectionProviderObservations( + ctx context.Context, +) ([]recovery.ProtectionProviderObservation, error) { + rows, err := s.db.QueryContext(ctx, ` + SELECT + id, provider, source, scope, job_state, + history_completeness, permissions, verification_expected, + observed_at_ms, ingested_at_ms, evidence_json + FROM ( + SELECT *, + ROW_NUMBER() OVER ( + PARTITION BY provider, scope + ORDER BY observed_at_ms DESC, id DESC + ) AS rn + FROM protection_provider_observations + ) + WHERE rn = 1 + ORDER BY provider, scope + `) + if err != nil { + return nil, err + } + defer rows.Close() + + out := make([]recovery.ProtectionProviderObservation, 0, 16) + for rows.Next() { + var ( + observation recovery.ProtectionProviderObservation + provider, jobState string + historyCompleteness, permissions string + verificationExpected int + observedAtMs, ingestedAtMs int64 + evidenceRaw string + ) + if err := rows.Scan( + &observation.ID, + &provider, + &observation.Source, + &observation.Scope, + &jobState, + &historyCompleteness, + &permissions, + &verificationExpected, + &observedAtMs, + &ingestedAtMs, + &evidenceRaw, + ); err != nil { + return nil, err + } + observation.Provider = recovery.Provider(provider) + observation.JobState = recovery.Outcome(jobState) + observation.HistoryCompleteness = + recovery.ProtectionHistoryCompleteness(historyCompleteness) + observation.Permissions = operationaltrust.EvidencePermissions(permissions) + observation.VerificationExpected = verificationExpected != 0 + observation.ObservedAt = time.UnixMilli(observedAtMs).UTC() + observation.IngestedAt = time.UnixMilli(ingestedAtMs).UTC() + if err := json.Unmarshal([]byte(evidenceRaw), &observation.Evidence); err != nil { + continue + } + if err := observation.Validate(); err != nil { + continue + } + out = append(out, observation) + } + return out, rows.Err() +} + +func (s *Store) writeProtectionPostures( + ctx context.Context, + subjectKeys []string, + pointsByKey map[string][]recovery.RecoveryPoint, + observations []recovery.ProtectionProviderObservation, +) error { + now := time.Now().UTC() + nowMs := now.UnixMilli() + tx, err := s.db.BeginTx(ctx, nil) + if err != nil { + return err + } + defer func() { _ = tx.Rollback() }() + upsert, err := tx.PrepareContext(ctx, ` + INSERT INTO protection_postures ( + subject_key, subject_resource_id, state, + posture_json, evaluated_at_ms, updated_at_ms + ) VALUES (?, ?, ?, ?, ?, ?) + ON CONFLICT(subject_key) DO UPDATE SET + subject_resource_id=excluded.subject_resource_id, + state=excluded.state, + posture_json=excluded.posture_json, + evaluated_at_ms=excluded.evaluated_at_ms, + updated_at_ms=excluded.updated_at_ms + `) + if err != nil { + return err + } + defer upsert.Close() + + for _, key := range subjectKeys { + points := pointsByKey[key] + subjectResourceID := "" + for _, point := range points { + if value := strings.TrimSpace(point.SubjectResourceID); value != "" { + subjectResourceID = value + break + } + } + if subjectResourceID == "" { + if _, err := tx.ExecContext( + ctx, + `DELETE FROM protection_postures WHERE subject_key = ?`, + key, + ); err != nil { + return err + } + continue + } + posture := recovery.BuildProtectionPostureFromPointsAt( + subjectResourceID, + points, + observations, + recovery.DefaultProtectionPosturePolicy, + now, + ) + if err := posture.Validate(); err != nil { + return err + } + postureJSON, err := json.Marshal(posture) + if err != nil { + return err + } + if _, err := upsert.ExecContext( + ctx, + key, + subjectResourceID, + string(posture.State), + string(postureJSON), + posture.EvaluatedAt.UTC().UnixMilli(), + nowMs, + ); err != nil { + return err + } + } + return tx.Commit() +} + +func (s *Store) ListProtectionPostures( + ctx context.Context, + query recovery.ProtectionPostureQuery, +) ([]recovery.ProtectionPosture, int, error) { + if err := s.ensureInitialized(); err != nil { + return nil, 0, err + } + if query.State != "" && !query.State.Valid() { + return nil, 0, fmt.Errorf("invalid protection posture state %q", query.State) + } + if ctx == nil { + ctx = context.Background() + } + + requestedIDs := normalizeProtectionStrings(query.SubjectResourceIDs) + if len(requestedIDs) > 0 { + return s.listRequestedProtectionPostures(ctx, requestedIDs, query.State) + } + + limit := normalizeLimit(query.Limit) + page := normalizePage(query.Page) + offset := (page - 1) * limit + whereSQL := "" + args := make([]any, 0, 3) + if query.State != "" { + whereSQL = "WHERE state = ?" + args = append(args, string(query.State)) + } + var total int + if err := s.db.QueryRowContext( + ctx, + "SELECT COUNT(*) FROM protection_postures "+whereSQL, + args..., + ).Scan(&total); err != nil { + return nil, 0, err + } + args = append(args, limit, offset) + rows, err := s.db.QueryContext(ctx, ` + SELECT posture_json + FROM protection_postures + `+whereSQL+` + ORDER BY + CASE state + WHEN 'attention' THEN 0 + WHEN 'unprotected' THEN 1 + WHEN 'unknown' THEN 2 + ELSE 3 + END, + evaluated_at_ms DESC, + subject_resource_id + LIMIT ? OFFSET ? + `, args...) + if err != nil { + return nil, 0, err + } + defer rows.Close() + + postures := make([]recovery.ProtectionPosture, 0, limit) + for rows.Next() { + var postureJSON string + if err := rows.Scan(&postureJSON); err != nil { + return nil, 0, err + } + var posture recovery.ProtectionPosture + if err := json.Unmarshal([]byte(postureJSON), &posture); err != nil { + return nil, 0, err + } + postures = append(postures, posture) + } + return postures, total, rows.Err() +} + +func (s *Store) listRequestedProtectionPostures( + ctx context.Context, + requestedIDs []string, + state recovery.ProtectionState, +) ([]recovery.ProtectionPosture, int, error) { + placeholders := strings.TrimSuffix(strings.Repeat("?,", len(requestedIDs)), ",") + args := make([]any, len(requestedIDs)) + for i := range requestedIDs { + args[i] = requestedIDs[i] + } + rows, err := s.db.QueryContext(ctx, ` + SELECT subject_key, subject_resource_id + FROM protection_postures + WHERE subject_resource_id IN (`+placeholders+`) + `, args...) + if err != nil { + return nil, 0, err + } + defer rows.Close() + + subjectKeyByID := make(map[string]string, len(requestedIDs)) + subjectKeys := make([]string, 0, len(requestedIDs)) + for rows.Next() { + var subjectKey, subjectResourceID string + if err := rows.Scan(&subjectKey, &subjectResourceID); err != nil { + return nil, 0, err + } + subjectKey = strings.TrimSpace(subjectKey) + subjectResourceID = strings.TrimSpace(subjectResourceID) + if subjectKey == "" || subjectResourceID == "" { + continue + } + subjectKeyByID[subjectResourceID] = subjectKey + subjectKeys = append(subjectKeys, subjectKey) + } + if err := rows.Err(); err != nil { + return nil, 0, err + } + if err := rows.Close(); err != nil { + return nil, 0, err + } + + now := time.Now().UTC() + pointsByKey := make(map[string][]recovery.RecoveryPoint) + var observations []recovery.ProtectionProviderObservation + if len(subjectKeys) > 0 { + pointsByKey, err = s.loadProtectionPointsForKeys(ctx, normalizeProtectionStrings(subjectKeys)) + if err != nil { + return nil, 0, err + } + observations, err = s.listLatestProtectionProviderObservations(ctx) + if err != nil { + return nil, 0, err + } + } + out := make([]recovery.ProtectionPosture, 0, len(requestedIDs)) + for _, subjectResourceID := range requestedIDs { + subjectKey := subjectKeyByID[subjectResourceID] + var posture recovery.ProtectionPosture + if subjectKey == "" { + posture = recovery.DeriveProtectionPostureAt( + subjectResourceID, + nil, + recovery.DefaultProtectionPosturePolicy, + now, + ) + } else { + posture = recovery.BuildProtectionPostureFromPointsAt( + subjectResourceID, + pointsByKey[subjectKey], + observations, + recovery.DefaultProtectionPosturePolicy, + now, + ) + } + if state != "" && posture.State != state { + continue + } + out = append(out, posture) + } + return out, len(out), nil +} + +func normalizeProtectionStrings(values []string) []string { + set := make(map[string]struct{}, len(values)) + for _, value := range values { + if value = strings.TrimSpace(value); value != "" { + set[value] = struct{}{} + } + } + out := make([]string, 0, len(set)) + for value := range set { + out = append(out, value) + } + sort.Strings(out) + return out +} diff --git a/internal/recovery/store/store_posture_test.go b/internal/recovery/store/store_posture_test.go new file mode 100644 index 000000000..6fb5767f3 --- /dev/null +++ b/internal/recovery/store/store_posture_test.go @@ -0,0 +1,499 @@ +package store + +import ( + "context" + "encoding/json" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust" + "github.com/rcourtman/pulse-go-rewrite/internal/recovery" +) + +func TestStoreProtectionPosturePersistsProviderAwareTruth(t *testing.T) { + t.Parallel() + + store, err := Open(filepath.Join(t.TempDir(), "recovery.db")) + if err != nil { + t.Fatalf("Open() error = %v", err) + } + defer store.Close() + + now := time.Now().UTC().Truncate(time.Millisecond) + backupTime := now.Add(-time.Hour) + verified := true + point := recovery.RecoveryPoint{ + ID: "pbs-backup:vm-100", + Provider: recovery.ProviderProxmoxPBS, + Kind: recovery.KindBackup, + Mode: recovery.ModeRemote, + Outcome: recovery.OutcomeSuccess, + CompletedAt: &backupTime, + Verified: &verified, + SubjectResourceID: "resource:vm-100", + RepositoryResourceID: "resource:pbs-store-a", + ProviderScope: "pbs-main", + } + evidence, err := recovery.NewRecoveryPointEvidence( + point, + "pbs-backup-inventory", + now, + ) + if err != nil { + t.Fatalf("NewRecoveryPointEvidence() error = %v", err) + } + point.Evidence = evidence + + complete, err := recovery.NewProtectionProviderObservation( + recovery.ProviderProxmoxPBS, + "pbs-backup-enumeration", + "pbs-main", + recovery.OutcomeSuccess, + recovery.ProtectionHistoryComplete, + operationaltrust.EvidencePermissionsSufficient, + true, + now, + now, + nil, + ) + if err != nil { + t.Fatalf("complete observation error = %v", err) + } + if err := store.UpsertProtectionProviderObservations( + context.Background(), + []recovery.ProtectionProviderObservation{complete}, + ); err != nil { + t.Fatalf("UpsertProtectionProviderObservations() error = %v", err) + } + if err := store.UpsertPoints( + context.Background(), + []recovery.RecoveryPoint{point}, + ); err != nil { + t.Fatalf("UpsertPoints() error = %v", err) + } + + got, total, err := store.ListProtectionPostures( + context.Background(), + recovery.ProtectionPostureQuery{ + SubjectResourceIDs: []string{"resource:vm-100", "resource:vm-404"}, + }, + ) + if err != nil { + t.Fatalf("ListProtectionPostures() error = %v", err) + } + if total != 2 || len(got) != 2 { + t.Fatalf("postures total=%d len=%d, want 2/2", total, len(got)) + } + byID := map[string]recovery.ProtectionPosture{} + for _, posture := range got { + byID[posture.SubjectResourceID] = posture + } + protected := byID["resource:vm-100"] + if protected.State != recovery.ProtectionStateProtected { + t.Fatalf("vm-100 state = %q, want protected; posture=%#v", protected.State, protected) + } + if protected.Coverage != recovery.ProtectionCoverageComplete { + t.Fatalf("vm-100 coverage = %q, want complete", protected.Coverage) + } + if protected.Verification != recovery.ProtectionVerificationVerified { + t.Fatalf("vm-100 verification = %q, want verified", protected.Verification) + } + if len(protected.ProviderStates) != 1 { + t.Fatalf("vm-100 provider states = %d, want 1", len(protected.ProviderStates)) + } + if byID["resource:vm-404"].State != recovery.ProtectionStateUnknown { + t.Fatalf( + "vm-404 state = %q, want unknown", + byID["resource:vm-404"].State, + ) + } + + attentionList, total, err := store.ListProtectionPostures( + context.Background(), + recovery.ProtectionPostureQuery{ + State: recovery.ProtectionStateProtected, + Page: 1, + Limit: 10, + }, + ) + if err != nil { + t.Fatalf("ListProtectionPostures(protected) error = %v", err) + } + if total != 1 || len(attentionList) != 1 { + t.Fatalf("protected postures total=%d len=%d, want 1/1", total, len(attentionList)) + } + + partialAt := now.Add(time.Minute) + partial, err := recovery.NewProtectionProviderObservation( + recovery.ProviderProxmoxPBS, + "pbs-backup-enumeration", + "pbs-main", + recovery.OutcomeWarning, + recovery.ProtectionHistoryPartial, + operationaltrust.EvidencePermissionsPartial, + true, + partialAt, + partialAt, + &operationaltrust.EvidenceReason{ + Code: "pbs_partial_enumeration", + Message: "One PBS datastore could not be enumerated.", + }, + ) + if err != nil { + t.Fatalf("partial observation error = %v", err) + } + if err := store.UpsertProtectionProviderObservations( + context.Background(), + []recovery.ProtectionProviderObservation{partial}, + ); err != nil { + t.Fatalf("UpsertProtectionProviderObservations(partial) error = %v", err) + } + got, _, err = store.ListProtectionPostures( + context.Background(), + recovery.ProtectionPostureQuery{ + SubjectResourceIDs: []string{"resource:vm-100"}, + }, + ) + if err != nil { + t.Fatalf("ListProtectionPostures(after partial) error = %v", err) + } + if got[0].State != recovery.ProtectionStateAttention { + t.Fatalf("state after partial = %q, want attention", got[0].State) + } + if got[0].Coverage != recovery.ProtectionCoveragePartial { + t.Fatalf("coverage after partial = %q, want partial", got[0].Coverage) + } +} + +func TestStoreProtectionSchemaMigratesLegacyRecoveryDatabase(t *testing.T) { + t.Parallel() + + dbPath := filepath.Join(t.TempDir(), "recovery.db") + store, err := Open(dbPath) + if err != nil { + t.Fatalf("Open() error = %v", err) + } + defer store.Close() + + for _, table := range []string{ + "protection_provider_observations", + "protection_postures", + } { + var name string + if err := store.db.QueryRow( + `SELECT name FROM sqlite_master WHERE type = 'table' AND name = ?`, + table, + ).Scan(&name); err != nil { + t.Fatalf("lookup table %s: %v", table, err) + } + if name != table { + t.Fatalf("table = %q, want %q", name, table) + } + } + + columns := map[string]bool{} + rows, err := store.db.Query(`PRAGMA table_info(recovery_points)`) + if err != nil { + t.Fatalf("PRAGMA table_info: %v", err) + } + defer rows.Close() + for rows.Next() { + var ( + cid int + name string + columnType string + notNull int + defaultValue any + primaryKey int + ) + if err := rows.Scan( + &cid, + &name, + &columnType, + ¬Null, + &defaultValue, + &primaryKey, + ); err != nil { + t.Fatalf("scan table_info: %v", err) + } + columns[name] = true + } + for _, column := range []string{"provider_scope", "evidence_id", "evidence_json"} { + if !columns[column] { + t.Fatalf("missing migrated recovery_points column %q", column) + } + } +} + +func TestStoreRequestedProtectionPosturesReevaluateAtReadTime(t *testing.T) { + t.Parallel() + + store, err := Open(filepath.Join(t.TempDir(), "recovery.db")) + if err != nil { + t.Fatalf("Open() error = %v", err) + } + defer store.Close() + + now := time.Now().UTC().Truncate(time.Millisecond) + oldBackup := now.Add(-8 * 24 * time.Hour) + verified := true + point := recovery.RecoveryPoint{ + ID: "pbs-backup:stale-vm", + Provider: recovery.ProviderProxmoxPBS, + Kind: recovery.KindBackup, + Mode: recovery.ModeRemote, + Outcome: recovery.OutcomeSuccess, + CompletedAt: &oldBackup, + Verified: &verified, + SubjectResourceID: "resource:stale-vm", + ProviderScope: "pbs-main", + } + evidence, err := recovery.NewRecoveryPointEvidence(point, "pbs-backup-inventory", now) + if err != nil { + t.Fatalf("NewRecoveryPointEvidence() error = %v", err) + } + point.Evidence = evidence + observation, err := recovery.NewProtectionProviderObservation( + recovery.ProviderProxmoxPBS, + "pbs-backup-enumeration", + "pbs-main", + recovery.OutcomeSuccess, + recovery.ProtectionHistoryComplete, + operationaltrust.EvidencePermissionsSufficient, + true, + now, + now, + nil, + ) + if err != nil { + t.Fatalf("NewProtectionProviderObservation() error = %v", err) + } + if err := store.UpsertProtectionProviderObservations( + context.Background(), + []recovery.ProtectionProviderObservation{observation}, + ); err != nil { + t.Fatalf("UpsertProtectionProviderObservations() error = %v", err) + } + if err := store.UpsertPoints(context.Background(), []recovery.RecoveryPoint{point}); err != nil { + t.Fatalf("UpsertPoints() error = %v", err) + } + + // Corrupt the materialized snapshot into a reassuring answer. A requested + // batch must derive from points + current provider evidence instead of + // trusting a posture that can age between collection cycles. + fake := recovery.DeriveProtectionPostureAt( + point.SubjectResourceID, + nil, + recovery.DefaultProtectionPosturePolicy, + now, + ) + fake.State = recovery.ProtectionStateProtected + fake.Freshness = recovery.ProtectionFreshnessCurrent + fake.Verification = recovery.ProtectionVerificationVerified + fake.Coverage = recovery.ProtectionCoverageComplete + fake.Explanation = "incorrect stored posture" + fakeJSON, err := json.Marshal(fake) + if err != nil { + t.Fatalf("json.Marshal(fake) error = %v", err) + } + if _, err := store.db.Exec( + `UPDATE protection_postures SET state = 'protected', posture_json = ?`, + string(fakeJSON), + ); err != nil { + t.Fatalf("update materialized posture: %v", err) + } + + got, _, err := store.ListProtectionPostures( + context.Background(), + recovery.ProtectionPostureQuery{ + SubjectResourceIDs: []string{point.SubjectResourceID}, + }, + ) + if err != nil { + t.Fatalf("ListProtectionPostures() error = %v", err) + } + if len(got) != 1 { + t.Fatalf("postures len = %d, want 1", len(got)) + } + if got[0].State != recovery.ProtectionStateAttention { + t.Fatalf("state = %q, want attention; posture=%#v", got[0].State, got[0]) + } + if got[0].Freshness != recovery.ProtectionFreshnessStale { + t.Fatalf("freshness = %q, want stale", got[0].Freshness) + } + if got[0].Explanation == "incorrect stored posture" { + t.Fatal("requested posture trusted stale materialized JSON") + } +} + +func TestStoreProtectionPostureBatchUsesIndexedBoundedReads(t *testing.T) { + t.Parallel() + + store, err := Open(filepath.Join(t.TempDir(), "recovery.db")) + if err != nil { + t.Fatalf("Open() error = %v", err) + } + defer store.Close() + + rows, err := store.db.Query(` + EXPLAIN QUERY PLAN + SELECT subject_key, subject_resource_id + FROM protection_postures + WHERE subject_resource_id IN (?, ?, ?) + `, "resource:a", "resource:b", "resource:c") + if err != nil { + t.Fatalf("EXPLAIN QUERY PLAN error = %v", err) + } + defer rows.Close() + var details []string + for rows.Next() { + var id, parent, notUsed int + var detail string + if err := rows.Scan(&id, &parent, ¬Used, &detail); err != nil { + t.Fatalf("scan query plan: %v", err) + } + details = append(details, detail) + } + if err := rows.Err(); err != nil { + t.Fatalf("query plan rows: %v", err) + } + plan := strings.Join(details, "\n") + if !strings.Contains(plan, "SEARCH protection_postures") || + !strings.Contains(plan, "subject_resource_id=?") { + t.Fatalf("posture batch query is not index-backed:\n%s", plan) + } +} + +func TestStoreProtectionEvidenceRetentionRefreshesMaterializedPosture(t *testing.T) { + t.Parallel() + + store, err := Open(filepath.Join(t.TempDir(), "recovery.db")) + if err != nil { + t.Fatalf("Open() error = %v", err) + } + defer store.Close() + store.retention = time.Hour + store.lastPrune = time.Now().UTC() + + now := time.Now().UTC().Truncate(time.Millisecond) + old := now.Add(-2 * time.Hour) + point := recovery.RecoveryPoint{ + ID: "pbs-backup:expired", + Provider: recovery.ProviderProxmoxPBS, + Kind: recovery.KindBackup, + Mode: recovery.ModeRemote, + Outcome: recovery.OutcomeSuccess, + CompletedAt: &old, + SubjectResourceID: "resource:expired", + ProviderScope: "pbs-main", + } + observation, err := recovery.NewProtectionProviderObservation( + recovery.ProviderProxmoxPBS, + "pbs-backup-enumeration", + "pbs-main", + recovery.OutcomeSuccess, + recovery.ProtectionHistoryComplete, + operationaltrust.EvidencePermissionsSufficient, + false, + old, + old, + nil, + ) + if err != nil { + t.Fatalf("NewProtectionProviderObservation() error = %v", err) + } + if err := store.UpsertProtectionProviderObservations( + context.Background(), + []recovery.ProtectionProviderObservation{observation}, + ); err != nil { + t.Fatalf("UpsertProtectionProviderObservations() error = %v", err) + } + if err := store.UpsertPoints(context.Background(), []recovery.RecoveryPoint{point}); err != nil { + t.Fatalf("UpsertPoints() error = %v", err) + } + + store.lastPrune = time.Time{} + store.maybePrune(context.Background()) + + for _, table := range []string{ + "recovery_points", + "protection_provider_observations", + "protection_postures", + } { + var count int + if err := store.db.QueryRow("SELECT COUNT(*) FROM " + table).Scan(&count); err != nil { + t.Fatalf("count %s: %v", table, err) + } + if count != 0 { + t.Fatalf("%s count = %d, want 0 after retention prune", table, count) + } + } +} + +func TestStoreProtectionMetadataBackfillAddsScopeAndTypedEvidence(t *testing.T) { + t.Parallel() + + store, err := Open(filepath.Join(t.TempDir(), "recovery.db")) + if err != nil { + t.Fatalf("Open() error = %v", err) + } + defer store.Close() + + observedAt := time.Now().UTC().Add(-time.Hour).Truncate(time.Millisecond) + updatedAt := observedAt.Add(time.Minute) + if _, err := store.db.Exec(` + INSERT INTO recovery_points ( + id, provider, kind, mode, outcome, + completed_at_ms, subject_key, subject_resource_id, + repository_ref_json, created_at_ms, updated_at_ms + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + `, + "pbs-backup:legacy", + string(recovery.ProviderProxmoxPBS), + string(recovery.KindBackup), + string(recovery.ModeRemote), + string(recovery.OutcomeSuccess), + observedAt.UnixMilli(), + "res:resource:legacy", + "resource:legacy", + `{"type":"proxmox-pbs-datastore","namespace":"pbs-main","name":"store-a"}`, + observedAt.UnixMilli(), + updatedAt.UnixMilli(), + ); err != nil { + t.Fatalf("insert legacy recovery point: %v", err) + } + + if err := store.BackfillProtectionMetadata(context.Background()); err != nil { + t.Fatalf("BackfillProtectionMetadata() error = %v", err) + } + + var providerScope, evidenceID, evidenceRaw string + if err := store.db.QueryRow(` + SELECT provider_scope, evidence_id, evidence_json + FROM recovery_points + WHERE id = 'pbs-backup:legacy' + `).Scan(&providerScope, &evidenceID, &evidenceRaw); err != nil { + t.Fatalf("read migrated recovery point: %v", err) + } + if providerScope != "pbs-main" { + t.Fatalf("provider_scope = %q, want pbs-main", providerScope) + } + var evidence operationaltrust.EvidenceEnvelope + if err := json.Unmarshal([]byte(evidenceRaw), &evidence); err != nil { + t.Fatalf("decode evidence: %v", err) + } + if evidenceID == "" || evidence.ID != evidenceID { + t.Fatalf("evidence id mismatch column=%q payload=%q", evidenceID, evidence.ID) + } + if evidence.Source.Collector != "recovery-point-migration" { + t.Fatalf("collector = %q, want recovery-point-migration", evidence.Source.Collector) + } + if !evidence.ObservedAt.Equal(observedAt) { + t.Fatalf("observedAt = %s, want %s", evidence.ObservedAt, observedAt) + } + if !evidence.IngestedAt.Equal(updatedAt) { + t.Fatalf("ingestedAt = %s, want %s", evidence.IngestedAt, updatedAt) + } +} diff --git a/internal/recovery/types.go b/internal/recovery/types.go index 0f8f46fa0..af40bb7fb 100644 --- a/internal/recovery/types.go +++ b/internal/recovery/types.go @@ -60,3 +60,55 @@ const ( VerifyIntentStale VerifyIntent = model.VerifyIntentStale VerifyIntentUnknown VerifyIntent = model.VerifyIntentUnknown ) + +type ProtectionState = model.ProtectionState + +const ( + ProtectionStateProtected ProtectionState = model.ProtectionStateProtected + ProtectionStateAttention ProtectionState = model.ProtectionStateAttention + ProtectionStateUnprotected ProtectionState = model.ProtectionStateUnprotected + ProtectionStateUnknown ProtectionState = model.ProtectionStateUnknown +) + +type ProtectionFreshness = model.ProtectionFreshness + +const ( + ProtectionFreshnessCurrent ProtectionFreshness = model.ProtectionFreshnessCurrent + ProtectionFreshnessStale ProtectionFreshness = model.ProtectionFreshnessStale + ProtectionFreshnessUnknown ProtectionFreshness = model.ProtectionFreshnessUnknown +) + +type ProtectionVerification = model.ProtectionVerification + +const ( + ProtectionVerificationVerified ProtectionVerification = model.ProtectionVerificationVerified + ProtectionVerificationUnverified ProtectionVerification = model.ProtectionVerificationUnverified + ProtectionVerificationStale ProtectionVerification = model.ProtectionVerificationStale + ProtectionVerificationUnknown ProtectionVerification = model.ProtectionVerificationUnknown +) + +type ProtectionCoverage = model.ProtectionCoverage + +const ( + ProtectionCoverageComplete ProtectionCoverage = model.ProtectionCoverageComplete + ProtectionCoveragePartial ProtectionCoverage = model.ProtectionCoveragePartial + ProtectionCoverageNone ProtectionCoverage = model.ProtectionCoverageNone + ProtectionCoverageUnknown ProtectionCoverage = model.ProtectionCoverageUnknown +) + +type ProtectionHistoryCompleteness = model.ProtectionHistoryCompleteness + +const ( + ProtectionHistoryComplete ProtectionHistoryCompleteness = model.ProtectionHistoryComplete + ProtectionHistoryPartial ProtectionHistoryCompleteness = model.ProtectionHistoryPartial + ProtectionHistoryUnavailable ProtectionHistoryCompleteness = model.ProtectionHistoryUnavailable + ProtectionHistoryUnknown ProtectionHistoryCompleteness = model.ProtectionHistoryUnknown +) + +type ProtectionProviderState = model.ProtectionProviderState +type ProtectionPosture = model.ProtectionPosture +type ProtectionProviderObservation = model.ProtectionProviderObservation +type ProtectionPosturePolicy = model.ProtectionPosturePolicy +type ProtectionPosturePolicyPayload = model.ProtectionPosturePolicyPayload +type ProtectionPostureQuery = model.ProtectionPostureQuery +type ProtectionProviderSummary = model.ProtectionProviderSummary diff --git a/scripts/release_control/canonical_completion_guard_test.py b/scripts/release_control/canonical_completion_guard_test.py index f9d92876d..d64ac83ac 100644 --- a/scripts/release_control/canonical_completion_guard_test.py +++ b/scripts/release_control/canonical_completion_guard_test.py @@ -230,6 +230,7 @@ class CanonicalCompletionGuardTest(unittest.TestCase): "proxmox-backup-identity-monitoring", "container-entrypoint-runtime", "mock-runtime-fixtures", + "pbs-protection-evidence-runtime", "monitoring-runtime", ], ) diff --git a/tests/integration/tests/90-operational-trust-protection-posture.spec.ts b/tests/integration/tests/90-operational-trust-protection-posture.spec.ts new file mode 100644 index 000000000..243452c2e --- /dev/null +++ b/tests/integration/tests/90-operational-trust-protection-posture.spec.ts @@ -0,0 +1,81 @@ +import { expect, test } from "@playwright/test"; +import { ensureAuthenticated } from "./helpers"; + +const DESKTOP_VIEWPORT = { width: 1440, height: 900 }; + +test.describe("Operational trust protection posture", () => { + test.setTimeout(180_000); + + test("uses one bounded posture query and keeps evidence in the workload drill-down", async ({ + page, + }, testInfo) => { + test.skip( + testInfo.project.name.startsWith("mobile-"), + "The desktop journey proves the full evidence drill-down", + ); + + const postureRequests: string[] = []; + page.on("request", (request) => { + if (request.url().includes("/api/recovery/postures")) { + postureRequests.push(request.url()); + } + }); + + await page.setViewportSize(DESKTOP_VIEWPORT); + await ensureAuthenticated(page); + await page.goto("/proxmox/backups", { waitUntil: "domcontentloaded" }); + await page.getByRole("button", { name: "Coverage" }).click(); + + const coverageTable = page + .locator("div.overflow-x-auto") + .filter({ has: page.locator('th:has-text("Posture")') }) + .first(); + await expect(coverageTable).toBeVisible(); + await expect.poll(() => postureRequests.length).toBe(1); + + const postureURL = new URL(postureRequests[0]); + const requestedResourceIDs = postureURL.searchParams.getAll("resourceId"); + expect(requestedResourceIDs.length).toBeGreaterThan(0); + expect(requestedResourceIDs.length).toBeLessThanOrEqual(200); + + // The default monitor stays compact. Explanation and provider/restore + // evidence appear only after the operator asks for the row's details. + await expect(coverageTable.getByText("Provider evidence")).toHaveCount(0); + const detailToggle = coverageTable + .locator('button[aria-label^="Expand details for"]') + .first(); + await expect(detailToggle).toBeVisible(); + await detailToggle.click(); + + const detailRow = coverageTable.locator("[data-inline-detail-for]").first(); + await expect(detailRow).toBeVisible(); + await expect(detailRow).toContainText( + /Protected:|Attention:|Unprotected:|Unknown:/, + ); + await expect(detailRow).toContainText( + /Restore evidence|No restore evidence has been discovered/, + ); + }); + + test("contains wide evidence tables on a phone-sized viewport", async ({ + page, + }, testInfo) => { + test.skip( + !testInfo.project.name.startsWith("mobile-"), + "The mobile projects own responsive containment", + ); + + await ensureAuthenticated(page); + await page.goto("/proxmox/backups", { waitUntil: "domcontentloaded" }); + await page.getByRole("button", { name: "Coverage" }).click(); + await expect(page.locator('th:has-text("Posture")').first()).toBeVisible(); + + const viewport = await page.evaluate(() => ({ + bodyScrollWidth: document.body.scrollWidth, + clientWidth: document.documentElement.clientWidth, + })); + expect(viewport.bodyScrollWidth).toBeLessThanOrEqual( + viewport.clientWidth + 1, + ); + }); +});