gsadmin/pulse
1
0
Fork 0
mirror of https://github.com/rcourtman/Pulse.git synced 2026-09-10 02:25:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

Build canonical protection posture

Browse Source
This commit is contained in:
rcourtman
2026-07-19 01:46:22 +01:00
parent cf0486492e
commit 93dbdd91cf
44 changed files with 4872 additions and 256 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/API.md
+7 -1
View File
@@ -484,7 +484,8 @@ Detailed storage usage per node and pool.
### Recovery (formerly Backups / Snapshots)
Pulse v6 uses the recovery API to provide a platform-agnostic view of backup and snapshot artifacts.
See `docs/architecture/RECOVERY_CONTRACT.md` for the provider-neutral contract (subjects, points, rollups, and filter semantics).
See `docs/architecture/RECOVERY_CONTRACT.md` for the provider-neutral contract
(subjects, points, rollups, posture, and filter semantics).
- `GET /api/recovery/points`
- Query params:
@@ -494,6 +495,11 @@ See `docs/architecture/RECOVERY_CONTRACT.md` for the provider-neutral contract (
- Normalized filters: `q`, `cluster`, `node`, `namespace`, `scope=workload`, `verification` (`verified` | `unverified` | `unknown`)
- `GET /api/recovery/rollups`
- Query params: `provider`, `kind`, `mode`, `outcome`, `subjectResourceId`, `rollupId`, `from` (RFC3339), `to` (RFC3339), `page`, `limit`
- `GET /api/recovery/postures`
- Returns server-derived per-resource protection posture and provider evidence quality.
- Query params: repeated `resourceId` values (maximum 200), `state` (`protected` | `attention` | `unprotected` | `unknown`), `page`, `limit` (maximum 200)
- Batch clients must make one bounded request per 200 resource ids, never one request per table row.
- Unknown identity, permission, history, or collection completeness remains `unknown`; clients must not infer a healthier state from raw backup or snapshot artifacts.
- `GET /api/recovery/series`
- Returns per-day counts for the activity chart.
- Query params: same filters as `/api/recovery/points` (except paging), plus `tzOffsetMinutes` (integer; UTC offset minutes for day bucketing)
docs/RECOVERY.md
+42 -5
View File
@@ -7,7 +7,7 @@ Pulse v6 includes a **provider-neutral recovery view** that aggregates backup, s
Recovery is event-first and answers two questions:
1. **"What happened?"** → The **Recovery events** table shows individual recovery points (artifacts) with timestamps, outcomes, and sizes.
2. **"What is covered?"** -> The **Protection coverage** action opens a rollup review for healthy, stale, failed, warning, running, unknown, and never-succeeded items.
2. **"What can I actually recover?"** → **Protection coverage** shows the canonical posture for each resource: protected, attention, unprotected, or unknown.
## Supported Providers
@@ -47,6 +47,25 @@ A rollup groups recovery points for a subject to show:
- **Latest point** — when was the most recent successful backup/snapshot?
- **Health** — are there recent failures or warnings?
### Protection Posture (A Trust Decision)
A protection posture combines subject-linked recovery points with the latest
provider collection evidence. It deliberately keeps four operator-facing
states:
- **Protected** — a qualifying current recovery point is linked to the resource
and complete provider evidence does not invalidate the claim.
- **Attention** — evidence exists, but it is stale, failing, incomplete, or
unverified when verification is expected.
- **Unprotected** — complete evidence confirms that no qualifying protection
exists.
- **Unknown** — identity, permissions, provider history, or collection
completeness cannot support a stronger claim.
A backup or snapshot artifact may still be shown while posture is unknown.
Artifacts answer what Pulse found; posture answers what Pulse can safely claim.
Snapshot presence alone is never presented as independent recovery.
## Navigating Recovery
### Recovery Events
@@ -64,15 +83,17 @@ Shows individual recovery points. Key columns:
### Protection Coverage
Available from the Recovery header action and compatibility routes when posture or freshness needs review. It shows one row per protected item (or per item + method when multiple backup methods exist). Key columns:
The Proxmox **Backups → Coverage** view shows one row per workload. The default
table stays compact; expanding a row reveals the plain-language posture reason,
provider evidence quality, and individual restore artifacts.
| Column | Description |
|---|---|
| Item | The protected resource (VM name, dataset path, etc.) |
| Item Type | Canonical resource category |
| Platform | Which provider created the latest point (PVE, PBS, TrueNAS, Kubernetes) |
| Latest Point | Most recent successful recovery point timestamp |
| Status | Healthy, stale, failed, warning, running, unknown, or never succeeded |
| Posture | Protected, attention, unprotected, or unknown |
| Restore | Most recent successful recovery point timestamp |
| Provider columns | Latest PBS, PVE, or guest-snapshot artifact where available |
### Filtering
@@ -89,6 +110,7 @@ Both workspaces support:
|---|---|---|
| `GET` | `/api/recovery/points` | List individual recovery points |
| `GET` | `/api/recovery/rollups` | List subject rollups (protection coverage) |
| `GET` | `/api/recovery/postures` | List canonical per-resource protection postures |
| `GET` | `/api/recovery/series` | Time-series data for recovery charts |
| `GET` | `/api/recovery/facets` | Available filter facets (providers, kinds, outcomes) |
@@ -106,6 +128,13 @@ All recovery endpoints support:
| `subject` | Filter by subject reference |
| `limit` | Max results (default: 500) |
`/api/recovery/postures` has a deliberately bounded table contract. Supply one
or more repeated `resourceId` parameters for a resource or batch lookup (at
most 200), or omit them for a paged list. It also accepts `state`, `page`, and
`limit`; `state=attention` returns the actionable attention list. The response
includes the posture policy and provider evidence states so clients do not
re-derive trust from raw artifacts.
## Troubleshooting
### No recovery data showing
@@ -129,6 +158,14 @@ Click the row to expand the details drawer, which shows the provider-specific er
- **TrueNAS**: Replication target unreachable, dataset locked, insufficient space
- **PVE**: Backup storage full, vzdump process error
### Current backups show an unknown posture
Expand the workload row and inspect the limiting evidence. Pulse uses unknown
when the current poll cannot prove provider-history completeness, permission
scope, or subject identity. Fix the reported collection or access gap and wait
for the next provider poll; Pulse does not promote retained backup artifacts to
protected while that uncertainty remains.
## See Also
- [PBS Integration](PBS.md) — Proxmox Backup Server monitoring
docs/release-control/v6/internal/records/operational-trust-protection-posture-2026-07-19.md
+132
View File
@@ -0,0 +1,132 @@
# Operational Trust Protection Posture Record
Date: 2026-07-19
## Scope
This record closes the Phase 2 protection-posture slice in
`OPERATIONAL_TRUST_IMPLEMENTATION_SPEC.md`. It does not claim completion of
the Patrol attention workbench, availability attachment, governed actions, or
the full Operational Trust specification.
## User job and evidence
The least-expert plausible user's job is: “Tell me which workloads I can
actually recover, which need attention, and why.”
Relevant issue history shows why artifact presence cannot be the answer:
1. [#1541](https://github.com/rcourtman/Pulse/issues/1541) reports current PBS
backups displayed with an incorrect unverified conclusion.
2. [#81](https://github.com/rcourtman/Pulse/issues/81) records a token that can
list datastores but cannot enumerate backup snapshots, proving that partial
permission must remain visible rather than becoming a healthy empty result.
3. [#1389](https://github.com/rcourtman/Pulse/issues/1389) records VM and
container backups mis-correlated by numeric id, proving that provider-scoped
subject identity is part of protection evidence.
4. [#1592](https://github.com/rcourtman/Pulse/issues/1592) and
[#1437](https://github.com/rcourtman/Pulse/issues/1437) report missing PVE
backups and snapshots, proving that one provider's visible artifacts do not
establish complete cross-provider coverage.
5. [#1056](https://github.com/rcourtman/Pulse/issues/1056) and
[#1162](https://github.com/rcourtman/Pulse/issues/1162) show the operator
cost of treating retained orphan artifacts as current workload protection.
## Canonical runtime result
1. `internal/recovery/model/posture.go` defines the provider-neutral posture,
provider-state, policy, query, and provider-observation contracts. The only
customer-facing states are protected, attention, unprotected, and unknown.
2. `internal/recovery/posture.go` derives posture by canonical resource id and
provider scope. Protected requires a current subject-linked backup plus
complete sufficient collection evidence. Attention covers actionable stale,
failed, partial, or expected-but-unverified evidence. Unprotected requires
complete evidence of no qualifying backup. Unknown is mandatory for
identity, permission, history, or collection uncertainty.
3. Backup and snapshot semantics remain distinct. A snapshot alone never
proves independent recovery, provider job success without a subject-linked
point never becomes subject protection, and a limitation from an unrelated
provider cannot erase a confirmed complete PBS recovery.
4. Recovery-point persistence adds provider scope and typed evidence
additively. Existing databases migrate in place, and legacy points receive
explicit unknown-quality evidence rather than invented certainty.
5. Provider observations and materialized postures have indexed, bounded,
retained storage. Requested postures are re-evaluated from current points
and observations at read time so a reassuring stored row cannot age into a
lie. Point, observation, and posture retention share the 90-day boundary.
6. PBS is the first explicit provider adapter. Every poll records complete,
partial, unavailable, or denied collection evidence separately from its
subject-linked recovery points. Collection evidence is persisted before the
point batch and reconciliation, so a large point write, timeout, cached
artifact path, or failed enumeration cannot preserve a healthier claim.
7. `GET /api/recovery/postures` supplies one-resource, bounded batch, paged
list, and `state=attention` reads under `monitoring:read`. Resource batches
are capped at 200, return unknown for requested ids without sufficient
evidence, and include policy plus provider evidence for drill-down.
8. `useProtectionPostures` sorts and deduplicates canonical resource ids,
fetches one batch per 200 resources, and exposes a keyed read model. Proxmox
passes exact unified-resource ids and does not parse table keys or derive
posture from raw PBS, PVE, or guest-snapshot artifacts.
9. The Proxmox Backups coverage table keeps the default monitor compact. Each
workload has one canonical status and latest restore time; the row
drill-down contains the plain-language reason, human-readable provider
evidence quality, and bounded restore artifacts.
10. `docs/API.md` and `docs/RECOVERY.md` document the bounded API, the
artifact-versus-posture distinction, the four states, and unknown-state
troubleshooting.
## Failure, migration, retention, and performance proof
The Phase 2 proof covers:
1. the provider-aware derivation truth table, including stale, failed,
unverified, partial, denied, missing-identity, snapshot-only, complete-empty,
and mixed-provider cases
2. schema migration and legacy evidence backfill
3. read-time re-evaluation over a deliberately corrupted reassuring
materialization
4. indexed query plans and bounded 200-resource API batches
5. recovery-point, provider-observation, and materialized-posture retention
6. complete, partial, unavailable, and denied PBS collection mappings
7. provider-observation persistence before a deliberately failed point write
8. API authorization, validation, pagination, attention filtering, missing
requested resources, and compatibility serialization
9. frontend batch normalization, one-call table integration, state filtering,
evidence drill-down, type-check, and Proxmox regression tests
## Live product and user-lens proof
The changed surface was exercised against the managed live runtime at
`/proxmox/backups`, through the deepest row expansion, at desktop and 390px
mobile widths.
1. Distance to goal is the existing Proxmox Backups route, the Coverage
selector, then one workload expansion for the reason and evidence.
2. The live pass found an inherited presentation defect: the explanation and
provider evidence were initially repeated inside every PBS table cell. That
detail was demoted to the expansion. The resulting workload row is 33px
high and the default table remains scannable.
3. Default-visible elements all support the job: the four-state summary
identifies fleet posture, filters narrow the queue, the table answers per
workload, and the expansion explains the claim. Raw artifacts are retained
only as forensic restore evidence.
4. Provider codes were replaced with product vocabulary in the expansion
(`Proxmox Backup Server`, `Proxmox VE`), and evidence-quality values are
presented as readable labels.
5. The live dataset honestly remained unknown where no complete provider
collection observation had yet been persisted, despite current verified PBS
artifacts. This is the required fail-closed behavior, not a healthy guess.
6. The phone-sized page has no body-level horizontal overflow; wide evidence
tables scroll inside their bounded table wrappers.
7. The durable Playwright guard proves one bounded posture request, compact
default presentation, evidence-on-expansion, and mobile containment.
The Phase 2 surface is classified as `product`.
## Boundary carried forward
Phase 2 supplies canonical protection truth but does not invent a work queue.
Phase 3 must project lifecycle records into typed attention items and consume
this posture as one input. Patrol must not parse recovery metadata, raw
artifacts, provider job payloads, or table presentation state to create its own
protection verdict.
docs/release-control/v6/internal/status.json
+25
View File
@@ -8727,6 +8727,16 @@
"path": "docs/release-control/v6/internal/records/operational-trust-alert-state-boundary-2026-07-18.md",
"kind": "file"
},
{
"repo": "pulse",
"path": "docs/release-control/v6/internal/records/operational-trust-lifecycle-evidence-notification-linkage-2026-07-19.md",
"kind": "file"
},
{
"repo": "pulse",
"path": "docs/release-control/v6/internal/records/operational-trust-protection-posture-2026-07-19.md",
"kind": "file"
},
{
"repo": "pulse",
"path": "docs/release-control/v6/internal/records/protection-posture-attention-queue-strong-version-2026-06-25.md",
@@ -8776,6 +8786,21 @@
"repo": "pulse",
"path": "internal/monitoring/proxmox_action_observer.go",
"kind": "file"
},
{
"repo": "pulse",
"path": "internal/recovery/posture.go",
"kind": "file"
},
{
"repo": "pulse",
"path": "internal/recovery/store/store_posture.go",
"kind": "file"
},
{
"repo": "pulse",
"path": "tests/integration/tests/90-operational-trust-protection-posture.spec.ts",
"kind": "file"
}
]
},
docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+6
View File
@@ -4619,3 +4619,9 @@ header alone is not identity. This lets a Basic-authenticated qualification
operator approve or reject an agent action while preserving the agent token's
separate reporting and command authority; invalid credentials must never reach
the action handler or acquire an operator principal.
The adjacent recovery handlers under `internal/api/` do not widen this agent
lifecycle boundary. Protection posture is a read-only `monitoring:read`
projection over recovery points and provider collection evidence. It does not
register agents, issue or rotate credentials, interpret observer responses,
or grant backup, restore, command, or remote-configuration authority.
docs/release-control/v6/internal/subsystems/api-contracts.md
+17
View File
@@ -7764,3 +7764,20 @@ invalid Basic credentials attach no principal and fail before the handler.
This is required for qualification clients and local operators to read,
approve, reject, and execute governed actions without weakening the same
capability checks for sessions, proxy users, or API tokens.
### Protection posture transport
`GET /api/recovery/postures` is the authenticated `monitoring:read` transport
for the canonical storage/recovery posture model. It accepts one or repeated
canonical `resourceId` values, rejects more than 200 unique IDs, and supports a
bounded state-filtered paginated list. Responses carry typed posture data,
provider-specific history and permission limitations, evidence references,
the server evaluation policy, evaluation timestamps, and stable pagination
metadata. A requested resource with insufficient evidence returns an explicit
`unknown` posture rather than disappearing or inheriting a frontend guess.
Existing recovery point and rollup routes remain additive compatibility
boundaries. Recovery points now expose provider scope and typed evidence, and
rollups expose their already-owned verification intent and last-verification
time. Supported legacy provider, subject, and display aliases remain readable;
they do not create a parallel posture authority.
docs/release-control/v6/internal/subsystems/frontend-primitives.md
+16
View File
@@ -5048,3 +5048,19 @@ the matching Open or History subtab from server-authored lifecycle state, and
removes the query when the dialog closes. Feature pages may summarize action
context, but they must not recreate approve, reject, run, progress, or outcome
controls outside the shared Actions review.
### Protection posture presentation boundary
Platform coverage tables render the storage/recovery-owned four-state
`ProtectionPosture` contract without inventing age, failure, verification, or
coverage states from raw browser payloads. Protected uses the shared success
tone, attention uses warning, unprotected uses danger, and unknown uses muted
presentation. The compact row remains actionable: plain-language rationale and
provider history/permission limitations live one disclosure deeper beside
bounded restore evidence. A posture fetch failure must keep the evidence
inspectable and show unknown, never a locally inferred healthy fallback.
Protection table controls continue to compose shared filter, table, status,
counter, loading, error, disclosure, and inline-detail primitives. The bounded
batch hook retains fulfilled values during refresh and issues at most one
request per 200 resource rows, rather than placing a request under each row.
docs/release-control/v6/internal/subsystems/monitoring.md
+22
View File
@@ -155,6 +155,7 @@ resource health.
54. `internal/monitoring/monitor_backups.go`
55. `internal/monitoring/resource_stale_thresholds.go`
56. `internal/monitoring/recovery_ingest.go`
56a. `internal/monitoring/pbs_protection_observation.go`
57. `internal/monitoring/multi_tenant_monitor.go`
58. `internal/monitoring/proxmox_action_observer.go`
@@ -1733,3 +1734,24 @@ delivery status and timestamp alongside guest power state. Downstream Patrol
transition detection consumes that status instead of inventing a fixed stale
window: a stopped guest can have fresh inventory, while a stale source cannot
authoritatively prove either a stopped transition or recovery.
### PBS protection evidence collection
Direct PBS backup enumeration emits two separate storage/recovery inputs:
subject-linked recovery points and one typed provider observation for the
polled PBS instance. A complete poll records complete history with sufficient
permissions; a partially successful poll records partial history and the
appropriate partial or unknown permission posture; total transient failure
records unavailable history; total terminal authorization failure records
denied access. Retained backup points survive failed enumeration, but the new
provider observation immediately prevents those cached points from being
presented as current protection truth.
PBS mapping attaches provider scope and a typed evidence envelope to every
successfully enumerated recovery point. Identity correlation is confirmed only
for direct canonical identity and inferred only for an auditable unique
provider-scoped guest match. Monitoring persists the collection observation
before point reconciliation so completeness and permission failure cannot be
lost behind a successful cached-artifact path. Shared protection semantics stay
in `internal/recovery/`; PBS monitoring owns only this explicit evidence-quality
adapter.
docs/release-control/v6/internal/subsystems/registry.json
+57 -2
View File
@@ -418,6 +418,14 @@
],
"rationale": "Proxmox backup server table rows are both a storage/recovery backup-health surface and a unified-resource platform-table consumer boundary"
},
{
"path": "frontend-modern/src/features/proxmox/ProxmoxCoverageTable.tsx",
"subsystems": [
"storage-recovery",
"unified-resources"
],
"rationale": "Proxmox workload coverage rows are both a storage/recovery protection-posture surface and a unified-resource identity consumer boundary"
},
{
"path": "frontend-modern/src/features/proxmox/ProxmoxRecoverableTable.tsx",
"subsystems": [
@@ -2538,6 +2546,23 @@
"internal/api/diagnostics_memory_test.go"
]
},
{
"id": "protection-posture-api-contract",
"label": "bounded protection posture API proof",
"match_prefixes": [],
"match_files": [
"internal/api/recovery_handlers.go",
"internal/api/router_routes_monitoring.go"
],
"allow_same_subsystem_tests": false,
"test_prefixes": [],
"exact_files": [
"frontend-modern/src/hooks/__tests__/useProtectionPostures.test.ts",
"frontend-modern/src/types/recovery.ts",
"internal/api/recovery_handlers_test.go",
"internal/api/route_inventory_test.go"
]
},
{
"id": "backend-payload-contracts",
"label": "backend API payload proof",
@@ -5151,6 +5176,24 @@
"tests/integration/tests/43-platform-mock-runtime.spec.ts"
]
},
{
"id": "pbs-protection-evidence-runtime",
"label": "PBS protection evidence collection proof",
"match_prefixes": [],
"match_files": [
"internal/monitoring/monitor_backups.go",
"internal/monitoring/pbs_protection_observation.go",
"internal/monitoring/recovery_ingest.go"
],
"allow_same_subsystem_tests": false,
"test_prefixes": [],
"exact_files": [
"internal/monitoring/pbs_protection_observation_test.go",
"internal/monitoring/recovery_ingest_test.go",
"internal/recovery/mapper/proxmox/mapper_test.go",
"internal/recovery/posture_test.go"
]
},
{
"id": "monitoring-runtime",
"label": "monitoring runtime proof",
@@ -6267,8 +6310,10 @@
"frontend-modern/src/features/proxmox/proxmoxBackupSourcePresentation.ts",
"frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx",
"frontend-modern/src/features/proxmox/proxmoxBackupsTableShared.tsx",
"frontend-modern/src/features/proxmox/ProxmoxCoverageTable.tsx",
"frontend-modern/src/features/proxmox/ProxmoxPageSurface.tsx",
"frontend-modern/src/features/proxmox/ProxmoxRecoverableTable.tsx",
"frontend-modern/src/hooks/useProtectionPostures.ts",
"frontend-modern/src/hooks/useRecoveryPoints.ts",
"frontend-modern/src/types/recovery.ts",
"frontend-modern/src/utils/recoveryArtifactModePresentation.ts",
@@ -6302,7 +6347,9 @@
"internal/recovery/idgen/idgen_test.go",
"internal/recovery/manager/manager_test.go",
"internal/recovery/mapper/proxmox/mapper_test.go",
"internal/recovery/posture_test.go",
"internal/recovery/recovery_test.go",
"internal/recovery/store/store_posture_test.go",
"internal/recovery/store/store_queryplan_test.go",
"internal/recovery/store/store_rollups_test.go",
"internal/recovery/store/store_series_test.go",
@@ -6314,6 +6361,7 @@
"label": "recovery product surface proof",
"match_prefixes": [],
"match_files": [
"frontend-modern/src/hooks/useProtectionPostures.ts",
"frontend-modern/src/hooks/useRecoveryPoints.ts",
"frontend-modern/src/types/recovery.ts",
"frontend-modern/src/utils/recoveryArtifactModePresentation.ts",
@@ -6326,8 +6374,11 @@
"allow_same_subsystem_tests": false,
"test_prefixes": [],
"exact_files": [
"frontend-modern/src/api/__tests__/recoveryTransport.test.ts",
"frontend-modern/src/hooks/__tests__/useProtectionPostures.test.ts",
"frontend-modern/src/utils/__tests__/frontendResourceTypeBoundaries.test.ts",
"tests/integration/tests/17-proxmox-backups-layout.spec.ts"
"tests/integration/tests/17-proxmox-backups-layout.spec.ts",
"tests/integration/tests/90-operational-trust-protection-posture.spec.ts"
]
},
{
@@ -6370,16 +6421,20 @@
"frontend-modern/src/features/proxmox/proxmoxBackupSourcePresentation.ts",
"frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx",
"frontend-modern/src/features/proxmox/proxmoxBackupsTableShared.tsx",
"frontend-modern/src/features/proxmox/ProxmoxCoverageTable.tsx",
"frontend-modern/src/features/proxmox/ProxmoxPageSurface.tsx",
"frontend-modern/src/features/proxmox/ProxmoxRecoverableTable.tsx"
],
"allow_same_subsystem_tests": false,
"test_prefixes": [],
"exact_files": [
"frontend-modern/src/features/proxmox/__tests__/proxmoxBackupRecoveryModel.branchcov.test.ts",
"frontend-modern/src/features/proxmox/__tests__/proxmoxBackupRecoveryModel.test.ts",
"frontend-modern/src/features/proxmox/__tests__/proxmoxBackupSourcePresentation.test.ts",
"frontend-modern/src/features/proxmox/__tests__/ProxmoxBackupsTable.test.tsx",
"frontend-modern/src/features/proxmox/__tests__/proxmoxBackupsTableShared.test.tsx"
"frontend-modern/src/features/proxmox/__tests__/proxmoxBackupsTableShared.test.tsx",
"frontend-modern/src/features/proxmox/__tests__/ProxmoxCoverageTable.test.tsx",
"tests/integration/tests/90-operational-trust-protection-posture.spec.ts"
]
},
{
docs/release-control/v6/internal/subsystems/storage-recovery.md
+36 -3
View File
@@ -53,6 +53,11 @@ state.
24. `frontend-modern/src/features/proxmox/ProxmoxRecoverableTable.tsx`
25. `frontend-modern/src/features/proxmox/proxmoxBackupsTableShared.tsx`
26. `frontend-modern/src/features/proxmox/proxmoxBackupSourcePresentation.ts`
27. `internal/recovery/model/posture.go`
28. `internal/recovery/posture.go`
29. `internal/recovery/store/store_posture.go`
30. `frontend-modern/src/hooks/useProtectionPostures.ts`
31. `frontend-modern/src/features/proxmox/ProxmoxCoverageTable.tsx`
## Shared Boundaries
@@ -90,9 +95,10 @@ knowledge persistence. Any future storage action must cross the explicit
a recovery-provider read helper or compatibility alias.
1. `frontend-modern/src/features/proxmox/ProxmoxBackupServersTable.tsx` shared with `unified-resources`: Proxmox backup server table rows are both a storage/recovery backup-health surface and a unified-resource platform-table consumer boundary.
2. `frontend-modern/src/features/proxmox/ProxmoxRecoverableTable.tsx` shared with `unified-resources`: Proxmox recoverable workload table rows are both a storage/recovery coverage surface and a unified-resource platform-table consumer boundary.
3. `internal/api/setup_script_render.go` shared with `agent-lifecycle`, `api-contracts`: the generated Proxmox setup-script is a shared boundary across agent lifecycle (forced-command keys, install/uninstall edits), API contracts (rendered token shape and encoded rerun URL), and storage/recovery (backup visibility grants, Pulse-managed temperature SSH keys, and SMART disk-temperature collection).
4. `internal/proxmoxidentity/backup_identity.go` shared with `alerts`, `monitoring`: Proxmox PBS backup subject identity is a shared runtime boundary for monitoring backup freshness, backup-age alert attribution, and recovery-point guest mapping.
2. `frontend-modern/src/features/proxmox/ProxmoxCoverageTable.tsx` shared with `unified-resources`: Proxmox workload coverage rows are both a storage/recovery protection-posture surface and a unified-resource identity consumer boundary.
3. `frontend-modern/src/features/proxmox/ProxmoxRecoverableTable.tsx` shared with `unified-resources`: Proxmox recoverable workload table rows are both a storage/recovery coverage surface and a unified-resource platform-table consumer boundary.
4. `internal/api/setup_script_render.go` shared with `agent-lifecycle`, `api-contracts`: the generated Proxmox setup-script is a shared boundary across agent lifecycle (forced-command keys, install/uninstall edits), API contracts (rendered token shape and encoded rerun URL), and storage/recovery (backup visibility grants, Pulse-managed temperature SSH keys, and SMART disk-temperature collection).
5. `internal/proxmoxidentity/backup_identity.go` shared with `alerts`, `monitoring`: Proxmox PBS backup subject identity is a shared runtime boundary for monitoring backup freshness, backup-age alert attribution, and recovery-point guest mapping.
Storage resource projections may show alert indicators from the canonical
active-alert read model only while detector evaluation is enabled. They must
@@ -2155,6 +2161,33 @@ while storage detail drawers and filter controls must route summary series IDs,
source tones, and disk metrics through the shared storage helpers instead of
reconstructing them from local table state.
### Canonical protection posture
`ProtectionPosture` is the storage/recovery-owned subject read model over
recovery points and typed provider collection evidence. Its only public states
are `protected`, `attention`, `unprotected`, and `unknown`. A protected claim
requires a current subject-linked backup supported by complete, sufficiently
authorized provider history, plus verification when the provider declares it
expected. Stale, failed, partial, or unverified evidence is attention;
unprotected requires complete evidence of no qualifying backup; unavailable
identity, history, collection, or permission evidence fails to unknown.
Snapshots remain distinct and never prove independent recovery by themselves.
The per-tenant recovery store persists provider observations and a materialized
posture index, refreshes it after writes, reconciliation, migration, identity
backfill, purge, and retention, and re-evaluates requested resource batches at
read time so freshness cannot age into a reassuring stale snapshot. Legacy
recovery rows receive additive provider scope and typed evidence during
migration. Provider observations and recovery points share bounded retention.
`GET /api/recovery/postures` is the bounded read boundary for one resource,
resource-table batches, state-filtered attention lists, policy, evaluation
time, provider limitations, and evidence references. The Proxmox coverage
surface requests canonical resource IDs in batches of at most 200 and never
derives posture from raw PVE/PBS artifacts. Raw backup files, snapshots, tasks,
and provider state remain expandable recovery evidence. Missing or failed
posture reads render unknown rather than falling back to browser inference.
The adjacent shared authentication boundary must remain live when storage or
recovery browser reads authenticated with local credentials overlap agent
API-token reports. Local credential snapshots inside `checkAuth` use the
docs/release-control/v6/internal/subsystems/unified-resources.md
+17 -5
View File
@@ -574,13 +574,14 @@ container inventory table.
10. `frontend-modern/src/components/Infrastructure/useUnifiedResourceTableState.ts` shared with `performance-and-scalability`: unified resource table state, grouping, and windowing are both a canonical unified-resource consumer surface and a fleet-scale performance hot-path boundary.
11. `frontend-modern/src/components/Infrastructure/useUnifiedResourceTableViewportSync.ts` shared with `performance-and-scalability`: unified resource table viewport sync and selected-row reveal are both a canonical unified-resource consumer surface and a fleet-scale performance hot-path boundary.
12. `frontend-modern/src/features/proxmox/ProxmoxBackupServersTable.tsx` shared with `storage-recovery`: Proxmox backup server table rows are both a storage/recovery backup-health surface and a unified-resource platform-table consumer boundary.
13. `frontend-modern/src/features/proxmox/ProxmoxRecoverableTable.tsx` shared with `storage-recovery`: Proxmox recoverable workload table rows are both a storage/recovery coverage surface and a unified-resource platform-table consumer boundary.
14. `frontend-modern/src/routing/routePreload.ts` shared with `frontend-primitives`, `performance-and-scalability`: the app-shell route preload registry is a canonical frontend shell boundary, an authenticated hot-path performance boundary, and the entry point for the unified-resource Actions workspace.
15. `frontend-modern/src/utils/platformSupportManifest.generated.ts` shared with `frontend-primitives`: the generated platform support projection is both a canonical unified-resource platform union boundary and a shared frontend source/platform vocabulary boundary.
13. `frontend-modern/src/features/proxmox/ProxmoxCoverageTable.tsx` shared with `storage-recovery`: Proxmox workload coverage rows are both a storage/recovery protection-posture surface and a unified-resource identity consumer boundary.
14. `frontend-modern/src/features/proxmox/ProxmoxRecoverableTable.tsx` shared with `storage-recovery`: Proxmox recoverable workload table rows are both a storage/recovery coverage surface and a unified-resource platform-table consumer boundary.
15. `frontend-modern/src/routing/routePreload.ts` shared with `frontend-primitives`, `performance-and-scalability`: the app-shell route preload registry is a canonical frontend shell boundary, an authenticated hot-path performance boundary, and the entry point for the unified-resource Actions workspace.
16. `frontend-modern/src/utils/platformSupportManifest.generated.ts` shared with `frontend-primitives`: the generated platform support projection is both a canonical unified-resource platform union boundary and a shared frontend source/platform vocabulary boundary.
It must carry the manifest `surface_kind` distinction so `docker` remains
machine-readable as a `runtime-lens` while owning infrastructure sources
remain `platform` entries.
16. `frontend-modern/src/utils/sourcePlatforms.ts` shared with `frontend-primitives`: the source platform normalizer is both a canonical unified-resource source adapter boundary and a shared frontend source/platform vocabulary boundary.
17. `frontend-modern/src/utils/sourcePlatforms.ts` shared with `frontend-primitives`: the source platform normalizer is both a canonical unified-resource source adapter boundary and a shared frontend source/platform vocabulary boundary.
That shared vocabulary boundary owns the generic `docker` platform label:
selectors, badges, and filter options render it as "Docker / Podman" so
v5 Docker users can still find the runtime surface while Podman-backed
@@ -602,7 +603,7 @@ container inventory table.
display/source family; `platformScopes` is the overlap set used when a
runtime workload belongs to both Docker and an owning infrastructure
platform.
17. `internal/api/resources.go` shared with `api-contracts`: the unified resource endpoint is both a backend payload contract surface and a unified-resource runtime boundary.
18. `internal/api/resources.go` shared with `api-contracts`: the unified resource endpoint is both a backend payload contract surface and a unified-resource runtime boundary.
`/api/resources` type filters must accept URL-encoded comma-separated lists
from browser query builders exactly like literal comma separators, so Docker
/ Podman runtime pages do not lose `docker-host` inventory while requesting
@@ -1789,6 +1790,17 @@ through the canonical resource model, but unified-resource consumers must not
reintroduce removed workload aliases or feature-local resource-type shims just
to satisfy one table, drawer, or badge surface.
### Protection posture identity consumer
`ProxmoxCoverageTable` remains a unified-resource identity consumer while
storage/recovery owns protection truth. Live VM/LXC rows carry the exact
canonical `Resource.id` into one bounded posture batch; the table must not parse
its presentation key, VMID, name, node, or instance to mint a replacement
resource identity. Orphaned backup artifacts have no canonical live resource
ID and therefore render unknown posture while retaining their forensic backup
detail. Unified resources own row identity only; they do not derive backup
freshness, provider completeness, verification, or protection state.
### APT Product Trust browser projection
The durable Actions inbox and resource action history consume the canonical APT
frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx
+40 -17
View File
@@ -6,6 +6,7 @@ import { useSearchParams } from '@solidjs/router';
import { FilterBar, type FilterDef, type FilterSelectOption } from '@/components/shared/FilterBar';
import { FilterSegmentedControl } from '@/components/shared/FilterToolbar';
import { useBreakpoint } from '@/hooks/useBreakpoint';
import { useProtectionPostures } from '@/hooks/useProtectionPostures';
import { apiFetch } from '@/utils/apiClient';
import {
PlatformErrorState,
@@ -37,7 +38,6 @@ import {
import {
buildProxmoxBackupRecoveryModel,
coverageRowMatchesSearch,
isCoverageAttention,
recoverableArtifactMatchesSearch,
type RecoverableArtifact,
} from './proxmoxBackupRecoveryModel';
@@ -99,6 +99,9 @@ export const ProxmoxBackupsTable: Component<{
const [backups, { refetch }] = createResource<PVEBackupsPayload>(fetchPVEBackups);
const [pbsBackups] = createResource<PBSBackupsPayload>(fetchPBSBackups);
const { isMobile } = useBreakpoint();
const protectionPostures = useProtectionPostures(() =>
(props.workloads ?? []).map((workload) => workload.id),
);
// Structured scope filters (node, type) live in the URL so the view is
// shareable, survives reload, and can be captured by FilterBar saved views.
@@ -201,6 +204,7 @@ export const ProxmoxBackupsTable: Component<{
snapshots: snapshots(),
tasks: tasks(),
nowMs: nowMs(),
protectionPostures: protectionPostures.postureByResourceID(),
}),
);
@@ -245,9 +249,7 @@ export const ProxmoxBackupsTable: Component<{
const list = recoveryModel().coverageRows.filter((row) => {
if (!nodeMatches(row.workload.node)) return false;
if (!typeMatches(row.workload.type)) return false;
if (filter === 'attention' && !isCoverageAttention(row.posture)) return false;
if (filter === 'current' && row.posture !== 'current') return false;
if (filter === 'uncovered' && row.posture !== 'uncovered') return false;
if (filter !== 'all' && row.posture !== filter) return false;
return coverageRowMatchesSearch(row, term);
});
const sortKey = coverageSortKey();
@@ -296,13 +298,18 @@ export const ProxmoxBackupsTable: Component<{
const liveHealthSummary = createMemo(() => {
const live = recoveryModel().coverageRows.filter((row) => !row.isOrphaned);
return {
current: live.filter((row) => row.posture === 'current').length,
attention: live.filter((row) => isCoverageAttention(row.posture)).length,
uncovered: live.filter((row) => row.posture === 'uncovered').length,
protected: live.filter((row) => row.posture === 'protected').length,
attention: live.filter((row) => row.posture === 'attention').length,
unprotected: live.filter((row) => row.posture === 'unprotected').length,
unknown: live.filter((row) => row.posture === 'unknown').length,
};
});
const view = (): BackupView =>
selectedView() ?? (liveHealthSummary().attention > 0 ? 'coverage' : 'date');
selectedView() ??
(liveHealthSummary().attention + liveHealthSummary().unprotected + liveHealthSummary().unknown >
0
? 'coverage'
: 'date');
const setView = (next: BackupView): void => {
setSelectedView(next);
};
@@ -462,6 +469,15 @@ export const ProxmoxBackupsTable: Component<{
}
>
<div class="space-y-3">
<Show when={protectionPostures.response.error}>
<div
role="status"
class="rounded-md border border-amber-300 bg-amber-50 px-3 py-2 text-xs text-amber-900 dark:border-amber-700 dark:bg-amber-950/40 dark:text-amber-100"
>
Protection posture is unavailable. Pulse is showing restore evidence without guessing
whether workloads are protected.
</div>
</Show>
<Show when={(props.servers?.length ?? 0) > 0}>
<ProxmoxBackupServersTable servers={props.servers ?? []} backups={pbsArtifacts()} />
</Show>
@@ -472,8 +488,8 @@ export const ProxmoxBackupsTable: Component<{
title="Backup health"
tail={
<span>
{liveTotalCount()} targets ·{' '}
{recoveryModel().coverageSummary.recoverableArtifacts} restore points
{liveTotalCount()} targets · {recoveryModel().coverageSummary.recoverableArtifacts}{' '}
restore points
<Show when={recoveryModel().coverageSummary.withPBS > 0}>
{' · '}
{recoveryModel().coverageSummary.withPBS} with{' '}
@@ -487,9 +503,9 @@ export const ProxmoxBackupsTable: Component<{
}
segments={[
{
key: 'current',
value: liveHealthSummary().current,
label: 'current',
key: 'protected',
value: liveHealthSummary().protected,
label: 'protected',
toneClass: 'bg-emerald-500',
},
{
@@ -500,11 +516,18 @@ export const ProxmoxBackupsTable: Component<{
muted: liveHealthSummary().attention === 0,
},
{
key: 'uncovered',
value: liveHealthSummary().uncovered,
label: 'uncovered',
key: 'unprotected',
value: liveHealthSummary().unprotected,
label: 'unprotected',
toneClass: 'bg-red-500',
muted: liveHealthSummary().uncovered === 0,
muted: liveHealthSummary().unprotected === 0,
},
{
key: 'unknown',
value: liveHealthSummary().unknown,
label: 'unknown',
toneClass: 'bg-slate-400',
muted: liveHealthSummary().unknown === 0,
},
]}
/>
frontend-modern/src/features/proxmox/ProxmoxCoverageTable.tsx
+54 -4
View File
@@ -32,9 +32,10 @@ import {
const coveragePostureVariant = (
posture: WorkloadCoverageRow['posture'],
): StatusIndicatorVariant => {
if (posture === 'current') return 'success';
if (posture === 'uncovered' || posture === 'failed' || posture === 'stale') return 'danger';
return 'warning';
if (posture === 'protected') return 'success';
if (posture === 'unprotected') return 'danger';
if (posture === 'attention') return 'warning';
return 'muted';
};
// Colour marks the exception, not the baseline: healthy rows keep neutral text
@@ -54,6 +55,17 @@ const taskWordVariant = (label: string): StatusIndicatorVariant => {
return 'warning';
};
const providerLabel = (provider: string): string => {
if (provider === 'proxmox-pbs') return 'Proxmox Backup Server';
if (provider === 'proxmox-pve') return 'Proxmox VE';
return provider;
};
const evidenceQualityLabel = (value: string): string => {
if (!value) return 'Unknown';
return value.charAt(0).toUpperCase() + value.slice(1);
};
// "Workload coverage" table: one row per workload answering "does this have a
// backup?" across PBS snapshots / PVE backup files / guest snapshots, each
// expanding to its restore evidence. Presentational — the parent owns the
@@ -266,10 +278,14 @@ export function ProxmoxCoverageTable(props: {
<StatusDot
size="sm"
variant={coveragePostureVariant(row.posture)}
title={getWorkloadRecoveryPostureLabel(row.posture)}
title={
row.protectionPosture?.explanation ??
'Pulse does not have enough provider evidence to determine protection.'
}
ariaHidden
/>
<span
title={row.protectionPosture?.explanation}
class={`truncate text-[11px] font-medium ${statusWordToneClass(
coveragePostureVariant(row.posture),
)}`}
@@ -374,6 +390,40 @@ export function ProxmoxCoverageTable(props: {
colspan={columnCount()}
data-inline-detail-for={row.key}
>
<div class="mb-2 rounded-md border border-border-subtle bg-surface px-2 py-1.5 text-[11px] text-base-content">
<span class="font-medium">
{getWorkloadRecoveryPostureLabel(row.posture)}:
</span>{' '}
{row.protectionPosture?.explanation ??
'Pulse cannot determine this workload’s protection because no complete provider evidence is linked to it.'}
</div>
<Show when={(row.protectionPosture?.providerStates.length ?? 0) > 0}>
<div class="mb-2 overflow-hidden rounded-md border border-border-subtle">
<div class="bg-surface-alt px-2 py-1 text-[11px] font-medium text-base-content">
Provider evidence
</div>
<div class="divide-y divide-border-subtle">
<For each={row.protectionPosture?.providerStates ?? []}>
{(provider) => (
<div class="grid grid-cols-2 gap-x-3 gap-y-0.5 px-2 py-1.5 text-[11px] sm:grid-cols-4">
<span class="font-medium text-base-content">
{providerLabel(provider.provider)}
</span>
<span class="text-muted">
Job {evidenceQualityLabel(provider.jobState)}
</span>
<span class="text-muted">
History {evidenceQualityLabel(provider.historyCompleteness)}
</span>
<span class="text-muted">
Access {evidenceQualityLabel(provider.permissions)}
</span>
</div>
)}
</For>
</div>
</div>
</Show>
<Show
when={evidence().length > 0}
fallback={
frontend-modern/src/features/proxmox/__tests__/ProxmoxBackupsTable.test.tsx
+40 -5
View File
@@ -25,9 +25,11 @@ const renderInRouter = (component: () => JSX.Element) =>
));
const apiFetchMock = vi.hoisted(() => vi.fn());
const apiFetchJSONMock = vi.hoisted(() => vi.fn());
vi.mock('@/utils/apiClient', () => ({
apiFetch: apiFetchMock,
apiFetchJSON: apiFetchJSONMock,
}));
const jsonResponse = (payload: unknown) =>
@@ -110,12 +112,40 @@ const pbsPayload = {
meta: { totalBackups: 1 },
};
function mockBackupAPIs() {
function mockBackupAPIs(state: 'protected' | 'attention' = 'protected') {
apiFetchMock.mockImplementation((url: string) => {
if (url === '/api/backups/pbs') return Promise.resolve(jsonResponse(pbsPayload));
if (url === '/api/backups/pve') return Promise.resolve(jsonResponse(pvePayload));
return Promise.resolve(jsonResponse({}));
});
apiFetchJSONMock.mockResolvedValue({
data: [
{
subjectResourceId: 'ct-112',
state,
lastAttemptAt: '2026-05-25T02:00:00Z',
lastSuccessfulPointAt: '2026-05-25T01:34:25Z',
lastVerifiedAt: '2026-05-25T01:34:25Z',
freshness: 'current',
verification: 'verified',
coverage: 'complete',
providerStates: [],
repositoryResourceIds: [],
evidenceIds: ['evidence-1'],
explanation:
state === 'protected'
? 'A current verified backup is available.'
: 'The latest provider job needs attention.',
evaluatedAt: '2026-05-25T02:05:00Z',
},
],
policy: {
freshnessWindowSeconds: 604800,
verificationWindowSeconds: 604800,
requireVerification: true,
},
meta: { page: 1, limit: 200, total: 1, totalPages: 1 },
});
}
const workloadResource = {
@@ -167,11 +197,12 @@ const expectCanonicalPlatformTableShell = (table: HTMLElement): void => {
afterEach(() => {
cleanup();
apiFetchMock.mockReset();
apiFetchJSONMock.mockReset();
});
describe('ProxmoxBackupsTable', () => {
it('defaults to coverage when protection needs attention and keeps the dated feed one click away', async () => {
mockBackupAPIs();
mockBackupAPIs('attention');
renderInRouter(() => (
<ProxmoxBackupsTable
@@ -216,6 +247,10 @@ describe('ProxmoxBackupsTable', () => {
}
expect(apiFetchMock).toHaveBeenCalledWith('/api/backups/pbs');
expect(apiFetchMock).toHaveBeenCalledWith('/api/backups/pve');
expect(apiFetchJSONMock).toHaveBeenCalledTimes(1);
const postureURL = new URL(apiFetchJSONMock.mock.calls[0][0], 'https://pulse.invalid');
expect(postureURL.pathname).toBe('/api/recovery/postures');
expect(postureURL.searchParams.getAll('resourceId')).toEqual(['ct-112']);
});
it('offers By date / Coverage views and no legacy sub-tab tree', async () => {
@@ -260,10 +295,10 @@ describe('ProxmoxBackupsTable', () => {
await screen.findAllByText('pbs-docker');
await fireEvent.click(screen.getByRole('button', { name: /coverage/i }));
// Coverage is the posture view; the workload's recent backup reads
// "Current".
// Coverage is the posture view; the server-owned workload posture reads
// "Protected".
expect(screen.getByRole('columnheader', { name: /posture/i })).toBeInTheDocument();
expect(screen.getAllByText('Current').length).toBeGreaterThan(0);
expect(screen.getAllByText('Protected').length).toBeGreaterThan(0);
// Per-source detail is one click down inside the workload's row.
await fireEvent.click(screen.getByRole('button', { name: /expand details for pbs-docker/i }));
frontend-modern/src/features/proxmox/__tests__/ProxmoxCoverageTable.test.tsx
+74 -1
View File
@@ -21,8 +21,30 @@ const row = {
pbsCount: 1,
archiveCount: 0,
snapshotCount: 0,
posture: 'current',
posture: 'protected',
postureRank: 0,
protectionPosture: {
subjectResourceId: 'resource:vm:100',
state: 'protected',
freshness: 'current',
verification: 'verified',
coverage: 'complete',
providerStates: [
{
provider: 'proxmox-pbs',
source: 'pbs-backup-enumeration',
scope: 'pbs-main',
jobState: 'success',
historyCompleteness: 'complete',
permissions: 'sufficient',
evidenceIds: ['evidence-provider'],
},
],
repositoryResourceIds: [],
evidenceIds: ['evidence-provider'],
explanation: 'A current verified backup is available from complete provider history.',
evaluatedAt: '2026-07-19T00:00:00Z',
},
} as unknown as WorkloadCoverageRow;
const headerTexts = () =>
@@ -69,4 +91,55 @@ describe('ProxmoxCoverageTable column visibility', () => {
expect(document.body.textContent).not.toContain('ID 100');
expect(document.body.textContent).not.toContain('Node pve1');
});
it('keeps provider evidence in the workload drill-down instead of every table row', () => {
const { unmount } = render(() => (
<ProxmoxCoverageTable
rows={[row]}
hasAnyRows
emptyIcon={<span />}
emptyTitle=""
emptyDescription=""
sortKey={(() => 'posture') as Accessor<CoverageSortKey>}
sortDirection={() => 'asc'}
onSort={() => {}}
expandedKeys={new Set<string>()}
onToggleExpand={() => {}}
showPbsColumn={true}
showArchiveColumn={false}
showSnapshotColumn={false}
showTaskColumn={false}
/>
));
expect(document.body.textContent).not.toContain('Provider evidence');
expect(document.body.textContent).not.toContain(
'A current verified backup is available from complete provider history.',
);
unmount();
render(() => (
<ProxmoxCoverageTable
rows={[row]}
hasAnyRows
emptyIcon={<span />}
emptyTitle=""
emptyDescription=""
sortKey={(() => 'posture') as Accessor<CoverageSortKey>}
sortDirection={() => 'asc'}
onSort={() => {}}
expandedKeys={new Set<string>(['w1'])}
onToggleExpand={() => {}}
showPbsColumn={true}
showArchiveColumn={false}
showSnapshotColumn={false}
showTaskColumn={false}
/>
));
expect(document.body.textContent).toContain('Provider evidence');
expect(document.body.textContent).toContain('Proxmox Backup Server');
expect(document.body.textContent).toContain('History Complete');
expect(document.body.textContent).toContain('Access Sufficient');
});
});
frontend-modern/src/features/proxmox/__tests__/proxmoxBackupRecoveryModel.branchcov.test.ts
+60 -107
View File
@@ -1,6 +1,7 @@
import { describe, expect, it } from 'vitest';
import type { BackupTask, GuestSnapshot, PBSBackup } from '@/types/api';
import type { BackupTask, PBSBackup } from '@/types/api';
import type { ProtectionPosture, ProtectionState } from '@/types/recovery';
import type { Resource } from '@/types/resource';
import {
@@ -53,18 +54,6 @@ const pbsBackup = (overrides: Partial<PBSBackup> = {}): PBSBackup => ({
...overrides,
});
const snap = (overrides: Partial<GuestSnapshot> = {}): GuestSnapshot => ({
id: 'snap-100',
name: 'pre-upgrade',
node: 'node-a',
instance: 'inst-a',
type: 'vm',
vmid: 100,
time: '2026-07-09T00:00:00Z',
vmstate: false,
...overrides,
});
const task = (overrides: Partial<BackupTask> = {}): BackupTask => ({
id: 'task-100',
node: 'node-a',
@@ -78,8 +67,25 @@ const task = (overrides: Partial<BackupTask> = {}): BackupTask => ({
type ModelInput = Parameters<typeof buildProxmoxBackupRecoveryModel>[0];
const buildModel = (input: Partial<ModelInput>): ProxmoxBackupRecoveryModel =>
buildProxmoxBackupRecoveryModel({
const posture = (resourceId: string, state: ProtectionState): ProtectionPosture => ({
subjectResourceId: resourceId,
state,
freshness: state === 'protected' ? 'current' : 'unknown',
verification: state === 'protected' ? 'verified' : 'unknown',
coverage: state === 'unprotected' ? 'none' : state === 'unknown' ? 'unknown' : 'complete',
providerStates: [],
repositoryResourceIds: [],
evidenceIds: [],
explanation: `Canonical ${state} fixture`,
evaluatedAt: new Date(NOW).toISOString(),
});
const buildModel = (input: Partial<ModelInput>): ProxmoxBackupRecoveryModel => {
const workloads = input.workloads ?? [];
const protectionPostures =
input.protectionPostures ??
new Map(workloads.map((resource) => [resource.id, posture(resource.id, 'protected')]));
return buildProxmoxBackupRecoveryModel({
workloads: [],
pbsBackups: [],
archives: [],
@@ -87,7 +93,9 @@ const buildModel = (input: Partial<ModelInput>): ProxmoxBackupRecoveryModel =>
tasks: [],
nowMs: NOW,
...input,
protectionPostures,
});
};
// ---------------------------------------------------------------------------
// getWorkloadRecoveryPostureLabel — uncovered switch cases
@@ -96,97 +104,37 @@ const buildModel = (input: Partial<ModelInput>): ProxmoxBackupRecoveryModel =>
describe('getWorkloadRecoveryPostureLabel uncovered switch cases', () => {
it.each<[WorkloadRecoveryPosture, string]>([
['aging', 'Aging'],
['stale', 'Stale'],
['snapshot-only', 'Snapshot only'],
['uncovered', 'Uncovered'],
['unverified', 'Unverified'],
['protected', 'Protected'],
['attention', 'Needs attention'],
['unprotected', 'Unprotected'],
['unknown', 'Unknown'],
])('renders posture %s as %j', (posture, expected) => {
expect(getWorkloadRecoveryPostureLabel(posture)).toBe(expected);
});
});
// ---------------------------------------------------------------------------
// buildPosture — uncovered posture arms (private; exercised via the model).
// Sibling tests cover 'current', 'uncovered', and the 'failed' posture when a
// failed task started AFTER the recovery (startedMs >= createdMs).
// ---------------------------------------------------------------------------
describe('buildPosture uncovered arms (via buildProxmoxBackupRecoveryModel)', () => {
it('classifies a 14-day-old PBS recovery as "aging"', () => {
describe('canonical protection posture ownership', () => {
it('uses the server posture even when raw artifacts suggest a different answer', () => {
const resource = workload({});
const model = buildModel({
workloads: [workload({})],
pbsBackups: [pbsBackup({ backupTime: isoDaysAgo(14) })],
workloads: [resource],
pbsBackups: [pbsBackup({ backupTime: isoDaysAgo(1), verified: true })],
protectionPostures: new Map([[resource.id, posture(resource.id, 'attention')]]),
});
expect(model.coverageRows[0].posture).toBe('aging');
expect(model.coverageRows[0].postureRank).toBe(4);
});
it('classifies a 45-day-old PBS recovery as "stale"', () => {
const model = buildModel({
workloads: [workload({})],
pbsBackups: [pbsBackup({ backupTime: isoDaysAgo(45) })],
});
expect(model.coverageRows[0].posture).toBe('stale');
expect(model.coverageRows[0].postureRank).toBe(2);
});
it('flags a recent unverified PBS recovery as "unverified"', () => {
const model = buildModel({
workloads: [workload({})],
pbsBackups: [pbsBackup({ backupTime: isoDaysAgo(1), verified: false })],
});
expect(model.coverageRows[0].posture).toBe('unverified');
expect(model.coverageRows[0].postureRank).toBe(2);
});
it('flags a guest-snapshot-only workload as "snapshot-only"', () => {
const model = buildModel({
workloads: [workload({})],
snapshots: [snap({})],
});
expect(model.coverageRows[0].posture).toBe('snapshot-only');
expect(model.coverageRows[0].postureRank).toBe(3);
expect(model.coverageRows[0].snapshotCount).toBe(1);
expect(model.coverageRows[0].pbsCount).toBe(0);
});
it('treats a failed task with an unparseable start time as the latest failure', () => {
// startedMs is undefined -> the `latestTask.startedMs === undefined` arm of
// the failedTask disjunction fires even though the recovery has a createdMs.
const model = buildModel({
workloads: [workload({})],
pbsBackups: [pbsBackup({ backupTime: isoDaysAgo(1) })],
tasks: [
task({
status: 'failed',
startTime: '',
type: 'vm',
}),
],
});
expect(model.coverageRows[0].posture).toBe('failed');
expect(model.coverageRows[0].posture).toBe('attention');
expect(model.coverageRows[0].postureRank).toBe(0);
expect(model.coverageRows[0].latestTask?.label).toBe('Failed');
expect(model.coverageRows[0].latestTask?.startedMs).toBeUndefined();
expect(model.coverageRows[0].protectionPosture?.explanation).toBe(
'Canonical attention fixture',
);
});
it('does not flag "failed" when a failed task predates the latest recovery', () => {
// failedTask is false because startedMs < latestRecovery.createdMs, so the
// failedTask conjunction is false and posture falls through to the age band.
it('reports unknown when no canonical posture is available', () => {
const model = buildModel({
workloads: [workload({})],
pbsBackups: [pbsBackup({ backupTime: isoDaysAgo(1) })],
tasks: [
task({
status: 'failed',
startTime: isoDaysAgo(10),
type: 'vm',
}),
],
protectionPostures: new Map(),
});
expect(model.coverageRows[0].posture).toBe('current');
expect(model.coverageRows[0].latestTask?.label).toBe('Failed');
expect(model.coverageRows[0].posture).toBe('unknown');
expect(model.coverageRows[0].postureRank).toBe(2);
});
});
@@ -324,9 +272,7 @@ describe('resourceBackupType branches (via candidate type)', () => {
describe('resourceNode branches (via candidate node)', () => {
it('prefers proxmox.nodeName when present', () => {
const model = buildModel({
workloads: [
workload({ id: 'vm-721', proxmox: { vmid: 721, nodeName: 'named-a' } }),
],
workloads: [workload({ id: 'vm-721', proxmox: { vmid: 721, nodeName: 'named-a' } })],
});
expect(model.coverageRows[0].workload.node).toBe('named-a');
});
@@ -340,9 +286,7 @@ describe('resourceNode branches (via candidate node)', () => {
it('falls back to parentName when no proxmox node field is set', () => {
const model = buildModel({
workloads: [
workload({ id: 'vm-723', parentName: 'parent-x', proxmox: { vmid: 723 } }),
],
workloads: [workload({ id: 'vm-723', parentName: 'parent-x', proxmox: { vmid: 723 } })],
});
expect(model.coverageRows[0].workload.node).toBe('parent-x');
});
@@ -527,9 +471,10 @@ describe('buildProxmoxBackupRecoveryModel summary, sort, and empty edges', () =>
expect(model.recoverableArtifacts).toHaveLength(0);
expect(model.coverageSummary).toEqual({
totalWorkloads: 0,
current: 0,
protected: 0,
attention: 0,
uncovered: 0,
unprotected: 0,
unknown: 0,
withPBS: 0,
recoverableArtifacts: 0,
totalBytes: 0,
@@ -550,12 +495,17 @@ describe('buildProxmoxBackupRecoveryModel summary, sort, and empty edges', () =>
expect(model.coverageSummary.totalBytes).toBe(7_000);
expect(model.coverageSummary.recoverableArtifacts).toBe(2);
expect(model.coverageSummary.withPBS).toBe(2);
expect(model.coverageSummary.current).toBe(2);
expect(model.coverageSummary.protected).toBe(2);
expect(model.coverageSummary.attention).toBe(0);
});
it('sorts rows by postureRank asc, then latestRecovery createdMs desc', () => {
const model = buildModel({
const postures = new Map([
['vm-830', posture('vm-830', 'unprotected')],
['vm-840', posture('vm-840', 'protected')],
['vm-850', posture('vm-850', 'protected')],
]);
const canonicalModel = buildModel({
workloads: [
workload({ id: 'vm-840', proxmox: { vmid: 840, node: 'a' } }),
workload({ id: 'vm-830', proxmox: { vmid: 830, node: 'b' } }),
@@ -565,11 +515,14 @@ describe('buildProxmoxBackupRecoveryModel summary, sort, and empty edges', () =>
pbsBackup({ id: 'pbs-840', vmid: '840', backupTime: isoDaysAgo(1) }),
pbsBackup({ id: 'pbs-850', vmid: '850', backupTime: isoDaysAgo(2) }),
],
protectionPostures: postures,
});
// 830 has no recovery -> uncovered (rank 1) sorts first.
// 840 and 850 are current (rank 5); tie broken by newer recovery first.
expect(model.coverageRows.map((row) => row.workload.vmid)).toEqual(['830', '840', '850']);
expect(model.coverageSummary.uncovered).toBe(1);
expect(canonicalModel.coverageRows.map((row) => row.workload.vmid)).toEqual([
'830',
'840',
'850',
]);
expect(canonicalModel.coverageSummary.unprotected).toBe(1);
});
it('sorts recoverable artifacts by createdMs desc', () => {
frontend-modern/src/features/proxmox/__tests__/proxmoxBackupRecoveryModel.test.ts
+29 -11
View File
@@ -1,6 +1,7 @@
import { describe, expect, it } from 'vitest';
import type { BackupTask, GuestSnapshot, PBSBackup, StorageBackup } from '@/types/api';
import type { ProtectionPosture, ProtectionState } from '@/types/recovery';
import type { Resource } from '@/types/resource';
import {
buildProxmoxBackupRecoveryModel,
@@ -83,8 +84,21 @@ const task = (overrides: Partial<BackupTask> = {}): BackupTask => ({
...overrides,
});
const protectionPosture = (resourceId: string, state: ProtectionState): ProtectionPosture => ({
subjectResourceId: resourceId,
state,
freshness: state === 'protected' ? 'current' : 'unknown',
verification: state === 'protected' ? 'verified' : 'unknown',
coverage: state === 'unprotected' ? 'none' : state === 'unknown' ? 'unknown' : 'complete',
providerStates: [],
repositoryResourceIds: [],
evidenceIds: [],
explanation: `Canonical ${state} fixture`,
evaluatedAt: '2026-05-26T08:00:00Z',
});
describe('proxmoxBackupRecoveryModel', () => {
it('classifies backup ages with the same thresholds used by coverage posture', () => {
it('classifies backup ages for restore-point presentation', () => {
const nowMs = Date.parse('2026-05-26T08:00:00Z');
expect(getRecoveryAgeBand(nowMs - 2 * 24 * 60 * 60 * 1000, nowMs)).toBe('current');
@@ -101,6 +115,7 @@ describe('proxmoxBackupRecoveryModel', () => {
snapshots: [snapshot()],
tasks: [task()],
nowMs: Date.parse('2026-05-26T08:00:00Z'),
protectionPostures: new Map([['vm-112', protectionPosture('vm-112', 'protected')]]),
});
expect(model.coverageRows).toHaveLength(1);
@@ -112,7 +127,7 @@ describe('proxmoxBackupRecoveryModel', () => {
expect(row.archiveCount).toBe(1);
expect(row.snapshotCount).toBe(1);
expect(row.latestTask?.label).toBe('OK');
expect(getWorkloadRecoveryPostureLabel(row.posture)).toBe('Current');
expect(getWorkloadRecoveryPostureLabel(row.posture)).toBe('Protected');
expect(model.recoverableArtifacts.map((artifact) => artifact.sourceLabel)).toEqual([
'PBS',
'PVE file',
@@ -140,7 +155,7 @@ describe('proxmoxBackupRecoveryModel', () => {
);
});
it('surfaces a failed latest backup task as workload attention', () => {
it('uses canonical workload attention while retaining the failed task as evidence', () => {
const model = buildProxmoxBackupRecoveryModel({
workloads: [workload({})],
pbsBackups: [pbsBackup({ backupTime: '2026-05-25T01:34:25Z' })],
@@ -155,27 +170,30 @@ describe('proxmoxBackupRecoveryModel', () => {
}),
],
nowMs: Date.parse('2026-05-26T08:00:00Z'),
protectionPostures: new Map([['vm-112', protectionPosture('vm-112', 'attention')]]),
});
const row = model.coverageRows[0];
expect(row.posture).toBe('failed');
expect(getWorkloadRecoveryPostureLabel(row.posture)).toBe('Failed latest task');
expect(row.posture).toBe('attention');
expect(getWorkloadRecoveryPostureLabel(row.posture)).toBe('Needs attention');
expect(coverageRowMatchesSearch(row, 'storage unavailable')).toBe(true);
});
it('keeps inventory workloads with no restore point visible as uncovered', () => {
it('keeps an inventory workload with canonical unprotected evidence visible', () => {
const resource = workload({ id: 'vm-200', proxmox: { vmid: 200, node: 'delly' } });
const model = buildProxmoxBackupRecoveryModel({
workloads: [workload({ id: 'vm-200', proxmox: { vmid: 200, node: 'delly' } })],
workloads: [resource],
pbsBackups: [],
archives: [],
snapshots: [],
tasks: [],
nowMs: Date.parse('2026-05-26T08:00:00Z'),
protectionPostures: new Map([[resource.id, protectionPosture(resource.id, 'unprotected')]]),
});
expect(model.coverageRows).toHaveLength(1);
expect(model.coverageRows[0].posture).toBe('uncovered');
expect(model.coverageSummary.uncovered).toBe(1);
expect(model.coverageRows[0].posture).toBe('unprotected');
expect(model.coverageSummary.unprotected).toBe(1);
});
it('does not treat a linked Pulse agent facet as recovery evidence or authority', () => {
@@ -200,8 +218,8 @@ describe('proxmoxBackupRecoveryModel', () => {
});
expect(model.coverageRows).toHaveLength(1);
expect(model.coverageRows[0].posture).toBe('uncovered');
expect(model.coverageSummary.uncovered).toBe(1);
expect(model.coverageRows[0].posture).toBe('unknown');
expect(model.coverageSummary.unknown).toBe(1);
expect(model.coverageRows[0].pbsCount).toBe(0);
expect(model.coverageRows[0].archiveCount).toBe(0);
expect(model.coverageRows[0].snapshotCount).toBe(0);
frontend-modern/src/features/proxmox/proxmoxBackupRecoveryModel.ts
+33 -48
View File
@@ -1,4 +1,5 @@
import type { BackupTask, GuestSnapshot, PBSBackup, StorageBackup } from '@/types/api';
import type { ProtectionPosture, ProtectionState } from '@/types/recovery';
import type { Resource } from '@/types/resource';
import {
@@ -9,17 +10,11 @@ import {
export type RecoverableSourceKind = ProxmoxBackupSourceKind;
export type WorkloadRecoveryPosture =
| 'current'
| 'aging'
| 'stale'
| 'snapshot-only'
| 'failed'
| 'uncovered'
| 'unverified';
export type WorkloadRecoveryPosture = ProtectionState;
export interface WorkloadReference {
key: string;
resourceId?: string;
type: 'vm' | 'ct' | 'host' | 'unknown';
typeLabel: string;
vmid: string;
@@ -70,6 +65,7 @@ export interface WorkloadCoverageRow {
snapshotCount: number;
posture: WorkloadRecoveryPosture;
postureRank: number;
protectionPosture?: ProtectionPosture;
// True only when a VM/LXC row exists because a backup/task referenced a VMID
// with no matching live inventory guest. Host backups can also carry a
// `backup:` key, but they are first-class backup targets, not orphaned guests.
@@ -81,9 +77,10 @@ export interface ProxmoxBackupRecoveryModel {
recoverableArtifacts: RecoverableArtifact[];
coverageSummary: {
totalWorkloads: number;
current: number;
protected: number;
attention: number;
uncovered: number;
unprotected: number;
unknown: number;
withPBS: number;
recoverableArtifacts: number;
totalBytes: number;
@@ -97,6 +94,7 @@ interface BuildModelInput {
snapshots: readonly GuestSnapshot[];
tasks: readonly BackupTask[];
nowMs: number;
protectionPostures?: ReadonlyMap<string, ProtectionPosture>;
}
interface WorkloadCandidate extends WorkloadReference {
@@ -242,6 +240,7 @@ function buildCandidateFromResource(resource: Resource): WorkloadCandidate | nul
const instance = resourceInstance(resource);
return {
key: `resource:${resource.id}`,
resourceId: resource.id,
type,
typeLabel: typeLabel(type),
vmid,
@@ -380,47 +379,28 @@ function taskDurationSeconds(task: BackupTask): number | undefined {
return Math.round((end - start) / 1000);
}
function buildPosture(row: WorkloadRowDraft, nowMs: number) {
const failedTask =
row.latestTask?.label === 'Failed' &&
(row.latestRecovery?.createdMs === undefined ||
row.latestTask.startedMs === undefined ||
row.latestTask.startedMs >= row.latestRecovery.createdMs);
if (failedTask) return { posture: 'failed' as const, rank: 0 };
if (!row.latestRecovery) return { posture: 'uncovered' as const, rank: 1 };
if (row.latestRecovery.sourceKind === 'pbs' && row.latestRecovery.verified === false) {
return { posture: 'unverified' as const, rank: 2 };
}
const hasExternalBackup = row.latestPBS !== undefined || row.latestArchive !== undefined;
if (!hasExternalBackup && row.latestSnapshot)
return { posture: 'snapshot-only' as const, rank: 3 };
const ageBand = getRecoveryAgeBand(row.latestRecovery.createdMs, nowMs);
if (ageBand === 'current') return { posture: 'current' as const, rank: 5 };
if (ageBand === 'aging') return { posture: 'aging' as const, rank: 4 };
return { posture: 'stale' as const, rank: 2 };
function protectionPostureRank(posture: WorkloadRecoveryPosture): number {
if (posture === 'attention') return 0;
if (posture === 'unprotected') return 1;
if (posture === 'unknown') return 2;
return 3;
}
export function getWorkloadRecoveryPostureLabel(posture: WorkloadRecoveryPosture): string {
switch (posture) {
case 'current':
return 'Current';
case 'aging':
return 'Aging';
case 'stale':
return 'Stale';
case 'snapshot-only':
return 'Snapshot only';
case 'failed':
return 'Failed latest task';
case 'uncovered':
return 'Uncovered';
case 'unverified':
return 'Unverified';
case 'protected':
return 'Protected';
case 'attention':
return 'Needs attention';
case 'unprotected':
return 'Unprotected';
case 'unknown':
return 'Unknown';
}
}
export function isCoverageAttention(posture: WorkloadRecoveryPosture): boolean {
return posture !== 'current';
return posture === 'attention';
}
export function buildProxmoxBackupRecoveryModel(
@@ -569,11 +549,15 @@ export function buildProxmoxBackupRecoveryModel(
}
const coverageRows = Array.from(rows.values()).map((row) => {
const posture = buildPosture(row, input.nowMs);
const protectionPosture = row.workload.resourceId
? input.protectionPostures?.get(row.workload.resourceId)
: undefined;
const posture = protectionPosture?.state ?? 'unknown';
return {
...row,
posture: posture.posture,
postureRank: posture.rank,
posture,
postureRank: protectionPostureRank(posture),
protectionPosture,
isOrphaned:
!row.key.startsWith('resource:') &&
(row.workload.type === 'vm' || row.workload.type === 'ct'),
@@ -592,9 +576,10 @@ export function buildProxmoxBackupRecoveryModel(
recoverableArtifacts: artifacts,
coverageSummary: {
totalWorkloads: coverageRows.length,
current: coverageRows.filter((row) => row.posture === 'current').length,
protected: coverageRows.filter((row) => row.posture === 'protected').length,
attention: coverageRows.filter((row) => isCoverageAttention(row.posture)).length,
uncovered: coverageRows.filter((row) => row.posture === 'uncovered').length,
unprotected: coverageRows.filter((row) => row.posture === 'unprotected').length,
unknown: coverageRows.filter((row) => row.posture === 'unknown').length,
withPBS: coverageRows.filter((row) => row.pbsCount > 0).length,
recoverableArtifacts: artifacts.length,
totalBytes,
frontend-modern/src/features/proxmox/proxmoxBackupsTableModel.ts
+3 -14
View File
@@ -11,14 +11,9 @@ import { getWorkloadTypePresentation } from '@/utils/workloadTypePresentation';
export type BackupTabId = 'coverage' | 'recoverable' | 'sources' | 'tasks';
export type SourceDetailTabId = 'pbs' | 'snapshots' | 'archives';
export type CoverageFilterValue = 'all' | 'attention' | 'current' | 'uncovered';
export type CoverageFilterValue = 'all' | 'attention' | 'protected' | 'unprotected' | 'unknown';
export type RecoverableFilterValue =
| 'all'
| 'pbs'
| 'archive'
| 'snapshot'
| 'verified'
| 'unverified';
'all' | 'pbs' | 'archive' | 'snapshot' | 'verified' | 'unverified';
export type SnapshotFilterValue = 'all' | 'recent' | 'stale' | 'with-ram';
// One guest's aggregated snapshot inventory, the row shape for the Snapshots
@@ -159,13 +154,7 @@ export function cmpBool(a: boolean, b: boolean, direction: 'asc' | 'desc'): numb
// `asc` (A→Z). Boolean columns default to `desc` so "true" sorts first.
export type CoverageSortKey =
| 'posture'
| 'workload'
| 'latest'
| 'pbs'
| 'archive'
| 'snapshot'
| 'task';
'posture' | 'workload' | 'latest' | 'pbs' | 'archive' | 'snapshot' | 'task';
export const COVERAGE_SORT_DEFAULT_DIRECTION: Record<CoverageSortKey, 'asc' | 'desc'> = {
posture: 'asc',
workload: 'asc',
frontend-modern/src/features/proxmox/proxmoxBackupsTableShared.tsx
+9 -4
View File
@@ -116,17 +116,22 @@ export const COVERAGE_FILTERS: FilterOption<CoverageFilterValue>[] = [
leading: filterChipStatusDot('bg-amber-500'),
},
{
value: 'current',
label: 'Current',
value: 'protected',
label: 'Protected',
tone: 'success',
leading: filterChipStatusDot('bg-emerald-500'),
},
{
value: 'uncovered',
label: 'Uncovered',
value: 'unprotected',
label: 'Unprotected',
tone: 'danger',
leading: filterChipStatusDot('bg-red-500'),
},
{
value: 'unknown',
label: 'Unknown',
leading: filterChipStatusDot('bg-base-content/40'),
},
];
const recoverableSourceFilterOption = (
frontend-modern/src/hooks/__tests__/useProtectionPostures.test.ts
+31
View File
@@ -0,0 +1,31 @@
import { describe, expect, it } from 'vitest';
import {
MAX_PROTECTION_POSTURE_BATCH_SIZE,
buildProtectionPostureBatchURL,
normalizeProtectionPostureResourceIDs,
} from '@/hooks/useProtectionPostures';
describe('useProtectionPostures transport', () => {
it('deduplicates and sorts resource IDs for a stable bounded cache key and request', () => {
expect(normalizeProtectionPostureResourceIDs(['vm:b', ' vm:a ', '', 'vm:b'])).toEqual([
'vm:a',
'vm:b',
]);
const url = new URL(
buildProtectionPostureBatchURL(['vm:b', 'vm:a', 'vm:b']),
'https://pulse.invalid',
);
expect(url.pathname).toBe('/api/recovery/postures');
expect(url.searchParams.getAll('resourceId')).toEqual(['vm:a', 'vm:b']);
expect(url.searchParams.get('limit')).toBe(String(MAX_PROTECTION_POSTURE_BATCH_SIZE));
});
it('rejects a batch larger than the server contract', () => {
const resourceIDs = Array.from(
{ length: MAX_PROTECTION_POSTURE_BATCH_SIZE + 1 },
(_, index) => `vm:${index}`,
);
expect(() => buildProtectionPostureBatchURL(resourceIDs)).toThrow(/limited to 200/);
});
});
frontend-modern/src/hooks/useProtectionPostures.ts
+115
View File
@@ -0,0 +1,115 @@
import { type Accessor, createMemo } from 'solid-js';
import { createNonSuspendingQuery } from '@/hooks/createNonSuspendingQuery';
import type {
ProtectionPosture,
ProtectionPosturePolicy,
ProtectionPosturesResponse,
} from '@/types/recovery';
import { apiFetchJSON } from '@/utils/apiClient';
const PROTECTION_POSTURES_URL = '/api/recovery/postures';
export const MAX_PROTECTION_POSTURE_BATCH_SIZE = 200;
const REFRESH_MS = 30_000;
const EMPTY_POLICY: ProtectionPosturePolicy = {
freshnessWindowSeconds: 0,
verificationWindowSeconds: 0,
requireVerification: false,
};
const EMPTY_RESPONSE: ProtectionPosturesResponse = {
data: [],
policy: EMPTY_POLICY,
meta: { page: 1, limit: MAX_PROTECTION_POSTURE_BATCH_SIZE, total: 0, totalPages: 0 },
};
export function normalizeProtectionPostureResourceIDs(
resourceIDs: readonly string[] | null | undefined,
): string[] {
return [
...new Set(
(resourceIDs ?? []).map((resourceID) => resourceID.trim()).filter((resourceID) => resourceID),
),
].sort((left, right) => left.localeCompare(right));
}
export function buildProtectionPostureBatchURL(resourceIDs: readonly string[]): string {
const normalized = normalizeProtectionPostureResourceIDs(resourceIDs);
if (normalized.length > MAX_PROTECTION_POSTURE_BATCH_SIZE) {
throw new Error(
`Protection posture batches are limited to ${MAX_PROTECTION_POSTURE_BATCH_SIZE} resource IDs.`,
);
}
const params = new URLSearchParams();
for (const resourceID of normalized) {
params.append('resourceId', resourceID);
}
params.set('limit', String(MAX_PROTECTION_POSTURE_BATCH_SIZE));
return `${PROTECTION_POSTURES_URL}?${params.toString()}`;
}
async function fetchProtectionPostures(resourceIDs: readonly string[]) {
return apiFetchJSON<ProtectionPosturesResponse>(buildProtectionPostureBatchURL(resourceIDs));
}
/**
* Fetch server-owned protection posture in bounded batches. A fleet may exceed
* one API batch, but request count scales by 200-row pages rather than by row.
*/
async function fetchAllProtectionPostures(
resourceIDs: readonly string[],
): Promise<ProtectionPosturesResponse> {
const batches: string[][] = [];
for (let offset = 0; offset < resourceIDs.length; offset += MAX_PROTECTION_POSTURE_BATCH_SIZE) {
batches.push(resourceIDs.slice(offset, offset + MAX_PROTECTION_POSTURE_BATCH_SIZE));
}
const responses = await Promise.all(batches.map(fetchProtectionPostures));
const data = responses.flatMap((response) => response.data ?? []);
return {
data,
policy: responses[0]?.policy ?? EMPTY_POLICY,
meta: {
page: 1,
limit: MAX_PROTECTION_POSTURE_BATCH_SIZE,
total: data.length,
totalPages: responses.length,
},
};
}
export function useProtectionPostures(resourceIDs: Accessor<readonly string[] | null | undefined>) {
const source = createMemo<string | null>(() => {
const normalized = normalizeProtectionPostureResourceIDs(resourceIDs());
return normalized.length > 0 ? JSON.stringify(normalized) : null;
});
const state = createNonSuspendingQuery<ProtectionPosturesResponse, string>({
source,
cacheKey: (key) => `protection-postures:${key}`,
fetcher: (key) => fetchAllProtectionPostures(JSON.parse(key) as string[]),
initialValue: EMPTY_RESPONSE,
pollMs: REFRESH_MS,
});
const postures = createMemo<ProtectionPosture[]>(() => state.value().data ?? []);
const postureByResourceID = createMemo<ReadonlyMap<string, ProtectionPosture>>(
() => new Map(postures().map((posture) => [posture.subjectResourceId, posture] as const)),
);
const policy = createMemo(() => state.value().policy ?? EMPTY_POLICY);
return {
response: {
get error() {
return state.error();
},
get loading() {
return state.loading();
},
},
postures,
postureByResourceID,
policy,
refetch: state.refetch,
resolvedOnce: state.resolvedOnce,
};
}
frontend-modern/src/types/recovery.ts
+53 -6
View File
@@ -1,13 +1,10 @@
import type { EvidenceEnvelope, EvidencePermissions } from '@/types/operationalTrust';
export type RecoveryPlatform = string;
export type RecoveryKind = 'snapshot' | 'backup' | 'other' | (string & {});
export type RecoveryMode = 'snapshot' | 'local' | 'remote' | (string & {});
export type RecoveryOutcome =
| 'success'
| 'warning'
| 'failed'
| 'running'
| 'unknown'
| (string & {});
'success' | 'warning' | 'failed' | 'running' | 'unknown' | (string & {});
export interface RecoveryExternalRef {
type: string;
@@ -59,6 +56,8 @@ export interface RecoveryPoint {
itemResourceId?: string;
repositoryResourceId?: string;
providerScope?: string;
evidence?: EvidenceEnvelope | null;
itemRef?: RecoveryExternalRef | null;
subjectRef?: RecoveryExternalRef | null;
repositoryRef?: RecoveryExternalRef | null;
@@ -146,3 +145,51 @@ export interface RecoveryPointsFacets {
hasVerification?: boolean;
hasEntityId?: boolean;
}
export type ProtectionState = 'protected' | 'attention' | 'unprotected' | 'unknown';
export type ProtectionFreshness = 'current' | 'stale' | 'unknown';
export type ProtectionVerification = 'verified' | 'unverified' | 'stale' | 'unknown';
export type ProtectionCoverage = 'complete' | 'partial' | 'none' | 'unknown';
export type ProtectionHistoryCompleteness = 'complete' | 'partial' | 'unavailable' | 'unknown';
export interface ProtectionProviderState {
provider: RecoveryPlatform;
source: string;
scope: string;
jobState: RecoveryOutcome;
historyCompleteness: ProtectionHistoryCompleteness;
permissions: EvidencePermissions;
lastAttemptAt?: string | null;
lastSuccessAt?: string | null;
lastVerifiedAt?: string | null;
evidenceIds: string[];
verificationExpected?: boolean;
}
export interface ProtectionPosture {
subjectResourceId: string;
state: ProtectionState;
lastAttemptAt?: string | null;
lastSuccessfulPointAt?: string | null;
lastVerifiedAt?: string | null;
freshness: ProtectionFreshness;
verification: ProtectionVerification;
coverage: ProtectionCoverage;
providerStates: ProtectionProviderState[];
repositoryResourceIds: string[];
evidenceIds: string[];
explanation: string;
evaluatedAt: string;
}
export interface ProtectionPosturePolicy {
freshnessWindowSeconds: number;
verificationWindowSeconds: number;
requireVerification: boolean;
}
export interface ProtectionPosturesResponse {
data: ProtectionPosture[];
policy: ProtectionPosturePolicy;
meta: RecoveryResponseMeta;
}
internal/api/recovery_handlers.go
+259 -8
View File
@@ -11,6 +11,7 @@ import (
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/mock"
"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
"github.com/rcourtman/pulse-go-rewrite/internal/recovery"
recoverymanager "github.com/rcourtman/pulse-go-rewrite/internal/recovery/manager"
kubernetesmapper "github.com/rcourtman/pulse-go-rewrite/internal/recovery/mapper/kubernetes"
@@ -63,14 +64,16 @@ type recoveryPointPayload struct {
Encrypted *bool `json:"encrypted,omitempty"`
Immutable *bool `json:"immutable,omitempty"`
ItemResourceID string `json:"itemResourceId,omitempty"`
SubjectResourceID string `json:"subjectResourceId,omitempty"`
RepositoryResourceID string `json:"repositoryResourceId,omitempty"`
ItemRef *recovery.ExternalRef `json:"itemRef,omitempty"`
SubjectRef *recovery.ExternalRef `json:"subjectRef,omitempty"`
RepositoryRef *recovery.ExternalRef `json:"repositoryRef,omitempty"`
Details map[string]any `json:"details,omitempty"`
Display *recovery.RecoveryPointDisplay `json:"display,omitempty"`
ItemResourceID string `json:"itemResourceId,omitempty"`
SubjectResourceID string `json:"subjectResourceId,omitempty"`
RepositoryResourceID string `json:"repositoryResourceId,omitempty"`
ItemRef *recovery.ExternalRef `json:"itemRef,omitempty"`
SubjectRef *recovery.ExternalRef `json:"subjectRef,omitempty"`
RepositoryRef *recovery.ExternalRef `json:"repositoryRef,omitempty"`
Details map[string]any `json:"details,omitempty"`
ProviderScope string `json:"providerScope,omitempty"`
Evidence *operationaltrust.EvidenceEnvelope `json:"evidence,omitempty"`
Display *recovery.RecoveryPointDisplay `json:"display,omitempty"`
}
type recoveryRollupPayload struct {
@@ -85,6 +88,8 @@ type recoveryRollupPayload struct {
LastOutcome recovery.Outcome `json:"lastOutcome"`
Platforms []recovery.Provider `json:"platforms,omitempty"`
Providers []recovery.Provider `json:"providers,omitempty"`
VerifyIntent recovery.VerifyIntent `json:"verifyIntent,omitempty"`
LastVerifiedAt *time.Time `json:"lastVerifiedAt,omitempty"`
}
func buildRecoveryPointPayload(point recovery.RecoveryPoint) recoveryPointPayload {
@@ -108,6 +113,8 @@ func buildRecoveryPointPayload(point recovery.RecoveryPoint) recoveryPointPayloa
SubjectRef: point.SubjectRef,
RepositoryRef: point.RepositoryRef,
Details: point.Details,
ProviderScope: point.ProviderScope,
Evidence: point.Evidence,
Display: point.Display,
}
}
@@ -125,6 +132,8 @@ func buildRecoveryRollupPayload(rollup recovery.ProtectionRollup) recoveryRollup
LastOutcome: rollup.LastOutcome,
Platforms: rollup.Providers,
Providers: rollup.Providers,
VerifyIntent: rollup.VerifyIntent,
LastVerifiedAt: rollup.LastVerifiedAt,
}
}
@@ -699,6 +708,248 @@ func (h *RecoveryHandlers) HandleListRollups(w http.ResponseWriter, r *http.Requ
}
}
const maxProtectionPostureResourceIDs = 200
type protectionPosturesResponse struct {
Data []recovery.ProtectionPosture `json:"data"`
Policy recovery.ProtectionPosturePolicyPayload `json:"policy"`
Meta struct {
Page int `json:"page"`
Limit int `json:"limit"`
Total int `json:"total"`
TotalPages int `json:"totalPages"`
} `json:"meta"`
}
func parseProtectionPostureResourceIDs(qs url.Values) []string {
values := make([]string, 0, len(qs["resourceId"])+len(qs["resourceIds"]))
values = append(values, qs["resourceId"]...)
values = append(values, qs["resourceIds"]...)
unique := make(map[string]struct{}, len(values))
for _, value := range values {
for _, part := range strings.Split(value, ",") {
if part = strings.TrimSpace(part); part != "" {
unique[part] = struct{}{}
}
}
}
out := make([]string, 0, len(unique))
for value := range unique {
out = append(out, value)
}
sort.Strings(out)
return out
}
func (h *RecoveryHandlers) HandleListProtectionPostures(
w http.ResponseWriter,
r *http.Request,
) {
if r.Method != http.MethodGet {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
return
}
qs := r.URL.Query()
resourceIDs := parseProtectionPostureResourceIDs(qs)
if len(resourceIDs) > maxProtectionPostureResourceIDs {
writeErrorResponse(
w,
http.StatusBadRequest,
"too_many_resource_ids",
"At most 200 resource IDs may be evaluated in one posture batch.",
map[string]string{"limit": strconv.Itoa(maxProtectionPostureResourceIDs)},
)
return
}
state := recovery.ProtectionState(strings.TrimSpace(qs.Get("state")))
if state != "" && !state.Valid() {
writeErrorResponse(
w,
http.StatusBadRequest,
"invalid_protection_state",
"Protection state must be protected, attention, unprotected, or unknown.",
map[string]string{"state": string(state)},
)
return
}
page := parseIntQuery(qs, "page", 1)
limit := parseIntQuery(qs, "limit", 100)
if limit <= 0 {
limit = 100
}
if limit > maxProtectionPostureResourceIDs {
limit = maxProtectionPostureResourceIDs
}
query := recovery.ProtectionPostureQuery{
SubjectResourceIDs: resourceIDs,
State: state,
Page: page,
Limit: limit,
}
var (
postures []recovery.ProtectionPosture
total int
err error
)
if mock.IsMockEnabled() {
postures, total = mockProtectionPostures(
mock.CurrentFixtureGraph().RecoveryPoints(),
query,
time.Now().UTC(),
)
} else {
orgID := GetOrgID(r.Context())
store, storeErr := h.storeForOrg(orgID)
if storeErr != nil {
http.Error(
w,
sanitizeErrorForClient(storeErr, "Internal server error"),
http.StatusInternalServerError,
)
return
}
postures, total, err = store.ListProtectionPostures(r.Context(), query)
if err != nil {
http.Error(
w,
sanitizeErrorForClient(err, "Internal server error"),
http.StatusInternalServerError,
)
return
}
}
var response protectionPosturesResponse
response.Data = postures
response.Policy = recovery.DefaultProtectionPosturePolicy.Payload()
response.Meta.Page = page
response.Meta.Limit = limit
response.Meta.Total = total
if total == 0 {
response.Meta.TotalPages = 0
} else {
response.Meta.TotalPages = (total + limit - 1) / limit
}
if err := utils.WriteJSONResponse(w, response); err != nil {
log.Error().Err(err).Msg("Failed to serialize protection posture response")
}
}
func mockProtectionPostures(
points []recovery.RecoveryPoint,
query recovery.ProtectionPostureQuery,
now time.Time,
) ([]recovery.ProtectionPosture, int) {
pointsByResource := make(map[string][]recovery.RecoveryPoint)
observationsByKey := make(map[string]recovery.ProtectionProviderObservation)
for _, point := range points {
resourceID := strings.TrimSpace(point.SubjectResourceID)
if resourceID == "" {
continue
}
pointsByResource[resourceID] = append(pointsByResource[resourceID], point)
scope := recovery.ProviderScopeForPoint(point)
key := string(point.Provider) + "\x00" + scope
if _, exists := observationsByKey[key]; exists {
continue
}
observation, err := recovery.NewProtectionProviderObservation(
point.Provider,
"mock-complete-recovery-fixture",
scope,
recovery.OutcomeUnknown,
recovery.ProtectionHistoryComplete,
operationaltrust.EvidencePermissionsSufficient,
point.Provider == recovery.ProviderProxmoxPBS,
now,
now,
nil,
)
if err == nil {
observationsByKey[key] = observation
}
}
observations := make([]recovery.ProtectionProviderObservation, 0, len(observationsByKey))
for _, observation := range observationsByKey {
observations = append(observations, observation)
}
resourceIDs := append([]string(nil), query.SubjectResourceIDs...)
if len(resourceIDs) == 0 {
for resourceID := range pointsByResource {
resourceIDs = append(resourceIDs, resourceID)
}
sort.Strings(resourceIDs)
}
postures := make([]recovery.ProtectionPosture, 0, len(resourceIDs))
for _, resourceID := range resourceIDs {
posture := recovery.BuildProtectionPostureFromPointsAt(
resourceID,
pointsByResource[resourceID],
observations,
recovery.DefaultProtectionPosturePolicy,
now,
)
if query.State != "" && posture.State != query.State {
continue
}
postures = append(postures, posture)
}
sort.SliceStable(postures, func(i, j int) bool {
iRank := protectionPostureStateRank(postures[i].State)
jRank := protectionPostureStateRank(postures[j].State)
if iRank != jRank {
return iRank < jRank
}
return postures[i].SubjectResourceID < postures[j].SubjectResourceID
})
total := len(postures)
if len(query.SubjectResourceIDs) > 0 {
return postures, total
}
return paginateProtectionPostures(postures, query.Page, query.Limit), total
}
func protectionPostureStateRank(state recovery.ProtectionState) int {
switch state {
case recovery.ProtectionStateAttention:
return 0
case recovery.ProtectionStateUnprotected:
return 1
case recovery.ProtectionStateUnknown:
return 2
default:
return 3
}
}
func paginateProtectionPostures(
postures []recovery.ProtectionPosture,
page int,
limit int,
) []recovery.ProtectionPosture {
if len(postures) == 0 {
return []recovery.ProtectionPosture{}
}
if page <= 0 {
page = 1
}
if limit <= 0 {
limit = 100
}
start := (page - 1) * limit
if start >= len(postures) {
return []recovery.ProtectionPosture{}
}
end := start + limit
if end > len(postures) {
end = len(postures)
}
return postures[start:end]
}
func filterRecoveryPointsForRollups(all []recovery.RecoveryPoint, opts recovery.ListPointsOptions) []recovery.RecoveryPoint {
if len(all) == 0 {
return nil
internal/api/recovery_handlers_test.go
+135
View File
@@ -8,10 +8,12 @@ import (
"net/http/httptest"
"net/url"
"path/filepath"
"strconv"
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/config"
"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
"github.com/rcourtman/pulse-go-rewrite/internal/recovery"
recoverymanager "github.com/rcourtman/pulse-go-rewrite/internal/recovery/manager"
_ "modernc.org/sqlite"
@@ -207,10 +209,13 @@ func TestHandleListRollupsExposeCanonicalPlatformsPayload(t *testing.T) {
}
func TestBuildRecoveryRollupPayloadExposesCanonicalItemResourceIDField(t *testing.T) {
verifiedAt := time.Date(2026, 7, 19, 6, 0, 0, 0, time.UTC)
payload := buildRecoveryRollupPayload(recovery.ProtectionRollup{
RollupID: "res:vm-123",
SubjectResourceID: "vm-123",
LastOutcome: recovery.Outcome("success"),
VerifyIntent: recovery.VerifyIntentVerified,
LastVerifiedAt: &verifiedAt,
})
if payload.ItemResourceID != "vm-123" {
@@ -219,6 +224,136 @@ func TestBuildRecoveryRollupPayloadExposesCanonicalItemResourceIDField(t *testin
if payload.SubjectResourceID != "vm-123" {
t.Fatalf("payload.SubjectResourceID = %q, want %q", payload.SubjectResourceID, "vm-123")
}
if payload.VerifyIntent != recovery.VerifyIntentVerified {
t.Fatalf("payload.VerifyIntent = %q, want verified", payload.VerifyIntent)
}
if payload.LastVerifiedAt == nil || !payload.LastVerifiedAt.Equal(verifiedAt) {
t.Fatalf("payload.LastVerifiedAt = %v, want %v", payload.LastVerifiedAt, verifiedAt)
}
}
func TestHandleListProtectionPosturesReturnsBoundedCanonicalBatch(t *testing.T) {
t.Parallel()
mtp := config.NewMultiTenantPersistence(t.TempDir())
manager := recoverymanager.New(mtp)
store, err := manager.StoreForOrg("default")
if err != nil {
t.Fatalf("StoreForOrg(default): %v", err)
}
now := time.Now().UTC().Truncate(time.Millisecond)
completedAt := now.Add(-time.Hour)
verified := true
point := recovery.RecoveryPoint{
ID: "pbs-backup:vm-123",
Provider: recovery.ProviderProxmoxPBS,
Kind: recovery.KindBackup,
Mode: recovery.ModeRemote,
Outcome: recovery.OutcomeSuccess,
CompletedAt: &completedAt,
Verified: &verified,
SubjectResourceID: "vm-123",
ProviderScope: "pbs-main",
}
evidence, err := recovery.NewRecoveryPointEvidence(
point,
"pbs-backup-inventory",
now,
)
if err != nil {
t.Fatalf("NewRecoveryPointEvidence() error = %v", err)
}
point.Evidence = evidence
observation, err := recovery.NewProtectionProviderObservation(
recovery.ProviderProxmoxPBS,
"pbs-backup-enumeration",
"pbs-main",
recovery.OutcomeSuccess,
recovery.ProtectionHistoryComplete,
operationaltrust.EvidencePermissionsSufficient,
true,
now,
now,
nil,
)
if err != nil {
t.Fatalf("NewProtectionProviderObservation() error = %v", err)
}
if err := store.UpsertProtectionProviderObservations(
context.Background(),
[]recovery.ProtectionProviderObservation{observation},
); err != nil {
t.Fatalf("UpsertProtectionProviderObservations() error = %v", err)
}
if err := store.UpsertPoints(context.Background(), []recovery.RecoveryPoint{point}); err != nil {
t.Fatalf("UpsertPoints() error = %v", err)
}
handler := NewRecoveryHandlers(manager)
request := httptest.NewRequest(
http.MethodGet,
"/api/recovery/postures?resourceId=vm-123&resourceId=vm-missing",
nil,
)
response := httptest.NewRecorder()
handler.HandleListProtectionPostures(response, request)
if response.Code != http.StatusOK {
t.Fatalf(
"HandleListProtectionPostures() status = %d, want 200 body=%s",
response.Code,
response.Body.String(),
)
}
var payload struct {
Data []recovery.ProtectionPosture `json:"data"`
Policy struct {
FreshnessWindowSeconds int64 `json:"freshnessWindowSeconds"`
VerificationWindowSeconds int64 `json:"verificationWindowSeconds"`
} `json:"policy"`
Meta struct {
Total int `json:"total"`
Limit int `json:"limit"`
} `json:"meta"`
}
if err := json.Unmarshal(response.Body.Bytes(), &payload); err != nil {
t.Fatalf("json.Unmarshal() error = %v", err)
}
if payload.Meta.Total != 2 || len(payload.Data) != 2 {
t.Fatalf("postures total=%d len=%d, want 2/2", payload.Meta.Total, len(payload.Data))
}
byID := make(map[string]recovery.ProtectionPosture)
for _, posture := range payload.Data {
byID[posture.SubjectResourceID] = posture
}
if byID["vm-123"].State != recovery.ProtectionStateProtected {
t.Fatalf("vm-123 posture = %#v, want protected", byID["vm-123"])
}
if byID["vm-missing"].State != recovery.ProtectionStateUnknown {
t.Fatalf("vm-missing posture = %#v, want unknown", byID["vm-missing"])
}
if payload.Policy.FreshnessWindowSeconds <= 0 ||
payload.Policy.VerificationWindowSeconds <= 0 {
t.Fatalf("policy = %#v, want positive server evaluation windows", payload.Policy)
}
}
func TestHandleListProtectionPosturesRejectsUnboundedBatch(t *testing.T) {
t.Parallel()
values := url.Values{}
for i := 0; i <= maxProtectionPostureResourceIDs; i++ {
values.Add("resourceId", "resource-"+strconv.Itoa(i))
}
request := httptest.NewRequest(
http.MethodGet,
"/api/recovery/postures?"+values.Encode(),
nil,
)
response := httptest.NewRecorder()
NewRecoveryHandlers(nil).HandleListProtectionPostures(response, request)
if response.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want 400 body=%s", response.Code, response.Body.String())
}
}
func TestBuildRecoveryRollupPayloadExposesCanonicalItemRefField(t *testing.T) {
internal/api/route_inventory_test.go
+1
View File
@@ -409,6 +409,7 @@ var allRouteAllowlist = []string{
"/api/recovery/series",
"/api/recovery/facets",
"/api/recovery/rollups",
"/api/recovery/postures",
"/api/resources",
"/api/resources/storage-incidents",
"/api/resources/storage-summary",
internal/api/router_routes_monitoring.go
+1
View File
@@ -29,6 +29,7 @@ func (r *Router) registerMonitoringResourceRoutes(
r.mux.HandleFunc("/api/recovery/series", RequireAuth(r.config, RequireScope(config.ScopeMonitoringRead, r.recoveryHandlers.HandleListSeries)))
r.mux.HandleFunc("/api/recovery/facets", RequireAuth(r.config, RequireScope(config.ScopeMonitoringRead, r.recoveryHandlers.HandleListFacets)))
r.mux.HandleFunc("/api/recovery/rollups", RequireAuth(r.config, RequireScope(config.ScopeMonitoringRead, r.recoveryHandlers.HandleListRollups)))
r.mux.HandleFunc("/api/recovery/postures", RequireAuth(r.config, RequireScope(config.ScopeMonitoringRead, r.recoveryHandlers.HandleListProtectionPostures)))
r.registerReplicationRoutes()
r.registerPVEBackupsRoutes()
r.registerPBSBackupsRoutes()
internal/monitoring/monitor_backups.go
+46 -2
View File
@@ -1582,8 +1582,30 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien
Int("count", len(allBackups)).
Msg("PBS backups fetched")
protectionObservedAt := time.Now().UTC()
protectionObservation, protectionObservationErr :=
buildPBSProtectionProviderObservation(
instanceName,
datastoreCount,
datastoreFetches,
datastoreErrors,
datastoreTerminalFailures,
protectionObservedAt,
)
if protectionObservationErr != nil {
log.Warn().
Err(protectionObservationErr).
Str("instance", instanceName).
Msg("Failed to build PBS protection provider observation")
}
// Decide whether to keep existing backups when all queries failed
if shouldPreservePBSBackupsWithTerminal(datastoreCount, datastoreFetches, datastoreTerminalFailures) {
if protectionObservationErr == nil {
m.ingestProtectionProviderObservationsAsync(
[]recovery.ProtectionProviderObservation{protectionObservation},
)
}
log.Warn().
Str("instance", instanceName).
Int("datastores", datastoreCount).
@@ -1600,8 +1622,30 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien
// Best-effort ingestion into recovery store (for rollups / unified backups UX).
candidates := buildPBSGuestCandidates(m.GetUnifiedReadStateOrSnapshot())
m.ingestAndReconcileRecoveryPointsAsync(
proxmoxrecoverymapper.FromPBSBackups(allBackups, candidates),
points, evidenceErr := proxmoxrecoverymapper.FromPBSBackupsWithEvidence(
allBackups,
candidates,
protectionObservedAt,
)
if evidenceErr != nil {
log.Warn().
Err(evidenceErr).
Str("instance", instanceName).
Msg("Failed to attach PBS recovery evidence; preserving existing points")
if protectionObservationErr == nil {
m.ingestProtectionProviderObservationsAsync(
[]recovery.ProtectionProviderObservation{protectionObservation},
)
}
return
}
observations := []recovery.ProtectionProviderObservation{}
if protectionObservationErr == nil {
observations = append(observations, protectionObservation)
}
m.ingestAndReconcileRecoveryPointsWithObservationsAsync(
points,
observations,
recoveryReconcileScope{
provider: string(recovery.ProviderProxmoxPBS),
idPrefix: "pbs-backup:",
internal/monitoring/pbs_protection_observation.go
+86
View File
@@ -0,0 +1,86 @@
package monitoring
import (
"fmt"
"strings"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
"github.com/rcourtman/pulse-go-rewrite/internal/recovery"
)
func buildPBSProtectionProviderObservation(
instanceName string,
datastoreCount int,
datastoreFetches int,
datastoreErrors int,
datastoreTerminalFailures int,
observedAt time.Time,
) (recovery.ProtectionProviderObservation, error) {
instanceName = strings.TrimSpace(instanceName)
if instanceName == "" {
return recovery.ProtectionProviderObservation{}, fmt.Errorf(
"PBS protection observation requires an instance",
)
}
if observedAt.IsZero() {
return recovery.ProtectionProviderObservation{}, fmt.Errorf(
"PBS protection observation requires an observation time",
)
}
jobState := recovery.OutcomeSuccess
historyCompleteness := recovery.ProtectionHistoryComplete
permissions := operationaltrust.EvidencePermissionsSufficient
var reason *operationaltrust.EvidenceReason
switch {
case datastoreErrors == 0:
// A successful empty enumeration is complete evidence that the
// connection currently exposes no backup history.
case datastoreFetches > 0:
jobState = recovery.OutcomeWarning
historyCompleteness = recovery.ProtectionHistoryPartial
permissions = operationaltrust.EvidencePermissionsUnknown
reason = &operationaltrust.EvidenceReason{
Code: "pbs_partial_enumeration",
Message: "Some PBS datastore or namespace history could not be enumerated.",
}
if datastoreTerminalFailures > 0 {
permissions = operationaltrust.EvidencePermissionsPartial
reason = &operationaltrust.EvidenceReason{
Code: "pbs_partial_provider_access",
Message: "PBS authorized only part of the configured backup-history scope.",
}
}
default:
jobState = recovery.OutcomeFailed
historyCompleteness = recovery.ProtectionHistoryUnavailable
permissions = operationaltrust.EvidencePermissionsUnknown
reason = &operationaltrust.EvidenceReason{
Code: "pbs_collection_unavailable",
Message: "PBS backup history could not be collected; retained points may be stale.",
}
if datastoreCount > 0 &&
datastoreTerminalFailures >= datastoreCount {
permissions = operationaltrust.EvidencePermissionsDenied
reason = &operationaltrust.EvidenceReason{
Code: "pbs_provider_access_denied",
Message: "PBS rejected every configured datastore history request.",
}
}
}
return recovery.NewProtectionProviderObservation(
recovery.ProviderProxmoxPBS,
"pbs-backup-enumeration",
instanceName,
jobState,
historyCompleteness,
permissions,
true,
observedAt,
observedAt,
reason,
)
}
internal/monitoring/pbs_protection_observation_test.go
+117
View File
@@ -0,0 +1,117 @@
package monitoring
import (
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
"github.com/rcourtman/pulse-go-rewrite/internal/recovery"
)
func TestBuildPBSProtectionProviderObservationMapsEvidenceQuality(t *testing.T) {
t.Parallel()
now := time.Date(2026, 7, 19, 8, 0, 0, 0, time.UTC)
tests := []struct {
name string
datastores int
fetches int
errors int
terminalFailures int
wantJob recovery.Outcome
wantHistory recovery.ProtectionHistoryCompleteness
wantPermissions operationaltrust.EvidencePermissions
wantReason string
}{
{
name: "complete enumeration",
datastores: 2,
fetches: 2,
wantJob: recovery.OutcomeSuccess,
wantHistory: recovery.ProtectionHistoryComplete,
wantPermissions: operationaltrust.EvidencePermissionsSufficient,
},
{
name: "partial transient enumeration",
datastores: 2,
fetches: 1,
errors: 1,
wantJob: recovery.OutcomeWarning,
wantHistory: recovery.ProtectionHistoryPartial,
wantPermissions: operationaltrust.EvidencePermissionsUnknown,
wantReason: "pbs_partial_enumeration",
},
{
name: "partial provider access",
datastores: 2,
fetches: 1,
errors: 1,
terminalFailures: 1,
wantJob: recovery.OutcomeWarning,
wantHistory: recovery.ProtectionHistoryPartial,
wantPermissions: operationaltrust.EvidencePermissionsPartial,
wantReason: "pbs_partial_provider_access",
},
{
name: "provider timeout",
datastores: 2,
errors: 2,
wantJob: recovery.OutcomeFailed,
wantHistory: recovery.ProtectionHistoryUnavailable,
wantPermissions: operationaltrust.EvidencePermissionsUnknown,
wantReason: "pbs_collection_unavailable",
},
{
name: "provider denies every datastore",
datastores: 2,
errors: 2,
terminalFailures: 2,
wantJob: recovery.OutcomeFailed,
wantHistory: recovery.ProtectionHistoryUnavailable,
wantPermissions: operationaltrust.EvidencePermissionsDenied,
wantReason: "pbs_provider_access_denied",
},
}
for _, test := range tests {
test := test
t.Run(test.name, func(t *testing.T) {
t.Parallel()
got, err := buildPBSProtectionProviderObservation(
"pbs-main",
test.datastores,
test.fetches,
test.errors,
test.terminalFailures,
now,
)
if err != nil {
t.Fatalf("buildPBSProtectionProviderObservation() error = %v", err)
}
if got.JobState != test.wantJob {
t.Fatalf("job state = %q, want %q", got.JobState, test.wantJob)
}
if got.HistoryCompleteness != test.wantHistory {
t.Fatalf(
"history completeness = %q, want %q",
got.HistoryCompleteness,
test.wantHistory,
)
}
if got.Permissions != test.wantPermissions {
t.Fatalf("permissions = %q, want %q", got.Permissions, test.wantPermissions)
}
if test.wantReason == "" {
if got.Evidence.Reason != nil {
t.Fatalf("reason = %#v, want nil", got.Evidence.Reason)
}
} else if got.Evidence.Reason == nil ||
got.Evidence.Reason.Code != test.wantReason {
t.Fatalf("reason = %#v, want %q", got.Evidence.Reason, test.wantReason)
}
if err := got.Validate(); err != nil {
t.Fatalf("Validate() error = %v", err)
}
})
}
}
internal/monitoring/recovery_ingest.go
+46 -4
View File
@@ -24,8 +24,9 @@ type recoveryReconcileScope struct {
// recoveryIngestBatch is one poll cycle's worth of recovery points, plus an
// optional reconcile scope when the points are a complete enumeration.
type recoveryIngestBatch struct {
points []recovery.RecoveryPoint
reconcile *recoveryReconcileScope
points []recovery.RecoveryPoint
observations []recovery.ProtectionProviderObservation
reconcile *recoveryReconcileScope
}
func (m *Monitor) ingestRecoveryPointsAsync(points []recovery.RecoveryPoint) {
@@ -39,8 +40,29 @@ func (m *Monitor) ingestAndReconcileRecoveryPointsAsync(points []recovery.Recove
m.enqueueRecoveryIngest(recoveryIngestBatch{points: points, reconcile: &scope})
}
func (m *Monitor) ingestAndReconcileRecoveryPointsWithObservationsAsync(
points []recovery.RecoveryPoint,
observations []recovery.ProtectionProviderObservation,
scope recoveryReconcileScope,
) {
m.enqueueRecoveryIngest(recoveryIngestBatch{
points: points,
observations: observations,
reconcile: &scope,
})
}
func (m *Monitor) ingestProtectionProviderObservationsAsync(
observations []recovery.ProtectionProviderObservation,
) {
m.enqueueRecoveryIngest(recoveryIngestBatch{observations: observations})
}
func (m *Monitor) enqueueRecoveryIngest(batch recoveryIngestBatch) {
if m == nil || (len(batch.points) == 0 && batch.reconcile == nil) {
if m == nil ||
(len(batch.points) == 0 &&
len(batch.observations) == 0 &&
batch.reconcile == nil) {
return
}
@@ -53,6 +75,7 @@ func (m *Monitor) enqueueRecoveryIngest(batch recoveryIngestBatch) {
m.recoveryIngestMu.Unlock()
log.Debug().
Int("points", len(batch.points)).
Int("provider_observations", len(batch.observations)).
Msg("Queued recovery point ingest behind active batch")
return
}
@@ -88,7 +111,10 @@ func (m *Monitor) runRecoveryPointIngestLoop(batches []recoveryIngestBatch) {
}
func (m *Monitor) ingestRecoveryPointsBestEffort(ctx context.Context, batch recoveryIngestBatch) {
if m == nil || (len(batch.points) == 0 && batch.reconcile == nil) {
if m == nil ||
(len(batch.points) == 0 &&
len(batch.observations) == 0 &&
batch.reconcile == nil) {
return
}
@@ -110,6 +136,22 @@ func (m *Monitor) ingestRecoveryPointsBestEffort(ctx context.Context, batch reco
return
}
// Collection-wide evidence is the authority for how much provider history
// Pulse could actually see. Persist it before any point mutation so a large
// point batch, refresh timeout, or reconciliation failure cannot leave
// retained artifacts looking more trustworthy than the poll that produced
// them.
if err := store.UpsertProtectionProviderObservations(
ctx,
batch.observations,
); err != nil {
log.Warn().
Err(err).
Str("org_id", orgID).
Int("provider_observations", len(batch.observations)).
Msg("Failed to upsert provider protection observations")
return
}
if err := store.UpsertPoints(ctx, batch.points); err != nil {
log.Warn().Err(err).Str("org_id", orgID).Int("points", len(batch.points)).Msg("Failed to upsert recovery points from backup polling")
// Do not reconcile against a batch that failed to land; deleting on
internal/monitoring/recovery_ingest_test.go
+105
View File
@@ -6,6 +6,7 @@ import (
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/config"
"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
"github.com/rcourtman/pulse-go-rewrite/internal/recovery"
recoverymanager "github.com/rcourtman/pulse-go-rewrite/internal/recovery/manager"
)
@@ -162,3 +163,107 @@ func TestIngestRecoveryPointsBestEffortSkipsReconcileWhenUpsertFails(t *testing.
t.Fatal("reconcile must be skipped when the upsert fails; vzdump-101 was deleted")
}
}
func TestIngestRecoveryPointsBestEffortPersistsProviderObservationAndPosture(t *testing.T) {
t.Parallel()
m, manager := recoveryIngestTestMonitor(t)
now := time.Now().UTC().Truncate(time.Millisecond)
point := recoveryIngestTestPoint("pve-backup:pve1-vzdump-100", "pve1", now)
point.SubjectResourceID = "resource:vm-100"
observation, err := recovery.NewProtectionProviderObservation(
recovery.ProviderProxmoxPVE,
"pve-backup-enumeration",
"pve1",
recovery.OutcomeSuccess,
recovery.ProtectionHistoryComplete,
operationaltrust.EvidencePermissionsSufficient,
false,
now,
now,
nil,
)
if err != nil {
t.Fatalf("NewProtectionProviderObservation() error = %v", err)
}
m.ingestRecoveryPointsBestEffort(context.Background(), recoveryIngestBatch{
points: []recovery.RecoveryPoint{point},
observations: []recovery.ProtectionProviderObservation{
observation,
},
})
store, err := manager.StoreForOrg("default")
if err != nil {
t.Fatalf("StoreForOrg(default): %v", err)
}
postures, _, err := store.ListProtectionPostures(
context.Background(),
recovery.ProtectionPostureQuery{
SubjectResourceIDs: []string{"resource:vm-100"},
},
)
if err != nil {
t.Fatalf("ListProtectionPostures() error = %v", err)
}
if len(postures) != 1 ||
postures[0].State != recovery.ProtectionStateProtected {
t.Fatalf("postures = %#v, want one protected posture", postures)
}
}
func TestIngestRecoveryPointsBestEffortPersistsProviderObservationBeforePointFailure(t *testing.T) {
t.Parallel()
m, manager := recoveryIngestTestMonitor(t)
store, err := manager.StoreForOrg("default")
if err != nil {
t.Fatalf("StoreForOrg(default): %v", err)
}
now := time.Now().UTC().Truncate(time.Millisecond)
existing := recoveryIngestTestPoint("pve-backup:pve1-vzdump-100", "pve1", now)
existing.SubjectResourceID = "resource:vm-100"
if err := store.UpsertPoints(context.Background(), []recovery.RecoveryPoint{existing}); err != nil {
t.Fatalf("UpsertPoints(existing): %v", err)
}
observation, err := recovery.NewProtectionProviderObservation(
recovery.ProviderProxmoxPVE,
"pve-backup-enumeration",
"pve1",
recovery.OutcomeSuccess,
recovery.ProtectionHistoryComplete,
operationaltrust.EvidencePermissionsSufficient,
false,
now,
now,
nil,
)
if err != nil {
t.Fatalf("NewProtectionProviderObservation() error = %v", err)
}
invalidPoint := existing
invalidPoint.ID = ""
m.ingestRecoveryPointsBestEffort(context.Background(), recoveryIngestBatch{
points: []recovery.RecoveryPoint{invalidPoint},
observations: []recovery.ProtectionProviderObservation{observation},
})
postures, _, err := store.ListProtectionPostures(
context.Background(),
recovery.ProtectionPostureQuery{
SubjectResourceIDs: []string{"resource:vm-100"},
},
)
if err != nil {
t.Fatalf("ListProtectionPostures() error = %v", err)
}
if len(postures) != 1 ||
postures[0].State != recovery.ProtectionStateProtected {
t.Fatalf(
"postures = %#v, want provider observation to survive point failure",
postures,
)
}
}
internal/recovery/mapper/proxmox/mapper.go
+32
View File
@@ -461,6 +461,38 @@ func FromPBSBackups(backups []models.PBSBackup, candidatesByKey map[string][]Gue
return out
}
// FromPBSBackupsWithEvidence is the operational-trust adapter for direct PBS
// inventory. It preserves the supported RecoveryPoint payload while attaching
// explicit provider scope and typed evidence to every successfully enumerated
// backup. Collection-wide completeness and permission state are recorded
// separately as a ProtectionProviderObservation by the polling owner.
func FromPBSBackupsWithEvidence(
backups []models.PBSBackup,
candidatesByKey map[string][]GuestCandidate,
ingestedAt time.Time,
) ([]recovery.RecoveryPoint, error) {
points := FromPBSBackups(backups, candidatesByKey)
if len(points) == 0 {
return []recovery.RecoveryPoint{}, nil
}
if ingestedAt.IsZero() {
return nil, fmt.Errorf("PBS recovery evidence ingestion time is required")
}
for i := range points {
points[i].ProviderScope = recovery.ProviderScopeForPoint(points[i])
evidence, err := recovery.NewRecoveryPointEvidence(
points[i],
"pbs-backup-inventory",
ingestedAt,
)
if err != nil {
return nil, fmt.Errorf("map PBS backup %q evidence: %w", points[i].ID, err)
}
points[i].Evidence = evidence
}
return points, nil
}
func sizePtr(v int64) *int64 {
if v <= 0 {
return nil
internal/recovery/mapper/proxmox/mapper_test.go
+60
View File
@@ -403,3 +403,63 @@ func TestFromPBSBackups_DisambiguatesCandidatesByNamespace(t *testing.T) {
t.Fatalf("SubjectRef = %#v, want linked debian-go guest", result[0].SubjectRef)
}
}
func TestFromPBSBackupsWithEvidenceAddsProviderScopeAndCorrelation(t *testing.T) {
t.Parallel()
backupTime := time.Date(2026, 7, 19, 6, 0, 0, 0, time.UTC)
ingestedAt := backupTime.Add(2 * time.Minute)
backups := []models.PBSBackup{
{
ID: "pbs-main:store-a:vm/100/2026-07-19T06:00:00Z",
VMID: "100",
Instance: "pbs-main",
Datastore: "store-a",
BackupType: "vm",
BackupTime: backupTime,
Verified: true,
},
}
candidates := map[string][]GuestCandidate{
"vm:100": {
{
SourceID: "vm-100",
ResourceType: unifiedresources.ResourceTypeVM,
DisplayName: "database",
InstanceName: "pve-main",
NodeName: "pve-a",
VMID: 100,
},
},
}
points, err := FromPBSBackupsWithEvidence(backups, candidates, ingestedAt)
if err != nil {
t.Fatalf("FromPBSBackupsWithEvidence() error = %v", err)
}
if len(points) != 1 {
t.Fatalf("points = %d, want 1", len(points))
}
point := points[0]
if point.ProviderScope != "pbs-main" {
t.Fatalf("provider scope = %q, want pbs-main", point.ProviderScope)
}
if point.Evidence == nil {
t.Fatal("expected typed PBS evidence")
}
if point.Evidence.ObservedAt != backupTime {
t.Fatalf("observedAt = %v, want %v", point.Evidence.ObservedAt, backupTime)
}
if point.Evidence.Correlation == nil {
t.Fatal("expected auditable canonical guest correlation")
}
if point.Evidence.Correlation.CandidateCount != 1 {
t.Fatalf(
"candidate count = %d, want 1",
point.Evidence.Correlation.CandidateCount,
)
}
if err := point.Evidence.Validate(); err != nil {
t.Fatalf("evidence Validate() error = %v", err)
}
}
internal/recovery/model/posture.go
+441
View File
@@ -0,0 +1,441 @@
package model
import (
"errors"
"fmt"
"sort"
"strings"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
)
type ProtectionState string
const (
ProtectionStateProtected ProtectionState = "protected"
ProtectionStateAttention ProtectionState = "attention"
ProtectionStateUnprotected ProtectionState = "unprotected"
ProtectionStateUnknown ProtectionState = "unknown"
)
func (state ProtectionState) Valid() bool {
switch state {
case ProtectionStateProtected,
ProtectionStateAttention,
ProtectionStateUnprotected,
ProtectionStateUnknown:
return true
default:
return false
}
}
type ProtectionFreshness string
const (
ProtectionFreshnessCurrent ProtectionFreshness = "current"
ProtectionFreshnessStale ProtectionFreshness = "stale"
ProtectionFreshnessUnknown ProtectionFreshness = "unknown"
)
func (freshness ProtectionFreshness) Valid() bool {
switch freshness {
case ProtectionFreshnessCurrent,
ProtectionFreshnessStale,
ProtectionFreshnessUnknown:
return true
default:
return false
}
}
type ProtectionVerification string
const (
ProtectionVerificationVerified ProtectionVerification = "verified"
ProtectionVerificationUnverified ProtectionVerification = "unverified"
ProtectionVerificationStale ProtectionVerification = "stale"
ProtectionVerificationUnknown ProtectionVerification = "unknown"
)
func (verification ProtectionVerification) Valid() bool {
switch verification {
case ProtectionVerificationVerified,
ProtectionVerificationUnverified,
ProtectionVerificationStale,
ProtectionVerificationUnknown:
return true
default:
return false
}
}
type ProtectionCoverage string
const (
ProtectionCoverageComplete ProtectionCoverage = "complete"
ProtectionCoveragePartial ProtectionCoverage = "partial"
ProtectionCoverageNone ProtectionCoverage = "none"
ProtectionCoverageUnknown ProtectionCoverage = "unknown"
)
func (coverage ProtectionCoverage) Valid() bool {
switch coverage {
case ProtectionCoverageComplete,
ProtectionCoveragePartial,
ProtectionCoverageNone,
ProtectionCoverageUnknown:
return true
default:
return false
}
}
type ProtectionHistoryCompleteness string
const (
ProtectionHistoryComplete ProtectionHistoryCompleteness = "complete"
ProtectionHistoryPartial ProtectionHistoryCompleteness = "partial"
ProtectionHistoryUnavailable ProtectionHistoryCompleteness = "unavailable"
ProtectionHistoryUnknown ProtectionHistoryCompleteness = "unknown"
)
func (completeness ProtectionHistoryCompleteness) Valid() bool {
switch completeness {
case ProtectionHistoryComplete,
ProtectionHistoryPartial,
ProtectionHistoryUnavailable,
ProtectionHistoryUnknown:
return true
default:
return false
}
}
type ProtectionProviderState struct {
Provider Provider `json:"provider"`
Source string `json:"source"`
Scope string `json:"scope"`
JobState Outcome `json:"jobState"`
HistoryCompleteness ProtectionHistoryCompleteness `json:"historyCompleteness"`
Permissions operationaltrust.EvidencePermissions `json:"permissions"`
LastAttemptAt *time.Time `json:"lastAttemptAt,omitempty"`
LastSuccessAt *time.Time `json:"lastSuccessAt,omitempty"`
LastVerifiedAt *time.Time `json:"lastVerifiedAt,omitempty"`
EvidenceIDs []string `json:"evidenceIds"`
VerificationExpected bool `json:"verificationExpected,omitempty"`
}
func (state ProtectionProviderState) Clone() ProtectionProviderState {
clone := state
clone.LastAttemptAt = cloneTime(state.LastAttemptAt)
clone.LastSuccessAt = cloneTime(state.LastSuccessAt)
clone.LastVerifiedAt = cloneTime(state.LastVerifiedAt)
clone.EvidenceIDs = append([]string(nil), state.EvidenceIDs...)
return clone
}
func (state ProtectionProviderState) Validate() error {
if strings.TrimSpace(string(state.Provider)) == "" {
return errors.New("protection provider is required")
}
if strings.TrimSpace(state.Source) == "" {
return errors.New("protection provider source is required")
}
if strings.TrimSpace(state.Scope) == "" {
return errors.New("protection provider scope is required")
}
if !validOutcome(state.JobState) {
return fmt.Errorf("protection provider job state %q is invalid", state.JobState)
}
if !state.HistoryCompleteness.Valid() {
return fmt.Errorf(
"protection provider history completeness %q is invalid",
state.HistoryCompleteness,
)
}
if !validEvidencePermissions(state.Permissions) {
return fmt.Errorf("protection provider permissions %q are invalid", state.Permissions)
}
if !sortedUniqueStrings(state.EvidenceIDs) {
return errors.New("protection provider evidence ids must be sorted and unique")
}
return nil
}
type ProtectionPosture struct {
SubjectResourceID string `json:"subjectResourceId"`
State ProtectionState `json:"state"`
LastAttemptAt *time.Time `json:"lastAttemptAt,omitempty"`
LastSuccessfulPointAt *time.Time `json:"lastSuccessfulPointAt,omitempty"`
LastVerifiedAt *time.Time `json:"lastVerifiedAt,omitempty"`
Freshness ProtectionFreshness `json:"freshness"`
Verification ProtectionVerification `json:"verification"`
Coverage ProtectionCoverage `json:"coverage"`
ProviderStates []ProtectionProviderState `json:"providerStates"`
RepositoryResourceIDs []string `json:"repositoryResourceIds"`
EvidenceIDs []string `json:"evidenceIds"`
Explanation string `json:"explanation"`
EvaluatedAt time.Time `json:"evaluatedAt"`
}
func (posture ProtectionPosture) Clone() ProtectionPosture {
clone := posture
clone.LastAttemptAt = cloneTime(posture.LastAttemptAt)
clone.LastSuccessfulPointAt = cloneTime(posture.LastSuccessfulPointAt)
clone.LastVerifiedAt = cloneTime(posture.LastVerifiedAt)
clone.ProviderStates = make([]ProtectionProviderState, len(posture.ProviderStates))
for i := range posture.ProviderStates {
clone.ProviderStates[i] = posture.ProviderStates[i].Clone()
}
clone.RepositoryResourceIDs = append([]string(nil), posture.RepositoryResourceIDs...)
clone.EvidenceIDs = append([]string(nil), posture.EvidenceIDs...)
return clone
}
func (posture ProtectionPosture) Validate() error {
if strings.TrimSpace(posture.SubjectResourceID) == "" {
return errors.New("protection posture subject resource id is required")
}
if !posture.State.Valid() {
return fmt.Errorf("protection posture state %q is invalid", posture.State)
}
if !posture.Freshness.Valid() {
return fmt.Errorf("protection posture freshness %q is invalid", posture.Freshness)
}
if !posture.Verification.Valid() {
return fmt.Errorf("protection posture verification %q is invalid", posture.Verification)
}
if !posture.Coverage.Valid() {
return fmt.Errorf("protection posture coverage %q is invalid", posture.Coverage)
}
if posture.EvaluatedAt.IsZero() {
return errors.New("protection posture evaluation time is required")
}
if strings.TrimSpace(posture.Explanation) == "" {
return errors.New("protection posture explanation is required")
}
if !sortedUniqueStrings(posture.RepositoryResourceIDs) {
return errors.New("protection repository resource ids must be sorted and unique")
}
if !sortedUniqueStrings(posture.EvidenceIDs) {
return errors.New("protection evidence ids must be sorted and unique")
}
for i := range posture.ProviderStates {
if err := posture.ProviderStates[i].Validate(); err != nil {
return fmt.Errorf("protection provider state %d: %w", i, err)
}
if i > 0 && compareProviderStates(
posture.ProviderStates[i-1],
posture.ProviderStates[i],
) >= 0 {
return errors.New("protection provider states must be sorted and unique")
}
}
return nil
}
type ProtectionProviderObservation struct {
ID string `json:"id"`
Provider Provider `json:"provider"`
Source string `json:"source"`
Scope string `json:"scope"`
JobState Outcome `json:"jobState"`
HistoryCompleteness ProtectionHistoryCompleteness `json:"historyCompleteness"`
Permissions operationaltrust.EvidencePermissions `json:"permissions"`
VerificationExpected bool `json:"verificationExpected,omitempty"`
ObservedAt time.Time `json:"observedAt"`
IngestedAt time.Time `json:"ingestedAt"`
Evidence operationaltrust.EvidenceEnvelope `json:"evidence"`
}
func (observation ProtectionProviderObservation) Clone() ProtectionProviderObservation {
clone := observation
clone.Evidence = observation.Evidence.Clone()
return clone
}
func (observation ProtectionProviderObservation) Validate() error {
if strings.TrimSpace(observation.ID) == "" {
return errors.New("protection provider observation id is required")
}
if strings.TrimSpace(string(observation.Provider)) == "" {
return errors.New("protection provider observation provider is required")
}
if strings.TrimSpace(observation.Source) == "" {
return errors.New("protection provider observation source is required")
}
if strings.TrimSpace(observation.Scope) == "" {
return errors.New("protection provider observation scope is required")
}
if !validOutcome(observation.JobState) {
return fmt.Errorf(
"protection provider observation job state %q is invalid",
observation.JobState,
)
}
if !observation.HistoryCompleteness.Valid() {
return fmt.Errorf(
"protection provider observation history completeness %q is invalid",
observation.HistoryCompleteness,
)
}
if !validEvidencePermissions(observation.Permissions) {
return fmt.Errorf(
"protection provider observation permissions %q are invalid",
observation.Permissions,
)
}
if observation.ObservedAt.IsZero() || observation.IngestedAt.IsZero() {
return errors.New("protection provider observation times are required")
}
if !observation.Evidence.ObservedAt.Equal(observation.ObservedAt) {
return errors.New("provider observation and evidence observation times must match")
}
if !observation.Evidence.IngestedAt.Equal(observation.IngestedAt) {
return errors.New("provider observation and evidence ingestion times must match")
}
if observation.Evidence.ID != observation.ID {
return errors.New("provider observation id must match its evidence id")
}
if err := observation.Evidence.Validate(); err != nil {
return fmt.Errorf("provider observation evidence: %w", err)
}
return nil
}
type ProtectionPosturePolicy struct {
FreshnessWindow time.Duration `json:"-"`
VerificationWindow time.Duration `json:"-"`
RequireVerification bool `json:"requireVerification"`
}
func (policy ProtectionPosturePolicy) Validate() error {
if policy.FreshnessWindow <= 0 {
return errors.New("protection freshness window must be positive")
}
if policy.VerificationWindow <= 0 {
return errors.New("protection verification window must be positive")
}
return nil
}
type ProtectionPosturePolicyPayload struct {
FreshnessWindowSeconds int64 `json:"freshnessWindowSeconds"`
VerificationWindowSeconds int64 `json:"verificationWindowSeconds"`
RequireVerification bool `json:"requireVerification"`
}
func (policy ProtectionPosturePolicy) Payload() ProtectionPosturePolicyPayload {
return ProtectionPosturePolicyPayload{
FreshnessWindowSeconds: int64(policy.FreshnessWindow / time.Second),
VerificationWindowSeconds: int64(policy.VerificationWindow / time.Second),
RequireVerification: policy.RequireVerification,
}
}
type ProtectionPostureQuery struct {
SubjectResourceIDs []string
State ProtectionState
Page int
Limit int
}
type ProtectionProviderSummary struct {
Provider Provider
Source string
Scope string
JobState Outcome
HistoryCompleteness ProtectionHistoryCompleteness
Permissions operationaltrust.EvidencePermissions
VerificationExpected bool
LastAttemptAt *time.Time
LastSuccessAt *time.Time
LastVerifiedAt *time.Time
BackupPointCount int
SnapshotPointCount int
RepositoryResourceIDs []string
EvidenceIDs []string
}
func (summary ProtectionProviderSummary) normalize() ProtectionProviderSummary {
summary.Provider = Provider(strings.TrimSpace(string(summary.Provider)))
summary.Source = strings.TrimSpace(summary.Source)
summary.Scope = strings.TrimSpace(summary.Scope)
summary.RepositoryResourceIDs = normalizeSortedStrings(summary.RepositoryResourceIDs)
summary.EvidenceIDs = normalizeSortedStrings(summary.EvidenceIDs)
if !validOutcome(summary.JobState) {
summary.JobState = OutcomeUnknown
}
if !summary.HistoryCompleteness.Valid() {
summary.HistoryCompleteness = ProtectionHistoryUnknown
}
if !validEvidencePermissions(summary.Permissions) {
summary.Permissions = operationaltrust.EvidencePermissionsUnknown
}
return summary
}
func cloneTime(value *time.Time) *time.Time {
if value == nil {
return nil
}
cloned := value.UTC()
return &cloned
}
func validOutcome(outcome Outcome) bool {
switch outcome {
case OutcomeSuccess, OutcomeWarning, OutcomeFailed, OutcomeRunning, OutcomeUnknown:
return true
default:
return false
}
}
func validEvidencePermissions(value operationaltrust.EvidencePermissions) bool {
switch value {
case operationaltrust.EvidencePermissionsSufficient,
operationaltrust.EvidencePermissionsPartial,
operationaltrust.EvidencePermissionsDenied,
operationaltrust.EvidencePermissionsUnknown:
return true
default:
return false
}
}
func normalizeSortedStrings(values []string) []string {
unique := make(map[string]struct{}, len(values))
for _, value := range values {
value = strings.TrimSpace(value)
if value != "" {
unique[value] = struct{}{}
}
}
out := make([]string, 0, len(unique))
for value := range unique {
out = append(out, value)
}
sort.Strings(out)
return out
}
func sortedUniqueStrings(values []string) bool {
for i, value := range values {
if strings.TrimSpace(value) == "" {
return false
}
if i > 0 && values[i-1] >= value {
return false
}
}
return true
}
func compareProviderStates(a, b ProtectionProviderState) int {
aKey := strings.Join([]string{string(a.Provider), a.Scope, a.Source}, "\x00")
bKey := strings.Join([]string{string(b.Provider), b.Scope, b.Source}, "\x00")
return strings.Compare(aKey, bKey)
}
internal/recovery/model/types.go
+13 -1
View File
@@ -1,6 +1,10 @@
package model
import "time"
import (
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
)
// Provider identifies the system that produced a recovery point.
// This is intentionally a string (not an enum) to keep forward-compatibility with new platforms.
@@ -83,6 +87,14 @@ type RecoveryPoint struct {
// Provider-specific details for drill-down (kept small).
Details map[string]any `json:"details,omitempty"`
// ProviderScope is the stable provider-local collection scope used to
// correlate this point with provider history and permission observations.
ProviderScope string `json:"providerScope,omitempty"`
// Evidence carries the typed provenance for this recovery point. Older
// persisted rows may omit it and are treated as unknown-quality evidence.
Evidence *operationaltrust.EvidenceEnvelope `json:"evidence,omitempty"`
// Display contains normalized, provider-agnostic fields intended for UIs.
// These fields are derived at ingest time and/or backfilled in the store.
Display *RecoveryPointDisplay `json:"display,omitempty"`
internal/recovery/posture.go
+673
View File
@@ -0,0 +1,673 @@
package recovery
import (
"fmt"
"sort"
"strings"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
)
const (
DefaultProtectionFreshnessWindow = 7 * 24 * time.Hour
DefaultProtectionVerificationWindow = 7 * 24 * time.Hour
)
var DefaultProtectionPosturePolicy = ProtectionPosturePolicy{
FreshnessWindow: DefaultProtectionFreshnessWindow,
VerificationWindow: DefaultProtectionVerificationWindow,
RequireVerification: false,
}
func ProviderScopeForPoint(point RecoveryPoint) string {
if scope := strings.TrimSpace(point.ProviderScope); scope != "" {
return scope
}
switch point.Provider {
case ProviderProxmoxPBS:
if point.RepositoryRef != nil {
if scope := strings.TrimSpace(point.RepositoryRef.Namespace); scope != "" {
return scope
}
}
case ProviderProxmoxPVE:
if scope := recoveryDetailString(point, "instance"); scope != "" {
return scope
}
case ProviderKubernetes:
if scope := recoveryDetailString(point, "k8sClusterId"); scope != "" {
return scope
}
case ProviderTrueNAS:
if scope := recoveryDetailString(point, "connectionId"); scope != "" {
return scope
}
}
if point.SubjectRef != nil {
if scope := strings.TrimSpace(point.SubjectRef.Namespace); scope != "" {
return scope
}
}
return "provider-default"
}
func NewRecoveryPointEvidence(
point RecoveryPoint,
collector string,
ingestedAt time.Time,
) (*operationaltrust.EvidenceEnvelope, error) {
collector = strings.TrimSpace(collector)
if collector == "" {
return nil, fmt.Errorf("recovery evidence collector is required")
}
observedAt := recoveryPointObservedAt(point)
if observedAt.IsZero() {
return nil, fmt.Errorf("recovery point observation time is required")
}
if ingestedAt.IsZero() {
return nil, fmt.Errorf("recovery point ingestion time is required")
}
scope := ProviderScopeForPoint(point)
source := operationaltrust.EvidenceSource{
Provider: string(point.Provider),
Collector: collector,
Instance: scope,
}
subject := operationaltrust.EvidenceSubject{}
if resourceID := strings.TrimSpace(point.SubjectResourceID); resourceID != "" {
subject.ResourceID = resourceID
} else {
subject.ProviderRef = SubjectKeyForPoint(point)
subject.ProviderScope = scope
}
evidenceID, err := operationaltrust.NewEvidenceID(
source,
subject,
observedAt,
point.ID,
)
if err != nil {
return nil, err
}
envelope := &operationaltrust.EvidenceEnvelope{
ID: evidenceID,
Source: source,
Subject: subject,
ObservedAt: observedAt,
IngestedAt: ingestedAt.UTC(),
Completeness: operationaltrust.EvidenceComplete,
Confidence: operationaltrust.EvidenceConfirmed,
Permissions: operationaltrust.EvidencePermissionsSufficient,
PayloadRef: &operationaltrust.EvidencePayloadRef{
Kind: "recovery-point",
ID: strings.TrimSpace(point.ID),
},
}
if subject.ResourceID != "" && point.Provider == ProviderProxmoxPBS {
index := DeriveIndex(point)
matched := map[string]string{
"providerScope": scope,
}
if value := strings.TrimSpace(index.ItemType); value != "" {
matched["itemType"] = value
}
if value := strings.TrimSpace(index.EntityIDLabel); value != "" {
matched["entityId"] = value
}
envelope.Confidence = operationaltrust.EvidenceInferred
envelope.Reason = &operationaltrust.EvidenceReason{
Code: "provider_identity_correlation",
Message: "The PBS subject was linked through a unique provider-scoped guest match.",
}
envelope.Correlation = &operationaltrust.IdentityCorrelation{
Rule: "provider_scoped_guest_identity",
MatchedFields: matched,
CandidateCount: 1,
}
}
if err := envelope.Validate(); err != nil {
return nil, err
}
return envelope, nil
}
func NewProtectionProviderObservation(
provider Provider,
source string,
scope string,
jobState Outcome,
historyCompleteness ProtectionHistoryCompleteness,
permissions operationaltrust.EvidencePermissions,
verificationExpected bool,
observedAt time.Time,
ingestedAt time.Time,
reason *operationaltrust.EvidenceReason,
) (ProtectionProviderObservation, error) {
source = strings.TrimSpace(source)
scope = strings.TrimSpace(scope)
evidenceSource := operationaltrust.EvidenceSource{
Provider: strings.TrimSpace(string(provider)),
Collector: source,
Instance: scope,
}
evidenceSubject := operationaltrust.EvidenceSubject{
ProviderRef: strings.TrimSpace(string(provider)) + ":" + scope,
ProviderScope: scope,
}
evidenceID, err := operationaltrust.NewEvidenceID(
evidenceSource,
evidenceSubject,
observedAt,
fmt.Sprintf(
"%s:%s:%s:%s",
jobState,
historyCompleteness,
permissions,
scope,
),
)
if err != nil {
return ProtectionProviderObservation{}, err
}
completeness := operationaltrust.EvidenceUnavailable
switch historyCompleteness {
case ProtectionHistoryComplete:
completeness = operationaltrust.EvidenceComplete
case ProtectionHistoryPartial:
completeness = operationaltrust.EvidencePartial
}
confidence := operationaltrust.EvidenceUnknown
if historyCompleteness == ProtectionHistoryComplete &&
permissions == operationaltrust.EvidencePermissionsSufficient {
confidence = operationaltrust.EvidenceConfirmed
}
if (completeness != operationaltrust.EvidenceComplete ||
confidence != operationaltrust.EvidenceConfirmed ||
permissions != operationaltrust.EvidencePermissionsSufficient) &&
reason == nil {
reason = &operationaltrust.EvidenceReason{
Code: "provider_history_limited",
Message: "Provider history could not support a complete protection assertion.",
}
}
observation := ProtectionProviderObservation{
ID: evidenceID,
Provider: provider,
Source: source,
Scope: scope,
JobState: jobState,
HistoryCompleteness: historyCompleteness,
Permissions: permissions,
VerificationExpected: verificationExpected,
ObservedAt: observedAt.UTC(),
IngestedAt: ingestedAt.UTC(),
Evidence: operationaltrust.EvidenceEnvelope{
ID: evidenceID,
Source: evidenceSource,
Subject: evidenceSubject,
ObservedAt: observedAt.UTC(),
IngestedAt: ingestedAt.UTC(),
Completeness: completeness,
Confidence: confidence,
Reason: reason,
Permissions: permissions,
PayloadRef: &operationaltrust.EvidencePayloadRef{
Kind: "protection-provider-observation",
ID: evidenceID,
},
},
}
if err := observation.Validate(); err != nil {
return ProtectionProviderObservation{}, err
}
return observation, nil
}
func BuildProtectionPostureFromPointsAt(
subjectResourceID string,
points []RecoveryPoint,
observations []ProtectionProviderObservation,
policy ProtectionPosturePolicy,
now time.Time,
) ProtectionPosture {
subjectResourceID = strings.TrimSpace(subjectResourceID)
type aggregate struct {
summary ProtectionProviderSummary
lastEventAt time.Time
}
byProviderScope := make(map[string]*aggregate)
latestObservations := make(map[string]ProtectionProviderObservation)
for _, observation := range observations {
if err := observation.Validate(); err != nil {
continue
}
key := providerScopeKey(observation.Provider, observation.Scope)
current, exists := latestObservations[key]
if !exists ||
observation.ObservedAt.After(current.ObservedAt) ||
(observation.ObservedAt.Equal(current.ObservedAt) && observation.ID > current.ID) {
latestObservations[key] = observation.Clone()
}
}
for _, point := range points {
if strings.TrimSpace(point.SubjectResourceID) != subjectResourceID {
continue
}
scope := ProviderScopeForPoint(point)
key := providerScopeKey(point.Provider, scope)
agg := byProviderScope[key]
if agg == nil {
agg = &aggregate{
summary: ProtectionProviderSummary{
Provider: point.Provider,
Source: "legacy-recovery-point",
Scope: scope,
JobState: OutcomeUnknown,
HistoryCompleteness: ProtectionHistoryUnknown,
Permissions: operationaltrust.EvidencePermissionsUnknown,
},
}
byProviderScope[key] = agg
}
eventAt := recoveryPointObservedAt(point)
if !eventAt.IsZero() {
if agg.summary.LastAttemptAt == nil || eventAt.After(*agg.summary.LastAttemptAt) {
value := eventAt.UTC()
agg.summary.LastAttemptAt = &value
}
if agg.lastEventAt.IsZero() || eventAt.After(agg.lastEventAt) {
agg.lastEventAt = eventAt
if validRecoveryOutcome(point.Outcome) {
agg.summary.JobState = point.Outcome
} else {
agg.summary.JobState = OutcomeUnknown
}
}
}
switch point.Kind {
case KindBackup:
agg.summary.BackupPointCount++
if point.Outcome == OutcomeSuccess && !eventAt.IsZero() {
if agg.summary.LastSuccessAt == nil || eventAt.After(*agg.summary.LastSuccessAt) {
value := eventAt.UTC()
agg.summary.LastSuccessAt = &value
}
}
if point.Outcome == OutcomeSuccess &&
point.Verified != nil &&
*point.Verified &&
!eventAt.IsZero() {
if agg.summary.LastVerifiedAt == nil || eventAt.After(*agg.summary.LastVerifiedAt) {
value := eventAt.UTC()
agg.summary.LastVerifiedAt = &value
}
}
case KindSnapshot:
agg.summary.SnapshotPointCount++
}
if value := strings.TrimSpace(point.RepositoryResourceID); value != "" {
agg.summary.RepositoryResourceIDs = append(
agg.summary.RepositoryResourceIDs,
value,
)
}
if point.Evidence != nil {
agg.summary.EvidenceIDs = append(agg.summary.EvidenceIDs, point.Evidence.ID)
if source := strings.TrimSpace(point.Evidence.Source.Collector); source != "" {
agg.summary.Source = source
}
}
}
for key, agg := range byProviderScope {
observation, ok := latestObservations[key]
if !ok {
continue
}
agg.summary.Source = observation.Source
agg.summary.HistoryCompleteness = observation.HistoryCompleteness
agg.summary.Permissions = observation.Permissions
agg.summary.VerificationExpected = observation.VerificationExpected
agg.summary.EvidenceIDs = append(agg.summary.EvidenceIDs, observation.Evidence.ID)
if !observation.ObservedAt.Before(agg.lastEventAt) &&
observation.JobState != OutcomeUnknown {
agg.summary.JobState = observation.JobState
}
}
summaries := make([]ProtectionProviderSummary, 0, len(byProviderScope))
for _, agg := range byProviderScope {
agg.summary.RepositoryResourceIDs = sortedUnique(agg.summary.RepositoryResourceIDs)
agg.summary.EvidenceIDs = sortedUnique(agg.summary.EvidenceIDs)
summaries = append(summaries, agg.summary)
}
return DeriveProtectionPostureAt(subjectResourceID, summaries, policy, now)
}
func DeriveProtectionPostureAt(
subjectResourceID string,
summaries []ProtectionProviderSummary,
policy ProtectionPosturePolicy,
now time.Time,
) ProtectionPosture {
subjectResourceID = strings.TrimSpace(subjectResourceID)
if err := policy.Validate(); err != nil {
policy = DefaultProtectionPosturePolicy
}
if now.IsZero() {
now = time.Now().UTC()
} else {
now = now.UTC()
}
posture := ProtectionPosture{
SubjectResourceID: subjectResourceID,
State: ProtectionStateUnknown,
Freshness: ProtectionFreshnessUnknown,
Verification: ProtectionVerificationUnknown,
Coverage: ProtectionCoverageUnknown,
ProviderStates: []ProtectionProviderState{},
RepositoryResourceIDs: []string{},
EvidenceIDs: []string{},
Explanation: "Pulse has no complete provider history linked to this resource, so protection is unknown.",
EvaluatedAt: now,
}
hasUnknownBlock := false
hasPartial := false
hasCompleteHistory := false
hasQualifyingSuccess := false
hasSupportedQualifyingSuccess := false
hasInvalidatingFailure := false
hasSnapshotsOnly := false
verificationExpected := policy.RequireVerification
supportedVerificationExpected := policy.RequireVerification
var supportedLastSuccessAt *time.Time
var supportedLastVerifiedAt *time.Time
for _, summary := range summaries {
if strings.TrimSpace(string(summary.Provider)) == "" ||
strings.TrimSpace(summary.Scope) == "" {
continue
}
if !validRecoveryOutcome(summary.JobState) {
summary.JobState = OutcomeUnknown
}
if !summary.HistoryCompleteness.Valid() {
summary.HistoryCompleteness = ProtectionHistoryUnknown
}
if !validProtectionPermissions(summary.Permissions) {
summary.Permissions = operationaltrust.EvidencePermissionsUnknown
}
summary.Source = strings.TrimSpace(summary.Source)
if summary.Source == "" {
summary.Source = "legacy-recovery-point"
}
summary.RepositoryResourceIDs = sortedUnique(summary.RepositoryResourceIDs)
summary.EvidenceIDs = sortedUnique(summary.EvidenceIDs)
state := ProtectionProviderState{
Provider: summary.Provider,
Source: summary.Source,
Scope: strings.TrimSpace(summary.Scope),
JobState: summary.JobState,
HistoryCompleteness: summary.HistoryCompleteness,
Permissions: summary.Permissions,
LastAttemptAt: cloneProtectionTime(summary.LastAttemptAt),
LastSuccessAt: cloneProtectionTime(summary.LastSuccessAt),
LastVerifiedAt: cloneProtectionTime(summary.LastVerifiedAt),
EvidenceIDs: append([]string(nil), summary.EvidenceIDs...),
VerificationExpected: summary.VerificationExpected,
}
posture.ProviderStates = append(posture.ProviderStates, state)
posture.RepositoryResourceIDs = append(
posture.RepositoryResourceIDs,
summary.RepositoryResourceIDs...,
)
posture.EvidenceIDs = append(posture.EvidenceIDs, summary.EvidenceIDs...)
posture.LastAttemptAt = latestProtectionTime(posture.LastAttemptAt, summary.LastAttemptAt)
posture.LastSuccessfulPointAt = latestProtectionTime(
posture.LastSuccessfulPointAt,
summary.LastSuccessAt,
)
posture.LastVerifiedAt = latestProtectionTime(posture.LastVerifiedAt, summary.LastVerifiedAt)
switch summary.HistoryCompleteness {
case ProtectionHistoryComplete:
hasCompleteHistory = true
case ProtectionHistoryPartial:
hasPartial = true
case ProtectionHistoryUnavailable, ProtectionHistoryUnknown:
hasUnknownBlock = true
}
switch summary.Permissions {
case operationaltrust.EvidencePermissionsSufficient:
case operationaltrust.EvidencePermissionsPartial:
hasPartial = true
case operationaltrust.EvidencePermissionsDenied,
operationaltrust.EvidencePermissionsUnknown:
hasUnknownBlock = true
}
if summary.LastSuccessAt != nil {
hasQualifyingSuccess = true
}
if summary.LastSuccessAt != nil &&
summary.HistoryCompleteness == ProtectionHistoryComplete &&
summary.Permissions == operationaltrust.EvidencePermissionsSufficient {
hasSupportedQualifyingSuccess = true
supportedLastSuccessAt = latestProtectionTime(
supportedLastSuccessAt,
summary.LastSuccessAt,
)
supportedLastVerifiedAt = latestProtectionTime(
supportedLastVerifiedAt,
summary.LastVerifiedAt,
)
if summary.VerificationExpected {
supportedVerificationExpected = true
}
}
if summary.SnapshotPointCount > 0 && summary.BackupPointCount == 0 {
hasSnapshotsOnly = true
}
if summary.VerificationExpected {
verificationExpected = true
}
if summary.JobState == OutcomeFailed &&
summary.LastAttemptAt != nil &&
(summary.LastSuccessAt == nil ||
!summary.LastAttemptAt.Before(*summary.LastSuccessAt)) {
hasInvalidatingFailure = true
}
}
sort.Slice(posture.ProviderStates, func(i, j int) bool {
return providerStateSortKey(posture.ProviderStates[i]) <
providerStateSortKey(posture.ProviderStates[j])
})
posture.RepositoryResourceIDs = sortedUnique(posture.RepositoryResourceIDs)
posture.EvidenceIDs = sortedUnique(posture.EvidenceIDs)
stateLastSuccessAt := posture.LastSuccessfulPointAt
stateLastVerifiedAt := posture.LastVerifiedAt
stateVerificationExpected := verificationExpected
if hasSupportedQualifyingSuccess {
stateLastSuccessAt = supportedLastSuccessAt
stateLastVerifiedAt = supportedLastVerifiedAt
stateVerificationExpected = supportedVerificationExpected
}
if stateLastSuccessAt != nil {
if now.Sub(*stateLastSuccessAt) <= policy.FreshnessWindow {
posture.Freshness = ProtectionFreshnessCurrent
} else {
posture.Freshness = ProtectionFreshnessStale
}
}
if stateLastSuccessAt != nil {
switch {
case stateLastVerifiedAt != nil &&
now.Sub(*stateLastVerifiedAt) <= policy.VerificationWindow:
posture.Verification = ProtectionVerificationVerified
case stateLastVerifiedAt != nil:
posture.Verification = ProtectionVerificationStale
case stateVerificationExpected:
posture.Verification = ProtectionVerificationUnverified
}
}
switch {
case len(posture.ProviderStates) == 0:
posture.Coverage = ProtectionCoverageUnknown
case hasUnknownBlock:
posture.Coverage = ProtectionCoverageUnknown
case hasPartial:
posture.Coverage = ProtectionCoveragePartial
case hasCompleteHistory && !hasQualifyingSuccess:
posture.Coverage = ProtectionCoverageNone
default:
posture.Coverage = ProtectionCoverageComplete
}
switch {
case len(posture.ProviderStates) == 0:
// Keep the initialized unknown explanation.
case hasSupportedQualifyingSuccess &&
posture.Freshness == ProtectionFreshnessCurrent &&
!hasInvalidatingFailure &&
!hasPartial &&
(!stateVerificationExpected || posture.Verification == ProtectionVerificationVerified):
posture.State = ProtectionStateProtected
if posture.Verification == ProtectionVerificationVerified {
posture.Explanation = "A current subject-linked backup is available and has recent verification evidence."
} else {
posture.Explanation = "A current subject-linked backup is available from complete provider history."
}
if hasUnknownBlock {
posture.Explanation += " Another linked provider has unavailable history, but it does not invalidate the confirmed recovery point."
}
case hasSupportedQualifyingSuccess:
posture.State = ProtectionStateAttention
switch {
case hasInvalidatingFailure:
posture.Explanation = "A backup exists, but a newer provider failure needs attention before Pulse can call this resource protected."
case posture.Freshness == ProtectionFreshnessStale:
posture.Explanation = "The strongest subject-linked backup is older than the configured freshness window."
case stateVerificationExpected && posture.Verification != ProtectionVerificationVerified:
posture.Explanation = "A current backup exists, but its verification evidence is missing or stale."
default:
posture.Explanation = "Recovery evidence exists, but provider history or permissions are incomplete."
}
case hasUnknownBlock:
posture.State = ProtectionStateUnknown
posture.Explanation = "Provider history or permissions are unavailable, so Pulse cannot make a stronger protection claim."
case hasQualifyingSuccess || hasPartial:
posture.State = ProtectionStateAttention
posture.Explanation = "Recovery evidence exists, but provider history or permissions are incomplete."
case hasCompleteHistory:
posture.State = ProtectionStateUnprotected
if hasSnapshotsOnly {
posture.Explanation = "Provider history is complete, but only snapshots are present; snapshots alone do not prove independent recovery."
} else {
posture.Explanation = "Provider history is complete, but no qualifying subject-linked backup exists."
}
default:
posture.State = ProtectionStateUnknown
}
return posture
}
func recoveryPointObservedAt(point RecoveryPoint) time.Time {
if point.CompletedAt != nil && !point.CompletedAt.IsZero() {
return point.CompletedAt.UTC()
}
if point.StartedAt != nil && !point.StartedAt.IsZero() {
return point.StartedAt.UTC()
}
return time.Time{}
}
func recoveryDetailString(point RecoveryPoint, key string) string {
if point.Details == nil {
return ""
}
value, _ := point.Details[key].(string)
return strings.TrimSpace(value)
}
func providerScopeKey(provider Provider, scope string) string {
return strings.TrimSpace(string(provider)) + "\x00" + strings.TrimSpace(scope)
}
func providerStateSortKey(state ProtectionProviderState) string {
return strings.Join([]string{
strings.TrimSpace(string(state.Provider)),
strings.TrimSpace(state.Scope),
strings.TrimSpace(state.Source),
}, "\x00")
}
func sortedUnique(values []string) []string {
set := make(map[string]struct{}, len(values))
for _, value := range values {
if value = strings.TrimSpace(value); value != "" {
set[value] = struct{}{}
}
}
out := make([]string, 0, len(set))
for value := range set {
out = append(out, value)
}
sort.Strings(out)
return out
}
func latestProtectionTime(current, candidate *time.Time) *time.Time {
if candidate == nil {
return cloneProtectionTime(current)
}
if current == nil || candidate.After(*current) {
return cloneProtectionTime(candidate)
}
return cloneProtectionTime(current)
}
func cloneProtectionTime(value *time.Time) *time.Time {
if value == nil {
return nil
}
clone := value.UTC()
return &clone
}
func validRecoveryOutcome(value Outcome) bool {
switch value {
case OutcomeSuccess, OutcomeWarning, OutcomeFailed, OutcomeRunning, OutcomeUnknown:
return true
default:
return false
}
}
func validProtectionPermissions(value operationaltrust.EvidencePermissions) bool {
switch value {
case operationaltrust.EvidencePermissionsSufficient,
operationaltrust.EvidencePermissionsPartial,
operationaltrust.EvidencePermissionsDenied,
operationaltrust.EvidencePermissionsUnknown:
return true
default:
return false
}
}
internal/recovery/posture_test.go
+321
View File
@@ -0,0 +1,321 @@
package recovery
import (
"strings"
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
)
func protectionTestSummary(
now time.Time,
lastSuccess *time.Time,
) ProtectionProviderSummary {
return ProtectionProviderSummary{
Provider: ProviderProxmoxPBS,
Source: "pbs-backup-enumeration",
Scope: "pbs-main",
JobState: OutcomeSuccess,
HistoryCompleteness: ProtectionHistoryComplete,
Permissions: operationaltrust.EvidencePermissionsSufficient,
VerificationExpected: true,
LastAttemptAt: cloneProtectionTime(lastSuccess),
LastSuccessAt: cloneProtectionTime(lastSuccess),
LastVerifiedAt: cloneProtectionTime(lastSuccess),
BackupPointCount: 1,
RepositoryResourceIDs: []string{
"repository:pbs-main/store-a",
},
EvidenceIDs: []string{"evidence:point-1", "evidence:provider-1"},
}
}
func TestDeriveProtectionPostureTruthTable(t *testing.T) {
t.Parallel()
now := time.Date(2026, 7, 19, 8, 0, 0, 0, time.UTC)
recent := now.Add(-2 * time.Hour)
stale := now.Add(-8 * 24 * time.Hour)
policy := DefaultProtectionPosturePolicy
tests := []struct {
name string
summaries []ProtectionProviderSummary
wantState ProtectionState
wantFresh ProtectionFreshness
wantVerify ProtectionVerification
wantCoverage ProtectionCoverage
wantText string
}{
{
name: "current verified PBS backup is protected",
summaries: []ProtectionProviderSummary{protectionTestSummary(now, &recent)},
wantState: ProtectionStateProtected,
wantFresh: ProtectionFreshnessCurrent,
wantVerify: ProtectionVerificationVerified,
wantCoverage: ProtectionCoverageComplete,
wantText: "current subject-linked backup",
},
{
name: "stale success needs attention",
summaries: []ProtectionProviderSummary{
protectionTestSummary(now, &stale),
},
wantState: ProtectionStateAttention,
wantFresh: ProtectionFreshnessStale,
wantVerify: ProtectionVerificationStale,
wantCoverage: ProtectionCoverageComplete,
wantText: "older than",
},
{
name: "missing expected verification needs attention",
summaries: func() []ProtectionProviderSummary {
summary := protectionTestSummary(now, &recent)
summary.LastVerifiedAt = nil
return []ProtectionProviderSummary{summary}
}(),
wantState: ProtectionStateAttention,
wantFresh: ProtectionFreshnessCurrent,
wantVerify: ProtectionVerificationUnverified,
wantCoverage: ProtectionCoverageComplete,
wantText: "verification evidence",
},
{
name: "newer provider failure invalidates protected claim",
summaries: func() []ProtectionProviderSummary {
summary := protectionTestSummary(now, &recent)
failedAt := now.Add(-time.Hour)
summary.JobState = OutcomeFailed
summary.LastAttemptAt = &failedAt
return []ProtectionProviderSummary{summary}
}(),
wantState: ProtectionStateAttention,
wantFresh: ProtectionFreshnessCurrent,
wantVerify: ProtectionVerificationVerified,
wantCoverage: ProtectionCoverageComplete,
wantText: "newer provider failure",
},
{
name: "complete snapshot-only history is unprotected",
summaries: []ProtectionProviderSummary{
{
Provider: ProviderProxmoxPVE,
Source: "pve-snapshot-enumeration",
Scope: "pve-main",
JobState: OutcomeSuccess,
HistoryCompleteness: ProtectionHistoryComplete,
Permissions: operationaltrust.EvidencePermissionsSufficient,
SnapshotPointCount: 3,
EvidenceIDs: []string{"evidence:snapshot"},
},
},
wantState: ProtectionStateUnprotected,
wantFresh: ProtectionFreshnessUnknown,
wantVerify: ProtectionVerificationUnknown,
wantCoverage: ProtectionCoverageNone,
wantText: "snapshots alone",
},
{
name: "partial provider history is attention when a backup exists",
summaries: func() []ProtectionProviderSummary {
summary := protectionTestSummary(now, &recent)
summary.HistoryCompleteness = ProtectionHistoryPartial
summary.Permissions = operationaltrust.EvidencePermissionsPartial
return []ProtectionProviderSummary{summary}
}(),
wantState: ProtectionStateAttention,
wantFresh: ProtectionFreshnessCurrent,
wantVerify: ProtectionVerificationVerified,
wantCoverage: ProtectionCoveragePartial,
wantText: "incomplete",
},
{
name: "permission denied is unknown",
summaries: func() []ProtectionProviderSummary {
summary := protectionTestSummary(now, &recent)
summary.HistoryCompleteness = ProtectionHistoryUnavailable
summary.Permissions = operationaltrust.EvidencePermissionsDenied
return []ProtectionProviderSummary{summary}
}(),
wantState: ProtectionStateUnknown,
wantFresh: ProtectionFreshnessCurrent,
wantVerify: ProtectionVerificationVerified,
wantCoverage: ProtectionCoverageUnknown,
wantText: "permissions are unavailable",
},
{
name: "confirmed PBS recovery is not invalidated by an unknown legacy provider",
summaries: func() []ProtectionProviderSummary {
confirmed := protectionTestSummary(now, &recent)
legacy := ProtectionProviderSummary{
Provider: ProviderProxmoxPVE,
Source: "legacy-recovery-point",
Scope: "pve-main",
JobState: OutcomeSuccess,
HistoryCompleteness: ProtectionHistoryUnknown,
Permissions: operationaltrust.EvidencePermissionsUnknown,
LastAttemptAt: &recent,
LastSuccessAt: &recent,
BackupPointCount: 1,
EvidenceIDs: []string{"evidence:legacy-pve"},
}
return []ProtectionProviderSummary{confirmed, legacy}
}(),
wantState: ProtectionStateProtected,
wantFresh: ProtectionFreshnessCurrent,
wantVerify: ProtectionVerificationVerified,
wantCoverage: ProtectionCoverageUnknown,
wantText: "does not invalidate",
},
{
name: "no provider evidence is unknown",
summaries: nil,
wantState: ProtectionStateUnknown,
wantFresh: ProtectionFreshnessUnknown,
wantVerify: ProtectionVerificationUnknown,
wantCoverage: ProtectionCoverageUnknown,
wantText: "no complete provider history",
},
}
for _, test := range tests {
test := test
t.Run(test.name, func(t *testing.T) {
t.Parallel()
got := DeriveProtectionPostureAt(
"resource:vm-100",
test.summaries,
policy,
now,
)
if got.State != test.wantState {
t.Fatalf("state = %q, want %q; posture=%#v", got.State, test.wantState, got)
}
if got.Freshness != test.wantFresh {
t.Fatalf("freshness = %q, want %q", got.Freshness, test.wantFresh)
}
if got.Verification != test.wantVerify {
t.Fatalf("verification = %q, want %q", got.Verification, test.wantVerify)
}
if got.Coverage != test.wantCoverage {
t.Fatalf("coverage = %q, want %q", got.Coverage, test.wantCoverage)
}
if !strings.Contains(got.Explanation, test.wantText) {
t.Fatalf("explanation = %q, want substring %q", got.Explanation, test.wantText)
}
if err := got.Validate(); err != nil {
t.Fatalf("Validate() error = %v", err)
}
})
}
}
func TestBuildProtectionPostureFromPointsUsesLatestProviderObservation(t *testing.T) {
t.Parallel()
now := time.Date(2026, 7, 19, 8, 0, 0, 0, time.UTC)
completedAt := now.Add(-time.Hour)
verified := true
point := RecoveryPoint{
ID: "pbs-backup:vm-100-2026-07-19",
Provider: ProviderProxmoxPBS,
Kind: KindBackup,
Mode: ModeRemote,
Outcome: OutcomeSuccess,
CompletedAt: &completedAt,
Verified: &verified,
SubjectResourceID: "resource:vm-100",
ProviderScope: "pbs-main",
}
envelope, err := NewRecoveryPointEvidence(point, "pbs-backup-inventory", now)
if err != nil {
t.Fatalf("NewRecoveryPointEvidence() error = %v", err)
}
point.Evidence = envelope
older, err := NewProtectionProviderObservation(
ProviderProxmoxPBS,
"pbs-backup-enumeration",
"pbs-main",
OutcomeFailed,
ProtectionHistoryUnavailable,
operationaltrust.EvidencePermissionsUnknown,
true,
now.Add(-2*time.Hour),
now.Add(-2*time.Hour),
&operationaltrust.EvidenceReason{Code: "pbs_timeout"},
)
if err != nil {
t.Fatalf("older observation error = %v", err)
}
current, err := NewProtectionProviderObservation(
ProviderProxmoxPBS,
"pbs-backup-enumeration",
"pbs-main",
OutcomeSuccess,
ProtectionHistoryComplete,
operationaltrust.EvidencePermissionsSufficient,
true,
now,
now,
nil,
)
if err != nil {
t.Fatalf("current observation error = %v", err)
}
got := BuildProtectionPostureFromPointsAt(
"resource:vm-100",
[]RecoveryPoint{point},
[]ProtectionProviderObservation{current, older},
DefaultProtectionPosturePolicy,
now,
)
if got.State != ProtectionStateProtected {
t.Fatalf("state = %q, want protected; posture=%#v", got.State, got)
}
if len(got.ProviderStates) != 1 {
t.Fatalf("provider states = %d, want 1", len(got.ProviderStates))
}
if got.ProviderStates[0].HistoryCompleteness != ProtectionHistoryComplete {
t.Fatalf(
"history completeness = %q, want complete",
got.ProviderStates[0].HistoryCompleteness,
)
}
if len(got.EvidenceIDs) != 2 {
t.Fatalf("evidence ids = %#v, want point and latest provider evidence", got.EvidenceIDs)
}
}
func TestNewProtectionProviderObservationRequiresTypedLimitation(t *testing.T) {
t.Parallel()
now := time.Date(2026, 7, 19, 8, 0, 0, 0, time.UTC)
observation, err := NewProtectionProviderObservation(
ProviderProxmoxPBS,
"pbs-backup-enumeration",
"pbs-main",
OutcomeFailed,
ProtectionHistoryUnavailable,
operationaltrust.EvidencePermissionsDenied,
true,
now,
now,
&operationaltrust.EvidenceReason{
Code: "pbs_access_denied",
Message: "PBS did not authorize backup history enumeration.",
},
)
if err != nil {
t.Fatalf("NewProtectionProviderObservation() error = %v", err)
}
if observation.Evidence.Reason == nil ||
observation.Evidence.Reason.Code != "pbs_access_denied" {
t.Fatalf("reason = %#v, want pbs_access_denied", observation.Evidence.Reason)
}
if observation.Evidence.Permissions != operationaltrust.EvidencePermissionsDenied {
t.Fatalf("permissions = %q, want denied", observation.Evidence.Permissions)
}
}
internal/recovery/store/store.go
+216 -8
View File
@@ -13,6 +13,7 @@ import (
"sync"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
"github.com/rcourtman/pulse-go-rewrite/internal/recovery"
"github.com/rs/zerolog/log"
@@ -362,6 +363,9 @@ func Open(dbPath string) (*Store, error) {
// Best-effort backfill for PBS guest rows that predate canonical historical continuity.
// This should never block startup; missing backfills only affect protected-item accuracy.
_ = store.BackfillHistoricalProxmoxPBSGuestIdentity(context.Background())
// Best-effort backfill for provider scopes and the materialized canonical
// protection posture read model.
_ = store.BackfillProtectionMetadata(context.Background())
if err := hardenSQLiteArtifacts(dbPath); err != nil {
_ = db.Close()
return nil, fmt.Errorf("failed to secure recovery db files: %w", err)
@@ -394,6 +398,35 @@ func (s *Store) PurgeStalePVEPBSBackups(ctx context.Context) error {
ctx = context.Background()
}
rows, err := s.db.QueryContext(ctx, `
SELECT DISTINCT subject_key
FROM recovery_points
WHERE id LIKE 'pve-backup:%'
AND json_extract(details_json, '$.isPBS') = 1
AND subject_key IS NOT NULL AND TRIM(subject_key) != ''
`)
if err != nil {
return err
}
affectedSubjectKeys := make(map[string]struct{})
for rows.Next() {
var key string
if err := rows.Scan(&key); err != nil {
_ = rows.Close()
return err
}
if key = strings.TrimSpace(key); key != "" {
affectedSubjectKeys[key] = struct{}{}
}
}
if err := rows.Err(); err != nil {
_ = rows.Close()
return err
}
if err := rows.Close(); err != nil {
return err
}
res, err := s.db.ExecContext(ctx, `
DELETE FROM recovery_points
WHERE id LIKE 'pve-backup:%'
@@ -408,6 +441,15 @@ func (s *Store) PurgeStalePVEPBSBackups(ctx context.Context) error {
return err
}
if deleted > 0 {
if err := s.RefreshProtectionPostures(
ctx,
sortedStringSet(affectedSubjectKeys),
); err != nil {
return fmt.Errorf(
"refresh protection postures after stale PVE/PBS purge: %w",
err,
)
}
log.Info().
Int64("deleted", deleted).
Msg("Purged stale PVE-sourced PBS backup entries - PBS direct is now authoritative")
@@ -447,7 +489,7 @@ func (s *Store) ReconcileInstancePoints(ctx context.Context, provider, idPrefix,
// repository_ref_json.namespace. The id prefix keeps the classes apart, so
// matching either field never crosses providers.
rows, err := s.db.QueryContext(ctx, `
SELECT id FROM recovery_points
SELECT id, subject_key FROM recovery_points
WHERE provider = ?
AND id LIKE ? || '%'
AND (
@@ -461,13 +503,18 @@ func (s *Store) ReconcileInstancePoints(ctx context.Context, provider, idPrefix,
defer rows.Close()
var stale []string
affectedSubjectKeys := make(map[string]struct{})
for rows.Next() {
var id string
if err := rows.Scan(&id); err != nil {
var subjectKey sql.NullString
if err := rows.Scan(&id, &subjectKey); err != nil {
return 0, err
}
if _, ok := keep[id]; !ok {
stale = append(stale, id)
if key := strings.TrimSpace(subjectKey.String); key != "" {
affectedSubjectKeys[key] = struct{}{}
}
}
}
if err := rows.Err(); err != nil {
@@ -512,6 +559,15 @@ func (s *Store) ReconcileInstancePoints(ctx context.Context, provider, idPrefix,
if err = tx.Commit(); err != nil {
return 0, err
}
if err := s.RefreshProtectionPostures(
ctx,
sortedStringSet(affectedSubjectKeys),
); err != nil {
return deleted, fmt.Errorf(
"refresh protection postures after recovery point reconciliation: %w",
err,
)
}
if deleted > 0 {
log.Info().
@@ -543,6 +599,9 @@ func (s *Store) initSchema() error {
verified INTEGER,
encrypted INTEGER,
immutable INTEGER,
provider_scope TEXT,
evidence_id TEXT,
evidence_json TEXT,
subject_key TEXT,
repository_key TEXT,
subject_resource_id TEXT,
@@ -575,6 +634,37 @@ func (s *Store) initSchema() error {
CREATE INDEX IF NOT EXISTS idx_recovery_points_subject_key_completed
ON recovery_points(subject_key, completed_at_ms);
CREATE TABLE IF NOT EXISTS protection_provider_observations (
id TEXT PRIMARY KEY,
provider TEXT NOT NULL,
source TEXT NOT NULL,
scope TEXT NOT NULL,
job_state TEXT NOT NULL,
history_completeness TEXT NOT NULL,
permissions TEXT NOT NULL,
verification_expected INTEGER NOT NULL DEFAULT 0,
observed_at_ms INTEGER NOT NULL,
ingested_at_ms INTEGER NOT NULL,
evidence_json TEXT NOT NULL,
created_at_ms INTEGER NOT NULL,
updated_at_ms INTEGER NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_protection_provider_observations_scope
ON protection_provider_observations(provider, scope, observed_at_ms DESC);
CREATE TABLE IF NOT EXISTS protection_postures (
subject_key TEXT PRIMARY KEY,
subject_resource_id TEXT NOT NULL UNIQUE,
state TEXT NOT NULL,
posture_json TEXT NOT NULL,
evaluated_at_ms INTEGER NOT NULL,
updated_at_ms INTEGER NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_protection_postures_state
ON protection_postures(state, evaluated_at_ms DESC, subject_resource_id);
`
if _, err := s.db.Exec(schema); err != nil {
return err
@@ -587,6 +677,15 @@ func (s *Store) initSchema() error {
if err := s.ensureColumn("recovery_points", "repository_key", "TEXT"); err != nil {
return err
}
if err := s.ensureColumn("recovery_points", "provider_scope", "TEXT"); err != nil {
return err
}
if err := s.ensureColumn("recovery_points", "evidence_id", "TEXT"); err != nil {
return err
}
if err := s.ensureColumn("recovery_points", "evidence_json", "TEXT"); err != nil {
return err
}
// Migration: normalized index columns for efficient filtering/searching and consistent UI display.
for _, col := range []struct {
@@ -623,6 +722,12 @@ func (s *Store) initSchema() error {
CREATE INDEX IF NOT EXISTS idx_recovery_points_namespace_completed
ON recovery_points(namespace_label, completed_at_ms);
CREATE INDEX IF NOT EXISTS idx_recovery_points_subject_provider_scope
ON recovery_points(subject_key, provider, provider_scope, completed_at_ms);
CREATE INDEX IF NOT EXISTS idx_recovery_points_provider_scope
ON recovery_points(provider, provider_scope, completed_at_ms);
`
if _, err := s.db.Exec(postMigrationIndexes); err != nil {
return err
@@ -975,6 +1080,7 @@ func (s *Store) UpsertPoints(ctx context.Context, points []recovery.RecoveryPoin
id, provider, kind, mode, outcome,
started_at_ms, completed_at_ms, size_bytes,
verified, encrypted, immutable,
provider_scope, evidence_id, evidence_json,
subject_key, repository_key,
subject_resource_id, repository_resource_id,
subject_ref_json, repository_ref_json, details_json,
@@ -982,7 +1088,7 @@ func (s *Store) UpsertPoints(ctx context.Context, points []recovery.RecoveryPoin
cluster_label, node_host_label, namespace_label, entity_id_label,
repository_label, details_summary,
created_at_ms, updated_at_ms
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(id) DO UPDATE SET
provider=excluded.provider,
kind=excluded.kind,
@@ -994,6 +1100,9 @@ func (s *Store) UpsertPoints(ctx context.Context, points []recovery.RecoveryPoin
verified=excluded.verified,
encrypted=excluded.encrypted,
immutable=excluded.immutable,
provider_scope=excluded.provider_scope,
evidence_id=excluded.evidence_id,
evidence_json=excluded.evidence_json,
subject_key=excluded.subject_key,
repository_key=excluded.repository_key,
subject_resource_id=excluded.subject_resource_id,
@@ -1018,6 +1127,7 @@ func (s *Store) UpsertPoints(ctx context.Context, points []recovery.RecoveryPoin
}
defer stmt.Close()
affectedSubjectKeys := make(map[string]struct{}, len(points))
for _, p := range points {
adoptHistoricalProxmoxPBSGuestIdentity(&p, proxmoxPBSIdentities, proxmoxPBSLooseIdentities)
@@ -1045,6 +1155,10 @@ func (s *Store) UpsertPoints(ctx context.Context, points []recovery.RecoveryPoin
if err != nil {
return err
}
evidence, err := marshalJSON(p.Evidence)
if err != nil {
return err
}
var size any
if p.SizeBytes != nil {
@@ -1054,7 +1168,15 @@ func (s *Store) UpsertPoints(ctx context.Context, points []recovery.RecoveryPoin
subjectRID := strings.TrimSpace(p.SubjectResourceID)
repoRID := strings.TrimSpace(p.RepositoryResourceID)
subjectKey := recovery.SubjectKeyForPoint(p)
if strings.TrimSpace(subjectKey) != "" {
affectedSubjectKeys[strings.TrimSpace(subjectKey)] = struct{}{}
}
repoKey := recovery.RepositoryKey(p.Provider, repoRID, p.RepositoryRef)
providerScope := recovery.ProviderScopeForPoint(p)
evidenceID := ""
if p.Evidence != nil {
evidenceID = strings.TrimSpace(p.Evidence.ID)
}
idx := recovery.DeriveIndex(p)
if _, err := stmt.ExecContext(
@@ -1070,6 +1192,9 @@ func (s *Store) UpsertPoints(ctx context.Context, points []recovery.RecoveryPoin
boolPtrToDB(p.Verified),
boolPtrToDB(p.Encrypted),
boolPtrToDB(p.Immutable),
providerScope,
evidenceID,
nullStringToAny(evidence),
strings.TrimSpace(subjectKey),
strings.TrimSpace(repoKey),
subjectRID,
@@ -1110,6 +1235,12 @@ func (s *Store) UpsertPoints(ctx context.Context, points []recovery.RecoveryPoin
if err := tx.Commit(); err != nil {
return err
}
if err := s.RefreshProtectionPostures(
ctx,
sortedStringSet(affectedSubjectKeys),
); err != nil {
return fmt.Errorf("refresh protection postures after recovery point upsert: %w", err)
}
return nil
}
@@ -1137,7 +1268,8 @@ func (s *Store) BackfillHistoricalProxmoxPBSGuestIdentity(ctx context.Context) e
id, provider, kind, mode, outcome,
subject_resource_id, repository_resource_id,
subject_ref_json, repository_ref_json, details_json,
subject_label, item_type, namespace_label, entity_id_label
subject_label, item_type, namespace_label, entity_id_label,
subject_key
FROM recovery_points
WHERE provider = 'proxmox-pbs'
LIMIT `+fmt.Sprint(maxBackfillRows)+`
@@ -1162,6 +1294,7 @@ func (s *Store) BackfillHistoricalProxmoxPBSGuestIdentity(ctx context.Context) e
itemType sql.NullString
namespaceLabel sql.NullString
entityIDLabel sql.NullString
subjectKey sql.NullString
}
items := make([]item, 0, 256)
@@ -1182,6 +1315,7 @@ func (s *Store) BackfillHistoricalProxmoxPBSGuestIdentity(ctx context.Context) e
&r.itemType,
&r.namespaceLabel,
&r.entityIDLabel,
&r.subjectKey,
); err != nil {
return err
}
@@ -1223,6 +1357,7 @@ func (s *Store) BackfillHistoricalProxmoxPBSGuestIdentity(ctx context.Context) e
}
defer stmt.Close()
affectedSubjectKeys := make(map[string]struct{})
for _, item := range items {
key := recovery.ProxmoxPBSGuestContinuityKey(
item.subjectLabel.String,
@@ -1289,9 +1424,10 @@ func (s *Store) BackfillHistoricalProxmoxPBSGuestIdentity(ctx context.Context) e
return err
}
nextSubjectKey := recovery.SubjectKeyForPoint(p)
if _, err := stmt.ExecContext(
ctx,
recovery.SubjectKeyForPoint(p),
nextSubjectKey,
p.SubjectResourceID,
nullStringToAny(subjectRefJSON),
strings.TrimSpace(idx.SubjectLabel),
@@ -1308,11 +1444,26 @@ func (s *Store) BackfillHistoricalProxmoxPBSGuestIdentity(ctx context.Context) e
); err != nil {
return err
}
if key := strings.TrimSpace(item.subjectKey.String); key != "" {
affectedSubjectKeys[key] = struct{}{}
}
if key := strings.TrimSpace(nextSubjectKey); key != "" {
affectedSubjectKeys[key] = struct{}{}
}
}
if err := tx.Commit(); err != nil {
return err
}
if err := s.RefreshProtectionPostures(
ctx,
sortedStringSet(affectedSubjectKeys),
); err != nil {
return fmt.Errorf(
"refresh protection postures after PBS identity backfill: %w",
err,
)
}
return nil
}
@@ -1334,11 +1485,58 @@ func (s *Store) maybePrune(ctx context.Context) {
s.lastPrune = now
cutoffMs := now.Add(-s.retention).UnixMilli()
// Best-effort cleanup; never block writes if pruning fails.
_, _ = s.db.ExecContext(ctx, `
// Best-effort cleanup; never block writes if pruning fails. Capture every
// subject whose posture can change before deleting either source points or
// the provider evidence used to qualify their history.
affectedSubjectKeys := make(map[string]struct{})
rows, err := s.db.QueryContext(ctx, `
SELECT DISTINCT subject_key
FROM recovery_points
WHERE COALESCE(completed_at_ms, started_at_ms, updated_at_ms) < ?
AND subject_key IS NOT NULL AND TRIM(subject_key) != ''
UNION
SELECT DISTINCT rp.subject_key
FROM recovery_points rp
JOIN protection_provider_observations observation
ON observation.provider = rp.provider
AND observation.scope = rp.provider_scope
WHERE observation.observed_at_ms < ?
AND rp.subject_key IS NOT NULL AND TRIM(rp.subject_key) != ''
`, cutoffMs, cutoffMs)
if err == nil {
for rows.Next() {
var key string
if scanErr := rows.Scan(&key); scanErr != nil {
err = scanErr
break
}
if key = strings.TrimSpace(key); key != "" {
affectedSubjectKeys[key] = struct{}{}
}
}
if rowsErr := rows.Err(); err == nil {
err = rowsErr
}
if closeErr := rows.Close(); err == nil {
err = closeErr
}
}
if err != nil {
return
}
if _, err = s.db.ExecContext(ctx, `
DELETE FROM recovery_points
WHERE COALESCE(completed_at_ms, started_at_ms, updated_at_ms) < ?
`, cutoffMs)
`, cutoffMs); err != nil {
return
}
if _, err = s.db.ExecContext(ctx, `
DELETE FROM protection_provider_observations
WHERE observed_at_ms < ?
`, cutoffMs); err != nil {
return
}
_ = s.RefreshProtectionPostures(ctx, sortedStringSet(affectedSubjectKeys))
}
func nullStringToAny(s sql.NullString) any {
@@ -1464,6 +1662,7 @@ func (s *Store) ListPoints(ctx context.Context, opts recovery.ListPointsOptions)
id, provider, kind, mode, outcome,
started_at_ms, completed_at_ms, size_bytes,
verified, encrypted, immutable,
provider_scope, evidence_json,
subject_resource_id, repository_resource_id,
subject_ref_json, repository_ref_json, details_json
, subject_label, subject_type, item_type, is_workload,
@@ -1489,6 +1688,7 @@ func (s *Store) ListPoints(ctx context.Context, opts recovery.ListPointsOptions)
var startedMs, completedMs sql.NullInt64
var sizeBytes sql.NullInt64
var verified, encrypted, immutable sql.NullInt64
var providerScope, evidenceRaw sql.NullString
var subjectRID, repoRID sql.NullString
var subjectRefRaw, repoRefRaw, detailsRaw sql.NullString
var subjectLabel, subjectType sql.NullString
@@ -1509,6 +1709,8 @@ func (s *Store) ListPoints(ctx context.Context, opts recovery.ListPointsOptions)
&verified,
&encrypted,
&immutable,
&providerScope,
&evidenceRaw,
&subjectRID,
&repoRID,
&subjectRefRaw,
@@ -1543,6 +1745,12 @@ func (s *Store) ListPoints(ctx context.Context, opts recovery.ListPointsOptions)
p.Verified = dbToBoolPtr(verified)
p.Encrypted = dbToBoolPtr(encrypted)
p.Immutable = dbToBoolPtr(immutable)
p.ProviderScope = strings.TrimSpace(providerScope.String)
var evidence operationaltrust.EvidenceEnvelope
_ = decodeRecoveryJSONField(p.ID, "evidence_json", evidenceRaw, &evidence)
if strings.TrimSpace(evidence.ID) != "" {
p.Evidence = &evidence
}
if subjectRID.Valid {
p.SubjectResourceID = subjectRID.String
internal/recovery/store/store_posture.go
+769
View File
@@ -0,0 +1,769 @@
package store
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"sort"
"strings"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
"github.com/rcourtman/pulse-go-rewrite/internal/recovery"
)
const protectionRefreshChunkSize = 200
func sortedStringSet(values map[string]struct{}) []string {
out := make([]string, 0, len(values))
for value := range values {
value = strings.TrimSpace(value)
if value != "" {
out = append(out, value)
}
}
sort.Strings(out)
return out
}
func (s *Store) BackfillProtectionMetadata(ctx context.Context) error {
if err := s.ensureInitialized(); err != nil {
return err
}
if ctx == nil {
ctx = context.Background()
}
const maxBackfillRows = 5000
rows, err := s.db.QueryContext(ctx, `
SELECT id, provider, kind, mode, outcome,
started_at_ms, completed_at_ms, updated_at_ms,
subject_resource_id, subject_ref_json,
repository_resource_id, repository_ref_json, details_json,
evidence_id, evidence_json
FROM recovery_points
WHERE provider_scope IS NULL OR TRIM(provider_scope) = ''
OR (
(evidence_json IS NULL OR TRIM(evidence_json) = '')
AND COALESCE(completed_at_ms, started_at_ms, 0) > 0
)
LIMIT `+fmt.Sprint(maxBackfillRows))
if err != nil {
return err
}
type backfillRow struct {
id string
provider string
kind string
mode string
outcome string
startedMs sql.NullInt64
completedMs sql.NullInt64
updatedAtMs int64
subjectRID sql.NullString
subjectRefRaw sql.NullString
repoRID sql.NullString
repoRefRaw sql.NullString
detailsRaw sql.NullString
evidenceID sql.NullString
evidenceRaw sql.NullString
}
items := make([]backfillRow, 0, 256)
for rows.Next() {
var item backfillRow
if err := rows.Scan(
&item.id,
&item.provider,
&item.kind,
&item.mode,
&item.outcome,
&item.startedMs,
&item.completedMs,
&item.updatedAtMs,
&item.subjectRID,
&item.subjectRefRaw,
&item.repoRID,
&item.repoRefRaw,
&item.detailsRaw,
&item.evidenceID,
&item.evidenceRaw,
); err != nil {
_ = rows.Close()
return err
}
items = append(items, item)
}
if err := rows.Err(); err != nil {
_ = rows.Close()
return err
}
if err := rows.Close(); err != nil {
return err
}
if len(items) > 0 {
tx, err := s.db.BeginTx(ctx, nil)
if err != nil {
return err
}
defer func() { _ = tx.Rollback() }()
stmt, err := tx.PrepareContext(
ctx,
`UPDATE recovery_points
SET provider_scope = ?,
evidence_id = CASE
WHEN evidence_id IS NULL OR TRIM(evidence_id) = '' THEN ?
ELSE evidence_id
END,
evidence_json = CASE
WHEN evidence_json IS NULL OR TRIM(evidence_json) = '' THEN ?
ELSE evidence_json
END
WHERE id = ?`,
)
if err != nil {
return err
}
defer stmt.Close()
for _, item := range items {
point := recovery.RecoveryPoint{
ID: strings.TrimSpace(item.id),
Provider: recovery.Provider(strings.TrimSpace(item.provider)),
Kind: recovery.Kind(strings.TrimSpace(item.kind)),
Mode: recovery.Mode(strings.TrimSpace(item.mode)),
Outcome: recovery.Outcome(strings.TrimSpace(item.outcome)),
StartedAt: millisToTimePtr(item.startedMs),
CompletedAt: millisToTimePtr(item.completedMs),
SubjectResourceID: strings.TrimSpace(item.subjectRID.String),
RepositoryResourceID: strings.TrimSpace(item.repoRID.String),
}
_ = unmarshalJSON(item.subjectRefRaw, &point.SubjectRef)
_ = unmarshalJSON(item.repoRefRaw, &point.RepositoryRef)
_ = unmarshalJSON(item.detailsRaw, &point.Details)
var evidenceID any
var evidenceJSON any
if strings.TrimSpace(item.evidenceRaw.String) == "" {
ingestedAt := time.UnixMilli(item.updatedAtMs).UTC()
evidence, evidenceErr := recovery.NewRecoveryPointEvidence(
point,
"recovery-point-migration",
ingestedAt,
)
if evidenceErr == nil {
encoded, marshalErr := json.Marshal(evidence)
if marshalErr != nil {
return marshalErr
}
evidenceID = evidence.ID
evidenceJSON = string(encoded)
}
}
if _, err := stmt.ExecContext(
ctx,
recovery.ProviderScopeForPoint(point),
evidenceID,
evidenceJSON,
point.ID,
); err != nil {
return err
}
}
if err := tx.Commit(); err != nil {
return err
}
}
keyRows, err := s.db.QueryContext(ctx, `
SELECT DISTINCT subject_key
FROM recovery_points
WHERE subject_key IS NOT NULL AND TRIM(subject_key) != ''
ORDER BY subject_key
LIMIT `+fmt.Sprint(maxBackfillRows))
if err != nil {
return err
}
keys := make([]string, 0, 512)
for keyRows.Next() {
var key string
if err := keyRows.Scan(&key); err != nil {
_ = keyRows.Close()
return err
}
if key = strings.TrimSpace(key); key != "" {
keys = append(keys, key)
}
}
if err := keyRows.Err(); err != nil {
_ = keyRows.Close()
return err
}
if err := keyRows.Close(); err != nil {
return err
}
return s.RefreshProtectionPostures(ctx, keys)
}
func (s *Store) UpsertProtectionProviderObservations(
ctx context.Context,
observations []recovery.ProtectionProviderObservation,
) error {
if err := s.ensureInitialized(); err != nil {
return err
}
if len(observations) == 0 {
return nil
}
if ctx == nil {
ctx = context.Background()
}
s.maybePrune(ctx)
nowMs := time.Now().UTC().UnixMilli()
tx, err := s.db.BeginTx(ctx, nil)
if err != nil {
return err
}
defer func() { _ = tx.Rollback() }()
stmt, err := tx.PrepareContext(ctx, `
INSERT INTO protection_provider_observations (
id, provider, source, scope, job_state,
history_completeness, permissions, verification_expected,
observed_at_ms, ingested_at_ms, evidence_json,
created_at_ms, updated_at_ms
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(id) DO UPDATE SET
provider=excluded.provider,
source=excluded.source,
scope=excluded.scope,
job_state=excluded.job_state,
history_completeness=excluded.history_completeness,
permissions=excluded.permissions,
verification_expected=excluded.verification_expected,
observed_at_ms=excluded.observed_at_ms,
ingested_at_ms=excluded.ingested_at_ms,
evidence_json=excluded.evidence_json,
updated_at_ms=excluded.updated_at_ms
`)
if err != nil {
return err
}
defer stmt.Close()
type providerScope struct {
provider recovery.Provider
scope string
}
affectedScopes := make(map[providerScope]struct{}, len(observations))
for _, observation := range observations {
if err := observation.Validate(); err != nil {
return err
}
evidenceJSON, err := json.Marshal(observation.Evidence)
if err != nil {
return err
}
verificationExpected := 0
if observation.VerificationExpected {
verificationExpected = 1
}
if _, err := stmt.ExecContext(
ctx,
observation.ID,
string(observation.Provider),
observation.Source,
observation.Scope,
string(observation.JobState),
string(observation.HistoryCompleteness),
string(observation.Permissions),
verificationExpected,
observation.ObservedAt.UTC().UnixMilli(),
observation.IngestedAt.UTC().UnixMilli(),
string(evidenceJSON),
nowMs,
nowMs,
); err != nil {
return err
}
affectedScopes[providerScope{
provider: observation.Provider,
scope: strings.TrimSpace(observation.Scope),
}] = struct{}{}
}
if err := tx.Commit(); err != nil {
return err
}
affectedKeys := make(map[string]struct{})
for scope := range affectedScopes {
rows, err := s.db.QueryContext(ctx, `
SELECT DISTINCT subject_key
FROM recovery_points
WHERE provider = ? AND provider_scope = ?
AND subject_key IS NOT NULL AND TRIM(subject_key) != ''
`, string(scope.provider), scope.scope)
if err != nil {
return err
}
for rows.Next() {
var key string
if err := rows.Scan(&key); err != nil {
_ = rows.Close()
return err
}
if key = strings.TrimSpace(key); key != "" {
affectedKeys[key] = struct{}{}
}
}
if err := rows.Err(); err != nil {
_ = rows.Close()
return err
}
if err := rows.Close(); err != nil {
return err
}
}
return s.RefreshProtectionPostures(ctx, sortedStringSet(affectedKeys))
}
func (s *Store) RefreshProtectionPostures(ctx context.Context, subjectKeys []string) error {
if err := s.ensureInitialized(); err != nil {
return err
}
subjectKeys = normalizeProtectionStrings(subjectKeys)
if len(subjectKeys) == 0 {
return nil
}
if ctx == nil {
ctx = context.Background()
}
observations, err := s.listLatestProtectionProviderObservations(ctx)
if err != nil {
return err
}
for start := 0; start < len(subjectKeys); start += protectionRefreshChunkSize {
end := start + protectionRefreshChunkSize
if end > len(subjectKeys) {
end = len(subjectKeys)
}
chunk := subjectKeys[start:end]
pointsByKey, err := s.loadProtectionPointsForKeys(ctx, chunk)
if err != nil {
return err
}
if err := s.writeProtectionPostures(
ctx,
chunk,
pointsByKey,
observations,
); err != nil {
return err
}
}
return nil
}
func (s *Store) loadProtectionPointsForKeys(
ctx context.Context,
subjectKeys []string,
) (map[string][]recovery.RecoveryPoint, error) {
placeholders := strings.TrimSuffix(strings.Repeat("?,", len(subjectKeys)), ",")
args := make([]any, len(subjectKeys))
for i := range subjectKeys {
args[i] = subjectKeys[i]
}
rows, err := s.db.QueryContext(ctx, `
SELECT
subject_key, id, provider, kind, mode, outcome,
started_at_ms, completed_at_ms, verified,
subject_resource_id, repository_resource_id,
subject_ref_json, repository_ref_json, details_json,
provider_scope, evidence_json
FROM recovery_points
WHERE subject_key IN (`+placeholders+`)
ORDER BY subject_key,
COALESCE(completed_at_ms, started_at_ms, updated_at_ms) DESC,
updated_at_ms DESC,
id DESC
`, args...)
if err != nil {
return nil, err
}
defer rows.Close()
out := make(map[string][]recovery.RecoveryPoint, len(subjectKeys))
for rows.Next() {
var (
subjectKey string
point recovery.RecoveryPoint
provider, kind string
mode, outcome string
startedMs, completedMs sql.NullInt64
verified sql.NullInt64
subjectRID, repoRID sql.NullString
subjectRefRaw sql.NullString
repoRefRaw, detailsRaw sql.NullString
providerScope sql.NullString
evidenceRaw sql.NullString
)
if err := rows.Scan(
&subjectKey,
&point.ID,
&provider,
&kind,
&mode,
&outcome,
&startedMs,
&completedMs,
&verified,
&subjectRID,
&repoRID,
&subjectRefRaw,
&repoRefRaw,
&detailsRaw,
&providerScope,
&evidenceRaw,
); err != nil {
return nil, err
}
point.Provider = recovery.Provider(provider)
point.Kind = recovery.Kind(kind)
point.Mode = recovery.Mode(mode)
point.Outcome = recovery.Outcome(outcome)
point.StartedAt = millisToTimePtr(startedMs)
point.CompletedAt = millisToTimePtr(completedMs)
point.Verified = dbToBoolPtr(verified)
point.SubjectResourceID = strings.TrimSpace(subjectRID.String)
point.RepositoryResourceID = strings.TrimSpace(repoRID.String)
point.ProviderScope = strings.TrimSpace(providerScope.String)
_ = unmarshalJSON(subjectRefRaw, &point.SubjectRef)
_ = unmarshalJSON(repoRefRaw, &point.RepositoryRef)
_ = unmarshalJSON(detailsRaw, &point.Details)
var evidence operationaltrust.EvidenceEnvelope
_ = unmarshalJSON(evidenceRaw, &evidence)
if strings.TrimSpace(evidence.ID) != "" {
point.Evidence = &evidence
}
subjectKey = strings.TrimSpace(subjectKey)
out[subjectKey] = append(out[subjectKey], point)
}
return out, rows.Err()
}
func (s *Store) listLatestProtectionProviderObservations(
ctx context.Context,
) ([]recovery.ProtectionProviderObservation, error) {
rows, err := s.db.QueryContext(ctx, `
SELECT
id, provider, source, scope, job_state,
history_completeness, permissions, verification_expected,
observed_at_ms, ingested_at_ms, evidence_json
FROM (
SELECT *,
ROW_NUMBER() OVER (
PARTITION BY provider, scope
ORDER BY observed_at_ms DESC, id DESC
) AS rn
FROM protection_provider_observations
)
WHERE rn = 1
ORDER BY provider, scope
`)
if err != nil {
return nil, err
}
defer rows.Close()
out := make([]recovery.ProtectionProviderObservation, 0, 16)
for rows.Next() {
var (
observation recovery.ProtectionProviderObservation
provider, jobState string
historyCompleteness, permissions string
verificationExpected int
observedAtMs, ingestedAtMs int64
evidenceRaw string
)
if err := rows.Scan(
&observation.ID,
&provider,
&observation.Source,
&observation.Scope,
&jobState,
&historyCompleteness,
&permissions,
&verificationExpected,
&observedAtMs,
&ingestedAtMs,
&evidenceRaw,
); err != nil {
return nil, err
}
observation.Provider = recovery.Provider(provider)
observation.JobState = recovery.Outcome(jobState)
observation.HistoryCompleteness =
recovery.ProtectionHistoryCompleteness(historyCompleteness)
observation.Permissions = operationaltrust.EvidencePermissions(permissions)
observation.VerificationExpected = verificationExpected != 0
observation.ObservedAt = time.UnixMilli(observedAtMs).UTC()
observation.IngestedAt = time.UnixMilli(ingestedAtMs).UTC()
if err := json.Unmarshal([]byte(evidenceRaw), &observation.Evidence); err != nil {
continue
}
if err := observation.Validate(); err != nil {
continue
}
out = append(out, observation)
}
return out, rows.Err()
}
func (s *Store) writeProtectionPostures(
ctx context.Context,
subjectKeys []string,
pointsByKey map[string][]recovery.RecoveryPoint,
observations []recovery.ProtectionProviderObservation,
) error {
now := time.Now().UTC()
nowMs := now.UnixMilli()
tx, err := s.db.BeginTx(ctx, nil)
if err != nil {
return err
}
defer func() { _ = tx.Rollback() }()
upsert, err := tx.PrepareContext(ctx, `
INSERT INTO protection_postures (
subject_key, subject_resource_id, state,
posture_json, evaluated_at_ms, updated_at_ms
) VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT(subject_key) DO UPDATE SET
subject_resource_id=excluded.subject_resource_id,
state=excluded.state,
posture_json=excluded.posture_json,
evaluated_at_ms=excluded.evaluated_at_ms,
updated_at_ms=excluded.updated_at_ms
`)
if err != nil {
return err
}
defer upsert.Close()
for _, key := range subjectKeys {
points := pointsByKey[key]
subjectResourceID := ""
for _, point := range points {
if value := strings.TrimSpace(point.SubjectResourceID); value != "" {
subjectResourceID = value
break
}
}
if subjectResourceID == "" {
if _, err := tx.ExecContext(
ctx,
`DELETE FROM protection_postures WHERE subject_key = ?`,
key,
); err != nil {
return err
}
continue
}
posture := recovery.BuildProtectionPostureFromPointsAt(
subjectResourceID,
points,
observations,
recovery.DefaultProtectionPosturePolicy,
now,
)
if err := posture.Validate(); err != nil {
return err
}
postureJSON, err := json.Marshal(posture)
if err != nil {
return err
}
if _, err := upsert.ExecContext(
ctx,
key,
subjectResourceID,
string(posture.State),
string(postureJSON),
posture.EvaluatedAt.UTC().UnixMilli(),
nowMs,
); err != nil {
return err
}
}
return tx.Commit()
}
func (s *Store) ListProtectionPostures(
ctx context.Context,
query recovery.ProtectionPostureQuery,
) ([]recovery.ProtectionPosture, int, error) {
if err := s.ensureInitialized(); err != nil {
return nil, 0, err
}
if query.State != "" && !query.State.Valid() {
return nil, 0, fmt.Errorf("invalid protection posture state %q", query.State)
}
if ctx == nil {
ctx = context.Background()
}
requestedIDs := normalizeProtectionStrings(query.SubjectResourceIDs)
if len(requestedIDs) > 0 {
return s.listRequestedProtectionPostures(ctx, requestedIDs, query.State)
}
limit := normalizeLimit(query.Limit)
page := normalizePage(query.Page)
offset := (page - 1) * limit
whereSQL := ""
args := make([]any, 0, 3)
if query.State != "" {
whereSQL = "WHERE state = ?"
args = append(args, string(query.State))
}
var total int
if err := s.db.QueryRowContext(
ctx,
"SELECT COUNT(*) FROM protection_postures "+whereSQL,
args...,
).Scan(&total); err != nil {
return nil, 0, err
}
args = append(args, limit, offset)
rows, err := s.db.QueryContext(ctx, `
SELECT posture_json
FROM protection_postures
`+whereSQL+`
ORDER BY
CASE state
WHEN 'attention' THEN 0
WHEN 'unprotected' THEN 1
WHEN 'unknown' THEN 2
ELSE 3
END,
evaluated_at_ms DESC,
subject_resource_id
LIMIT ? OFFSET ?
`, args...)
if err != nil {
return nil, 0, err
}
defer rows.Close()
postures := make([]recovery.ProtectionPosture, 0, limit)
for rows.Next() {
var postureJSON string
if err := rows.Scan(&postureJSON); err != nil {
return nil, 0, err
}
var posture recovery.ProtectionPosture
if err := json.Unmarshal([]byte(postureJSON), &posture); err != nil {
return nil, 0, err
}
postures = append(postures, posture)
}
return postures, total, rows.Err()
}
func (s *Store) listRequestedProtectionPostures(
ctx context.Context,
requestedIDs []string,
state recovery.ProtectionState,
) ([]recovery.ProtectionPosture, int, error) {
placeholders := strings.TrimSuffix(strings.Repeat("?,", len(requestedIDs)), ",")
args := make([]any, len(requestedIDs))
for i := range requestedIDs {
args[i] = requestedIDs[i]
}
rows, err := s.db.QueryContext(ctx, `
SELECT subject_key, subject_resource_id
FROM protection_postures
WHERE subject_resource_id IN (`+placeholders+`)
`, args...)
if err != nil {
return nil, 0, err
}
defer rows.Close()
subjectKeyByID := make(map[string]string, len(requestedIDs))
subjectKeys := make([]string, 0, len(requestedIDs))
for rows.Next() {
var subjectKey, subjectResourceID string
if err := rows.Scan(&subjectKey, &subjectResourceID); err != nil {
return nil, 0, err
}
subjectKey = strings.TrimSpace(subjectKey)
subjectResourceID = strings.TrimSpace(subjectResourceID)
if subjectKey == "" || subjectResourceID == "" {
continue
}
subjectKeyByID[subjectResourceID] = subjectKey
subjectKeys = append(subjectKeys, subjectKey)
}
if err := rows.Err(); err != nil {
return nil, 0, err
}
if err := rows.Close(); err != nil {
return nil, 0, err
}
now := time.Now().UTC()
pointsByKey := make(map[string][]recovery.RecoveryPoint)
var observations []recovery.ProtectionProviderObservation
if len(subjectKeys) > 0 {
pointsByKey, err = s.loadProtectionPointsForKeys(ctx, normalizeProtectionStrings(subjectKeys))
if err != nil {
return nil, 0, err
}
observations, err = s.listLatestProtectionProviderObservations(ctx)
if err != nil {
return nil, 0, err
}
}
out := make([]recovery.ProtectionPosture, 0, len(requestedIDs))
for _, subjectResourceID := range requestedIDs {
subjectKey := subjectKeyByID[subjectResourceID]
var posture recovery.ProtectionPosture
if subjectKey == "" {
posture = recovery.DeriveProtectionPostureAt(
subjectResourceID,
nil,
recovery.DefaultProtectionPosturePolicy,
now,
)
} else {
posture = recovery.BuildProtectionPostureFromPointsAt(
subjectResourceID,
pointsByKey[subjectKey],
observations,
recovery.DefaultProtectionPosturePolicy,
now,
)
}
if state != "" && posture.State != state {
continue
}
out = append(out, posture)
}
return out, len(out), nil
}
func normalizeProtectionStrings(values []string) []string {
set := make(map[string]struct{}, len(values))
for _, value := range values {
if value = strings.TrimSpace(value); value != "" {
set[value] = struct{}{}
}
}
out := make([]string, 0, len(set))
for value := range set {
out = append(out, value)
}
sort.Strings(out)
return out
}
internal/recovery/store/store_posture_test.go
+499
View File
@@ -0,0 +1,499 @@
package store
import (
"context"
"encoding/json"
"path/filepath"
"strings"
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
"github.com/rcourtman/pulse-go-rewrite/internal/recovery"
)
func TestStoreProtectionPosturePersistsProviderAwareTruth(t *testing.T) {
t.Parallel()
store, err := Open(filepath.Join(t.TempDir(), "recovery.db"))
if err != nil {
t.Fatalf("Open() error = %v", err)
}
defer store.Close()
now := time.Now().UTC().Truncate(time.Millisecond)
backupTime := now.Add(-time.Hour)
verified := true
point := recovery.RecoveryPoint{
ID: "pbs-backup:vm-100",
Provider: recovery.ProviderProxmoxPBS,
Kind: recovery.KindBackup,
Mode: recovery.ModeRemote,
Outcome: recovery.OutcomeSuccess,
CompletedAt: &backupTime,
Verified: &verified,
SubjectResourceID: "resource:vm-100",
RepositoryResourceID: "resource:pbs-store-a",
ProviderScope: "pbs-main",
}
evidence, err := recovery.NewRecoveryPointEvidence(
point,
"pbs-backup-inventory",
now,
)
if err != nil {
t.Fatalf("NewRecoveryPointEvidence() error = %v", err)
}
point.Evidence = evidence
complete, err := recovery.NewProtectionProviderObservation(
recovery.ProviderProxmoxPBS,
"pbs-backup-enumeration",
"pbs-main",
recovery.OutcomeSuccess,
recovery.ProtectionHistoryComplete,
operationaltrust.EvidencePermissionsSufficient,
true,
now,
now,
nil,
)
if err != nil {
t.Fatalf("complete observation error = %v", err)
}
if err := store.UpsertProtectionProviderObservations(
context.Background(),
[]recovery.ProtectionProviderObservation{complete},
); err != nil {
t.Fatalf("UpsertProtectionProviderObservations() error = %v", err)
}
if err := store.UpsertPoints(
context.Background(),
[]recovery.RecoveryPoint{point},
); err != nil {
t.Fatalf("UpsertPoints() error = %v", err)
}
got, total, err := store.ListProtectionPostures(
context.Background(),
recovery.ProtectionPostureQuery{
SubjectResourceIDs: []string{"resource:vm-100", "resource:vm-404"},
},
)
if err != nil {
t.Fatalf("ListProtectionPostures() error = %v", err)
}
if total != 2 || len(got) != 2 {
t.Fatalf("postures total=%d len=%d, want 2/2", total, len(got))
}
byID := map[string]recovery.ProtectionPosture{}
for _, posture := range got {
byID[posture.SubjectResourceID] = posture
}
protected := byID["resource:vm-100"]
if protected.State != recovery.ProtectionStateProtected {
t.Fatalf("vm-100 state = %q, want protected; posture=%#v", protected.State, protected)
}
if protected.Coverage != recovery.ProtectionCoverageComplete {
t.Fatalf("vm-100 coverage = %q, want complete", protected.Coverage)
}
if protected.Verification != recovery.ProtectionVerificationVerified {
t.Fatalf("vm-100 verification = %q, want verified", protected.Verification)
}
if len(protected.ProviderStates) != 1 {
t.Fatalf("vm-100 provider states = %d, want 1", len(protected.ProviderStates))
}
if byID["resource:vm-404"].State != recovery.ProtectionStateUnknown {
t.Fatalf(
"vm-404 state = %q, want unknown",
byID["resource:vm-404"].State,
)
}
attentionList, total, err := store.ListProtectionPostures(
context.Background(),
recovery.ProtectionPostureQuery{
State: recovery.ProtectionStateProtected,
Page: 1,
Limit: 10,
},
)
if err != nil {
t.Fatalf("ListProtectionPostures(protected) error = %v", err)
}
if total != 1 || len(attentionList) != 1 {
t.Fatalf("protected postures total=%d len=%d, want 1/1", total, len(attentionList))
}
partialAt := now.Add(time.Minute)
partial, err := recovery.NewProtectionProviderObservation(
recovery.ProviderProxmoxPBS,
"pbs-backup-enumeration",
"pbs-main",
recovery.OutcomeWarning,
recovery.ProtectionHistoryPartial,
operationaltrust.EvidencePermissionsPartial,
true,
partialAt,
partialAt,
&operationaltrust.EvidenceReason{
Code: "pbs_partial_enumeration",
Message: "One PBS datastore could not be enumerated.",
},
)
if err != nil {
t.Fatalf("partial observation error = %v", err)
}
if err := store.UpsertProtectionProviderObservations(
context.Background(),
[]recovery.ProtectionProviderObservation{partial},
); err != nil {
t.Fatalf("UpsertProtectionProviderObservations(partial) error = %v", err)
}
got, _, err = store.ListProtectionPostures(
context.Background(),
recovery.ProtectionPostureQuery{
SubjectResourceIDs: []string{"resource:vm-100"},
},
)
if err != nil {
t.Fatalf("ListProtectionPostures(after partial) error = %v", err)
}
if got[0].State != recovery.ProtectionStateAttention {
t.Fatalf("state after partial = %q, want attention", got[0].State)
}
if got[0].Coverage != recovery.ProtectionCoveragePartial {
t.Fatalf("coverage after partial = %q, want partial", got[0].Coverage)
}
}
func TestStoreProtectionSchemaMigratesLegacyRecoveryDatabase(t *testing.T) {
t.Parallel()
dbPath := filepath.Join(t.TempDir(), "recovery.db")
store, err := Open(dbPath)
if err != nil {
t.Fatalf("Open() error = %v", err)
}
defer store.Close()
for _, table := range []string{
"protection_provider_observations",
"protection_postures",
} {
var name string
if err := store.db.QueryRow(
`SELECT name FROM sqlite_master WHERE type = 'table' AND name = ?`,
table,
).Scan(&name); err != nil {
t.Fatalf("lookup table %s: %v", table, err)
}
if name != table {
t.Fatalf("table = %q, want %q", name, table)
}
}
columns := map[string]bool{}
rows, err := store.db.Query(`PRAGMA table_info(recovery_points)`)
if err != nil {
t.Fatalf("PRAGMA table_info: %v", err)
}
defer rows.Close()
for rows.Next() {
var (
cid int
name string
columnType string
notNull int
defaultValue any
primaryKey int
)
if err := rows.Scan(
&cid,
&name,
&columnType,
&notNull,
&defaultValue,
&primaryKey,
); err != nil {
t.Fatalf("scan table_info: %v", err)
}
columns[name] = true
}
for _, column := range []string{"provider_scope", "evidence_id", "evidence_json"} {
if !columns[column] {
t.Fatalf("missing migrated recovery_points column %q", column)
}
}
}
func TestStoreRequestedProtectionPosturesReevaluateAtReadTime(t *testing.T) {
t.Parallel()
store, err := Open(filepath.Join(t.TempDir(), "recovery.db"))
if err != nil {
t.Fatalf("Open() error = %v", err)
}
defer store.Close()
now := time.Now().UTC().Truncate(time.Millisecond)
oldBackup := now.Add(-8 * 24 * time.Hour)
verified := true
point := recovery.RecoveryPoint{
ID: "pbs-backup:stale-vm",
Provider: recovery.ProviderProxmoxPBS,
Kind: recovery.KindBackup,
Mode: recovery.ModeRemote,
Outcome: recovery.OutcomeSuccess,
CompletedAt: &oldBackup,
Verified: &verified,
SubjectResourceID: "resource:stale-vm",
ProviderScope: "pbs-main",
}
evidence, err := recovery.NewRecoveryPointEvidence(point, "pbs-backup-inventory", now)
if err != nil {
t.Fatalf("NewRecoveryPointEvidence() error = %v", err)
}
point.Evidence = evidence
observation, err := recovery.NewProtectionProviderObservation(
recovery.ProviderProxmoxPBS,
"pbs-backup-enumeration",
"pbs-main",
recovery.OutcomeSuccess,
recovery.ProtectionHistoryComplete,
operationaltrust.EvidencePermissionsSufficient,
true,
now,
now,
nil,
)
if err != nil {
t.Fatalf("NewProtectionProviderObservation() error = %v", err)
}
if err := store.UpsertProtectionProviderObservations(
context.Background(),
[]recovery.ProtectionProviderObservation{observation},
); err != nil {
t.Fatalf("UpsertProtectionProviderObservations() error = %v", err)
}
if err := store.UpsertPoints(context.Background(), []recovery.RecoveryPoint{point}); err != nil {
t.Fatalf("UpsertPoints() error = %v", err)
}
// Corrupt the materialized snapshot into a reassuring answer. A requested
// batch must derive from points + current provider evidence instead of
// trusting a posture that can age between collection cycles.
fake := recovery.DeriveProtectionPostureAt(
point.SubjectResourceID,
nil,
recovery.DefaultProtectionPosturePolicy,
now,
)
fake.State = recovery.ProtectionStateProtected
fake.Freshness = recovery.ProtectionFreshnessCurrent
fake.Verification = recovery.ProtectionVerificationVerified
fake.Coverage = recovery.ProtectionCoverageComplete
fake.Explanation = "incorrect stored posture"
fakeJSON, err := json.Marshal(fake)
if err != nil {
t.Fatalf("json.Marshal(fake) error = %v", err)
}
if _, err := store.db.Exec(
`UPDATE protection_postures SET state = 'protected', posture_json = ?`,
string(fakeJSON),
); err != nil {
t.Fatalf("update materialized posture: %v", err)
}
got, _, err := store.ListProtectionPostures(
context.Background(),
recovery.ProtectionPostureQuery{
SubjectResourceIDs: []string{point.SubjectResourceID},
},
)
if err != nil {
t.Fatalf("ListProtectionPostures() error = %v", err)
}
if len(got) != 1 {
t.Fatalf("postures len = %d, want 1", len(got))
}
if got[0].State != recovery.ProtectionStateAttention {
t.Fatalf("state = %q, want attention; posture=%#v", got[0].State, got[0])
}
if got[0].Freshness != recovery.ProtectionFreshnessStale {
t.Fatalf("freshness = %q, want stale", got[0].Freshness)
}
if got[0].Explanation == "incorrect stored posture" {
t.Fatal("requested posture trusted stale materialized JSON")
}
}
func TestStoreProtectionPostureBatchUsesIndexedBoundedReads(t *testing.T) {
t.Parallel()
store, err := Open(filepath.Join(t.TempDir(), "recovery.db"))
if err != nil {
t.Fatalf("Open() error = %v", err)
}
defer store.Close()
rows, err := store.db.Query(`
EXPLAIN QUERY PLAN
SELECT subject_key, subject_resource_id
FROM protection_postures
WHERE subject_resource_id IN (?, ?, ?)
`, "resource:a", "resource:b", "resource:c")
if err != nil {
t.Fatalf("EXPLAIN QUERY PLAN error = %v", err)
}
defer rows.Close()
var details []string
for rows.Next() {
var id, parent, notUsed int
var detail string
if err := rows.Scan(&id, &parent, &notUsed, &detail); err != nil {
t.Fatalf("scan query plan: %v", err)
}
details = append(details, detail)
}
if err := rows.Err(); err != nil {
t.Fatalf("query plan rows: %v", err)
}
plan := strings.Join(details, "\n")
if !strings.Contains(plan, "SEARCH protection_postures") ||
!strings.Contains(plan, "subject_resource_id=?") {
t.Fatalf("posture batch query is not index-backed:\n%s", plan)
}
}
func TestStoreProtectionEvidenceRetentionRefreshesMaterializedPosture(t *testing.T) {
t.Parallel()
store, err := Open(filepath.Join(t.TempDir(), "recovery.db"))
if err != nil {
t.Fatalf("Open() error = %v", err)
}
defer store.Close()
store.retention = time.Hour
store.lastPrune = time.Now().UTC()
now := time.Now().UTC().Truncate(time.Millisecond)
old := now.Add(-2 * time.Hour)
point := recovery.RecoveryPoint{
ID: "pbs-backup:expired",
Provider: recovery.ProviderProxmoxPBS,
Kind: recovery.KindBackup,
Mode: recovery.ModeRemote,
Outcome: recovery.OutcomeSuccess,
CompletedAt: &old,
SubjectResourceID: "resource:expired",
ProviderScope: "pbs-main",
}
observation, err := recovery.NewProtectionProviderObservation(
recovery.ProviderProxmoxPBS,
"pbs-backup-enumeration",
"pbs-main",
recovery.OutcomeSuccess,
recovery.ProtectionHistoryComplete,
operationaltrust.EvidencePermissionsSufficient,
false,
old,
old,
nil,
)
if err != nil {
t.Fatalf("NewProtectionProviderObservation() error = %v", err)
}
if err := store.UpsertProtectionProviderObservations(
context.Background(),
[]recovery.ProtectionProviderObservation{observation},
); err != nil {
t.Fatalf("UpsertProtectionProviderObservations() error = %v", err)
}
if err := store.UpsertPoints(context.Background(), []recovery.RecoveryPoint{point}); err != nil {
t.Fatalf("UpsertPoints() error = %v", err)
}
store.lastPrune = time.Time{}
store.maybePrune(context.Background())
for _, table := range []string{
"recovery_points",
"protection_provider_observations",
"protection_postures",
} {
var count int
if err := store.db.QueryRow("SELECT COUNT(*) FROM " + table).Scan(&count); err != nil {
t.Fatalf("count %s: %v", table, err)
}
if count != 0 {
t.Fatalf("%s count = %d, want 0 after retention prune", table, count)
}
}
}
func TestStoreProtectionMetadataBackfillAddsScopeAndTypedEvidence(t *testing.T) {
t.Parallel()
store, err := Open(filepath.Join(t.TempDir(), "recovery.db"))
if err != nil {
t.Fatalf("Open() error = %v", err)
}
defer store.Close()
observedAt := time.Now().UTC().Add(-time.Hour).Truncate(time.Millisecond)
updatedAt := observedAt.Add(time.Minute)
if _, err := store.db.Exec(`
INSERT INTO recovery_points (
id, provider, kind, mode, outcome,
completed_at_ms, subject_key, subject_resource_id,
repository_ref_json, created_at_ms, updated_at_ms
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`,
"pbs-backup:legacy",
string(recovery.ProviderProxmoxPBS),
string(recovery.KindBackup),
string(recovery.ModeRemote),
string(recovery.OutcomeSuccess),
observedAt.UnixMilli(),
"res:resource:legacy",
"resource:legacy",
`{"type":"proxmox-pbs-datastore","namespace":"pbs-main","name":"store-a"}`,
observedAt.UnixMilli(),
updatedAt.UnixMilli(),
); err != nil {
t.Fatalf("insert legacy recovery point: %v", err)
}
if err := store.BackfillProtectionMetadata(context.Background()); err != nil {
t.Fatalf("BackfillProtectionMetadata() error = %v", err)
}
var providerScope, evidenceID, evidenceRaw string
if err := store.db.QueryRow(`
SELECT provider_scope, evidence_id, evidence_json
FROM recovery_points
WHERE id = 'pbs-backup:legacy'
`).Scan(&providerScope, &evidenceID, &evidenceRaw); err != nil {
t.Fatalf("read migrated recovery point: %v", err)
}
if providerScope != "pbs-main" {
t.Fatalf("provider_scope = %q, want pbs-main", providerScope)
}
var evidence operationaltrust.EvidenceEnvelope
if err := json.Unmarshal([]byte(evidenceRaw), &evidence); err != nil {
t.Fatalf("decode evidence: %v", err)
}
if evidenceID == "" || evidence.ID != evidenceID {
t.Fatalf("evidence id mismatch column=%q payload=%q", evidenceID, evidence.ID)
}
if evidence.Source.Collector != "recovery-point-migration" {
t.Fatalf("collector = %q, want recovery-point-migration", evidence.Source.Collector)
}
if !evidence.ObservedAt.Equal(observedAt) {
t.Fatalf("observedAt = %s, want %s", evidence.ObservedAt, observedAt)
}
if !evidence.IngestedAt.Equal(updatedAt) {
t.Fatalf("ingestedAt = %s, want %s", evidence.IngestedAt, updatedAt)
}
}
internal/recovery/types.go
+52
View File
@@ -60,3 +60,55 @@ const (
VerifyIntentStale VerifyIntent = model.VerifyIntentStale
VerifyIntentUnknown VerifyIntent = model.VerifyIntentUnknown
)
type ProtectionState = model.ProtectionState
const (
ProtectionStateProtected ProtectionState = model.ProtectionStateProtected
ProtectionStateAttention ProtectionState = model.ProtectionStateAttention
ProtectionStateUnprotected ProtectionState = model.ProtectionStateUnprotected
ProtectionStateUnknown ProtectionState = model.ProtectionStateUnknown
)
type ProtectionFreshness = model.ProtectionFreshness
const (
ProtectionFreshnessCurrent ProtectionFreshness = model.ProtectionFreshnessCurrent
ProtectionFreshnessStale ProtectionFreshness = model.ProtectionFreshnessStale
ProtectionFreshnessUnknown ProtectionFreshness = model.ProtectionFreshnessUnknown
)
type ProtectionVerification = model.ProtectionVerification
const (
ProtectionVerificationVerified ProtectionVerification = model.ProtectionVerificationVerified
ProtectionVerificationUnverified ProtectionVerification = model.ProtectionVerificationUnverified
ProtectionVerificationStale ProtectionVerification = model.ProtectionVerificationStale
ProtectionVerificationUnknown ProtectionVerification = model.ProtectionVerificationUnknown
)
type ProtectionCoverage = model.ProtectionCoverage
const (
ProtectionCoverageComplete ProtectionCoverage = model.ProtectionCoverageComplete
ProtectionCoveragePartial ProtectionCoverage = model.ProtectionCoveragePartial
ProtectionCoverageNone ProtectionCoverage = model.ProtectionCoverageNone
ProtectionCoverageUnknown ProtectionCoverage = model.ProtectionCoverageUnknown
)
type ProtectionHistoryCompleteness = model.ProtectionHistoryCompleteness
const (
ProtectionHistoryComplete ProtectionHistoryCompleteness = model.ProtectionHistoryComplete
ProtectionHistoryPartial ProtectionHistoryCompleteness = model.ProtectionHistoryPartial
ProtectionHistoryUnavailable ProtectionHistoryCompleteness = model.ProtectionHistoryUnavailable
ProtectionHistoryUnknown ProtectionHistoryCompleteness = model.ProtectionHistoryUnknown
)
type ProtectionProviderState = model.ProtectionProviderState
type ProtectionPosture = model.ProtectionPosture
type ProtectionProviderObservation = model.ProtectionProviderObservation
type ProtectionPosturePolicy = model.ProtectionPosturePolicy
type ProtectionPosturePolicyPayload = model.ProtectionPosturePolicyPayload
type ProtectionPostureQuery = model.ProtectionPostureQuery
type ProtectionProviderSummary = model.ProtectionProviderSummary
scripts/release_control/canonical_completion_guard_test.py
+1
View File
@@ -230,6 +230,7 @@ class CanonicalCompletionGuardTest(unittest.TestCase):
"proxmox-backup-identity-monitoring",
"container-entrypoint-runtime",
"mock-runtime-fixtures",
"pbs-protection-evidence-runtime",
"monitoring-runtime",
],
)
tests/integration/tests/90-operational-trust-protection-posture.spec.ts
+81
View File
@@ -0,0 +1,81 @@
import { expect, test } from "@playwright/test";
import { ensureAuthenticated } from "./helpers";
const DESKTOP_VIEWPORT = { width: 1440, height: 900 };
test.describe("Operational trust protection posture", () => {
test.setTimeout(180_000);
test("uses one bounded posture query and keeps evidence in the workload drill-down", async ({
page,
}, testInfo) => {
test.skip(
testInfo.project.name.startsWith("mobile-"),
"The desktop journey proves the full evidence drill-down",
);
const postureRequests: string[] = [];
page.on("request", (request) => {
if (request.url().includes("/api/recovery/postures")) {
postureRequests.push(request.url());
}
});
await page.setViewportSize(DESKTOP_VIEWPORT);
await ensureAuthenticated(page);
await page.goto("/proxmox/backups", { waitUntil: "domcontentloaded" });
await page.getByRole("button", { name: "Coverage" }).click();
const coverageTable = page
.locator("div.overflow-x-auto")
.filter({ has: page.locator('th:has-text("Posture")') })
.first();
await expect(coverageTable).toBeVisible();
await expect.poll(() => postureRequests.length).toBe(1);
const postureURL = new URL(postureRequests[0]);
const requestedResourceIDs = postureURL.searchParams.getAll("resourceId");
expect(requestedResourceIDs.length).toBeGreaterThan(0);
expect(requestedResourceIDs.length).toBeLessThanOrEqual(200);
// The default monitor stays compact. Explanation and provider/restore
// evidence appear only after the operator asks for the row's details.
await expect(coverageTable.getByText("Provider evidence")).toHaveCount(0);
const detailToggle = coverageTable
.locator('button[aria-label^="Expand details for"]')
.first();
await expect(detailToggle).toBeVisible();
await detailToggle.click();
const detailRow = coverageTable.locator("[data-inline-detail-for]").first();
await expect(detailRow).toBeVisible();
await expect(detailRow).toContainText(
/Protected:|Attention:|Unprotected:|Unknown:/,
);
await expect(detailRow).toContainText(
/Restore evidence|No restore evidence has been discovered/,
);
});
test("contains wide evidence tables on a phone-sized viewport", async ({
page,
}, testInfo) => {
test.skip(
!testInfo.project.name.startsWith("mobile-"),
"The mobile projects own responsive containment",
);
await ensureAuthenticated(page);
await page.goto("/proxmox/backups", { waitUntil: "domcontentloaded" });
await page.getByRole("button", { name: "Coverage" }).click();
await expect(page.locator('th:has-text("Posture")').first()).toBeVisible();
const viewport = await page.evaluate(() => ({
bodyScrollWidth: document.body.scrollWidth,
clientWidth: document.documentElement.clientWidth,
}));
expect(viewport.bodyScrollWidth).toBeLessThanOrEqual(
viewport.clientWidth + 1,
);
});
});