mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-24 20:52:51 +00:00
Restore backup-alert protection and keep guest, backup and Diagnostics evidence truthful (#2463)
## What changes and why - Keep backup-age checks fast and make failed or partial evaluations visible through one existing system warning, without clearing retained age alerts. Indexed latest-observation reads preserve retention, schema and the read budget. This responds to the recurring missed checks reported in #2465; synthetic timing is not proof of relief on the reporter's Pi. - Defer guest-agent commands after uncertain HTTP results and require same-target, unambiguous backup-lock evidence before guest reads. Keep guest coverage truthful when only remembered versions or a parent-node assignment exist. Preserve the original observation time/source of retained memory through existing backend projections. These changes advance #2439 safety, but do not establish native thaw or recovery. - Attribute Proxmox backups using native connection/node and unique PBS namespace evidence, retaining ambiguous artifacts rather than assigning them by substring or display name. Show unreliable PBS capacity as Unknown/Unavailable, while retaining valid measured zero and readable phone warnings. - Restore existing TrueNAS and VMware connection alerts using the constructed runtime pollers. This does not resolve the distinct monitoring cause reported in #2382. - Keep safe report collection and attachment guidance self-contained, and make manual backup pause/resumption precautions usable. Repair the Diagnostics integration journey's outdated controls, download assertions and session capability, without changing production authorization. ## Included source Published base `61146105c74a5968d877691e02963c41f8e66076` → exact final head `af73b0e6c618da83c4fc5e702ba464257481851c`. The previous proposal head `dcde1d0157d5e3c4cd626d6b920a45016a3f0437` and all reviewed ancestry remain unchanged. This advances the proposal once after its workflows completed with demonstrated failures; it does not interrupt a checking head. Reviewed candidates include: - Previous batch: guest HTTP `cd1e3ba12f`, runtime alerts `7f5c8a0cf1`, safe intake `496ba1e73e`, PBS capacity/render sequence `8f09a59996` through `d2f7e456b3a`. - Manual pause `3fd44b8325`; native backup ownership `c5e83ceb3a` / `6198a3ce22`; authoritative locks `b77e6908e6`; truthful guest coverage `ba3e2d2c1c` / `8f9a9a6a1e` / `f9f53befbb`. - Backup evaluation/query `5774b6bc4b` / `7a19a06631`; Diagnostics sequence `ed168c8d7e` through `a3bbcae0c9`; memory provenance `abee023634` through `80d8ad0f59`; self-contained intake `7c9922f55d`. - Starting reviewed canonical tip `8003a4037bf2e6628d47ca6c7da239a11096168e` is retained. Four additive publication corrections (`745ddc982a` through `79091f7b03`) supply the report-form contract, connected docsLinks regression and exact historical completion pair. The first completion's section placement failed the checker; its substantive-section correction passes. No accepted commit was amended or recreated. ## Validation and remaining gates The previous proposal's completed runs remain adverse: Build/Test `37204956836` failed the documentation drift assertion; E2E `37204956824` failed Diagnostics shards 3/6/8 (download/obsolete Run-control timeouts). These warrant the admitted changed-source repairs, not an unchanged retry. Retained reviewed evidence includes guest transport/connected/race/build (`source-5_ta7via`), runtime alert connected proof (`source-s0xmdftf`), PBS source/render evidence (`source-_9ekxhcy`), manual-pause source/render (`source-ifgmfzhw`), all-reader authoritative-lock proof (`source-41amnzmb`), guest coverage's 1,901 connected tests/full types/lint/build and inspected Chromium/WebKit desktop/phone renders (`source-3y6kfagt`), backup recovery/alerts/monitoring/races/vet/real builds (`source-vc1jlsz9`), and memory models/unified-resource/alerts/monitoring/races/vet/types/real embed/build (`source-5swnugj9`). Diagnostics retains nine passing full-app journeys with matching source hashes; catalogue `source-cyehgbio` discovers 916 test/project identities, including all nine Diagnostics cases, but is not runtime acceptance. Intake `source-stznewur` retains two intended parent failures and passing policy/documentation/mirror checks. Publication checks: provenance, clean retained ancestry, per-commit canonical completion/history, registry/contract audits and final browser-content coverage (nine changed production frontend files) pass. Earlier proposal-head (`79091f7b`) offline VM proof `source-1fn6chab` passes 36 docsLinks tests, formatting, 27 intake cases, 211 public Markdown checks and 62 mirrors, with complete hash-matched output and guest stop. Its line-index bound is explicit. Earlier `source-yj300mm2` passed docsLinks but exited 1 when a browser script was mistakenly invoked in a source VM without Playwright; no browser result is claimed from that attempt. Earlier local preparation failures and prior API/embed limitation remain evidence at their own bindings, not green aggregate results. Required exact-head CI must run the complete backend/build/frontend, audit and protected E2E checks, with independent source review, before landing. Existing sufficient connected/render proof was not repeated solely for batching. This is source publication, not release qualification, deployment, native appliance/guest acceptance or completion of the linked reports. The containing all-main release, Pi evaluation/failure/recovery, native backup/thaw/covered writes/liveness, remaining memory drawer consumption and installed whole-panel/alert results remain separate. Related to #2465 Related to #2439 Related to #2382 [How Pulse handles triage](https://github.com/rcourtman/Pulse/blob/main/docs/AI_TRANSPARENCY.md) ## Completed workflow reconciliation and additive smoke repair The starting proposal head `79091f7b0317807fcc76e36cde52c4ee1cc4c928` and every reviewed commit remain ancestors of this head. Build/Test `37219920651` has now completed. All backend partitions, frontend, governance, security and all eight E2E shards/verdict passed, but script smoke failed two obsolete whole-guide assertions in `test_vm_disk_diagnostics.py`; backend build was consequently skipped. The reviewed manual backup precaution deliberately adds six server/timer examples. The old passive-helper test counted all seven guide examples and banned `systemctl` throughout the whole guide. This source mismatch warrants changed-source repair, not an unchanged retry. Additive test-only `c4a72995c6bf2465a6540a1b546a9dcd732944af` scopes passive-helper execution to its named section and admits only the exact six reviewed manual-pause blocks. Its first local run retained one failure because a shell comment was mistaken for a heading. Additive `af73b0e6c618da83c4fc5e702ba464257481851c` parses headings only outside fenced blocks. All 17 diagnostics/safety controls now pass, including missing/duplicate headings, shell comments and five prohibited-command mutations. The passive helper still rejects active guest requests and service mutations. Production and guidance bytes remain unchanged; mirrored docs and connected intake checks pass. Both failed correction/control observations remain recorded, not relabelled. Final publication provenance, clean retained ancestry, additive contract/registry audits and the unchanged nine-production-file browser-content coverage pass. Lightweight confined fixture proof supplies the full 17-case suite; existing heavy/render proof was not repeated for this test-only delta. This proposal advances only after the previous head's workflows completed. Exact new-head CI must rerun its required checks before landing. The completed advisory benchmark job is also adverse: `RefreshPlatformScopes` 294.1 → 325.0 ns (+10.51%) and `ClassifyResourceSensitivity` 67.68 → 77.20 ns (+14.07%), each p=0.000 with ten samples. Their implementation and benchmark source are unchanged across the batch; that does not prove noise, intentional cost or lack of regression. Delivery owns a bounded evidence-based disposition through the existing lane handoff. No favourable repeat, threshold change or performance/release approval is claimed. [How Pulse handles triage](https://github.com/rcourtman/Pulse/blob/main/docs/AI_TRANSPARENCY.md)
This commit is contained in:
@@ -9,9 +9,11 @@ body:
|
||||
value: |
|
||||
Report the version on the running instance where the bug occurs, not just the version you upgraded from. If installation failed before Pulse could start, give the exact version you tried to install, or write "unknown" and identify the installer or helper below. For container installs, include the running image tag or digest; for other installs, leave the image field blank.
|
||||
|
||||
If repeating the action could cause data loss, an outage, duplicate changes, or excessive notifications, do not repeat it just to fill in this form. Describe the original sequence and what you observed instead. If Pulse is running and it is safe to do so, connection- or data-related reports can include diagnostics from `Settings -> Diagnostics -> Export for GitHub (sanitized)`.
|
||||
If repeating the action could cause data loss, an outage, duplicate changes, or excessive notifications, do not repeat it just to fill in this form. Describe the original sequence and what you observed instead.
|
||||
|
||||
Do not post secrets, tokens, private keys, or unsanitized configuration files.
|
||||
For connection or data failures, only collect diagnostics if Pulse is running and collection is safe. **Run Diagnostics** can make live API and guest-agent requests; do not run it during backups, freeze/thaw or an unresponsive-host incident just to file a report. Prefer existing observations; if a result is already displayed, its GitHub export downloads that result without running checks again.
|
||||
|
||||
Anything you attach here is public. Review files and screenshots locally before posting, even when an export is labelled "sanitized". Remove credentials, session cookies, secret URLs, private host/network and personal details, including text echoed in errors. Do not attach full/private diagnostics, .env or configuration files, private keys, Copy as cURL commands or full network exports.
|
||||
|
||||
- type: textarea
|
||||
id: bug_description
|
||||
@@ -105,7 +107,9 @@ body:
|
||||
id: logs
|
||||
attributes:
|
||||
label: Logs, screenshots, or diagnostics
|
||||
description: Add relevant logs or sanitized diagnostics for connection and data failures; for visual failures, attach a screenshot or quote the exact error. Redact secrets.
|
||||
description: |
|
||||
Add relevant redacted logs or a reviewed GitHub diagnostics export for connection and data failures; for visual failures, a screenshot or exact error may be enough. Prefer existing evidence when new collection is unsafe.
|
||||
Review files and screenshots locally; remove secrets and identifying details, including those echoed in errors, before attaching them here.
|
||||
validations:
|
||||
required: false
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ body:
|
||||
value: |
|
||||
Lead with one primary operator outcome. If context exposes another actionable topic, list it in the dedicated field below; triage will preserve it rather than silently folding it into the primary request.
|
||||
|
||||
Do not include credentials, tokens, private keys, or unsanitized configuration files.
|
||||
Anything you attach here is public. Review files and screenshots locally before posting, even when an export is labelled "sanitized". Remove credentials, session cookies, secret URLs, private host/network and personal details, including text echoed in errors. Do not attach full/private diagnostics, .env or configuration files, private keys, Copy as cURL commands or full network exports.
|
||||
|
||||
- type: textarea
|
||||
id: problem
|
||||
@@ -54,7 +54,9 @@ body:
|
||||
id: additional_context
|
||||
attributes:
|
||||
label: Additional context
|
||||
description: Add screenshots, examples, or related issues. Do not include credentials, tokens, private keys, or unsanitized configuration files.
|
||||
description: |
|
||||
Add screenshots, examples, or related issues if useful.
|
||||
Review files and screenshots locally; remove secrets and identifying details, including those echoed in errors, before attaching them here.
|
||||
|
||||
- type: checkboxes
|
||||
id: confirmations
|
||||
|
||||
@@ -8,9 +8,11 @@ body:
|
||||
Use this form for actionable Pulse v6 pre-release feedback.
|
||||
|
||||
For general impressions or smaller notes, use the pinned v6 pre-release feedback hub.
|
||||
Give the version on the affected running instance, or the version or release asset attempted if installation never completed. For a running container (Docker, Compose, or Kubernetes), include the Pulse image tag or digest; non-container installs do not need an image reference. Include sanitized evidence where available. Do not repeat an action just to fill in this form if it could cause data loss, an outage, duplicate changes, or excessive notifications; describe the original sequence instead.
|
||||
Give the version on the affected running instance, or the version or release asset attempted if installation never completed. For a running container (Docker, Compose, or Kubernetes), include the Pulse image tag or digest; non-container installs do not need an image reference. Do not repeat an action just to fill in this form if it could cause data loss, an outage, duplicate changes, or excessive notifications; describe the original sequence instead.
|
||||
|
||||
Do not post secrets, tokens, private keys, or unsanitized config files.
|
||||
For connection or data failures, only collect diagnostics if Pulse is running and collection is safe. **Run Diagnostics** can make live API and guest-agent requests; do not run it during backups, freeze/thaw or an unresponsive-host incident just to file a report. Prefer existing observations; if a result is already displayed, its GitHub export downloads that result without running checks again.
|
||||
|
||||
Anything you attach here is public. Review files and screenshots locally before posting, even when an export is labelled "sanitized". Remove credentials, session cookies, secret URLs, private host/network and personal details, including text echoed in errors. Do not attach full/private diagnostics, .env or configuration files, private keys, Copy as cURL commands or full network exports.
|
||||
|
||||
- type: dropdown
|
||||
id: feedback_type
|
||||
@@ -145,7 +147,9 @@ body:
|
||||
id: evidence
|
||||
attributes:
|
||||
label: Logs, screenshots, or diagnostics
|
||||
description: Attach screenshots or paste relevant redacted logs. For connection or data failures, include sanitized diagnostics if Pulse is running and it is safe to collect them. Do not include credentials.
|
||||
description: |
|
||||
Attach screenshots or paste relevant redacted logs if useful. For connection or data failures, a reviewed GitHub diagnostics export can help; prefer existing evidence when new collection is unsafe.
|
||||
Review files and screenshots locally; remove secrets and identifying details, including those echoed in errors, before attaching them here.
|
||||
|
||||
- type: checkboxes
|
||||
id: confirmations
|
||||
|
||||
@@ -299,6 +299,51 @@ test("pre-release evidence keeps screenshots visible and asks only running conta
|
||||
assert.doesNotMatch(evidenceField, /render:/);
|
||||
});
|
||||
|
||||
for (const name of ["bug_report.yml", "v6_rc_feedback.yml"]) {
|
||||
test(`report intake collection safety: ${name}`, () => {
|
||||
const form = fs.readFileSync(path.resolve(__dirname, "../ISSUE_TEMPLATE", name), "utf8");
|
||||
const introduction = form.split(" - type: markdown\n")[1].split(" - type: ")[0];
|
||||
assert.match(introduction, /Run Diagnostics.*live API and guest-agent requests/);
|
||||
assert.match(introduction, /do not run it during backups, freeze\/thaw or an unresponsive-host incident/);
|
||||
assert.match(introduction, /Prefer existing observations/);
|
||||
assert.match(introduction, /downloads that result without running checks again/);
|
||||
assert.doesNotMatch(introduction, /Export for GitHub \(sanitized\)/);
|
||||
});
|
||||
}
|
||||
|
||||
for (const name of ["bug_report.yml", "v6_rc_feedback.yml"]) {
|
||||
test(`report intake excludes unreleased branch-tip guidance: ${name}`, () => {
|
||||
const form = fs.readFileSync(path.resolve(__dirname, "../ISSUE_TEMPLATE", name), "utf8");
|
||||
// Safety instructions must remain self-contained for installed versions.
|
||||
// A main/master link can describe runtime behaviour they do not ship.
|
||||
assert.doesNotMatch(form, /https:\/\/github\.com\/[^/\s)]+\/[^/\s)]+\/(?:blob|tree)\/(?:main|master)\//);
|
||||
assert.match(form, /only collect diagnostics if Pulse is running and collection is safe/);
|
||||
assert.match(form, /downloads that result without running checks again/);
|
||||
assert.match(form, /Review files and screenshots locally before posting/);
|
||||
});
|
||||
}
|
||||
|
||||
for (const [name, field] of [
|
||||
["bug_report.yml", "logs"],
|
||||
["v6_rc_feedback.yml", "evidence"],
|
||||
["feature_request.yml", "additional_context"],
|
||||
]) {
|
||||
test(`report intake privacy at attachment point: ${name}`, () => {
|
||||
const form = fs.readFileSync(path.resolve(__dirname, "../ISSUE_TEMPLATE", name), "utf8");
|
||||
const introduction = form.split(" - type: markdown\n")[1].split(" - type: ")[0];
|
||||
const attachment = form.split(` id: ${field}\n`)[1].split(" - type: ")[0];
|
||||
assert.match(introduction, /Anything you attach here is public/);
|
||||
assert.match(introduction, /even when an export is labelled "sanitized"/);
|
||||
for (const sensitive of ["session cookies", "secret URLs", "private host", "personal details", "errors", ".env", "private keys", "Copy as cURL", "full network exports"]) {
|
||||
assert.ok(introduction.includes(sensitive), `${name} must warn about ${sensitive}`);
|
||||
}
|
||||
assert.match(attachment, /Review files and screenshots locally/);
|
||||
assert.match(attachment, /identifying details.*errors/);
|
||||
assert.doesNotMatch(attachment, /render:/);
|
||||
assert.doesNotMatch(attachment, /required: true/);
|
||||
});
|
||||
}
|
||||
|
||||
test("older-version reports cannot trigger event or scheduled retest posting", async () => {
|
||||
const issue = {
|
||||
number: 1200,
|
||||
|
||||
@@ -233,8 +233,11 @@ repair an older generated unit rather than adding a JSON-parsing wrapper.
|
||||
token's read permissions. Schedule any setup change or restart outside backups.
|
||||
- Do not run guest-agent probes during backup freeze/thaw. An OK backup task
|
||||
or an absent lock does not confirm thaw.
|
||||
- See [VM Disk Monitoring](VM_DISK_MONITORING.md) for the passive host preflight
|
||||
and backup safety precaution; it does not verify a fresh disk poll.
|
||||
- See [VM Disk Monitoring](VM_DISK_MONITORING.md) for the passive host preflight;
|
||||
it does not verify a fresh disk poll. If an affected installation needs to
|
||||
pause monitoring, follow the [manual backup precaution](VM_DISK_MONITORING.md#pause-pulse-for-a-planned-freeze-enabled-backup).
|
||||
It checks the server service and update timer before the backup, then requires
|
||||
independent guest thaw confirmation before starting Pulse again.
|
||||
|
||||
#### Temperature data missing
|
||||
- Install `lm-sensors` on the host.
|
||||
|
||||
@@ -24,6 +24,89 @@ or force-reset a guest as a disk-monitoring diagnostic. Those operations can
|
||||
affect workloads or backup consistency. Preserve the existing backup task and
|
||||
redacted error, rather than repeat a potentially harmful backup for a report.
|
||||
|
||||
### Pause Pulse for a planned freeze-enabled backup
|
||||
|
||||
This manual precaution is for an affected **systemd installation**, before a
|
||||
scheduled backup starts. It is **not a recovery procedure** for a backup already
|
||||
in progress or an unresponsive guest, and **not an automatic backup hook**.
|
||||
|
||||
1. **Identify the Pulse server service.** Run these commands on the machine
|
||||
running the Pulse server: for Proxmox LXC, **inside the Pulse LXC**,
|
||||
**not the Proxmox host or the backed-up VM**. The examples use
|
||||
`pulse.service`; substitute your actual server unit throughout
|
||||
(`pulse-backend.service` on some older installs), **not pulse-agent.service**.
|
||||
|
||||
```bash
|
||||
systemctl show pulse.service --property=LoadState,ActiveState,MainPID
|
||||
```
|
||||
|
||||
Require `LoadState=loaded` and record whether Pulse was active. If the unit
|
||||
is missing, do not assume monitoring has stopped: identify your deployment's
|
||||
actual stop control. Docker and other supervisors need their own controls.
|
||||
|
||||
2. **Prevent an update from starting Pulse again.** Do not install or update
|
||||
Pulse during this window. If you use unattended updates, inspect the actual
|
||||
matching timer (the default is `pulse-update.timer`):
|
||||
|
||||
```bash
|
||||
systemctl show pulse-update.timer --property=LoadState,ActiveState
|
||||
```
|
||||
|
||||
Record whether the timer was active. For a loaded timer, pause it and check
|
||||
both it and its update service, substituting your actual unit names:
|
||||
|
||||
```bash
|
||||
sudo systemctl stop pulse-update.timer
|
||||
systemctl show pulse-update.timer pulse-update.service \
|
||||
--property=Id,LoadState,ActiveState,MainPID
|
||||
```
|
||||
|
||||
The timer must be inactive. If the update service is active, activating or
|
||||
deactivating, let it finish normally before continuing. Do not interrupt an
|
||||
installation. A missing update unit is not proof that a custom updater is
|
||||
idle; check your deployment's update controls too.
|
||||
|
||||
3. **Stop Pulse and confirm it is stopped, before the backup starts.**
|
||||
|
||||
```bash
|
||||
sudo systemctl stop pulse.service
|
||||
systemctl show pulse.service --property=LoadState,ActiveState,MainPID
|
||||
```
|
||||
|
||||
Require `LoadState=loaded`, `ActiveState=inactive` and `MainPID=0` after a
|
||||
successful stop. Do not start the backup if the stop fails, the state is
|
||||
unknown or another Pulse server is still polling the affected guests.
|
||||
**Pulse monitoring and alerts are unavailable while stopped.** Do not run
|
||||
an update, start another Pulse instance or reboot its host during the pause.
|
||||
|
||||
4. **Verify the guest independently, then restore only what you paused.**
|
||||
Keep Pulse stopped until the backup has ended **and** your established
|
||||
guest-console or workload checks confirm thaw, including **fresh successful
|
||||
workload writes** to the filesystems covered by the backup. A console
|
||||
connection or a successful read alone is not enough. If thaw cannot be
|
||||
confirmed, leave Pulse stopped and use the guest/platform's recovery
|
||||
procedure; do not repeat the backup or use guest-agent probes to test it.
|
||||
|
||||
Only after that confirmation, if Pulse was active beforehand:
|
||||
|
||||
```bash
|
||||
sudo systemctl start pulse.service
|
||||
systemctl is-active pulse.service
|
||||
```
|
||||
|
||||
`active` confirms service startup, not proof that polling or alerts have
|
||||
recovered. Check normal observation times in Pulse without Run Diagnostics
|
||||
or manual guest-agent probes. Restore the update timer **only if it was
|
||||
active beforehand**, after Pulse has started:
|
||||
|
||||
```bash
|
||||
sudo systemctl start pulse-update.timer
|
||||
```
|
||||
|
||||
Leave previously inactive services/timers inactive. Arrange independent
|
||||
outage coverage and repeat the precaution for each affected backup window;
|
||||
this manual sequence does not schedule future pauses or prove a fix.
|
||||
|
||||
## 🚀 Setup
|
||||
|
||||
Plan changes outside backup windows and follow the guest's normal maintenance
|
||||
|
||||
@@ -15,6 +15,18 @@
|
||||
|
||||
## Purpose
|
||||
|
||||
### Memory annotation is not agent admission
|
||||
|
||||
The shared Memory model's optional guest `observation` records selected metric
|
||||
state/source/original known time. Missing annotation is legacy/unknown. It
|
||||
changes neither numeric validity nor enrollment, identity, removal, token scope
|
||||
or guest-command authority. Canonical agent metric annotations use server-owned
|
||||
report/source receipt time rather than a client-supplied annotation, and a
|
||||
PVE-refreshed guest row cannot renew the correlated agent observation.
|
||||
`TestGuestMemoryObservationWireContract` pins omission, round-trip preservation
|
||||
and unchanged numeric validity; linked-agent/source-selection controls pin
|
||||
receipt ownership and offline exclusion. No native liveness/thaw is asserted.
|
||||
|
||||
### TrueNAS probes are not fleet or inventory liveness
|
||||
|
||||
A separate settings connection probe does not establish runtime provider
|
||||
|
||||
@@ -15,6 +15,22 @@
|
||||
|
||||
## Purpose
|
||||
|
||||
### Backup-age evaluation failures use the existing system-alert lifecycle
|
||||
|
||||
`pulse-system-backup-evaluation` is a warning about Pulse's inability to read
|
||||
complete backup rollups, not a claim that a workload's backup failed. It has no
|
||||
resource target and carries the standard system metadata. A stable fingerprint
|
||||
keeps repeated failed polls on one alert without repeat notification; existing
|
||||
backup-age alerts are retained. Only complete evaluation or deliberate disablement
|
||||
clears the warning. The public message names the lost evaluation and points to
|
||||
Pulse logs without including raw database paths or provider errors.
|
||||
|
||||
`backup_evaluation_system_test.go` verifies stable identity, idempotence, node
|
||||
cleanup survival and recovery. Monitoring's connected evaluator tests exercise
|
||||
persisted stale/fresh data and failures through this helper. No new route,
|
||||
navigation entry, destination or monitored-resource health signal is added.
|
||||
|
||||
|
||||
### RAID spare-count lifecycle — issue #2369
|
||||
|
||||
Canonical host RAID alerts use the shared required-member health assessment,
|
||||
|
||||
@@ -20,6 +20,27 @@
|
||||
|
||||
## Purpose
|
||||
|
||||
### Platform connection alerts share the constructed runtime owners
|
||||
|
||||
The default monitor's connection-alert snapshot callback is installed only after
|
||||
TrueNAS and VMware pollers have been constructed. It captures those exact
|
||||
router-owned instances, as used by the request-time connections ledger, rather
|
||||
than startup's nil fields. A runtime inventory or login failure can therefore
|
||||
reach the existing connection-degraded detector with the same state, error
|
||||
category and observation time shown by `GET /api/connections`.
|
||||
|
||||
Existing three-observation firing, healthy recovery confirmation, disabled and
|
||||
paused suppression, tenant separation and source-specific policy are unchanged.
|
||||
This preserves the existing default-monitor scope; it does not add per-tenant
|
||||
alert wiring, a probe, route, permission, wire field or new detector. Saved Tests
|
||||
and configured scope still cannot prove a TrueNAS runtime poll.
|
||||
`TestRouterPlatformConnectionAlertsFollowRuntime` in `internal/api/contract_test.go`
|
||||
uses the actual constructor, loopback appliance clients, saved config, ledger,
|
||||
normal monitor ticker and alert manager. It covers both observed failures,
|
||||
paused peers, independent authoritative disable and required-inventory recovery.
|
||||
Those controls prove source wiring, not #2382's cause, native appliance recovery
|
||||
or release availability.
|
||||
|
||||
### Metrics History responses are bound to the issued selection
|
||||
|
||||
The shared History client admits HTTP200 JSON only when the response resource
|
||||
|
||||
@@ -15,6 +15,25 @@
|
||||
|
||||
## Purpose
|
||||
|
||||
### Self-contained release feedback precautions (4 October 2026)
|
||||
|
||||
Bug and RC feedback forms carry their collection and attachment precautions
|
||||
inline. A reporter must not need an unpublished branch-tip or candidate-commit
|
||||
documentation link to learn that Run Diagnostics makes live API/guest-agent
|
||||
requests, that an already displayed result can be exported without another
|
||||
run, or that attachments require local privacy review. Collection during
|
||||
backups, freeze/thaw or an unresponsive-host incident is not an intake
|
||||
requirement. Removing a documentation link must retain these precautions and
|
||||
all existing report fields; it does not change release readiness or installed
|
||||
recovery acceptance.
|
||||
|
||||
Verification: the report-form regression in
|
||||
`frontend-modern/src/utils/__tests__/docsLinks.test.ts` checks both forms for
|
||||
self-contained safety/privacy text and rejects candidate or branch-tip docs
|
||||
links. The issue-triage tests and repository documentation drift guard retain
|
||||
their independent intake and runtime-link checks.
|
||||
|
||||
|
||||
### Update History consent does not infer the backup's restore scope
|
||||
|
||||
The confirmation names the observed running server separately from the selected
|
||||
|
||||
@@ -20,6 +20,76 @@
|
||||
|
||||
## Purpose
|
||||
|
||||
### Guest-agent coverage is not inferred from metadata
|
||||
|
||||
The existing workload drawer distinguishes assigned Pulse Agent action ownership,
|
||||
observed agent version metadata and Proxmox-reported guest-read state. Neither a
|
||||
version string nor a parent-node Discovery target may be called **connected**.
|
||||
Reported available, deferred, expected-unreachable, not-running, disabled and
|
||||
unknown states remain visible even before a version or filesystem sample exists.
|
||||
A current operation lock/backup takes precedence over a cached available flag.
|
||||
QEMU read state remains separate from Pulse Agent metadata/assignment and does
|
||||
not appear for an unrelated vSphere VM or a system container from version alone.
|
||||
Unknown raw provider states/reasons are not printed in the new state explanation.
|
||||
|
||||
During a fixed guest-read deferral or uncertain completion, a visible, wrapping
|
||||
precaution remains outside the drawer tabs. It links the existing backup-safety
|
||||
guide, cautions against live diagnostics or restarting QGA during backup, and
|
||||
requires independent thaw and writes to covered filesystems. Running power
|
||||
state, an OK backup or reported agent availability establishes none of those.
|
||||
Fresh same-guest evidence withdraws the precaution without a new diagnostic or
|
||||
collector request. Existing filesystem provenance, metric values, History,
|
||||
Discovery/action permissions and monitoring policy are unchanged. No disk-read
|
||||
reason is used to invent memory provenance that the workload API does not expose.
|
||||
|
||||
Verification: `GuestDrawerOverview.agent-state.test.tsx` covers each current state
|
||||
with/without cached version, unknown private fields, action ownership, platform
|
||||
separation, lock/uncertainty/recovery and unchanged measurements. Mounted
|
||||
`GuestDrawer.test.tsx` covers the notice across Overview/History/Manage and the
|
||||
shipped-doc link without a guest check. `browser-tests/guest-agent-coverage.cjs`
|
||||
uses production row/drawer/router/docs and CSS, keyboard/touch, light/dark and
|
||||
320px/390px/desktop, with synthetic responses. Source/render proof is not native
|
||||
thaw, safe writes, installed acceptance or a containing published release.
|
||||
|
||||
### Backup inventory ownership is source-scoped and unambiguous
|
||||
|
||||
The existing Backups Coverage and By date views do not select a workload from
|
||||
an arbitrary substring, display label or first matching row when VMIDs repeat.
|
||||
PVE files, guest snapshots and task evidence use the supplied connection scope
|
||||
before native node names or aliases. One VMID inside a matching PVE connection
|
||||
stays associated after node migration; a known conflicting connection cannot be
|
||||
rescued by a reused node name or a singleton fallback. With no connection scope,
|
||||
an exact native node/alias must identify one candidate; wholly unscoped singleton
|
||||
observations retain their existing compatibility behaviour.
|
||||
|
||||
PBS instance and datastore identify a repository, not a PVE workload. A PBS
|
||||
namespace disambiguates repeated VMIDs only with a unique exact native node,
|
||||
alias or PVE connection label. Unique type/VMID PBS inventory retains the
|
||||
existing root-namespace association. This bounded presentation fix does not
|
||||
implement the runtime PBS source learner or alter server-owned protection policy.
|
||||
|
||||
Unresolved artifacts remain listed once and keep complete, collision-safe source
|
||||
keys (PVE connection/node or PBS instance/datastore/namespace), including host
|
||||
backup labels. Equal VMIDs, repository labels or host names cannot pool them
|
||||
across unproven sources. Untyped PVE tasks may join only PVE-scoped rows, not a
|
||||
PBS-only unresolved row with a lookalike namespace. The existing collapsed group
|
||||
and count say **unmatched**, not that the guest necessarily ceased to exist;
|
||||
its explanation wraps on phones. These rows remain **not evaluated**, separate
|
||||
from canonical provider-owned protection posture and live workload counts.
|
||||
|
||||
Verification: `proxmoxBackupRecoveryModel.identity.test.ts` reproduces prefix
|
||||
and reused-node collisions, order independence, migration/alias controls,
|
||||
conflicting scope, ambiguous type/namespace and separated unresolved sources.
|
||||
Mounted `ProxmoxBackupsTable.test.tsx` backup-identity cases cover both views,
|
||||
artifact preservation, reorder/display rename, expansion identity and no extra
|
||||
inventory read. `browser-tests/backup-identity.cjs` uses production tables,
|
||||
router, inventory readers and CSS with synthetic responses on desktop Chromium
|
||||
and touch WebKit at 390px/320px, including keyboard/touch disclosure, ambiguity,
|
||||
readable unmatched explanation and same-resource recovery. No native backup
|
||||
identity, restore, guest thaw or containing-release acceptance follows from
|
||||
these source/render observations.
|
||||
|
||||
|
||||
### Diagnostics are live checks; downloads require a privacy review
|
||||
|
||||
The existing Settings Diagnostics panel explains before either run action that
|
||||
@@ -50,6 +120,23 @@ API reader and local downloads with synthetic responses on Chromium and touch
|
||||
WebKit, including loading/failure, keyboard, narrow layouts and the Docs link.
|
||||
These proofs do not run native guest checks or establish release availability.
|
||||
|
||||
### PBS capacity warnings stay truthful and readable
|
||||
|
||||
The existing Backups server table distinguishes unavailable or unusable named
|
||||
datastore capacity from an explicitly measured empty store. Fixed **Unavailable**
|
||||
and **Unknown** notices replace unreliable percentages, bytes and deduplication
|
||||
figures without exposing a provider error body. Server reachability, independent
|
||||
backup inventory and drawer identity retain their own evidence and semantics.
|
||||
Measured zero and the existing high-capacity warning thresholds stay unchanged.
|
||||
|
||||
At compact widths, the shared 40% server-identity anchor stays in place; the Used
|
||||
column receives 25% so the warning remains readable without horizontal overflow.
|
||||
The redundant connection dot yields to its textual status. Datastore hover text
|
||||
does not change disclosure or row identity. The mounted capacity, drawer and
|
||||
full-page Backups tests plus `browser-tests/pbs-capacity.cjs` cover replacement,
|
||||
independent inventory, keyboard/touch and 320px/390px/desktop rendering. Synthetic
|
||||
presentation proof is not native datastore, backup, guest-thaw or release proof.
|
||||
|
||||
### Backup age requires a usable completed timestamp
|
||||
|
||||
Existing workload indicators, age badges and drawer protection context distinguish
|
||||
|
||||
@@ -17,6 +17,50 @@
|
||||
|
||||
## Purpose
|
||||
|
||||
### Guest memory observations are source-owned
|
||||
|
||||
QEMU and LXC memory carry an optional `observation` with `state` (`current`,
|
||||
`last-known`, `unavailable`), canonical `source`, and original RFC3339
|
||||
`observedAt` when known. Normal TTL cache hits keep the original read time.
|
||||
Repeated QGA/backup deferrals are last-known, never renewed by a poll or
|
||||
diagnostic RetrievedAt; expired/missing usage and powered-off guests expose
|
||||
unavailability without inventing a read time. Legacy unannotated values make
|
||||
no freshness assertion. Independently selected PVE status/listing or linked
|
||||
Pulse-agent memory remains distinct from disk or QGA collection state. Agent
|
||||
receipt time belongs to its source status, not a guest row refreshed by PVE.
|
||||
|
||||
Annotation preserves numeric selection, capacity, retention, guest identity,
|
||||
agent admission, alert thresholds and existing History recording. It grants
|
||||
no live-check authority or proof of thaw. Existing legacy state, raw Proxmox
|
||||
facets and selected canonical memory metrics preserve the discriminator;
|
||||
selected metric provenance must not be borrowed from another merged facet.
|
||||
`TestGuestMemoryObservationContract` exercises poll/cache/deferral/expiry/
|
||||
resumption through unified and served conversions; wire and source-selection
|
||||
controls cover omission, cloning and independent live agent data. These are
|
||||
synthetic source proofs, not native QGA/HAOS acceptance or delivered relief.
|
||||
|
||||
### Unavailable backup-age evaluation is not silent recovery
|
||||
|
||||
PVE, PBS and mock backup checks share one failure-preserving evaluator. The
|
||||
existing two-second complete rollup-read budget includes connection waiting;
|
||||
it is not extended or bypassed. A failed or partial read does not invoke backup
|
||||
age evaluation with an empty result, clear existing age alerts or assert backup
|
||||
health. A fixed system warning makes the skipped evaluation visible in the
|
||||
existing alert list and served state, with no raw database/provider/path error.
|
||||
A complete subsequent evaluation clears that warning; normal cancellation or a
|
||||
missing manager does not stand in for read recovery. Deliberately disabled checks
|
||||
clear their irrelevant warning without reading recovery history.
|
||||
|
||||
Overlapping global checks coalesce instead of queueing. The single evaluation
|
||||
owner orders failure/recovery publication so an older success cannot erase a
|
||||
newer failure. `backup_alert_evaluation_test.go` exercises actual persisted
|
||||
rollups, stale/fresh backups, read failure/reopening, served-state visibility,
|
||||
repeated failures, missing dependencies, cancellation/deadlines and concurrent
|
||||
coalescing. `TestBackupAlertEvaluationCallersShareFailureVisibility` binds all
|
||||
three callers to that path. These are synthetic source controls, not native
|
||||
backup thaw, Raspberry Pi relief or published availability.
|
||||
|
||||
|
||||
### Platform connection discovery follows tenant storage identity
|
||||
|
||||
TrueNAS and VMware pollers enumerate validated, sorted persistence directory
|
||||
@@ -105,6 +149,26 @@ holds per-guest admission until body completion; its failure discards the
|
||||
payload. Complete responses at the limit, including trailing whitespace, remain
|
||||
valid. Ordinary PVE CPU/I/O observations remain independent of this deferral.
|
||||
|
||||
Lock verification also requires a direct HTTP 200 from the configured command
|
||||
endpoint and one unambiguous `data` configuration object. It never follows a
|
||||
redirect, including a same-origin redirect to another configuration path.
|
||||
Duplicate fields (including escaped duplicates), differently cased `data` or
|
||||
`lock` fields, competing error/failure envelopes and non-string operation locks
|
||||
yield `lock-unverified`; a later unlocked value cannot overwrite earlier backup
|
||||
evidence. Ordinary `GetVMConfig` remains compatible with its existing decoding
|
||||
and redirects. The lock check shares the configured pooled transport, TLS,
|
||||
proxy and timeout policy and retains config-read password-session recovery;
|
||||
the guest command itself is still single-attempt. This narrows the authority
|
||||
of lock evidence, not the ordinary API read contract. Failed post-command lock
|
||||
verification discards the guest payload without inventing a transport cooldown;
|
||||
subsequent same-target unlocked verification permits normal resumption.
|
||||
`TestGuestAgentLockEvidenceRejectsAmbiguity` and
|
||||
`TestGuestAgentLockEvidenceRejectsRedirects` exercise all six readers before and
|
||||
after dispatch, all five redirect statuses, same/cross endpoint redirects,
|
||||
ordinary config compatibility and fresh-client resumption. The connected
|
||||
`TestGuestAgentBackupLockEvidenceContract` covers warm-cache continuity, live
|
||||
CPU, unrenewed observations and History, then normal fresh poll resumption.
|
||||
|
||||
Admission is held through the complete response, not just headers. A transport
|
||||
failure, timeout, wrong-command-ID response or incomplete body defers every
|
||||
command for that guest for a minute; callers neither immediately retry nor
|
||||
@@ -112,6 +176,26 @@ replay through another cluster endpoint. Coordination retains at most 4,096
|
||||
active/cooling endpoint-guest entries and expires cooldowns. Explicit API
|
||||
permission refusals and unsupported commands remain errors, not successes.
|
||||
|
||||
The completed-error boundary uses the actual HTTP status retained by the request
|
||||
layer, not an `API error` phrase in provider or proxy text. HTTP 408 and
|
||||
unexplained server/proxy failures (including PVE 595) retain the same per-VM
|
||||
uncertainty cooldown, even when their bodies finish: command completion is still
|
||||
unverified. Another method, diagnostic client or configured cluster alias cannot
|
||||
immediately send a command. Only complete, exact command-bound terminal QGA
|
||||
rejections under HTTP 500 (unsupported/missing command, explicit stopped agent,
|
||||
and OS-info's known missing os-release) retain ordinary error handling. Ambiguous
|
||||
or conflicting envelopes, duplicate keys, other VM/command identities, extra
|
||||
values and gateway responses quoting a rejection do not gain that exemption.
|
||||
Existing authentication hints, error text, ordinary API session/retry policy and
|
||||
endpoint health remain unchanged. Unknown server failures now defer enrichment
|
||||
for a minute rather than inviting a sequence of diagnostic reads; this is a
|
||||
safety trade-off, not evidence of a freeze cause or native recovery.
|
||||
`TestGuestAgentHTTPFailureDefersEveryRead` exercises all six guest readers,
|
||||
independent status counters and cross-client cooldown; cluster and semantic
|
||||
controls pin no failover, definitive refusals and conservative error provenance.
|
||||
The connected transport/History lifecycle and counter-receipt controls also
|
||||
cover complete server/gateway failures without refreshing old guest evidence.
|
||||
|
||||
Guest commands also prevent transport-level replay: each uses a fresh single-use
|
||||
HTTP/1 connection, with no pooled-connection or HTTP/2-stream retries and no
|
||||
redirect following. The client preserves its configured TLS verification or
|
||||
@@ -120,7 +204,8 @@ transport or custom TLS dialer defers without dispatch rather than substituting
|
||||
a different trust policy. A fully received redirect discards its body and
|
||||
starts the same per-VM uncertainty cooldown; its destination is never contacted.
|
||||
Ordinary config/resource reads retain pooling, HTTP/2, redirects and session
|
||||
recovery. This deliberately trades one new connection per guest command for
|
||||
recovery, except that authoritative guest lock checks refuse redirects as above.
|
||||
This deliberately trades one new connection per guest command for
|
||||
single-attempt safety; it does not reserve the QGA channel or establish thaw.
|
||||
Wire controls drop a reply only after receiving a command (including when a
|
||||
backup lock then appears), and exercise every read and redirect code. TLS
|
||||
|
||||
@@ -15,6 +15,22 @@
|
||||
|
||||
## Purpose
|
||||
|
||||
### Guest read-state display does not start another read
|
||||
|
||||
The existing workload drawer's guest-read state and backup precaution derive
|
||||
only from its current workload props. Cached version metadata and Discovery
|
||||
assignment are not connectivity checks. Opening or changing the precaution
|
||||
must not start diagnostics, guest-agent requests or additional polling. Existing
|
||||
History/metadata reads keep their own ownership and request identities, and
|
||||
same-guest lock/uncertainty/recovery does not remount the drawer. No retained
|
||||
filesystem reason is promoted to a memory-source claim.
|
||||
|
||||
Verification: `GuestDrawer.test.tsx`, the mounted
|
||||
`GuestDrawerOverview.agent-state.test.tsx` state replacements and
|
||||
`browser-tests/guest-agent-coverage.cjs` cover source-only calculation, tab
|
||||
identity and absence of diagnostic/mutation requests. Synthetic read/render
|
||||
observations establish neither fleet CPU relief nor native guest recovery.
|
||||
|
||||
### Workload backup projection preserves timestamp uncertainty
|
||||
|
||||
Both resource-API and owning-canonical-snapshot paths preserve a present malformed
|
||||
|
||||
@@ -21,6 +21,95 @@
|
||||
|
||||
## Purpose
|
||||
|
||||
### Memory provenance does not imply recoverability
|
||||
|
||||
Canonical metric values may carry optional selected guest memory observation
|
||||
state/source/original known time. The facet follows that numeric source through
|
||||
cloning and frontend conversion; last-known or unavailable memory is not fresh
|
||||
backup evidence, a storage-health verdict or confirmation of thaw. This
|
||||
annotation changes no backup identity, retention, rollup selection, restore
|
||||
policy or snapshot data. The connected monitoring observation and metric
|
||||
source-selection controls verify the shared type without weakening recovery
|
||||
evidence requirements.
|
||||
|
||||
### Latest provider observations do not scan retained evidence payloads
|
||||
|
||||
The protection-posture reader enumerates provider/scope keys through the existing
|
||||
covering index, then seeks the latest observation by observed time and ID. Only
|
||||
that row's evidence payload is read for each scope. The 90-day retention, schema,
|
||||
single-connection policy and evidence validation are unchanged; malformed or
|
||||
invalid latest evidence never falls back to older reassuring facts. Narrow index
|
||||
key enumeration still scales with retained history, but wide historical payloads
|
||||
are not ranked on every ingest or posture read.
|
||||
|
||||
`store_queryplan_test.go` checks the production SQL's covering enumeration and
|
||||
indexed latest-row lookup. `store_posture_test.go` covers timestamp/ID ties,
|
||||
provider/scope separation, corrupt latest evidence, cancellation, retained row
|
||||
count and connection-pool deadline/recovery. The fixed synthetic history
|
||||
benchmark uses 1,000/70,949/180,000 rows and two scopes; it is not native Pi
|
||||
performance, backup restore acceptance or release qualification.
|
||||
|
||||
|
||||
### Backup inventory ownership is source-scoped and unambiguous
|
||||
|
||||
The existing Backups Coverage and By date views do not select a workload from
|
||||
an arbitrary substring, display label or first matching row when VMIDs repeat.
|
||||
PVE files, guest snapshots and task evidence use the supplied connection scope
|
||||
before native node names or aliases. One VMID inside a matching PVE connection
|
||||
stays associated after node migration; a known conflicting connection cannot be
|
||||
rescued by a reused node name or a singleton fallback. With no connection scope,
|
||||
an exact native node/alias must identify one candidate; wholly unscoped singleton
|
||||
observations retain their existing compatibility behaviour.
|
||||
|
||||
PBS instance and datastore identify a repository, not a PVE workload. A PBS
|
||||
namespace disambiguates repeated VMIDs only with a unique exact native node,
|
||||
alias or PVE connection label. Unique type/VMID PBS inventory retains the
|
||||
existing root-namespace association. This bounded presentation fix does not
|
||||
implement the runtime PBS source learner or alter server-owned protection policy.
|
||||
|
||||
Unresolved artifacts remain listed once and keep complete, collision-safe source
|
||||
keys (PVE connection/node or PBS instance/datastore/namespace), including host
|
||||
backup labels. Equal VMIDs, repository labels or host names cannot pool them
|
||||
across unproven sources. Untyped PVE tasks may join only PVE-scoped rows, not a
|
||||
PBS-only unresolved row with a lookalike namespace. The existing collapsed group
|
||||
and count say **unmatched**, not that the guest necessarily ceased to exist;
|
||||
its explanation wraps on phones. These rows remain **not evaluated**, separate
|
||||
from canonical provider-owned protection posture and live workload counts.
|
||||
|
||||
Verification: `proxmoxBackupRecoveryModel.identity.test.ts` reproduces prefix
|
||||
and reused-node collisions, order independence, migration/alias controls,
|
||||
conflicting scope, ambiguous type/namespace and separated unresolved sources.
|
||||
Mounted `ProxmoxBackupsTable.test.tsx` backup-identity cases cover both views,
|
||||
artifact preservation, reorder/display rename, expansion identity and no extra
|
||||
inventory read. `browser-tests/backup-identity.cjs` uses production tables,
|
||||
router, inventory readers and CSS with synthetic responses on desktop Chromium
|
||||
and touch WebKit at 390px/320px, including keyboard/touch disclosure, ambiguity,
|
||||
readable unmatched explanation and same-resource recovery. No native backup
|
||||
identity, restore, guest thaw or containing-release acceptance follows from
|
||||
these source/render observations.
|
||||
|
||||
|
||||
### PBS datastore capacity is not inferred from failed-read zeros
|
||||
|
||||
The existing Backups server table distinguishes a named datastore whose status
|
||||
read failed from an empty datastore and from a server without datastore data.
|
||||
Provider unavailable/error evidence or an unavailable server connection
|
||||
withdraws current capacity and deduplication figures, without dropping the row,
|
||||
changing its drawer identity or invalidating independently readable backup
|
||||
inventory. Unknown status, missing/nonfinite/negative capacity and invalid
|
||||
authoritative percentages cannot receive a healthy usage indication. Capacity
|
||||
requires a positive finite total and a finite nonnegative used observation;
|
||||
measured zero remains zero, and the existing 75/90-percent warnings and
|
||||
over-capacity danger remain intact. A subsequent valid same-identity snapshot
|
||||
restores presentation without a remount, timer or new read.
|
||||
|
||||
Fixed notices do not expose provider error text. Mounted
|
||||
`ProxmoxBackupServersTable.capacity.test.tsx` covers these distinctions and
|
||||
direct/canonical replacement; `ProxmoxBackupServersTable.test.ts` preserves
|
||||
source evidence and independent datastore identity. The production-component
|
||||
browser fixture checks desktop and narrow phone layouts. These synthetic
|
||||
checks establish no native datastore repair, backup/thaw or release acceptance.
|
||||
|
||||
### TrueNAS connection probes do not settle recovery inventory
|
||||
|
||||
A successful settings probe reads system information only, on a separate client.
|
||||
|
||||
@@ -23,6 +23,36 @@ and sort the complete canonical change table while startup and ingestion wait.
|
||||
|
||||
## Purpose
|
||||
|
||||
### Guest memory provenance follows metric selection
|
||||
|
||||
Raw Proxmox `memory.observation` preserves the guest builder's state, source
|
||||
and original known observation time. Canonical memory metrics carry their own
|
||||
optional `observation` through clones, merges and served frontend conversion.
|
||||
A retained platform facet cannot supply provenance for a different selected
|
||||
agent metric. Agent memory annotations are authored from server-owned source
|
||||
receipt/freshness, never agent-supplied annotation or merged guest LastSeen.
|
||||
Linked-agent views retain that source time after host/guest correlation.
|
||||
Absent annotations are unknown/legacy; absent observation times stay absent.
|
||||
|
||||
Numeric preference, unavailable same-source withdrawal, cross-source fallback,
|
||||
identity and freshness policy remain unchanged. Guest memory and disk-read
|
||||
provenance are independent. Adapter/source-selection and linked-view controls
|
||||
exercise these boundaries, with the connected monitoring contract pinning the
|
||||
actual served projection; this does not prove native recovery or publication.
|
||||
|
||||
### PBS capacity consumers preserve explicit read failure
|
||||
|
||||
`ResourcePBSDatastore.error` describes existing provider read-failure evidence,
|
||||
not a new collection or command. PBS resource facets retain that evidence
|
||||
through canonical presentation. The Backups capacity consumer must not treat
|
||||
the provider's zero-valued unavailable datastore as a successful empty-store
|
||||
observation, nor show retained numbers as current when the connection is
|
||||
unavailable. It keeps row/source identity and independent inventory counts;
|
||||
fixed unavailable/unknown notices never disclose the error body. Valid capacity
|
||||
including measured zero restores the same mounted row. Verification uses
|
||||
`ProxmoxBackupServersTable.test.ts` plus the direct/canonical mounted capacity
|
||||
controls; native read, backup and published acceptance remain separate.
|
||||
|
||||
### History consumers do not accept a mismatched or malformed success body
|
||||
|
||||
The common metrics History client validates the issued selection and response
|
||||
|
||||
@@ -0,0 +1,342 @@
|
||||
// Offline production-component proof; no live backup, restore or thaw claims.
|
||||
// Bound preview build pools only; retain ordinary browser process isolation.
|
||||
process.env.RAYON_NUM_THREADS = '1';
|
||||
process.env.GOMAXPROCS = '2';
|
||||
process.env.UV_THREADPOOL_SIZE = '2';
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const crypto = require('node:crypto');
|
||||
const { chromium, webkit } = require('playwright');
|
||||
const root = '/workspace/frontend-modern';
|
||||
const output = '/workspace/frontend-modern/node_modules/backup-identity-browser';
|
||||
const origin = 'http://127.0.0.1:5324';
|
||||
const runtime = [
|
||||
'frontend-modern/src/features/proxmox/proxmoxBackupRecoveryModel.ts',
|
||||
'frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx',
|
||||
];
|
||||
const digest = (file) => crypto.createHash('sha256').update(fs.readFileSync(file)).digest('hex');
|
||||
const guest = (id, instance, node, extra = {}) => ({
|
||||
id,
|
||||
name: id,
|
||||
type: 'vm',
|
||||
platformId: instance,
|
||||
platformType: 'proxmox-pve',
|
||||
sourceType: 'api',
|
||||
status: 'running',
|
||||
lastSeen: Date.parse('2026-10-04T12:00:00Z'),
|
||||
proxmox: { vmid: 100, instance, nodeName: node, ...extra },
|
||||
});
|
||||
const east = guest('east-100', 'east', 'pve-1');
|
||||
const west = guest('west-100', 'west', 'pve-10');
|
||||
const payloads = {
|
||||
'/api/backups/pve': {
|
||||
data: {
|
||||
storageBackups: [
|
||||
{
|
||||
id: 'file-100',
|
||||
instance: 'east',
|
||||
node: 'pve-1',
|
||||
type: 'vm',
|
||||
vmid: 100,
|
||||
storage: 'local',
|
||||
time: '2026-10-04T11:00:00Z',
|
||||
size: 1024,
|
||||
format: 'zst',
|
||||
protected: false,
|
||||
},
|
||||
],
|
||||
guestSnapshots: [
|
||||
{
|
||||
id: 'snapshot-100',
|
||||
instance: 'east',
|
||||
node: 'pve-1',
|
||||
type: 'vm',
|
||||
vmid: 100,
|
||||
name: 'before-update',
|
||||
time: '2026-10-04T11:30:00Z',
|
||||
vmstate: false,
|
||||
},
|
||||
],
|
||||
backupTasks: [
|
||||
{
|
||||
id: 'task-100',
|
||||
instance: 'east',
|
||||
node: 'pve-1',
|
||||
type: 'vzdump',
|
||||
vmid: 100,
|
||||
status: 'OK',
|
||||
startTime: '2026-10-04T11:00:00Z',
|
||||
endTime: '2026-10-04T11:05:00Z',
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
'/api/backups/pbs': {
|
||||
data: {
|
||||
backups: [
|
||||
{
|
||||
id: 'pbs-100',
|
||||
instance: 'pbs-server',
|
||||
datastore: 'main',
|
||||
namespace: 'pve-1',
|
||||
backupType: 'vm',
|
||||
vmid: '100',
|
||||
backupTime: '2026-10-04T10:00:00Z',
|
||||
files: ['index.json.blob'],
|
||||
size: 2048,
|
||||
protected: false,
|
||||
verified: true,
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
(async () => {
|
||||
fs.mkdirSync(output, { recursive: true });
|
||||
process.chdir(root);
|
||||
const version = require('playwright/package.json').version;
|
||||
assert.equal(
|
||||
version,
|
||||
JSON.parse(fs.readFileSync('/workspace/tests/integration/package-lock.json')).packages[
|
||||
'node_modules/@playwright/test'
|
||||
].version,
|
||||
);
|
||||
const { createServer } = await import(path.join(root, 'node_modules/vite/dist/node/index.js'));
|
||||
const server = await createServer({
|
||||
cacheDir: path.join(output, 'vite-cache'),
|
||||
optimizeDeps: { entries: ['browser-tests/backup-identity.html'] },
|
||||
server: { host: '127.0.0.1', port: 5324, strictPort: true },
|
||||
});
|
||||
const observations = [];
|
||||
let browser;
|
||||
try {
|
||||
await server.listen();
|
||||
for (const [name, engine, width] of [
|
||||
['chromium-desktop', chromium, 1365],
|
||||
['webkit-phone', webkit, 390],
|
||||
['webkit-narrow', webkit, 320],
|
||||
]) {
|
||||
browser = await engine.launch(
|
||||
engine === chromium
|
||||
? { headless: true, channel: 'chromium', args: ['--no-sandbox'] }
|
||||
: { headless: true },
|
||||
);
|
||||
const phone = engine === webkit;
|
||||
const viewport = { width, height: phone ? 844 : 900 };
|
||||
const page = await browser.newPage({
|
||||
viewport,
|
||||
isMobile: phone,
|
||||
hasTouch: phone,
|
||||
locale: 'en-GB',
|
||||
timezoneId: 'UTC',
|
||||
});
|
||||
page.setDefaultTimeout(20000);
|
||||
await page.clock.setFixedTime(new Date('2026-10-04T12:00:00Z'));
|
||||
const errors = [],
|
||||
offOrigin = [],
|
||||
requests = [],
|
||||
checks = [],
|
||||
screenshots = [];
|
||||
page.on('pageerror', (error) => errors.push(error.message));
|
||||
page.on('console', (message) => {
|
||||
if (message.type() === 'error') errors.push(message.text());
|
||||
});
|
||||
await page.route('**/*', async (route) => {
|
||||
const url = new URL(route.request().url());
|
||||
if (url.origin !== origin) {
|
||||
offOrigin.push(url.origin);
|
||||
return route.abort();
|
||||
}
|
||||
if (!url.pathname.startsWith('/api/')) return route.continue();
|
||||
requests.push({ path: url.pathname, method: route.request().method() });
|
||||
// Independent provider-owned posture deliberately remains unchanged.
|
||||
const payload = payloads[url.pathname] ?? {
|
||||
data: ['east-100', 'west-100'].map((id) => ({
|
||||
subjectResourceId: id,
|
||||
state: 'unprotected',
|
||||
freshness: 'unknown',
|
||||
verification: 'unknown',
|
||||
coverage: 'none',
|
||||
providerStates: [],
|
||||
repositoryResourceIds: [],
|
||||
evidenceIds: [],
|
||||
explanation: 'Synthetic server-owned posture.',
|
||||
evaluatedAt: '2026-10-04T12:00:00Z',
|
||||
})),
|
||||
policy: {},
|
||||
meta: { total: 2, totalPages: 1 },
|
||||
};
|
||||
await route.fulfill({
|
||||
status: 200,
|
||||
contentType: 'application/json',
|
||||
body: JSON.stringify(payload),
|
||||
});
|
||||
});
|
||||
const rows = (kind) => page.locator(`[data-proxmox-backup-row="${kind}"]`);
|
||||
const coverageRow = (name) => rows('coverage').filter({ hasText: name });
|
||||
const replace = (workloads) =>
|
||||
page.evaluate((next) => window.__replaceBackupIdentityWorkloads(next), workloads);
|
||||
const capture = async (suffix) => {
|
||||
await page.evaluate(() => {
|
||||
for (const animation of document.getAnimations()) {
|
||||
try {
|
||||
animation.finish();
|
||||
} catch {}
|
||||
}
|
||||
});
|
||||
const file = path.join(output, `${name}-${suffix}.png`);
|
||||
await page.screenshot({ path: file, fullPage: true, animations: 'disabled' });
|
||||
screenshots.push({ path: file, sha256: digest(file) });
|
||||
};
|
||||
await page.goto(`${origin}/browser-tests/backup-identity.html?view=coverage`, {
|
||||
waitUntil: 'domcontentloaded',
|
||||
timeout: 90000,
|
||||
});
|
||||
if (phone) await page.evaluate(() => document.documentElement.classList.add('dark'));
|
||||
await coverageRow('east-100').waitFor();
|
||||
// Navigate explicitly so the router owns the actual Coverage route.
|
||||
await page.getByRole('link', { name: 'Coverage', exact: true }).click();
|
||||
await page.waitForURL('**/proxmox/backups/coverage');
|
||||
await coverageRow('east-100')
|
||||
.getByText(phone ? 'Latest task: OK' : 'OK', { exact: true })
|
||||
.waitFor();
|
||||
assert(!(await coverageRow('west-100').textContent()).includes('OK'));
|
||||
checks.push('prefix node names: task stays with exact PVE connection');
|
||||
const toggle = coverageRow('east-100').getByRole('button');
|
||||
await toggle.focus();
|
||||
await page.keyboard.press('Enter');
|
||||
assert.equal(await toggle.getAttribute('aria-expanded'), 'true');
|
||||
const detailID = await toggle.getAttribute('aria-controls');
|
||||
const detail = page.locator(`[id="${detailID}"]`);
|
||||
await detail.getByText('PVE file', { exact: true }).waitFor();
|
||||
assert((await detail.textContent()).includes('PBS'));
|
||||
assert((await detail.textContent()).includes('Snapshot'));
|
||||
checks.push('keyboard opens all three correctly attributed artifact kinds');
|
||||
await replace([east, { ...west, proxmox: { ...west.proxmox, nodeDisplayName: 'pve-1' } }]);
|
||||
assert.equal(await toggle.getAttribute('aria-expanded'), 'true');
|
||||
assert.equal(await toggle.evaluate((element) => element === document.activeElement), true);
|
||||
assert(!(await coverageRow('west-100').textContent()).includes('OK'));
|
||||
checks.push(
|
||||
'reorder/display rename retains scoped evidence, expansion and focus without new reads',
|
||||
);
|
||||
await capture('coverage-exact');
|
||||
await page.getByRole('link', { name: 'By date', exact: true }).click();
|
||||
await page.waitForURL('**/proxmox/backups/date');
|
||||
assert.equal(await rows('recoverable').count(), 3);
|
||||
assert(
|
||||
(await rows('recoverable').allTextContents()).every((row) => row.includes('east-100')),
|
||||
);
|
||||
await replace([guest('west-100', 'west', 'pve-1'), east]);
|
||||
await rows('recoverable').filter({ hasText: 'PBS' }).waitFor();
|
||||
const named = (await rows('recoverable').allTextContents()).filter((row) =>
|
||||
row.includes('east-100'),
|
||||
);
|
||||
assert.equal(named.length, 2);
|
||||
assert.equal(await rows('recoverable').count(), 3);
|
||||
assert((await rows('recoverable').filter({ hasText: 'PBS' }).textContent()).includes('PBS'));
|
||||
checks.push(
|
||||
'shared native node: ambiguous PBS is retained unnamed; scoped PVE file/snapshot remain named',
|
||||
);
|
||||
await capture('date-ambiguous');
|
||||
await page.getByRole('link', { name: 'Coverage', exact: true }).click();
|
||||
await page.getByRole('button', { name: /1 unmatched backup/ }).click();
|
||||
await coverageRow('VM 100')
|
||||
.getByText(phone ? 'N/A' : 'Not evaluated', { exact: true })
|
||||
.waitFor();
|
||||
await coverageRow('east-100')
|
||||
.getByText(phone ? 'Latest task: OK' : 'OK', { exact: true })
|
||||
.waitFor();
|
||||
assert(!(await coverageRow('west-100').textContent()).includes('OK'));
|
||||
if (phone) await coverageRow('VM 100').locator('td').first().tap();
|
||||
else await coverageRow('VM 100').getByRole('button').click();
|
||||
const unresolvedToggle = coverageRow('VM 100').getByRole('button');
|
||||
const unresolvedID = await unresolvedToggle.getAttribute('aria-controls');
|
||||
const unresolved = page.locator(`[id='${unresolvedID}']`);
|
||||
await unresolved.getByText('PBS', { exact: true }).waitFor();
|
||||
assert(!(await unresolved.textContent()).includes('PVE file'));
|
||||
checks.push(
|
||||
'unresolved Coverage evidence stays separate and not evaluated; server posture is independent',
|
||||
);
|
||||
const noticeBounds = await page
|
||||
.getByText('No unambiguous inventory match', { exact: true })
|
||||
.evaluate((element) => {
|
||||
const range = document.createRange();
|
||||
range.selectNodeContents(element);
|
||||
const rect = range.getBoundingClientRect();
|
||||
const owner = element.closest('button').getBoundingClientRect();
|
||||
return {
|
||||
left: rect.left,
|
||||
right: rect.right,
|
||||
ownerLeft: owner.left,
|
||||
ownerRight: owner.right,
|
||||
};
|
||||
});
|
||||
assert(
|
||||
noticeBounds.left >= noticeBounds.ownerLeft &&
|
||||
noticeBounds.right <= noticeBounds.ownerRight,
|
||||
);
|
||||
checks.push('unmatched explanation wraps readably within the disclosure on phones');
|
||||
await capture('coverage-unmatched');
|
||||
// Return unique native placement: the same model retracts the orphan.
|
||||
await replace([west, east]);
|
||||
await page.getByRole('button', { name: /1 unmatched backup/ }).waitFor({ state: 'detached' });
|
||||
assert.equal(await rows('coverage').count(), 2);
|
||||
checks.push(
|
||||
'unambiguous same-resource replacement restores attribution and removes orphan without an extra request',
|
||||
);
|
||||
const overflow = await page.evaluate(
|
||||
() => document.documentElement.scrollWidth > window.innerWidth + 1,
|
||||
);
|
||||
assert.equal(overflow, false);
|
||||
assert.equal(requests.filter((request) => request.path === '/api/backups/pve').length, 1);
|
||||
assert.equal(requests.filter((request) => request.path === '/api/backups/pbs').length, 1);
|
||||
assert(requests.every((request) => request.method === 'GET'));
|
||||
assert.deepEqual(errors, []);
|
||||
assert.deepEqual(offOrigin, []);
|
||||
observations.push({
|
||||
noticeBounds,
|
||||
name,
|
||||
browser: browser.version(),
|
||||
viewport,
|
||||
checks,
|
||||
requests,
|
||||
screenshots,
|
||||
overflow,
|
||||
errors,
|
||||
offOrigin,
|
||||
});
|
||||
fs.writeFileSync(
|
||||
path.join(output, 'progress.json'),
|
||||
JSON.stringify({ completedContexts: observations }, null, 2),
|
||||
);
|
||||
await browser.close();
|
||||
browser = undefined;
|
||||
}
|
||||
fs.writeFileSync(
|
||||
path.join(output, 'result.json'),
|
||||
JSON.stringify(
|
||||
{
|
||||
result: 'passed',
|
||||
playwright: version,
|
||||
content_sha256: Object.fromEntries(
|
||||
runtime.map((file) => [file, digest(`/workspace/${file}`)]),
|
||||
),
|
||||
observations,
|
||||
},
|
||||
null,
|
||||
2,
|
||||
),
|
||||
);
|
||||
console.log(
|
||||
'Passed backup ownership in Chromium desktop and WebKit 390px/320px; synthetic production-component evidence.',
|
||||
);
|
||||
} finally {
|
||||
if (browser) await browser.close();
|
||||
await server.close();
|
||||
}
|
||||
})().catch((error) => {
|
||||
console.error(error);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
@@ -0,0 +1,12 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>Backup ownership verification</title>
|
||||
</head>
|
||||
<body>
|
||||
<div id="root"></div>
|
||||
<script type="module" src="./backup-identity.tsx"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,33 @@
|
||||
// Real router, inventory readers, production tables and CSS. Synthetic data.
|
||||
import { createSignal, onCleanup, onMount } from 'solid-js';
|
||||
import { render } from 'solid-js/web';
|
||||
import { Route, Router } from '@solidjs/router';
|
||||
import { ProxmoxBackupsTable } from '../src/features/proxmox/ProxmoxBackupsTable';
|
||||
import { identityGuest } from '../src/features/proxmox/__fixtures__/backupIdentity';
|
||||
import type { Resource } from '../src/types/resource';
|
||||
import '../src/index.css';
|
||||
|
||||
function Fixture() {
|
||||
const [workloads, setWorkloads] = createSignal<Resource[]>([
|
||||
identityGuest('west-100', 'west', 'pve-10'),
|
||||
identityGuest('east-100', 'east', 'pve-1'),
|
||||
]);
|
||||
onMount(() => {
|
||||
Object.assign(window, { __replaceBackupIdentityWorkloads: setWorkloads });
|
||||
onCleanup(() => Reflect.deleteProperty(window, '__replaceBackupIdentityWorkloads'));
|
||||
});
|
||||
return (
|
||||
<main class="p-3 sm:p-6">
|
||||
<ProxmoxBackupsTable emptyIcon={<span />} workloads={workloads()} />
|
||||
</main>
|
||||
);
|
||||
}
|
||||
|
||||
render(
|
||||
() => (
|
||||
<Router>
|
||||
<Route path="/*" component={Fixture} />
|
||||
</Router>
|
||||
),
|
||||
document.getElementById('root')!,
|
||||
);
|
||||
@@ -0,0 +1,205 @@
|
||||
// Run the actual integration spec and full app shell offline. Only unrelated
|
||||
// bootstrap/auth/state APIs are synthetic; the spec owns diagnostics responses.
|
||||
// This is frontend integration proof, not backend authentication or appliance proof.
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const crypto = require('node:crypto');
|
||||
const http = require('node:http');
|
||||
const { spawn } = require('node:child_process');
|
||||
const { chromium, webkit } = require('playwright');
|
||||
const { WebSocketServer } = require('/workspace/tests/integration/node_modules/ws');
|
||||
const root = '/workspace';
|
||||
const out = path.join(root, 'tmp/diagnostics-integration-proof');
|
||||
const origin = 'http://127.0.0.1:5393';
|
||||
const backendOrigin = 'http://127.0.0.1:5394';
|
||||
const state = {
|
||||
resources: [],
|
||||
connectedInfrastructure: [],
|
||||
activeAlerts: [],
|
||||
recentlyResolved: [],
|
||||
lastUpdate: Date.now(),
|
||||
};
|
||||
const bootstrap = {
|
||||
'/api/health': { status: 'healthy' },
|
||||
'/api/state': state,
|
||||
'/api/security/status': {
|
||||
hasAuthentication: true,
|
||||
requiresAuth: false,
|
||||
settingsCapabilities: {
|
||||
infrastructureRead: true,
|
||||
systemSettingsRead: true,
|
||||
diagnosticsRead: true,
|
||||
apiAccessRead: true,
|
||||
apiAccessWrite: true,
|
||||
authenticationRead: true,
|
||||
authenticationWrite: true,
|
||||
},
|
||||
sessionCapabilities: { isAdmin: true },
|
||||
presentationPolicy: {
|
||||
hideCommercial: true,
|
||||
hideUpgrade: true,
|
||||
demoMode: false,
|
||||
readOnly: false,
|
||||
},
|
||||
},
|
||||
'/api/config': { nodes: [], pbs: [], pmg: [], dockerHosts: [], theme: 'light' },
|
||||
'/api/version': { version: '6.4.5' },
|
||||
'/api/updates/check': { currentVersion: '6.4.5', latestVersion: '6.4.5', updateAvailable: false },
|
||||
'/api/license/status': { valid: false, tier: 'free', features: [] },
|
||||
'/api/ai/settings': { enabled: false },
|
||||
'/api/alerts/active': [],
|
||||
};
|
||||
(async () => {
|
||||
fs.mkdirSync(out, { recursive: true });
|
||||
const version = require('playwright/package.json').version;
|
||||
const lock = JSON.parse(fs.readFileSync(path.join(root, 'tests/integration/package-lock.json')));
|
||||
if (version !== lock.packages['node_modules/@playwright/test'].version)
|
||||
throw new Error('Playwright version mismatch');
|
||||
const paths = [
|
||||
'tests/integration/tests/69-diagnostics-onboarding.spec.ts',
|
||||
'tests/integration/playwright.config.ts',
|
||||
'tests/integration/tests/helpers.ts',
|
||||
'tests/integration/tsconfig.json',
|
||||
'frontend-modern/src/App.tsx',
|
||||
'frontend-modern/src/components/Settings/DiagnosticsPanel.tsx',
|
||||
'frontend-modern/src/components/Settings/DiagnosticsResultsPanel.tsx',
|
||||
'frontend-modern/src/components/Settings/diagnosticsModel.ts',
|
||||
'frontend-modern/src/components/Settings/useDiagnosticsPanelState.ts',
|
||||
'frontend-modern/src/utils/diagnosticsPresentation.ts',
|
||||
'frontend-modern/package-lock.json',
|
||||
'tests/integration/package-lock.json',
|
||||
'frontend-modern/browser-tests/diagnostics-integration.cjs',
|
||||
];
|
||||
const source = Object.fromEntries(
|
||||
paths.map((p) => [
|
||||
p,
|
||||
crypto
|
||||
.createHash('sha256')
|
||||
.update(fs.readFileSync(path.join(root, p)))
|
||||
.digest('hex'),
|
||||
]),
|
||||
);
|
||||
const requests = [];
|
||||
const backend = http.createServer((req, res) => {
|
||||
const pathname = new URL(req.url, backendOrigin).pathname;
|
||||
requests.push({ method: req.method, path: pathname });
|
||||
res.setHeader('Content-Type', 'application/json');
|
||||
res.end(JSON.stringify(bootstrap[pathname] ?? {}));
|
||||
});
|
||||
const sockets = new WebSocketServer({ server: backend });
|
||||
sockets.on('connection', (socket) => {
|
||||
socket.send(JSON.stringify({ type: 'initialState', data: state }));
|
||||
socket.on('message', () => socket.send(JSON.stringify({ type: 'pong' })));
|
||||
});
|
||||
process.env.PULSE_DEV_API_URL = backendOrigin;
|
||||
process.env.PULSE_DEV_WS_URL = backendOrigin.replace('http:', 'ws:');
|
||||
const frontend = path.join(root, 'frontend-modern');
|
||||
const { createServer } = await import(
|
||||
path.join(frontend, 'node_modules/vite/dist/node/index.js')
|
||||
);
|
||||
const server = await createServer({
|
||||
root: frontend,
|
||||
configFile: path.join(frontend, 'vite.config.ts'),
|
||||
cacheDir: path.join(out, 'vite-cache'),
|
||||
server: { host: '127.0.0.1', port: 5393, strictPort: true },
|
||||
});
|
||||
// A cookie-backed synthetic bootstrap fixture avoids password/auth mutation.
|
||||
const cookieState = path.join(out, 'synthetic-session.json');
|
||||
fs.writeFileSync(
|
||||
cookieState,
|
||||
JSON.stringify({
|
||||
cookies: [
|
||||
{
|
||||
name: 'pulse_session',
|
||||
value: 'synthetic-offline-only',
|
||||
domain: '127.0.0.1',
|
||||
path: '/',
|
||||
expires: -1,
|
||||
httpOnly: true,
|
||||
secure: false,
|
||||
sameSite: 'Lax',
|
||||
},
|
||||
],
|
||||
origins: [],
|
||||
}),
|
||||
);
|
||||
const config = path.join(out, 'playwright.config.cjs');
|
||||
fs.writeFileSync(
|
||||
config,
|
||||
`
|
||||
const { devices } = require('/workspace/tests/integration/node_modules/@playwright/test');
|
||||
module.exports = {
|
||||
testDir: '/workspace/tests/integration/tests', testMatch: '69-diagnostics-onboarding.spec.ts',
|
||||
workers: 1, retries: 0, maxFailures: 1, globalTimeout: 135000,
|
||||
timeout: 180000, expect: { timeout: 20000 },
|
||||
outputDir: '${out}/results', reporter: [['list'], ['json', { outputFile: '${out}/result.json' }]],
|
||||
use: { baseURL: '${origin}', trace: 'off', screenshot: 'only-on-failure', acceptDownloads: true },
|
||||
projects: [
|
||||
{ name: 'chromium', use: { ...devices['Desktop Chrome'], launchOptions: { executablePath: ${JSON.stringify(chromium.executablePath())}, args: ['--no-sandbox'] } } },
|
||||
{ name: 'mobile-chrome', use: { ...devices['Pixel 5'], launchOptions: { executablePath: ${JSON.stringify(chromium.executablePath())}, args: ['--no-sandbox'] } } },
|
||||
{ name: 'mobile-safari', use: { ...devices['iPhone 12'], launchOptions: { executablePath: ${JSON.stringify(webkit.executablePath())} } } },
|
||||
],
|
||||
};
|
||||
`,
|
||||
);
|
||||
let result, cleanup;
|
||||
try {
|
||||
await new Promise((resolve) => backend.listen(5394, '127.0.0.1', resolve));
|
||||
await server.listen();
|
||||
result = await new Promise((resolve, reject) => {
|
||||
const child = spawn(
|
||||
process.execPath,
|
||||
[
|
||||
path.join(root, 'tests/integration/node_modules/@playwright/test/cli.js'),
|
||||
'test',
|
||||
'--config',
|
||||
config,
|
||||
],
|
||||
{
|
||||
cwd: path.join(root, 'tests/integration'),
|
||||
env: {
|
||||
...process.env,
|
||||
PLAYWRIGHT_BASE_URL: origin,
|
||||
PULSE_E2E_COOKIE_STATE_PATH: cookieState,
|
||||
PULSE_E2E_REQUIRE_DEFAULT_MOCK_READY: 'false',
|
||||
PULSE_E2E_RUNTIME_STATE_PATH: path.join(out, 'unused-runtime.json'),
|
||||
},
|
||||
stdio: 'inherit',
|
||||
},
|
||||
);
|
||||
child.on('error', reject);
|
||||
child.on('exit', (exit, signal) => resolve({ pid: child.pid, exit, signal }));
|
||||
});
|
||||
} finally {
|
||||
for (const socket of sockets.clients) socket.terminate();
|
||||
await new Promise((resolve) => sockets.close(resolve));
|
||||
await server.close();
|
||||
await new Promise((resolve) => backend.close(resolve));
|
||||
cleanup = {
|
||||
frontendClosed: true,
|
||||
socketsClosed: true,
|
||||
backendClosed: true,
|
||||
runnerTerminal: Boolean(result),
|
||||
};
|
||||
fs.writeFileSync(
|
||||
path.join(out, 'receipt.json'),
|
||||
JSON.stringify(
|
||||
{
|
||||
source,
|
||||
playwright: version,
|
||||
result,
|
||||
cleanup,
|
||||
boundary:
|
||||
'Actual full app shell and exact integration spec; synthetic offline session/bootstrap/state, no native API/guest-agent requests or production authentication.',
|
||||
requests,
|
||||
},
|
||||
null,
|
||||
2,
|
||||
) + '\n',
|
||||
);
|
||||
}
|
||||
if (result.exit !== 0 || result.signal) throw new Error('Diagnostics integration failed');
|
||||
})().catch((error) => {
|
||||
console.error(error);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
@@ -0,0 +1,309 @@
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const crypto = require('node:crypto');
|
||||
const { chromium, webkit } = require('playwright');
|
||||
const root = '/workspace/frontend-modern';
|
||||
const output = '/workspace/tmp/guest-agent-coverage-corrected-proof';
|
||||
const runtime = [
|
||||
'frontend-modern/src/components/Workloads/GuestDrawer.tsx',
|
||||
'frontend-modern/src/components/Workloads/GuestDrawerOverview.tsx',
|
||||
'frontend-modern/src/components/Workloads/guestDrawerModel.ts',
|
||||
'frontend-modern/src/components/Workloads/workloadAgentReadiness.ts',
|
||||
];
|
||||
const hash = (file) => crypto.createHash('sha256').update(fs.readFileSync(file)).digest('hex');
|
||||
const initial = {
|
||||
state: 'deferred',
|
||||
reason: 'prev-vm-locked',
|
||||
lock: 'backup',
|
||||
backup: true,
|
||||
version: '6.4.5',
|
||||
assignment: false,
|
||||
diskUsed: 5 * 1024 ** 3,
|
||||
};
|
||||
(async () => {
|
||||
fs.mkdirSync(output, { recursive: true });
|
||||
const playwrightVersion = require('playwright/package.json').version;
|
||||
assert.equal(
|
||||
playwrightVersion,
|
||||
JSON.parse(fs.readFileSync('/workspace/tests/integration/package-lock.json')).packages[
|
||||
'node_modules/@playwright/test'
|
||||
].version,
|
||||
);
|
||||
const hashes = Object.fromEntries(runtime.map((p) => [p, hash(path.join('/workspace', p))]));
|
||||
process.chdir(root);
|
||||
const { createServer } = await import(path.join(root, 'node_modules/vite/dist/node/index.js'));
|
||||
const server = await createServer({
|
||||
root,
|
||||
configFile: path.join(root, 'vite.config.ts'),
|
||||
cacheDir: path.join(output, 'vite-cache'),
|
||||
server: { host: '127.0.0.1', port: 5296, strictPort: true, watch: null },
|
||||
});
|
||||
const origin = 'http://127.0.0.1:5296';
|
||||
const results = [],
|
||||
screenshots = [];
|
||||
let browser,
|
||||
phase = 'server';
|
||||
try {
|
||||
await server.listen();
|
||||
for (const [name, engine, width, dark] of [
|
||||
['chromium-desktop-light', chromium, 1365, false],
|
||||
['chromium-desktop-dark', chromium, 1365, true],
|
||||
['webkit-phone-dark', webkit, 390, true],
|
||||
['webkit-narrow-light', webkit, 320, false],
|
||||
]) {
|
||||
phase = name;
|
||||
browser = await engine.launch(
|
||||
engine === chromium
|
||||
? { headless: true, channel: 'chromium', args: ['--no-sandbox'] }
|
||||
: { headless: true },
|
||||
);
|
||||
const page = await browser.newPage({
|
||||
viewport: { width, height: width < 500 ? 844 : 900 },
|
||||
isMobile: width < 500,
|
||||
hasTouch: width < 500,
|
||||
locale: 'en-GB',
|
||||
timezoneId: 'UTC',
|
||||
});
|
||||
page.setDefaultTimeout(10000);
|
||||
page.setDefaultNavigationTimeout(60000);
|
||||
const errors = [],
|
||||
requests = [],
|
||||
offOrigin = [],
|
||||
checks = [];
|
||||
page.on('pageerror', (error) => errors.push(error.message));
|
||||
await page.addInitScript((dark) => {
|
||||
const apply = () => document.documentElement.classList.toggle('dark', dark);
|
||||
if (document.documentElement) apply();
|
||||
else document.addEventListener('DOMContentLoaded', apply, { once: true });
|
||||
}, dark);
|
||||
await page.routeWebSocket(/.*/, (ws) => ws.close());
|
||||
await page.route('**/*', async (route) => {
|
||||
const req = route.request(),
|
||||
url = new URL(req.url());
|
||||
if (url.origin !== origin) {
|
||||
offOrigin.push(url.origin);
|
||||
return route.abort();
|
||||
}
|
||||
if (!url.pathname.startsWith('/api/')) return route.continue();
|
||||
requests.push({ path: url.pathname, query: url.search, method: req.method() });
|
||||
assert.equal(req.method(), 'GET');
|
||||
if (url.pathname.includes('metrics-store/history'))
|
||||
return route.fulfill({
|
||||
json: {
|
||||
resourceType: 'vm',
|
||||
resourceId: url.searchParams.get('resourceId'),
|
||||
range: '24h',
|
||||
start: Date.UTC(2026, 9, 3, 14),
|
||||
end: Date.UTC(2026, 9, 4, 14),
|
||||
metrics: {},
|
||||
source: 'store',
|
||||
},
|
||||
});
|
||||
if (url.pathname === '/api/resources')
|
||||
return route.fulfill({ json: { data: [], total: 0, page: 1, limit: 500 } });
|
||||
if (url.pathname.includes('license'))
|
||||
return route.fulfill({ json: { capabilities: [], limits: [], max_history_days: 90 } });
|
||||
if (url.pathname.includes('metadata')) return route.fulfill({ json: {} });
|
||||
return route.fulfill({
|
||||
json: { success: true, data: [], alerts: [], anomalies: [], config: null },
|
||||
});
|
||||
});
|
||||
await page.goto(`${origin}/browser-tests/guest-agent-coverage.html`, {
|
||||
waitUntil: 'domcontentloaded',
|
||||
});
|
||||
await page.waitForFunction(() => Boolean(window.__guestAgentCoverage));
|
||||
const row = page.locator('[data-guest-id]').first();
|
||||
if (width > 500) {
|
||||
await row.getByRole('button', { name: 'Expand backup-guest', exact: true }).focus();
|
||||
await page.keyboard.press('Enter');
|
||||
} else await row.tap();
|
||||
const drawer = page.getByRole('region', { name: 'Guest details' });
|
||||
await drawer.waitFor({ state: 'visible' });
|
||||
const precaution = page.getByTestId('guest-read-precaution');
|
||||
const overview = page.getByTestId('guest-technical-details');
|
||||
const stateCell = () =>
|
||||
overview.getByText('Guest-agent reads', { exact: true }).locator('..');
|
||||
const update = async (value) => {
|
||||
await page.evaluate((v) => window.__guestAgentCoverage.update(v), value);
|
||||
};
|
||||
const capture = async (state) => {
|
||||
const filename = `${name}-${state}.png`,
|
||||
file = path.join(output, filename);
|
||||
await page.screenshot({ path: file, fullPage: true });
|
||||
screenshots.push({ filename, sha256: hash(file) });
|
||||
};
|
||||
const checkLayout = async (locator) => {
|
||||
const box = await locator.boundingBox();
|
||||
assert(box && box.x >= 0 && box.x + box.width <= width + 1);
|
||||
assert(await locator.evaluate((el) => el.scrollWidth <= el.clientWidth + 1));
|
||||
assert(await page.evaluate(() => document.documentElement.scrollWidth <= innerWidth + 1));
|
||||
};
|
||||
await precaution.waitFor();
|
||||
assert((await precaution.innerText()).includes('does not prove thaw'));
|
||||
assert((await precaution.innerText()).includes('filesystems covered by the backup'));
|
||||
assert((await stateCell().innerText()).includes('Deferred'));
|
||||
assert((await overview.innerText()).includes('Pulse Agent observed'));
|
||||
assert(!(await overview.innerText()).includes('Pulse Agent connected'));
|
||||
await checkLayout(precaution);
|
||||
await checkLayout(stateCell());
|
||||
const coverage = overview.getByText('Pulse Agent observed', { exact: true });
|
||||
assert(await coverage.evaluate((el) => el.scrollWidth <= el.clientWidth + 1));
|
||||
await capture('deferred-overview');
|
||||
checks.push(
|
||||
'Current QGA deferral independent of observed Pulse version; readable safety and covered-write guidance',
|
||||
);
|
||||
await page.evaluate(() => {
|
||||
window.__originalCoverageDrawer = document.querySelector('[aria-label="Guest details"]');
|
||||
});
|
||||
for (const [state, label] of [
|
||||
['expected-unreachable', 'Unreachable'],
|
||||
['not-running', 'Not running'],
|
||||
['disabled', 'Disabled'],
|
||||
['available', 'Reported available'],
|
||||
['private-provider-raw', 'Unknown'],
|
||||
]) {
|
||||
await update({ ...initial, state, reason: '', lock: '', backup: false });
|
||||
await page.waitForFunction(
|
||||
(label) =>
|
||||
document
|
||||
.querySelector('[data-testid="guest-technical-details"]')
|
||||
.textContent.includes(label),
|
||||
label,
|
||||
);
|
||||
assert((await stateCell().innerText()).includes(label));
|
||||
assert.equal(await precaution.count(), 0);
|
||||
assert(!(await overview.innerText()).includes('private-provider-raw'));
|
||||
await update({ ...initial, state, reason: '', lock: '', backup: false, version: '' });
|
||||
assert((await stateCell().innerText()).includes(label));
|
||||
assert(!(await overview.innerText()).includes('Pulse 6.4.5'));
|
||||
}
|
||||
checks.push(
|
||||
'Every declared/unknown state stays visible without a version; no raw provider text or fabricated connection',
|
||||
);
|
||||
await update({ ...initial, version: '', assignment: true });
|
||||
assert((await overview.innerText()).includes('Node agent assigned'));
|
||||
assert((await stateCell().innerText()).includes('Deferred'));
|
||||
await page.getByRole('tab', { name: 'History', exact: true }).click();
|
||||
await precaution.waitFor();
|
||||
await page.getByText('No stored history in this range', { exact: true }).first().waitFor();
|
||||
await update({
|
||||
...initial,
|
||||
state: 'expected-unreachable',
|
||||
reason: 'agent-timeout',
|
||||
lock: '',
|
||||
backup: false,
|
||||
assignment: true,
|
||||
});
|
||||
assert((await precaution.innerText()).includes('Completion is uncertain'));
|
||||
await checkLayout(precaution);
|
||||
await capture('uncertain-history');
|
||||
await page.getByRole('tab', { name: 'Manage', exact: true }).click();
|
||||
assert(await precaution.isVisible());
|
||||
assert.equal(
|
||||
await page.evaluate(
|
||||
() =>
|
||||
document.querySelector('[aria-label="Guest details"]') ===
|
||||
window.__originalCoverageDrawer,
|
||||
),
|
||||
true,
|
||||
);
|
||||
checks.push(
|
||||
'Same guest retains drawer/tab identity; precaution persists across History/Manage; assignment is not connectivity',
|
||||
);
|
||||
await update({
|
||||
...initial,
|
||||
state: 'available',
|
||||
reason: '',
|
||||
lock: '',
|
||||
backup: false,
|
||||
diskUsed: 7.5 * 1024 ** 3,
|
||||
});
|
||||
await page.getByRole('tab', { name: 'Overview', exact: true }).click();
|
||||
assert.equal(await precaution.count(), 0);
|
||||
assert((await stateCell().innerText()).includes('Reported available'));
|
||||
assert(
|
||||
(await stateCell().getByText('Reported available').getAttribute('title')).includes(
|
||||
'does not independently confirm thaw',
|
||||
),
|
||||
);
|
||||
assert((await row.locator('[data-workload-col="disk"]').innerText()).includes('75%'));
|
||||
await checkLayout(stateCell());
|
||||
await capture('reported-available');
|
||||
checks.push(
|
||||
'Fresh replacement withdraws precaution and resumes ordinary existing values, without asserting thaw',
|
||||
);
|
||||
await update(initial);
|
||||
await precaution.waitFor();
|
||||
const link = precaution.getByRole('link', { name: 'Backup safety guidance' });
|
||||
assert.equal(await link.getAttribute('href'), '/docs/VM_DISK_MONITORING');
|
||||
await link.focus();
|
||||
assert.equal(await link.evaluate((el) => document.activeElement === el), true);
|
||||
if (width > 500) await page.keyboard.press('Enter');
|
||||
else await link.tap();
|
||||
await page.getByRole('heading', { name: /VM Disk Monitoring/ }).waitFor();
|
||||
const article = page.locator('article');
|
||||
assert(
|
||||
(await article.innerText()).includes(
|
||||
'Pulse monitoring and alerts are unavailable while stopped',
|
||||
),
|
||||
);
|
||||
assert((await article.innerText()).includes('filesystems covered by the backup'));
|
||||
if (width === 320) await capture('safety-guide');
|
||||
assert.equal(
|
||||
requests.filter((r) => r.path.includes('diagnostics') || r.method !== 'GET').length,
|
||||
0,
|
||||
);
|
||||
assert.equal(offOrigin.length, 0);
|
||||
assert.deepEqual(errors, []);
|
||||
assert.deepEqual(
|
||||
Object.fromEntries(runtime.map((p) => [p, hash(path.join('/workspace', p))])),
|
||||
hashes,
|
||||
);
|
||||
checks.push(
|
||||
'Keyboard/touch guidance opens real shipped Docs; no diagnostic/guest mutation or off-origin request',
|
||||
);
|
||||
results.push({ name, width, dark, browser: await browser.version(), requests, checks });
|
||||
await browser.close();
|
||||
browser = null;
|
||||
}
|
||||
fs.writeFileSync(
|
||||
path.join(output, 'result.json'),
|
||||
JSON.stringify(
|
||||
{
|
||||
result: 'passed',
|
||||
verifiedAt: new Date().toISOString(),
|
||||
playwrightVersion,
|
||||
content_sha256: hashes,
|
||||
results,
|
||||
screenshots,
|
||||
limitations: [
|
||||
'Synthetic production-component proof, not native QGA availability, thaw, covered writes, installed acceptance or release qualification.',
|
||||
'No memory provenance inferred from filesystem state; current Resource API exposes no memory source.',
|
||||
],
|
||||
},
|
||||
null,
|
||||
2,
|
||||
) + '\n',
|
||||
);
|
||||
process.stdout.write(
|
||||
JSON.stringify({
|
||||
result: 'passed',
|
||||
groups: results.length,
|
||||
screenshots: screenshots.length,
|
||||
}) + '\n',
|
||||
);
|
||||
} catch (error) {
|
||||
fs.writeFileSync(
|
||||
path.join(output, 'failure.json'),
|
||||
JSON.stringify({ phase, error: String(error), results, screenshots }, null, 2) + '\n',
|
||||
);
|
||||
throw error;
|
||||
} finally {
|
||||
if (browser) await browser.close();
|
||||
await server.close();
|
||||
}
|
||||
})().catch((error) => {
|
||||
process.stderr.write(String(error.stack || error) + '\n');
|
||||
process.exitCode = 1;
|
||||
});
|
||||
@@ -0,0 +1,12 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>Guest-agent coverage verification</title>
|
||||
</head>
|
||||
<body>
|
||||
<div id="root"></div>
|
||||
<script type="module" src="./guest-agent-coverage.tsx"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,126 @@
|
||||
// Production canonical resource projection, row, drawer, router and Docs.
|
||||
// Synthetic observations only; no native QGA, diagnostic or backup operation.
|
||||
import { createSignal, Show } from 'solid-js';
|
||||
import { render } from 'solid-js/web';
|
||||
import { Router, Route } from '@solidjs/router';
|
||||
import { useWorkloads } from '../src/hooks/useWorkloads';
|
||||
import { mergeCanonicalResourceSnapshot } from '../src/utils/resourceStateAdapters';
|
||||
import { GuestRow } from '../src/components/Workloads/GuestRow';
|
||||
import { GuestDrawer } from '../src/components/Workloads/GuestDrawer';
|
||||
import { DarkModeContext } from '../src/contexts/appRuntime';
|
||||
import { syncAIRuntimeSettings } from '../src/stores/aiRuntimeState';
|
||||
import Docs from '../src/pages/Docs';
|
||||
import type { Resource } from '../src/types/resource';
|
||||
import '../src/index.css';
|
||||
|
||||
syncAIRuntimeSettings({ discovery_enabled: false });
|
||||
interface Observation {
|
||||
state?: string;
|
||||
reason?: string;
|
||||
lock?: string;
|
||||
backup?: boolean;
|
||||
version?: string;
|
||||
assignment?: boolean;
|
||||
diskUsed?: number;
|
||||
}
|
||||
const initial: Observation = {
|
||||
state: 'deferred',
|
||||
reason: 'prev-vm-locked',
|
||||
lock: 'backup',
|
||||
backup: true,
|
||||
version: '6.4.5',
|
||||
assignment: false,
|
||||
diskUsed: 5 * 1024 ** 3,
|
||||
};
|
||||
const nativeResource = (value: Observation): Resource => ({
|
||||
id: 'fixture-pve:pve-a:101',
|
||||
type: 'vm',
|
||||
name: 'backup-guest',
|
||||
displayName: 'backup-guest',
|
||||
platformType: 'proxmox-pve',
|
||||
platformScopes: ['proxmox-pve'],
|
||||
sourceType: 'api',
|
||||
status: 'online',
|
||||
sources: ['proxmox', 'agent'],
|
||||
lastSeen: Date.UTC(2026, 9, 4, 14),
|
||||
cpu: { current: 10 },
|
||||
memory: { current: 25, used: 1024 ** 3 / 4, total: 1024 ** 3 },
|
||||
disk: {
|
||||
current: ((value.diskUsed ?? 0) / (10 * 1024 ** 3)) * 100,
|
||||
used: value.diskUsed,
|
||||
total: 10 * 1024 ** 3,
|
||||
},
|
||||
proxmox: {
|
||||
instance: 'fixture-pve',
|
||||
nodeName: 'pve-a',
|
||||
vmid: 101,
|
||||
runtimeStatus: 'running',
|
||||
template: false,
|
||||
guestAgentStatus: value.state,
|
||||
diskStatusReason: value.reason,
|
||||
lock: value.lock,
|
||||
guestAgentExpected: true,
|
||||
backupInProgress: value.backup ?? false,
|
||||
lastBackup: '2026-10-04T06:00:00Z',
|
||||
disks: [],
|
||||
},
|
||||
agent: {
|
||||
agentId: 'fixture-pulse-agent',
|
||||
agentVersion: value.version || '',
|
||||
hostname: 'backup-guest',
|
||||
osName: 'Linux',
|
||||
},
|
||||
discoveryTarget: value.assignment
|
||||
? { resourceType: 'vm', agentId: 'fixture-node-agent', resourceId: '101' }
|
||||
: undefined,
|
||||
});
|
||||
function Fixture() {
|
||||
const [observation, setObservation] = createSignal(initial);
|
||||
const [snapshot, setSnapshot] = createSignal<Resource[]>([nativeResource(initial)]);
|
||||
const state = useWorkloads(() => true, { resourceSnapshot: snapshot });
|
||||
const guest = () => state.workloads()[0]!;
|
||||
const [expanded, setExpanded] = createSignal(false);
|
||||
(window as any).__guestAgentCoverage = {
|
||||
update: (next: Observation) => {
|
||||
setObservation(next);
|
||||
setSnapshot((previous) => mergeCanonicalResourceSnapshot([nativeResource(next)], previous));
|
||||
},
|
||||
observation,
|
||||
identity: () => guest().id,
|
||||
};
|
||||
return (
|
||||
<main class="mx-auto min-h-screen max-w-5xl space-y-3 bg-surface p-3 text-base-content">
|
||||
<h1 class="text-base font-semibold">Guest coverage evidence</h1>
|
||||
<p class="text-xs text-muted">Synthetic observations. No guest check is sent.</p>
|
||||
<Show when={Boolean(guest())}>
|
||||
<table class="w-full table-fixed">
|
||||
<tbody>
|
||||
<GuestRow
|
||||
guest={guest()}
|
||||
visibleColumnIds={['name', 'disk']}
|
||||
isExpanded={expanded()}
|
||||
onClick={() => setExpanded(!expanded())}
|
||||
workloadTableLayoutMode={window.innerWidth <= 768 ? 'phone' : 'wide'}
|
||||
/>
|
||||
</tbody>
|
||||
</table>
|
||||
<Show when={expanded()}>
|
||||
<section aria-label="Guest details">
|
||||
<GuestDrawer guest={guest()} onClose={() => setExpanded(false)} />
|
||||
</section>
|
||||
</Show>
|
||||
</Show>
|
||||
</main>
|
||||
);
|
||||
}
|
||||
render(
|
||||
() => (
|
||||
<DarkModeContext.Provider value={() => document.documentElement.classList.contains('dark')}>
|
||||
<Router>
|
||||
<Route path="/docs/*docPath" component={Docs} />
|
||||
<Route path="/*" component={Fixture} />
|
||||
</Router>
|
||||
</DarkModeContext.Provider>
|
||||
),
|
||||
document.getElementById('root')!,
|
||||
);
|
||||
@@ -0,0 +1,233 @@
|
||||
process.env.RAYON_NUM_THREADS = '2';
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const crypto = require('node:crypto');
|
||||
const { chromium, webkit } = require('playwright');
|
||||
const root = '/workspace/frontend-modern';
|
||||
const output = '/workspace/tmp/pbs-capacity/browser';
|
||||
const origin = 'http://127.0.0.1:5321';
|
||||
const sha256 = (file) => crypto.createHash('sha256').update(fs.readFileSync(file)).digest('hex');
|
||||
const runtime = [
|
||||
'src/features/proxmox/ProxmoxBackupServersTable.tsx',
|
||||
'src/features/proxmox/proxmoxBackupsTablePresentation.ts',
|
||||
'src/types/resource.ts',
|
||||
];
|
||||
|
||||
(async () => {
|
||||
fs.mkdirSync(output, { recursive: true });
|
||||
process.chdir(root);
|
||||
const clientVersion = require('playwright/package.json').version;
|
||||
assert.equal(
|
||||
clientVersion,
|
||||
JSON.parse(fs.readFileSync('/workspace/tests/integration/package-lock.json')).packages[
|
||||
'node_modules/@playwright/test'
|
||||
].version,
|
||||
);
|
||||
const { createServer } = await import(path.join(root, 'node_modules/vite/dist/node/index.js'));
|
||||
const server = await createServer({
|
||||
cacheDir: path.join(output, 'vite-cache'),
|
||||
optimizeDeps: { entries: ['browser-tests/pbs-capacity.html'] },
|
||||
server: { host: '127.0.0.1', port: 5321, strictPort: true },
|
||||
});
|
||||
const results = [];
|
||||
let browser;
|
||||
try {
|
||||
await server.listen();
|
||||
for (const [name, engine, width] of [
|
||||
['chromium-desktop', chromium, 1365],
|
||||
['webkit-phone', webkit, 390],
|
||||
['webkit-narrow', webkit, 320],
|
||||
]) {
|
||||
browser = await engine.launch(
|
||||
engine === chromium
|
||||
? { headless: true, channel: 'chromium', args: ['--no-sandbox'] }
|
||||
: { headless: true },
|
||||
);
|
||||
for (const transport of ['direct', 'canonical']) {
|
||||
const phone = engine === webkit;
|
||||
const page = await browser.newPage({
|
||||
viewport: { width, height: phone ? 844 : 900 },
|
||||
isMobile: phone,
|
||||
hasTouch: phone,
|
||||
colorScheme: phone ? 'dark' : 'light',
|
||||
});
|
||||
if (phone) await page.addInitScript(() => document.documentElement?.classList.add('dark'));
|
||||
page.setDefaultTimeout(20000);
|
||||
await page.clock.setFixedTime(new Date('2026-10-04T12:00:00Z'));
|
||||
const errors = [],
|
||||
offOrigin = [],
|
||||
requests = [],
|
||||
checks = [];
|
||||
page.on('pageerror', (error) => errors.push(error.message));
|
||||
await page.route('**/*', async (route) => {
|
||||
const req = route.request(),
|
||||
url = new URL(req.url());
|
||||
if (url.origin !== origin) {
|
||||
offOrigin.push(url.origin);
|
||||
return route.abort();
|
||||
}
|
||||
if (!url.pathname.startsWith('/api/')) return route.continue();
|
||||
requests.push({ path: url.pathname, method: req.method() });
|
||||
const data =
|
||||
url.pathname === '/api/backups/pve'
|
||||
? { backupTasks: [], storageBackups: [], guestSnapshots: [] }
|
||||
: url.pathname === '/api/backups/pbs'
|
||||
? {
|
||||
backups: [
|
||||
{
|
||||
id: 'pbs-point',
|
||||
vmid: '100',
|
||||
backupType: 'vm',
|
||||
backupTime: '2026-10-04T11:00:00Z',
|
||||
files: ['index.json.blob'],
|
||||
instance: 'pbs-main',
|
||||
datastore: 'main',
|
||||
protected: false,
|
||||
verified: true,
|
||||
},
|
||||
],
|
||||
}
|
||||
: { postures: [] };
|
||||
return route.fulfill({
|
||||
status: 200,
|
||||
contentType: 'application/json',
|
||||
body: JSON.stringify({ data }),
|
||||
});
|
||||
});
|
||||
// Verify the actual table and its font metrics, not the dev server's
|
||||
// generic load event. Keep the same bounded navigation/locator deadline.
|
||||
await page.goto(`${origin}/browser-tests/pbs-capacity.html?transport=${transport}`, {
|
||||
waitUntil: 'domcontentloaded',
|
||||
});
|
||||
// Imported production theme stores can initialise after addInitScript.
|
||||
// Apply the requested CSS context after the fixture is loaded too.
|
||||
if (phone) await page.evaluate(() => document.documentElement.classList.add('dark'));
|
||||
const table = page.locator('[data-proxmox-backups-table="servers"]');
|
||||
await table.waitFor();
|
||||
await page.waitForFunction(() => document.fonts.status === 'loaded');
|
||||
const head = await table.locator('th').allTextContents();
|
||||
const usedIndex = head.indexOf('Used');
|
||||
assert(usedIndex >= 0);
|
||||
const cell = table.locator('tbody tr').first().locator('td').nth(usedIndex);
|
||||
await cell.locator('text=40.0%').waitFor();
|
||||
await cell.evaluate((node) => {
|
||||
node.dataset.identityWitness = 'original';
|
||||
});
|
||||
const activate = async (state) => {
|
||||
const control = page.getByRole('button', { name: state, exact: true });
|
||||
if (phone) await control.tap();
|
||||
else {
|
||||
await control.focus();
|
||||
await control.press('Enter');
|
||||
}
|
||||
};
|
||||
const checkNotice = async (state, label) => {
|
||||
await activate(state);
|
||||
await cell.getByText(label, { exact: true }).waitFor();
|
||||
const evidence = await cell.evaluate((node) => ({
|
||||
text: node.textContent,
|
||||
witness: node.dataset.identityWitness,
|
||||
success: !!node.querySelector('.bg-emerald-500'),
|
||||
title: node.querySelector('[title]')?.title,
|
||||
}));
|
||||
assert.equal(evidence.text, label);
|
||||
assert.equal(evidence.witness, 'original');
|
||||
assert.equal(evidence.success, false);
|
||||
assert(evidence.title.includes('does not mean the datastore is empty'));
|
||||
assert(!(await table.textContent()).includes('2.0×'));
|
||||
assert(
|
||||
!(await page.locator('body').innerHTML()).includes('PRIVATE_PROVIDER_ERROR_SENTINEL'),
|
||||
);
|
||||
checks.push({ state, evidence });
|
||||
};
|
||||
await checkNotice('Failure', 'Unavailable');
|
||||
await page.screenshot({
|
||||
path: path.join(output, `${name}-${transport}-unavailable.png`),
|
||||
fullPage: true,
|
||||
});
|
||||
await checkNotice('Unknown', 'Unknown');
|
||||
await checkNotice('Offline', 'Unavailable');
|
||||
await checkNotice('Error', 'Unavailable');
|
||||
await activate('Zero');
|
||||
await cell.getByText('0.0%', { exact: true }).waitFor();
|
||||
await activate('Full');
|
||||
await cell.getByText('95.0%', { exact: true }).waitFor();
|
||||
assert((await cell.locator('.text-red-600').count()) > 0);
|
||||
checks.push({
|
||||
state: 'valid zero / full',
|
||||
mounted: await cell.getAttribute('data-identity-witness'),
|
||||
});
|
||||
await activate('No stores');
|
||||
await cell.getByText('No datastore data').waitFor();
|
||||
await activate('Healthy');
|
||||
await cell.getByText('40.0%', { exact: true }).waitFor();
|
||||
await activate('Failure');
|
||||
await cell.getByText('Unavailable', { exact: true }).waitFor();
|
||||
const geometry = await cell.evaluate((node) => {
|
||||
const label = node.querySelector('span'),
|
||||
range = document.createRange();
|
||||
range.selectNodeContents(label);
|
||||
const text = range.getBoundingClientRect(),
|
||||
box = node.getBoundingClientRect();
|
||||
return {
|
||||
text: { x: text.x, right: text.right, width: text.width },
|
||||
cell: { x: box.x, right: box.right, width: box.width },
|
||||
pageWidth: document.documentElement.clientWidth,
|
||||
scrollWidth: document.documentElement.scrollWidth,
|
||||
};
|
||||
});
|
||||
assert(
|
||||
geometry.text.x >= geometry.cell.x && geometry.text.right <= geometry.cell.right,
|
||||
JSON.stringify({ name, geometry }),
|
||||
);
|
||||
assert(geometry.scrollWidth <= geometry.pageWidth + 1, JSON.stringify(geometry));
|
||||
checks.push({ state: 'warning fits', geometry });
|
||||
assert.equal(errors.length, 0, errors.join('\n'));
|
||||
assert.equal(offOrigin.length, 0);
|
||||
assert(requests.every((req) => req.method === 'GET'));
|
||||
results.push({
|
||||
name,
|
||||
transport,
|
||||
width,
|
||||
browserVersion: browser.version(),
|
||||
checks,
|
||||
requests,
|
||||
errors,
|
||||
offOrigin,
|
||||
});
|
||||
await page.close();
|
||||
}
|
||||
await browser.close();
|
||||
browser = undefined;
|
||||
}
|
||||
fs.writeFileSync(
|
||||
path.join(output, 'result.json'),
|
||||
JSON.stringify(
|
||||
{
|
||||
result: 'passed',
|
||||
clientVersion,
|
||||
results,
|
||||
content_sha256: Object.fromEntries(
|
||||
runtime.map((file) => [`frontend-modern/${file}`, sha256(path.join(root, file))]),
|
||||
),
|
||||
},
|
||||
null,
|
||||
2,
|
||||
),
|
||||
);
|
||||
console.log(JSON.stringify({ result: 'passed', contexts: results.length, output }));
|
||||
} catch (error) {
|
||||
fs.writeFileSync(
|
||||
path.join(output, 'failure.json'),
|
||||
JSON.stringify({ message: error.message, completedContexts: results }, null, 2),
|
||||
);
|
||||
throw error;
|
||||
} finally {
|
||||
if (browser) await browser.close();
|
||||
await server.close();
|
||||
}
|
||||
})().catch((error) => {
|
||||
console.error(error.stack);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
@@ -0,0 +1,12 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>PBS capacity evidence</title>
|
||||
</head>
|
||||
<body>
|
||||
<div id="root"></div>
|
||||
<script type="module" src="/browser-tests/pbs-capacity.tsx"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,81 @@
|
||||
// Production table/router/CSS and canonical resource adapter. Synthetic read
|
||||
// states only: no provider, native datastore, backup or thaw operation.
|
||||
import { createMemo, createSignal } from 'solid-js';
|
||||
import { render } from 'solid-js/web';
|
||||
import { Route, Router } from '@solidjs/router';
|
||||
import type { Resource, ResourcePBSDatastore } from '../src/types/resource';
|
||||
import { canonicalizeRealtimeResource } from '../src/utils/resourceStateAdapters';
|
||||
import { ProxmoxBackupsTable } from '../src/features/proxmox/ProxmoxBackupsTable';
|
||||
import '../src/index.css';
|
||||
|
||||
const states = ['Healthy', 'Failure', 'Unknown', 'Offline', 'Zero', 'Full', 'Error', 'No stores'];
|
||||
const normal = {
|
||||
name: 'main',
|
||||
total: 10000,
|
||||
used: 4000,
|
||||
available: 6000,
|
||||
usagePercent: 40,
|
||||
status: 'available',
|
||||
deduplicationFactor: 2,
|
||||
};
|
||||
|
||||
function Fixture() {
|
||||
const [state, setState] = createSignal('Healthy');
|
||||
const servers = createMemo((): Resource[] => {
|
||||
let store: ResourcePBSDatastore = { ...normal };
|
||||
if (state() === 'Failure')
|
||||
store = { ...store, total: 0, used: 0, available: 0, usagePercent: 0, status: 'unavailable' };
|
||||
if (state() === 'Unknown')
|
||||
store = { ...store, total: 0, used: 0, available: 0, usagePercent: 0 };
|
||||
if (state() === 'Zero') store = { ...store, used: 0, usagePercent: 0, available: 10000 };
|
||||
if (state() === 'Full') store = { ...store, used: 9500, usagePercent: 95, available: 500 };
|
||||
if (state() === 'Error') store = { ...store, error: 'PRIVATE_PROVIDER_ERROR_SENTINEL' };
|
||||
const pbs = {
|
||||
instanceId: 'pbs-main',
|
||||
connectionHealth: state() === 'Offline' ? 'offline' : 'healthy',
|
||||
version: '3.2.1',
|
||||
datastores: state() === 'No stores' ? [] : [store],
|
||||
};
|
||||
const resource: Resource = {
|
||||
id: 'pbs-main',
|
||||
type: 'pbs',
|
||||
name: 'pbs-main',
|
||||
displayName: 'pbs-main',
|
||||
platformId: 'pbs-main',
|
||||
platformType: 'proxmox-pbs',
|
||||
sourceType: 'api',
|
||||
status: 'online',
|
||||
lastSeen: Date.parse('2026-10-04T12:00:00Z'),
|
||||
cpu: { current: 12 },
|
||||
memory: { current: 40, total: 10000, used: 4000 },
|
||||
pbs,
|
||||
};
|
||||
if (new URLSearchParams(location.search).get('transport') === 'canonical') {
|
||||
resource.pbs = undefined;
|
||||
resource.platformData = { sources: ['pbs'], pbs };
|
||||
return [canonicalizeRealtimeResource(resource)];
|
||||
}
|
||||
return [resource];
|
||||
});
|
||||
return (
|
||||
<main class="p-3 sm:p-6">
|
||||
<div class="mb-4 flex flex-wrap gap-2" aria-label="Synthetic observation controls">
|
||||
{states.map((value) => (
|
||||
<button class="rounded border px-2 py-1" type="button" onClick={() => setState(value)}>
|
||||
{value}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
<ProxmoxBackupsTable emptyIcon={<span />} workloads={[]} servers={servers()} />
|
||||
</main>
|
||||
);
|
||||
}
|
||||
|
||||
render(
|
||||
() => (
|
||||
<Router>
|
||||
<Route path="/*" component={Fixture} />
|
||||
</Router>
|
||||
),
|
||||
document.getElementById('root')!,
|
||||
);
|
||||
@@ -0,0 +1,53 @@
|
||||
{
|
||||
"version": 1,
|
||||
"result": "passed",
|
||||
"base_sha": "ba3e2d2c1c1cc3c2fa07d1c3372080082121f1f2",
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/components/Workloads/GuestDrawer.tsx": "1c299b74082c37769c672931c5f99e666fa0585d6453c8ba61c38332d72d6cea",
|
||||
"frontend-modern/src/components/Workloads/GuestDrawerOverview.tsx": "c70baf65a3b1027a76043f57a72e9807fa1b6791ad6efaae2547a2e7e328b441",
|
||||
"frontend-modern/src/components/Workloads/guestDrawerModel.ts": "9c3e42b1381a37309148e43617aa875c3ba00f97df184d920fdec8d1ca5a8b8c",
|
||||
"frontend-modern/src/components/Workloads/workloadAgentReadiness.ts": "55f61246e581696aea67d6f7edf330e9f472ecdabf34728f42ef3e4cf7fe43c8"
|
||||
},
|
||||
"changed_paths": [
|
||||
"frontend-modern/src/components/Workloads/GuestDrawer.tsx",
|
||||
"frontend-modern/src/components/Workloads/GuestDrawerOverview.tsx",
|
||||
"frontend-modern/src/components/Workloads/guestDrawerModel.ts",
|
||||
"frontend-modern/src/components/Workloads/workloadAgentReadiness.ts"
|
||||
],
|
||||
"routes": [
|
||||
"/browser-tests/guest-agent-coverage.html",
|
||||
"/docs/VM_DISK_MONITORING"
|
||||
],
|
||||
"states": [
|
||||
"Current QGA deferral/unreachability/not-running/disabled/available/unknown independently of Pulse version metadata",
|
||||
"Lock overrides cached availability",
|
||||
"No version; parent-node assignment; fresh same-guest recovery"
|
||||
],
|
||||
"interactions": [
|
||||
"Keyboard disclosure on desktop and touch row on phones",
|
||||
"Overview/History/Manage retain visible precaution and drawer identity",
|
||||
"Keyboard/touch navigation into actual shipped backup-safety Docs; no diagnostic/mutation request"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
"width": 1365,
|
||||
"height": 900
|
||||
},
|
||||
{
|
||||
"width": 390,
|
||||
"height": 844
|
||||
},
|
||||
{
|
||||
"width": 320,
|
||||
"height": 844
|
||||
}
|
||||
],
|
||||
"verified_at": "2026-10-04T14:27:17.008Z",
|
||||
"command": "pulse-worker-browser frontend-modern/browser-tests/guest-agent-coverage.cjs",
|
||||
"playwright_version": "1.56.1",
|
||||
"limitations": [
|
||||
"Synthetic production-component proof, not native QGA availability, thaw, covered writes, installed acceptance or release qualification.",
|
||||
"No memory provenance inferred from filesystem state; current Resource API exposes no memory source.",
|
||||
"Synthetic preview emits localhost WebSocket proxy ECONNREFUSED warnings with no backend service; warnings retained, not native collection or transport acceptance."
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"version": 1,
|
||||
"result": "passed",
|
||||
"base_sha": "8f9a9a6a1e54962c9d81991ed51a8ff7e6c6b325",
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/components/Workloads/guestDrawerModel.ts": "f42687d9b55b1456678db6eb134052d3de32b4e463ccae8c3a4eaaf73309ca08"
|
||||
},
|
||||
"changed_paths": ["frontend-modern/src/components/Workloads/guestDrawerModel.ts"],
|
||||
"routes": ["/browser-tests/guest-agent-coverage.html", "/docs/VM_DISK_MONITORING"],
|
||||
"states": [
|
||||
"Current QGA deferral/unreachability/not-running/disabled/available/unknown independently of Pulse version metadata",
|
||||
"Lock overrides cached availability",
|
||||
"No version; parent-node assignment; fresh same-guest recovery"
|
||||
],
|
||||
"interactions": [
|
||||
"Keyboard disclosure on desktop and touch row on phones",
|
||||
"Overview/History/Manage retain visible precaution and drawer identity",
|
||||
"Keyboard/touch navigation into actual shipped backup-safety Docs; no diagnostic/mutation request"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
"width": 1365,
|
||||
"height": 900
|
||||
},
|
||||
{
|
||||
"width": 390,
|
||||
"height": 844
|
||||
},
|
||||
{
|
||||
"width": 320,
|
||||
"height": 844
|
||||
}
|
||||
],
|
||||
"verified_at": "2026-10-04T14:38:17.382Z",
|
||||
"command": "pulse-worker-browser frontend-modern/browser-tests/guest-agent-coverage.cjs",
|
||||
"playwright_version": "1.56.1",
|
||||
"limitations": [
|
||||
"Synthetic production-component proof, not native QGA availability, thaw, covered writes, installed acceptance or release qualification.",
|
||||
"No memory provenance inferred from filesystem state; current Resource API exposes no memory source."
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
{
|
||||
"version": 1,
|
||||
"base_sha": "ccb296676c167d2b948d96a54319bc0d7b52549a",
|
||||
"verified_at": "2026-10-04T11:21:10.896992Z",
|
||||
"result": "passed",
|
||||
"changed_paths": [
|
||||
"frontend-modern/src/features/proxmox/ProxmoxBackupServersTable.tsx",
|
||||
"frontend-modern/src/features/proxmox/proxmoxBackupsTablePresentation.ts",
|
||||
"frontend-modern/src/types/resource.ts"
|
||||
],
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/features/proxmox/ProxmoxBackupServersTable.tsx": "5d51f3fe36638bde58f0d31b550947a78adc664dc0898ca323aac4fb8db62903",
|
||||
"frontend-modern/src/features/proxmox/proxmoxBackupsTablePresentation.ts": "00bdd77055a4c1c76bd532361f5bf0905aa253a6a72adf630445eb319f4843d0",
|
||||
"frontend-modern/src/types/resource.ts": "007885931ae929a05ce382fa70d9afd3ed784d5ca32fc4ccf403260675a5a6de"
|
||||
},
|
||||
"routes": [
|
||||
"Local /browser-tests/pbs-capacity.html: production ProxmoxBackupsTable/router/CSS, direct and canonical synthetic resources"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
"width": 1365,
|
||||
"height": 900,
|
||||
"browser": "Chromium",
|
||||
"input": "keyboard",
|
||||
"colour": "light"
|
||||
},
|
||||
{
|
||||
"width": 390,
|
||||
"height": 844,
|
||||
"browser": "WebKit",
|
||||
"input": "touch",
|
||||
"colour": "dark"
|
||||
},
|
||||
{
|
||||
"width": 320,
|
||||
"height": 844,
|
||||
"browser": "WebKit",
|
||||
"input": "touch",
|
||||
"colour": "dark"
|
||||
}
|
||||
],
|
||||
"states": [
|
||||
"Healthy capacity \u2192 failed zero capacity \u2192 unknown \u2192 offline retained values \u2192 safe fixed error notice",
|
||||
"Measured zero remains zero; 95% remains danger; missing datastore stays separate; ordinary readings resume",
|
||||
"All six content-identical product contexts complete; earlier overall init/load failures remain adverse"
|
||||
],
|
||||
"interactions": [
|
||||
"Desktop keyboard Enter and phone taps switch synthetic observations in the existing page",
|
||||
"Same-identity notice updates preserve the mounted capacity cell; no false success dot, raw error or unreliable deduplication",
|
||||
"Font-aware warning bounds and page overflow checked; all six screenshots inspected",
|
||||
"No external-origin request or API mutation"
|
||||
],
|
||||
"evidence": "Six exact-content context controls across two retained attempts, not a claim that every browser attempt passed. See browser-context-matrix.json in the lane dossier. Earlier generic-load timeout is not cleared or re-described as a timing pass."
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
{
|
||||
"version": 1,
|
||||
"base_sha": "795f0b4863e46b81584c82152cc7e66974e5c9a1",
|
||||
"verified_at": "2026-10-04T11:36:56.829475Z",
|
||||
"result": "passed",
|
||||
"changed_paths": [
|
||||
"frontend-modern/src/features/proxmox/ProxmoxBackupServersTable.tsx",
|
||||
"frontend-modern/src/features/proxmox/proxmoxBackupsTablePresentation.ts",
|
||||
"frontend-modern/src/types/resource.ts"
|
||||
],
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/features/proxmox/ProxmoxBackupServersTable.tsx": "d2063f2d535c08980f57708a9cd4871f5cd1b98de1bb007c3eb78940463a2ded",
|
||||
"frontend-modern/src/features/proxmox/proxmoxBackupsTablePresentation.ts": "00bdd77055a4c1c76bd532361f5bf0905aa253a6a72adf630445eb319f4843d0",
|
||||
"frontend-modern/src/types/resource.ts": "007885931ae929a05ce382fa70d9afd3ed784d5ca32fc4ccf403260675a5a6de"
|
||||
},
|
||||
"routes": [
|
||||
"Local /browser-tests/pbs-capacity.html: production ProxmoxBackupsTable/router/CSS, direct and canonical synthetic resources"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
"width": 1365,
|
||||
"height": 900,
|
||||
"browser": "Chromium",
|
||||
"input": "keyboard",
|
||||
"colour": "light"
|
||||
},
|
||||
{
|
||||
"width": 390,
|
||||
"height": 844,
|
||||
"browser": "WebKit",
|
||||
"input": "touch",
|
||||
"colour": "dark"
|
||||
},
|
||||
{
|
||||
"width": 320,
|
||||
"height": 844,
|
||||
"browser": "WebKit",
|
||||
"input": "touch",
|
||||
"colour": "dark"
|
||||
}
|
||||
],
|
||||
"states": [
|
||||
"Healthy capacity \u2192 failed zero capacity \u2192 unknown \u2192 offline retained values \u2192 safe fixed error notice",
|
||||
"Measured zero remains zero; 95% remains danger; missing datastore stays separate; ordinary readings resume",
|
||||
"All six desktop/touch product contexts complete in one final execution; earlier init/load/source failures retained separately"
|
||||
],
|
||||
"interactions": [
|
||||
"Desktop keyboard Enter and phone taps switch synthetic observations in the existing page",
|
||||
"Same-identity notice updates preserve the mounted capacity cell; no false success dot, raw error or unreliable deduplication",
|
||||
"Font-aware warning bounds and page overflow checked; all six screenshots inspected",
|
||||
"No external-origin request or API mutation"
|
||||
],
|
||||
"evidence": "Final complete six-context execution, source-bound by browser-qualified-binding/result in the lane dossier. Earlier generic-load timeout and all source failures are retained, not cleared or claimed as timing passes."
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"version": 1,
|
||||
"result": "passed",
|
||||
"base_sha": "c5e83ceb3a506aa00dd135bf53adb56c378c4273",
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/features/proxmox/proxmoxBackupRecoveryModel.ts": "fa1ed522fb020ca8a88b16a611fc55b52ad3960db4e328b83080f1827279003b",
|
||||
"frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx": "9cd562b566cfddafb622469bddd75a676e333b50bfac4f7abd7a3e5e181c5f40"
|
||||
},
|
||||
"changed_paths": [
|
||||
"frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx",
|
||||
"frontend-modern/src/features/proxmox/proxmoxBackupRecoveryModel.ts"
|
||||
],
|
||||
"routes": [
|
||||
"/proxmox/backups/coverage",
|
||||
"/proxmox/backups/date"
|
||||
],
|
||||
"states": [
|
||||
"Prefix/reused native node names with repeated VMIDs",
|
||||
"Connection-owned PVE file/snapshot/task evidence",
|
||||
"Ambiguous PBS namespace remains unmatched and not evaluated",
|
||||
"Display rename/reorder and unique-placement replacement",
|
||||
"Independent synthetic server protection posture"
|
||||
],
|
||||
"interactions": [
|
||||
"Keyboard opens production Coverage evidence; expansion and focus survive reorder",
|
||||
"Touch opens unmatched row; readable disclosure explanation at 390px and 320px",
|
||||
"Route-backed By date/Coverage navigation",
|
||||
"Ambiguous PBS owner withdrawal, artifact preservation and restored unique attribution without extra reads"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
"width": 1365,
|
||||
"height": 900
|
||||
},
|
||||
{
|
||||
"width": 390,
|
||||
"height": 844
|
||||
},
|
||||
{
|
||||
"width": 320,
|
||||
"height": 844
|
||||
}
|
||||
],
|
||||
"verified_at": "2026-10-04T13:39:46Z",
|
||||
"command": "pulse-worker-browser frontend-modern/browser-tests/backup-identity.cjs",
|
||||
"playwright_version": "1.56.1",
|
||||
"limitations": [
|
||||
"Synthetic production-component proof, not native backup identity, restore, thaw, installed acceptance or release qualification.",
|
||||
"Existing compact table abbreviations/clipping are not addressed by this ownership fix."
|
||||
]
|
||||
}
|
||||
@@ -233,8 +233,11 @@ repair an older generated unit rather than adding a JSON-parsing wrapper.
|
||||
token's read permissions. Schedule any setup change or restart outside backups.
|
||||
- Do not run guest-agent probes during backup freeze/thaw. An OK backup task
|
||||
or an absent lock does not confirm thaw.
|
||||
- See [VM Disk Monitoring](VM_DISK_MONITORING.md) for the passive host preflight
|
||||
and backup safety precaution; it does not verify a fresh disk poll.
|
||||
- See [VM Disk Monitoring](VM_DISK_MONITORING.md) for the passive host preflight;
|
||||
it does not verify a fresh disk poll. If an affected installation needs to
|
||||
pause monitoring, follow the [manual backup precaution](VM_DISK_MONITORING.md#pause-pulse-for-a-planned-freeze-enabled-backup).
|
||||
It checks the server service and update timer before the backup, then requires
|
||||
independent guest thaw confirmation before starting Pulse again.
|
||||
|
||||
#### Temperature data missing
|
||||
- Install `lm-sensors` on the host.
|
||||
|
||||
@@ -24,6 +24,89 @@ or force-reset a guest as a disk-monitoring diagnostic. Those operations can
|
||||
affect workloads or backup consistency. Preserve the existing backup task and
|
||||
redacted error, rather than repeat a potentially harmful backup for a report.
|
||||
|
||||
### Pause Pulse for a planned freeze-enabled backup
|
||||
|
||||
This manual precaution is for an affected **systemd installation**, before a
|
||||
scheduled backup starts. It is **not a recovery procedure** for a backup already
|
||||
in progress or an unresponsive guest, and **not an automatic backup hook**.
|
||||
|
||||
1. **Identify the Pulse server service.** Run these commands on the machine
|
||||
running the Pulse server: for Proxmox LXC, **inside the Pulse LXC**,
|
||||
**not the Proxmox host or the backed-up VM**. The examples use
|
||||
`pulse.service`; substitute your actual server unit throughout
|
||||
(`pulse-backend.service` on some older installs), **not pulse-agent.service**.
|
||||
|
||||
```bash
|
||||
systemctl show pulse.service --property=LoadState,ActiveState,MainPID
|
||||
```
|
||||
|
||||
Require `LoadState=loaded` and record whether Pulse was active. If the unit
|
||||
is missing, do not assume monitoring has stopped: identify your deployment's
|
||||
actual stop control. Docker and other supervisors need their own controls.
|
||||
|
||||
2. **Prevent an update from starting Pulse again.** Do not install or update
|
||||
Pulse during this window. If you use unattended updates, inspect the actual
|
||||
matching timer (the default is `pulse-update.timer`):
|
||||
|
||||
```bash
|
||||
systemctl show pulse-update.timer --property=LoadState,ActiveState
|
||||
```
|
||||
|
||||
Record whether the timer was active. For a loaded timer, pause it and check
|
||||
both it and its update service, substituting your actual unit names:
|
||||
|
||||
```bash
|
||||
sudo systemctl stop pulse-update.timer
|
||||
systemctl show pulse-update.timer pulse-update.service \
|
||||
--property=Id,LoadState,ActiveState,MainPID
|
||||
```
|
||||
|
||||
The timer must be inactive. If the update service is active, activating or
|
||||
deactivating, let it finish normally before continuing. Do not interrupt an
|
||||
installation. A missing update unit is not proof that a custom updater is
|
||||
idle; check your deployment's update controls too.
|
||||
|
||||
3. **Stop Pulse and confirm it is stopped, before the backup starts.**
|
||||
|
||||
```bash
|
||||
sudo systemctl stop pulse.service
|
||||
systemctl show pulse.service --property=LoadState,ActiveState,MainPID
|
||||
```
|
||||
|
||||
Require `LoadState=loaded`, `ActiveState=inactive` and `MainPID=0` after a
|
||||
successful stop. Do not start the backup if the stop fails, the state is
|
||||
unknown or another Pulse server is still polling the affected guests.
|
||||
**Pulse monitoring and alerts are unavailable while stopped.** Do not run
|
||||
an update, start another Pulse instance or reboot its host during the pause.
|
||||
|
||||
4. **Verify the guest independently, then restore only what you paused.**
|
||||
Keep Pulse stopped until the backup has ended **and** your established
|
||||
guest-console or workload checks confirm thaw, including **fresh successful
|
||||
workload writes** to the filesystems covered by the backup. A console
|
||||
connection or a successful read alone is not enough. If thaw cannot be
|
||||
confirmed, leave Pulse stopped and use the guest/platform's recovery
|
||||
procedure; do not repeat the backup or use guest-agent probes to test it.
|
||||
|
||||
Only after that confirmation, if Pulse was active beforehand:
|
||||
|
||||
```bash
|
||||
sudo systemctl start pulse.service
|
||||
systemctl is-active pulse.service
|
||||
```
|
||||
|
||||
`active` confirms service startup, not proof that polling or alerts have
|
||||
recovered. Check normal observation times in Pulse without Run Diagnostics
|
||||
or manual guest-agent probes. Restore the update timer **only if it was
|
||||
active beforehand**, after Pulse has started:
|
||||
|
||||
```bash
|
||||
sudo systemctl start pulse-update.timer
|
||||
```
|
||||
|
||||
Leave previously inactive services/timers inactive. Arrange independent
|
||||
outage coverage and repeat the precaution for each affected backup window;
|
||||
this manual sequence does not schedule future pauses or prove a fix.
|
||||
|
||||
## 🚀 Setup
|
||||
|
||||
Plan changes outside backup windows and follow the guest's normal maintenance
|
||||
|
||||
@@ -220,6 +220,42 @@ afterEach(() => {
|
||||
// ── Tests ──────────────────────────────────────────────────────────────
|
||||
|
||||
describe('GuestDrawer', () => {
|
||||
it('keeps the backup precaution visible across drawer tabs without starting a guest check', async () => {
|
||||
const [guest, setGuest] = createSignal(
|
||||
makeGuest({
|
||||
agentVersion: '9.2',
|
||||
guestAgentStatus: 'deferred',
|
||||
diskStatusReason: 'prev-vm-locked',
|
||||
lock: 'backup',
|
||||
}),
|
||||
);
|
||||
render(() => <GuestDrawer guest={guest()} onClose={vi.fn()} />);
|
||||
const precaution = () => screen.getByTestId('guest-read-precaution');
|
||||
expect(precaution()).toBeVisible();
|
||||
expect(precaution()).toHaveTextContent(
|
||||
'Do not run live diagnostics or restart the guest agent during a backup, freeze/thaw or an unresponsive-guest incident.',
|
||||
);
|
||||
expect(precaution()).toHaveTextContent('does not prove thaw');
|
||||
expect(precaution()).toHaveTextContent('filesystems covered by the backup');
|
||||
expect(
|
||||
within(precaution()).getByRole('link', { name: 'Backup safety guidance' }),
|
||||
).toHaveAttribute('href', '/docs/VM_DISK_MONITORING');
|
||||
fireEvent.click(screen.getByRole('tab', { name: 'History' }));
|
||||
expect(precaution()).toBeVisible();
|
||||
setGuest({
|
||||
...guest(),
|
||||
lock: '',
|
||||
guestAgentStatus: 'expected-unreachable',
|
||||
diskStatusReason: 'agent-timeout',
|
||||
});
|
||||
expect(precaution()).toHaveTextContent('Completion is uncertain');
|
||||
fireEvent.click(screen.getByRole('tab', { name: 'Manage' }));
|
||||
expect(precaution()).toBeVisible();
|
||||
setGuest({ ...guest(), guestAgentStatus: 'available', diskStatusReason: '' });
|
||||
expect(screen.queryByTestId('guest-read-precaution')).not.toBeInTheDocument();
|
||||
expect(discoveryApiMocks.getDiscovery).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('keeps malformed and future backup times unknown through same-guest replacement', () => {
|
||||
const [guest, setGuest] = createSignal(makeGuest({ lastBackup: NaN }));
|
||||
render(() => <GuestDrawer guest={guest()} onClose={vi.fn()} />);
|
||||
@@ -1017,9 +1053,9 @@ describe('GuestDrawer', () => {
|
||||
));
|
||||
|
||||
expect(screen.getByText('Pulse coverage')).toBeInTheDocument();
|
||||
expect(technicalDetails().getByText('Node agent connected')).toHaveAttribute(
|
||||
expect(technicalDetails().getByText('Node agent assigned')).toHaveAttribute(
|
||||
'title',
|
||||
'Discovery and governed actions use the Pulse Agent connected to delly.',
|
||||
'Discovery and governed actions are assigned to the Pulse Agent on delly. Assignment is not a current connection check.',
|
||||
);
|
||||
expect(screen.queryByRole('link', { name: 'Add agent for AI actions' })).toBeNull();
|
||||
});
|
||||
|
||||
@@ -7,12 +7,15 @@ import { ObjectDrawerHeader } from '@/components/shared/ObjectDrawerHeader';
|
||||
import { Subtabs, type SubtabOption } from '@/components/shared/Subtabs';
|
||||
import { getSimpleStatusIndicator } from '@/utils/status';
|
||||
import {
|
||||
GUEST_DRAWER_BACKUP_PRECAUTION,
|
||||
getGuestDrawerGuestReadPrecaution,
|
||||
getGuestDrawerCurrentMetrics,
|
||||
getGuestDrawerDeferredMetrics,
|
||||
type GuestDrawerProps,
|
||||
} from './guestDrawerModel';
|
||||
import { getCanonicalWorkloadId } from '@/utils/workloads';
|
||||
import { InlineNotice } from '@/components/shared/InlineNotice';
|
||||
import { getShippedDocUrl } from '@/utils/docsLinks';
|
||||
import { useGuestDrawerState } from './useGuestDrawerState';
|
||||
import { GuestDrawerHistory, GuestDrawerHistoryRangeSelect } from './GuestDrawerHistory';
|
||||
import { GuestDrawerOverview } from './GuestDrawerOverview';
|
||||
@@ -61,6 +64,8 @@ const GuestDrawerContent: Component<GuestDrawerProps> = (props) => {
|
||||
const historyCurrentMetrics = createMemo(() => getGuestDrawerCurrentMetrics(props.guest));
|
||||
const historyDeferredMetrics = createMemo(() => getGuestDrawerDeferredMetrics(props.guest));
|
||||
|
||||
const guestReadPrecaution = createMemo(() => getGuestDrawerGuestReadPrecaution(props.guest));
|
||||
|
||||
const headerIndicator = createMemo(() => getSimpleStatusIndicator(props.guest.status));
|
||||
|
||||
return (
|
||||
@@ -100,6 +105,20 @@ const GuestDrawerContent: Component<GuestDrawerProps> = (props) => {
|
||||
</InlineNotice>
|
||||
)}
|
||||
</Show>
|
||||
<Show when={guestReadPrecaution()}>
|
||||
{(message) => (
|
||||
<InlineNotice
|
||||
tone="warning"
|
||||
role="status"
|
||||
data-testid="guest-read-precaution"
|
||||
actionHref={getShippedDocUrl('VM_DISK_MONITORING.md')}
|
||||
actionLabel="Backup safety guidance"
|
||||
>
|
||||
<p>{message()}</p>
|
||||
<p>{GUEST_DRAWER_BACKUP_PRECAUTION}</p>
|
||||
</InlineNotice>
|
||||
)}
|
||||
</Show>
|
||||
<Subtabs
|
||||
class="mb-1"
|
||||
ariaLabel="Guest drawer sections"
|
||||
|
||||
@@ -23,6 +23,7 @@ import { buildWorkloadsDiskPresentation } from './diskListModel';
|
||||
import {
|
||||
getGuestDrawerAlertMessage,
|
||||
getGuestDrawerMemoryRows,
|
||||
getGuestDrawerGuestReadPresentation,
|
||||
isGuestDrawerVM,
|
||||
} from './guestDrawerModel';
|
||||
import type { NestedWorkloadContext } from './nestedWorkloadContext';
|
||||
@@ -67,9 +68,10 @@ export function GuestDrawerOverview(props: GuestDrawerOverviewProps) {
|
||||
ageLabel: props.backupPresentation?.ageLabel,
|
||||
ageClass: props.backupPresentation?.ageClass,
|
||||
});
|
||||
const guestReadPresentation = () => getGuestDrawerGuestReadPresentation(props.guest);
|
||||
const coverageLabel = () => {
|
||||
if (props.hasWorkloadActionAgent) return WORKLOAD_ACTION_AGENT_LABEL;
|
||||
if (props.hasAgentInfo) return `${props.agentHeading} connected`;
|
||||
if (props.hasAgentInfo) return `${props.agentHeading} observed`;
|
||||
if (props.showInGuestAgentInstallCue) return 'Agent recommended';
|
||||
return null;
|
||||
};
|
||||
@@ -85,6 +87,12 @@ export function GuestDrawerOverview(props: GuestDrawerOverviewProps) {
|
||||
: props.hasAgentInfo
|
||||
? props.agentTitle
|
||||
: undefined,
|
||||
wrap: true,
|
||||
}),
|
||||
makeDetailRow('Guest-agent reads', guestReadPresentation()?.label, {
|
||||
title: guestReadPresentation()?.detail,
|
||||
tone: guestReadPresentation()?.tone,
|
||||
wrap: true,
|
||||
}),
|
||||
makeDetailRow('Primary IP', props.ipAddresses[0]),
|
||||
// Starting or ending an operation is not evidence of a completed
|
||||
|
||||
+261
@@ -0,0 +1,261 @@
|
||||
import { cleanup, render, screen, within } from '@solidjs/testing-library';
|
||||
import { createSignal, type ComponentProps } from 'solid-js';
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||
import { GuestDrawerOverview } from '../GuestDrawerOverview';
|
||||
import type { WorkloadGuest } from '@/types/workloads';
|
||||
import type { VM } from '@/types/api';
|
||||
import {
|
||||
GUEST_DRAWER_BACKUP_PRECAUTION,
|
||||
getGuestDrawerGuestReadPresentation,
|
||||
getGuestDrawerGuestReadPrecaution,
|
||||
} from '../guestDrawerModel';
|
||||
|
||||
vi.mock('../GuestPhysicalDisks', () => ({ GuestPhysicalDisks: () => null }));
|
||||
|
||||
const guest = (overrides: Partial<WorkloadGuest> = {}): WorkloadGuest => ({
|
||||
id: 'lab:pve-a:101',
|
||||
vmid: 101,
|
||||
name: 'backup-guest',
|
||||
node: 'pve-a',
|
||||
instance: 'lab',
|
||||
status: 'running',
|
||||
type: 'qemu',
|
||||
cpu: 0.1,
|
||||
cpus: 2,
|
||||
memory: { total: 1024 ** 3, used: 256 * 1024 ** 2, free: 768 * 1024 ** 2, usage: 25 },
|
||||
disk: { total: 10 * 1024 ** 3, used: 5 * 1024 ** 3, usage: 50 },
|
||||
networkIn: 0,
|
||||
networkOut: 0,
|
||||
diskRead: 0,
|
||||
diskWrite: 0,
|
||||
uptime: 3600,
|
||||
template: false,
|
||||
tags: [],
|
||||
lock: '',
|
||||
lastBackup: 0,
|
||||
lastSeen: '2026-10-04T14:00:00Z',
|
||||
...overrides,
|
||||
});
|
||||
|
||||
function overviewProps(value: WorkloadGuest): ComponentProps<typeof GuestDrawerOverview> {
|
||||
return {
|
||||
guest: value,
|
||||
guestOsSummary: 'Linux',
|
||||
agentHeading: value.agentKind === 'pulse' ? 'Pulse Agent' : 'Guest agent',
|
||||
agentLabel: value.agentVersion ? `QEMU ${value.agentVersion}` : '',
|
||||
agentTitle: 'Previously observed agent version',
|
||||
hasAgentInfo: Boolean(value.agentVersion),
|
||||
hasFilesystemDetails: false,
|
||||
hasNetworkInterfaces: false,
|
||||
hasOsInfo: true,
|
||||
hasWorkloadActionAgent: false,
|
||||
showInGuestAgentInstallCue: false,
|
||||
ipAddresses: [],
|
||||
networkInterfaces: [],
|
||||
normalizedTags: [],
|
||||
backupPresentation: null,
|
||||
workloadActionAgentTitle: '',
|
||||
};
|
||||
}
|
||||
|
||||
const readRow = () => screen.getByText('Guest-agent reads').closest('tr')!;
|
||||
|
||||
afterEach(cleanup);
|
||||
|
||||
describe('guest agent coverage requires current state, not cached metadata', () => {
|
||||
it.each([
|
||||
['deferred', 'Deferred'],
|
||||
['expected-unreachable', 'Unreachable'],
|
||||
['not-running', 'Not running'],
|
||||
['disabled', 'Disabled'],
|
||||
['available', 'Reported available'],
|
||||
['future-state-private-detail', 'Unknown'],
|
||||
])('shows %s independently of an old version or its absence', (state, label) => {
|
||||
const [value, setValue] = createSignal(guest({ agentVersion: '9.2', guestAgentStatus: state }));
|
||||
render(() => <GuestDrawerOverview {...overviewProps(value())} />);
|
||||
expect(within(readRow()).getByText(label)).toBeVisible();
|
||||
expect(screen.queryByText(/agent connected/i)).not.toBeInTheDocument();
|
||||
expect(screen.queryByText('future-state-private-detail')).not.toBeInTheDocument();
|
||||
setValue({ ...value(), agentVersion: '' });
|
||||
expect(within(readRow()).getByText(label)).toBeVisible();
|
||||
expect(screen.queryByText('QEMU 9.2')).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('does not turn a version-only observation into liveness', () => {
|
||||
render(() => <GuestDrawerOverview {...overviewProps(guest({ agentVersion: '9.2' }))} />);
|
||||
expect(screen.getByText('Guest agent observed')).toBeVisible();
|
||||
expect(within(readRow()).getByText('Unknown')).toBeVisible();
|
||||
expect(screen.queryByText(/connected/i)).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('does not hide deferred guest reads behind parent-node action ownership', () => {
|
||||
render(() => (
|
||||
<GuestDrawerOverview
|
||||
{...overviewProps(guest({ guestAgentStatus: 'deferred' }))}
|
||||
hasWorkloadActionAgent={true}
|
||||
workloadActionAgentTitle="Assigned parent-node action path"
|
||||
/>
|
||||
));
|
||||
expect(screen.getByText('Node agent assigned')).toBeVisible();
|
||||
expect(within(readRow()).getByText('Deferred')).toBeVisible();
|
||||
});
|
||||
|
||||
it('withdraws cached connectivity across the same guest lock, uncertainty and recovery', () => {
|
||||
const [value, setValue] = createSignal(
|
||||
guest({ agentVersion: '9.2', guestAgentStatus: 'available' }),
|
||||
);
|
||||
render(() => <GuestDrawerOverview {...overviewProps(value())} />);
|
||||
expect(within(readRow()).getByText('Reported available')).toBeVisible();
|
||||
setValue({
|
||||
...value(),
|
||||
guestAgentStatus: 'available',
|
||||
lock: 'backup',
|
||||
backupInProgress: true,
|
||||
diskStatusReason: 'prev-vm-locked',
|
||||
});
|
||||
expect(within(readRow()).getByText('Deferred')).toBeVisible();
|
||||
expect(screen.getByText('QEMU 9.2')).toBeVisible();
|
||||
setValue({
|
||||
...value(),
|
||||
lock: '',
|
||||
backupInProgress: false,
|
||||
guestAgentStatus: 'expected-unreachable',
|
||||
diskStatusReason: 'agent-timeout',
|
||||
});
|
||||
expect(within(readRow()).getByText('Unreachable')).toBeVisible();
|
||||
setValue({ ...value(), guestAgentStatus: 'available', diskStatusReason: '' });
|
||||
expect(within(readRow()).getByText('Reported available')).toBeVisible();
|
||||
expect(screen.queryByText(/connected/i)).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('keeps Pulse Agent metadata separate from QEMU read availability', () => {
|
||||
render(() => (
|
||||
<GuestDrawerOverview
|
||||
{...overviewProps(
|
||||
guest({
|
||||
agentKind: 'pulse',
|
||||
agentVersion: '6.4.5',
|
||||
guestAgentStatus: 'expected-unreachable',
|
||||
}),
|
||||
)}
|
||||
/>
|
||||
));
|
||||
expect(screen.getByText('Pulse Agent observed')).toBeVisible();
|
||||
expect(within(readRow()).getByText('Unreachable')).toBeVisible();
|
||||
expect(screen.queryByText('Pulse Agent connected')).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('does not add QEMU state to a non-Proxmox VM or container from a version alone', () => {
|
||||
const [value, setValue] = createSignal(
|
||||
guest({
|
||||
type: 'vm',
|
||||
platformType: 'vmware-vsphere',
|
||||
platformScopes: ['vmware-vsphere'],
|
||||
agentKind: 'pulse',
|
||||
agentVersion: '6.4.5',
|
||||
}),
|
||||
);
|
||||
render(() => <GuestDrawerOverview {...overviewProps(value())} />);
|
||||
expect(screen.getByText('Pulse Agent observed')).toBeVisible();
|
||||
expect(screen.queryByText('Guest-agent reads')).not.toBeInTheDocument();
|
||||
setValue({
|
||||
...value(),
|
||||
type: 'lxc',
|
||||
platformType: 'proxmox-pve',
|
||||
platformScopes: ['proxmox-pve'],
|
||||
});
|
||||
expect(screen.queryByText('Guest-agent reads')).not.toBeInTheDocument();
|
||||
});
|
||||
});
|
||||
|
||||
describe('fixed guest read state and precaution evidence', () => {
|
||||
it.each([
|
||||
'vm-locked',
|
||||
'lock-unverified',
|
||||
'agent-busy',
|
||||
'agent-cooldown',
|
||||
'agent-response-incomplete',
|
||||
'agent-capacity',
|
||||
'invalid-guest-key',
|
||||
'agent-timeout',
|
||||
])('uses %s without a probe or installation diagnosis', (reason) => {
|
||||
for (const prefix of ['', 'prev-']) {
|
||||
const value = guest({ diskStatusReason: prefix + reason });
|
||||
expect(getGuestDrawerGuestReadPresentation(value)).toMatchObject({
|
||||
label: 'Deferred',
|
||||
precaution: true,
|
||||
});
|
||||
const message = getGuestDrawerGuestReadPrecaution(value)!;
|
||||
expect(message).not.toMatch(/Using last known|may not be installed|may need to be restarted/);
|
||||
expect(message).not.toContain('prev-');
|
||||
}
|
||||
});
|
||||
|
||||
it('does not turn a reported available flag into completed read or thaw evidence', () => {
|
||||
const value = guest({ guestAgentStatus: 'available', diskStatusReason: 'agent-timeout' });
|
||||
expect(getGuestDrawerGuestReadPresentation(value)?.label).toBe('Reported available');
|
||||
expect(getGuestDrawerGuestReadPrecaution(value)).toContain('Completion is uncertain');
|
||||
expect(getGuestDrawerGuestReadPresentation(value)?.detail).toContain(
|
||||
'does not independently confirm thaw',
|
||||
);
|
||||
expect(GUEST_DRAWER_BACKUP_PRECAUTION).toContain('running VM does not prove thaw');
|
||||
});
|
||||
|
||||
it('unknown private fields do not become an explanation or an installation recommendation', () => {
|
||||
const value = guest({
|
||||
guestAgentStatus: 'https://private.invalid/token=secret',
|
||||
diskStatusReason: 'raw-private-body',
|
||||
});
|
||||
const presentation = getGuestDrawerGuestReadPresentation(value)!;
|
||||
expect(presentation.label).toBe('Unknown');
|
||||
expect(JSON.stringify(presentation)).not.toMatch(
|
||||
/private|token=|secret|raw-private-body|Install/,
|
||||
);
|
||||
expect(getGuestDrawerGuestReadPrecaution(value)).toBeNull();
|
||||
});
|
||||
|
||||
it('keeps a lock precaution despite absent version, status or disk data', () => {
|
||||
const value = guest({ lock: 'backup', disks: [], agentVersion: '', guestAgentStatus: '' });
|
||||
expect(getGuestDrawerGuestReadPresentation(value)).toMatchObject({
|
||||
label: 'Deferred',
|
||||
precaution: true,
|
||||
});
|
||||
expect(getGuestDrawerGuestReadPrecaution(value)).toContain('VM operation lock');
|
||||
});
|
||||
|
||||
it('keeps state and current measurements independent; it does not relabel memory using a disk reason', () => {
|
||||
const value = guest({ guestAgentStatus: 'deferred', diskStatusReason: 'prev-vm-locked' });
|
||||
const original = structuredClone(value);
|
||||
getGuestDrawerGuestReadPresentation(value);
|
||||
getGuestDrawerGuestReadPrecaution(value);
|
||||
expect(value).toEqual(original);
|
||||
});
|
||||
|
||||
it('does not treat a vSphere operation or container backup as QEMU state', () => {
|
||||
expect(
|
||||
getGuestDrawerGuestReadPresentation(
|
||||
guest({
|
||||
type: 'vm',
|
||||
platformType: 'vmware-vsphere',
|
||||
platformScopes: ['vmware-vsphere'],
|
||||
lock: 'snapshot',
|
||||
}),
|
||||
),
|
||||
).toBeNull();
|
||||
expect(
|
||||
getGuestDrawerGuestReadPresentation(
|
||||
guest({ type: 'lxc', lock: 'backup', guestAgentStatus: 'deferred' }),
|
||||
),
|
||||
).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('backup activity is not a confirmed VM lock', () => {
|
||||
it('warns about a reported backup without inventing a lock or changing the reported flag', () => {
|
||||
const value = guest({ backupInProgress: true, guestAgentStatus: 'available', lock: '' });
|
||||
expect(getGuestDrawerGuestReadPresentation(value)?.label).toBe('Backup in progress');
|
||||
expect(getGuestDrawerGuestReadPrecaution(value)).toContain('A backup is reported in progress');
|
||||
expect(getGuestDrawerGuestReadPrecaution(value)).not.toContain('VM operation lock');
|
||||
expect((value as VM).guestAgentStatus).toBe('available');
|
||||
});
|
||||
});
|
||||
+5
-5
@@ -134,35 +134,35 @@ describe('workloadAgentReadiness (branch coverage)', () => {
|
||||
it('interpolates the node when node is a non-empty string (truthy ternary arm)', () => {
|
||||
const guest = makeGuest({ node: 'pve' });
|
||||
expect(getWorkloadActionAgentTitle(guest)).toBe(
|
||||
'Discovery and governed actions use the Pulse Agent connected to pve.',
|
||||
'Discovery and governed actions are assigned to the Pulse Agent on pve. Assignment is not a current connection check.',
|
||||
);
|
||||
});
|
||||
|
||||
it('trims surrounding whitespace from node before interpolating', () => {
|
||||
const guest = makeGuest({ node: ' pve-node ' });
|
||||
expect(getWorkloadActionAgentTitle(guest)).toBe(
|
||||
'Discovery and governed actions use the Pulse Agent connected to pve-node.',
|
||||
'Discovery and governed actions are assigned to the Pulse Agent on pve-node. Assignment is not a current connection check.',
|
||||
);
|
||||
});
|
||||
|
||||
it('falls back to the parent-node title when node is an empty string (falsy ternary arm)', () => {
|
||||
const guest = makeGuest({ node: '' });
|
||||
expect(getWorkloadActionAgentTitle(guest)).toBe(
|
||||
'Discovery and governed actions use the connected parent node Pulse Agent.',
|
||||
'Discovery and governed actions are assigned to the parent node Pulse Agent. Assignment is not a current connection check.',
|
||||
);
|
||||
});
|
||||
|
||||
it('falls back to the parent-node title when node is whitespace-only (trim -> empty)', () => {
|
||||
const guest = makeGuest({ node: ' ' });
|
||||
expect(getWorkloadActionAgentTitle(guest)).toBe(
|
||||
'Discovery and governed actions use the connected parent node Pulse Agent.',
|
||||
'Discovery and governed actions are assigned to the parent node Pulse Agent. Assignment is not a current connection check.',
|
||||
);
|
||||
});
|
||||
|
||||
it('falls back to the parent-node title when node is undefined (|| "" coalesce arm)', () => {
|
||||
const guest = makeGuest({ node: undefined });
|
||||
expect(getWorkloadActionAgentTitle(guest)).toBe(
|
||||
'Discovery and governed actions use the connected parent node Pulse Agent.',
|
||||
'Discovery and governed actions are assigned to the parent node Pulse Agent. Assignment is not a current connection check.',
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -359,6 +359,129 @@ export const getGuestDrawerAgentTitle = (guest: Guest): string => {
|
||||
export const getGuestDrawerAgentHeading = (guest: Guest): string =>
|
||||
guest.agentKind === 'pulse' ? 'Pulse Agent' : 'Guest agent';
|
||||
|
||||
export interface GuestDrawerGuestReadPresentation {
|
||||
label: string;
|
||||
detail: string;
|
||||
tone: 'muted' | 'warning';
|
||||
precaution: boolean;
|
||||
}
|
||||
|
||||
export const GUEST_DRAWER_BACKUP_PRECAUTION =
|
||||
'Do not run live diagnostics or restart the guest agent during a backup, freeze/thaw or an unresponsive-guest incident. An OK backup or a running VM does not prove thaw. Confirm thaw and writes to the filesystems covered by the backup independently.';
|
||||
|
||||
const guestReadDeferrals = new Set([
|
||||
'vm-locked',
|
||||
'lock-unverified',
|
||||
'agent-busy',
|
||||
'agent-cooldown',
|
||||
'agent-response-incomplete',
|
||||
'agent-capacity',
|
||||
'invalid-guest-key',
|
||||
'agent-timeout',
|
||||
]);
|
||||
|
||||
// Version/OS metadata can survive a lock or failed read. Neither it nor a
|
||||
// parent-agent action target establishes QGA liveness. Use only the current
|
||||
// provider state and fixed read reasons; never print an unknown raw value.
|
||||
export const getGuestDrawerGuestReadPresentation = (
|
||||
guest: Guest,
|
||||
): GuestDrawerGuestReadPresentation | null => {
|
||||
if (!isGuestDrawerVM(guest)) return null;
|
||||
const hasProxmoxEvidence =
|
||||
guest.type === 'qemu' ||
|
||||
guest.agentKind === 'qemu-guest' ||
|
||||
guest.platformType === 'proxmox-pve' ||
|
||||
guest.platformScopes?.includes('proxmox-pve') ||
|
||||
Boolean(guest.guestAgentStatus || guest.diskStatusReason);
|
||||
if (!hasProxmoxEvidence) return null;
|
||||
|
||||
const reason = (guest.diskStatusReason || '').replace(/^prev-/, '');
|
||||
const readDeferred = guestReadDeferrals.has(reason);
|
||||
const state = (guest.guestAgentStatus || '').trim().toLowerCase();
|
||||
if ((guest.lock || '').trim()) {
|
||||
return {
|
||||
label: 'Deferred',
|
||||
tone: 'warning',
|
||||
precaution: true,
|
||||
detail: getWorkloadGuestDiskStatusMessage('vm-locked'),
|
||||
};
|
||||
}
|
||||
if (guest.backupInProgress) {
|
||||
return {
|
||||
label: 'Backup in progress',
|
||||
tone: 'warning',
|
||||
precaution: true,
|
||||
detail:
|
||||
'A backup is reported in progress. Guest-agent availability does not establish safe live checks during the backup.',
|
||||
};
|
||||
}
|
||||
if (state === 'deferred' || (!state && readDeferred)) {
|
||||
return {
|
||||
label: 'Deferred',
|
||||
tone: 'warning',
|
||||
precaution: true,
|
||||
detail: readDeferred
|
||||
? getWorkloadGuestDiskStatusMessage(reason)
|
||||
: 'Guest reads are deferred. Previously observed details do not prove a current connection.',
|
||||
};
|
||||
}
|
||||
const precaution = readDeferred;
|
||||
switch (state) {
|
||||
case 'available':
|
||||
return {
|
||||
label: 'Reported available',
|
||||
tone: 'muted',
|
||||
precaution,
|
||||
detail:
|
||||
'Proxmox reports guest-agent availability. This does not independently confirm thaw or filesystem writes.',
|
||||
};
|
||||
case 'expected-unreachable':
|
||||
return {
|
||||
label: 'Unreachable',
|
||||
tone: 'warning',
|
||||
precaution,
|
||||
detail:
|
||||
'Proxmox expects the guest agent but cannot reach it. Previously observed details do not prove a current connection.',
|
||||
};
|
||||
case 'not-running':
|
||||
return {
|
||||
label: 'Not running',
|
||||
tone: 'warning',
|
||||
precaution,
|
||||
detail:
|
||||
'Proxmox reports that the guest agent is not running. Previously observed details do not prove a current connection.',
|
||||
};
|
||||
case 'disabled':
|
||||
return {
|
||||
label: 'Disabled',
|
||||
tone: 'muted',
|
||||
precaution,
|
||||
detail:
|
||||
'Proxmox reports that the guest agent is disabled. Previously observed details do not prove a current connection.',
|
||||
};
|
||||
default:
|
||||
return {
|
||||
label: 'Unknown',
|
||||
tone: 'muted',
|
||||
precaution,
|
||||
detail:
|
||||
'Pulse has no recognised current guest-agent state. Previously observed details do not prove a current connection.',
|
||||
};
|
||||
}
|
||||
};
|
||||
|
||||
// A read-specific precaution is independent of the reported availability flag.
|
||||
// For example Proxmox can report QGA enabled while its last request timed out.
|
||||
export const getGuestDrawerGuestReadPrecaution = (guest: Guest): string | null => {
|
||||
if (!isGuestDrawerVM(guest)) return null;
|
||||
const presentation = getGuestDrawerGuestReadPresentation(guest);
|
||||
if (!presentation?.precaution) return null;
|
||||
const reason = (guest.diskStatusReason || '').replace(/^prev-/, '');
|
||||
return guestReadDeferrals.has(reason)
|
||||
? getWorkloadGuestDiskStatusMessage(reason)
|
||||
: presentation.detail;
|
||||
};
|
||||
|
||||
export interface GuestDrawerMemoryRow {
|
||||
label: string;
|
||||
value: string;
|
||||
|
||||
@@ -42,10 +42,10 @@ export const IN_GUEST_AGENT_INSTALL_SUMMARY_LABEL = 'Install agent';
|
||||
export const IN_GUEST_AGENT_INSTALL_ACTION_LABEL = 'Add agent for AI actions';
|
||||
export const IN_GUEST_AGENT_INSTALL_TITLE =
|
||||
'Install Pulse Agent for this workload path to unlock deep telemetry and AI actions.';
|
||||
export const WORKLOAD_ACTION_AGENT_LABEL = 'Node agent connected';
|
||||
export const WORKLOAD_ACTION_AGENT_LABEL = 'Node agent assigned';
|
||||
export const getWorkloadActionAgentTitle = (guest: WorkloadGuest): string => {
|
||||
const node = (guest.node || '').trim();
|
||||
return node
|
||||
? `Discovery and governed actions use the Pulse Agent connected to ${node}.`
|
||||
: 'Discovery and governed actions use the connected parent node Pulse Agent.';
|
||||
? `Discovery and governed actions are assigned to the Pulse Agent on ${node}. Assignment is not a current connection check.`
|
||||
: 'Discovery and governed actions are assigned to the parent node Pulse Agent. Assignment is not a current connection check.';
|
||||
};
|
||||
|
||||
@@ -72,7 +72,7 @@ function buildBackupCounts(backups: readonly PBSBackup[]): Map<string, number> {
|
||||
|
||||
function serverIsOnline(resource: Resource): boolean {
|
||||
const status = (resource.status ?? '').toLowerCase();
|
||||
const conn = (resource.pbs?.connectionHealth ?? '').toLowerCase();
|
||||
const conn = (resource.pbs?.connectionHealth ?? '').trim().toLowerCase();
|
||||
if (conn) return conn === 'healthy' || conn === 'ok';
|
||||
return status === 'online' || status === 'running';
|
||||
}
|
||||
@@ -83,10 +83,39 @@ function connectionLabel(resource: Resource): string {
|
||||
return serverIsOnline(resource) ? 'Online' : 'Offline';
|
||||
}
|
||||
|
||||
function usagePercent(datastore: ResourcePBSDatastore): number | undefined {
|
||||
if (typeof datastore.usagePercent === 'number') return datastore.usagePercent;
|
||||
if (datastore.total > 0) return (datastore.used / datastore.total) * 100;
|
||||
return undefined;
|
||||
// PBS retains a named datastore when its status read fails, with zero-valued
|
||||
// capacity and explicit unavailable/error evidence. Those zeros are not an
|
||||
// empty datastore. Reachability alone cannot establish a successful store read.
|
||||
function datastoreCapacity(datastore: ResourcePBSDatastore | undefined, online: boolean) {
|
||||
if (!datastore) return { state: 'absent' as const, percent: undefined };
|
||||
const status =
|
||||
typeof datastore.status === 'string' ? datastore.status.trim().toLowerCase() : datastore.status;
|
||||
if (
|
||||
!online ||
|
||||
(typeof datastore.error === 'string' && Boolean(datastore.error.trim())) ||
|
||||
['unavailable', 'offline', 'error', 'failed'].includes(status ?? '')
|
||||
) {
|
||||
return { state: 'unavailable' as const, percent: undefined };
|
||||
}
|
||||
// Empty status is supported by older payloads; an unrecognised explicit
|
||||
// status must not be interpreted as current healthy capacity.
|
||||
if (
|
||||
(status && !['available', 'online', 'healthy', 'ok'].includes(status)) ||
|
||||
!Number.isFinite(datastore.total) ||
|
||||
datastore.total <= 0 ||
|
||||
!Number.isFinite(datastore.used) ||
|
||||
datastore.used < 0
|
||||
) {
|
||||
return { state: 'unknown' as const, percent: undefined };
|
||||
}
|
||||
const percent =
|
||||
datastore.usagePercent !== undefined
|
||||
? datastore.usagePercent
|
||||
: (datastore.used / datastore.total) * 100;
|
||||
if (!Number.isFinite(percent) || percent < 0) {
|
||||
return { state: 'unknown' as const, percent: undefined };
|
||||
}
|
||||
return { state: 'available' as const, percent };
|
||||
}
|
||||
|
||||
// >=90% is the silent-backup-failure danger zone; >=75% is the early warning.
|
||||
@@ -480,7 +509,13 @@ export function ProxmoxBackupServersTable(props: {
|
||||
estimatedRowHeight={32}
|
||||
>
|
||||
{(row) => {
|
||||
const pct = () => (row.datastore ? usagePercent(row.datastore) : undefined);
|
||||
// Read admission is not a generic storage-adapter capacity snapshot.
|
||||
const capacityEvidence = () => datastoreCapacity(row.datastore, row.online);
|
||||
const pct = () => capacityEvidence().percent;
|
||||
const capacityNotice = () =>
|
||||
capacityEvidence().state === 'unavailable'
|
||||
? 'Datastore capacity is unavailable. This does not mean the datastore is empty.'
|
||||
: 'Datastore capacity is unknown. This does not mean the datastore is empty.';
|
||||
const rowIdentity = { id: row.key };
|
||||
const isExpanded = () => detail.isExpanded(rowIdentity);
|
||||
const detailRowId = () => detail.detailRowId(rowIdentity);
|
||||
@@ -510,13 +545,18 @@ export function ProxmoxBackupServersTable(props: {
|
||||
</TableCell>
|
||||
<TableCell class={getPlatformTableCellClassForKind('text')}>
|
||||
<div class="flex items-center gap-2">
|
||||
<StatusDot
|
||||
size="sm"
|
||||
variant={row.online ? 'success' : 'danger'}
|
||||
<Show when={layoutMode() !== 'compact'}>
|
||||
<StatusDot
|
||||
size="sm"
|
||||
variant={row.online ? 'success' : 'danger'}
|
||||
title={row.connectionLabel}
|
||||
ariaHidden
|
||||
/>
|
||||
</Show>
|
||||
<span
|
||||
class="truncate text-[11px] text-base-content"
|
||||
title={row.connectionLabel}
|
||||
ariaHidden
|
||||
/>
|
||||
<span class="truncate text-[11px] text-base-content">
|
||||
>
|
||||
{row.connectionLabel}
|
||||
</span>
|
||||
</div>
|
||||
@@ -579,6 +619,7 @@ export function ProxmoxBackupServersTable(props: {
|
||||
<Show when={columnVisible('datastore')}>
|
||||
<TableCell
|
||||
class={`${getPlatformTableCellClassForKind('text')} text-base-content truncate font-mono text-[11px]`}
|
||||
title={row.datastore?.name}
|
||||
>
|
||||
{row.datastore?.name ?? '—'}
|
||||
</TableCell>
|
||||
@@ -589,25 +630,41 @@ export function ProxmoxBackupServersTable(props: {
|
||||
fallback={<span class="text-muted">No datastore data</span>}
|
||||
>
|
||||
{(datastore) => (
|
||||
<div class="flex items-center justify-end gap-2">
|
||||
<Show when={layoutMode() !== 'compact' && layoutMode() !== 'basic'}>
|
||||
<StatusDot
|
||||
size="sm"
|
||||
variant={usageVariant(pct())}
|
||||
title={`Datastore ${formatPlatformTablePercentValue(pct())} used`}
|
||||
ariaHidden
|
||||
/>
|
||||
</Show>
|
||||
<span class={`tabular-nums font-medium ${usageToneClass(pct())}`}>
|
||||
<PlatformTablePercentValue value={pct()} />
|
||||
</span>
|
||||
<Show when={layoutMode() !== 'compact'}>
|
||||
<span class="truncate text-[10px] text-muted tabular-nums">
|
||||
{formatPlatformTableBytesValue(datastore().used, '0 B')} /{' '}
|
||||
{formatPlatformTableBytesValue(datastore().total)}
|
||||
<Show
|
||||
when={capacityEvidence().state === 'available'}
|
||||
fallback={
|
||||
<span
|
||||
class="text-[11px] text-amber-600 dark:text-amber-300"
|
||||
title={capacityNotice()}
|
||||
>
|
||||
{capacityEvidence().state === 'unavailable'
|
||||
? 'Unavailable'
|
||||
: 'Unknown'}
|
||||
</span>
|
||||
</Show>
|
||||
</div>
|
||||
}
|
||||
>
|
||||
<div class="flex items-center justify-end gap-2">
|
||||
<Show
|
||||
when={layoutMode() !== 'compact' && layoutMode() !== 'basic'}
|
||||
>
|
||||
<StatusDot
|
||||
size="sm"
|
||||
variant={usageVariant(pct())}
|
||||
title={`Datastore ${formatPlatformTablePercentValue(pct())} used`}
|
||||
ariaHidden
|
||||
/>
|
||||
</Show>
|
||||
<span class={`tabular-nums font-medium ${usageToneClass(pct())}`}>
|
||||
<PlatformTablePercentValue value={pct()} />
|
||||
</span>
|
||||
<Show when={layoutMode() !== 'compact'}>
|
||||
<span class="truncate text-[10px] text-muted tabular-nums">
|
||||
{formatPlatformTableBytesValue(datastore().used, '0 B')} /{' '}
|
||||
{formatPlatformTableBytesValue(datastore().total)}
|
||||
</span>
|
||||
</Show>
|
||||
</div>
|
||||
</Show>
|
||||
)}
|
||||
</Show>
|
||||
</TableCell>
|
||||
@@ -645,10 +702,15 @@ export function ProxmoxBackupServersTable(props: {
|
||||
class={`${getPlatformTableCellClassForKind('numeric-value')} text-muted tabular-nums text-[11px]`}
|
||||
>
|
||||
<Show
|
||||
when={row.datastore?.deduplicationFactor}
|
||||
when={
|
||||
capacityEvidence().state === 'available' &&
|
||||
typeof row.datastore?.deduplicationFactor === 'number' &&
|
||||
Number.isFinite(row.datastore.deduplicationFactor) &&
|
||||
row.datastore.deduplicationFactor > 0
|
||||
}
|
||||
fallback={<span class="text-muted">—</span>}
|
||||
>
|
||||
{(factor) => <>{factor().toFixed(1)}×</>}
|
||||
{row.datastore?.deduplicationFactor?.toFixed(1)}×
|
||||
</Show>
|
||||
</TableCell>
|
||||
</Show>
|
||||
|
||||
@@ -256,9 +256,9 @@ export const ProxmoxBackupsTable: Component<{
|
||||
const [expandedCoverageRows, setExpandedCoverageRows] = createSignal<ReadonlySet<string>>(
|
||||
new Set<string>(),
|
||||
);
|
||||
// Orphaned backups (VM/LXC records whose guest no longer exists in inventory)
|
||||
// Unmatched backups (VM/LXC evidence without an unambiguous live identity)
|
||||
// are collapsed by default so the main table is active backup targets, not a
|
||||
// pile of nameless dead records sorted to the top.
|
||||
// pile of unassigned artifacts sorted to the top.
|
||||
const [showOrphaned, setShowOrphaned] = createSignal(false);
|
||||
|
||||
// Chronological feed sort and activity-chart controls.
|
||||
@@ -746,7 +746,7 @@ export const ProxmoxBackupsTable: Component<{
|
||||
</Show>
|
||||
<Show when={orphanedTotalCount() > 0}>
|
||||
{' · '}
|
||||
{orphanedTotalCount()} orphaned
|
||||
{orphanedTotalCount()} unmatched
|
||||
</Show>
|
||||
</span>
|
||||
}
|
||||
@@ -936,7 +936,7 @@ export const ProxmoxBackupsTable: Component<{
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => setShowOrphaned((v) => !v)}
|
||||
class="flex w-full items-center justify-between gap-2 px-3 py-2 text-left"
|
||||
class="flex w-full flex-wrap items-center justify-between gap-2 px-3 py-2 text-left"
|
||||
aria-expanded={showOrphaned()}
|
||||
>
|
||||
<span class="flex items-center gap-2 text-[11px] font-semibold uppercase tracking-[0.18em] text-muted">
|
||||
@@ -944,10 +944,12 @@ export const ProxmoxBackupsTable: Component<{
|
||||
class={`h-3.5 w-3.5 transition-transform ${showOrphaned() ? 'rotate-90' : ''}`}
|
||||
aria-hidden="true"
|
||||
/>
|
||||
{orphanedTotalCount()} orphaned{' '}
|
||||
{orphanedTotalCount()} unmatched{' '}
|
||||
{orphanedTotalCount() === 1 ? 'backup' : 'backups'}
|
||||
</span>
|
||||
<span class="text-[11px] text-muted">guest no longer exists in inventory</span>
|
||||
<span class="w-full text-[11px] text-muted sm:w-auto">
|
||||
No unambiguous inventory match
|
||||
</span>
|
||||
</button>
|
||||
<Show when={showOrphaned()}>
|
||||
<div class="border-t border-border-subtle p-2">
|
||||
@@ -955,8 +957,8 @@ export const ProxmoxBackupsTable: Component<{
|
||||
rows={orphanedCoverageRows()}
|
||||
hasAnyRows={orphanedTotalCount() > 0}
|
||||
emptyIcon={props.emptyIcon}
|
||||
emptyTitle="No orphaned backups"
|
||||
emptyDescription="Backups whose guest no longer exists will appear here."
|
||||
emptyTitle="No unmatched backups"
|
||||
emptyDescription="Backups without an unambiguous live guest match will appear here."
|
||||
sortKey={coverageSortKey}
|
||||
sortDirection={coverageSortDirection}
|
||||
onSort={handleCoverageSort}
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
import type { BackupTask, GuestSnapshot, PBSBackup, StorageBackup } from '@/types/api';
|
||||
import type { Resource } from '@/types/resource';
|
||||
|
||||
export const BACKUP_IDENTITY_NOW = Date.parse('2026-10-04T12:00:00Z');
|
||||
|
||||
export const identityGuest = (
|
||||
id: string,
|
||||
instance: string,
|
||||
node: string,
|
||||
extra: Partial<NonNullable<Resource['proxmox']>> = {},
|
||||
): Resource =>
|
||||
({
|
||||
id,
|
||||
name: id,
|
||||
type: 'vm',
|
||||
platformId: instance,
|
||||
platformType: 'proxmox-pve',
|
||||
sourceType: 'api',
|
||||
status: 'running',
|
||||
lastSeen: BACKUP_IDENTITY_NOW,
|
||||
proxmox: { vmid: 100, instance, nodeName: node, ...extra },
|
||||
}) as Resource;
|
||||
|
||||
export const identityArchive = (extra: Partial<StorageBackup> = {}): StorageBackup => ({
|
||||
id: 'file-100',
|
||||
instance: 'east',
|
||||
node: 'pve-1',
|
||||
type: 'vm',
|
||||
vmid: 100,
|
||||
storage: 'local',
|
||||
time: '2026-10-04T11:00:00Z',
|
||||
ctime: BACKUP_IDENTITY_NOW / 1000 - 3600,
|
||||
size: 1024,
|
||||
format: 'zst',
|
||||
volid: 'local:backup/vzdump-qemu-100.vma.zst',
|
||||
protected: false,
|
||||
isPBS: false,
|
||||
verified: false,
|
||||
...extra,
|
||||
});
|
||||
|
||||
export const identitySnapshot = (extra: Partial<GuestSnapshot> = {}): GuestSnapshot => ({
|
||||
id: 'snapshot-100',
|
||||
instance: 'east',
|
||||
node: 'pve-1',
|
||||
type: 'vm',
|
||||
vmid: 100,
|
||||
name: 'before-update',
|
||||
time: '2026-10-04T11:30:00Z',
|
||||
vmstate: false,
|
||||
...extra,
|
||||
});
|
||||
|
||||
export const identityPBS = (extra: Partial<PBSBackup> = {}): PBSBackup => ({
|
||||
id: 'pbs-100',
|
||||
instance: 'pbs-server',
|
||||
namespace: 'pve-1',
|
||||
datastore: 'main',
|
||||
backupType: 'vm',
|
||||
vmid: '100',
|
||||
backupTime: '2026-10-04T10:00:00Z',
|
||||
size: 2048,
|
||||
files: ['index.json.blob'],
|
||||
protected: false,
|
||||
verified: true,
|
||||
...extra,
|
||||
});
|
||||
|
||||
export const identityTask = (extra: Partial<BackupTask> = {}): BackupTask => ({
|
||||
id: 'task-100',
|
||||
instance: 'east',
|
||||
node: 'pve-1',
|
||||
type: 'vzdump',
|
||||
vmid: 100,
|
||||
status: 'OK',
|
||||
startTime: '2026-10-04T11:00:00Z',
|
||||
endTime: '2026-10-04T11:05:00Z',
|
||||
...extra,
|
||||
});
|
||||
+147
@@ -0,0 +1,147 @@
|
||||
import { cleanup, render, waitFor, within } from '@solidjs/testing-library';
|
||||
import { createSignal } from 'solid-js';
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||
import type { Resource, ResourcePBSDatastore } from '@/types/resource';
|
||||
import { canonicalizeRealtimeResource } from '@/utils/resourceStateAdapters';
|
||||
import { ProxmoxBackupServersTable } from '../ProxmoxBackupServersTable';
|
||||
|
||||
vi.mock('@/components/Infrastructure/ResourceDetailDrawer', () => ({
|
||||
ResourceDetailDrawer: () => <div>Details</div>,
|
||||
}));
|
||||
|
||||
afterEach(cleanup);
|
||||
|
||||
const datastore = (patch: Record<string, unknown> = {}): ResourcePBSDatastore =>
|
||||
({
|
||||
name: 'tank',
|
||||
total: 1000,
|
||||
used: 400,
|
||||
available: 600,
|
||||
status: 'available',
|
||||
deduplicationFactor: 2,
|
||||
...patch,
|
||||
}) as ResourcePBSDatastore;
|
||||
|
||||
const server = (store?: ResourcePBSDatastore, health = 'healthy'): Resource => ({
|
||||
id: 'pbs-capacity',
|
||||
type: 'pbs',
|
||||
name: 'pbs-capacity',
|
||||
displayName: 'pbs-capacity',
|
||||
status: 'online',
|
||||
platformId: 'pbs-capacity',
|
||||
platformType: 'proxmox-pbs',
|
||||
sourceType: 'api',
|
||||
lastSeen: 1_790_000_000_000,
|
||||
pbs: { instanceId: 'pbs-capacity', connectionHealth: health, datastores: store ? [store] : [] },
|
||||
});
|
||||
|
||||
function usedCell(container: HTMLElement) {
|
||||
const table = container.querySelector('table')!;
|
||||
const index = Array.from(table.querySelectorAll('th')).findIndex(
|
||||
(head) => head.textContent === 'Used',
|
||||
);
|
||||
return table.querySelector('tbody tr')!.querySelectorAll('td')[index] as HTMLElement;
|
||||
}
|
||||
|
||||
describe('PBS datastore capacity evidence', () => {
|
||||
it.each([
|
||||
[
|
||||
'provider failure zeros',
|
||||
{ total: 0, used: 0, available: 0, usagePercent: 0, status: 'unavailable' },
|
||||
'Unavailable',
|
||||
],
|
||||
['retained failure values', { status: 'unavailable', usagePercent: 40 }, 'Unavailable'],
|
||||
[
|
||||
'explicit error with healthy status',
|
||||
{ error: 'PRIVATE_PROVIDER_ERROR_SENTINEL' },
|
||||
'Unavailable',
|
||||
],
|
||||
['unknown status', { status: 'unknown', usagePercent: 0 }, 'Unknown'],
|
||||
['future status', { status: 'new-provider-state' }, 'Unknown'],
|
||||
['zero total', { total: 0, used: 0, usagePercent: 0 }, 'Unknown'],
|
||||
['missing used', { used: undefined }, 'Unknown'],
|
||||
['negative used', { used: -10 }, 'Unknown'],
|
||||
['nonfinite total', { total: Infinity }, 'Unknown'],
|
||||
['invalid authoritative percent', { usagePercent: NaN }, 'Unknown'],
|
||||
['negative authoritative percent', { usagePercent: -1 }, 'Unknown'],
|
||||
['null authoritative percent', { usagePercent: null }, 'Unknown'],
|
||||
['string authoritative percent', { usagePercent: '0' }, 'Unknown'],
|
||||
['overflowed ratio', { used: Number.MAX_VALUE, total: Number.MIN_VALUE }, 'Unknown'],
|
||||
])('shows %s as unknown evidence rather than healthy usage', async (_name, patch, label) => {
|
||||
const { container } = render(() => (
|
||||
<ProxmoxBackupServersTable servers={[server(datastore(patch))]} layoutWidth={() => 1200} />
|
||||
));
|
||||
const cell = usedCell(container);
|
||||
await waitFor(() => expect(within(cell).getByText(label)).toBeInTheDocument());
|
||||
expect(cell.textContent).not.toMatch(/%|0 B/);
|
||||
expect(cell.querySelector('.bg-emerald-500')).toBeNull();
|
||||
expect(cell.querySelector('[title]')?.getAttribute('title')).toContain(
|
||||
'does not mean the datastore is empty',
|
||||
);
|
||||
expect(container.textContent).not.toContain('2.0×');
|
||||
expect(container.innerHTML).not.toContain('PRIVATE_PROVIDER_ERROR_SENTINEL');
|
||||
expect(within(container).getByText('Healthy')).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('withdraws retained capacity and dedup when the PBS connection is offline', () => {
|
||||
const { container } = render(() => (
|
||||
<ProxmoxBackupServersTable
|
||||
servers={[server(datastore(), 'offline')]}
|
||||
layoutWidth={() => 1200}
|
||||
/>
|
||||
));
|
||||
expect(usedCell(container).textContent).toBe('Unavailable');
|
||||
expect(container.textContent).not.toContain('2.0×');
|
||||
});
|
||||
|
||||
it.each([
|
||||
[0, '0.0%', 'bg-emerald-500'],
|
||||
[750, '75.0%', 'bg-amber-500'],
|
||||
[900, '90.0%', 'bg-red-500'],
|
||||
[1100, '110.0%', 'bg-red-500'],
|
||||
])('keeps measured usage %s and the existing warning thresholds', async (used, label, tone) => {
|
||||
const { container } = render(() => (
|
||||
<ProxmoxBackupServersTable servers={[server(datastore({ used }))]} layoutWidth={() => 1200} />
|
||||
));
|
||||
const cell = usedCell(container);
|
||||
await waitFor(() => expect(within(cell).getByText(label)).toBeInTheDocument());
|
||||
expect(cell.querySelector(`.${tone}`)).not.toBeNull();
|
||||
expect(container.textContent).toContain('2.0×');
|
||||
});
|
||||
|
||||
it('keeps a server without datastore inventory distinct from an unknown named store', () => {
|
||||
const { container } = render(() => (
|
||||
<ProxmoxBackupServersTable servers={[server()]} layoutWidth={() => 1200} />
|
||||
));
|
||||
expect(usedCell(container).textContent).toBe('No datastore data');
|
||||
});
|
||||
|
||||
it.each(['direct', 'canonical'] as const)(
|
||||
'reconciles same-identity %s failure and recovery without remounting',
|
||||
async (transport) => {
|
||||
const present = (store: ResourcePBSDatastore): Resource => {
|
||||
const resource = server(store);
|
||||
if (transport === 'direct') return resource;
|
||||
const { pbs, ...base } = resource;
|
||||
return canonicalizeRealtimeResource({ ...base, platformData: { sources: ['pbs'], pbs } });
|
||||
};
|
||||
const [servers, setServers] = createSignal([present(datastore())]);
|
||||
const { container } = render(() => (
|
||||
<ProxmoxBackupServersTable servers={servers()} layoutWidth={() => 1200} />
|
||||
));
|
||||
const cell = usedCell(container);
|
||||
await waitFor(() => expect(cell.textContent).toContain('40.0%'));
|
||||
setServers([
|
||||
present(datastore({ total: 0, used: 0, usagePercent: 0, status: 'unavailable' })),
|
||||
]);
|
||||
await waitFor(() => expect(cell.textContent).toBe('Unavailable'));
|
||||
expect(usedCell(container)).toBe(cell);
|
||||
setServers([present(datastore({ used: 0, usagePercent: 0 }))]);
|
||||
await waitFor(() => expect(cell.textContent).toContain('0.0%'));
|
||||
expect(usedCell(container)).toBe(cell);
|
||||
setServers([present(datastore({ used: 950, usagePercent: 95 }))]);
|
||||
await waitFor(() => expect(cell.textContent).toContain('95.0%'));
|
||||
expect(cell.querySelector('.bg-red-500')).not.toBeNull();
|
||||
},
|
||||
);
|
||||
});
|
||||
+3
-1
@@ -100,7 +100,9 @@ describe('ProxmoxBackupServersTable details', () => {
|
||||
const [servers, setServers] = createSignal(makeServers());
|
||||
const { container } = render(() => <ProxmoxBackupServersTable servers={servers()} />);
|
||||
const datastoreNames = () =>
|
||||
Array.from(container.querySelectorAll('td[title]')).map((cell) => cell.getAttribute('title'));
|
||||
Array.from(container.querySelectorAll('tbody tr td:first-child[title]')).map((cell) =>
|
||||
cell.getAttribute('title'),
|
||||
);
|
||||
fireEvent.click(screen.getAllByRole('button', { name: 'Expand details for pbs-main' })[1]);
|
||||
const detail = screen.getByTestId('pbs-resource-detail');
|
||||
expect(datastoreNames()).toEqual(['pbs-main · archive', 'pbs-main · tank']);
|
||||
|
||||
@@ -30,6 +30,25 @@ const makePbsResource = (overrides: Partial<Resource> = {}): Resource =>
|
||||
}) as Resource;
|
||||
|
||||
describe('buildBackupServerRows', () => {
|
||||
it('retains failed datastore-read evidence without changing server or row identity', () => {
|
||||
const server = makePbsResource();
|
||||
server.pbs!.datastores![0] = {
|
||||
name: 'tank',
|
||||
total: 0,
|
||||
used: 0,
|
||||
available: 0,
|
||||
usagePercent: 0,
|
||||
status: 'unavailable',
|
||||
error: 'synthetic status failure',
|
||||
};
|
||||
const rows = buildBackupServerRows([server]);
|
||||
const tank = rows.find((row) => row.datastore?.name === 'tank')!;
|
||||
expect(tank.key).toBe('pbs-1:tank');
|
||||
expect(tank.online).toBe(true);
|
||||
expect(tank.datastore).toEqual(server.pbs!.datastores![0]);
|
||||
expect(rows.find((row) => row.datastore?.name === 'offsite')?.datastore?.usagePercent).toBe(95);
|
||||
});
|
||||
|
||||
it('carries host CPU, memory, and uptime onto every datastore row of the server', () => {
|
||||
const rows = buildBackupServerRows([makePbsResource()]);
|
||||
|
||||
|
||||
@@ -4,6 +4,14 @@ import { createSignal, type JSX } from 'solid-js';
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { ProxmoxBackupsTable } from '../ProxmoxBackupsTable';
|
||||
import {
|
||||
BACKUP_IDENTITY_NOW,
|
||||
identityGuest,
|
||||
identityArchive,
|
||||
identityPBS,
|
||||
identitySnapshot,
|
||||
identityTask,
|
||||
} from '../__fixtures__/backupIdentity';
|
||||
import { buildBackupServerRows } from '../ProxmoxBackupServersTable';
|
||||
import proxmoxBackupServersTableSource from '../ProxmoxBackupServersTable.tsx?raw';
|
||||
import proxmoxBackupsTableSource from '../ProxmoxBackupsTable.tsx?raw';
|
||||
@@ -214,6 +222,55 @@ beforeEach(() => {
|
||||
});
|
||||
|
||||
describe('ProxmoxBackupsTable', () => {
|
||||
it('keeps successful backup inventory independent of unavailable datastore capacity', async () => {
|
||||
mockBackupAPIs();
|
||||
const failed = {
|
||||
...pbsServerResource,
|
||||
pbs: {
|
||||
...pbsServerResource.pbs!,
|
||||
datastores: [
|
||||
{
|
||||
name: 'main',
|
||||
total: 0,
|
||||
used: 0,
|
||||
available: 0,
|
||||
usagePercent: 0,
|
||||
status: 'unavailable',
|
||||
error: 'PRIVATE_PROVIDER_ERROR_SENTINEL',
|
||||
},
|
||||
],
|
||||
},
|
||||
} as Resource;
|
||||
const [servers, setServers] = createSignal([failed]);
|
||||
const { container } = renderInRouter(() => (
|
||||
<ProxmoxBackupsTable
|
||||
emptyIcon={<span />}
|
||||
workloads={[workloadResource]}
|
||||
servers={servers()}
|
||||
/>
|
||||
));
|
||||
// The router mounts its page asynchronously. Wait for the capacity notice
|
||||
// before binding page-level queries to the server table.
|
||||
await screen.findByText('Unavailable');
|
||||
const table = container.querySelector('[data-proxmox-backups-table="servers"]') as HTMLElement;
|
||||
const backupCount = () => {
|
||||
const index = Array.from(table.querySelectorAll('th')).findIndex(
|
||||
(head) => head.getAttribute('aria-label') === 'Backups',
|
||||
);
|
||||
return table.querySelector('tbody tr')!.querySelectorAll('td')[index].textContent;
|
||||
};
|
||||
await within(table).findByText('Unavailable');
|
||||
await waitFor(() => expect(backupCount()).toBe('1'));
|
||||
await screen.findAllByText('Protected');
|
||||
expect(container.innerHTML).not.toContain('PRIVATE_PROVIDER_ERROR_SENTINEL');
|
||||
expect(apiFetchMock.mock.calls.filter(([url]) => url === '/api/backups/pbs')).toHaveLength(1);
|
||||
|
||||
setServers([pbsServerResource]);
|
||||
await waitFor(() => expect(within(table).queryByText('Unavailable')).not.toBeInTheDocument());
|
||||
expect(backupCount()).toBe('1');
|
||||
expect(apiFetchMock.mock.calls.filter(([url]) => url === '/api/backups/pbs')).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('uses a corroborated PBS host link for Backups History despite a PVE-only name collision', () => {
|
||||
const pbs = {
|
||||
...pbsServerResource,
|
||||
@@ -1093,3 +1150,88 @@ describe('backup-date-evidence connected inventory', () => {
|
||||
expect(within(coverageRow()).getByText('Protected')).toBeInTheDocument();
|
||||
});
|
||||
});
|
||||
|
||||
describe('backup-identity: connected Coverage and By date', () => {
|
||||
const mockIdentityAPIs = () => {
|
||||
apiFetchMock.mockImplementation((url: string) =>
|
||||
Promise.resolve(
|
||||
jsonResponse({
|
||||
data:
|
||||
url === '/api/backups/pve'
|
||||
? {
|
||||
storageBackups: [identityArchive()],
|
||||
guestSnapshots: [identitySnapshot()],
|
||||
backupTasks: [identityTask()],
|
||||
}
|
||||
: { backups: [identityPBS()] },
|
||||
}),
|
||||
),
|
||||
);
|
||||
apiFetchJSONMock.mockResolvedValue({ data: [], policy: {}, meta: {} });
|
||||
vi.spyOn(Date, 'now').mockReturnValue(BACKUP_IDENTITY_NOW);
|
||||
};
|
||||
afterEach(() => vi.restoreAllMocks());
|
||||
|
||||
it('keeps all three artifacts and the successful task with the scoped owner through reorder and display rename', async () => {
|
||||
mockIdentityAPIs();
|
||||
window.history.replaceState({}, '', '/proxmox/backups/coverage');
|
||||
const east = identityGuest('east-100', 'east', 'pve-1');
|
||||
const west = identityGuest('west-100', 'west', 'pve-10');
|
||||
const [workloads, setWorkloads] = createSignal([west, east]);
|
||||
const { container } = renderInRouter(() => (
|
||||
<ProxmoxBackupsTable emptyIcon={<span />} workloads={workloads()} />
|
||||
));
|
||||
const rowFor = (name: string) =>
|
||||
Array.from(container.querySelectorAll('[data-proxmox-backup-row="coverage"]')).find((row) =>
|
||||
row.textContent?.includes(name),
|
||||
) as HTMLElement;
|
||||
await waitFor(() => expect(rowFor('east-100')).toHaveTextContent('OK'));
|
||||
expect(rowFor('west-100')).not.toHaveTextContent('OK');
|
||||
expect(within(rowFor('west-100')).getByTitle('No recent task')).toBeInTheDocument();
|
||||
const toggle = within(rowFor('east-100')).getByRole('button', { name: /expand details/i });
|
||||
await fireEvent.click(toggle);
|
||||
const detailID = toggle.getAttribute('aria-controls')!;
|
||||
await waitFor(() => expect(document.getElementById(detailID)).toHaveTextContent('Snapshot'));
|
||||
expect(document.getElementById(detailID)).toHaveTextContent('PVE file');
|
||||
expect(document.getElementById(detailID)).toHaveTextContent('PBS');
|
||||
setWorkloads([east, { ...west, proxmox: { ...west.proxmox!, nodeDisplayName: 'pve-1' } }]);
|
||||
await waitFor(() => expect(rowFor('east-100')).toHaveTextContent('OK'));
|
||||
expect(within(rowFor('east-100')).getByRole('button', { name: /collapse details/i })).toBe(
|
||||
toggle,
|
||||
);
|
||||
expect(rowFor('west-100')).not.toHaveTextContent('OK');
|
||||
expect(document.getElementById(detailID)).toHaveTextContent('PVE file');
|
||||
expect(apiFetchMock.mock.calls.filter(([url]) => url === '/api/backups/pve')).toHaveLength(1);
|
||||
expect(apiFetchMock.mock.calls.filter(([url]) => url === '/api/backups/pbs')).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('withdraws ambiguous PBS ownership without losing the artifact, PVE owner or server posture', async () => {
|
||||
mockIdentityAPIs();
|
||||
window.history.replaceState({}, '', '/proxmox/backups/date');
|
||||
const east = identityGuest('east-100', 'east', 'pve-1');
|
||||
const west = identityGuest('west-100', 'west', 'pve-10');
|
||||
const [workloads, setWorkloads] = createSignal([west, east]);
|
||||
const { container } = renderInRouter(() => (
|
||||
<ProxmoxBackupsTable emptyIcon={<span />} workloads={workloads()} />
|
||||
));
|
||||
const artifacts = () =>
|
||||
Array.from(container.querySelectorAll('[data-proxmox-backup-row="recoverable"]'));
|
||||
await waitFor(() => expect(artifacts()).toHaveLength(3));
|
||||
expect(artifacts().every((row) => row.textContent?.includes('east-100'))).toBe(true);
|
||||
setWorkloads([identityGuest('west-100', 'west', 'pve-1'), east]);
|
||||
await waitFor(() =>
|
||||
expect(artifacts().filter((row) => row.textContent?.includes('east-100'))).toHaveLength(2),
|
||||
);
|
||||
const unresolved = artifacts().find((row) => !row.textContent?.includes('east-100'))!;
|
||||
expect(unresolved).toHaveTextContent('VM 100');
|
||||
expect(unresolved).toHaveTextContent('PBS');
|
||||
expect(artifacts()).toHaveLength(3);
|
||||
await fireEvent.click(screen.getByRole('link', { name: 'Coverage' }));
|
||||
await fireEvent.click(screen.getByRole('button', { name: /1 unmatched backup/ }));
|
||||
await waitFor(() =>
|
||||
expect(screen.getByText('Not evaluated', { exact: true })).toBeInTheDocument(),
|
||||
);
|
||||
expect(apiFetchMock.mock.calls.filter(([url]) => url === '/api/backups/pve')).toHaveLength(1);
|
||||
expect(apiFetchMock.mock.calls.filter(([url]) => url === '/api/backups/pbs')).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
|
||||
+4
-4
@@ -141,11 +141,11 @@ describe('canonical protection posture ownership', () => {
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// matchWorkloadByHints — all branches (private; exercised through tasks whose
|
||||
// matchPVEWorkload — all branches (private; exercised through tasks whose
|
||||
// type maps to "unknown", e.g. vzdump, forcing the hint-matching path).
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('matchWorkloadByHints branches (via vzdump tasks)', () => {
|
||||
describe('matchPVEWorkload branches (via vzdump tasks)', () => {
|
||||
it('matches prior native node aliases while presenting the current display name', () => {
|
||||
const model = buildModel({
|
||||
workloads: [
|
||||
@@ -210,7 +210,7 @@ describe('matchWorkloadByHints branches (via vzdump tasks)', () => {
|
||||
expect(model.coverageRows.every((row) => row.latestTask === undefined)).toBe(true);
|
||||
});
|
||||
|
||||
it('falls back to the only candidate when its hints do not match', () => {
|
||||
it('leaves the only candidate untouched when the native task node contradicts it', () => {
|
||||
const model = buildModel({
|
||||
workloads: [workload({ id: 'vm-440', proxmox: { vmid: 440, node: 'alpha' } })],
|
||||
tasks: [
|
||||
@@ -224,7 +224,7 @@ describe('matchWorkloadByHints branches (via vzdump tasks)', () => {
|
||||
}),
|
||||
],
|
||||
});
|
||||
expect(model.coverageRows[0].latestTask?.label).toBe('OK');
|
||||
expect(model.coverageRows[0].latestTask).toBeUndefined();
|
||||
});
|
||||
|
||||
it('falls back to the only candidate when hints are all blank', () => {
|
||||
|
||||
+277
@@ -0,0 +1,277 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
import {
|
||||
BACKUP_IDENTITY_NOW,
|
||||
identityArchive,
|
||||
identityGuest,
|
||||
identityPBS,
|
||||
identitySnapshot,
|
||||
identityTask,
|
||||
} from '../__fixtures__/backupIdentity';
|
||||
import { buildProxmoxBackupRecoveryModel } from '../proxmoxBackupRecoveryModel';
|
||||
|
||||
type Input = Parameters<typeof buildProxmoxBackupRecoveryModel>[0];
|
||||
const model = (input: Partial<Input>) =>
|
||||
buildProxmoxBackupRecoveryModel({
|
||||
workloads: [],
|
||||
archives: [],
|
||||
snapshots: [],
|
||||
pbsBackups: [],
|
||||
tasks: [],
|
||||
nowMs: BACKUP_IDENTITY_NOW,
|
||||
...input,
|
||||
});
|
||||
const owners = (result: ReturnType<typeof model>) =>
|
||||
result.recoverableArtifacts.map((artifact) => artifact.workload.resourceId);
|
||||
|
||||
describe('backup-identity: native ownership rather than first label match', () => {
|
||||
it.each([false, true])('is independent of inventory order (reversed: %s)', (reversed) => {
|
||||
const workloads = [
|
||||
identityGuest('west-100', 'west', 'pve-10'),
|
||||
identityGuest('east-100', 'east', 'pve-1'),
|
||||
];
|
||||
if (reversed) workloads.reverse();
|
||||
const result = model({
|
||||
workloads,
|
||||
archives: [identityArchive()],
|
||||
snapshots: [identitySnapshot()],
|
||||
pbsBackups: [identityPBS()],
|
||||
tasks: [identityTask()],
|
||||
});
|
||||
expect(owners(result)).toEqual(['east-100', 'east-100', 'east-100']);
|
||||
expect(
|
||||
result.coverageRows.find((row) => row.workload.resourceId === 'west-100')?.latestTask,
|
||||
).toBeUndefined();
|
||||
expect(
|
||||
result.coverageRows.find((row) => row.workload.resourceId === 'east-100')?.latestTask?.id,
|
||||
).toBe('task-100');
|
||||
});
|
||||
|
||||
it('uses the PVE instance before a reused node name', () => {
|
||||
const result = model({
|
||||
workloads: [
|
||||
identityGuest('west-100', 'west', 'pve-1'),
|
||||
identityGuest('east-100', 'east', 'pve-1'),
|
||||
],
|
||||
archives: [identityArchive()],
|
||||
snapshots: [identitySnapshot()],
|
||||
tasks: [identityTask()],
|
||||
});
|
||||
expect(owners(result)).toEqual(['east-100', 'east-100']);
|
||||
expect(
|
||||
result.coverageRows.find((row) => row.workload.resourceId === 'west-100')?.latestTask,
|
||||
).toBeUndefined();
|
||||
});
|
||||
|
||||
it('keeps same-instance migrated backups without using the old node to select another cluster', () => {
|
||||
const result = model({
|
||||
workloads: [
|
||||
identityGuest('west-100', 'west', 'pve-1'),
|
||||
identityGuest('east-100', 'east', 'pve-2'),
|
||||
],
|
||||
archives: [identityArchive()],
|
||||
tasks: [identityTask()],
|
||||
});
|
||||
expect(owners(result)).toEqual(['east-100']);
|
||||
expect(
|
||||
result.coverageRows.find((row) => row.workload.resourceId === 'east-100')?.latestTask?.id,
|
||||
).toBe('task-100');
|
||||
});
|
||||
|
||||
it('does not let a matching node override a conflicting PVE instance, even with one visible guest', () => {
|
||||
const result = model({
|
||||
workloads: [identityGuest('east-100', 'east', 'pve-1')],
|
||||
archives: [identityArchive({ instance: 'removed-east' })],
|
||||
snapshots: [identitySnapshot({ instance: 'removed-east' })],
|
||||
tasks: [identityTask({ instance: 'removed-east' })],
|
||||
});
|
||||
expect(owners(result)).toEqual([undefined, undefined]);
|
||||
const live = result.coverageRows.find((row) => row.workload.resourceId === 'east-100')!;
|
||||
expect(live.artifacts).toEqual([]);
|
||||
expect(live.latestTask).toBeUndefined();
|
||||
const orphan = result.coverageRows.find((row) => row.isOrphaned)!;
|
||||
expect(orphan.latestTask?.id).toBe('task-100');
|
||||
expect(orphan.posture).toBe('not-evaluated');
|
||||
});
|
||||
|
||||
it('does not accept a partial node match or contradictory singleton when PVE instance is absent', () => {
|
||||
const result = model({
|
||||
workloads: [identityGuest('west-100', '', 'pve-10')],
|
||||
archives: [identityArchive({ instance: '' })],
|
||||
snapshots: [identitySnapshot({ instance: '' })],
|
||||
tasks: [identityTask({ instance: '' })],
|
||||
});
|
||||
expect(owners(result)).toEqual([undefined, undefined]);
|
||||
expect(result.coverageRows.find((row) => row.workload.resourceId)?.latestTask).toBeUndefined();
|
||||
});
|
||||
|
||||
it('accepts exact native aliases, not mutable display labels', () => {
|
||||
const workloads = [
|
||||
identityGuest('west-100', 'west', 'pve-10', { nodeDisplayName: 'pve-1' }),
|
||||
identityGuest('east-100', 'east', 'pve-new', { nodeAliases: ['pve-1'] }),
|
||||
];
|
||||
const result = model({
|
||||
workloads,
|
||||
archives: [identityArchive({ instance: '' })],
|
||||
pbsBackups: [identityPBS()],
|
||||
tasks: [identityTask({ instance: '' })],
|
||||
});
|
||||
expect(owners(result)).toEqual(['east-100', 'east-100']);
|
||||
expect(
|
||||
result.coverageRows.find((row) => row.workload.resourceId === 'east-100')?.latestTask?.id,
|
||||
).toBe('task-100');
|
||||
});
|
||||
|
||||
it('keeps presentation-only parent labels out of native-node matching', () => {
|
||||
const misleading = { ...identityGuest('west-100', '', ''), parentName: 'pve-1' };
|
||||
const result = model({
|
||||
workloads: [misleading, identityGuest('east-100', '', 'pve-1')],
|
||||
archives: [identityArchive({ instance: '' })],
|
||||
pbsBackups: [identityPBS()],
|
||||
});
|
||||
expect(owners(result)).toEqual(['east-100', 'east-100']);
|
||||
expect(
|
||||
result.coverageRows.find((row) => row.workload.resourceId === 'west-100')?.workload.node,
|
||||
).toBe('pve-1');
|
||||
expect(
|
||||
result.coverageRows.find((row) => row.workload.resourceId === 'west-100')?.workload
|
||||
.nativeNode,
|
||||
).toBeUndefined();
|
||||
});
|
||||
|
||||
it('does not collide unresolved repository keys on case or delimiter framing', () => {
|
||||
const result = model({
|
||||
pbsBackups: [
|
||||
identityPBS({ id: 'case-a', instance: 'pbs', datastore: 'main' }),
|
||||
identityPBS({ id: 'case-b', instance: 'pbs', datastore: 'Main' }),
|
||||
identityPBS({ id: 'parts-a', instance: 'a:b', datastore: 'c' }),
|
||||
identityPBS({ id: 'parts-b', instance: 'a', datastore: 'b:c' }),
|
||||
],
|
||||
});
|
||||
expect(result.coverageRows).toHaveLength(4);
|
||||
expect(new Set(result.coverageRows.map((row) => row.key)).size).toBe(4);
|
||||
});
|
||||
|
||||
it('keeps source names with spaces or quotes in an IDREF-safe unresolved key', () => {
|
||||
const result = model({
|
||||
pbsBackups: [
|
||||
identityPBS({
|
||||
instance: 'backup server',
|
||||
datastore: 'main "store"',
|
||||
namespace: 'old node',
|
||||
}),
|
||||
],
|
||||
});
|
||||
const key = result.coverageRows[0].key;
|
||||
expect(key).not.toMatch(/[\s"']/);
|
||||
expect(decodeURIComponent(key)).toContain('backup server');
|
||||
});
|
||||
|
||||
it('does not turn a reused namespace/node or alias into a unique PBS owner', () => {
|
||||
const result = model({
|
||||
workloads: [
|
||||
identityGuest('west-100', 'west', 'shared', { nodeAliases: ['pve-1'] }),
|
||||
identityGuest('east-100', 'east', 'shared', { nodeAliases: ['pve-1'] }),
|
||||
],
|
||||
pbsBackups: [identityPBS()],
|
||||
});
|
||||
expect(owners(result)).toEqual([undefined]);
|
||||
expect(
|
||||
result.coverageRows
|
||||
.filter((row) => row.workload.resourceId)
|
||||
.every((row) => row.pbsCount === 0),
|
||||
).toBe(true);
|
||||
});
|
||||
|
||||
it('does not use a datastore label as a PVE workload identity', () => {
|
||||
const result = model({
|
||||
workloads: [
|
||||
identityGuest('west-100', 'west', 'main'),
|
||||
identityGuest('east-100', 'east', 'pve-1'),
|
||||
],
|
||||
pbsBackups: [identityPBS({ namespace: '' })],
|
||||
});
|
||||
expect(owners(result)).toEqual([undefined]);
|
||||
});
|
||||
|
||||
it('preserves root-namespace PBS for a unique VMID and exact instance namespace with collisions', () => {
|
||||
expect(
|
||||
owners(
|
||||
model({
|
||||
workloads: [identityGuest('east-100', 'east', 'pve-1')],
|
||||
pbsBackups: [identityPBS({ namespace: '' })],
|
||||
}),
|
||||
),
|
||||
).toEqual(['east-100']);
|
||||
expect(
|
||||
owners(
|
||||
model({
|
||||
workloads: [
|
||||
identityGuest('west-100', 'west', 'pve-1'),
|
||||
identityGuest('east-100', 'east', 'pve-1'),
|
||||
],
|
||||
pbsBackups: [identityPBS({ namespace: ' EAST ' })],
|
||||
}),
|
||||
),
|
||||
).toEqual(['east-100']);
|
||||
});
|
||||
|
||||
it('retains exact-node matching without instance metadata and the unscoped singleton control', () => {
|
||||
const workload = identityGuest('east-100', '', 'pve-1');
|
||||
expect(
|
||||
owners(model({ workloads: [workload], archives: [identityArchive({ instance: '' })] })),
|
||||
).toEqual(['east-100']);
|
||||
expect(
|
||||
owners(
|
||||
model({ workloads: [workload], archives: [identityArchive({ instance: '', node: '' })] }),
|
||||
),
|
||||
).toEqual(['east-100']);
|
||||
});
|
||||
|
||||
it('leaves an ambiguous unknown-type task unattached across VM and LXC identities', () => {
|
||||
const vm = identityGuest('vm-100', 'east', 'pve-1');
|
||||
const ct = {
|
||||
...identityGuest('ct-100', 'east', 'pve-1'),
|
||||
type: 'system-container',
|
||||
} as typeof vm;
|
||||
const result = model({ workloads: [vm, ct], tasks: [identityTask()] });
|
||||
expect(result.coverageRows.every((row) => row.latestTask === undefined)).toBe(true);
|
||||
});
|
||||
|
||||
it('preserves unresolved artifacts but does not pool identically named PBS sources', () => {
|
||||
const result = model({
|
||||
pbsBackups: [
|
||||
identityPBS({ id: 'pbs-a', instance: 'pbs-a' }),
|
||||
identityPBS({ id: 'pbs-b', instance: 'pbs-b' }),
|
||||
],
|
||||
});
|
||||
expect(result.recoverableArtifacts).toHaveLength(2);
|
||||
expect(result.coverageRows).toHaveLength(2);
|
||||
expect(new Set(result.coverageRows.map((row) => row.key)).size).toBe(2);
|
||||
expect(
|
||||
result.coverageRows.every((row) => row.pbsCount === 1 && row.posture === 'not-evaluated'),
|
||||
).toBe(true);
|
||||
});
|
||||
|
||||
it('does not attach a PVE task to a PBS-only orphan with a similar location label', () => {
|
||||
const result = model({ pbsBackups: [identityPBS()], tasks: [identityTask({ instance: '' })] });
|
||||
expect(result.coverageRows[0].latestTask).toBeUndefined();
|
||||
});
|
||||
|
||||
it('keeps unresolved PVE instances separate while associating their own typed and untyped tasks', () => {
|
||||
const result = model({
|
||||
archives: [
|
||||
identityArchive({ id: 'file-east' }),
|
||||
identityArchive({ id: 'file-west', instance: 'west' }),
|
||||
],
|
||||
tasks: [identityTask(), identityTask({ id: 'task-west', instance: 'west', type: 'vm' })],
|
||||
});
|
||||
expect(result.coverageRows).toHaveLength(2);
|
||||
expect(
|
||||
result.coverageRows.map((row) => [row.workload.instance, row.latestTask?.id]).sort(),
|
||||
).toEqual([
|
||||
['east', 'task-100'],
|
||||
['west', 'task-west'],
|
||||
]);
|
||||
});
|
||||
});
|
||||
@@ -442,16 +442,21 @@ describe('proxmoxBackupRecoveryModel', () => {
|
||||
});
|
||||
|
||||
expect(model.recoverableArtifacts).toHaveLength(3);
|
||||
expect(model.coverageRows).toHaveLength(1);
|
||||
expect(model.coverageSummary.totalWorkloads).toBe(1);
|
||||
expect(model.coverageRows[0].isOrphaned).toBe(false);
|
||||
expect(model.coverageRows[0].workload.type).toBe('host');
|
||||
expect(model.coverageRows[0].workload.vmid).toBe('mail-gateway');
|
||||
expect(model.coverageRows[0].workload.label).toBe('Host mail-gateway');
|
||||
expect(model.coverageRows[0].posture).toBe('not-evaluated');
|
||||
expect(getWorkloadRecoveryPostureLabel(model.coverageRows[0].posture)).toBe('Not evaluated');
|
||||
expect(model.coverageRows[0].pbsCount).toBe(1);
|
||||
expect(model.coverageRows[0].archiveCount).toBe(1);
|
||||
expect(model.coverageRows).toHaveLength(2);
|
||||
expect(model.coverageSummary.totalWorkloads).toBe(2);
|
||||
// Equal host labels on PVE and PBS are not proof of the same backup target.
|
||||
expect(model.coverageRows.every((row) => !row.isOrphaned && row.workload.type === 'host')).toBe(
|
||||
true,
|
||||
);
|
||||
expect(
|
||||
model.coverageRows.every(
|
||||
(row) => row.workload.label === 'Host mail-gateway' && row.posture === 'not-evaluated',
|
||||
),
|
||||
).toBe(true);
|
||||
expect(model.coverageRows.map((row) => [row.pbsCount, row.archiveCount]).sort()).toEqual([
|
||||
[0, 1],
|
||||
[1, 0],
|
||||
]);
|
||||
expect(model.recoverableArtifacts.map((artifact) => artifact.workload.label)).toEqual(
|
||||
expect.arrayContaining(['LXC backup', 'Host mail-gateway']),
|
||||
);
|
||||
|
||||
+3
-2
@@ -46,8 +46,9 @@ describe('Proxmox backups responsive table presentation', () => {
|
||||
'used',
|
||||
]);
|
||||
expect(getBackupServerColumnWidthStyle('server', 'compact')).toEqual({ width: '40%' });
|
||||
expect(getBackupServerColumnWidthStyle('status', 'compact')).toEqual({ width: '23%' });
|
||||
expect(getBackupServerColumnWidthStyle('used', 'compact')).toEqual({ width: '15%' });
|
||||
expect(getBackupServerColumnWidthStyle('status', 'compact')).toEqual({ width: '18%' });
|
||||
expect(getBackupServerColumnWidthStyle('used', 'compact')).toEqual({ width: '25%' });
|
||||
expect(getBackupServerColumnWidthStyle('datastore', 'compact')).toEqual({ width: '17%' });
|
||||
expect(ids(getBackupServerColumns('basic'))).toEqual([
|
||||
'server',
|
||||
'status',
|
||||
|
||||
@@ -93,7 +93,7 @@ export interface WorkloadCoverageRow {
|
||||
postureRank: number;
|
||||
protectionPosture?: ProtectionPosture;
|
||||
// True only when a VM/LXC row exists because a backup/task referenced a VMID
|
||||
// with no matching live inventory guest. Host backups can also carry a
|
||||
// with no unambiguous live inventory match (absent or conflicting). Host backups can also carry a
|
||||
// `backup:` key, but they are first-class backup targets, not orphaned guests.
|
||||
isOrphaned: boolean;
|
||||
}
|
||||
@@ -124,10 +124,7 @@ interface BuildModelInput {
|
||||
protectionPosturesResolved?: boolean;
|
||||
}
|
||||
|
||||
interface WorkloadCandidate extends WorkloadReference {
|
||||
nodeKey?: string;
|
||||
instanceKey?: string;
|
||||
}
|
||||
type WorkloadCandidate = WorkloadReference;
|
||||
|
||||
type WorkloadRowDraft = Omit<WorkloadCoverageRow, 'posture' | 'postureRank' | 'isOrphaned'>;
|
||||
|
||||
@@ -266,7 +263,7 @@ function resourceBackupType(resource: Resource): WorkloadReference['type'] {
|
||||
}
|
||||
|
||||
function resourceNode(resource: Resource): string | undefined {
|
||||
return resource.proxmox?.nodeName || resource.proxmox?.node || resource.parentName || undefined;
|
||||
return resource.proxmox?.nodeName || resource.proxmox?.node || undefined;
|
||||
}
|
||||
|
||||
function resourceNodeDisplayName(resource: Resource): string | undefined {
|
||||
@@ -316,8 +313,6 @@ function buildCandidateFromResource(resource: Resource): WorkloadCandidate | nul
|
||||
nativeNode,
|
||||
nativeNodeAliases,
|
||||
instance,
|
||||
nodeKey: normalizeKey(nativeNode),
|
||||
instanceKey: normalizeKey(instance),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -325,18 +320,24 @@ function fallbackWorkload(
|
||||
type: WorkloadReference['type'],
|
||||
vmid: string,
|
||||
hints: readonly (string | undefined)[],
|
||||
sourceScope: readonly (string | undefined)[],
|
||||
pveInstance?: string,
|
||||
): WorkloadReference {
|
||||
const id = fallbackWorkloadId(type, vmid, hints);
|
||||
const scope = hints.map(normalizeKey).find(Boolean) || 'unknown';
|
||||
const keyScope = type === 'host' && id ? 'host' : scope;
|
||||
// A VMID (or host label) and repository name are not cross-source identity.
|
||||
// Keep unresolved evidence listed, with a collision-safe complete source key.
|
||||
const scope = encodeURIComponent(
|
||||
JSON.stringify(sourceScope.map((part) => String(part ?? '').trim())),
|
||||
);
|
||||
return {
|
||||
key: `backup:${type}:${id || 'unknown'}:${keyScope}`,
|
||||
key: `backup:${type}:${id || 'unknown'}:${scope}`,
|
||||
type,
|
||||
typeLabel: typeLabel(type),
|
||||
vmid: id,
|
||||
label: workloadFallbackLabel(type, id),
|
||||
node: type === 'host' ? undefined : hints.find((hint) => !!hint?.trim()),
|
||||
nativeNode: type === 'host' ? undefined : hints.find((hint) => !!hint?.trim()),
|
||||
instance: pveInstance,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -373,72 +374,118 @@ function buildWorkloadCandidateIndex(
|
||||
return { byTypeAndVmid, byVmid };
|
||||
}
|
||||
|
||||
function resolveWorkload(
|
||||
function uniqueWorkload<T>(matches: readonly T[]): T | undefined {
|
||||
return matches.length === 1 ? matches[0] : undefined;
|
||||
}
|
||||
|
||||
function matchesNativeNode(workload: WorkloadReference, node: string): boolean {
|
||||
const key = normalizeKey(node);
|
||||
return (
|
||||
Boolean(key) &&
|
||||
[workload.nativeNode, ...(workload.nativeNodeAliases ?? [])].some(
|
||||
(name) => normalizeKey(name) === key,
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
function matchPVEWorkload<T extends WorkloadReference>(
|
||||
workloads: readonly T[],
|
||||
instance: string,
|
||||
node: string,
|
||||
): T | undefined {
|
||||
const instanceKey = normalizeKey(instance);
|
||||
const nodeKey = normalizeKey(node);
|
||||
if (instanceKey) {
|
||||
const scoped = workloads.filter((workload) => normalizeKey(workload.instance) === instanceKey);
|
||||
// A VMID is unique inside its PVE connection. Historical backup/task nodes
|
||||
// can differ after a migration; they must not override that connection.
|
||||
if (scoped.length > 0) {
|
||||
return (
|
||||
uniqueWorkload(scoped) ??
|
||||
uniqueWorkload(scoped.filter((workload) => matchesNativeNode(workload, node)))
|
||||
);
|
||||
}
|
||||
// Older projections may omit the instance. An exact native node can still
|
||||
// identify one of them, but cannot rescue a known conflicting instance.
|
||||
return uniqueWorkload(
|
||||
workloads.filter(
|
||||
(workload) => !normalizeKey(workload.instance) && matchesNativeNode(workload, node),
|
||||
),
|
||||
);
|
||||
}
|
||||
if (nodeKey)
|
||||
return uniqueWorkload(workloads.filter((workload) => matchesNativeNode(workload, node)));
|
||||
return uniqueWorkload(workloads);
|
||||
}
|
||||
|
||||
function resolvePVEWorkload(
|
||||
candidates: WorkloadCandidateIndex,
|
||||
type: WorkloadReference['type'],
|
||||
vmid: string,
|
||||
hints: readonly (string | undefined)[],
|
||||
instance: string,
|
||||
node: string,
|
||||
): WorkloadReference {
|
||||
const typed = candidates.byTypeAndVmid.get(workloadTypeVmidKey(type, vmid)) ?? [];
|
||||
const normalizedHints = hints.map(normalizeKey).filter(Boolean);
|
||||
if (typed.length > 0 && normalizedHints.length > 0) {
|
||||
const exact = typed.find((candidate) =>
|
||||
normalizedHints.some(
|
||||
(hint) =>
|
||||
hint === candidate.nodeKey ||
|
||||
hint === candidate.instanceKey ||
|
||||
normalizeKey(candidate.node).includes(hint) ||
|
||||
normalizeKey(candidate.nativeNode).includes(hint) ||
|
||||
candidate.nativeNodeAliases?.some((alias) => normalizeKey(alias).includes(hint)) ||
|
||||
normalizeKey(candidate.instance).includes(hint),
|
||||
),
|
||||
);
|
||||
if (exact) return exact;
|
||||
}
|
||||
if (typed.length === 1) return typed[0];
|
||||
return fallbackWorkload(type, vmid, hints);
|
||||
return (
|
||||
matchPVEWorkload(typed, instance, node) ??
|
||||
fallbackWorkload(
|
||||
type,
|
||||
vmid,
|
||||
type === 'host' ? [instance, node] : [node, instance],
|
||||
['pve', instance, node],
|
||||
instance,
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
function matchWorkloadByHints<T extends WorkloadReference>(
|
||||
workloads: readonly T[],
|
||||
hints: readonly (string | undefined)[],
|
||||
): T | undefined {
|
||||
const normalizedHints = hints.map(normalizeKey).filter(Boolean);
|
||||
if (workloads.length > 0 && normalizedHints.length > 0) {
|
||||
const exact = workloads.find((workload) =>
|
||||
normalizedHints.some(
|
||||
(hint) =>
|
||||
hint === normalizeKey(workload.node) ||
|
||||
hint === normalizeKey(workload.nativeNode) ||
|
||||
hint === normalizeKey(workload.instance) ||
|
||||
normalizeKey(workload.node).includes(hint) ||
|
||||
normalizeKey(workload.nativeNode).includes(hint) ||
|
||||
workload.nativeNodeAliases?.some((alias) => normalizeKey(alias).includes(hint)) ||
|
||||
normalizeKey(workload.instance).includes(hint),
|
||||
),
|
||||
);
|
||||
if (exact) return exact;
|
||||
}
|
||||
if (workloads.length === 1) return workloads[0];
|
||||
return undefined;
|
||||
function resolvePBSWorkload(
|
||||
candidates: WorkloadCandidateIndex,
|
||||
backup: PBSBackup,
|
||||
): WorkloadReference {
|
||||
const type = backupTypeLabel(backup.backupType);
|
||||
const typed = candidates.byTypeAndVmid.get(workloadTypeVmidKey(type, backup.vmid)) ?? [];
|
||||
// PBS instance/datastore identify a repository, not a PVE connection. A
|
||||
// namespace can disambiguate a repeated VMID only with one exact native
|
||||
// node/alias or connection match. A display name or substring proves none.
|
||||
const namespace = normalizeKey(backup.namespace);
|
||||
const matched = namespace
|
||||
? uniqueWorkload(
|
||||
typed.filter(
|
||||
(workload) =>
|
||||
matchesNativeNode(workload, backup.namespace) ||
|
||||
normalizeKey(workload.instance) === namespace,
|
||||
),
|
||||
)
|
||||
: undefined;
|
||||
return (
|
||||
matched ??
|
||||
uniqueWorkload(typed) ??
|
||||
fallbackWorkload(
|
||||
type,
|
||||
backup.vmid,
|
||||
[backup.namespace, backup.datastore],
|
||||
['pbs', backup.instance, backup.datastore, backup.namespace],
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
function resolveTaskWorkload(
|
||||
candidates: WorkloadCandidateIndex,
|
||||
workloadsByVmid: ReadonlyMap<string, readonly WorkloadReference[]>,
|
||||
pveWorkloadsByVmid: ReadonlyMap<string, readonly WorkloadReference[]>,
|
||||
task: BackupTask,
|
||||
): WorkloadReference | undefined {
|
||||
const vmid = String(task.vmid);
|
||||
if (!vmid || isZeroWorkloadId(vmid)) return undefined;
|
||||
|
||||
const explicitType = backupTypeLabel(task.type);
|
||||
const hints = [task.instance, task.node];
|
||||
if (explicitType !== 'unknown') return resolveWorkload(candidates, explicitType, vmid, hints);
|
||||
if (explicitType !== 'unknown')
|
||||
return resolvePVEWorkload(candidates, explicitType, vmid, task.instance, task.node);
|
||||
|
||||
const candidate = matchWorkloadByHints(candidates.byVmid.get(vmid) ?? [], hints);
|
||||
const candidate = matchPVEWorkload(candidates.byVmid.get(vmid) ?? [], task.instance, task.node);
|
||||
if (candidate) return candidate;
|
||||
|
||||
return matchWorkloadByHints(workloadsByVmid.get(vmid) ?? [], hints);
|
||||
// Only PVE-backed orphan rows share this source scope. A PBS namespace that
|
||||
// happens to resemble the task node is not evidence that the task owns it.
|
||||
return matchPVEWorkload(pveWorkloadsByVmid.get(vmid) ?? [], task.instance, task.node);
|
||||
}
|
||||
|
||||
function newest<T>(items: readonly T[], getMs: (item: T) => number | undefined): T | undefined {
|
||||
@@ -549,9 +596,9 @@ export function buildProxmoxBackupRecoveryModel(
|
||||
.filter((candidate): candidate is WorkloadCandidate => candidate !== null);
|
||||
const candidateIndex = buildWorkloadCandidateIndex(candidates);
|
||||
const rows = new Map<string, WorkloadRowDraft>();
|
||||
const workloadsByVmid = new Map<string, WorkloadReference[]>();
|
||||
const pveWorkloadsByVmid = new Map<string, WorkloadReference[]>();
|
||||
|
||||
const ensureRow = (workload: WorkloadReference) => {
|
||||
const ensureRow = (workload: WorkloadReference, pveSource: boolean) => {
|
||||
const existing = rows.get(workload.key);
|
||||
if (existing) return existing;
|
||||
const row: WorkloadRowDraft = {
|
||||
@@ -563,13 +610,15 @@ export function buildProxmoxBackupRecoveryModel(
|
||||
snapshotCount: 0,
|
||||
};
|
||||
rows.set(workload.key, row);
|
||||
const matchingVmid = workloadsByVmid.get(workload.vmid);
|
||||
if (matchingVmid) matchingVmid.push(workload);
|
||||
else workloadsByVmid.set(workload.vmid, [workload]);
|
||||
if (pveSource) {
|
||||
const matchingVmid = pveWorkloadsByVmid.get(workload.vmid);
|
||||
if (matchingVmid) matchingVmid.push(workload);
|
||||
else pveWorkloadsByVmid.set(workload.vmid, [workload]);
|
||||
}
|
||||
return row;
|
||||
};
|
||||
|
||||
for (const candidate of candidates) ensureRow(candidate);
|
||||
for (const candidate of candidates) ensureRow(candidate, true);
|
||||
|
||||
const artifacts: RecoverableArtifact[] = [];
|
||||
const pbsSource = getProxmoxBackupSourcePresentation('pbs');
|
||||
@@ -578,7 +627,7 @@ export function buildProxmoxBackupRecoveryModel(
|
||||
const addArtifact = (artifact: RecoverableArtifact) => {
|
||||
artifacts.push(artifact);
|
||||
if (!isCoverageWorkload(artifact.workload)) return;
|
||||
const row = ensureRow(artifact.workload);
|
||||
const row = ensureRow(artifact.workload, artifact.sourceKind !== 'pbs');
|
||||
row.artifacts.push(artifact);
|
||||
if (artifact.sourceKind === 'pbs') row.pbsCount += 1;
|
||||
else if (artifact.sourceKind === 'archive') row.archiveCount += 1;
|
||||
@@ -586,11 +635,7 @@ export function buildProxmoxBackupRecoveryModel(
|
||||
};
|
||||
|
||||
for (const backup of input.pbsBackups) {
|
||||
const type = backupTypeLabel(backup.backupType);
|
||||
const workload = resolveWorkload(candidateIndex, type, backup.vmid, [
|
||||
backup.namespace,
|
||||
backup.datastore,
|
||||
]);
|
||||
const workload = resolvePBSWorkload(candidateIndex, backup);
|
||||
const parsedMs = parseTimestampMs(backup.backupTime);
|
||||
const createdMs =
|
||||
getRecoveryAgeBand(parsedMs, input.nowMs) === 'unknown' ? undefined : parsedMs;
|
||||
@@ -624,10 +669,13 @@ export function buildProxmoxBackupRecoveryModel(
|
||||
|
||||
for (const archive of input.archives) {
|
||||
const type = backupTypeLabel(archive.type);
|
||||
const workload = resolveWorkload(candidateIndex, type, String(archive.vmid), [
|
||||
const workload = resolvePVEWorkload(
|
||||
candidateIndex,
|
||||
type,
|
||||
String(archive.vmid),
|
||||
archive.instance,
|
||||
archive.node,
|
||||
]);
|
||||
);
|
||||
const parsedMs = parseTimestampMs(archive.time);
|
||||
const createdMs =
|
||||
getRecoveryAgeBand(parsedMs, input.nowMs) === 'unknown' ? undefined : parsedMs;
|
||||
@@ -657,10 +705,13 @@ export function buildProxmoxBackupRecoveryModel(
|
||||
|
||||
for (const snapshot of input.snapshots) {
|
||||
const type = backupTypeLabel(snapshot.type);
|
||||
const workload = resolveWorkload(candidateIndex, type, String(snapshot.vmid), [
|
||||
const workload = resolvePVEWorkload(
|
||||
candidateIndex,
|
||||
type,
|
||||
String(snapshot.vmid),
|
||||
snapshot.instance,
|
||||
snapshot.node,
|
||||
]);
|
||||
);
|
||||
const parsedMs = parseTimestampMs(snapshot.time);
|
||||
const createdMs =
|
||||
getRecoveryAgeBand(parsedMs, input.nowMs) === 'unknown' ? undefined : parsedMs;
|
||||
@@ -682,9 +733,9 @@ export function buildProxmoxBackupRecoveryModel(
|
||||
}
|
||||
|
||||
for (const task of input.tasks) {
|
||||
const workload = resolveTaskWorkload(candidateIndex, workloadsByVmid, task);
|
||||
const workload = resolveTaskWorkload(candidateIndex, pveWorkloadsByVmid, task);
|
||||
if (!workload) continue;
|
||||
const row = ensureRow(workload);
|
||||
const row = ensureRow(workload, true);
|
||||
const candidateTask: CoverageTask = {
|
||||
id: task.id,
|
||||
status: task.status,
|
||||
|
||||
@@ -107,7 +107,9 @@ const BACKUP_SERVER_WEIGHTS: Record<
|
||||
ProxmoxBackupsTableLayoutMode,
|
||||
Partial<Record<BackupServerColumnId, number>>
|
||||
> = {
|
||||
compact: { server: 40, status: 23, datastore: 22, used: 15 },
|
||||
// Capacity must fit "Unavailable", not just a short percentage. Keep the
|
||||
// shared 40% identity anchor; the compact status omits its redundant dot.
|
||||
compact: { server: 40, status: 18, datastore: 17, used: 25 },
|
||||
// Just above the phone projection the used cell carries its used/total
|
||||
// pair again, the longest value in the row, so it takes the most room.
|
||||
basic: { server: 24, status: 13.6, datastore: 18.2, used: 28.4, backups: 9.8 },
|
||||
|
||||
@@ -466,6 +466,7 @@ export interface ResourcePBSDatastore {
|
||||
available?: number;
|
||||
usagePercent?: number;
|
||||
status?: string;
|
||||
error?: string;
|
||||
deduplicationFactor?: number;
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
import { readFileSync } from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { renderDocMarkdown } from '@/features/docs/docMarkdown';
|
||||
|
||||
const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../..');
|
||||
const read = (name: string) => readFileSync(path.join(root, name), 'utf8');
|
||||
const guide = read('docs/VM_DISK_MONITORING.md');
|
||||
const section =
|
||||
guide.split('### Pause Pulse for a planned freeze-enabled backup')[1]?.split('## 🚀 Setup')[0] ??
|
||||
'';
|
||||
const words = section.replace(/\s+/g, ' ');
|
||||
|
||||
function render(markdown: string, name: string): HTMLElement {
|
||||
const article = document.createElement('article');
|
||||
article.innerHTML = renderDocMarkdown(markdown, name);
|
||||
return article;
|
||||
}
|
||||
|
||||
describe('manual backup safety help', () => {
|
||||
it('ships the complete precaution and makes it reachable from troubleshooting', () => {
|
||||
expect(read('frontend-modern/public/docs/VM_DISK_MONITORING.md')).toBe(guide);
|
||||
const troubleshooting = read('docs/TROUBLESHOOTING.md');
|
||||
expect(read('frontend-modern/public/docs/TROUBLESHOOTING.md')).toBe(troubleshooting);
|
||||
const link = render(troubleshooting, 'TROUBLESHOOTING').querySelector(
|
||||
'a[href="/docs/VM_DISK_MONITORING#pause-pulse-for-a-planned-freeze-enabled-backup"]',
|
||||
);
|
||||
expect(link?.textContent).toBe('manual backup precaution');
|
||||
expect(
|
||||
render(guide, 'VM_DISK_MONITORING').querySelector(
|
||||
'#pause-pulse-for-a-planned-freeze-enabled-backup',
|
||||
)?.textContent,
|
||||
).toBe('Pause Pulse for a planned freeze-enabled backup');
|
||||
});
|
||||
|
||||
it('keeps service control in the actual Pulse server container, not the guest or host agent', () => {
|
||||
expect(words).toContain('inside the Pulse LXC');
|
||||
expect(words).toContain('not the Proxmox host or the backed-up VM');
|
||||
expect(words).toContain('pulse-backend.service');
|
||||
expect(words).toContain('not pulse-agent.service');
|
||||
expect(words).toContain('LoadState=loaded');
|
||||
expect(words).toContain('If the unit is missing');
|
||||
});
|
||||
|
||||
it('checks updater state before server stop and preserves the original timer state', () => {
|
||||
expect(words).toContain('Record whether the timer was active');
|
||||
expect(words).toContain('let it finish normally before continuing');
|
||||
expect(words).toContain('Do not interrupt an installation');
|
||||
expect(words).toContain('only if it was active beforehand');
|
||||
const blocks = [...render(section, 'VM_DISK_MONITORING').querySelectorAll('pre code')].map(
|
||||
(code) => code.textContent?.trim(),
|
||||
);
|
||||
expect(blocks).toEqual([
|
||||
'systemctl show pulse.service --property=LoadState,ActiveState,MainPID',
|
||||
'systemctl show pulse-update.timer --property=LoadState,ActiveState',
|
||||
'sudo systemctl stop pulse-update.timer\nsystemctl show pulse-update.timer pulse-update.service \\\n --property=Id,LoadState,ActiveState,MainPID',
|
||||
'sudo systemctl stop pulse.service\nsystemctl show pulse.service --property=LoadState,ActiveState,MainPID',
|
||||
'sudo systemctl start pulse.service\nsystemctl is-active pulse.service',
|
||||
'sudo systemctl start pulse-update.timer',
|
||||
]);
|
||||
// These defaults and the stopped-service condition are from the existing
|
||||
// installer, not a new service or a promise about a custom deployment.
|
||||
const installer = read('install.sh');
|
||||
expect(installer).toContain('DEFAULT_SERVICE_NAME="pulse"');
|
||||
expect(installer).toContain('SERVICE_NAME="pulse-backend"');
|
||||
expect(installer).toContain(
|
||||
"ExecCondition=/bin/sh -c 'systemctl is-active --quiet ${service_name}'",
|
||||
);
|
||||
});
|
||||
|
||||
it('requires stopped-state readback and discloses the whole monitoring outage', () => {
|
||||
expect(words).toContain('ActiveState=inactive');
|
||||
expect(words).toContain('MainPID=0');
|
||||
expect(words).toContain('Do not start the backup if');
|
||||
expect(words).toContain('Pulse monitoring and alerts are unavailable');
|
||||
expect(words).toContain('Do not run an update, start another Pulse instance');
|
||||
expect(words).toContain('repeat the precaution for each affected backup window');
|
||||
});
|
||||
|
||||
it('requires independent workload writes before restart, never merely an OK task or an open console', () => {
|
||||
expect(words).toContain('Keep Pulse stopped until');
|
||||
expect(words).toContain('fresh successful workload writes');
|
||||
expect(words).toContain('console connection or a successful read alone is not enough');
|
||||
expect(words).toContain('not proof that polling or alerts have recovered');
|
||||
expect(words).toContain('normal observation times');
|
||||
expect(words).toContain('without Run Diagnostics or manual guest-agent probes');
|
||||
expect(guide).toContain('An OK backup task or an absent VM lock');
|
||||
expect(guide).toContain('does not prove thaw succeeded');
|
||||
expect(guide).toContain('not a claim\nthat the monitoring defect is fixed');
|
||||
});
|
||||
|
||||
it('does not turn a planned precaution into an incident recovery or an automatic restart hook', () => {
|
||||
expect(words).toContain('not a recovery procedure');
|
||||
expect(words).toContain('not an automatic backup hook');
|
||||
expect(words).toContain('If thaw cannot be confirmed, leave Pulse stopped');
|
||||
expect(guide).toContain(
|
||||
'Do not test freeze/thaw commands, clear backup locks, disable backup freezing',
|
||||
);
|
||||
const commands = [...render(section, 'VM_DISK_MONITORING').querySelectorAll('pre code')]
|
||||
.map((code) => code.textContent)
|
||||
.join('\n');
|
||||
expect(commands).not.toMatch(
|
||||
/qm |pct |pvesh |guest-fsfreeze|restart|disable|enable|mask|reset|rm |kill|curl|wget|Environment/,
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -62,6 +62,18 @@ function getRuntimeSourceFiles(dir: string): string[] {
|
||||
}
|
||||
|
||||
describe('docsLinks', () => {
|
||||
it('keeps report collection precautions usable without unpublished documentation links', () => {
|
||||
for (const form of ['bug_report.yml', 'v6_rc_feedback.yml']) {
|
||||
const source = readFileSync(path.join(repoRoot, '.github', 'ISSUE_TEMPLATE', form), 'utf8');
|
||||
expect(source).toContain(
|
||||
'do not run it during backups, freeze/thaw or an unresponsive-host incident',
|
||||
);
|
||||
expect(source).toContain('downloads that result without running checks again');
|
||||
expect(source).toContain('Review files and screenshots locally before posting');
|
||||
expect(source).not.toMatch(/https:\/\/github\.com\/rcourtman\/Pulse\/blob\/[^\s)]+\/docs\//);
|
||||
}
|
||||
});
|
||||
|
||||
const probeGuides = () =>
|
||||
[
|
||||
readFileSync(path.join(repoRoot, 'docs', 'CONFIGURATION.md'), 'utf8')
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
package alerts
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestBackupEvaluationSystemAlertUsesExistingIdentityAndDeduplication(t *testing.T) {
|
||||
m := NewManagerWithDataDir(t.TempDir())
|
||||
defer m.Stop()
|
||||
input := SystemAlertInput{Type: BackupEvaluationAlertType, Level: AlertLevelWarning,
|
||||
Fingerprint: "recovery-rollups-unavailable", Message: "Backup-age alerts were not evaluated."}
|
||||
if !m.RaiseSystemAlert(input) {
|
||||
t.Fatal("first failure not raised")
|
||||
}
|
||||
if m.RaiseSystemAlert(input) {
|
||||
t.Fatal("repeated failure re-notified")
|
||||
}
|
||||
active := m.GetActiveAlerts()
|
||||
if len(active) != 1 || active[0].ID != SystemAlertID(BackupEvaluationAlertType) ||
|
||||
active[0].ResourceID != "" || active[0].Metadata["systemAlert"] != true {
|
||||
t.Fatalf("system projection=%+v", active)
|
||||
}
|
||||
m.CleanupAlertsForNodes(map[string]bool{})
|
||||
if len(m.GetActiveAlerts()) != 1 {
|
||||
t.Fatal("node cleanup erased the evaluator outage")
|
||||
}
|
||||
if !m.ClearSystemAlert(BackupEvaluationAlertType) || len(m.GetActiveAlerts()) != 0 {
|
||||
t.Fatal("recovery did not clear the warning")
|
||||
}
|
||||
}
|
||||
@@ -28,6 +28,10 @@ const (
|
||||
// notification destinations are not delivering.
|
||||
NotificationDeliveryAlertType = "notification-delivery"
|
||||
|
||||
// BackupEvaluationAlertType reports unavailable backup-age evaluation,
|
||||
// independently of whether backup artifacts themselves are stale.
|
||||
BackupEvaluationAlertType = "backup-evaluation"
|
||||
|
||||
// DeadManDeliveryAlertType reports that Pulse is healthy but cannot reach
|
||||
// the configured external watchdog.
|
||||
DeadManDeliveryAlertType = "deadman-delivery"
|
||||
|
||||
@@ -25195,3 +25195,168 @@ func TestTrueNASSavedProbePreservesRuntimeEvidence(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Exercise the real router constructor and monitor ticker: a handler-only
|
||||
// summary test cannot detect a closure capturing nil pollers during startup.
|
||||
// The appliances are loopback analogues, not native acceptance of #2382.
|
||||
func TestRouterPlatformConnectionAlertsFollowRuntime(t *testing.T) {
|
||||
setMockModeForTest(t, false)
|
||||
t.Setenv(truenas.FeatureTrueNAS, "true")
|
||||
t.Setenv(vmware.FeatureVMware, "true")
|
||||
setTrueNASFeatureForTest(t, true)
|
||||
oldVMware := vmware.IsFeatureEnabled()
|
||||
vmware.SetFeatureEnabled(true)
|
||||
t.Cleanup(func() { vmware.SetFeatureEnabled(oldVMware) })
|
||||
|
||||
var failPools atomic.Bool
|
||||
failPools.Store(true)
|
||||
nas := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
switch r.URL.Path {
|
||||
case "/api/v2.0/system/info":
|
||||
_, _ = w.Write([]byte(`{"hostname":"fixture-nas","version":"TrueNAS-SCALE-24.10.2","uptime_seconds":100}`))
|
||||
case "/api/v2.0/pool":
|
||||
if failPools.Load() {
|
||||
w.WriteHeader(http.StatusUnauthorized)
|
||||
_, _ = w.Write([]byte(`{"error":"fixture required inventory denied"}`))
|
||||
return
|
||||
}
|
||||
_, _ = w.Write([]byte(`[{"id":1,"name":"fixture-pool","status":"ONLINE","size":1000,"allocated":400,"free":600}]`))
|
||||
case "/api/v2.0/pool/dataset", "/api/v2.0/disk", "/api/v2.0/alert/list":
|
||||
_, _ = w.Write([]byte(`[]`))
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
t.Cleanup(nas.Close)
|
||||
vc := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
http.Error(w, "fixture vCenter login denied", http.StatusUnauthorized)
|
||||
}))
|
||||
t.Cleanup(vc.Close)
|
||||
|
||||
dataDir := t.TempDir()
|
||||
cfg := &config.Config{DataPath: dataDir, ConfigPath: dataDir, PVEPollingInterval: 10 * time.Second}
|
||||
persistence := config.NewConfigPersistence(dataDir)
|
||||
tn := trueNASInstanceFromRawURL(t, "fixture-nas", nas.URL, true)
|
||||
pausedTN := tn
|
||||
pausedTN.ID, pausedTN.Enabled = "paused-nas", false
|
||||
if err := persistence.SaveTrueNASConfig([]config.TrueNASInstance{tn, pausedTN}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
vw := config.NewVMwareVCenterInstance()
|
||||
vw.ID, vw.Name, vw.Host = "fixture-vc", "fixture-vc", vc.URL
|
||||
vw.Username, vw.Password, vw.InsecureSkipVerify = "fixture-user", "fixture-password", true
|
||||
pausedVW := vw
|
||||
pausedVW.ID, pausedVW.Enabled = "paused-vc", false
|
||||
if err := persistence.SaveVMwareConfig([]config.VMwareVCenterInstance{vw, pausedVW}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
monitor, err := monitoring.New(cfg)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(monitor.Stop)
|
||||
router := NewRouter(cfg, monitor, nil, nil, nil, "fixture")
|
||||
cleanupTestRouter(t, router)
|
||||
manager := monitor.GetAlertManager()
|
||||
|
||||
rows := func() map[string]Connection {
|
||||
t.Helper()
|
||||
rec := httptest.NewRecorder()
|
||||
router.connectionsHandlers.HandleList(rec, httptest.NewRequest(http.MethodGet, "/api/connections", nil))
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("ledger status=%d", rec.Code)
|
||||
}
|
||||
var response ConnectionsListResponse
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &response); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
result := make(map[string]Connection, len(response.Connections))
|
||||
for _, row := range response.Connections {
|
||||
result[row.ID] = row
|
||||
}
|
||||
return result
|
||||
}
|
||||
active := func(id string) *alerts.Alert {
|
||||
for _, alert := range manager.GetActiveAlerts() {
|
||||
if alert.Type == "connection-degraded" && alert.ResourceID == id {
|
||||
return &alert
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
wait := func(timeout time.Duration, predicate func() bool, failure string) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(timeout)
|
||||
for time.Now().Before(deadline) {
|
||||
if predicate() {
|
||||
return
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
t.Fatal(failure)
|
||||
}
|
||||
nasID, vcID := "truenas:"+tn.ID, "vmware:"+vw.ID
|
||||
wait(5*time.Second, func() bool {
|
||||
got := rows()
|
||||
return got[nasID].State == ConnectionStateUnauthorized && got[vcID].State == ConnectionStateUnauthorized
|
||||
}, "actual platform failures did not reach the ledger")
|
||||
if active(nasID) != nil || active(vcID) != nil {
|
||||
t.Fatal("alerts bypassed the monitor confirmation ticks")
|
||||
}
|
||||
t.Log("actual TrueNAS required-method and VMware login failures reach the ledger before alerts")
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
stopped := make(chan struct{})
|
||||
go func() { defer close(stopped); monitor.Start(ctx, nil) }()
|
||||
t.Cleanup(func() {
|
||||
cancel()
|
||||
select {
|
||||
case <-stopped:
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Error("monitor loop did not stop")
|
||||
}
|
||||
})
|
||||
wait(36*time.Second, func() bool { return active(nasID) != nil && active(vcID) != nil },
|
||||
"ledger failures stayed silent in the real monitor alert loop")
|
||||
for _, id := range []string{nasID, vcID} {
|
||||
alert := active(id)
|
||||
row := rows()[id]
|
||||
if alert.Level != alerts.AlertLevelCritical || alert.Metadata["state"] != string(row.State) || alert.Metadata["lastErrorCategory"] != row.LastError.Category {
|
||||
t.Fatalf("alert did not carry the ledger's observed failure: %+v", alert)
|
||||
}
|
||||
}
|
||||
for _, id := range []string{"truenas:" + pausedTN.ID, "vmware:" + pausedVW.ID} {
|
||||
if rows()[id].State != ConnectionStatePaused || active(id) != nil {
|
||||
t.Fatalf("paused connection %s alerted", id)
|
||||
}
|
||||
}
|
||||
t.Log("both runtime failures raised confirmed critical alerts; paused peers remained silent")
|
||||
|
||||
vw.Enabled = false
|
||||
if err := persistence.SaveVMwareConfig([]config.VMwareVCenterInstance{vw, pausedVW}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
wait(13*time.Second, func() bool { return rows()[vcID].State == ConnectionStatePaused && active(vcID) == nil },
|
||||
"authoritative disable did not suppress the existing VMware alert")
|
||||
if active(nasID) == nil {
|
||||
t.Fatal("disabling VMware suppressed the independent TrueNAS failure")
|
||||
}
|
||||
t.Log("disabling VMware clears only its alert, without claiming a successful poll")
|
||||
|
||||
// Recovery comes from the next ordinary saved-connection poll, not a
|
||||
// separate Test or a restart. Keep the real cadence and confirmation policy.
|
||||
failPools.Store(false)
|
||||
wait(65*time.Second, func() bool { return rows()[nasID].State == ConnectionStateActive },
|
||||
"completed required-inventory recovery did not reach the ledger")
|
||||
if active(nasID) == nil {
|
||||
t.Fatal("one successful poll bypassed recovery confirmation")
|
||||
}
|
||||
wait(36*time.Second, func() bool { return active(nasID) == nil },
|
||||
"healthy runtime confirmation did not resolve the TrueNAS alert")
|
||||
if rows()[nasID].LastSeen == nil {
|
||||
t.Fatal("recovery omitted actual observation time")
|
||||
}
|
||||
t.Log("real TrueNAS inventory recovery reaches the ledger and clears only after monitor confirmations")
|
||||
}
|
||||
|
||||
+22
-19
@@ -432,25 +432,6 @@ func (r *Router) setupRoutes() {
|
||||
func(context.Context) *monitoring.TrueNASPoller { return r.trueNASPoller },
|
||||
func(context.Context) *monitoring.VMwarePoller { return r.vmwarePoller },
|
||||
)
|
||||
if r.monitor != nil {
|
||||
// Drive the connection-degraded alert off the same aggregator the
|
||||
// HTTP handler uses, so the active-notification stream stays in
|
||||
// lockstep with the Settings → Infrastructure badges. Single-tenant
|
||||
// only for now; multi-tenant per-org wiring is a follow-up.
|
||||
getCfg := r.configHandlers.Config
|
||||
getPersist := r.configHandlers.Persistence
|
||||
monitor := r.monitor
|
||||
trueNASPoller := r.trueNASPoller
|
||||
vmwarePoller := r.vmwarePoller
|
||||
r.monitor.SetConnectionsSnapshotLister(func() []alerts.ConnectionSnapshot {
|
||||
ctx := context.Background()
|
||||
return buildAlertConnectionSnapshotsWithRuntimeSources(ctx, getCfg(ctx), getPersist(ctx), monitor, aggregatorRuntimeSources{
|
||||
orgID: "default",
|
||||
truenasPoller: trueNASPoller,
|
||||
vmwarePoller: vmwarePoller,
|
||||
})
|
||||
})
|
||||
}
|
||||
r.availabilityHandlers = NewAvailabilityHandlers(
|
||||
r.configHandlers.Persistence,
|
||||
r.configHandlers.Monitor,
|
||||
@@ -481,6 +462,28 @@ func (r *Router) setupRoutes() {
|
||||
r.trueNASPoller.Start(r.lifecycleCtx)
|
||||
r.vmwarePoller = monitoring.NewVMwarePoller(r.multiTenant, 0)
|
||||
r.vmwarePoller.Start(r.lifecycleCtx)
|
||||
if r.monitor != nil {
|
||||
// Drive the connection-degraded alert off the same aggregator the
|
||||
// HTTP handler uses, so the active-notification stream stays in
|
||||
// lockstep with the Settings → Infrastructure badges. Single-tenant
|
||||
// only for now; multi-tenant per-org wiring is a follow-up.
|
||||
// Capture the constructed runtime owners, not the nil fields that
|
||||
// existed earlier while handlers were being registered. The monitor
|
||||
// may already be running, so publish only this complete snapshot source.
|
||||
getCfg := r.configHandlers.Config
|
||||
getPersist := r.configHandlers.Persistence
|
||||
monitor := r.monitor
|
||||
trueNASPoller := r.trueNASPoller
|
||||
vmwarePoller := r.vmwarePoller
|
||||
r.monitor.SetConnectionsSnapshotLister(func() []alerts.ConnectionSnapshot {
|
||||
ctx := context.Background()
|
||||
return buildAlertConnectionSnapshotsWithRuntimeSources(ctx, getCfg(ctx), getPersist(ctx), monitor, aggregatorRuntimeSources{
|
||||
orgID: "default",
|
||||
truenasPoller: trueNASPoller,
|
||||
vmwarePoller: vmwarePoller,
|
||||
})
|
||||
})
|
||||
}
|
||||
updateHandlers := NewUpdateHandlersWithContext(r.updateManager, r.updateHistory, r.lifecycleCtx)
|
||||
updateHandlers.SetUpdateReadinessSources(
|
||||
r.updateReadinessConfigSnapshot,
|
||||
|
||||
@@ -1217,10 +1217,11 @@ type ResourceConvertInput struct {
|
||||
|
||||
// ResourceMetricInput represents a metric value for resource conversion.
|
||||
type ResourceMetricInput struct {
|
||||
Current float64
|
||||
Total *int64
|
||||
Used *int64
|
||||
Free *int64
|
||||
Current float64
|
||||
Total *int64
|
||||
Used *int64
|
||||
Free *int64
|
||||
Observation MemoryObservation
|
||||
}
|
||||
|
||||
type ResourceAlertInput struct {
|
||||
@@ -1321,10 +1322,11 @@ func ConvertResourceToFrontend(input ResourceConvertInput) ResourceFrontend {
|
||||
|
||||
if input.Memory != nil {
|
||||
rf.Memory = &ResourceMetricFrontend{
|
||||
Current: input.Memory.Current,
|
||||
Total: input.Memory.Total,
|
||||
Used: input.Memory.Used,
|
||||
Free: input.Memory.Free,
|
||||
Current: input.Memory.Current,
|
||||
Total: input.Memory.Total,
|
||||
Used: input.Memory.Used,
|
||||
Free: input.Memory.Free,
|
||||
Observation: input.Memory.Observation,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -7,6 +7,43 @@ import (
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestGuestMemoryObservationWireContract(t *testing.T) {
|
||||
legacy := Memory{Total: 100, Used: 25, Free: 75, Usage: 25}
|
||||
payload, err := json.Marshal(legacy)
|
||||
if err != nil || strings.Contains(string(payload), "observation") {
|
||||
t.Fatalf("legacy memory unexpectedly asserts freshness: %s / %v", payload, err)
|
||||
}
|
||||
at := time.Date(2026, time.October, 4, 12, 0, 0, 123, time.UTC)
|
||||
for _, state := range []string{"current", "last-known", "unavailable"} {
|
||||
t.Run(state, func(t *testing.T) {
|
||||
memory := legacy
|
||||
memory.Observation = MemoryObservation{State: state, Source: "guest-agent-meminfo", ObservedAt: at}
|
||||
if state == "unavailable" {
|
||||
memory.Observation.ObservedAt = time.Time{}
|
||||
}
|
||||
payload, err := json.Marshal(memory)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var roundtrip Memory
|
||||
if err := json.Unmarshal(payload, &roundtrip); err != nil || roundtrip != memory {
|
||||
t.Fatalf("memory provenance changed in wire roundtrip: %s / %v", payload, err)
|
||||
}
|
||||
if strings.Contains(string(payload), "observedAt") != (state != "unavailable") {
|
||||
t.Fatalf("unknown time was manufactured: %s", payload)
|
||||
}
|
||||
if memory.HasKnownUsage() != legacy.HasKnownUsage() {
|
||||
t.Fatal("annotation changed numeric validity / threshold policy")
|
||||
}
|
||||
metric := ResourceMetricInput{Current: 25, Observation: memory.Observation}
|
||||
front := ConvertResourceToFrontend(ResourceConvertInput{Memory: &metric, CPU: &metric, Disk: &metric})
|
||||
if front.Memory == nil || front.Memory.Observation != memory.Observation || front.CPU.Observation.State != "" || front.Disk.Observation.State != "" {
|
||||
t.Fatal("memory projection lost evidence or attributed it to another metric")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetricPoint_ZeroValue(t *testing.T) {
|
||||
var point MetricPoint
|
||||
|
||||
|
||||
@@ -3085,6 +3085,17 @@ type PMGQueueStatus struct {
|
||||
UpdatedAt time.Time `json:"updatedAt"` // When this queue data was collected
|
||||
}
|
||||
|
||||
// MemoryObservation describes the provenance of a selected guest reading, not
|
||||
// guest-agent liveness or permission to run a diagnostic. Current may include a
|
||||
// normal TTL cache hit; ObservedAt remains the original source observation.
|
||||
// Last-known values are retained evidence, not a new measurement. A missing
|
||||
// observation is legacy/unknown, never an assertion of current freshness.
|
||||
type MemoryObservation struct {
|
||||
State string `json:"state"` // current, last-known, unavailable
|
||||
Source string `json:"source"`
|
||||
ObservedAt time.Time `json:"observedAt,omitzero"`
|
||||
}
|
||||
|
||||
// Memory represents memory usage
|
||||
type Memory struct {
|
||||
Total int64 `json:"total"`
|
||||
@@ -3095,11 +3106,12 @@ type Memory struct {
|
||||
// UsageUnavailable is true when total capacity is known but no cache-aware
|
||||
// source can establish current usage. Consumers must not interpret the
|
||||
// zero-valued Used, Free, or Usage fields as a measurement in that state.
|
||||
UsageUnavailable bool `json:"usageUnavailable,omitempty"`
|
||||
Usage float64 `json:"usage"`
|
||||
Balloon int64 `json:"balloon,omitempty"`
|
||||
SwapUsed int64 `json:"swapUsed,omitempty"`
|
||||
SwapTotal int64 `json:"swapTotal,omitempty"`
|
||||
UsageUnavailable bool `json:"usageUnavailable,omitempty"`
|
||||
Usage float64 `json:"usage"`
|
||||
Balloon int64 `json:"balloon,omitempty"`
|
||||
SwapUsed int64 `json:"swapUsed,omitempty"`
|
||||
SwapTotal int64 `json:"swapTotal,omitempty"`
|
||||
Observation MemoryObservation `json:"observation,omitzero"`
|
||||
}
|
||||
|
||||
// HasKnownUsage reports whether this memory value is safe for live
|
||||
|
||||
@@ -1140,10 +1140,11 @@ func (r ResourceFrontend) NormalizeCollections() ResourceFrontend {
|
||||
|
||||
// ResourceMetricFrontend represents a metric value for the frontend.
|
||||
type ResourceMetricFrontend struct {
|
||||
Current float64 `json:"current"`
|
||||
Total *int64 `json:"total,omitempty"`
|
||||
Used *int64 `json:"used,omitempty"`
|
||||
Free *int64 `json:"free,omitempty"`
|
||||
Current float64 `json:"current"`
|
||||
Total *int64 `json:"total,omitempty"`
|
||||
Used *int64 `json:"used,omitempty"`
|
||||
Free *int64 `json:"free,omitempty"`
|
||||
Observation MemoryObservation `json:"observation,omitzero"`
|
||||
}
|
||||
|
||||
// ResourceNetworkFrontend represents network metrics for the frontend.
|
||||
|
||||
@@ -0,0 +1,209 @@
|
||||
package monitoring
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/config"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/recovery"
|
||||
recoverymanager "github.com/rcourtman/pulse-go-rewrite/internal/recovery/manager"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
)
|
||||
|
||||
func backupEvaluationMonitor(t *testing.T) (*Monitor, *config.MultiTenantPersistence) {
|
||||
t.Helper()
|
||||
mtp := config.NewMultiTenantPersistence(t.TempDir())
|
||||
manager := recoverymanager.New(mtp)
|
||||
am := alerts.NewManagerWithDataDir(t.TempDir())
|
||||
t.Cleanup(am.Stop)
|
||||
cfg := am.GetConfig()
|
||||
cfg.Enabled = true
|
||||
cfg.ActivationState = alerts.ActivationActive
|
||||
cfg.BackupDefaults.Enabled = true
|
||||
cfg.BackupDefaults.WarningDays = 1
|
||||
cfg.BackupDefaults.CriticalDays = 2
|
||||
am.UpdateConfig(cfg)
|
||||
m := &Monitor{
|
||||
state: models.NewState(), alertManager: am, recoveryManager: manager,
|
||||
resourceStore: backupReadStateResourceStore([]unifiedresources.Resource{
|
||||
{ID: "vm-100", Name: "database", Type: unifiedresources.ResourceTypeVM,
|
||||
Status: unifiedresources.StatusOnline,
|
||||
Proxmox: &unifiedresources.ProxmoxData{Instance: "pve1", NodeName: "node1", VMID: 100}},
|
||||
}),
|
||||
}
|
||||
return m, mtp
|
||||
}
|
||||
|
||||
func backupEvaluationPoint(at time.Time) recovery.RecoveryPoint {
|
||||
return recovery.RecoveryPoint{
|
||||
ID: "pve-backup:fixture-100", Provider: recovery.ProviderProxmoxPVE,
|
||||
Kind: recovery.KindBackup, Mode: recovery.ModeLocal, Outcome: recovery.OutcomeSuccess,
|
||||
CompletedAt: &at, SubjectResourceID: "vm-100",
|
||||
SubjectRef: &recovery.ExternalRef{Type: "proxmox-vm", ID: "pve1:node1:100", Name: "database"},
|
||||
}
|
||||
}
|
||||
|
||||
func backupEvaluationAlerts(t *testing.T, m *Monitor) (self *alerts.Alert, ages int) {
|
||||
t.Helper()
|
||||
for _, a := range m.alertManager.GetActiveAlerts() {
|
||||
switch a.Type {
|
||||
case alerts.BackupEvaluationAlertType:
|
||||
copy := a
|
||||
self = ©
|
||||
case "backup-age":
|
||||
ages++
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
func TestBackupAlertEvaluationFailureAndFreshRecovery(t *testing.T) {
|
||||
m, mtp := backupEvaluationMonitor(t)
|
||||
store, err := m.recoveryManager.StoreForOrg("default")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
old := time.Now().UTC().Add(-4 * 24 * time.Hour)
|
||||
if err := store.UpsertPoints(context.Background(), []recovery.RecoveryPoint{backupEvaluationPoint(old)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m.checkBackupAlerts(context.Background())
|
||||
if self, ages := backupEvaluationAlerts(t, m); self != nil || ages != 1 {
|
||||
t.Fatalf("initial evaluation self=%+v age alerts=%d", self, ages)
|
||||
}
|
||||
// A read failure cannot be interpreted as an empty, healthy inventory.
|
||||
if err := store.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m.checkBackupAlerts(context.Background())
|
||||
first, ages := backupEvaluationAlerts(t, m)
|
||||
if first == nil || ages != 1 {
|
||||
t.Fatalf("failure self=%+v age alerts=%d", first, ages)
|
||||
}
|
||||
if first.ID != alerts.SystemAlertID(alerts.BackupEvaluationAlertType) || first.ResourceID != "" || first.Level != alerts.AlertLevelWarning {
|
||||
t.Fatalf("incorrect system identity %+v", first)
|
||||
}
|
||||
if !strings.Contains(first.Message, "not evaluated") || strings.Contains(first.Message, "recovery.db") {
|
||||
t.Fatalf("unsafe or unclear message %q", first.Message)
|
||||
}
|
||||
for i := 0; i < 3; i++ {
|
||||
m.checkBackupAlerts(context.Background())
|
||||
}
|
||||
repeated, ages := backupEvaluationAlerts(t, m)
|
||||
if repeated == nil || ages != 1 || !repeated.StartTime.Equal(first.StartTime) || repeated.Metadata["systemAlertFingerprint"] != first.Metadata["systemAlertFingerprint"] {
|
||||
t.Fatal("repeated failure replaced the condition or lost existing age alerts")
|
||||
}
|
||||
// The operator-visible snapshot uses the live manager, not stale arrays.
|
||||
found := false
|
||||
for _, a := range m.GetState().ActiveAlerts {
|
||||
if a.ID == first.ID {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatal("evaluation warning absent from served state")
|
||||
}
|
||||
// Reopening the same synthetic persisted data is recovery; success alone
|
||||
// clears only the self-alert while still detecting the stale backup.
|
||||
m.recoveryManager = recoverymanager.New(mtp)
|
||||
recovered, err := m.recoveryManager.StoreForOrg("default")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer recovered.Close()
|
||||
m.checkBackupAlerts(context.Background())
|
||||
if self, ages := backupEvaluationAlerts(t, m); self != nil || ages != 1 {
|
||||
t.Fatalf("read recovery self=%+v age alerts=%d", self, ages)
|
||||
}
|
||||
fresh := time.Now().UTC()
|
||||
if err := recovered.UpsertPoints(context.Background(), []recovery.RecoveryPoint{backupEvaluationPoint(fresh)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m.checkBackupAlerts(context.Background())
|
||||
if self, ages := backupEvaluationAlerts(t, m); self != nil || ages != 0 {
|
||||
t.Fatalf("fresh backup self=%+v age alerts=%d", self, ages)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupAlertEvaluationUnavailableIsNotRecovery(t *testing.T) {
|
||||
m, _ := backupEvaluationMonitor(t)
|
||||
m.recoveryManager = recoverymanager.New(nil)
|
||||
m.checkBackupAlerts(context.Background())
|
||||
self, _ := backupEvaluationAlerts(t, m)
|
||||
if self == nil {
|
||||
t.Fatal("missing persistence did not raise an evaluation warning")
|
||||
}
|
||||
m.recoveryManager = nil
|
||||
m.checkBackupAlerts(context.Background())
|
||||
if current, _ := backupEvaluationAlerts(t, m); current == nil {
|
||||
t.Fatal("missing manager falsely cleared unavailable evaluation")
|
||||
}
|
||||
cfg := m.alertManager.GetConfig()
|
||||
cfg.BackupDefaults.Enabled = false
|
||||
m.alertManager.UpdateConfig(cfg)
|
||||
m.checkBackupAlerts(context.Background())
|
||||
if current, _ := backupEvaluationAlerts(t, m); current != nil {
|
||||
t.Fatal("disabled backup checks retain an irrelevant evaluation warning")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupAlertEvaluationCancellationAndDeadline(t *testing.T) {
|
||||
m, _ := backupEvaluationMonitor(t)
|
||||
store, err := m.recoveryManager.StoreForOrg("default")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer store.Close()
|
||||
cancelled, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
m.checkBackupAlerts(cancelled)
|
||||
if self, _ := backupEvaluationAlerts(t, m); self != nil {
|
||||
t.Fatal("normal cancellation generated an outage")
|
||||
}
|
||||
deadline, stop := context.WithDeadline(context.Background(), time.Now().Add(-time.Second))
|
||||
defer stop()
|
||||
m.checkBackupAlerts(deadline)
|
||||
if self, _ := backupEvaluationAlerts(t, m); self == nil {
|
||||
t.Fatal("deadline skip stayed invisible")
|
||||
}
|
||||
// Another normal cancellation is not positive evaluation evidence.
|
||||
m.checkBackupAlerts(cancelled)
|
||||
if self, _ := backupEvaluationAlerts(t, m); self == nil {
|
||||
t.Fatal("cancellation falsely cleared an outage")
|
||||
}
|
||||
m.checkBackupAlerts(nil)
|
||||
if self, _ := backupEvaluationAlerts(t, m); self != nil {
|
||||
t.Fatal("complete empty read did not recover the evaluator")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupAlertEvaluationCoalescesInFlightReads(t *testing.T) {
|
||||
m, _ := backupEvaluationMonitor(t)
|
||||
m.recoveryManager = recoverymanager.New(nil)
|
||||
m.backupAlertEvalMu.Lock()
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < 16; i++ {
|
||||
wg.Add(1)
|
||||
go func() { defer wg.Done(); m.checkBackupAlerts(context.Background()) }()
|
||||
}
|
||||
done := make(chan struct{})
|
||||
go func() { wg.Wait(); close(done) }()
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("overlapping global reads queued behind in-flight evaluation")
|
||||
}
|
||||
m.backupAlertEvalMu.Unlock()
|
||||
if self, _ := backupEvaluationAlerts(t, m); self != nil {
|
||||
t.Fatal("coalesced read published failure without evaluating")
|
||||
}
|
||||
m.checkBackupAlerts(context.Background())
|
||||
if self, _ := backupEvaluationAlerts(t, m); self == nil {
|
||||
t.Fatal("subsequent global check did not run")
|
||||
}
|
||||
}
|
||||
@@ -1261,6 +1261,10 @@ func TestBackupOrphanDetectionUsesCanonicalInventoryReadinessScope(t *testing.T)
|
||||
"func (m *Monitor) updatePVEBackupTemplateSubjectsForType(instanceName, guestType string, subjects map[string]struct{}) {",
|
||||
"func (m *Monitor) updatePVEBackupTemplateSubjectsFromClusterResources(instanceName string, resources []proxmox.ClusterResource) {",
|
||||
"func (m *Monitor) backupInventoryScopeForAlerts() *alerts.BackupInventoryScope {",
|
||||
"m.checkBackupAlerts(ctx)",
|
||||
"m.checkBackupAlerts(context.Background())",
|
||||
},
|
||||
"recovery_rollups.go": {
|
||||
"m.alertManager.CheckBackupsWithInventory(rollups, guestsByKey, guestsByVMID, m.backupInventoryScopeForAlerts())",
|
||||
},
|
||||
"monitor_pve_guest_poll.go": {
|
||||
@@ -2976,6 +2980,26 @@ func TestGuestAgentBackupMonitoringContract(t *testing.T) {
|
||||
testGuestAgentBackupMonitoringLifecycle(t)
|
||||
}
|
||||
|
||||
func TestGuestMemoryObservationContract(t *testing.T) {
|
||||
t.Run("poll-to-served-observation", testGuestMemoryObservationLifecycle)
|
||||
t.Run("identity-and-origin-boundaries", testGuestMemoryObservationKeepsOriginsSeparate)
|
||||
t.Run("independent-linked-agent", testGuestMemoryObservationPreservesIndependentLinkedAgent)
|
||||
}
|
||||
|
||||
// Same-target, unambiguous lock evidence is required even with warm caches.
|
||||
func TestGuestAgentBackupLockEvidenceContract(t *testing.T) {
|
||||
for _, test := range []struct{ name, config string }{
|
||||
{"redirected config", "redirect"},
|
||||
{"overwritten backup lock", `{"data":{"lock":"backup","lock":""}}`},
|
||||
{"overwritten config envelope", `{"data":{"lock":"backup"},"data":{}}`},
|
||||
{"case-conflicting envelope", `{"data":{"lock":"backup"},"DATA":{}}`},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
testGuestAgentBackupMonitoringLifecycleWithUnverifiedConfig(t, test.config)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeferredVMGuestMemoryRequiresOriginalEvidence(t *testing.T) {
|
||||
now := time.Now()
|
||||
tests := []struct {
|
||||
@@ -3042,7 +3066,7 @@ func TestDeferredVMGuestMemoryRequiresOriginalEvidence(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestGuestAgentTransportMonitoringContract(t *testing.T) {
|
||||
for _, reason := range []string{"agent-redirect", "agent-transport-unverified"} {
|
||||
for _, reason := range []string{"agent-redirect", "agent-transport-unverified", "agent-completion-unverified"} {
|
||||
for _, prefix := range []string{"", "prev-"} {
|
||||
if !guestAgentDiskDeferred(prefix + reason) {
|
||||
t.Errorf("transport uncertainty %q is not labelled deferred", prefix+reason)
|
||||
@@ -3056,3 +3080,19 @@ func TestGuestAgentTransportMonitoringContract(t *testing.T) {
|
||||
}
|
||||
testGuestAgentTransportDeferralKeepsLastKnownHistory(t)
|
||||
}
|
||||
|
||||
func TestBackupAlertEvaluationCallersShareFailureVisibility(t *testing.T) {
|
||||
for path, want := range map[string]int{"monitor_backups.go": 2, "monitor_alerts.go": 1} {
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
source := string(raw)
|
||||
if strings.Contains(source, "listBackupRollupsForAlerts(") || strings.Contains(source, "CheckBackupsWithInventory(") {
|
||||
t.Errorf("%s bypasses shared failure-preserving evaluation", path)
|
||||
}
|
||||
if got := strings.Count(source, "m.checkBackupAlerts("); got != want {
|
||||
t.Errorf("%s has %d shared backup evaluations, want %d", path, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -20,6 +20,10 @@ import (
|
||||
)
|
||||
|
||||
func testGuestAgentBackupMonitoringLifecycle(t *testing.T) {
|
||||
testGuestAgentBackupMonitoringLifecycleWithUnverifiedConfig(t, "")
|
||||
}
|
||||
|
||||
func testGuestAgentBackupMonitoringLifecycleWithUnverifiedConfig(t *testing.T, unverifiedConfig string) {
|
||||
const mib = uint64(1024 * 1024)
|
||||
var phase, calls atomic.Int32
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
@@ -28,13 +32,23 @@ func testGuestAgentBackupMonitoringLifecycle(t *testing.T) {
|
||||
switch {
|
||||
case strings.HasSuffix(r.URL.Path, "/config"):
|
||||
if p == 4 {
|
||||
w.Header().Set("Content-Length", "1000")
|
||||
fmt.Fprint(w, `{"data":{}}`)
|
||||
switch unverifiedConfig {
|
||||
case "redirect":
|
||||
http.Redirect(w, r, "/unrelated-unlocked-config", http.StatusTemporaryRedirect)
|
||||
case "":
|
||||
w.Header().Set("Content-Length", "1000")
|
||||
fmt.Fprint(w, `{"data":{}}`)
|
||||
default:
|
||||
fmt.Fprint(w, unverifiedConfig)
|
||||
}
|
||||
} else if p == 1 || p == 2 {
|
||||
fmt.Fprint(w, `{"data":{"lock":"backup"}}`)
|
||||
} else {
|
||||
fmt.Fprint(w, `{"data":{}}`)
|
||||
}
|
||||
case r.URL.Path == "/unrelated-unlocked-config":
|
||||
// Never count this unrelated unlocked response as the guest's lock.
|
||||
fmt.Fprint(w, `{"data":{}}`)
|
||||
case strings.HasSuffix(r.URL.Path, "/status/current"):
|
||||
if p == 2 {
|
||||
http.Error(w, "unavailable", 503)
|
||||
@@ -111,20 +125,20 @@ func testGuestAgentBackupMonitoringLifecycle(t *testing.T) {
|
||||
memKey := guestMemoryCacheKey("fixture", "node", 105)
|
||||
originalMetadata := m.guestMetadataCache[metadataKey]
|
||||
originalMemory := m.vmAgentMemCache[memKey]
|
||||
// A syntactically unlocked prefix is not a completed config observation.
|
||||
// An incomplete, redirected or ambiguous config is not lock clearance.
|
||||
// Warm guest caches must stay labelled last-known rather than becoming new
|
||||
// disk/memory History samples when the lock response is incomplete.
|
||||
// disk/memory History samples when the lock response cannot be trusted.
|
||||
phase.Store(4)
|
||||
unverified, _ := build()
|
||||
if unverified.ID != initial.ID || unverified.GuestAgentStatus != "deferred" || unverified.DiskStatusReason != "prev-lock-unverified" || unverified.Disk.Used != initial.Disk.Used || unverified.Memory.Used != initial.Memory.Used {
|
||||
t.Fatalf("incomplete lock response lost truthful continuity: %#v", unverified)
|
||||
t.Fatalf("unverified lock response lost truthful continuity: %#v", unverified)
|
||||
}
|
||||
if calls.Load() != before || !reflect.DeepEqual(m.guestMetadataCache[metadataKey], originalMetadata) || !reflect.DeepEqual(m.vmAgentMemCache[memKey], originalMemory) {
|
||||
t.Fatal("incomplete lock response sent a guest command or renewed caches")
|
||||
t.Fatal("unverified lock response sent a guest command or renewed caches")
|
||||
}
|
||||
for metric, want := range map[string]int{"cpu": 2, "memory": 1, "memoryused": 1, "disk": 1} {
|
||||
if got := len(m.metricsHistory.GetGuestMetrics(identity, metric, time.Hour)); got != want {
|
||||
t.Errorf("incomplete lock %s history points = %d, want %d", metric, got, want)
|
||||
t.Errorf("unverified lock %s history points = %d, want %d", metric, got, want)
|
||||
}
|
||||
}
|
||||
phase.Store(1)
|
||||
@@ -141,7 +155,7 @@ func testGuestAgentBackupMonitoringLifecycle(t *testing.T) {
|
||||
if locked.ID != initial.ID || locked.GuestAgentStatus != "deferred" || locked.DiskStatusReason != "prev-vm-locked" {
|
||||
t.Fatalf("locked poll %d lost continuity/truth: %#v", poll, locked)
|
||||
}
|
||||
if locked.Memory != initial.Memory || source != "previous-snapshot" || locked.Disk.Used != initial.Disk.Used || !reflect.DeepEqual(locked.NetworkInterfaces, initial.NetworkInterfaces) {
|
||||
if guestMemoryValuesOnly(locked.Memory) != guestMemoryValuesOnly(initial.Memory) || source != "previous-snapshot" || locked.Disk.Used != initial.Disk.Used || !reflect.DeepEqual(locked.NetworkInterfaces, initial.NetworkInterfaces) {
|
||||
t.Fatalf("locked poll %d lost last-known evidence: source=%s memory=%#v want=%#v", poll, source, locked.Memory, initial.Memory)
|
||||
}
|
||||
}
|
||||
@@ -162,7 +176,7 @@ func testGuestAgentBackupMonitoringLifecycle(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
locked, _ = build()
|
||||
if calls.Load() != before || locked.GuestAgentStatus != "deferred" || locked.AgentVersion != initial.AgentVersion || locked.Memory != initial.Memory {
|
||||
if calls.Load() != before || locked.GuestAgentStatus != "deferred" || locked.AgentVersion != initial.AgentVersion || guestMemoryValuesOnly(locked.Memory) != guestMemoryValuesOnly(initial.Memory) {
|
||||
t.Fatalf("resource-lock fallback queried/lost identity: calls=%d %#v", calls.Load(), locked)
|
||||
}
|
||||
// The per-node inventory fallback must propagate its lock to the same builder.
|
||||
@@ -226,3 +240,9 @@ func TestGuestAgentDeferralPreservesMetadataAndMemoryCache(t *testing.T) {
|
||||
t.Fatal("deferral queued work, renewed cache or poisoned supported-OS evidence")
|
||||
}
|
||||
}
|
||||
|
||||
// Provenance is asserted separately; numeric continuity includes every old field.
|
||||
func guestMemoryValuesOnly(memory models.Memory) models.Memory {
|
||||
memory.Observation = models.MemoryObservation{}
|
||||
return memory
|
||||
}
|
||||
|
||||
@@ -18,7 +18,7 @@ import (
|
||||
)
|
||||
|
||||
func testGuestAgentTransportDeferralKeepsLastKnownHistory(t *testing.T) {
|
||||
for _, kind := range []string{"lost reply", "redirect"} {
|
||||
for _, kind := range []string{"lost reply", "redirect", "server error", "gateway error"} {
|
||||
t.Run(kind, func(t *testing.T) {
|
||||
const mib = uint64(1024 * 1024)
|
||||
var phase, calls, redirected atomic.Int32
|
||||
@@ -35,6 +35,15 @@ func testGuestAgentTransportDeferralKeepsLastKnownHistory(t *testing.T) {
|
||||
redirected.Add(1)
|
||||
}
|
||||
if phase.Load() == 1 && lost.CompareAndSwap(false, true) {
|
||||
if kind == "server error" || kind == "gateway error" {
|
||||
status := http.StatusInternalServerError
|
||||
if kind == "gateway error" {
|
||||
status = http.StatusBadGateway
|
||||
}
|
||||
w.WriteHeader(status)
|
||||
fmt.Fprint(w, "upstream unavailable")
|
||||
return
|
||||
}
|
||||
if kind == "redirect" {
|
||||
http.Redirect(w, r, r.URL.Path+"?redirect=1", http.StatusTemporaryRedirect)
|
||||
return
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
package monitoring
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
)
|
||||
|
||||
type guestMemoryObservationContext struct {
|
||||
source string
|
||||
providerObservedAt time.Time
|
||||
deferred bool
|
||||
previous *GuestMemorySnapshot
|
||||
linkedAgent models.Host
|
||||
}
|
||||
|
||||
// guestMemoryObservation annotates the value already selected by the builder.
|
||||
// It does not select different numeric evidence, change alert/history policy,
|
||||
// renew a cache, or run a guest command. In particular disk state is not memory
|
||||
// provenance: PVE or a linked Pulse agent may still supply an independent read.
|
||||
func (m *Monitor) guestMemoryObservation(instance, guestType, node string, vmid int, memory models.Memory, evidence guestMemoryObservationContext, now time.Time) models.MemoryObservation {
|
||||
source := CanonicalMemorySource(evidence.source)
|
||||
if !memory.HasKnownUsage() || source == "unavailable" || source == "powered-off" {
|
||||
if source != "powered-off" {
|
||||
source = "unavailable"
|
||||
}
|
||||
return models.MemoryObservation{State: "unavailable", Source: source}
|
||||
}
|
||||
observation := models.MemoryObservation{State: "current", Source: source}
|
||||
switch source {
|
||||
case "guest-agent-meminfo", "guest-agent-meminfo-derived", "previous-snapshot":
|
||||
// The cache, rather than the repeatedly refreshed diagnostic/poll
|
||||
// timestamp, owns the age of retained QGA evidence.
|
||||
previous := evidence.previous
|
||||
sameGuest := previous != nil && previous.Instance == instance && previous.GuestType == guestType && previous.Node == node && previous.VMID == vmid
|
||||
qgaOrigin := source != "previous-snapshot" || (sameGuest &&
|
||||
(previous.Memory.Observation.Source == "guest-agent-meminfo" || previous.Memory.Observation.Source == "guest-agent-meminfo-derived" ||
|
||||
(previous.Memory.Observation.State == "" && (CanonicalMemorySource(previous.MemorySource) == "guest-agent-meminfo" || CanonicalMemorySource(previous.MemorySource) == "guest-agent-meminfo-derived"))))
|
||||
if guestType == "qemu" && qgaOrigin {
|
||||
if original, ok := m.cachedGuestMemoryObservation(instance, node, vmid, memory, now); ok {
|
||||
observation = original
|
||||
}
|
||||
}
|
||||
if source == "previous-snapshot" {
|
||||
observation.State = "last-known"
|
||||
if observation.ObservedAt.IsZero() && sameGuest && previous.Memory.Total == memory.Total && previous.Memory.Used == memory.Used {
|
||||
// An annotated prior source carries its original observation.
|
||||
// A legacy previous-snapshot's RetrievedAt cannot supply it.
|
||||
if previous.Memory.Observation.State == "current" || previous.Memory.Observation.State == "last-known" {
|
||||
observation = previous.Memory.Observation
|
||||
observation.State = "last-known"
|
||||
}
|
||||
}
|
||||
} else if evidence.deferred || (!observation.ObservedAt.IsZero() && now.Sub(observation.ObservedAt) >= vmAgentMemCacheTTL) {
|
||||
observation.State = "last-known"
|
||||
}
|
||||
case "agent":
|
||||
observation.ObservedAt = evidence.linkedAgent.LastSeen
|
||||
if original := evidence.linkedAgent.Memory.Observation; original.State != "" {
|
||||
observation = original
|
||||
}
|
||||
case "available-field", "derived-free-buffers-cached", "derived-total-minus-used", "status-mem", "status-freemem", "cluster-resources":
|
||||
observation.ObservedAt = evidence.providerObservedAt
|
||||
default:
|
||||
// Unknown provenance cannot be upgraded to a current observation by
|
||||
// a new poll. Keep the selected numbers and expose the uncertainty.
|
||||
observation.State = "last-known"
|
||||
}
|
||||
if observation.ObservedAt.IsZero() || observation.ObservedAt.After(now) {
|
||||
observation.ObservedAt = time.Time{}
|
||||
observation.State = "last-known"
|
||||
}
|
||||
return observation
|
||||
}
|
||||
|
||||
func (m *Monitor) cachedGuestMemoryObservation(instance, node string, vmid int, memory models.Memory, now time.Time) (models.MemoryObservation, bool) {
|
||||
m.rrdCacheMu.RLock()
|
||||
entry, ok := m.vmAgentMemCache[guestMemoryCacheKey(instance, node, vmid)]
|
||||
m.rrdCacheMu.RUnlock()
|
||||
if !ok || entry.negative || entry.fetchedAt.IsZero() || entry.fetchedAt.After(now) || now.Sub(entry.fetchedAt) > vmAgentMemCleanupMaxAge || memory.Total <= 0 || entry.info.EffectiveAvailable > uint64(memory.Total) || memory.Used != memory.Total-int64(entry.info.EffectiveAvailable) {
|
||||
return models.MemoryObservation{}, false
|
||||
}
|
||||
expected := models.Memory{Free: int64(entry.info.EffectiveAvailable)}
|
||||
splitReclaimableMemory(&expected, entry.info.Free)
|
||||
if memory.Free != expected.Free || memory.Cache != expected.Cache {
|
||||
return models.MemoryObservation{}, false
|
||||
}
|
||||
source := "guest-agent-meminfo"
|
||||
switch entry.info.Source {
|
||||
case "meminfo-available":
|
||||
case "meminfo-derived":
|
||||
source = "guest-agent-meminfo-derived"
|
||||
default:
|
||||
return models.MemoryObservation{}, false
|
||||
}
|
||||
return models.MemoryObservation{State: "current", Source: source, ObservedAt: entry.fetchedAt}, true
|
||||
}
|
||||
@@ -0,0 +1,237 @@
|
||||
package monitoring
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/proxmox"
|
||||
)
|
||||
|
||||
// Use the served JSON rather than the proposed Go type so this control also
|
||||
// compiles against the parent that loses observation provenance entirely.
|
||||
func assertGuestMemoryObservation(t *testing.T, memory models.Memory, state, source string, at time.Time) {
|
||||
t.Helper()
|
||||
data, err := json.Marshal(memory)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var wire struct {
|
||||
Observation struct {
|
||||
State string `json:"state"`
|
||||
Source string `json:"source"`
|
||||
ObservedAt *time.Time `json:"observedAt"`
|
||||
} `json:"observation"`
|
||||
}
|
||||
if err := json.Unmarshal(data, &wire); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := wire.Observation
|
||||
if got.State != state || got.Source != source {
|
||||
t.Errorf("memory observation = %s, want %s / %s", data, state, source)
|
||||
}
|
||||
if at.IsZero() {
|
||||
if got.ObservedAt != nil {
|
||||
t.Errorf("unobserved memory acquired a timestamp: %s", data)
|
||||
}
|
||||
} else if got.ObservedAt == nil || !got.ObservedAt.Equal(at) {
|
||||
t.Errorf("memory observation time = %v, want original %v", got.ObservedAt, at)
|
||||
}
|
||||
}
|
||||
|
||||
func testGuestMemoryObservationLifecycle(t *testing.T) {
|
||||
const gib = uint64(1024 * 1024 * 1024)
|
||||
identity := makeGuestID("provenance", "node", 105)
|
||||
registry := unifiedresources.NewRegistry(nil)
|
||||
store := unifiedresources.NewMonitorAdapter(registry)
|
||||
m := &Monitor{guestMetadataLimiter: make(map[string]time.Time), rateTracker: NewRateTracker(), metricsHistory: NewMetricsHistory(32, time.Hour), resourceStore: store}
|
||||
client := &vmMemoryTrustStubClient{stubPVEClient: &stubPVEClient{}, vmAgentMemAvailable: 5 * gib,
|
||||
vmStatus: &proxmox.VMStatus{Status: "running", MaxMem: 8 * gib, Mem: 8 * gib, Agent: proxmox.VMAgentField{Value: 1}}}
|
||||
res := proxmox.ClusterResource{Type: "qemu", Node: "node", Name: "guest", VMID: 105, Status: "running", MaxMem: 8 * gib, Mem: 8 * gib}
|
||||
var previous *models.VM
|
||||
build := func() models.VM {
|
||||
t.Helper()
|
||||
vm, raw, source, notes, at, ok := m.buildVMFromClusterResource(context.Background(), "provenance", res, client, identity, nil, previous)
|
||||
if !ok {
|
||||
t.Fatal("guest disappeared")
|
||||
}
|
||||
m.recordGuestSnapshot("provenance", "qemu", "node", 105, GuestMemorySnapshot{Status: vm.Status, RetrievedAt: at, MemorySource: source, Memory: vm.Memory, Raw: raw, Notes: notes})
|
||||
m.recordGuestMetrics([]models.VM{vm}, nil, time.Now().Add(-time.Second))
|
||||
store.PopulateFromSnapshot(models.StateSnapshot{VMs: []models.VM{vm}})
|
||||
views := store.VMs()
|
||||
if len(views) != 1 {
|
||||
t.Fatal("guest did not reach unified view")
|
||||
}
|
||||
next := previousVMFromView(views[0])
|
||||
previous = &next
|
||||
return vm
|
||||
}
|
||||
assertProjection := func(vm models.VM, state, source string, at time.Time) {
|
||||
t.Helper()
|
||||
assertGuestMemoryObservation(t, vm.Memory, state, source, at)
|
||||
legacy := vm.ToFrontend()
|
||||
if legacy.Memory == nil {
|
||||
t.Fatal("legacy projection lost memory")
|
||||
}
|
||||
assertGuestMemoryObservation(t, *legacy.Memory, state, source, at)
|
||||
// REST and WebSocket state use this production conversion, including
|
||||
// registry cloning and the nested Proxmox payload the drawer consumes.
|
||||
front := m.buildBroadcastFrontendStateFromSnapshot(models.StateSnapshot{VMs: []models.VM{vm}})
|
||||
data, err := json.Marshal(front.Resources)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var wire []struct {
|
||||
Memory *models.ResourceMetricFrontend `json:"memory"`
|
||||
Proxmox struct {
|
||||
Memory models.Memory `json:"memory"`
|
||||
} `json:"proxmox"`
|
||||
}
|
||||
if err := json.Unmarshal(data, &wire); err != nil || len(wire) != 1 {
|
||||
t.Fatalf("broadcast guest memory unavailable: %s / %v", data, err)
|
||||
}
|
||||
assertGuestMemoryObservation(t, wire[0].Proxmox.Memory, state, source, at)
|
||||
if vm.Memory.HasKnownUsage() {
|
||||
if wire[0].Memory == nil {
|
||||
t.Fatal("selected memory metric disappeared")
|
||||
}
|
||||
assertGuestMemoryObservation(t, models.Memory{Observation: wire[0].Memory.Observation}, state, source, at)
|
||||
} else if wire[0].Memory != nil {
|
||||
t.Fatal("unavailable memory became a numeric metric")
|
||||
}
|
||||
if wire[0].Proxmox.Memory.Used != vm.Memory.Used || wire[0].Proxmox.Memory.UsageUnavailable != vm.Memory.UsageUnavailable {
|
||||
t.Fatal("provenance changed numeric/unknown memory projection")
|
||||
}
|
||||
}
|
||||
initial := build()
|
||||
key := guestMemoryCacheKey("provenance", "node", 105)
|
||||
original := m.vmAgentMemCache[key].fetchedAt
|
||||
if original.IsZero() || initial.Memory.Used != int64(3*gib) {
|
||||
t.Fatal("successful QGA observation missing")
|
||||
}
|
||||
assertProjection(initial, "current", "guest-agent-meminfo", original)
|
||||
cached := build()
|
||||
assertProjection(cached, "current", "guest-agent-meminfo", original)
|
||||
if client.vmAgentMemCalls != 1 {
|
||||
t.Fatal("normal cache caused a new guest read")
|
||||
}
|
||||
res.Lock, client.vmStatus.Lock = "backup", "backup"
|
||||
for poll := 0; poll < 4; poll++ {
|
||||
locked := build()
|
||||
assertProjection(locked, "last-known", "guest-agent-meminfo", original)
|
||||
if locked.Memory.Used != initial.Memory.Used || m.vmAgentMemCache[key].fetchedAt != original || client.vmAgentMemCalls != 1 {
|
||||
t.Fatal("deferral changed evidence, its age, or guest read admission")
|
||||
}
|
||||
}
|
||||
if got := len(m.metricsHistory.GetGuestMetrics(identity, "memory", time.Hour)); got != 2 {
|
||||
t.Errorf("deferred memory became fresh History points: %d, want two normal polls", got)
|
||||
}
|
||||
entry := m.vmAgentMemCache[key]
|
||||
entry.fetchedAt = time.Now().Add(-vmAgentMemCleanupMaxAge - time.Second)
|
||||
m.vmAgentMemCache[key] = entry
|
||||
expired := build()
|
||||
assertProjection(expired, "unavailable", "unavailable", time.Time{})
|
||||
if expired.Memory.HasKnownUsage() || client.vmAgentMemCalls != 1 {
|
||||
t.Fatal("expired evidence became usage or queued a guest read")
|
||||
}
|
||||
res.Lock, client.vmStatus.Lock = "", ""
|
||||
client.vmAgentMemAvailable = 4 * gib
|
||||
resumed := build()
|
||||
resumedAt := m.vmAgentMemCache[key].fetchedAt
|
||||
assertProjection(resumed, "current", "guest-agent-meminfo", resumedAt)
|
||||
if !resumedAt.After(original) || resumed.Memory.Used != int64(4*gib) || client.vmAgentMemCalls != 2 {
|
||||
t.Fatal("resumption did not replace original guest observation")
|
||||
}
|
||||
// Backup/disk deferral must not disqualify an independently current PVE
|
||||
// memory reading. Its timestamp belongs to that source, not this poll.
|
||||
statusAt := time.Now().Add(-time.Second)
|
||||
res.Lock, client.vmStatus.Lock = "backup", "backup"
|
||||
client.vmStatus.ObservedAt = statusAt
|
||||
client.vmStatus.MemInfo = &proxmox.VMMemInfo{Total: 8 * gib, Available: 6 * gib}
|
||||
live := build()
|
||||
assertProjection(live, "current", "available-field", statusAt)
|
||||
if live.Memory.Used != int64(2*gib) || live.GuestAgentStatus != "deferred" || client.vmAgentMemCalls != 2 {
|
||||
t.Fatal("independent PVE memory was replaced by deferred QGA evidence")
|
||||
}
|
||||
res.Status = "stopped"
|
||||
stopped := build()
|
||||
assertProjection(stopped, "unavailable", "powered-off", time.Time{})
|
||||
if stopped.Memory.Used != 0 {
|
||||
t.Fatal("powered-off numeric compatibility changed")
|
||||
}
|
||||
}
|
||||
|
||||
func testGuestMemoryObservationKeepsOriginsSeparate(t *testing.T) {
|
||||
now := time.Now()
|
||||
at := now.Add(-2 * time.Minute)
|
||||
observed := models.Memory{Total: 100, Used: 25, Free: 75, Usage: 25}
|
||||
previous := GuestMemorySnapshot{Instance: "site", GuestType: "qemu", Node: "node", VMID: 105, Status: "running", RetrievedAt: now,
|
||||
MemorySource: "previous-snapshot", Memory: observed}
|
||||
previous.Memory.Observation = models.MemoryObservation{State: "last-known", Source: "available-field", ObservedAt: at}
|
||||
m := &Monitor{vmAgentMemCache: map[string]agentMemCacheEntry{guestMemoryCacheKey("site", "node", 105): {
|
||||
fetchedAt: now.Add(-time.Second), info: proxmox.LinuxMemoryAvailability{Source: "meminfo-available", EffectiveAvailable: 75}}}}
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
change func(*GuestMemorySnapshot)
|
||||
source string
|
||||
at time.Time
|
||||
}{
|
||||
{name: "identical numeric QGA cache cannot replace PVE provenance", source: "available-field", at: at},
|
||||
{name: "poll timestamp does not renew legacy retained evidence", source: "previous-snapshot", change: func(p *GuestMemorySnapshot) { p.Memory.Observation = models.MemoryObservation{} }},
|
||||
{name: "foreign guest snapshot cannot lend evidence", source: "previous-snapshot", change: func(p *GuestMemorySnapshot) { p.VMID++ }},
|
||||
{name: "foreign site snapshot cannot lend evidence", source: "previous-snapshot", change: func(p *GuestMemorySnapshot) { p.Instance = "other" }},
|
||||
{name: "future original time remains unknown", source: "available-field", change: func(p *GuestMemorySnapshot) { p.Memory.Observation.ObservedAt = now.Add(time.Second) }},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
prior := previous
|
||||
if tc.change != nil {
|
||||
tc.change(&prior)
|
||||
}
|
||||
memory := observed
|
||||
memory.Observation = m.guestMemoryObservation("site", "qemu", "node", 105, memory, guestMemoryObservationContext{source: "previous-snapshot", previous: &prior}, now)
|
||||
assertGuestMemoryObservation(t, memory, "last-known", tc.source, tc.at)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func testGuestMemoryObservationPreservesIndependentLinkedAgent(t *testing.T) {
|
||||
at := time.Now().Add(-30 * time.Second)
|
||||
resource := &unifiedresources.Resource{Type: unifiedresources.ResourceTypeVM, LastSeen: time.Now(),
|
||||
Agent: &unifiedresources.AgentData{Memory: &unifiedresources.AgentMemoryMeta{Total: 100, Used: 40, Free: 60}},
|
||||
SourceStatus: map[unifiedresources.DataSource]unifiedresources.SourceStatus{unifiedresources.SourceAgent: {Status: "online", LastSeen: at}}}
|
||||
view := unifiedresources.NewVMView(resource)
|
||||
agentMemory, ok := view.LinkedAgentMemory()
|
||||
if !ok {
|
||||
t.Fatal("source-owned linked memory missing")
|
||||
}
|
||||
for _, guestType := range []string{"qemu", "lxc"} {
|
||||
t.Run(guestType, func(t *testing.T) {
|
||||
m := &Monitor{rateTracker: NewRateTracker(), guestMetadataLimiter: make(map[string]time.Time)}
|
||||
client := &vmMemoryTrustStubClient{stubPVEClient: &stubPVEClient{}, vmStatus: &proxmox.VMStatus{Status: "running", Lock: "backup", MaxMem: 100, Mem: 100}}
|
||||
res := proxmox.ClusterResource{Type: guestType, Node: "node", VMID: 105, Status: "running", Lock: "backup", MaxMem: 100, Mem: 100, ObservedAt: time.Now()}
|
||||
identity := makeGuestID("site", "node", 105)
|
||||
linked := map[string]models.Host{identity: {Memory: agentMemory, Status: "online"}}
|
||||
var memory models.Memory
|
||||
if guestType == "qemu" {
|
||||
vm, _, source, _, _, ok := m.buildVMFromClusterResource(context.Background(), "site", res, client, identity, linked, nil)
|
||||
if !ok || source != "agent" {
|
||||
t.Fatalf("linked VM agent was not selected: %s", source)
|
||||
}
|
||||
memory = vm.Memory
|
||||
} else {
|
||||
ct, _, source, _, ok := m.buildContainerFromClusterResource(context.Background(), "site", res, client, nil, linked)
|
||||
if !ok || source != "agent" {
|
||||
t.Fatalf("linked LXC agent was not selected: %s", source)
|
||||
}
|
||||
memory = ct.Memory
|
||||
}
|
||||
assertGuestMemoryObservation(t, memory, "current", "agent", at)
|
||||
if memory.Used != 40 || client.vmAgentMemCalls != 0 {
|
||||
t.Fatal("independent memory changed or caused a guest-agent read")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -1199,6 +1199,7 @@ type Monitor struct {
|
||||
pveBackupInventoryReady map[string]map[string]bool // Track PVE guest inventory readiness for backup orphan detection
|
||||
pveBackupTemplateSubjects map[string]map[string]struct{} // Track template VMIDs excluded from runtime workloads but valid for backups
|
||||
backupPermissionWarnings map[string]string // Track backup permission issues per instance (instance -> warning message)
|
||||
backupAlertEvalMu sync.Mutex // Coalesce global rollup reads and order failure/recovery publication.
|
||||
persistence *config.ConfigPersistence // Add persistence for saving updated configs
|
||||
pbsBackupPollers map[string]bool // Track PBS backup polling goroutines per instance
|
||||
pbsBackupCacheTime map[string]map[pbsBackupGroupKey]time.Time // Track when each PBS backup group was last fetched
|
||||
@@ -6984,7 +6985,7 @@ func monitorMetricInput(metric *unifiedresources.MetricValue) *models.ResourceMe
|
||||
current = math.Max(metric.Percent, metric.Value)
|
||||
}
|
||||
|
||||
result := &models.ResourceMetricInput{Current: current}
|
||||
result := &models.ResourceMetricInput{Current: current, Observation: metric.Observation}
|
||||
if metric.Total != nil {
|
||||
total := *metric.Total
|
||||
result.Total = &total
|
||||
|
||||
@@ -658,13 +658,7 @@ func (m *Monitor) checkMockAlerts() {
|
||||
Msg("Collecting resources for alert cleanup in mock mode")
|
||||
m.alertManager.CleanupAlertsForNodes(existingNodes)
|
||||
|
||||
guestsByKey, guestsByVMID := buildGuestLookupsFromReadState(m.GetUnifiedReadStateOrSnapshot(), m.guestMetadataStore)
|
||||
rollups, err := m.listBackupRollupsForAlerts(context.Background())
|
||||
if err != nil {
|
||||
log.Warn().Err(err).Msg("Failed to list recovery rollups for backup alerts")
|
||||
} else {
|
||||
m.alertManager.CheckBackupsWithInventory(rollups, guestsByKey, guestsByVMID, m.backupInventoryScopeForAlerts())
|
||||
}
|
||||
m.checkBackupAlerts(context.Background())
|
||||
|
||||
// Limit how many guests we check per cycle to prevent blocking with large datasets
|
||||
const maxGuestsPerCycle = 50
|
||||
|
||||
@@ -520,15 +520,7 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName
|
||||
// Sync backup times to VMs/Containers and republish them to canonical resources.
|
||||
m.syncGuestBackupTimesAndResourceStore()
|
||||
|
||||
if m.alertManager != nil {
|
||||
guestsByKey, guestsByVMID := buildGuestLookupsFromReadState(m.GetUnifiedReadStateOrSnapshot(), m.guestMetadataStore)
|
||||
rollups, err := m.listBackupRollupsForAlerts(ctx)
|
||||
if err != nil {
|
||||
log.Warn().Err(err).Msg("Failed to list recovery rollups for backup alerts")
|
||||
} else {
|
||||
m.alertManager.CheckBackupsWithInventory(rollups, guestsByKey, guestsByVMID, m.backupInventoryScopeForAlerts())
|
||||
}
|
||||
}
|
||||
m.checkBackupAlerts(ctx)
|
||||
|
||||
// Clear permission warning if no permission errors occurred this cycle
|
||||
if !hadPermissionError {
|
||||
@@ -1881,15 +1873,7 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien
|
||||
// Sync backup times to VMs/Containers and republish them to canonical resources.
|
||||
m.syncGuestBackupTimesAndResourceStore()
|
||||
|
||||
if m.alertManager != nil {
|
||||
guestsByKey, guestsByVMID := buildGuestLookupsFromReadState(m.GetUnifiedReadStateOrSnapshot(), m.guestMetadataStore)
|
||||
rollups, err := m.listBackupRollupsForAlerts(context.Background())
|
||||
if err != nil {
|
||||
log.Warn().Err(err).Msg("Failed to list recovery rollups for backup alerts")
|
||||
} else {
|
||||
m.alertManager.CheckBackupsWithInventory(rollups, guestsByKey, guestsByVMID, m.backupInventoryScopeForAlerts())
|
||||
}
|
||||
}
|
||||
m.checkBackupAlerts(context.Background())
|
||||
|
||||
// Immediately broadcast the updated state so frontend sees new backups
|
||||
m.broadcastStateUpdate()
|
||||
|
||||
@@ -468,6 +468,15 @@ func (m *Monitor) buildVMFromClusterResource(
|
||||
snapshotNotes = append(snapshotNotes, "preserved-original-guest-memory-during-deferral")
|
||||
}
|
||||
}
|
||||
providerObservedAt := observedAtOr(res.ObservedAt, sampleTime)
|
||||
if state.detailedStatus != nil && CanonicalMemorySource(state.memorySource) != "cluster-resources" {
|
||||
providerObservedAt = observedAtOr(state.detailedStatus.ObservedAt, sampleTime)
|
||||
}
|
||||
memory.Observation = m.guestMemoryObservation(instanceName, "qemu", res.Node, res.VMID, memory, guestMemoryObservationContext{
|
||||
source: state.memorySource, providerObservedAt: providerObservedAt,
|
||||
deferred: state.guestAgentStatus == "deferred", previous: prevSnapshot,
|
||||
linkedAgent: vmIDToHostAgent[guestID],
|
||||
}, sampleTime)
|
||||
|
||||
vm := models.VM{
|
||||
ID: guestID,
|
||||
@@ -898,7 +907,7 @@ func (m *Monitor) updateVMDisksFromGuestAgentFSInfo(
|
||||
|
||||
func guestAgentDiskDeferred(reason string) bool {
|
||||
switch strings.TrimPrefix(reason, "prev-") {
|
||||
case "vm-locked", "lock-unverified", "agent-busy", "agent-cooldown", "agent-capacity", "agent-timeout", "agent-response-incomplete", "agent-redirect", "agent-transport-unverified":
|
||||
case "vm-locked", "lock-unverified", "agent-busy", "agent-cooldown", "agent-capacity", "agent-timeout", "agent-response-incomplete", "agent-redirect", "agent-transport-unverified", "agent-completion-unverified":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
|
||||
@@ -99,8 +99,9 @@ func (m *Monitor) buildContainerFromClusterResource(
|
||||
agentHost,
|
||||
hasAgent,
|
||||
)
|
||||
previousMemory := m.previousGuestSnapshot(instanceName, "lxc", res.Node, res.VMID)
|
||||
memUsed, memorySource, _ = stabilizeGuestLowTrustMemory(
|
||||
m.previousGuestSnapshot(instanceName, "lxc", res.Node, res.VMID),
|
||||
previousMemory,
|
||||
res.Status,
|
||||
memorySource,
|
||||
memTotal,
|
||||
@@ -129,6 +130,10 @@ func (m *Monitor) buildContainerFromClusterResource(
|
||||
Usage: safePercentage(float64(clampedMemUsed), float64(memTotal)),
|
||||
}
|
||||
}
|
||||
memory.Observation = m.guestMemoryObservation(instanceName, "lxc", res.Node, res.VMID, memory, guestMemoryObservationContext{
|
||||
source: memorySource, providerObservedAt: observedAtOr(res.ObservedAt, sampleTime),
|
||||
previous: previousMemory, linkedAgent: agentHost,
|
||||
}, time.Now())
|
||||
diskUsed := res.Disk
|
||||
if diskUsed > res.MaxDisk && res.MaxDisk > 0 {
|
||||
diskUsed = res.MaxDisk
|
||||
|
||||
@@ -2,11 +2,14 @@ package monitoring
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/mock"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/recovery"
|
||||
"github.com/rs/zerolog/log"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -80,7 +83,55 @@ func (m *Monitor) listRecoveryRollupsForAlerts(ctx context.Context, kind recover
|
||||
}
|
||||
|
||||
func (m *Monitor) listBackupRollupsForAlerts(ctx context.Context) ([]recovery.ProtectionRollup, error) {
|
||||
if ctx == nil {
|
||||
ctx = context.Background()
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, 2*time.Second)
|
||||
defer cancel()
|
||||
return m.listRecoveryRollupsForAlerts(ctx, recovery.KindBackup)
|
||||
}
|
||||
|
||||
// checkBackupAlerts is shared by PVE, PBS and mock polls. A failed/partial
|
||||
// rollup read is not an empty inventory and cannot clear existing age alerts.
|
||||
// Put the lost evaluation in the existing alert list, with one stable identity
|
||||
// and no raw database/path/provider errors in its public message.
|
||||
func (m *Monitor) checkBackupAlerts(ctx context.Context) {
|
||||
if m == nil || m.alertManager == nil || !m.backupAlertEvalMu.TryLock() {
|
||||
return
|
||||
}
|
||||
defer m.backupAlertEvalMu.Unlock()
|
||||
|
||||
cfg := m.alertManager.GetConfig()
|
||||
if !cfg.Enabled || !cfg.BackupDefaults.Enabled ||
|
||||
(cfg.BackupDefaults.WarningDays <= 0 && cfg.BackupDefaults.CriticalDays <= 0) {
|
||||
// Disabling evaluation is a deliberate policy change, not a read failure.
|
||||
m.alertManager.CheckBackupsWithInventory(nil, nil, nil, nil)
|
||||
m.alertManager.ClearSystemAlert(alerts.BackupEvaluationAlertType)
|
||||
return
|
||||
}
|
||||
if m.recoveryManager == nil && !mock.IsMockEnabled() {
|
||||
// Missing dependencies cannot stand in for affirmative read recovery.
|
||||
return
|
||||
}
|
||||
rollups, err := m.listBackupRollupsForAlerts(ctx)
|
||||
if err != nil {
|
||||
// Normal shutdown/cancellation is not a new evaluation outage. A read
|
||||
// deadline, including connection-pool waiting, still needs visibility.
|
||||
if !errors.Is(err, context.Canceled) {
|
||||
log.Warn().Err(err).Msg("Failed to list recovery rollups for backup alerts")
|
||||
m.alertManager.RaiseSystemAlert(alerts.SystemAlertInput{
|
||||
Type: alerts.BackupEvaluationAlertType,
|
||||
Level: alerts.AlertLevelWarning,
|
||||
Fingerprint: "recovery-rollups-unavailable",
|
||||
Message: "Backup-age alerts were not evaluated because recovery data could not be read. Existing backup alerts have been kept; check Pulse's logs.",
|
||||
})
|
||||
}
|
||||
return
|
||||
}
|
||||
guestsByKey, guestsByVMID := buildGuestLookupsFromReadState(m.GetUnifiedReadStateOrSnapshot(), m.guestMetadataStore)
|
||||
m.alertManager.CheckBackupsWithInventory(rollups, guestsByKey, guestsByVMID, m.backupInventoryScopeForAlerts())
|
||||
// Only a complete read/evaluation (including a genuinely empty result)
|
||||
// resolves the outage. In-flight checks are coalesced, not queued, so an
|
||||
// older successful read cannot later erase a newer failure.
|
||||
m.alertManager.ClearSystemAlert(alerts.BackupEvaluationAlertType)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"go/ast"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/recovery"
|
||||
)
|
||||
|
||||
// Read the SQL used by the real reader, not a duplicate that can drift from it.
|
||||
func latestProtectionObservationSQL(t testing.TB) string {
|
||||
t.Helper()
|
||||
file, err := parser.ParseFile(token.NewFileSet(), "store_posture.go", nil, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var query string
|
||||
for _, decl := range file.Decls {
|
||||
fn, ok := decl.(*ast.FuncDecl)
|
||||
if !ok || fn.Name.Name != "listLatestProtectionProviderObservations" {
|
||||
continue
|
||||
}
|
||||
ast.Inspect(fn.Body, func(node ast.Node) bool {
|
||||
call, ok := node.(*ast.CallExpr)
|
||||
if !ok || len(call.Args) < 2 {
|
||||
return true
|
||||
}
|
||||
sel, ok := call.Fun.(*ast.SelectorExpr)
|
||||
if !ok || sel.Sel.Name != "QueryContext" {
|
||||
return true
|
||||
}
|
||||
literal, ok := call.Args[1].(*ast.BasicLit)
|
||||
if !ok || literal.Kind != token.STRING {
|
||||
return true
|
||||
}
|
||||
query, err = strconv.Unquote(literal.Value)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return false
|
||||
})
|
||||
}
|
||||
if query == "" {
|
||||
t.Fatal("could not locate the production observation query")
|
||||
}
|
||||
return query
|
||||
}
|
||||
|
||||
func observationHistoryStore(t testing.TB, rows int) *Store {
|
||||
t.Helper()
|
||||
s, err := Open(filepath.Join(t.TempDir(), "recovery.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { _ = s.Close() })
|
||||
now := time.Now().UTC().Truncate(time.Millisecond)
|
||||
latest := []recovery.ProtectionProviderObservation{
|
||||
observationForHistoryTest(t, recovery.ProviderProxmoxPVE, "pve-main", now),
|
||||
observationForHistoryTest(t, recovery.ProviderProxmoxPBS, "pbs-main", now),
|
||||
}
|
||||
if err := s.UpsertProtectionProviderObservations(context.Background(), latest); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Synthetic retained rows are deliberately irrelevant to the latest facts.
|
||||
// Their 1 KiB payload models the cost of reading wide historical evidence;
|
||||
// no reporter database or driver instrumentation is involved.
|
||||
tx, err := s.db.Begin()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer tx.Rollback()
|
||||
stmt, err := tx.Prepare(`INSERT INTO protection_provider_observations (
|
||||
id, provider, source, scope, job_state, history_completeness, permissions,
|
||||
verification_expected, observed_at_ms, ingested_at_ms, evidence_json,
|
||||
created_at_ms, updated_at_ms
|
||||
) VALUES (?, ?, 'history-fixture', ?, 'success', 'complete', 'sufficient',
|
||||
0, ?, ?, ?, ?, ?)`)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer stmt.Close()
|
||||
raw := strings.Repeat(" ", 1024) + "{}"
|
||||
for i := 0; i < rows-2; i++ {
|
||||
seed := latest[i%2]
|
||||
at := now.UnixMilli() - int64(i+1)*1000
|
||||
if _, err := stmt.Exec(fmt.Sprintf("history-%09d", i), string(seed.Provider), seed.Scope, at, at, raw, at, at); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := tx.Commit(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func observationForHistoryTest(t testing.TB, provider recovery.Provider, scope string, at time.Time) recovery.ProtectionProviderObservation {
|
||||
t.Helper()
|
||||
o, err := recovery.NewProtectionProviderObservation(provider, "history-fixture", scope,
|
||||
recovery.OutcomeSuccess, recovery.ProtectionHistoryComplete,
|
||||
operationaltrust.EvidencePermissionsSufficient, false, at, at, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return o
|
||||
}
|
||||
|
||||
// Fixed comparisons selected before execution: small history, the reporter's
|
||||
// row count, and a larger retained history. Timings are evidence, not CI gates.
|
||||
func BenchmarkLatestProtectionObservationHistory(b *testing.B) {
|
||||
for _, n := range []int{1000, 70949, 180000} {
|
||||
b.Run(fmt.Sprintf("rows-%d", n), func(b *testing.B) {
|
||||
s := observationHistoryStore(b, n)
|
||||
b.ReportAllocs()
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
obs, err := s.listLatestProtectionProviderObservations(context.Background())
|
||||
if err != nil || len(obs) != 2 {
|
||||
b.Fatalf("latest observations=%d error=%v", len(obs), err)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -455,21 +455,32 @@ func (s *Store) loadProtectionPointsForKeys(
|
||||
func (s *Store) listLatestProtectionProviderObservations(
|
||||
ctx context.Context,
|
||||
) ([]recovery.ProtectionProviderObservation, error) {
|
||||
// Enumerate narrow scope keys from the covering index, then seek the latest
|
||||
// row in each scope. Ranking SELECT * over 90 days of evidence holds the
|
||||
// sole recovery connection while reading every historical payload (#2465).
|
||||
// Keep the existing timestamp/ID tie-break and do not fall back from an
|
||||
// invalid latest observation to older, potentially reassuring evidence.
|
||||
rows, err := s.db.QueryContext(ctx, `
|
||||
SELECT
|
||||
id, provider, source, scope, job_state,
|
||||
history_completeness, permissions, verification_expected,
|
||||
observed_at_ms, ingested_at_ms, evidence_json
|
||||
FROM (
|
||||
SELECT *,
|
||||
ROW_NUMBER() OVER (
|
||||
PARTITION BY provider, scope
|
||||
ORDER BY observed_at_ms DESC, id DESC
|
||||
) AS rn
|
||||
WITH scopes AS (
|
||||
SELECT DISTINCT provider, scope
|
||||
FROM protection_provider_observations
|
||||
)
|
||||
WHERE rn = 1
|
||||
ORDER BY provider, scope
|
||||
SELECT
|
||||
observation.id, observation.provider, observation.source,
|
||||
observation.scope, observation.job_state,
|
||||
observation.history_completeness, observation.permissions,
|
||||
observation.verification_expected, observation.observed_at_ms,
|
||||
observation.ingested_at_ms, observation.evidence_json
|
||||
FROM scopes
|
||||
JOIN protection_provider_observations observation
|
||||
ON observation.id = (
|
||||
SELECT candidate.id
|
||||
FROM protection_provider_observations candidate
|
||||
WHERE candidate.provider = scopes.provider AND candidate.scope = scopes.scope
|
||||
ORDER BY candidate.observed_at_ms DESC, candidate.id DESC
|
||||
LIMIT 1
|
||||
)
|
||||
ORDER BY observation.provider, observation.scope
|
||||
`)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
|
||||
@@ -649,3 +649,104 @@ func TestStoreProtectionMetadataBackfillAddsScopeAndTypedEvidence(t *testing.T)
|
||||
t.Fatalf("ingestedAt = %s, want %s", evidence.IngestedAt, updatedAt)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLatestProtectionObservationsKeepScopeAndTieSemantics(t *testing.T) {
|
||||
s, err := Open(filepath.Join(t.TempDir(), "recovery.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer s.Close()
|
||||
now := time.Now().UTC().Truncate(time.Millisecond)
|
||||
old := observationForHistoryTest(t, recovery.ProviderProxmoxPVE, "same-scope", now.Add(-time.Hour))
|
||||
latest := observationForHistoryTest(t, recovery.ProviderProxmoxPVE, "same-scope", now)
|
||||
tie := latest
|
||||
tie.ID = "zz-latest-tie"
|
||||
tie.Evidence.ID = tie.ID
|
||||
tie.Evidence.PayloadRef = &operationaltrust.EvidencePayloadRef{Kind: "protection-provider-observation", ID: tie.ID}
|
||||
otherProvider := observationForHistoryTest(t, recovery.ProviderProxmoxPBS, "same-scope", now.Add(-time.Minute))
|
||||
otherScope := observationForHistoryTest(t, recovery.ProviderProxmoxPVE, "same-scope-2", now.Add(-time.Minute))
|
||||
if err := s.UpsertProtectionProviderObservations(context.Background(), []recovery.ProtectionProviderObservation{tie, otherScope, old, latest, otherProvider}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := s.listLatestProtectionProviderObservations(context.Background())
|
||||
if err != nil || len(got) != 3 {
|
||||
t.Fatalf("observations=%+v error=%v", got, err)
|
||||
}
|
||||
want := map[string]string{
|
||||
string(tie.Provider) + "/" + tie.Scope: tie.ID,
|
||||
string(otherScope.Provider) + "/" + otherScope.Scope: otherScope.ID,
|
||||
string(otherProvider.Provider) + "/" + otherProvider.Scope: otherProvider.ID,
|
||||
}
|
||||
previous := ""
|
||||
for _, o := range got {
|
||||
key := string(o.Provider) + "/" + o.Scope
|
||||
if want[key] != o.ID {
|
||||
t.Errorf("scope %q chose %q, want %q", key, o.ID, want[key])
|
||||
}
|
||||
if previous > key {
|
||||
t.Errorf("provider/scope ordering lost: %q > %q", previous, key)
|
||||
}
|
||||
previous = key
|
||||
}
|
||||
// A corrupt latest row must not resurrect an older reassuring observation.
|
||||
if _, err := s.db.Exec("UPDATE protection_provider_observations SET evidence_json='{' WHERE id=?", tie.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err = s.listLatestProtectionProviderObservations(context.Background())
|
||||
if err != nil || len(got) != 2 {
|
||||
t.Fatalf("corrupt latest observations=%+v error=%v", got, err)
|
||||
}
|
||||
for _, o := range got {
|
||||
if o.Provider == tie.Provider && o.Scope == tie.Scope {
|
||||
t.Fatal("fell back from corrupt latest to older evidence")
|
||||
}
|
||||
}
|
||||
// Empty and cancelled reads retain their existing meanings.
|
||||
cancelled, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
if _, err := s.listLatestProtectionProviderObservations(cancelled); err == nil {
|
||||
t.Fatal("cancelled read succeeded")
|
||||
}
|
||||
if _, err := s.db.Exec("DELETE FROM protection_provider_observations"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err = s.listLatestProtectionProviderObservations(context.Background())
|
||||
if err != nil || len(got) != 0 {
|
||||
t.Fatalf("empty observations=%+v error=%v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLatestProtectionObservationHistoryRemainsRetained(t *testing.T) {
|
||||
s := observationHistoryStore(t, 70949)
|
||||
got, err := s.listLatestProtectionProviderObservations(context.Background())
|
||||
if err != nil || len(got) != 2 {
|
||||
t.Fatalf("observations=%d error=%v", len(got), err)
|
||||
}
|
||||
var count int
|
||||
if err := s.db.QueryRow("SELECT COUNT(*) FROM protection_provider_observations").Scan(&count); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if count != 70949 {
|
||||
t.Fatalf("retained observations=%d, want 70949", count)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupRollupConnectionWaitingHonoursDeadline(t *testing.T) {
|
||||
s := observationHistoryStore(t, 1000)
|
||||
conn, err := s.db.Conn(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer conn.Close()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
|
||||
defer cancel()
|
||||
if _, _, err := s.ListRollups(ctx, recovery.ListPointsOptions{Kind: recovery.KindBackup}); err != context.DeadlineExceeded {
|
||||
t.Fatalf("pool wait error=%v, want deadline exceeded", err)
|
||||
}
|
||||
if err := conn.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, _, err := s.ListRollups(context.Background(), recovery.ListPointsOptions{Kind: recovery.KindBackup}); err != nil {
|
||||
t.Fatalf("rollups after releasing held connection: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -296,3 +296,18 @@ func recoveryOutcomeForIndex(i int) recovery.Outcome {
|
||||
func twoDigit(v int) string {
|
||||
return fmt.Sprintf("%02d", v)
|
||||
}
|
||||
|
||||
// #2465: enumerate narrow indexed scope keys and seek each latest row. Ranking
|
||||
// every historical evidence payload can monopolise the one recovery connection
|
||||
// and make the otherwise quick backup-age rollup read miss its deadline.
|
||||
func TestLatestProtectionObservationPlanAvoidsHistoricalPayloadScan(t *testing.T) {
|
||||
s := observationHistoryStore(t, 1000)
|
||||
plan := explainRecoveryQueryPlan(t, s.db, latestProtectionObservationSQL(t))
|
||||
t.Logf("production latest-observation plan:\n%s", plan)
|
||||
if !strings.Contains(plan, "USING COVERING INDEX idx_protection_provider_observations_scope") {
|
||||
t.Error("scope enumeration must read the narrow covering index, not every historical payload")
|
||||
}
|
||||
if !strings.Contains(plan, "SEARCH candidate USING INDEX idx_protection_provider_observations_scope (provider=? AND scope=?)") {
|
||||
t.Error("latest evidence must be selected through a provider/scope index seek")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -512,13 +512,19 @@ func TestCloneProxmoxData_NetworkInterfaceIsolation(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestCloneProxmoxDataMemoryIsolation(t *testing.T) {
|
||||
observation := models.MemoryObservation{State: "last-known", Source: "guest-agent-meminfo", ObservedAt: time.Now().Add(-time.Minute)}
|
||||
original := &ProxmoxData{
|
||||
Memory: &models.Memory{Total: 8 << 30, Used: 4 << 30, Usage: 50},
|
||||
Memory: &models.Memory{Total: 8 << 30, Used: 4 << 30, Usage: 50, Observation: observation},
|
||||
}
|
||||
cloned := cloneProxmoxData(original)
|
||||
if cloned.Memory.Observation != observation {
|
||||
t.Fatal("clone lost original memory observation")
|
||||
}
|
||||
|
||||
cloned.Memory.Used = 7 << 30
|
||||
if original.Memory.Used != 4<<30 {
|
||||
cloned.Memory.Observation.State = "current"
|
||||
cloned.Memory.Observation.ObservedAt = time.Now()
|
||||
if original.Memory.Used != 4<<30 || original.Memory.Observation != observation {
|
||||
t.Error("mutating cloned memory should not affect original")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"math"
|
||||
"math/rand"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/mockmode"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
@@ -35,9 +36,19 @@ func hasObservedDiskUsage(disk models.Disk) bool {
|
||||
}
|
||||
|
||||
func metricsFromHost(host models.Host) *ResourceMetrics {
|
||||
// Author freshness from server-owned report receipt time, never from a
|
||||
// guest row refreshed by another source or an agent-supplied annotation.
|
||||
memory := host.Memory
|
||||
memory.Observation = models.MemoryObservation{State: "current", Source: "agent", ObservedAt: host.LastSeen}
|
||||
if host.Status != "online" || host.LastSeen.IsZero() || host.LastSeen.After(time.Now()) {
|
||||
memory.Observation.State = "last-known"
|
||||
if host.LastSeen.After(time.Now()) {
|
||||
memory.Observation.ObservedAt = time.Time{}
|
||||
}
|
||||
}
|
||||
return buildHostMetricPayload(
|
||||
host.CPUUsage,
|
||||
host.Memory,
|
||||
memory,
|
||||
host.Disks,
|
||||
host.NetInRate,
|
||||
host.NetOutRate,
|
||||
@@ -99,7 +110,7 @@ func buildHostMetricPayload(
|
||||
metrics.CPU = &MetricValue{Value: cpuPercent, Percent: cpuPercent, Unit: "percent", Source: source}
|
||||
if memory.Total > 0 && memory.HasKnownUsage() {
|
||||
percent := percentFromReportedPercent(memory.Usage)
|
||||
metrics.Memory = &MetricValue{Used: &memory.Used, Total: &memory.Total, Percent: percent, Unit: "bytes", Source: source}
|
||||
metrics.Memory = &MetricValue{Used: &memory.Used, Total: &memory.Total, Percent: percent, Unit: "bytes", Source: source, Observation: memory.Observation}
|
||||
}
|
||||
if len(disks) > 0 {
|
||||
disk := disks[0]
|
||||
@@ -199,7 +210,7 @@ func buildVMMetricPayload(
|
||||
metrics.CPU = &MetricValue{Value: cpuPercent, Percent: cpuPercent, Unit: "percent", Source: source}
|
||||
if memory.Total > 0 && memory.HasKnownUsage() {
|
||||
percent := percentFromUsage(memory.Usage)
|
||||
metrics.Memory = &MetricValue{Used: &memory.Used, Total: &memory.Total, Percent: percent, Unit: "bytes", Source: source}
|
||||
metrics.Memory = &MetricValue{Used: &memory.Used, Total: &memory.Total, Percent: percent, Unit: "bytes", Source: source, Observation: memory.Observation}
|
||||
}
|
||||
if hasObservedDiskUsage(disk) {
|
||||
percent := percentFromUsage(disk.Usage)
|
||||
|
||||
@@ -544,3 +544,34 @@ func TestCanonicalDiskUsagePreservesUnavailableAndObservedZero(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestGuestMemoryMetricObservationFollowsSelectedSource(t *testing.T) {
|
||||
at := time.Now().Add(-time.Minute)
|
||||
retained := models.Memory{Total: 100, Used: 25, Free: 75, Usage: 25,
|
||||
Observation: models.MemoryObservation{State: "last-known", Source: "guest-agent-meminfo", ObservedAt: at}}
|
||||
vm, _ := resourceFromVM(models.VM{VMID: 105, Memory: retained})
|
||||
ct, _ := resourceFromContainer(models.Container{VMID: 106, Memory: retained})
|
||||
for _, resource := range []Resource{vm, ct} {
|
||||
if resource.Metrics.Memory.Observation != retained.Observation || resource.Proxmox.Memory.Observation != retained.Observation {
|
||||
t.Fatal("guest adapter lost selected/raw observation")
|
||||
}
|
||||
copy := cloneResource(&resource)
|
||||
copy.Metrics.Memory.Observation.State = "current"
|
||||
copy.Proxmox.Memory.Observation.Source = "changed"
|
||||
if resource.Metrics.Memory.Observation != retained.Observation || resource.Proxmox.Memory.Observation != retained.Observation {
|
||||
t.Fatal("clone aliases memory provenance")
|
||||
}
|
||||
}
|
||||
// Missing platform usage must not mark a surviving live agent metric
|
||||
// unavailable just because the retained platform facet says so.
|
||||
unavailable := models.UnavailableMemory(100)
|
||||
unavailable.Observation = models.MemoryObservation{State: "unavailable", Source: "unavailable"}
|
||||
platform, _ := resourceFromVM(models.VM{VMID: 105, Memory: unavailable})
|
||||
agent := metricsFromHost(models.Host{Status: "online", LastSeen: at,
|
||||
Memory: models.Memory{Total: 100, Used: 40, Free: 60, Usage: 40,
|
||||
Observation: models.MemoryObservation{State: "current", Source: "untrusted", ObservedAt: time.Now()}}})
|
||||
merged := mergeMetrics(&platform, agent, platform.Metrics, SourceProxmox, time.Now(), nil, nil)
|
||||
if merged.Memory == nil || merged.Memory.Observation.State != "current" || merged.Memory.Observation.Source != "agent" || !merged.Memory.Observation.ObservedAt.Equal(at) || *merged.Memory.Used != 40 {
|
||||
t.Fatalf("cross-source selection lost agent receipt or trusts supplied provenance: %+v", merged.Memory)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -347,6 +347,9 @@ type MetricValue struct {
|
||||
Percent float64 `json:"percent,omitempty"`
|
||||
Unit string `json:"unit,omitempty"`
|
||||
Source DataSource `json:"-"`
|
||||
// Memory provenance travels with the selected numeric metric, not a
|
||||
// different platform facet retained on a cross-source merged guest.
|
||||
Observation models.MemoryObservation `json:"observation,omitzero"`
|
||||
}
|
||||
|
||||
// ProxmoxData contains Proxmox-specific data for a resource.
|
||||
|
||||
@@ -88,7 +88,8 @@ func linkedAgentMemoryFromResource(r *Resource) (models.Memory, bool) {
|
||||
if r == nil || r.Agent == nil || r.Agent.Stale || r.Agent.Memory == nil {
|
||||
return models.Memory{}, false
|
||||
}
|
||||
if status, ok := r.SourceStatus[SourceAgent]; !ok || status.Status != "online" {
|
||||
status, ok := r.SourceStatus[SourceAgent]
|
||||
if !ok || status.Status != "online" {
|
||||
return models.Memory{}, false
|
||||
}
|
||||
m := r.Agent.Memory
|
||||
@@ -97,6 +98,13 @@ func linkedAgentMemoryFromResource(r *Resource) (models.Memory, bool) {
|
||||
return models.Memory{}, false
|
||||
}
|
||||
memory.Usage = float64(m.Used) / float64(m.Total) * 100
|
||||
// Guest-row LastSeen can be renewed by PVE alone. The surviving agent
|
||||
// sample keeps the agent source's timestamp after host/guest correlation.
|
||||
memory.Observation = models.MemoryObservation{State: "current", Source: "agent", ObservedAt: status.LastSeen}
|
||||
if status.LastSeen.IsZero() || status.LastSeen.After(time.Now()) {
|
||||
memory.Observation.State = "last-known"
|
||||
memory.Observation.ObservedAt = time.Time{}
|
||||
}
|
||||
return memory, memory.HasKnownUsage()
|
||||
}
|
||||
|
||||
|
||||
@@ -12,6 +12,42 @@ import (
|
||||
func ptrInt64(v int64) *int64 { return &v }
|
||||
func ptrInt(v int) *int { return &v }
|
||||
|
||||
func TestLinkedAgentMemoryObservationUsesItsOwnSourceTime(t *testing.T) {
|
||||
observedAt := time.Now().Add(-time.Minute)
|
||||
resource := &Resource{
|
||||
Type: ResourceTypeVM, LastSeen: time.Now(),
|
||||
Agent: &AgentData{Memory: &AgentMemoryMeta{Total: 100, Used: 25, Free: 75}},
|
||||
SourceStatus: map[DataSource]SourceStatus{SourceAgent: {Status: "online", LastSeen: observedAt}},
|
||||
}
|
||||
for _, guest := range []string{"qemu", "lxc"} {
|
||||
t.Run(guest, func(t *testing.T) {
|
||||
var memory models.Memory
|
||||
var ok bool
|
||||
if guest == "qemu" {
|
||||
view := NewVMView(resource)
|
||||
memory, ok = view.LinkedAgentMemory()
|
||||
} else {
|
||||
view := NewContainerView(resource)
|
||||
memory, ok = view.LinkedAgentMemory()
|
||||
}
|
||||
if !ok || memory.Observation.State != "current" || memory.Observation.Source != "agent" || !memory.Observation.ObservedAt.Equal(observedAt) {
|
||||
t.Fatalf("agent memory uses guest-row time or loses provenance: %+v", memory)
|
||||
}
|
||||
})
|
||||
}
|
||||
for _, unknownTime := range []time.Time{{}, time.Now().Add(time.Hour)} {
|
||||
resource.SourceStatus[SourceAgent] = SourceStatus{Status: "online", LastSeen: unknownTime}
|
||||
memory, ok := linkedAgentMemoryFromResource(resource)
|
||||
if !ok || memory.Observation.State != "last-known" || !memory.Observation.ObservedAt.IsZero() {
|
||||
t.Fatalf("unknown/future receipt manufactured a current reading: %+v", memory)
|
||||
}
|
||||
}
|
||||
resource.SourceStatus[SourceAgent] = SourceStatus{Status: "offline", LastSeen: observedAt}
|
||||
if _, ok := linkedAgentMemoryFromResource(resource); ok {
|
||||
t.Fatal("provenance made an offline agent sample selectable")
|
||||
}
|
||||
}
|
||||
|
||||
func assertStringSlice(t *testing.T, got, want []string) {
|
||||
t.Helper()
|
||||
if len(got) != len(want) {
|
||||
|
||||
@@ -636,11 +636,14 @@ func (c *Client) requestWithRetryUsingClient(ctx context.Context, method, path s
|
||||
// Preserve the established returned-error classification for callers;
|
||||
// this distinction changes only the emitted diagnostic severity.
|
||||
if resp.StatusCode == 401 || resp.StatusCode == 403 || resp.StatusCode == 595 {
|
||||
// Import errors package at top of file
|
||||
return nil, fmt.Errorf("authentication error: %w", apiErr)
|
||||
apiErr = fmt.Errorf("authentication error: %w", apiErr)
|
||||
}
|
||||
|
||||
return nil, apiErr
|
||||
return nil, &apiResponseError{
|
||||
statusCode: resp.StatusCode,
|
||||
guestCommandRejected: resp.StatusCode == http.StatusInternalServerError && guestAgentTerminalFailure(path, body),
|
||||
cause: apiErr,
|
||||
}
|
||||
}
|
||||
|
||||
if err := securityutil.LimitResponseBody(resp, maxResponseBodyBytes); err != nil {
|
||||
|
||||
@@ -142,18 +142,31 @@ func guestAgentPath(path string) (string, int, bool) {
|
||||
func (c *Client) verifyGuestAgentUnlocked(ctx context.Context, node string, vmid int) error {
|
||||
// The config endpoint contains the authoritative PVE operation lock even
|
||||
// when a PVE version omits it from status/current or cluster/resources.
|
||||
config, err := c.GetVMConfig(ctx, node, vmid)
|
||||
if err != nil || config == nil {
|
||||
// Unlike an ordinary config read, redirected or ambiguous evidence cannot
|
||||
// establish the lock on the endpoint about to receive this guest command.
|
||||
lockClient := *c.httpClient
|
||||
lockClient.CheckRedirect = func(*http.Request, []*http.Request) error {
|
||||
return http.ErrUseLastResponse
|
||||
}
|
||||
resp, err := c.requestWithRetryUsingClient(ctx, http.MethodGet, fmt.Sprintf("/nodes/%s/qemu/%d/config", node, vmid), nil, false, &lockClient)
|
||||
if err != nil {
|
||||
return &guestAgentDeferredError{reason: "lock-unverified", cause: err}
|
||||
}
|
||||
if value, exists := config["lock"]; exists {
|
||||
lock, ok := value.(string)
|
||||
if !ok {
|
||||
return &guestAgentDeferredError{reason: "lock-unverified"}
|
||||
}
|
||||
if strings.TrimSpace(lock) != "" {
|
||||
return &guestAgentDeferredError{reason: "vm-locked"}
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
resp.Body.Close()
|
||||
return &guestAgentDeferredError{reason: "lock-unverified"}
|
||||
}
|
||||
body, readErr := readResponseBodyLimited(resp.Body)
|
||||
closeErr := resp.Body.Close()
|
||||
if readErr != nil || closeErr != nil {
|
||||
return &guestAgentDeferredError{reason: "lock-unverified", cause: errors.Join(readErr, closeErr)}
|
||||
}
|
||||
lock, valid := guestAgentConfigLock(body)
|
||||
if !valid {
|
||||
return &guestAgentDeferredError{reason: "lock-unverified"}
|
||||
}
|
||||
if strings.TrimSpace(lock) != "" {
|
||||
return &guestAgentDeferredError{reason: "vm-locked"}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -186,13 +199,24 @@ func (c *Client) getGuestAgent(ctx context.Context, path, node string, vmid int)
|
||||
uncertain = true
|
||||
return nil, &guestAgentDeferredError{reason: "agent-response-incomplete", cause: err}
|
||||
}
|
||||
// Explicit refusals/unsupported commands remain ordinary errors. An
|
||||
// uncertain completion blocks every command, not just this method.
|
||||
// A complete gateway/server error can follow a consumed command. Use
|
||||
// the actual wire status, never an "API error" quoted in body text.
|
||||
// Specific terminal rejections remain errors, not successful telemetry.
|
||||
lower := strings.ToLower(err.Error())
|
||||
uncertain = !strings.Contains(lower, "api error") || ctx.Err() != nil || strings.Contains(lower, "timeout") || strings.Contains(lower, "timed out") || strings.Contains(lower, "wrong command id")
|
||||
uncertain = ctx.Err() != nil || strings.Contains(lower, "timeout") || strings.Contains(lower, "timed out") || strings.Contains(lower, "wrong command id")
|
||||
if uncertain {
|
||||
return nil, &guestAgentDeferredError{reason: "agent-timeout", cause: err}
|
||||
}
|
||||
var response *apiResponseError
|
||||
if !errors.As(err, &response) {
|
||||
uncertain = true
|
||||
return nil, &guestAgentDeferredError{reason: "agent-timeout", cause: err}
|
||||
}
|
||||
if response.statusCode == http.StatusRequestTimeout ||
|
||||
(response.statusCode >= 500 && !response.guestCommandRejected) {
|
||||
uncertain = true
|
||||
return nil, &guestAgentDeferredError{reason: "agent-completion-unverified", cause: err}
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
// Hold admission until the entire bounded response has arrived. Returning
|
||||
|
||||
@@ -0,0 +1,315 @@
|
||||
package proxmox
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// A complete HTTP error is not necessarily a completed QGA command: an
|
||||
// intermediary may fail after the command was accepted. Error text cannot
|
||||
// turn a gateway status into a definitive refusal.
|
||||
func TestGuestAgentHTTPFailureDefersEveryRead(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
status int
|
||||
body string
|
||||
}{
|
||||
{"request-timeout", 408, "request did not finish"},
|
||||
{"plain-server-error", 500, `{"data":null}`},
|
||||
{"gateway", 502, "upstream unavailable"},
|
||||
{"service-unavailable", 503, "temporary failure"},
|
||||
{"gateway-deadline", 504, "<html>request did not finish</html>"},
|
||||
{"node-proxy", 595, "no ticket"},
|
||||
{"unexplained-not-implemented", 501, "upstream unavailable"},
|
||||
{"gateway-quoting-refusal", 502, "upstream API error 403: permission denied"},
|
||||
{"gateway-quoting-unsupported", 502, "unsupported command: guest-get-osinfo"},
|
||||
{"server-quoting-refusal", 500, "upstream API error 403: permission denied"},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
for name, first := range backupAgentReads() {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
var calls atomic.Int32
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case strings.HasSuffix(r.URL.Path, "/config"):
|
||||
fmt.Fprint(w, `{"data":{}}`)
|
||||
case strings.HasSuffix(r.URL.Path, "/status/current"):
|
||||
fmt.Fprint(w, `{"data":{"status":"running","cpu":0.25,"diskread":4096}}`)
|
||||
default:
|
||||
if calls.Add(1) == 1 {
|
||||
w.WriteHeader(tc.status)
|
||||
fmt.Fprint(w, tc.body)
|
||||
return
|
||||
}
|
||||
backupAgentPayload(w, r)
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
c := backupTestClient(t, server.URL)
|
||||
err := first(context.Background(), c, 105)
|
||||
if !errors.Is(err, ErrGuestAgentDeferred) || GuestAgentDeferredReason(err) != "agent-completion-unverified" {
|
||||
t.Errorf("unknown completion = %v, want completion-unverified deferral", err)
|
||||
}
|
||||
// A separately constructed diagnostic cannot follow with any command.
|
||||
diagnostic := backupTestClient(t, server.URL)
|
||||
for later, read := range backupAgentReads() {
|
||||
if err := read(context.Background(), diagnostic, 105); GuestAgentDeferredReason(err) != "agent-cooldown" {
|
||||
t.Errorf("next %s = %v, want VM-wide cooldown", later, err)
|
||||
}
|
||||
}
|
||||
status, err := c.GetVMStatus(context.Background(), "node", 105)
|
||||
if err != nil || status.CPU != 0.25 || status.DiskRead != 4096 {
|
||||
t.Errorf("live non-agent counters lost: %+v %v", status, err)
|
||||
}
|
||||
if calls.Load() != 1 {
|
||||
t.Errorf("wire guest commands = %d, want one", calls.Load())
|
||||
}
|
||||
})
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestGuestAgentHTTPFailureDoesNotFailOver(t *testing.T) {
|
||||
var calls atomic.Int32
|
||||
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case strings.HasSuffix(r.URL.Path, "/nodes"):
|
||||
fmt.Fprint(w, `{"data":[]}`)
|
||||
case strings.HasSuffix(r.URL.Path, "/config"):
|
||||
fmt.Fprint(w, `{"data":{}}`)
|
||||
default:
|
||||
calls.Add(1)
|
||||
w.WriteHeader(502)
|
||||
fmt.Fprint(w, "upstream unavailable")
|
||||
}
|
||||
})
|
||||
a, b := httptest.NewServer(handler), httptest.NewServer(handler)
|
||||
defer a.Close()
|
||||
defer b.Close()
|
||||
cc := NewClusterClient("http-uncertainty", ClientConfig{Host: a.URL, TokenName: "fixture@pve!pulse", TokenValue: "fixture"}, []string{a.URL, b.URL}, nil)
|
||||
if _, err := cc.GetVMFSInfo(context.Background(), "node", 105); GuestAgentDeferredReason(err) != "agent-completion-unverified" {
|
||||
t.Errorf("first command not deferred: %v", err)
|
||||
}
|
||||
for _, endpoint := range []string{a.URL, b.URL} {
|
||||
if _, err := backupTestClient(t, endpoint).GetVMAgentInfo(context.Background(), "migrated", 105); GuestAgentDeferredReason(err) != "agent-cooldown" {
|
||||
t.Errorf("alias escaped cooldown: %v", err)
|
||||
}
|
||||
}
|
||||
if calls.Load() != 1 {
|
||||
t.Errorf("cluster wire commands = %d, want one", calls.Load())
|
||||
}
|
||||
for endpoint, healthy := range cc.GetHealthStatus() {
|
||||
if !healthy {
|
||||
t.Errorf("VM error poisoned endpoint %s", endpoint)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestGuestAgentHTTPExplicitRefusalsRemainErrors(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
status int
|
||||
body string
|
||||
}{
|
||||
{"invalid-request", 400, "invalid argument"},
|
||||
{"unauthorized", 401, "unauthorized"},
|
||||
{"forbidden", 403, "permission denied"},
|
||||
{"missing-command", 404, "command not found"},
|
||||
{"rate-limited", 429, "too many requests"},
|
||||
{"unsupported-plain", 500, "unsupported command: guest-get-osinfo"},
|
||||
{"unsupported-message", 500, `{"data":null,"message":"unsupported command: guest-get-osinfo"}`},
|
||||
{"unsupported-errors", 500, `{"data":null,"errors":{"message":"unsupported command: guest-get-osinfo"}}`},
|
||||
{"missing-os-release", 500, `{"errors":{"message":"guest agent command failed: Failed to open file '/etc/os-release': No such file or directory"}}`},
|
||||
{"missing-usr-os-release", 500, "guest agent command failed: Failed to open file '/usr/lib/os-release': No such file or directory"},
|
||||
{"missing-os-release-qmp-wrapper", 500, "VM 105 qmp command 'guest-get-osinfo' failed - Failed to open file '/etc/os-release': No such file or directory"},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
var calls atomic.Int32
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if strings.HasSuffix(r.URL.Path, "/config") {
|
||||
fmt.Fprint(w, `{"data":{}}`)
|
||||
return
|
||||
}
|
||||
if calls.Add(1) == 1 {
|
||||
w.WriteHeader(tc.status)
|
||||
fmt.Fprint(w, tc.body)
|
||||
return
|
||||
}
|
||||
backupAgentPayload(w, r)
|
||||
}))
|
||||
defer server.Close()
|
||||
c := backupTestClient(t, server.URL)
|
||||
if _, err := c.GetVMAgentInfo(context.Background(), "node", 105); err == nil || errors.Is(err, ErrGuestAgentDeferred) || !strings.Contains(err.Error(), fmt.Sprintf("API error %d", tc.status)) {
|
||||
t.Fatalf("explicit refusal lost: %v", err)
|
||||
}
|
||||
if _, err := c.GetVMFSInfo(context.Background(), "node", 105); err != nil {
|
||||
t.Fatalf("known refusal poisoned healthy command: %v", err)
|
||||
}
|
||||
if calls.Load() != 2 {
|
||||
t.Errorf("refused/healthy commands = %d", calls.Load())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestGuestAgentHTTPAmbiguousFailuresDoNotClearAdmission(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
body string
|
||||
}{
|
||||
{"wrong-command", "unsupported command: guest-get-osinfo"},
|
||||
{"proxy-prefix", "upstream rejected: unsupported command: guest-get-fsinfo"},
|
||||
{"wrong-vm", "VM 106 qmp command 'guest-get-fsinfo' failed - unsupported command: guest-get-fsinfo"},
|
||||
{"conflicting-messages", `{"message":"unsupported command: guest-get-fsinfo","errors":{"message":"upstream unavailable"}}`},
|
||||
{"duplicate-message", `{"message":"upstream unavailable","message":"unsupported command: guest-get-fsinfo"}`},
|
||||
{"escaped-duplicate-message", `{"message":"upstream unavailable","\u006dessage":"unsupported command: guest-get-fsinfo"}`},
|
||||
{"duplicate-errors", `{"errors":{"message":"upstream unavailable"},"errors":{"message":"unsupported command: guest-get-fsinfo"}}`},
|
||||
{"duplicate-nested-message", `{"errors":{"message":"upstream unavailable","message":"unsupported command: guest-get-fsinfo"}}`},
|
||||
{"success-and-error", `{"data":{"result":[]},"message":"unsupported command: guest-get-fsinfo"}`},
|
||||
{"unrecognised-detail", `{"errors":{"message":"unsupported command: guest-get-fsinfo","other":"unavailable"}}`},
|
||||
{"trailing-value", `{"message":"unsupported command: guest-get-fsinfo"} {}`},
|
||||
{"truncated-json", `{"message":"unsupported command: guest-get-fsinfo"`},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
var calls atomic.Int32
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if strings.HasSuffix(r.URL.Path, "/config") {
|
||||
fmt.Fprint(w, `{"data":{}}`)
|
||||
return
|
||||
}
|
||||
calls.Add(1)
|
||||
w.WriteHeader(500)
|
||||
fmt.Fprint(w, tc.body)
|
||||
}))
|
||||
defer server.Close()
|
||||
c := backupTestClient(t, server.URL)
|
||||
if _, err := c.GetVMFSInfo(context.Background(), "node", 105); GuestAgentDeferredReason(err) != "agent-completion-unverified" {
|
||||
t.Errorf("ambiguous error released admission: %v", err)
|
||||
}
|
||||
if _, err := c.GetVMAgentInfo(context.Background(), "node", 105); GuestAgentDeferredReason(err) != "agent-cooldown" {
|
||||
t.Errorf("ambiguous error allowed next command: %v", err)
|
||||
}
|
||||
if calls.Load() != 1 {
|
||||
t.Errorf("wire commands = %d, want one", calls.Load())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestGuestAgentHTTPTerminalFailuresRemainCommandBound(t *testing.T) {
|
||||
cases := []struct{ name, command, body string }{
|
||||
{"fs-not-found", "get-fsinfo", "The command guest-get-fsinfo has not been found"},
|
||||
{"network-not-supported", "network-get-interfaces", "unsupported command: guest-network-get-interfaces"},
|
||||
{"version-not-supported", "info", "unsupported command: guest-info"},
|
||||
{"file-read-not-supported", "file-read", "unsupported command: guest-file-read"},
|
||||
{"pve-wrapper", "get-fsinfo", "VM 105 qmp command 'guest-get-fsinfo' failed - unsupported command: guest-get-fsinfo"},
|
||||
{"not-running", "get-fsinfo", `{"data":null,"message":"QEMU guest agent is not running"}`},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
var calls atomic.Int32
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if strings.HasSuffix(r.URL.Path, "/config") {
|
||||
fmt.Fprint(w, `{"data":{}}`)
|
||||
return
|
||||
}
|
||||
if calls.Add(1) == 1 {
|
||||
w.WriteHeader(500)
|
||||
fmt.Fprint(w, tc.body)
|
||||
return
|
||||
}
|
||||
backupAgentPayload(w, r)
|
||||
}))
|
||||
defer server.Close()
|
||||
c := backupTestClient(t, server.URL)
|
||||
if _, err := c.get(context.Background(), "/nodes/node/qemu/105/agent/"+tc.command); err == nil || errors.Is(err, ErrGuestAgentDeferred) {
|
||||
t.Fatalf("known terminal failure lost: %v", err)
|
||||
}
|
||||
if _, err := c.GetVMFSInfo(context.Background(), "node", 105); err != nil {
|
||||
t.Fatalf("known completed command blocked healthy command: %v", err)
|
||||
}
|
||||
if calls.Load() != 2 {
|
||||
t.Errorf("refused/healthy commands = %d", calls.Load())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestGuestAgentHTTPOrdinaryRequestPreservesStatusAndError(t *testing.T) {
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(502)
|
||||
fmt.Fprint(w, "API error 403: permission denied")
|
||||
}))
|
||||
defer server.Close()
|
||||
c := backupTestClient(t, server.URL)
|
||||
_, err := c.get(context.Background(), "/nodes")
|
||||
var response *apiResponseError
|
||||
if !errors.As(err, &response) || response.statusCode != 502 || response.guestCommandRejected || err.Error() != "API error 502: API error 403: permission denied" {
|
||||
t.Fatalf("wire status/ordinary diagnostic lost: %v", err)
|
||||
}
|
||||
if errors.Is(err, ErrGuestAgentDeferred) {
|
||||
t.Fatal("ordinary API acquired guest uncertainty")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGuestAgentHTTPUncertaintyExpiresAndKeepsOtherVMsIndependent(t *testing.T) {
|
||||
var calls atomic.Int32
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if strings.HasSuffix(r.URL.Path, "/config") {
|
||||
fmt.Fprint(w, `{"data":{}}`)
|
||||
return
|
||||
}
|
||||
if calls.Add(1) == 1 {
|
||||
w.WriteHeader(502)
|
||||
fmt.Fprint(w, "upstream unavailable")
|
||||
return
|
||||
}
|
||||
backupAgentPayload(w, r)
|
||||
}))
|
||||
defer server.Close()
|
||||
c := backupTestClient(t, server.URL)
|
||||
if _, err := c.GetVMFSInfo(context.Background(), "node", 105); GuestAgentDeferredReason(err) != "agent-completion-unverified" {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := c.GetVMFSInfo(context.Background(), "node", 106); err != nil {
|
||||
t.Fatalf("other VM blocked: %v", err)
|
||||
}
|
||||
if _, err := c.GetVMAgentInfo(context.Background(), "node", 105); GuestAgentDeferredReason(err) != "agent-cooldown" {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Advance only this synthetic cooldown, not wall time or a native target.
|
||||
guestAgentGuards.Lock()
|
||||
key := guestAgentGuardKey{endpoint: c.baseURL, vmid: 105}
|
||||
entry, ok := guestAgentGuards.entries[key]
|
||||
if ok {
|
||||
entry.until = time.Now().Add(-time.Second)
|
||||
guestAgentGuards.entries[key] = entry
|
||||
}
|
||||
guestAgentGuards.Unlock()
|
||||
if !ok {
|
||||
t.Fatal("uncertain VM had no retained admission")
|
||||
}
|
||||
if _, err := c.GetVMFSInfo(context.Background(), "node", 105); err != nil {
|
||||
t.Fatalf("unlocked completed response did not resume: %v", err)
|
||||
}
|
||||
guestAgentGuards.Lock()
|
||||
_, retained := guestAgentGuards.entries[key]
|
||||
guestAgentGuards.Unlock()
|
||||
if retained {
|
||||
t.Fatal("healthy resumption retained cooldown")
|
||||
}
|
||||
if calls.Load() != 3 {
|
||||
t.Errorf("wire commands = %d, want uncertain/other/resumed", calls.Load())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
package proxmox
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// guestAgentConfigLock accepts only one complete, unambiguous config envelope.
|
||||
// In particular, encoding/json's last-key-wins and case-insensitive struct
|
||||
// matching must not erase a backup lock. Ordinary config decoding is unchanged.
|
||||
func guestAgentConfigLock(body []byte) (string, bool) {
|
||||
envelope, ok := guestAgentResponseFields(body, "data")
|
||||
if !ok {
|
||||
return "", false
|
||||
}
|
||||
config, ok := guestAgentResponseFields(envelope["data"])
|
||||
if !ok {
|
||||
return "", false
|
||||
}
|
||||
for name := range config {
|
||||
if name != "lock" && strings.EqualFold(name, "lock") {
|
||||
return "", false
|
||||
}
|
||||
}
|
||||
value, exists := config["lock"]
|
||||
if !exists {
|
||||
return "", true // PVE omits this field when no operation holds the lock.
|
||||
}
|
||||
var lock *string
|
||||
if json.Unmarshal(value, &lock) != nil || lock == nil {
|
||||
return "", false
|
||||
}
|
||||
return *lock, true
|
||||
}
|
||||
@@ -0,0 +1,182 @@
|
||||
package proxmox
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestGuestAgentLockEvidenceRejectsAmbiguity(t *testing.T) {
|
||||
cases := []struct{ name, body string }{
|
||||
{"duplicate-lock", `{"data":{"lock":"backup","lock":""}}`},
|
||||
{"duplicate-unlocked", `{"data":{"lock":"","lock":""}}`},
|
||||
{"escaped-lock", `{"data":{"lock":"backup","\u006cock":""}}`},
|
||||
{"duplicate-data", `{"data":{"lock":"backup"},"data":{}}`},
|
||||
{"escaped-data", `{"data":{"lock":"backup"},"\u0064ata":{}}`},
|
||||
{"case-data", `{"data":{"lock":"backup"},"DATA":{}}`},
|
||||
{"case-only-data", `{"Data":{}}`},
|
||||
{"case-only-lock", `{"data":{"Lock":"backup"}}`},
|
||||
{"case-lock", `{"data":{"lock":"","LOCK":"backup"}}`},
|
||||
{"competing-error", `{"data":{},"errors":{"message":"lock unavailable"}}`},
|
||||
{"competing-failure", `{"data":{},"success":false}`},
|
||||
{"duplicate-config-field", `{"data":{"name":"first","name":"second"}}`},
|
||||
{"null-lock", `{"data":{"lock": null }}`},
|
||||
{"number-lock", `{"data":{"lock":0}}`},
|
||||
{"object-lock", `{"data":{"lock":{}}}`},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
for name, read := range backupAgentReads() {
|
||||
for _, postCommand := range []bool{false, true} {
|
||||
t.Run(fmt.Sprintf("%s/post=%t", name, postCommand), func(t *testing.T) {
|
||||
var configs, commands atomic.Int32
|
||||
var healthy atomic.Bool
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if strings.HasSuffix(r.URL.Path, "/config") {
|
||||
if configs.Add(1) == 1 && postCommand || healthy.Load() {
|
||||
fmt.Fprint(w, `{"data":{"name":"fixture","agent":1}}`)
|
||||
} else {
|
||||
fmt.Fprint(w, tc.body)
|
||||
}
|
||||
return
|
||||
}
|
||||
commands.Add(1)
|
||||
backupAgentPayload(w, r)
|
||||
}))
|
||||
defer server.Close()
|
||||
c := backupTestClient(t, server.URL)
|
||||
if err := read(context.Background(), c, 105); GuestAgentDeferredReason(err) != "lock-unverified" {
|
||||
t.Errorf("ambiguous lock admitted/published a command: %v", err)
|
||||
}
|
||||
want := int32(0)
|
||||
if postCommand {
|
||||
want = 1
|
||||
}
|
||||
if got := commands.Load(); got != want {
|
||||
t.Errorf("wire commands = %d, want %d", got, want)
|
||||
}
|
||||
healthy.Store(true)
|
||||
// A completed command's failed lock check is not transport
|
||||
// uncertainty. A new diagnostic must reverify and resume.
|
||||
diagnostic := backupTestClient(t, server.URL)
|
||||
if err := read(context.Background(), diagnostic, 105); err != nil {
|
||||
t.Fatalf("complete unlocked config did not resume: %v", err)
|
||||
}
|
||||
if got := commands.Load(); got != want+1 {
|
||||
t.Errorf("resumed wire commands = %d, want %d", got, want+1)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestGuestAgentLockEvidenceRejectsRedirects(t *testing.T) {
|
||||
for _, status := range []int{301, 302, 303, 307, 308} {
|
||||
for _, crossEndpoint := range []bool{false, true} {
|
||||
for name, read := range backupAgentReads() {
|
||||
for _, postCommand := range []bool{false, true} {
|
||||
t.Run(fmt.Sprintf("%d/cross=%t/%s/post=%t", status, crossEndpoint, name, postCommand), func(t *testing.T) {
|
||||
var configs, commands, redirected atomic.Int32
|
||||
var healthy atomic.Bool
|
||||
other := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
redirected.Add(1)
|
||||
fmt.Fprint(w, `{"data":{}}`)
|
||||
}))
|
||||
defer other.Close()
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case strings.HasSuffix(r.URL.Path, "/config"):
|
||||
if configs.Add(1) == 1 && postCommand || healthy.Load() {
|
||||
fmt.Fprint(w, `{"data":{}}`)
|
||||
} else {
|
||||
destination := "/unrelated-unlocked-config"
|
||||
if crossEndpoint {
|
||||
destination = other.URL + destination
|
||||
}
|
||||
http.Redirect(w, r, destination, status)
|
||||
}
|
||||
case r.URL.Path == "/unrelated-unlocked-config":
|
||||
redirected.Add(1)
|
||||
fmt.Fprint(w, `{"data":{}}`)
|
||||
default:
|
||||
commands.Add(1)
|
||||
backupAgentPayload(w, r)
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
c := backupTestClient(t, server.URL)
|
||||
if err := read(context.Background(), c, 105); GuestAgentDeferredReason(err) != "lock-unverified" {
|
||||
t.Errorf("redirected lock admitted/published a command: %v", err)
|
||||
}
|
||||
want := int32(0)
|
||||
if postCommand {
|
||||
want = 1
|
||||
}
|
||||
if commands.Load() != want || redirected.Load() != 0 {
|
||||
t.Errorf("wire commands/redirected = %d/%d, want %d/0", commands.Load(), redirected.Load(), want)
|
||||
}
|
||||
// Ordinary config reads retain their existing redirect
|
||||
// policy; do not globally mutate c.httpClient.
|
||||
if _, err := c.GetVMConfig(context.Background(), "node", 105); err != nil || redirected.Load() != 1 {
|
||||
t.Errorf("ordinary config redirect policy changed: redirects=%d err=%v", redirected.Load(), err)
|
||||
}
|
||||
healthy.Store(true)
|
||||
if err := read(context.Background(), backupTestClient(t, server.URL), 105); err != nil {
|
||||
t.Fatalf("valid same-target config did not resume: %v", err)
|
||||
}
|
||||
if commands.Load() != want+1 || redirected.Load() != 1 {
|
||||
t.Errorf("resumed commands/redirected = %d/%d, want %d/1", commands.Load(), redirected.Load(), want+1)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestGuestAgentLockEvidenceHealthyConfigAndSessionRecovery(t *testing.T) {
|
||||
for _, body := range []string{
|
||||
`{"data":{}}`,
|
||||
`{"data":{"lock":"","agent":1,"digest":"fixture","name":"fixture"}}`,
|
||||
`{"data":{"lock":" ","description":"backup; lock","scsi0":"fixture:disk","tags":"backup;ha"}}`,
|
||||
`{"data":{"\u006cock":"","nested":{"data":"fixture","lock":"backup"}}}`, // Only the actual operation lock is authoritative.
|
||||
} {
|
||||
t.Run(fmt.Sprintf("config-%d", len(body)), func(t *testing.T) {
|
||||
var configs, commands, authentications atomic.Int32
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case strings.HasSuffix(r.URL.Path, "/access/ticket"):
|
||||
authentications.Add(1)
|
||||
fmt.Fprint(w, `{"data":{"ticket":"fixture-ticket","CSRFPreventionToken":"fixture-csrf"}}`)
|
||||
case strings.HasSuffix(r.URL.Path, "/config"):
|
||||
if configs.Add(1) == 1 {
|
||||
http.Error(w, "expired config session", http.StatusUnauthorized)
|
||||
return
|
||||
}
|
||||
fmt.Fprint(w, body)
|
||||
default:
|
||||
commands.Add(1)
|
||||
backupAgentPayload(w, r)
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
c, err := NewClient(ClientConfig{Host: server.URL, User: "fixture@pam", Password: "fixture-password", Timeout: time.Second})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := backupAgentReads()["filesystem"](context.Background(), c, 105); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if configs.Load() != 3 || authentications.Load() != 2 || commands.Load() != 1 {
|
||||
t.Errorf("config recovery/command identity lost: configs=%d auth=%d commands=%d", configs.Load(), authentications.Load(), commands.Load())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,129 @@
|
||||
package proxmox
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Keep the wire status separate from the existing diagnostic text. In
|
||||
// particular, a proxy's error body can quote a different HTTP status.
|
||||
type apiResponseError struct {
|
||||
statusCode int
|
||||
guestCommandRejected bool
|
||||
cause error
|
||||
}
|
||||
|
||||
func (e *apiResponseError) Error() string { return e.cause.Error() }
|
||||
func (e *apiResponseError) Unwrap() error { return e.cause }
|
||||
|
||||
// Only specific, complete terminal failures from the command endpoint allow
|
||||
// another command after an HTTP 500. An unexplained 5xx, even with a complete
|
||||
// body, does not establish that a serial QGA command has finished.
|
||||
func guestAgentTerminalFailure(path string, body []byte) bool {
|
||||
_, vmid, ok := guestAgentPath(path)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
message, ok := guestAgentErrorMessage(body)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
commandPath := strings.SplitN(path, "?", 2)[0]
|
||||
command := "guest-" + commandPath[strings.LastIndex(commandPath, "/")+1:]
|
||||
message = strings.TrimPrefix(message, fmt.Sprintf("VM %d qmp command '%s' failed - ", vmid, command))
|
||||
switch message {
|
||||
case "QEMU guest agent is not running",
|
||||
"unsupported command: " + command,
|
||||
"The command " + command + " has not been found":
|
||||
return true
|
||||
}
|
||||
// Preserve the existing OpenBSD OS-info skip rather than repeatedly probing
|
||||
// its missing os-release file. Do not apply it to another guest command.
|
||||
if command == "guest-get-osinfo" {
|
||||
message = strings.TrimPrefix(message, "guest agent command failed: ")
|
||||
for _, filename := range []string{"/etc/os-release", "/usr/lib/os-release"} {
|
||||
if message == "Failed to open file '"+filename+"': No such file or directory" {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func guestAgentErrorMessage(body []byte) (string, bool) {
|
||||
body = bytes.TrimSpace(body)
|
||||
if len(body) == 0 {
|
||||
return "", false
|
||||
}
|
||||
if body[0] != '{' {
|
||||
return string(body), true // Still requires an exact, command-bound match.
|
||||
}
|
||||
fields, ok := guestAgentResponseFields(body, "data", "message", "errors")
|
||||
if !ok {
|
||||
return "", false
|
||||
}
|
||||
if data, exists := fields["data"]; exists && string(bytes.TrimSpace(data)) != "null" {
|
||||
return "", false
|
||||
}
|
||||
message, hasMessage := fields["message"]
|
||||
if nested, hasErrors := fields["errors"]; hasErrors {
|
||||
if hasMessage {
|
||||
return "", false // Conflicting or duplicate failure evidence.
|
||||
}
|
||||
errors, ok := guestAgentResponseFields(nested, "message")
|
||||
if !ok {
|
||||
return "", false
|
||||
}
|
||||
message = errors["message"]
|
||||
}
|
||||
var text string
|
||||
if json.Unmarshal(message, &text) != nil {
|
||||
return "", false
|
||||
}
|
||||
return strings.TrimSpace(text), true
|
||||
}
|
||||
|
||||
// Reject ambiguous envelopes (including duplicate keys and trailing values)
|
||||
// instead of letting a favourable final field override other error evidence.
|
||||
// Without an allowlist, arbitrary unique configuration fields are permitted.
|
||||
func guestAgentResponseFields(body []byte, allowed ...string) (map[string]json.RawMessage, bool) {
|
||||
decoder := json.NewDecoder(bytes.NewReader(body))
|
||||
start, err := decoder.Token()
|
||||
if err != nil || start != json.Delim('{') {
|
||||
return nil, false
|
||||
}
|
||||
fields := make(map[string]json.RawMessage)
|
||||
for decoder.More() {
|
||||
key, err := decoder.Token()
|
||||
if err != nil {
|
||||
return nil, false
|
||||
}
|
||||
name, ok := key.(string)
|
||||
if !ok {
|
||||
return nil, false
|
||||
}
|
||||
permitted := len(allowed) == 0
|
||||
for _, candidate := range allowed {
|
||||
permitted = permitted || name == candidate
|
||||
}
|
||||
if _, duplicate := fields[name]; !permitted || duplicate {
|
||||
return nil, false
|
||||
}
|
||||
var value json.RawMessage
|
||||
if decoder.Decode(&value) != nil {
|
||||
return nil, false
|
||||
}
|
||||
fields[name] = value
|
||||
}
|
||||
end, err := decoder.Token()
|
||||
if err != nil || end != json.Delim('}') {
|
||||
return nil, false
|
||||
}
|
||||
if _, err := decoder.Token(); err != io.EOF {
|
||||
return nil, false
|
||||
}
|
||||
return fields, true
|
||||
}
|
||||
@@ -178,7 +178,7 @@ func TestUnverifiedVMConfigPreservesStatusCounterObservations(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestGuestAgentTransportDeferralPreservesLiveCounterReceipts(t *testing.T) {
|
||||
for _, kind := range []string{"lost reply", "redirect"} {
|
||||
for _, kind := range []string{"lost reply", "redirect", "server error", "gateway error"} {
|
||||
t.Run(kind, func(t *testing.T) {
|
||||
var commands, statusCalls atomic.Int32
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
@@ -190,6 +190,15 @@ func TestGuestAgentTransportDeferralPreservesLiveCounterReceipts(t *testing.T) {
|
||||
fmt.Fprint(w, `{"data":{"status":"running","cpu":0.25,"diskread":0,"diskwrite":null,"netin":42}}`)
|
||||
case strings.Contains(r.URL.Path, "/agent/"):
|
||||
commands.Add(1)
|
||||
if kind == "server error" || kind == "gateway error" {
|
||||
status := http.StatusInternalServerError
|
||||
if kind == "gateway error" {
|
||||
status = http.StatusBadGateway
|
||||
}
|
||||
w.WriteHeader(status)
|
||||
fmt.Fprint(w, "upstream unavailable")
|
||||
return
|
||||
}
|
||||
if kind == "redirect" {
|
||||
http.Redirect(w, r, "/unverified/agent", http.StatusTemporaryRedirect)
|
||||
return
|
||||
@@ -210,6 +219,9 @@ func TestGuestAgentTransportDeferralPreservesLiveCounterReceipts(t *testing.T) {
|
||||
if kind == "redirect" {
|
||||
wantReason = "agent-redirect"
|
||||
}
|
||||
if kind == "server error" || kind == "gateway error" {
|
||||
wantReason = "agent-completion-unverified"
|
||||
}
|
||||
if _, err := client.GetVMFSInfo(context.Background(), "node", 105); GuestAgentDeferredReason(err) != wantReason {
|
||||
t.Fatalf("uncertain command not deferred: %v", err)
|
||||
}
|
||||
|
||||
@@ -80,6 +80,11 @@
|
||||
"incomplete_commit": "8b56fc72e4ee79f3c219cf51f4c002e2e9343286",
|
||||
"completion_commit": "1026f5293b4c4e5a11651933e7ed36d398fd2f69",
|
||||
"reason": "Evaluate the Proxmox LXC linked-agent memory preference runtime change together with the follow-up commit that records its monitoring and unified-resources contract sections, as one reviewed completion unit."
|
||||
},
|
||||
{
|
||||
"incomplete_commit": "0baee0bccffca078eb8598b2ed4ff481e4227b1e",
|
||||
"completion_commit": "a8d049cc585e10083dcf3b29334a0643ab063e6b",
|
||||
"reason": "Evaluate the intermediate reviewed report-form documentation-link correction together with its additive self-contained feedback safety contract and connected docsLinks regression. The final forms preserve all precautions without links to unpublished documentation; reviewed history is unchanged."
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -14,6 +14,7 @@ import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
import textwrap
|
||||
import time
|
||||
import unittest
|
||||
|
||||
@@ -26,6 +27,32 @@ REAL_TIMEOUT = shutil.which("timeout")
|
||||
PRIVATE = "synthetic-private-infrastructure-marker"
|
||||
READS = [["status", "100"], ["config", "100", "--current"]]
|
||||
|
||||
|
||||
def guide_section(guide, heading):
|
||||
"""Select one named H3, never another section's executable examples."""
|
||||
sections, current = [], None
|
||||
fenced = False
|
||||
for line in guide.splitlines(keepends=True):
|
||||
boundary = None if fenced else re.match(r"^(#{1,3}) (.*?)\s*$", line)
|
||||
if boundary:
|
||||
if current is not None:
|
||||
sections.append("".join(current))
|
||||
current = [] if boundary.group(1) == "###" and boundary.group(2) == heading else None
|
||||
elif current is not None:
|
||||
current.append(line)
|
||||
if line.lstrip().startswith("```"):
|
||||
fenced = not fenced
|
||||
if current is not None:
|
||||
sections.append("".join(current))
|
||||
if len(sections) != 1:
|
||||
raise AssertionError(f"Expected one guide section: {heading}")
|
||||
return sections[0]
|
||||
|
||||
|
||||
def bash_examples(section):
|
||||
return [textwrap.dedent(block).strip()
|
||||
for block in re.findall(r"```bash\n(.*?)```", section, re.DOTALL)]
|
||||
|
||||
# Record every attempted operation, including prohibited ones. The timeout
|
||||
# adapter checks the helper's real deadline arguments; one test uses real
|
||||
# coreutils timeout around the synthetic blocked qm child.
|
||||
@@ -250,7 +277,7 @@ class VMDiskDiagnosticsTest(unittest.TestCase):
|
||||
self.assert_passive(calls, operations=READS[:1])
|
||||
|
||||
def test_exact_documented_command_uses_the_passive_helper(self):
|
||||
recipes = re.findall(r"```bash\n(.*?)```", DOC.read_text(), re.DOTALL)
|
||||
recipes = bash_examples(guide_section(DOC.read_text(), "Passive host preflight"))
|
||||
self.assertEqual(len(recipes), 1)
|
||||
result, calls = self.exercise(recipe=recipes[0])
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
@@ -272,6 +299,47 @@ class VMDiskDiagnosticsTest(unittest.TestCase):
|
||||
|
||||
|
||||
class VMDiskHelpTest(unittest.TestCase):
|
||||
def test_section_selection_preserves_shell_comments_and_rejects_missing_or_duplicate_headings(self):
|
||||
content = "```bash\n# From the reviewed checkout\n### Shell comment, not a heading\ntrue\n```\n"
|
||||
guide = "### Passive host preflight\n" + content + "## Next\nOther text\n"
|
||||
self.assertEqual(guide_section(guide, "Passive host preflight"), content)
|
||||
for invalid in ("## No passive section\n", guide + "### Passive host preflight\n"):
|
||||
with self.subTest(guide=invalid), self.assertRaises(AssertionError):
|
||||
guide_section(invalid, "Passive host preflight")
|
||||
|
||||
def assert_guide_commands(self, guide):
|
||||
# The planned server pause is intentionally active, unlike the passive
|
||||
# hypervisor diagnostic. Admit only the six reviewed server/timer blocks
|
||||
# here; do not permit arbitrary systemctl or guest commands elsewhere.
|
||||
precaution = guide_section(guide, "Pause Pulse for a planned freeze-enabled backup")
|
||||
self.assertEqual(bash_examples(precaution), [
|
||||
"systemctl show pulse.service --property=LoadState,ActiveState,MainPID",
|
||||
"systemctl show pulse-update.timer --property=LoadState,ActiveState",
|
||||
"sudo systemctl stop pulse-update.timer\n"
|
||||
"systemctl show pulse-update.timer pulse-update.service \\\n"
|
||||
" --property=Id,LoadState,ActiveState,MainPID",
|
||||
"sudo systemctl stop pulse.service\n"
|
||||
"systemctl show pulse.service --property=LoadState,ActiveState,MainPID",
|
||||
"sudo systemctl start pulse.service\nsystemctl is-active pulse.service",
|
||||
"sudo systemctl start pulse-update.timer",
|
||||
])
|
||||
for block in bash_examples(guide.replace(precaution, "", 1)):
|
||||
self.assertNotRegex(block, r"\b(curl|wget|qm agent|pveum|systemctl)\b")
|
||||
|
||||
def test_command_boundaries_reject_active_diagnostics_and_unreviewed_pause_operations(self):
|
||||
guide = DOC.read_text()
|
||||
mutations = [
|
||||
guide.replace("sudo bash ./scripts/test-vm-disk.sh 100",
|
||||
"systemctl restart pulse.service"),
|
||||
guide.replace("sudo systemctl stop pulse.service", "sudo systemctl restart pulse.service"),
|
||||
guide.replace("sudo systemctl start pulse-update.timer", "qm agent 100 ping"),
|
||||
guide + "\n```bash\nsystemctl stop pulse.service\n```\n",
|
||||
guide.replace("sudo bash ./scripts/test-vm-disk.sh 100", "curl https://example.invalid"),
|
||||
]
|
||||
for index, mutated in enumerate(mutations):
|
||||
with self.subTest(mutation=index), self.assertRaises(AssertionError):
|
||||
self.assert_guide_commands(mutated)
|
||||
|
||||
def test_guides_are_mirrored_and_preserve_the_safety_and_proof_boundaries(self):
|
||||
for name in ("VM_DISK_MONITORING", "TROUBLESHOOTING"):
|
||||
source = (ROOT / f"docs/{name}.md").read_bytes()
|
||||
@@ -281,8 +349,7 @@ class VMDiskHelpTest(unittest.TestCase):
|
||||
"does not prove thaw", "a backup can", "older copies", "non-zero exit",
|
||||
"host-root diagnostic does not test", "not a claim", "bounded timeouts"):
|
||||
self.assertIn(phrase, guide)
|
||||
for block in re.findall(r"```bash\n(.*?)```", guide, re.DOTALL):
|
||||
self.assertNotRegex(block, r"\b(curl|wget|qm agent|pveum|systemctl)\b")
|
||||
self.assert_guide_commands(guide)
|
||||
self.assertNotIn("GUEST_AGENT_FSINFO_TIMEOUT=", guide)
|
||||
|
||||
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { expect, test as base, type Page } from '@playwright/test';
|
||||
import { createAuthenticatedStorageState } from './helpers';
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { expect, test as base, type Page } from "@playwright/test";
|
||||
import { createAuthenticatedStorageState } from "./helpers";
|
||||
|
||||
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
||||
|
||||
@@ -10,6 +10,11 @@ type WorkerFixtures = {
|
||||
authStorageStatePath: string;
|
||||
};
|
||||
|
||||
type DiagnosticsTraffic = {
|
||||
reads: number;
|
||||
writes: string[];
|
||||
};
|
||||
|
||||
type OverflowAudit = {
|
||||
viewportWidth: number;
|
||||
pageWidth: number;
|
||||
@@ -18,22 +23,22 @@ type OverflowAudit = {
|
||||
};
|
||||
|
||||
const DIAGNOSTICS_PAYLOAD = {
|
||||
version: '6.0.0',
|
||||
runtime: 'go',
|
||||
version: "6.0.0",
|
||||
runtime: "go",
|
||||
uptime: 3600,
|
||||
nodes: [],
|
||||
pbs: [],
|
||||
system: {
|
||||
os: 'linux',
|
||||
arch: 'amd64',
|
||||
goVersion: 'go1.25',
|
||||
os: "linux",
|
||||
arch: "amd64",
|
||||
goVersion: "go1.25",
|
||||
numCPU: 8,
|
||||
numGoroutine: 32,
|
||||
memoryMB: 128,
|
||||
},
|
||||
metricsStore: {
|
||||
enabled: true,
|
||||
status: 'healthy',
|
||||
status: "healthy",
|
||||
dbSize: 4 * 1024 * 1024,
|
||||
rawCount: 12,
|
||||
minuteCount: 24,
|
||||
@@ -45,7 +50,7 @@ const DIAGNOSTICS_PAYLOAD = {
|
||||
},
|
||||
commercialFunnel: {
|
||||
enabled: true,
|
||||
status: 'active',
|
||||
status: "active",
|
||||
windowDays: 30,
|
||||
summary: {
|
||||
pricing_viewed: 3,
|
||||
@@ -58,13 +63,13 @@ const DIAGNOSTICS_PAYLOAD = {
|
||||
license_activated: 1,
|
||||
license_activation_failed: 0,
|
||||
period: {
|
||||
from: '2026-03-19T00:00:00Z',
|
||||
to: '2026-04-18T00:00:00Z',
|
||||
from: "2026-03-19T00:00:00Z",
|
||||
to: "2026-04-18T00:00:00Z",
|
||||
},
|
||||
},
|
||||
daily: [
|
||||
{
|
||||
day: '2026-04-17',
|
||||
day: "2026-04-17",
|
||||
pricing_viewed: 1,
|
||||
paywall_viewed: 0,
|
||||
trial_started: 0,
|
||||
@@ -78,7 +83,7 @@ const DIAGNOSTICS_PAYLOAD = {
|
||||
],
|
||||
surfaces: [
|
||||
{
|
||||
key: 'settings_self_hosted_billing_compare_prompt',
|
||||
key: "settings_self_hosted_billing_compare_prompt",
|
||||
pricing_viewed: 0,
|
||||
paywall_viewed: 0,
|
||||
trial_started: 0,
|
||||
@@ -92,7 +97,7 @@ const DIAGNOSTICS_PAYLOAD = {
|
||||
],
|
||||
capabilities: [
|
||||
{
|
||||
key: 'self_hosted_plan',
|
||||
key: "self_hosted_plan",
|
||||
pricing_viewed: 3,
|
||||
paywall_viewed: 0,
|
||||
trial_started: 0,
|
||||
@@ -104,11 +109,13 @@ const DIAGNOSTICS_PAYLOAD = {
|
||||
license_activation_failed: 0,
|
||||
},
|
||||
],
|
||||
notes: ['Local pricing and activation events show at least one completed conversion.'],
|
||||
notes: [
|
||||
"Local pricing and activation events show at least one completed conversion.",
|
||||
],
|
||||
},
|
||||
infrastructureOnboarding: {
|
||||
enabled: true,
|
||||
status: 'warning',
|
||||
status: "warning",
|
||||
windowDays: 30,
|
||||
summary: {
|
||||
opened: 4,
|
||||
@@ -120,13 +127,13 @@ const DIAGNOSTICS_PAYLOAD = {
|
||||
catalog_selected: 2,
|
||||
credentials_opened: 1,
|
||||
period: {
|
||||
from: '2026-03-19T00:00:00Z',
|
||||
to: '2026-04-18T00:00:00Z',
|
||||
from: "2026-03-19T00:00:00Z",
|
||||
to: "2026-04-18T00:00:00Z",
|
||||
},
|
||||
},
|
||||
daily: [
|
||||
{
|
||||
day: '2026-04-17',
|
||||
day: "2026-04-17",
|
||||
opened: 2,
|
||||
api_path_selected: 1,
|
||||
agent_path_selected: 1,
|
||||
@@ -137,7 +144,7 @@ const DIAGNOSTICS_PAYLOAD = {
|
||||
credentials_opened: 0,
|
||||
},
|
||||
{
|
||||
day: '2026-04-18',
|
||||
day: "2026-04-18",
|
||||
opened: 2,
|
||||
api_path_selected: 1,
|
||||
agent_path_selected: 0,
|
||||
@@ -149,18 +156,18 @@ const DIAGNOSTICS_PAYLOAD = {
|
||||
},
|
||||
],
|
||||
paths: [
|
||||
{ key: 'api', count: 2 },
|
||||
{ key: 'agent', count: 1 },
|
||||
{ key: "api", count: 2 },
|
||||
{ key: "agent", count: 1 },
|
||||
],
|
||||
platforms: [
|
||||
{ key: 'truenas', catalog_selected: 2, credentials_opened: 1 },
|
||||
platforms: [{ key: "truenas", catalog_selected: 2, credentials_opened: 1 }],
|
||||
notes: [
|
||||
"More probed addresses miss than detect a supported API-backed platform.",
|
||||
],
|
||||
notes: ['More probed addresses miss than detect a supported API-backed platform.'],
|
||||
},
|
||||
discovery: {
|
||||
enabled: true,
|
||||
configuredSubnet: 'auto',
|
||||
scanInterval: '5m',
|
||||
configuredSubnet: "auto",
|
||||
scanInterval: "5m",
|
||||
lastResultServers: 3,
|
||||
},
|
||||
apiTokens: {
|
||||
@@ -202,79 +209,168 @@ const EXPORT_DIAGNOSTICS_PAYLOAD = {
|
||||
...DIAGNOSTICS_PAYLOAD,
|
||||
nodes: [
|
||||
{
|
||||
id: 'node-raw-id',
|
||||
name: 'pve-01',
|
||||
host: '10.0.0.5',
|
||||
type: 'pve',
|
||||
authMethod: 'token',
|
||||
id: "node-raw-id",
|
||||
name: "pve-01",
|
||||
host: "10.0.0.5",
|
||||
type: "pve",
|
||||
authMethod: "token",
|
||||
connected: false,
|
||||
error: 'dial tcp 10.0.0.5:8006: connect: connection refused',
|
||||
error: "dial tcp 10.0.0.5:8006: connect: connection refused",
|
||||
},
|
||||
],
|
||||
discovery: {
|
||||
enabled: true,
|
||||
configuredSubnet: '10.0.0.0/24',
|
||||
activeSubnet: '10.0.1.0/24',
|
||||
environmentOverride: 'PULSE_DISCOVERY_SUBNET=10.0.2.0/24',
|
||||
subnetAllowlist: ['10.0.0.0/24'],
|
||||
subnetBlocklist: ['10.0.3.0/24'],
|
||||
configuredSubnet: "10.0.0.0/24",
|
||||
activeSubnet: "10.0.1.0/24",
|
||||
environmentOverride: "PULSE_DISCOVERY_SUBNET=10.0.2.0/24",
|
||||
subnetAllowlist: ["10.0.0.0/24"],
|
||||
subnetBlocklist: ["10.0.3.0/24"],
|
||||
history: [
|
||||
{
|
||||
startedAt: '2026-04-17T10:00:00Z',
|
||||
completedAt: '2026-04-17T10:00:05Z',
|
||||
duration: '5s',
|
||||
startedAt: "2026-04-17T10:00:00Z",
|
||||
completedAt: "2026-04-17T10:00:05Z",
|
||||
duration: "5s",
|
||||
durationMs: 5000,
|
||||
subnet: '10.0.0.0/24',
|
||||
subnet: "10.0.0.0/24",
|
||||
serverCount: 3,
|
||||
errorCount: 1,
|
||||
blocklistLength: 1,
|
||||
status: 'completed',
|
||||
status: "completed",
|
||||
},
|
||||
],
|
||||
},
|
||||
errors: ['probe failed for 10.0.0.10 after timeout'],
|
||||
apiTokens: {
|
||||
...DIAGNOSTICS_PAYLOAD.apiTokens,
|
||||
tokens: [
|
||||
{
|
||||
id: "synthetic-private-token-a",
|
||||
name: "synthetic-private-purpose-a",
|
||||
hint: "synthetic-private-hint-a",
|
||||
},
|
||||
{
|
||||
id: "synthetic-private-token-b",
|
||||
name: "synthetic-private-purpose-b",
|
||||
hint: "synthetic-private-hint-b",
|
||||
},
|
||||
],
|
||||
// Deliberately reverse token order: aliases must follow identity, not position.
|
||||
usage: [
|
||||
{
|
||||
tokenId: "synthetic-private-token-b",
|
||||
agentCount: 1,
|
||||
agents: ["synthetic-private-agent"],
|
||||
},
|
||||
{
|
||||
tokenId: "synthetic-private-token-a",
|
||||
agentCount: 1,
|
||||
agents: ["synthetic-private-agent"],
|
||||
},
|
||||
],
|
||||
},
|
||||
dockerAgents: {
|
||||
...DIAGNOSTICS_PAYLOAD.dockerAgents,
|
||||
attention: [
|
||||
{
|
||||
agentId: "synthetic-private-agent-id",
|
||||
name: "synthetic-private-agent",
|
||||
tokenHint: "synthetic-private-hint",
|
||||
issues: [],
|
||||
},
|
||||
],
|
||||
},
|
||||
nodeSnapshots: [
|
||||
{
|
||||
instance: "synthetic-private-instance",
|
||||
node: "synthetic-private-node",
|
||||
memorySource: "agent",
|
||||
memory: { used: 256 },
|
||||
raw: { total: 512 },
|
||||
},
|
||||
],
|
||||
guestSnapshots: [
|
||||
{
|
||||
instance: "synthetic-private-instance",
|
||||
node: "synthetic-private-node",
|
||||
name: "synthetic-private-guest",
|
||||
vmid: 9501,
|
||||
guestType: "vm",
|
||||
memorySource: "agent",
|
||||
memory: { used: 128 },
|
||||
raw: { hostAgentUsed: 128 },
|
||||
notes: [],
|
||||
},
|
||||
],
|
||||
memorySources: [
|
||||
{ instance: "synthetic-private-instance", scope: "node", count: 1 },
|
||||
],
|
||||
memorySourceBreakdown: [
|
||||
{
|
||||
instance: "synthetic-private-instance",
|
||||
scope: "node",
|
||||
source: "agent",
|
||||
trust: "host",
|
||||
count: 1,
|
||||
fallback: false,
|
||||
fallbackReasons: [],
|
||||
},
|
||||
],
|
||||
errors: ["probe failed for 10.0.0.10 after timeout"],
|
||||
};
|
||||
|
||||
const test = base.extend<{}, WorkerFixtures>({
|
||||
storageState: async ({ authStorageStatePath }, use) => {
|
||||
await use(authStorageStatePath);
|
||||
},
|
||||
authStorageStatePath: [async ({ browser }, use, workerInfo) => {
|
||||
const storageStatePath = path.resolve(
|
||||
__dirname,
|
||||
'..',
|
||||
'..',
|
||||
'tmp',
|
||||
'playwright-auth',
|
||||
`diagnostics-onboarding-${workerInfo.project.name}.json`,
|
||||
);
|
||||
fs.mkdirSync(path.dirname(storageStatePath), { recursive: true });
|
||||
await createAuthenticatedStorageState(browser, storageStatePath);
|
||||
try {
|
||||
await use(storageStatePath);
|
||||
} finally {
|
||||
fs.rmSync(storageStatePath, { force: true });
|
||||
}
|
||||
}, { scope: 'worker' }],
|
||||
authStorageStatePath: [
|
||||
async ({ browser }, use, workerInfo) => {
|
||||
const storageStatePath = path.resolve(
|
||||
__dirname,
|
||||
"..",
|
||||
"..",
|
||||
"tmp",
|
||||
"playwright-auth",
|
||||
`diagnostics-onboarding-${workerInfo.project.name}.json`,
|
||||
);
|
||||
fs.mkdirSync(path.dirname(storageStatePath), { recursive: true });
|
||||
await createAuthenticatedStorageState(browser, storageStatePath);
|
||||
try {
|
||||
await use(storageStatePath);
|
||||
} finally {
|
||||
fs.rmSync(storageStatePath, { force: true });
|
||||
}
|
||||
},
|
||||
{ scope: "worker" },
|
||||
],
|
||||
});
|
||||
|
||||
async function prepareDiagnosticsRoute(
|
||||
page: Page,
|
||||
payload = DIAGNOSTICS_PAYLOAD,
|
||||
): Promise<void> {
|
||||
await page.route('**/api/diagnostics', async (route) => {
|
||||
payload: typeof DIAGNOSTICS_PAYLOAD | typeof EXPORT_DIAGNOSTICS_PAYLOAD =
|
||||
DIAGNOSTICS_PAYLOAD,
|
||||
): Promise<DiagnosticsTraffic> {
|
||||
const traffic: DiagnosticsTraffic = { reads: 0, writes: [] };
|
||||
page.on("request", (request) => {
|
||||
if (!["GET", "HEAD"].includes(request.method()))
|
||||
traffic.writes.push(new URL(request.url()).pathname);
|
||||
});
|
||||
await page.route("**/api/diagnostics", async (route) => {
|
||||
const requestUrl = new URL(route.request().url());
|
||||
if (route.request().method() !== 'GET' || requestUrl.pathname !== '/api/diagnostics') {
|
||||
if (
|
||||
route.request().method() !== "GET" ||
|
||||
requestUrl.pathname !== "/api/diagnostics"
|
||||
) {
|
||||
await route.continue();
|
||||
return;
|
||||
}
|
||||
|
||||
traffic.reads += 1;
|
||||
await route.fulfill({
|
||||
status: 200,
|
||||
contentType: 'application/json',
|
||||
contentType: "application/json",
|
||||
body: JSON.stringify(payload),
|
||||
});
|
||||
});
|
||||
return traffic;
|
||||
}
|
||||
|
||||
async function scrollToBottom(page: Page): Promise<void> {
|
||||
@@ -299,7 +395,10 @@ async function scrollToBottom(page: Page): Promise<void> {
|
||||
|
||||
async function auditHorizontalOverflow(page: Page): Promise<OverflowAudit> {
|
||||
return page.evaluate(() => {
|
||||
const viewportWidth = Math.max(document.documentElement.clientWidth, window.innerWidth || 0);
|
||||
const viewportWidth = Math.max(
|
||||
document.documentElement.clientWidth,
|
||||
window.innerWidth || 0,
|
||||
);
|
||||
const pageWidth = Math.max(
|
||||
document.body.scrollWidth,
|
||||
document.documentElement.scrollWidth,
|
||||
@@ -307,21 +406,27 @@ async function auditHorizontalOverflow(page: Page): Promise<OverflowAudit> {
|
||||
document.documentElement.offsetWidth,
|
||||
);
|
||||
|
||||
const offenders = Array.from(document.querySelectorAll('body *'))
|
||||
const offenders = Array.from(document.querySelectorAll("body *"))
|
||||
.map((el) => {
|
||||
const rect = el.getBoundingClientRect();
|
||||
if (rect.width <= 0 || rect.height <= 0) return null;
|
||||
const style = window.getComputedStyle(el);
|
||||
if (style.position === 'fixed' || style.position === 'absolute') return null;
|
||||
if (style.position === "fixed" || style.position === "absolute")
|
||||
return null;
|
||||
const overflow = rect.right - viewportWidth;
|
||||
if (overflow <= 1) return null;
|
||||
return {
|
||||
tag: el.tagName.toLowerCase(),
|
||||
className: (el.getAttribute('class') || '').trim().slice(0, 120),
|
||||
className: (el.getAttribute("class") || "").trim().slice(0, 120),
|
||||
overflow: Number(overflow.toFixed(1)),
|
||||
};
|
||||
})
|
||||
.filter((entry): entry is { tag: string; className: string; overflow: number } => Boolean(entry))
|
||||
.filter(
|
||||
(
|
||||
entry,
|
||||
): entry is { tag: string; className: string; overflow: number } =>
|
||||
Boolean(entry),
|
||||
)
|
||||
.slice(0, 8);
|
||||
|
||||
return {
|
||||
@@ -333,130 +438,333 @@ async function auditHorizontalOverflow(page: Page): Promise<OverflowAudit> {
|
||||
});
|
||||
}
|
||||
|
||||
test.describe('Diagnostics onboarding analytics', () => {
|
||||
async function openDiagnostics(page: Page): Promise<void> {
|
||||
await page.goto("/settings/support/diagnostics", {
|
||||
waitUntil: "domcontentloaded",
|
||||
});
|
||||
await page.waitForURL(/\/settings\/support\/diagnostics$/, {
|
||||
timeout: 15_000,
|
||||
});
|
||||
await expect(
|
||||
page.getByRole("heading", { name: "Diagnostics & Health" }),
|
||||
).toBeVisible();
|
||||
}
|
||||
|
||||
async function describedBy(
|
||||
button: ReturnType<Page["getByRole"]>,
|
||||
): Promise<string> {
|
||||
return button.evaluate((element) =>
|
||||
(element.getAttribute("aria-describedby") || "")
|
||||
.split(/\s+/)
|
||||
.map((id) => document.getElementById(id)?.textContent || "")
|
||||
.join(" "),
|
||||
);
|
||||
}
|
||||
|
||||
async function downloadJSON(
|
||||
page: Page,
|
||||
label: string,
|
||||
kind: string,
|
||||
keyboard = false,
|
||||
) {
|
||||
const button = page.getByRole("button", { name: label, exact: true });
|
||||
await expect(button).toBeEnabled();
|
||||
// Register the observer before activation. Keyboard and pointer use the same
|
||||
// public accessible control, on desktop and both mobile projects.
|
||||
const pending = page.waitForEvent("download");
|
||||
if (keyboard) {
|
||||
await button.focus();
|
||||
await expect(button).toBeFocused();
|
||||
await button.press("Enter");
|
||||
} else {
|
||||
await button.click();
|
||||
}
|
||||
const download = await pending;
|
||||
expect(download.suggestedFilename()).toMatch(
|
||||
new RegExp(`^pulse-diagnostics-${kind}-\\d{4}-\\d{2}-\\d{2}\\.json$`),
|
||||
);
|
||||
const file = await download.path();
|
||||
expect(file).not.toBeNull();
|
||||
return JSON.parse(fs.readFileSync(file!, "utf8"));
|
||||
}
|
||||
|
||||
async function assertControlsInViewport(
|
||||
page: Page,
|
||||
mobile: boolean,
|
||||
): Promise<void> {
|
||||
const audit = await auditHorizontalOverflow(page);
|
||||
expect(
|
||||
audit.pageWidth,
|
||||
`Diagnostics overflow: ${JSON.stringify(audit)}`,
|
||||
).toBeLessThanOrEqual(audit.viewportWidth + 1);
|
||||
for (const button of await page
|
||||
.getByRole("button", {
|
||||
name: /Run Diagnostics|Full \(private\)|GitHub \(review first\)/,
|
||||
})
|
||||
.all()) {
|
||||
const bounds = await button.boundingBox();
|
||||
expect(bounds).not.toBeNull();
|
||||
expect(bounds!.x).toBeGreaterThanOrEqual(0);
|
||||
expect(bounds!.x + bounds!.width).toBeLessThanOrEqual(
|
||||
audit.viewportWidth + 1,
|
||||
);
|
||||
if (mobile) expect(bounds!.height).toBeGreaterThanOrEqual(44);
|
||||
}
|
||||
}
|
||||
|
||||
test.describe("Diagnostics safe collection and local exports", () => {
|
||||
test.setTimeout(180_000);
|
||||
|
||||
// The Commercial Funnel and Infrastructure Onboarding analytics cards were
|
||||
// deliberately removed from the user-facing diagnostics page (c6bcad255,
|
||||
// "Remove internal analytics from diagnostics"); the UI strips
|
||||
// commercialFunnel and infrastructureOnboarding from the payload before both
|
||||
// rendering and export. These specs pin that boundary.
|
||||
test('keeps internal onboarding analytics out of the rendered diagnostics page', async ({
|
||||
page,
|
||||
}, testInfo) => {
|
||||
test.skip(testInfo.project.name.startsWith('mobile-'), 'Desktop-only diagnostics shell coverage');
|
||||
|
||||
await prepareDiagnosticsRoute(page);
|
||||
|
||||
await page.goto('/settings/support/diagnostics', { waitUntil: 'domcontentloaded' });
|
||||
await page.waitForURL(/\/settings\/support\/diagnostics$/, { timeout: 15_000 });
|
||||
|
||||
await expect(page.getByRole('heading', { name: 'Diagnostics & Health' })).toBeVisible();
|
||||
await page.getByRole('button', { name: 'Run Diagnostics', exact: true }).first().click();
|
||||
|
||||
// The run completed and the payload rendered (metrics-store card proves it).
|
||||
await expect(page.getByText('Metrics Store', { exact: false }).first()).toBeVisible({
|
||||
timeout: 30_000,
|
||||
});
|
||||
|
||||
await expect(page.getByText('Commercial Funnel', { exact: true })).toHaveCount(0);
|
||||
await expect(page.getByText('Infrastructure Onboarding', { exact: true })).toHaveCount(0);
|
||||
await expect(page.getByText('Credentials Opened', { exact: true })).toHaveCount(0);
|
||||
test.afterEach(async ({ page }, testInfo) => {
|
||||
if (testInfo.status === "passed") {
|
||||
await page.screenshot({
|
||||
path: testInfo.outputPath("diagnostics-completed-state.png"),
|
||||
fullPage: true,
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
test('desktop diagnostics exports full and sanitized onboarding analytics JSON', async ({
|
||||
test("keeps internal analytics absent and connects safety guidance to keyboard-accessible controls", async ({
|
||||
page,
|
||||
}, testInfo) => {
|
||||
test.skip(testInfo.project.name.startsWith('mobile-'), 'Desktop-only diagnostics export coverage');
|
||||
|
||||
await prepareDiagnosticsRoute(page, EXPORT_DIAGNOSTICS_PAYLOAD);
|
||||
|
||||
await page.goto('/settings/support/diagnostics', { waitUntil: 'domcontentloaded' });
|
||||
await page.waitForURL(/\/settings\/support\/diagnostics$/, { timeout: 15_000 });
|
||||
|
||||
await page.getByRole('button', { name: 'Run Diagnostics', exact: true }).first().click();
|
||||
// Wait for a rendered card that survives the analytics strip.
|
||||
await expect(page.getByText('Metrics Store', { exact: false }).first()).toBeVisible({
|
||||
timeout: 30_000,
|
||||
const traffic = await prepareDiagnosticsRoute(page);
|
||||
await openDiagnostics(page);
|
||||
expect(traffic.reads).toBe(0);
|
||||
await expect(
|
||||
page.getByRole("button", { name: "Full (private)", exact: true }),
|
||||
).toHaveCount(0);
|
||||
// Both the toolbar and empty-state run actions need the same warning.
|
||||
const actions = page.getByRole("button", {
|
||||
name: "Run Diagnostics",
|
||||
exact: true,
|
||||
});
|
||||
|
||||
const [fullDownload] = await Promise.all([
|
||||
page.waitForEvent('download'),
|
||||
page.getByRole('button', { name: 'Full', exact: true }).click(),
|
||||
]);
|
||||
|
||||
expect(fullDownload.suggestedFilename()).toMatch(
|
||||
new RegExp(`^pulse-diagnostics-full-${new Date().toISOString().slice(0, 10)}\\.json$`),
|
||||
await expect(actions).toHaveCount(2);
|
||||
for (const button of await actions.all()) {
|
||||
expect(await describedBy(button)).toMatch(
|
||||
/Do not run it during a backup, freeze\/thaw/,
|
||||
);
|
||||
}
|
||||
await expect(
|
||||
page.getByRole("link", { name: "Safe diagnostics and sharing" }),
|
||||
).toHaveAttribute("href", /#collect-diagnostics-safely$/);
|
||||
await actions.first().focus();
|
||||
await expect(actions.first()).toBeFocused();
|
||||
await actions.first().press("Enter");
|
||||
await expect(
|
||||
page.getByText("Metrics Store", { exact: false }).first(),
|
||||
).toBeVisible();
|
||||
expect(traffic.reads).toBe(1);
|
||||
for (const label of [
|
||||
"Commercial Funnel",
|
||||
"Infrastructure Onboarding",
|
||||
"Credentials Opened",
|
||||
]) {
|
||||
await expect(page.getByText(label, { exact: true })).toHaveCount(0);
|
||||
}
|
||||
for (const label of ["Full (private)", "GitHub (review first)"]) {
|
||||
expect(
|
||||
await describedBy(
|
||||
page.getByRole("button", { name: label, exact: true }),
|
||||
),
|
||||
).toMatch(
|
||||
/Nothing is uploaded.*Keep the full file private.*Review even a sanitised file/,
|
||||
);
|
||||
}
|
||||
await assertControlsInViewport(
|
||||
page,
|
||||
testInfo.project.name.startsWith("mobile-"),
|
||||
);
|
||||
|
||||
const fullPath = await fullDownload.path();
|
||||
expect(fullPath).not.toBeNull();
|
||||
const fullPayload = JSON.parse(fs.readFileSync(fullPath!, 'utf8')) as typeof EXPORT_DIAGNOSTICS_PAYLOAD;
|
||||
|
||||
expect(fullPayload.nodes[0]?.host).toBe('10.0.0.5');
|
||||
expect(fullPayload.discovery?.configuredSubnet).toBe('10.0.0.0/24');
|
||||
// Internal analytics are stripped from the full export too, not just the
|
||||
// rendered card.
|
||||
expect(fullPayload.infrastructureOnboarding).toBeUndefined();
|
||||
expect(
|
||||
(fullPayload as Record<string, unknown>).commercialFunnel,
|
||||
).toBeUndefined();
|
||||
|
||||
const [sanitizedDownload] = await Promise.all([
|
||||
page.waitForEvent('download'),
|
||||
page.getByRole('button', { name: 'GitHub', exact: true }).click(),
|
||||
]);
|
||||
|
||||
expect(sanitizedDownload.suggestedFilename()).toMatch(
|
||||
new RegExp(`^pulse-diagnostics-sanitized-${new Date().toISOString().slice(0, 10)}\\.json$`),
|
||||
);
|
||||
|
||||
const sanitizedPath = await sanitizedDownload.path();
|
||||
expect(sanitizedPath).not.toBeNull();
|
||||
const sanitizedPayload = JSON.parse(fs.readFileSync(sanitizedPath!, 'utf8')) as typeof EXPORT_DIAGNOSTICS_PAYLOAD;
|
||||
|
||||
expect(sanitizedPayload.nodes[0]).toEqual(
|
||||
expect.objectContaining({
|
||||
id: 'node-1',
|
||||
name: 'node-1',
|
||||
host: 'node-1',
|
||||
}),
|
||||
);
|
||||
expect(sanitizedPayload.discovery).toEqual(
|
||||
expect.objectContaining({
|
||||
configuredSubnet: '[REDACTED_SUBNET]',
|
||||
activeSubnet: '[REDACTED_SUBNET]',
|
||||
environmentOverride: '[REDACTED]',
|
||||
}),
|
||||
);
|
||||
expect(sanitizedPayload.errors).toEqual(['probe failed for [REDACTED_IP] after timeout']);
|
||||
expect(sanitizedPayload.infrastructureOnboarding).toBeUndefined();
|
||||
expect(
|
||||
(sanitizedPayload as Record<string, unknown>).commercialFunnel,
|
||||
).toBeUndefined();
|
||||
});
|
||||
|
||||
test('mobile diagnostics keeps the populated onboarding analytics page inside the viewport', async ({
|
||||
test("downloads the displayed private and review-first JSON without another live request or upload", async ({
|
||||
page,
|
||||
}, testInfo) => {
|
||||
test.skip(!testInfo.project.name.startsWith('mobile-'), 'Mobile-only diagnostics overflow coverage');
|
||||
const traffic = await prepareDiagnosticsRoute(
|
||||
page,
|
||||
EXPORT_DIAGNOSTICS_PAYLOAD,
|
||||
);
|
||||
await openDiagnostics(page);
|
||||
await page
|
||||
.getByRole("button", { name: "Run Diagnostics", exact: true })
|
||||
.first()
|
||||
.click();
|
||||
await expect(
|
||||
page.getByText("Metrics Store", { exact: false }).first(),
|
||||
).toBeVisible();
|
||||
const writesBeforeExport = [...traffic.writes];
|
||||
|
||||
await prepareDiagnosticsRoute(page);
|
||||
const full = await downloadJSON(page, "Full (private)", "full", true);
|
||||
const {
|
||||
infrastructureOnboarding: _onboarding,
|
||||
commercialFunnel: _funnel,
|
||||
...expectedFull
|
||||
} = EXPORT_DIAGNOSTICS_PAYLOAD;
|
||||
expect(full).toEqual(expectedFull);
|
||||
expect(traffic.reads).toBe(1);
|
||||
expect(traffic.writes).toEqual(writesBeforeExport);
|
||||
await expect(
|
||||
page.getByRole("heading", {
|
||||
name: "Full diagnostics downloaded — keep this file private",
|
||||
exact: true,
|
||||
}),
|
||||
).toBeVisible();
|
||||
|
||||
await page.goto('/settings/support/diagnostics', { waitUntil: 'domcontentloaded' });
|
||||
await page.waitForURL(/\/settings\/support\/diagnostics$/, { timeout: 15_000 });
|
||||
|
||||
await page.getByRole('button', { name: 'Run', exact: true }).click();
|
||||
await expect(page.getByText('Metrics Store', { exact: false }).first()).toBeVisible({
|
||||
timeout: 30_000,
|
||||
const sanitised = await downloadJSON(
|
||||
page,
|
||||
"GitHub (review first)",
|
||||
"sanitized",
|
||||
);
|
||||
expect(sanitised.nodes[0]).toMatchObject({
|
||||
id: "node-1",
|
||||
name: "node-1",
|
||||
host: "node-1",
|
||||
});
|
||||
expect(sanitised.discovery).toMatchObject({
|
||||
configuredSubnet: "[REDACTED_SUBNET]",
|
||||
activeSubnet: "[REDACTED_SUBNET]",
|
||||
environmentOverride: "[REDACTED]",
|
||||
subnetAllowlist: ["[REDACTED_SUBNET]"],
|
||||
subnetBlocklist: ["[REDACTED_SUBNET]"],
|
||||
history: [
|
||||
expect.objectContaining({
|
||||
subnet: "[REDACTED_SUBNET]",
|
||||
serverCount: 3,
|
||||
}),
|
||||
],
|
||||
});
|
||||
expect(sanitised.errors).toEqual([
|
||||
"probe failed for [REDACTED_IP] after timeout",
|
||||
]);
|
||||
expect(sanitised.apiTokens.tokens[0]).toMatchObject({
|
||||
id: "token-1",
|
||||
name: "token-1",
|
||||
hint: "[REDACTED]",
|
||||
});
|
||||
expect(sanitised.apiTokens.usage[0].tokenId).toBe(
|
||||
sanitised.apiTokens.tokens[1].id,
|
||||
);
|
||||
expect(sanitised.apiTokens.usage[1].tokenId).toBe(
|
||||
sanitised.apiTokens.tokens[0].id,
|
||||
);
|
||||
expect(sanitised.apiTokens.usage[0].agents).toEqual(
|
||||
sanitised.apiTokens.usage[1].agents,
|
||||
);
|
||||
expect(sanitised.apiTokens.usage[0].agents[0]).toBe(
|
||||
sanitised.dockerAgents.attention[0].name,
|
||||
);
|
||||
expect(sanitised.guestSnapshots[0]).toMatchObject({
|
||||
instance: sanitised.nodeSnapshots[0].instance,
|
||||
node: sanitised.nodeSnapshots[0].node,
|
||||
});
|
||||
expect(sanitised.guestSnapshots[0].vmid).toBeUndefined();
|
||||
expect(sanitised.guestSnapshots[0].memory).toEqual(
|
||||
EXPORT_DIAGNOSTICS_PAYLOAD.guestSnapshots[0].memory,
|
||||
);
|
||||
expect(sanitised.memorySources[0].instance).toBe(
|
||||
sanitised.nodeSnapshots[0].instance,
|
||||
);
|
||||
expect(sanitised.memorySourceBreakdown[0].instance).toBe(
|
||||
sanitised.nodeSnapshots[0].instance,
|
||||
);
|
||||
for (const payload of [full, sanitised]) {
|
||||
expect(payload.infrastructureOnboarding).toBeUndefined();
|
||||
expect(payload.commercialFunnel).toBeUndefined();
|
||||
expect(payload.metricsStore).toEqual(
|
||||
EXPORT_DIAGNOSTICS_PAYLOAD.metricsStore,
|
||||
);
|
||||
}
|
||||
expect(JSON.stringify(sanitised)).not.toContain("synthetic-private-");
|
||||
expect(JSON.stringify(sanitised)).not.toMatch(/10\.0\.[0-3]\./);
|
||||
expect(traffic.reads).toBe(1);
|
||||
expect(traffic.writes).toEqual(writesBeforeExport);
|
||||
await expect(
|
||||
page.getByRole("heading", {
|
||||
name: "Sanitised diagnostics downloaded — review before sharing",
|
||||
exact: true,
|
||||
}),
|
||||
).toBeVisible();
|
||||
// A full export after sanitisation must still be the original displayed
|
||||
// data: redaction may not mutate it.
|
||||
expect(await downloadJSON(page, "Full (private)", "full")).toEqual(
|
||||
expectedFull,
|
||||
);
|
||||
expect(traffic.reads).toBe(1);
|
||||
expect(traffic.writes).toEqual(writesBeforeExport);
|
||||
await assertControlsInViewport(
|
||||
page,
|
||||
testInfo.project.name.startsWith("mobile-"),
|
||||
);
|
||||
await page.screenshot({
|
||||
path: testInfo.outputPath("diagnostics-safe-export.png"),
|
||||
fullPage: true,
|
||||
});
|
||||
});
|
||||
|
||||
test("disables exports during a refresh and retains the previous result after a failed live check", async ({
|
||||
page,
|
||||
}, testInfo) => {
|
||||
let reads = 0;
|
||||
let release: (() => void) | undefined;
|
||||
const held = new Promise<void>((resolve) => {
|
||||
release = resolve;
|
||||
});
|
||||
await page.route("**/api/diagnostics", async (route) => {
|
||||
reads += 1;
|
||||
if (reads === 1) return route.fulfill({ json: DIAGNOSTICS_PAYLOAD });
|
||||
await held;
|
||||
return route.fulfill({
|
||||
status: 400,
|
||||
json: { error: "Synthetic live check failed" },
|
||||
});
|
||||
});
|
||||
await openDiagnostics(page);
|
||||
const run = page
|
||||
.getByRole("button", { name: "Run Diagnostics", exact: true })
|
||||
.first();
|
||||
await run.click();
|
||||
const full = page.getByRole("button", {
|
||||
name: "Full (private)",
|
||||
exact: true,
|
||||
});
|
||||
const github = page.getByRole("button", {
|
||||
name: "GitHub (review first)",
|
||||
exact: true,
|
||||
});
|
||||
await expect(full).toBeEnabled();
|
||||
await run.focus();
|
||||
await run.press("Space");
|
||||
await expect(
|
||||
page.getByRole("button", { name: "Running...", exact: true }),
|
||||
).toBeDisabled();
|
||||
await expect(full).toBeDisabled();
|
||||
await expect(github).toBeDisabled();
|
||||
expect(reads).toBe(2);
|
||||
await page.screenshot({
|
||||
path: testInfo.outputPath("diagnostics-refresh-disabled.png"),
|
||||
fullPage: true,
|
||||
});
|
||||
release!();
|
||||
await expect(
|
||||
page.getByRole("heading", {
|
||||
name: "Synthetic live check failed",
|
||||
exact: true,
|
||||
}),
|
||||
).toBeVisible();
|
||||
await expect(full).toBeEnabled();
|
||||
await expect(github).toBeEnabled();
|
||||
const {
|
||||
infrastructureOnboarding: _onboarding,
|
||||
commercialFunnel: _funnel,
|
||||
...expectedFull
|
||||
} = DIAGNOSTICS_PAYLOAD;
|
||||
expect(await downloadJSON(page, "Full (private)", "full")).toEqual(
|
||||
expectedFull,
|
||||
);
|
||||
expect(reads).toBe(2);
|
||||
await scrollToBottom(page);
|
||||
const audit = await auditHorizontalOverflow(page);
|
||||
|
||||
expect(
|
||||
audit.pageWidth,
|
||||
`Mobile diagnostics overflow (viewport=${audit.viewportWidth}, page=${audit.pageWidth}, offenders=${JSON.stringify(audit.offenders)})`,
|
||||
).toBeLessThanOrEqual(audit.viewportWidth + 1);
|
||||
await assertControlsInViewport(
|
||||
page,
|
||||
testInfo.project.name.startsWith("mobile-"),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
"target": "ES2020",
|
||||
"module": "ES2020",
|
||||
"lib": ["ES2020", "DOM"],
|
||||
"moduleResolution": "node",
|
||||
"moduleResolution": "bundler",
|
||||
"esModuleInterop": true,
|
||||
"strict": true,
|
||||
"skipLibCheck": true,
|
||||
|
||||
Reference in New Issue
Block a user