gsadmin/pulse
1
0
Fork 0
mirror of https://github.com/rcourtman/Pulse.git synced 2026-10-24 04:33:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

Keep deferred guest readings, alerts and History truthful (#2472)

Browse Source 
## What changes and why
Keep guest monitoring truthful while reads are deferred: retained
memory/filesystem values must not become new alert decisions, new
History observations, current filesystem health, or continuous chart
lines across missing readings. Failed or empty metadata refreshes retain
the last useful observation time rather than manufacturing freshness.
Independent current readings and measured-zero recovery keep their
existing behaviour.

Manual guest Discovery pauses during the existing backup/read
precaution, preserves saved evidence, and keeps delayed outcomes
attached to their original guest and run. The published drawer-header
link to a saved web interface remains available separately from live
Discovery. The existing TrueNAS guide and shipped copy keep credentials
and responses private and bound individual requests.

The guest work follows the prolonged freeze reported in #2439 and
reproduced downstream observation/lifecycle gaps. It does not identify
the freeze's cause or prove native recovery. The guidance repair follows
inspection of unsafe existing recipes. No new collector, schema,
retention policy, threshold or product surface is introduced by this
batch.

## Included source and reconciliation
Current published base `efe6fc21ae10a6bf173cc17e6203aaf6c97dc130` →
proposed head `1914f5fb7b278b5fc4cc10a292e909a10714a2f8` (39 introduced
commits, including additive integrations).

- TrueNAS guidance: `cadfd1f4e1d9f909a45977f94aadd4f1324e82ef` →
`d589407a743a12fe492e93785d9e24c12bbb4fb5`.
- Discovery pause/outcome ownership:
`2cb3f6a9043e0bab9c54c5b5ef57b3e1cb994781` →
`bbaee0a3f43dc7906011be5c78d684f9a1327bae`, including additive recovery
and lifecycle controls.
- Alert admission: `b0f8b96c8a159ac242b85f2d09f8dfffd5c29c7a` →
`0fd7768e9d1379f8aa0126b33d4047b4c52c0d87`.
- History writer admission: `a15920eaebea00ab8c353ddd3861b9cd7ff7d379`.
- Filesystem presentation: `2525d76c04a9834c2130e97c1a529e164a9c7796` →
`5b75f21159fda3c01fbe6688aa9cfde68779fba8`.
- Metadata freshness: `f1e221a0fb9b0804f200c13490f8526424cb7e1c`.
- Missing-observation chart presentation:
`0ad41e4c9a309ce459dd717d16d343b46efcac8a` →
`5e7e49650fcdf5131b7c5006fe487336ab660e15`.
- Platform-aware Discovery fixture:
`a2608c9fadf76c242527a636ee08968fa16ddde8`; TrueNAS recipe
reconciliation `1f0d12af85ec7c2ad8df391b8e6958e80fa98179`; additive
upstream/header composition and content-bound browser receipts through
`b01dad904c3436e70bc01588ea1546fffaa05dc6`; batch-start upstream merge
`f1c7e413bac2e3306d2169bcd069c60325e56d5f`, then exact bounded
refreshed-upstream merges `07112b0e3ab2be174a5ac7a11f87ac0837edceb0` and
`1914f5fb7b278b5fc4cc10a292e909a10714a2f8`.

The previous proposal head `6a20d106242598bd6b466e9ae888ea931d284b30` is
completed-failed, not still validating. Native run `37239183481`
incorrectly required a Linux/PVE capability on Windows/macOS; Build and
Test `37239183545` extracted zero wrapped TrueNAS recipes. Reviewed
additive fixtures correct both without weakening runtime safety. The old
head cannot land independently, so this update includes those
corrections and the coherent observation batch. Every accepted commit
identity and all ancestry are retained. Published cluster labels, saved
guest links, unique demo workload names the newer storage priority/alert
explanations, and the published Alerts overview are preserved. Bounded
refresh receipt
`2916978f4e4b49d084752e973e19a5c0f7dd19f0bf4ac60cc31875876318528d`
completed at the exact observed upstream without changing source; the
subsequent full-SHA merge composes it additively. A second successful
bounded refresh
`59f1939979347921d99e262398d5347077bfce6faf6f42535050840a3a19820b`
incorporates the subsequently published Alerts overview in the same way.
Neither is a replay of a failed or uncertain attempt.

## Validation and remaining gates
Retained evidence includes connected alert/API/producer/recovery
controls (`source-p58_phvc`), Discovery/component/build/browser checks
(`source-gy5x8buj`), installer/docs controls (`source-7s0iorys`,
`source-uo6rxs61`), metadata connected/race/vet/build proof
(`source-ji23nb_c`), and ten connected suites plus focused safety
races/vet and native-target compilation (`source-seboen4h`). Browser
content receipts cover the final seven changed frontend runtime files,
including the composed saved-link/safety header and chart gaps.

The History presentation proof `source-ualisb1k` completed 22,571
frontend tests across 1,234 files, types/lint/format/build and
bundle/integrity checks; its overall exit 1 from trailing nonexistent
optional test paths remains recorded, not converted into a whole-command
pass. Four browser groups/32 checks and eight inspected captures
separately passed. Earlier reproductions, fixture failures and
reporting-limit failures remain evidence.

Publication-gate checks:
- `source-qt6m6mf_`: seven connected TrueNAS recipe controls, full mock
suite, scoped real agent/guest safety, alert observation and
metadata/History tests pass. Backend/docs source is byte-identical in
the final composition; this evidence was not repeated after
frontend-only upstream advances.
- `source-xy1lnmgt`: types pass and 281 of 283 scoped frontend controls
pass; two first-case History tests hit the unchanged 5,000ms timeout.
Exit 1 is retained and the following build did not run. No functional
assertion failure or cause was established.
- One predeclared single-worker/file-serial comparison,
`source-17bch24o`, passes all 23 controls in those two files with
unchanged source, assertions and timeouts. This establishes isolated
functional behaviour, not a diagnosis or clearance of parallel CI.
- `source-dvlfnm5t`: refreshed storage composition passes types, 1,177
tests/56 files and the real production/embed build. The event-parser
line-bound flag is recorded; the complete 26,474 raw bytes independently
match the receipt.
- Final exact-head `source-sjjtzjqm`: types, scoped published Alerts
overview/translation/presentation controls and the real production/embed
build pass. Complete raw output is independently hash-matched and the
guest stopped.
- All executed receipts, including the failed scoped run, retain
complete output and completed guest stop. Provenance, clean source,
diff/ancestry checks, all 24 introduced non-merge commits' parent-bound
canonical guards using only their original recorded trailers, and the
final seven-file browser-content receipt guard pass.

Required exact-head administrative source review and GitHub checks must
complete before landing; this update is not a blind failed-job retry or
a CI waiver. Native overlap, independent thaw, fresh writes on every
covered filesystem, workload liveness, truthful alert delivery and safe
restoration of only prior-active services/timers remain separate. UI
clearance, saved values, an OK backup or these source tests are not
recovery, a containing release or installed/reporter acceptance.

Related to #2439


[How Pulse handles
triage](https://github.com/rcourtman/Pulse/blob/main/docs/AI_TRANSPARENCY.md)
This commit is contained in:
pulse-triage[bot]
2026-10-05 01:09:46 +00:00
committed by GitHub
parent a36f643510 1914f5fb7b
commit 1d4c3aeb6e
53 changed files with 5025 additions and 188 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/TRUENAS.md
+94 -28
View File
@@ -101,48 +101,114 @@ All endpoints require admin authentication.
### Testing and adding a connection (API)
The **Settings → Infrastructure → Platform connections** flow is the simplest option. For an API client,
prepare the private Pulse header file as described in the
[API authentication guide](API.md). The Pulse API token authenticates to
Pulse; the separate TrueNAS API key belongs in the connection's JSON file.
Do not put either credential in shell arguments, URLs or a public thread.
Prefer **Settings → Infrastructure → Platform connections** for setup. These
API examples are for an authorised administrator using curl 7.76 or later,
not a way to diagnose stale polling. Testing makes a live system-information
request to TrueNAS; saving creates a connection and starts ordinary polling.
Do not use either during a backup, freeze/thaw or an unresponsive-host incident.
Create or protect the connection file before opening it in a trusted editor:
First prepare a private Pulse header file using the
[API token file procedure](API.md#api-token-recommended), with the admin access
needed for connection management. The Pulse token authenticates to Pulse;
the separate TrueNAS API key belongs in the connection's JSON file. Enter
credentials only in your local editor, never in arguments, URLs or a thread.
On a trusted machine, as the account running curl, prepare the connection
file below. Stop if either file-preparation step fails. This preserves an
existing file, restricts the directory and refuses symlinked credential paths
or a non-regular connection file before opening the editor:
```bash
umask 077
mkdir -p "$HOME/.config/pulse"
touch "$HOME/.config/pulse/truenas-connection.json"
chmod 600 "$HOME/.config/pulse/truenas-connection.json"
vi "$HOME/.config/pulse/truenas-connection.json"
(
set -eu
umask 077
config_dir="$HOME/.config/pulse"
connection_file="$config_dir/truenas-connection.json"
if [ -L "$HOME/.config" ] || [ -L "$config_dir" ] || [ -L "$connection_file" ]; then
printf 'Refusing a symlinked credential path.\n' >&2
exit 1
fi
if [ -e "$connection_file" ] && [ ! -f "$connection_file" ]; then
printf 'Connection file must be a regular file.\n' >&2
exit 1
fi
mkdir -p "$config_dir"
chmod 700 "$config_dir"
touch "$connection_file"
chmod 600 "$connection_file"
vi "$connection_file"
)
```
In that file, enter a JSON object with `name`, `host` (the TrueNAS HTTPS URL),
`username` (the key owner) and `apiKey`. Keep both credential files private;
do not paste their contents into a terminal command or upload them.
In the editor, save a JSON object with `name`, `host` (the TrueNAS HTTPS URL),
`username` (the key owner) and `apiKey`. Keep both credential files private and
outside repositories and diagnostics. Do not upload them.
Test the connection before saving it:
The following examples run **on the Pulse host**, using its direct loopback
address. They ignore curl configuration, do not follow redirects or retry,
and retain each response in a new private local file. Responses and errors can
contain identifying details or echoed credentials; review them locally, not in
a public terminal recording or thread. Keep `--disable` first so local curl
configuration cannot enable tracing or change the request.
Test once without saving:
```bash
curl --fail-with-body --silent --show-error \
--header "@$HOME/.config/pulse/api-header" \
--header 'Content-Type: application/json' \
--data-binary "@$HOME/.config/pulse/truenas-connection.json" \
http://127.0.0.1:7655/api/truenas/connections/test
(
set -eu
umask 077
result_file=$(mktemp "$HOME/.config/pulse/truenas-test.XXXXXX")
printf 'Private response file: %s\n' "$result_file"
status=$(curl --disable --fail-with-body --silent --show-error \
--connect-timeout 5 --max-time 20 --proto '=http' --noproxy 127.0.0.1 \
--header "@$HOME/.config/pulse/api-header" \
--header 'Content-Type: application/json' \
--data-binary "@$HOME/.config/pulse/truenas-connection.json" \
--output "$result_file" --write-out '%{http_code}' \
http://127.0.0.1:7655/api/truenas/connections/test)
printf 'HTTP %s\n' "$status"
[ "$status" = 200 ]
)
```
After checking the test response, save the connection with:
Require exit 0, HTTP **200** and `success: true` in that private test response.
A redirect, HTTP error or `000` is not a successful test. Test success is not
inventory, metric or guest-thaw acceptance.
Only after reviewing the test response, create the connection once:
```bash
curl --fail-with-body --silent --show-error \
--header "@$HOME/.config/pulse/api-header" \
--header 'Content-Type: application/json' \
--data-binary "@$HOME/.config/pulse/truenas-connection.json" \
http://127.0.0.1:7655/api/truenas/connections
(
set -eu
umask 077
result_file=$(mktemp "$HOME/.config/pulse/truenas-save.XXXXXX")
printf 'Private response file: %s\n' "$result_file"
status=$(curl --disable --fail-with-body --silent --show-error \
--connect-timeout 5 --max-time 20 --proto '=http' --noproxy 127.0.0.1 \
--header "@$HOME/.config/pulse/api-header" \
--header 'Content-Type: application/json' \
--data-binary "@$HOME/.config/pulse/truenas-connection.json" \
--output "$result_file" --write-out '%{http_code}' \
http://127.0.0.1:7655/api/truenas/connections)
printf 'HTTP %s\n' "$status"
[ "$status" = 201 ]
)
```
These Pulse URLs are loopback-only examples. For a remote Pulse server, use
its trusted HTTPS origin without disabling certificate verification.
A successful save returns HTTP **201** and the new connection's `id`, not
proof of a complete poll. If saving times out or loses its response, it may
already have created the connection. Check **Platform connections** or
`/api/truenas/connections` in your existing signed-in Pulse browser before
trying again; do not blindly repeat POST and create duplicates. Continue with
the [ordinary polling checks](#stale-truenas-data) for collection acceptance.
For remote Pulse access, replace the loopback origin in the relevant command
with your trusted **HTTPS** origin and change `--proto '=http'` to
`--proto '=https'`. Keep certificate verification enabled. For a private CA,
add `--cacert /path/to/trusted-ca.pem` using a CA obtained independently from
your administrator, not from the failing connection. Do not add `--insecure`,
`--location`, verbose/trace flags or automatic retries. Share only the relevant
HTTP status and a manually redacted error, never the full response or headers.
## Troubleshooting
docs/release-control/v6/internal/subsystems/alerts.md
+29
View File
@@ -15,6 +15,35 @@
## Purpose
### Retained guest observations are not alert evidence
Guest memory marked `last-known`, `unavailable` or an unknown observation state
cannot start, advance, refresh or resolve a memory incident. A VM filesystem
read with a nonempty `diskStatusReason`, including retained `prev-*` values,
or unavailable aggregate usage is likewise not current disk evidence. It cannot
evaluate either aggregate or per-filesystem thresholds, replace an aggregate
incident with filesystem incidents, or clear filesystem alerts as if a failed
or expired inventory were authoritative removal. Metric filters do not match
these unavailable memory/disk values, including a fabricated healthy zero.
Existing incidents retain their last trusted value, occurrence, acknowledgement,
observation time and lifecycle/delivery evidence until a current observation
resumes evaluation. Successful current inventory still resolves removed
filesystems. Explicit disablement, suppression and stopped-guest behavior remain
effective independently of telemetry, including per-filesystem disable overrides
on retained identities. Legacy unannotated memory and independent
current PVE/Pulse-agent memory/disks keep their existing policy, even while QGA
is deferred; current CPU and I/O continue to evaluate. Producers own freshness;
no new age threshold, QGA request, payload, alert family or UI surface is added.
`guest_observation_test.go` pins typed/pointer VM/LXC observation admission,
delayed breaches, canonical incident/acknowledgement/ledger continuity, filters,
fresh recovery and removal, independent metrics and deliberate disablement.
`TestGuestBackupDeferralAlertObservationLifecycle` connects actual HTTP client,
poll builder, retained snapshot, unified view, History and the production alert
call. These are synthetic source proofs, not native backup/thaw, containing
release, installed delivery or external recipient acceptance.
### Pulse system alerts retain their service scope in the existing UI
The `pulse-system-` identity prefix or explicit `systemAlert: true` metadata
docs/release-control/v6/internal/subsystems/frontend-primitives.md
+49
View File
@@ -249,6 +249,24 @@ Fresh linked Pulse-agent disk evidence with no reason remains unmarked, and
non-VM rows do not inherit QGA guidance. The cue adds no action or tab stop;
touch on it continues through the existing row disclosure to the full reason.
The opened guest Overview applies the same admission to every filesystem, not
only the aggregate row or its status sentence. A valid `prev-` value is labelled
**Last known** beside its mount identity and used/capacity bytes; retained values
do not receive current utilization bars or threshold colours. Other VM read
reasons and explicit disk telemetry unavailability show **Usage unavailable**,
retaining identity and known capacity without promoting old used bytes. Missing
or invalid retained usage is unavailable, not an empty healthy filesystem. The
read explanation precedes the filesystem breakdown and remains visible before
any sample. Clearing a lock alone cannot clear a still-supplied reason. A fresh
same-guest read restores the normal accessible utilization bar; independent LXC
and linked-agent readings with no VM reason retain their existing semantics.
Verification: `GuestDrawerOverview.filesystem-provenance.test.tsx`, the existing
filesystem regressions and builder controls in `DiskList.test.tsx`, plus
`browser-tests/guest-filesystem-provenance.cjs` using the full production guest
drawer on desktop/light/dark and touch WebKit. These are synthetic presentation
and input proofs, not native backup/thaw, covered-filesystem writes or resumption.
Verification: mounted `GuestRow.test.tsx` and `MetricMiniSparkline.test.tsx`
regressions and `browser-tests/guest-disk-provenance.cjs` exercise retained and
pre-sample states, display/lens changes, current-value semantics and same-row
@@ -8308,3 +8326,34 @@ chain at desktop and phone widths with keyboard/touch refresh, class-dark CSS,
403/503 controls and pending recovery. Unit controls separately cover current
request ownership and disposal. No mutation, route, navigation entry, queue
policy, permission or recipient-delivery contract is added by these states.
### Drawer History does not interpolate across missing observations
The shared `GuestDrawerHistory` renderer connects a series only across
consecutive stored times in that configured panel. If another series has a
stored observation at an intervening time and this series does not, its line
stops. Each isolated observation, including a measured zero, remains visible
as a point; separate continuous segments retain all their original values,
timestamps and common-window coordinates. Independently sampled series must
not imply simultaneous observations or a continuous trend through absent data.
A visible, SVG-described explanation names the affected configured series and
states the limit: missing readings do not identify an outage's cause or duration.
There is no elapsed-time/cadence heuristic, synthetic zero, invented observation,
new API field or collector change. Times outside the configured panel cannot
split its series. Entirely absent series retain the existing missing/fallback
policy. Live or retained snapshot carriers cannot repair a stored gap; only a
matching stored response can. Source/range/access isolation, transient retained
reads and exact-time keyboard/pointer/touch inspection remain in force.
`GuestDrawerHistory.gaps.test.tsx` and `guestHistorySegments.test.ts` cover
split lines, singleton recovery, zero rates, interleaved sampling, multiple
segments, configured-group isolation, invalid points, elapsed-time controls,
live updates, stored refresh and denied/transient reads. `GuestDrawer.test.tsx`
checks the full drawer's live-state recovery and unchanged request budget.
Updated pointer/sparse
regressions preserve their original missing-value and fallback obligations.
`browser-tests/history-observation-gaps.cjs` exercises the production full guest
drawer with synthetic APIs in Chromium/WebKit desktop and 390/320-pixel light
and dark views. This is presentation acceptance, not native backup/QGA safety,
provider outage attribution, installed recovery or release availability.
docs/release-control/v6/internal/subsystems/performance-and-scalability.md
+23
View File
@@ -1690,6 +1690,15 @@ without contention or cross-delivery.
lifecycle. Request handlers may reuse the cached manager but must not reopen
or remigrate SQLite on each role or assignment request, and shutdown must
close the owned manager set.
8. Guest Overview filesystem provenance is local presentation over the already
supplied snapshot: no additional query, timer, poll, History fetch or guest
command may be added to establish it. Its existing linear filesystem mapping
may label retained values and omit current utilization bars; it must preserve
ordinary current-value builder behaviour and must not mutate the snapshot.
`DiskList.test.tsx` verifies the pure builder's retained/unavailable/current
controls; mounted drawer regressions and the full-drawer browser proof cover
the shipped path. No performance or native workload improvement follows from
the presentation proof alone.
## Current State
@@ -3671,3 +3680,17 @@ backend, metrics-store, performance threshold or latency claim changes here.
including selection retention while a same-source window advances. Browser
receipts exercise the actual PBS table/drawer renderer with synthetic APIs,
not installed collection or a performance benchmark.
### Stored drawer History segmentation stays bounded
Missing-observation rendering uses the existing normalized, configured-group
timestamps and the unchanged maximum of 240 requested points per metric. The
segment builder indexes those timestamps once per series and walks its stored
points without nested nearest-point searches. It adds no request, polling timer,
collector or synthetic reading. Separate singletons/segments retain the source
data and shared time window; live updates cannot fill absent stored evidence.
The mounted `GuestDrawerHistory.gaps.test.tsx` controls pin one initial read and
only explicit matching refreshes; the full `GuestDrawer.test.tsx` pins the
unchanged cap and no live-recovery read. `guestHistorySegments.test.ts` pins
complete point preservation at the existing request bound. Neither claims a
timing benchmark, fleet CPU attribution or native incident relief.
frontend-modern/browser-tests/discovery-outcome-ownership.cjs
+301
View File
@@ -0,0 +1,301 @@
const assert = require('node:assert/strict');
const fs = require('node:fs');
const path = require('node:path');
const crypto = require('node:crypto');
const { chromium, webkit } = require('playwright');
const root = '/workspace/frontend-modern';
const output = '/workspace/tmp/discovery-ownership-proof/browser';
const origin = 'http://127.0.0.1:5333';
const runtime = [
'frontend-modern/src/components/Discovery/DiscoveryTab.tsx',
'frontend-modern/src/components/Discovery/useDiscoveryTabState.ts',
'frontend-modern/src/components/Workloads/GuestDrawer.tsx',
];
const hash = (file) => crypto.createHash('sha256').update(fs.readFileSync(file)).digest('hex');
const saved = (id, service = `Saved Home Assistant ${id}`) => ({
id: `vm:fixture-node-agent:${id}`,
resource_type: 'vm',
resource_id: id,
target_id: 'fixture-node-agent',
hostname: `guest-${id}`,
service_type: 'home-assistant',
service_name: service,
service_version: 'test',
category: 'home_automation',
cli_access: '',
facts: [],
config_paths: [],
data_paths: [],
log_paths: [],
ports: [],
user_notes: '',
user_secrets: {},
confidence: 0.9,
ai_reasoning: 'Synthetic saved evidence',
discovered_at: '2026-10-03T12:00:00Z',
updated_at: '2026-10-03T12:00:00Z',
discovery_engine_version: 1,
});
(async () => {
fs.mkdirSync(output, { recursive: true });
const report = {
result: 'incomplete',
playwright: require('playwright/package.json').version,
runtime_sha256: Object.fromEntries(runtime.map((p) => [p, hash('/workspace/' + p)])),
groups: [],
screenshots: [],
cleanup: {},
limits:
'Production DiscoveryTab/state/CSS, synthetic API and production event-bus progress. No native QGA, automatic collector, original freeze cause, thaw, covered filesystem writes, installed, physical-phone or release acceptance.',
};
assert.equal(
report.playwright,
JSON.parse(fs.readFileSync('/workspace/tests/integration/package-lock.json')).packages[
'node_modules/@playwright/test'
].version,
);
process.chdir(root);
const { createServer } = await import(path.join(root, 'node_modules/vite/dist/node/index.js'));
const server = await createServer({
root,
configFile: path.join(root, 'vite.config.ts'),
cacheDir: output + '/cache',
server: { host: '127.0.0.1', port: 5333, strictPort: true, watch: null },
});
let browser,
phase = 'server';
try {
await server.listen();
for (const [name, engine, width, dark] of [
['desktop-light', chromium, 1365, false],
['desktop-dark', chromium, 1365, true],
['phone-dark', webkit, 390, true],
['narrow-light', webkit, 320, false],
]) {
phase = name;
browser = await engine.launch(
engine === chromium
? { headless: true, channel: 'chromium', args: ['--no-sandbox'] }
: { headless: true },
);
const page = await browser.newPage({
viewport: { width, height: width < 500 ? 844 : 900 },
isMobile: width < 500,
hasTouch: width < 500,
locale: 'en-GB',
timezoneId: 'UTC',
});
page.setDefaultTimeout(10000);
page.setDefaultNavigationTimeout(60000);
const group = {
name,
width,
dark,
browser: browser.version(),
checks: [],
requests: [],
errors: [],
offOrigin: [],
};
report.groups.push(group);
page.on('pageerror', (error) => group.errors.push(error.message));
await page.routeWebSocket('**/ws*', () => {});
await page.addInitScript((dark) => {
const apply = () => document.documentElement.classList.toggle('dark', dark);
if (document.documentElement) apply();
else document.addEventListener('DOMContentLoaded', apply, { once: true });
}, dark);
let fail = false;
const releases = new Map();
const arrivals = new Map();
const stored = new Map();
await page.route('**/*', async (route) => {
const request = route.request(),
url = new URL(request.url());
if (url.origin !== origin) {
group.offOrigin.push(url.origin);
return route.abort();
}
if (!url.pathname.startsWith('/api/')) return route.continue();
group.requests.push({
path: url.pathname,
method: request.method(),
body: request.postData(),
});
const target = url.pathname.match(/^\/api\/discovery\/vm\/fixture-node-agent\/(101|102)$/);
if (target) {
const id = target[1];
if (request.method() === 'GET')
return route.fulfill({ json: stored.get(id) || saved(id) });
assert.equal(request.method(), 'POST');
if (fail)
return route.fulfill({
status: 409,
json: { error: 'Guest execution paused: backup lock observed' },
});
await new Promise((resolve) => {
releases.set(id, resolve);
arrivals.get(id)?.();
});
const result = saved(id, id === '101' ? 'Old guest result' : 'Current guest result');
stored.set(id, result);
return route.fulfill({ json: result });
}
assert.equal(request.method(), 'GET');
const responses = {
'/api/discovery/info/vm': {
ai_provider: { provider: 'test', model: 'test', is_local: true, label: 'Local' },
commands: [],
command_categories: [],
},
'/api/ai/agents': {
count: 1,
agents: [{ agent_id: 'fixture-node-agent', hostname: 'pve1' }],
},
'/api/ai/settings': { discovery_enabled: true, enabled: false },
'/api/license/runtime-capabilities': {
capabilities: [],
limits: [],
hosted_mode: false,
runtime: { build: 'community' },
blocked_capabilities: [],
},
};
return route.fulfill({ json: responses[url.pathname] || {} });
});
const capture = async (suffix) => {
await page.screenshot({ path: `${output}/${name}-${suffix}.png`, fullPage: true });
report.screenshots.push(`${name}-${suffix}.png`);
assert(await page.evaluate(() => document.documentElement.scrollWidth <= innerWidth + 1));
};
const emit = (progress) =>
page.evaluate((progress) => window.discoveryOwnership.progress(progress), progress);
const run = page.getByRole('button', { name: 'Run Discovery', exact: true });
const pressRun = async () => {
if (width >= 500) {
await run.focus();
await page.keyboard.press('Enter');
} else await run.tap();
};
await page.goto(origin + '/browser-tests/discovery-outcome-ownership.html');
await page.getByText('Saved Home Assistant 101', { exact: true }).waitFor();
await run.waitFor();
await page.waitForFunction(
() =>
!Array.from(document.querySelectorAll('button')).find(
(b) => b.textContent.trim() === 'Run Discovery',
).disabled,
);
const before = group.requests.length;
await emit({
resource_id: 'vm:fixture-node-agent:101',
status: 'running',
current_step: 'Collecting guest evidence',
percent_complete: 20,
});
await page.getByText('Collecting guest evidence', { exact: true }).waitFor();
await emit({
resource_id: 'vm:fixture-node-agent:101',
status: 'completed',
error: 'Guest execution paused: backup lock observed',
current_step: 'Guest execution paused',
percent_complete: 20,
});
const alert = page.getByRole('alert');
await alert.waitFor();
await page.waitForTimeout(650);
assert((await alert.innerText()).includes('Guest execution paused: backup lock observed'));
assert.equal(group.requests.length, before);
assert(await page.getByText('Saved Home Assistant 101', { exact: true }).isVisible());
assert.equal(await page.getByText('Discovery complete!', { exact: true }).count(), 0);
await capture('retained-background-pause');
group.checks.push(
'Background completed-with-error remains an accessible failure with saved evidence and no retry or refresh',
);
const dismiss = page.getByRole('button', { name: 'Dismiss Discovery error' });
await dismiss.focus();
await page.keyboard.press('Enter');
assert.equal(await alert.count(), 0);
group.checks.push('The error dismiss action has a keyboard-accessible name');
const firstPost = new Promise((resolve) => arrivals.set('101', resolve));
await pressRun();
await firstPost;
await page.waitForFunction(() => document.body.textContent.includes('Scanning...'));
await page.evaluate(() => window.discoveryOwnership.target('102'));
await page.getByText('Saved Home Assistant 102', { exact: true }).waitFor();
const secondPost = new Promise((resolve) => arrivals.set('102', resolve));
await pressRun();
await secondPost;
await page.waitForFunction(() => document.body.textContent.includes('Scanning...'));
assert(releases.has('101'));
assert(releases.has('102'));
releases.get('101')();
await page.waitForTimeout(150);
assert.equal(await page.getByText('Old guest result', { exact: true }).count(), 0);
assert.equal(await page.getByText('Discovery complete!', { exact: true }).count(), 0);
assert(await page.getByRole('button', { name: 'Scanning...', exact: true }).isDisabled());
await emit({
resource_id: 'vm:fixture-node-agent:102',
status: 'completed',
percent_complete: 100,
});
assert(await page.getByRole('button', { name: 'Scanning...', exact: true }).isDisabled());
assert.equal(await page.getByText('Discovery complete!', { exact: true }).count(), 0);
releases.get('102')();
await page.getByText('Current guest result', { exact: true }).waitFor();
await page.getByText('Discovery complete!', { exact: true }).waitFor();
group.checks.push(
'Late old-target HTTP and scanner completion cannot replace the current guest or finish its active HTTP request',
);
const posts = group.requests.filter((r) => r.method === 'POST');
assert.equal(posts.length, 2);
assert.deepEqual(
posts.map((p) => JSON.parse(p.body)),
[
{ force: true, hostname: 'guest-101' },
{ force: true, hostname: 'guest-102' },
],
);
await capture('current-guest-result');
await emit({ resource_id: 'vm:fixture-node-agent:102', status: 'completed' });
await page.waitForTimeout(650);
assert.equal(await page.getByText('Discovery complete!', { exact: true }).count(), 0);
assert(await page.getByText('Current guest result', { exact: true }).isVisible());
assert.equal(group.requests.filter((r) => r.method === 'POST').length, 2);
group.checks.push(
'Later completion cannot strand the old HTTP success banner or issue an automatic scan',
);
fail = true;
await pressRun();
await alert.waitFor();
assert((await alert.innerText()).includes('Guest execution paused: backup lock observed'));
assert(await page.getByText('Current guest result', { exact: true }).isVisible());
assert.equal(await page.getByText('Discovery complete!', { exact: true }).count(), 0);
await page.waitForTimeout(650);
assert.equal(group.requests.filter((r) => r.method === 'POST').length, 3);
group.checks.push(
'A failed explicit run retains prior saved evidence, visible failure and no automatic retry',
);
assert.deepEqual(group.errors, []);
assert.deepEqual(group.offOrigin, []);
await browser.close();
browser = null;
}
report.result = 'passed';
report.verified_at = new Date().toISOString();
} catch (error) {
report.result = 'failed';
report.phase = phase;
report.error = String(error.stack || error);
throw error;
} finally {
if (browser) await browser.close();
report.cleanup.browser_closed = true;
await server.close();
report.cleanup.server_closed = true;
fs.writeFileSync(output + '/result.json', JSON.stringify(report, null, 2) + '\n');
}
})().catch((error) => {
console.error(error.stack || String(error));
process.exitCode = 1;
});
frontend-modern/browser-tests/discovery-outcome-ownership.html
+12
View File
@@ -0,0 +1,12 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Discovery outcome ownership evidence</title>
</head>
<body>
<div id="root"></div>
<script type="module" src="./discovery-outcome-ownership.tsx"></script>
</body>
</html>
frontend-modern/browser-tests/discovery-outcome-ownership.tsx
+54
View File
@@ -0,0 +1,54 @@
import { createSignal } from 'solid-js';
import { render } from 'solid-js/web';
import { Router, Route } from '@solidjs/router';
import { DiscoveryTab } from '../src/components/Discovery/DiscoveryTab';
import { eventBus } from '../src/stores/events';
import { syncAIRuntimeSettings } from '../src/stores/aiRuntimeState';
import type { DiscoveryProgress } from '../src/types/discovery';
import '../src/index.css';
declare global {
interface Window {
discoveryOwnership: {
target: (id: string) => void;
progress: (progress: DiscoveryProgress) => void;
enabled: (enabled: boolean) => void;
};
}
}
function Fixture() {
const [id, setId] = createSignal('101');
const enabled = (discovery_enabled: boolean) =>
syncAIRuntimeSettings({ discovery_enabled } as Parameters<typeof syncAIRuntimeSettings>[0]);
enabled(true);
window.discoveryOwnership = {
target: setId,
progress: (progress) => eventBus.emit('ai_discovery_progress', progress),
enabled,
};
return (
<main class="mx-auto min-h-screen max-w-4xl space-y-3 bg-surface p-3 text-base-content">
<h1 class="text-base font-semibold">Discovery outcome ownership evidence</h1>
<p class="text-xs text-muted">Synthetic guest responses. No guest command is sent.</p>
<h2 class="text-sm font-medium">Guest {id()}</h2>
<DiscoveryTab
resourceType="vm"
agentId="fixture-node-agent"
resourceId={id()}
canonicalResourceId={`fixture-pve1-${id()}`}
hostname={`guest-${id()}`}
showManualRunAction
/>
</main>
);
}
render(
() => (
<Router>
<Route path="/*" component={Fixture} />
</Router>
),
document.getElementById('root')!,
);
frontend-modern/browser-tests/guest-discovery-completion.cjs
+19
View File
@@ -0,0 +1,19 @@
const { spawnSync } = require('node:child_process');
for (const script of ['guest-discovery-safety.cjs', 'discovery-outcome-ownership.cjs']) {
const result = spawnSync(
process.execPath,
['/workspace/frontend-modern/browser-tests/' + script],
{ stdio: 'inherit' },
);
if (result.status !== 0) {
console.error(
script,
'did not complete successfully',
result.status,
result.signal,
result.error?.message,
);
process.exit(result.status || 1);
}
}
console.log('Combined guest Discovery pause and outcome ownership browser proof passed.');
frontend-modern/browser-tests/guest-discovery-safety.cjs
+329
View File
@@ -0,0 +1,329 @@
const assert = require('node:assert/strict');
const fs = require('node:fs');
const path = require('node:path');
const crypto = require('node:crypto');
const { chromium, webkit } = require('playwright');
const root = '/workspace/frontend-modern';
const output = '/workspace/tmp/discovery-safety-proof/browser';
const runtime = [
'frontend-modern/src/components/Workloads/GuestDrawer.tsx',
'frontend-modern/src/components/Discovery/DiscoveryTab.tsx',
'frontend-modern/src/components/Discovery/useDiscoveryTabState.ts',
];
const hash = (file) => crypto.createHash('sha256').update(fs.readFileSync(file)).digest('hex');
const initial = {
state: 'deferred',
reason: 'prev-vm-locked',
lock: 'backup',
backup: true,
version: '6.4.5',
assignment: true,
diskUsed: 5 * 1024 ** 3,
};
const healthy = { ...initial, state: 'available', reason: '', lock: '', backup: false };
const saved = {
id: 'vm:fixture-node-agent:101',
resource_type: 'vm',
resource_id: '101',
target_id: 'fixture-node-agent',
hostname: 'backup-guest',
service_type: 'home-assistant',
service_name: 'Saved Home Assistant',
service_version: 'test',
category: 'home_automation',
cli_access: '',
facts: [],
config_paths: [],
data_paths: [],
log_paths: [],
ports: [],
user_notes: '',
user_secrets: {},
confidence: 0.9,
ai_reasoning: 'Synthetic saved evidence',
discovered_at: '2026-10-03T12:00:00Z',
updated_at: '2026-10-03T12:00:00Z',
discovery_engine_version: 1,
};
(async () => {
fs.mkdirSync(output, { recursive: true });
const report = {
runtime_sha256: Object.fromEntries(runtime.map((p) => [p, hash(path.join('/workspace', p))])),
playwright: require('playwright/package.json').version,
groups: [],
screenshots: [],
cleanup: {},
};
assert.equal(
report.playwright,
JSON.parse(fs.readFileSync('/workspace/tests/integration/package-lock.json')).packages[
'node_modules/@playwright/test'
].version,
);
process.chdir(root);
const { createServer } = await import(path.join(root, 'node_modules/vite/dist/node/index.js'));
const server = await createServer({
root,
configFile: path.join(root, 'vite.config.ts'),
cacheDir: path.join(output, 'vite-cache'),
server: { host: '127.0.0.1', port: 5297, strictPort: true, watch: null },
});
const origin = 'http://127.0.0.1:5297';
let browser,
phase = 'server';
try {
await server.listen();
for (const [name, engine, width, dark] of [
['chromium-desktop-light', chromium, 1365, false],
['chromium-desktop-dark', chromium, 1365, true],
['webkit-phone-dark', webkit, 390, true],
['webkit-narrow-light', webkit, 320, false],
]) {
phase = name;
browser = await engine.launch(
engine === chromium
? { headless: true, channel: 'chromium', args: ['--no-sandbox'] }
: { headless: true },
);
const page = await browser.newPage({
viewport: { width, height: width < 500 ? 844 : 900 },
isMobile: width < 500,
hasTouch: width < 500,
locale: 'en-GB',
timezoneId: 'UTC',
});
page.setDefaultTimeout(10000);
page.setDefaultNavigationTimeout(60000);
const errors = [],
requests = [],
offOrigin = [],
checks = [];
report.groups.push({
name,
width,
dark,
browser: browser.version(),
checks,
requests,
errors,
offOrigin,
});
await page.routeWebSocket('**/ws*', () => {});
let hold = false,
release;
page.on('pageerror', (e) => errors.push(e.message));
await page.addInitScript((dark) => {
const apply = () => document.documentElement.classList.toggle('dark', dark);
if (document.documentElement) apply();
else document.addEventListener('DOMContentLoaded', apply, { once: true });
}, dark);
await page.route('**/*', async (route) => {
const req = route.request(),
url = new URL(req.url());
if (url.origin !== origin) {
offOrigin.push(url.origin);
return route.abort();
}
if (!url.pathname.startsWith('/api/')) return route.continue();
requests.push({
path: url.pathname,
query: url.search,
method: req.method(),
body: req.postData(),
});
if (url.pathname === '/api/discovery/vm/fixture-node-agent/101') {
if (req.method() === 'POST') {
if (hold)
await new Promise((resolve) => {
release = resolve;
});
return route.fulfill({ json: saved });
}
return route.fulfill({ json: saved });
}
assert.equal(req.method(), 'GET');
if (url.pathname.startsWith('/api/discovery/info/'))
return route.fulfill({
json: {
ai_provider: { provider: 'test', model: 'test', is_local: true, label: 'Local' },
commands: [],
command_categories: [],
},
});
if (url.pathname === '/api/ai/agents')
return route.fulfill({
json: {
count: 1,
agents: [
{
agent_id: 'fixture-node-agent',
hostname: 'pve-a',
version: '6.4.5',
platform: 'linux',
},
],
},
});
if (url.pathname.includes('metrics-store/history'))
return route.fulfill({
json: {
resourceType: 'vm',
resourceId: url.searchParams.get('resourceId'),
range: '24h',
start: Date.UTC(2026, 9, 3, 12),
end: Date.UTC(2026, 9, 4, 12),
metrics: {},
source: 'store',
},
});
if (url.pathname === '/api/resources')
return route.fulfill({ json: { data: [], total: 0, page: 1, limit: 500 } });
if (url.pathname.includes('license'))
return route.fulfill({ json: { capabilities: [], limits: [], max_history_days: 90 } });
if (url.pathname.includes('metadata')) return route.fulfill({ json: {} });
return route.fulfill({
json: { success: true, data: [], alerts: [], anomalies: [], config: null },
});
});
await page.goto(`${origin}/browser-tests/guest-discovery-safety.html`, {
waitUntil: 'domcontentloaded',
});
await page.waitForFunction(() => Boolean(window.__guestDiscoverySafety));
const row = page.locator('[data-guest-id]').first();
if (width > 500) {
await row.getByRole('button', { name: 'Expand backup-guest', exact: true }).focus();
await page.keyboard.press('Enter');
} else await row.tap();
await page.getByRole('tab', { name: 'Discovery', exact: true }).click();
const run = page.getByRole('button', { name: 'Run Discovery', exact: true });
await run.waitFor();
const update = async (value) =>
page.evaluate((v) => window.__guestDiscoverySafety.update(v), value);
const postCount = () => requests.filter((r) => r.method === 'POST').length;
const block = page.getByTestId('discovery-run-block');
const capture = async (state) => {
const filename = `${name}-${state}.png`,
file = path.join(output, filename);
await run.scrollIntoViewIfNeeded();
await page.screenshot({ path: file, fullPage: true });
report.screenshots.push({ filename, sha256: hash(file) });
};
const layout = async () => {
assert(await page.evaluate(() => document.documentElement.scrollWidth <= innerWidth + 1));
for (const loc of [block, page.getByTestId('guest-read-precaution')]) {
const box = await loc.boundingBox();
assert(box && box.x >= 0 && box.x + box.width <= width + 1);
assert(await loc.evaluate((el) => el.scrollWidth <= el.clientWidth + 1));
}
};
await page.getByText('Saved Home Assistant', { exact: true }).first().waitFor();
assert(await run.isDisabled());
assert.equal(postCount(), 0);
assert((await block.innerText()).includes('does not prove thaw'));
assert(await run.getAttribute('aria-describedby'));
assert.equal(await block.getAttribute('id'), await run.getAttribute('aria-describedby'));
await layout();
await capture('locked-saved');
checks.push(
'Native canonical projection, row and real drawer/Discovery: assigned node agent, saved results readable, backup blocks live run with associated wrapping reason',
);
await page.evaluate(() => {
window.__originalSafetyButton = document.querySelector(
'[aria-describedby^="discovery-run-block-"]',
);
});
for (const reason of [
'vm-locked',
'lock-unverified',
'agent-busy',
'agent-cooldown',
'agent-response-incomplete',
'agent-capacity',
'invalid-guest-key',
'agent-timeout',
]) {
for (const prefix of ['', 'prev-']) {
await update({ ...healthy, reason: prefix + reason });
assert(await run.isDisabled());
assert.equal(postCount(), 0);
assert(
await page.evaluate(
() =>
document.querySelector('[aria-describedby^="discovery-run-block-"]') ===
window.__originalSafetyButton,
),
);
}
}
await update({ ...healthy, reason: 'prev-agent-timeout' });
await capture('uncertain-saved');
await page.getByRole('tab', { name: 'History', exact: true }).click();
assert(await page.getByTestId('guest-read-precaution').isVisible());
await page.getByRole('tab', { name: 'Discovery', exact: true }).click();
assert(await run.isDisabled());
checks.push(
'All eight read deferrals, retained variants and same-ID tab/snapshot updates block without remount, POST or automatic rescan',
);
await update(healthy);
assert(await run.isEnabled());
assert.equal(await block.count(), 0);
assert.equal(postCount(), 0);
await capture('ready-manual');
await run.focus();
await page.keyboard.press('Enter');
await page.waitForFunction(() => document.body.textContent.includes('Discovery complete!'));
assert.equal(postCount(), 1);
const first = requests.find((r) => r.method === 'POST');
assert.deepEqual(JSON.parse(first.body), { force: true, hostname: 'backup-guest' });
checks.push(
'Explicit healthy keyboard run sends exactly the original target/body once; clearing safety pause alone sends nothing',
);
await page.waitForFunction(() => !document.body.textContent.includes('Discovery complete!'));
hold = true;
const secondPost = page.waitForRequest(
(req) =>
req.method() === 'POST' &&
new URL(req.url()).pathname === '/api/discovery/vm/fixture-node-agent/101',
);
if (width < 500) await run.tap();
else await run.click();
await secondPost;
await page.waitForFunction(() => document.body.textContent.includes('Scanning...'));
assert.equal(postCount(), 2);
await update({ ...initial, reason: 'agent-timeout', lock: '', backup: false });
assert(await page.getByRole('button', { name: 'Scanning...', exact: true }).isDisabled());
assert(await block.isVisible());
assert((await page.locator('body').innerText()).includes('Scanning...'));
assert.equal(postCount(), 2);
assert(release);
release();
await page.waitForFunction(() => document.body.textContent.includes('Discovery complete!'));
assert(await run.isDisabled());
assert.equal(postCount(), 2);
await capture('inflight-completed-blocked');
checks.push(
'A newly blocked snapshot neither hides nor cancels dispatched work; original completion is shown and no further run is sent',
);
assert.deepEqual(errors, []);
assert.deepEqual(offOrigin, []);
await browser.close();
browser = null;
}
report.result = 'passed';
report.verified_at = new Date().toISOString();
} catch (error) {
report.result = 'failed';
report.phase = phase;
report.error = String(error.stack || error);
throw error;
} finally {
if (browser) await browser.close();
report.cleanup.browser_closed = true;
await server.close();
report.cleanup.server_closed = true;
fs.writeFileSync(path.join(output, 'result.json'), JSON.stringify(report, null, 2) + '\n');
}
})().catch((error) => {
console.error(error.stack || String(error));
process.exitCode = 1;
});
frontend-modern/browser-tests/guest-discovery-safety.html
+12
View File
@@ -0,0 +1,12 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Guest Discovery safety verification</title>
</head>
<body>
<div id="root"></div>
<script type="module" src="./guest-discovery-safety.tsx"></script>
</body>
</html>
frontend-modern/browser-tests/guest-discovery-safety.tsx
+126
View File
@@ -0,0 +1,126 @@
// Production canonical resource projection, row, drawer, router and Docs.
// Synthetic observations only; no native QGA, diagnostic or backup operation.
import { createSignal, Show } from 'solid-js';
import { render } from 'solid-js/web';
import { Router, Route } from '@solidjs/router';
import { useWorkloads } from '../src/hooks/useWorkloads';
import { mergeCanonicalResourceSnapshot } from '../src/utils/resourceStateAdapters';
import { GuestRow } from '../src/components/Workloads/GuestRow';
import { GuestDrawer } from '../src/components/Workloads/GuestDrawer';
import { DarkModeContext } from '../src/contexts/appRuntime';
import { syncAIRuntimeSettings } from '../src/stores/aiRuntimeState';
import Docs from '../src/pages/Docs';
import type { Resource } from '../src/types/resource';
import '../src/index.css';
syncAIRuntimeSettings({ discovery_enabled: true });
interface Observation {
state?: string;
reason?: string;
lock?: string;
backup?: boolean;
version?: string;
assignment?: boolean;
diskUsed?: number;
}
const initial: Observation = {
state: 'deferred',
reason: 'prev-vm-locked',
lock: 'backup',
backup: true,
version: '6.4.5',
assignment: true,
diskUsed: 5 * 1024 ** 3,
};
const nativeResource = (value: Observation): Resource => ({
id: 'fixture-pve:pve-a:101',
type: 'vm',
name: 'backup-guest',
displayName: 'backup-guest',
platformType: 'proxmox-pve',
platformScopes: ['proxmox-pve'],
sourceType: 'api',
status: 'online',
sources: ['proxmox', 'agent'],
lastSeen: Date.UTC(2026, 9, 4, 14),
cpu: { current: 10 },
memory: { current: 25, used: 1024 ** 3 / 4, total: 1024 ** 3 },
disk: {
current: ((value.diskUsed ?? 0) / (10 * 1024 ** 3)) * 100,
used: value.diskUsed,
total: 10 * 1024 ** 3,
},
proxmox: {
instance: 'fixture-pve',
nodeName: 'pve-a',
vmid: 101,
runtimeStatus: 'running',
template: false,
guestAgentStatus: value.state,
diskStatusReason: value.reason,
lock: value.lock,
guestAgentExpected: true,
backupInProgress: value.backup ?? false,
lastBackup: '2026-10-04T06:00:00Z',
disks: [],
},
agent: {
agentId: 'fixture-pulse-agent',
agentVersion: value.version || '',
hostname: 'backup-guest',
osName: 'Linux',
},
discoveryTarget: value.assignment
? { resourceType: 'vm', agentId: 'fixture-node-agent', resourceId: '101' }
: undefined,
});
function Fixture() {
const [observation, setObservation] = createSignal(initial);
const [snapshot, setSnapshot] = createSignal<Resource[]>([nativeResource(initial)]);
const state = useWorkloads(() => true, { resourceSnapshot: snapshot });
const guest = () => state.workloads()[0]!;
const [expanded, setExpanded] = createSignal(false);
(window as any).__guestDiscoverySafety = {
update: (next: Observation) => {
setObservation(next);
setSnapshot((previous) => mergeCanonicalResourceSnapshot([nativeResource(next)], previous));
},
observation,
identity: () => guest().id,
};
return (
<main class="mx-auto min-h-screen max-w-5xl space-y-3 bg-surface p-3 text-base-content">
<h1 class="text-base font-semibold">Guest Discovery safety evidence</h1>
<p class="text-xs text-muted">Synthetic observations. No guest check is sent.</p>
<Show when={Boolean(guest())}>
<table class="w-full table-fixed">
<tbody>
<GuestRow
guest={guest()}
visibleColumnIds={['name', 'disk']}
isExpanded={expanded()}
onClick={() => setExpanded(!expanded())}
workloadTableLayoutMode={window.innerWidth <= 768 ? 'phone' : 'wide'}
/>
</tbody>
</table>
<Show when={expanded()}>
<section aria-label="Guest details">
<GuestDrawer guest={guest()} onClose={() => setExpanded(false)} />
</section>
</Show>
</Show>
</main>
);
}
render(
() => (
<DarkModeContext.Provider value={() => document.documentElement.classList.contains('dark')}>
<Router>
<Route path="/docs/*docPath" component={Docs} />
<Route path="/*" component={Fixture} />
</Router>
</DarkModeContext.Provider>
),
document.getElementById('root')!,
);
frontend-modern/browser-tests/guest-filesystem-provenance.cjs
+136
View File
@@ -0,0 +1,136 @@
const assert = require('node:assert/strict');
const fs = require('node:fs');
const path = require('node:path');
const crypto = require('node:crypto');
const { chromium, webkit } = require('playwright');
const root = '/workspace/frontend-modern';
const output = '/workspace/tmp/filesystem-read-provenance/browser-final';
const origin = 'http://127.0.0.1:5324';
const runtime = [
'frontend-modern/src/components/Workloads/GuestDrawerOverview.tsx',
'frontend-modern/src/components/Workloads/diskListModel.ts',
];
const hash = file => crypto.createHash('sha256').update(fs.readFileSync(file)).digest('hex');
const settle = page => page.evaluate(async () => {
await document.fonts.ready;
await new Promise(resolve => requestAnimationFrame(() => requestAnimationFrame(resolve)));
});
(async () => {
fs.mkdirSync(output, { recursive: true });
const result = {
result: 'incomplete', playwright: require('playwright/package.json').version,
runtime_hashes: Object.fromEntries(runtime.map(file => [file, hash('/workspace/' + file)])),
cases: [], screenshots: [], cleanup: { browser: false, server: false },
limits: 'Real GuestRow/full GuestDrawer/DetailSectionTable and CSS; synthetic guest observations and APIs. No native QGA, backup, cause, thaw, writes, installed/reporter or release acceptance.',
};
assert.equal(result.playwright, JSON.parse(fs.readFileSync('/workspace/tests/integration/package-lock.json')).packages['node_modules/@playwright/test'].version);
process.chdir(root);
const { createServer } = await import(path.join(root, 'node_modules/vite/dist/node/index.js'));
const server = await createServer({root, configFile: path.join(root, 'vite.config.ts'), cacheDir: output + '/cache', server: { host: '127.0.0.1', port: 5324, strictPort: true }});
let browser;
try {
await server.listen();
for (const [name, engine, width, dark] of [
['chromium-desktop-light', chromium, 1365, false],
['chromium-desktop-dark', chromium, 1365, true],
['webkit-phone-dark', webkit, 390, true],
['webkit-narrow-light', webkit, 320, false],
]) {
browser = await engine.launch(engine === chromium ? { headless: true, channel: 'chromium', args: ['--no-sandbox'] } : { headless: true });
const context = await browser.newContext({viewport: { width, height: width <= 768 ? 844 : 900 }, isMobile: width <= 768, hasTouch: width <= 768, locale: 'en-GB', timezoneId: 'UTC'});
const page = await context.newPage();
page.setDefaultTimeout(15000);
page.setDefaultNavigationTimeout(60000);
const record = {name, browser: browser.version(), width, dark, errors: [], requests: [], checks: [], states: []};
result.cases.push(record);
page.on('pageerror', error => record.errors.push(error.message));
await page.route('**/*', route => {
const request = route.request(); const url = new URL(request.url());
if (url.origin !== origin) { record.requests.push({ method: request.method(), path: url.pathname, disposition: 'off-origin blocked' }); return route.abort(); }
if (url.pathname.startsWith('/api/')) {
record.requests.push({ method: request.method(), path: url.pathname, disposition: 'synthetic' });
return route.fulfill({status: 200, contentType: 'application/json', body: JSON.stringify({ success: true, data: [], alerts: [], anomalies: [], config: null })});
}
return route.continue();
});
await page.addInitScript(isDark => {
document.addEventListener('DOMContentLoaded', () => { if (isDark) document.documentElement.classList.add('dark'); });
window.__trustedInput = [];
document.addEventListener('click', event => window.__trustedInput.push({ type: 'click', trusted: event.isTrusted }), true);
document.addEventListener('keydown', event => window.__trustedInput.push({ type: event.key, trusted: event.isTrusted }), true);
}, dark);
await page.goto(`${origin}/browser-tests/guest-disk-provenance.html`);
await page.waitForFunction(() => window.__guestDiskProvenance);
const apply = async next => { await page.evaluate(next => window.__guestDiskProvenance.apply(next), next); await settle(page); };
await apply({ reason: 'prev-vm-locked', data: true, usage: 90, lock: 'backup', kind: 'qemu' });
const row = page.locator('[data-guest-id="fixture-pve:pve-a:101"]');
const disclosure = row.locator('[data-row-action]');
if (width <= 768) await row.locator('[data-workload-disk-read-status]').tap();
else { await disclosure.focus(); await page.keyboard.press('Enter'); }
const drawer = page.getByRole('region', {name: 'Guest details'});
await drawer.waitFor({state: 'visible'});
const technical = drawer.getByTestId('guest-technical-details');
const filesystem = technical.getByText('/data', {exact: true}).locator('xpath=ancestor::tr');
const progress = filesystem.getByRole('progressbar');
const text = async () => (await filesystem.innerText()).replace(/\s+/g, ' ').trim();
const capture = async state => {
await technical.scrollIntoViewIfNeeded(); await settle(page);
const file = `${name}-${state}.png`;
await technical.screenshot({path: path.join(output, file)});
result.screenshots.push({file, sha256: hash(path.join(output, file)), name, state});
};
assert.match(await text(), /Last known 90% · 9\.00 GB\/10\.0 GB · EXT4/);
assert.equal(await progress.count(), 0);
record.checks.push('Retained quantitative values are labelled last known beside their filesystem identity, with no current utilization bar.');
record.states.push({state: 'retained', text: await text()});
await capture('retained');
await apply({ lock: '' });
assert.match(await text(), /Last known 90%/); assert.equal(await progress.count(), 0);
record.checks.push('Clearing only the operation lock cannot promote a deferred reading.');
const reasons = await page.evaluate(() => window.__guestDiskProvenance.deferrals.map(item => item[0]));
for (const reason of reasons) {
await apply({reason: 'prev-' + reason});
assert.match(await text(), /Last known 90%/); assert.equal(await progress.count(), 0);
await apply({reason});
assert.match(await text(), /Usage unavailable · \?\/10\.0 GB · EXT4/);
assert.doesNotMatch(await text(), /90%|9\.00 GB/); assert.equal(await progress.count(), 0);
}
record.checks.push('All eight fixed read reasons distinguish retained data from unavailable numeric carriers.');
record.states.push({state: 'unavailable', text: await text()});
await capture('unavailable');
await apply({reason: '', usage: 30, lock: ''});
assert.match(await text(), /30% · 3\.00 GB\/10\.0 GB · EXT4/);
assert.equal(await progress.getAttribute('aria-valuenow'), '30');
assert.equal(await progress.getAttribute('aria-label'), 'Filesystem /data utilization');
assert.equal(await drawer.isVisible(), true);
record.checks.push('A same-guest fresh read restores the correct numeric value and accessible utilization bar without closing the drawer.');
record.states.push({state: 'fresh', text: await text()});
await capture('fresh');
await apply({reason: 'prev-vm-locked', kind: 'lxc'});
assert.match(await text(), /30% · 3\.00 GB\/10\.0 GB · EXT4/); assert.equal(await progress.getAttribute('aria-valuenow'), '30');
record.checks.push('Independent LXC filesystem evidence does not inherit a VM-only reason.');
await apply({kind: 'qemu', data: false, reason: 'vm-locked'});
assert.equal(await technical.getByText('/data', {exact: true}).count(), 0);
assert.match(await technical.innerText(), /Guest reads paused while Proxmox reports a VM operation lock/);
record.checks.push('With no prior filesystem sample, the readable explanation remains and no filesystem value is invented.');
const overflow = await technical.evaluate(el => ({width: el.clientWidth, scrollWidth: el.scrollWidth, documentWidth: document.documentElement.clientWidth, documentScroll: document.documentElement.scrollWidth}));
assert.ok(overflow.scrollWidth <= overflow.width + 1, JSON.stringify(overflow));
record.layout = overflow;
await drawer.getByRole('button', {name: 'Collapse backup-guest details'}).click();
assert.equal(await drawer.count(), 0);
record.input = await page.evaluate(() => window.__trustedInput);
assert.ok(record.input.some(event => event.trusted));
assert.deepEqual(record.errors, []);
assert.equal(record.requests.filter(request => !['GET','HEAD'].includes(request.method)).length, 0);
record.checks.push('Trusted keyboard/touch disclosure and close work; no uncaught page error, overflow or mutating request.');
await context.close(); await browser.close(); browser = null;
}
result.result = 'passed';
} catch (error) {
result.error = String(error.stack || error); throw error;
} finally {
if (browser) await browser.close(); result.cleanup.browser = true;
await server.close(); result.cleanup.server = true;
fs.writeFileSync(path.join(output, 'result.json'), JSON.stringify(result, null, 2) + '\n');
}
})().catch(error => { console.error(error); process.exitCode = 1; });
frontend-modern/browser-tests/history-observation-gaps.cjs
+339
View File
@@ -0,0 +1,339 @@
const assert = require('node:assert/strict');
const fs = require('node:fs');
const path = require('node:path');
const crypto = require('node:crypto');
const { chromium, webkit } = require('playwright');
const root = '/workspace/frontend-modern';
const output = '/workspace/tmp/history-observation-gaps/browser';
const origin = 'http://127.0.0.1:5317';
const runtime = [
'frontend-modern/src/components/Workloads/GuestDrawerHistory.tsx',
'frontend-modern/src/components/Workloads/guestDrawerModel.ts',
];
const hash = (file) => crypto.createHash('sha256').update(fs.readFileSync(file)).digest('hex');
const settle = (page) =>
page.evaluate(async () => {
await document.fonts.ready;
await new Promise((resolve) => requestAnimationFrame(() => requestAnimationFrame(resolve)));
});
(async () => {
fs.mkdirSync(output, { recursive: true });
const result = {
result: 'incomplete',
playwright: require('playwright/package.json').version,
runtime_hashes: Object.fromEntries(runtime.map((file) => [file, hash('/workspace/' + file)])),
cases: [],
screenshots: [],
limits:
'Production full GuestDrawer/History, client and CSS with synthetic local APIs. Not native guest/QGA/backup/thaw, installed recovery, physical-phone or release acceptance.',
};
assert.equal(
result.playwright,
JSON.parse(fs.readFileSync('/workspace/tests/integration/package-lock.json')).packages[
'node_modules/@playwright/test'
].version,
);
process.chdir(root);
const { createServer } = await import(path.join(root, 'node_modules/vite/dist/node/index.js'));
const server = await createServer({
root,
configFile: path.join(root, 'vite.config.ts'),
cacheDir: output + '/cache',
server: { host: '127.0.0.1', port: 5317, strictPort: true },
});
let browser;
try {
await server.listen();
for (const [name, engine, width, dark] of [
['chromium-desktop-light', chromium, 1365, false],
['chromium-desktop-dark', chromium, 1365, true],
['webkit-phone-dark', webkit, 390, true],
['webkit-narrow-light', webkit, 320, false],
]) {
browser = await engine.launch(
engine === chromium
? { headless: true, channel: 'chromium', args: ['--no-sandbox'] }
: { headless: true },
);
const page = await browser.newPage({
viewport: { width, height: width < 768 ? 844 : 900 },
isMobile: width < 768,
hasTouch: width < 768,
locale: 'en-GB',
timezoneId: 'UTC',
});
page.setDefaultTimeout(12000);
page.setDefaultNavigationTimeout(60000);
const record = {
name,
width,
dark,
browser: browser.version(),
errors: [],
requests: [],
checks: [],
};
result.cases.push(record);
let mode = 'gaps';
const start = Date.UTC(2026, 9, 4, 12);
const point = (minute, value) => ({
timestamp: start + minute * 60_000,
value,
min: value,
max: value,
});
const times = [0, 15, 30, 45, 60];
const metrics = () => ({
cpu: times.map((minute, index) => point(minute, 10 + index * 5)),
memory: (mode === 'complete' ? times : [0, 15, 60]).map((minute) =>
point(minute, 20 + minute / 6),
),
disk: (mode === 'complete' ? times : [0, 60]).map((minute) => point(minute, 50)),
netin: times.map((minute) => point(minute, 100)),
netout: (mode === 'complete' ? times : [0, 60]).map((minute) => point(minute, 0)),
diskread: (mode === 'complete' ? times : [0, 60]).map((minute) => point(minute, 0)),
diskwrite: times.map((minute) => point(minute, 200)),
});
page.on('pageerror', (error) => record.errors.push(error.message));
await page.route('**/*', async (route) => {
const url = new URL(route.request().url());
if (url.origin !== origin) {
record.requests.push({ origin: url.origin, blocked: true });
return route.abort();
}
if (!url.pathname.startsWith('/api/')) return route.continue();
record.requests.push({ path: url.pathname, method: route.request().method(), mode });
if (url.pathname === '/api/metrics-store/history') {
assert.equal(route.request().method(), 'GET');
assert.equal(url.searchParams.get('resourceId'), 'fixture:pve:101');
assert.equal(url.searchParams.get('maxPoints'), '240');
if (mode === '503' || mode === '403')
return route.fulfill({
status: Number(mode),
json: { error: 'private-fixture-detail' },
});
return route.fulfill({
json: {
resourceType: url.searchParams.get('resourceType'),
resourceId: url.searchParams.get('resourceId'),
range: url.searchParams.get('range'),
start,
end: start + 60 * 60_000,
metrics: metrics(),
source: 'store',
},
});
}
return route.fulfill({
json:
url.pathname === '/api/license/runtime-capabilities'
? {
capabilities: [],
limits: [],
max_history_days: 90,
hosted_mode: false,
runtime: { build: 'community' },
blocked_capabilities: [],
}
: { success: true, data: [], alerts: [], anomalies: [], config: null },
});
});
await page.addInitScript((isDark) => {
document.addEventListener('DOMContentLoaded', () =>
document.documentElement.classList.toggle('dark', isDark),
);
window.__trustedInputs = [];
for (const type of ['click', 'input', 'keydown'])
document.addEventListener(
type,
(event) => window.__trustedInputs.push({ type, trusted: event.isTrusted }),
true,
);
}, dark);
await page.goto(origin + '/browser-tests/history-observation-gaps.html');
const activate = async (locator) => (width < 768 ? locator.tap() : locator.click());
await activate(page.getByRole('tab', { name: 'History', exact: true }));
const chart = page.locator('[data-history-group="utilization"]');
const slider = chart.getByRole('slider', { name: 'Inspect Utilization history' });
await slider.waitFor();
await settle(page);
await page.evaluate(() => {
window.__originalHistoryChart = document.querySelector(
'[data-history-group="utilization"]',
);
});
const memoryPath = chart.locator('path[stroke="#f59e0b"]');
assert.equal(await memoryPath.count(), 1);
assert.equal((await memoryPath.getAttribute('d')).match(/L/g).length, 1);
assert.equal(await chart.locator('[data-history-observation="memory"]').count(), 1);
assert.equal(await chart.locator('path[stroke="#10b981"]').count(), 0);
assert.equal(await chart.locator('[data-history-observation="disk"]').count(), 2);
assert.equal(
(await chart.locator('path[stroke="#8b5cf6"]').getAttribute('d')).match(/L/g).length,
4,
);
assert.match(
await chart.locator('[data-history-gaps]').innerText(),
/Missing observations: Memory, Disk\./,
);
assert.equal(await page.locator('[data-history-gaps]').count(), 3);
const description = await chart
.getByRole('img', { name: 'Utilization history' })
.evaluate((el) =>
el
.getAttribute('aria-describedby')
.split(' ')
.map((id) => document.getElementById(id).textContent)
.join(' '),
);
assert.match(description, /Missing observations: Memory, Disk\./);
assert.match(description, /does not identify the cause or duration/);
const noteGeometry = await chart.locator('[data-history-gaps]').evaluate((el) => ({
width: el.clientWidth,
scrollWidth: el.scrollWidth,
colour: getComputedStyle(el).color,
}));
assert.ok(noteGeometry.scrollWidth <= noteGeometry.width + 1);
record.noteGeometry = noteGeometry;
record.checks.push(
'Actual stored gap splits only affected metrics; recovered singleton/zero samples stay visible; visible/accessibly linked explanation fits.',
);
const screenshot = async (suffix) => {
await settle(page);
const file = `${name}-${suffix}.png`;
await page.screenshot({ path: path.join(output, file), fullPage: true });
result.screenshots.push({
file,
sha256: hash(path.join(output, file)),
name,
state: suffix,
});
};
await screenshot('gaps');
await slider.focus();
await page.keyboard.press('Home');
await page.keyboard.press('ArrowRight');
await page.keyboard.press('ArrowRight');
assert.equal(await slider.inputValue(), '2');
assert.match(
await slider.getAttribute('aria-valuetext'),
/Memory no observation\. Disk no observation\./,
);
assert.match(await chart.innerText(), /Memory\s*-/);
assert.equal(await chart.locator('circle[r="3"]').count(), 1);
record.checks.push(
'Trusted Home/Arrow keys inspect the missing stored time without borrowing nearby/live memory or disk.',
);
if (width < 768) {
await slider.scrollIntoViewIfNeeded();
const box = await slider.boundingBox();
await page.touchscreen.tap(box.x + box.width * 0.75, box.y + box.height / 2);
assert.notEqual(await slider.inputValue(), '2');
record.checks.push('Trusted touch operates the native inspector.');
} else {
await page.getByRole('heading', { name: 'Drawer History missing observations' }).click();
const box = await chart.getByRole('img', { name: 'Utilization history' }).boundingBox();
await page.mouse.move(box.x + box.width * 0.537, box.y + box.height / 2);
assert.match(
await chart.locator('[data-testid="guest-history-hover-time"]').innerText(),
/12:30/,
);
assert.equal(await chart.locator('circle[r="3"]').count(), 1);
record.checks.push(
'Trusted pointer matches the exact stored time, not a synthetic line value.',
);
}
const reads = () =>
record.requests.filter((r) => r.path === '/api/metrics-store/history').length;
const before = reads();
await activate(page.getByRole('button', { name: 'Simulate live recovery' }));
assert.equal(reads(), before);
assert.equal(await chart.locator('[data-history-gaps]').count(), 1);
assert.equal(await chart.locator('[data-history-observation="memory"]').count(), 1);
assert.equal(
await page.evaluate(
() =>
window.__originalHistoryChart ===
document.querySelector('[data-history-group="utilization"]'),
),
true,
);
record.checks.push(
'Live status recovery cannot fill missing stored evidence, refetch or remount the chart.',
);
mode = '503';
await activate(page.getByRole('button', { name: 'Refresh history' }));
await page.getByRole('button', { name: 'Retry history' }).waitFor();
assert.equal(await chart.locator('[data-history-gaps]').count(), 1);
assert.equal(await chart.locator('path[stroke="#10b981"]').count(), 0);
assert.doesNotMatch(await page.locator('body').innerText(), /private-fixture-detail/);
record.checks.push('Transient read failure retains the gap and does not leak the raw error.');
mode = 'complete';
await activate(page.getByRole('button', { name: 'Retry history' }));
await page.waitForFunction(
() => document.querySelectorAll('[data-history-gaps]').length === 0,
);
assert.equal(await chart.locator('[data-history-observation]').count(), 0);
assert.equal((await memoryPath.getAttribute('d')).match(/L/g).length, 4);
assert.equal(
await page.evaluate(
() =>
window.__originalHistoryChart ===
document.querySelector('[data-history-group="utilization"]'),
),
true,
);
record.checks.push(
'Matching complete stored refresh joins actual consecutive observations; chart identity stays stable.',
);
await screenshot('complete');
mode = '403';
await activate(page.getByRole('button', { name: 'Refresh history' }));
await page.waitForFunction(
() => document.querySelectorAll('[data-testid="guest-history-group-chart"]').length === 0,
);
assert.equal(await page.locator('[data-history-gaps]').count(), 0);
assert.doesNotMatch(await page.locator('body').innerText(), /private-fixture-detail/);
record.checks.push('Access denial withdraws all stored chart/gap/inspection evidence.');
assert.equal(reads(), before + 3);
assert.equal(
await page.evaluate(() => document.documentElement.scrollWidth <= innerWidth + 1),
true,
);
assert.equal(
record.requests.some((r) => r.method && r.method !== 'GET'),
false,
);
assert.equal(
record.requests.some((r) => r.blocked),
false,
);
assert.deepEqual(record.errors, []);
assert.equal(
await page.evaluate(() => window.__trustedInputs.every((entry) => entry.trusted)),
true,
);
record.checks.push(
'No page error, horizontal overflow, external request or mutation; unchanged user-request read budget.',
);
await page.close();
await browser.close();
browser = null;
}
result.result = 'passed';
} catch (error) {
result.failure = error.stack || String(error);
throw error;
} finally {
fs.writeFileSync(path.join(output, 'result.json'), JSON.stringify(result, null, 2) + '\n');
if (browser) await browser.close();
await server.close();
}
process.stdout.write(JSON.stringify(result) + '\n');
})().catch((error) => {
process.stderr.write(error.stack + '\n');
process.exitCode = 1;
});
frontend-modern/browser-tests/history-observation-gaps.html
+12
View File
@@ -0,0 +1,12 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Drawer History missing-observation verification</title>
</head>
<body>
<div id="root"></div>
<script type="module" src="./history-observation-gaps.tsx"></script>
</body>
</html>
frontend-modern/browser-tests/history-observation-gaps.tsx
+87
View File
@@ -0,0 +1,87 @@
// Real production drawer/CSS. The browser supplies synthetic stored readings;
// no collector, guest-agent, backup, thaw or installation operation is run.
import { createSignal } from 'solid-js';
import { render } from 'solid-js/web';
import { Router, Route } from '@solidjs/router';
import { GuestDrawer } from '../src/components/Workloads/GuestDrawer';
import { DarkModeContext } from '../src/contexts/appRuntime';
import { syncAIRuntimeSettings } from '../src/stores/aiRuntimeState';
import type { WorkloadGuest } from '../src/types/workloads';
import '../src/index.css';
syncAIRuntimeSettings({ discovery_enabled: false });
function Fixture() {
const [recovered, setRecovered] = createSignal(false);
const guest = (): WorkloadGuest => ({
id: 'fixture:pve:101',
vmid: 101,
name: 'backup-guest',
node: 'pve',
instance: 'fixture',
status: 'running',
type: 'qemu',
cpu: 0.1,
cpus: 2,
memory: {
total: 1024 ** 3,
used: 256 * 1024 ** 2,
free: 768 * 1024 ** 2,
usage: 25,
observation: {
state: recovered() ? 'current' : 'last-known',
source: 'status-mem',
observedAt: '2026-10-04T12:01:00Z',
},
},
disk: { total: 10 * 1024 ** 3, used: 5 * 1024 ** 3, usage: 50 },
diskStatusReason: recovered() ? '' : 'prev-vm-locked',
lock: recovered() ? '' : 'backup',
guestAgentStatus: recovered() ? 'available' : 'deferred',
backupInProgress: !recovered(),
networkIn: 0,
networkOut: 0,
diskRead: 0,
diskWrite: 0,
uptime: 3600,
template: false,
lastBackup: 0,
tags: [],
lastSeen: '2026-10-04T12:04:00Z',
});
return (
<main class="min-h-screen space-y-3 bg-surface p-3 text-base-content">
<h1 class="text-base font-semibold">Drawer History missing observations</h1>
<p class="text-xs text-muted">
Synthetic readings. No backup or guest-agent command is sent.
</p>
<button
class="rounded-sm border border-border p-2 text-xs"
onClick={() => setRecovered(true)}
>
Simulate live recovery
</button>
<section class="mx-auto max-w-5xl" aria-label="Guest details">
<GuestDrawer guest={guest()} onClose={() => {}} />
</section>
</main>
);
}
render(
() => (
<Router>
<Route
path="*"
component={() => (
<DarkModeContext.Provider
value={() => document.documentElement.classList.contains('dark')}
>
<Fixture />
</DarkModeContext.Provider>
)}
/>
</Router>
),
document.getElementById('root')!,
);
frontend-modern/browser-verification/1014496b3e63eb6a.json
+60
View File
@@ -0,0 +1,60 @@
{
"version": 1,
"result": "passed",
"base_sha": "664b29f930b19f91b74f673914c85e3b806b54f7",
"changed_paths": [
"frontend-modern/src/components/Workloads/GuestDrawerOverview.tsx",
"frontend-modern/src/components/Workloads/diskListModel.ts"
],
"content_sha256": {
"frontend-modern/src/components/Workloads/GuestDrawerOverview.tsx": "16ccfd4c8a865c02d4ab3580d7872518cb599715245fb10ebbe3dfbe375cb104",
"frontend-modern/src/components/Workloads/diskListModel.ts": "8b43eeb3e08dff191039ee19ea74baa89703d6b8f89d801a5ff0f8cdef09dffc"
},
"routes": [
"/browser-tests/guest-disk-provenance.html \u2014 production Workloads GuestRow/full GuestDrawer; synthetic APIs, not the full App route"
],
"states": [
"Retained filesystem usage with each of eight prev- reasons: labelled last known, no current bars/threshold colours",
"Same eight non-retained read failures: unavailable used bytes, identity/capacity retained",
"Lock-only clearance retains the label; same-guest fresh read restores accessible 30% utilization",
"LXC independent reading and no-sample readable VM status"
],
"interactions": [
"Trusted desktop Enter disclosure and close",
"Trusted WebKit phone/narrow touch on the row provenance cue and close",
"Synthetic same-guest poll replacements while keeping the drawer open",
"Inspected every retained/unavailable/fresh screenshot in all four browser/theme/width groups"
],
"viewports": [
{
"width": 1365,
"height": 900,
"browser": "chromium-desktop-light",
"theme": "light"
},
{
"width": 1365,
"height": 900,
"browser": "chromium-desktop-dark",
"theme": "dark"
},
{
"width": 390,
"height": 844,
"browser": "webkit-phone-dark",
"theme": "dark"
},
{
"width": 320,
"height": 844,
"browser": "webkit-narrow-light",
"theme": "light"
}
],
"verified_at": "2026-10-04T21:47:04Z",
"playwright_version": "1.56.1",
"checks": 28,
"screenshots_inspected": 12,
"script": "frontend-modern/browser-tests/guest-filesystem-provenance.cjs",
"limits": "Real GuestRow/full GuestDrawer/DetailSectionTable and CSS; synthetic guest observations and APIs. No native QGA, backup, cause, thaw, writes, installed/reporter or release acceptance."
}
frontend-modern/browser-verification/430ebb1acb72552b.json
+67
View File
@@ -0,0 +1,67 @@
{
"version": 1,
"result": "passed",
"base_sha": "2cb3f6a9043e0bab9c54c5b5ef57b3e1cb994781",
"changed_paths": [
"frontend-modern/src/components/Discovery/DiscoveryTab.tsx",
"frontend-modern/src/components/Discovery/useDiscoveryTabState.ts",
"frontend-modern/src/components/Workloads/GuestDrawer.tsx"
],
"content_sha256": {
"frontend-modern/src/components/Workloads/GuestDrawer.tsx": "d8ba9c14cbfac0756f8df461bea07c40a39ef56ba907301b71e92286e410facc",
"frontend-modern/src/components/Discovery/DiscoveryTab.tsx": "01e2d687067df69d4f3eb6ea74742c9ce478f278976ee87c96c3567ff1b82896",
"frontend-modern/src/components/Discovery/useDiscoveryTabState.ts": "19388b7e163d5fff79416f4bcb9c47ba0718cede9a56dbdf331065ab09cc00ee"
},
"routes": [
"/browser-tests/guest-discovery-safety.html: production canonical resource projection, Workloads row, GuestDrawer and DiscoveryTab with synthetic APIs"
],
"states": [
"Desktop light/dark and 390px/320px WebKit phone: locked guest and saved results",
"Every declared deferral and retained variant; same-ID/tab changes",
"Healthy manual action; uncertain read; in-flight scan completion while further runs are blocked"
],
"interactions": [
"Keyboard expand/tab/explicit run and touch expand/run; accessible disabled reason; wrapping warning/saved service inspection",
"No POST on blocked/unlock/snapshot/tab changes; exactly one original-body POST per explicit enabled run",
"No remount, cancellation or false in-flight-stop; original completion remains visible and additional run is blocked"
],
"viewports": [
{
"width": 1365,
"height": 900
},
{
"width": 390,
"height": 844
},
{
"width": 320,
"height": 844
}
],
"verified_at": "2026-10-04T15:40:18.829Z",
"script": "frontend-modern/browser-tests/guest-discovery-safety.cjs",
"browsers": [
{
"group": "chromium-desktop-light",
"version": "141.0.7390.37"
},
{
"group": "chromium-desktop-dark",
"version": "141.0.7390.37"
},
{
"group": "webkit-phone-dark",
"version": "26.0"
},
{
"group": "webkit-narrow-light",
"version": "26.0"
}
],
"limitations": [
"Synthetic backend responses, not a native guest/backup or all-path backend admission proof.",
"Vite-transformed production components; production build is separately checked in the source-proof VM.",
"Independent thaw/covered-filesystem writes remain necessary; an unlocked snapshot does not prove them."
]
}
frontend-modern/browser-verification/51fcee98af1964fd.json
+59
View File
@@ -0,0 +1,59 @@
{
"version": 1,
"result": "passed",
"base_sha": "0ad41e4c9a309ce459dd717d16d343b46efcac8a",
"changed_paths": [
"frontend-modern/src/components/Workloads/GuestDrawerHistory.tsx",
"frontend-modern/src/components/Workloads/guestDrawerModel.ts"
],
"content_sha256": {
"frontend-modern/src/components/Workloads/GuestDrawerHistory.tsx": "8b94c3d9409f46bedf812b00ff9f3156da9ace6885ec04456ea5fed59d842127",
"frontend-modern/src/components/Workloads/guestDrawerModel.ts": "79e176dd5640190f6bf11d3a6bbddc63fcd18c37b19e4e7098163a95d542b0a5"
},
"routes": [
"/browser-tests/history-observation-gaps.html \u2014 production full GuestDrawer/History/client/CSS, not the full App route"
],
"states": [
"Stored CPU is continuous while memory and disk miss intervening group times; preserve split segments and recovery singleton/zero readings.",
"Live recovery alone preserves stored gaps and chart identity without a read.",
"Transient 503 retains data/gaps; matching complete stored refresh restores actual contiguous lines; denied 403 withdraws all history."
],
"interactions": [
"Trusted desktop click, Home/Arrow keyboard inspection and pointer at exact missing stored time.",
"Trusted WebKit 390/320 px native inspector touch, keyboard arrows and refresh/retry.",
"Trusted live-recovery action does not alter stored points, query budget or chart identity.",
"All eight captures visually inspected for gaps, isolated points, contiguous recovery, light/dark readability and 390/320 px wrapping."
],
"viewports": [
{
"width": 1365,
"height": 900,
"browser": "chromium-desktop-light",
"theme": "light"
},
{
"width": 1365,
"height": 900,
"browser": "chromium-desktop-dark",
"theme": "dark"
},
{
"width": 390,
"height": 844,
"browser": "webkit-phone-dark",
"theme": "dark"
},
{
"width": 320,
"height": 844,
"browser": "webkit-narrow-light",
"theme": "light"
}
],
"verified_at": "2026-10-04T22:39:45.443984Z",
"playwright_version": "1.56.1",
"checks": 32,
"screenshots_inspected": 8,
"script": "frontend-modern/browser-tests/history-observation-gaps.cjs",
"limits": "Production full GuestDrawer/History, client and CSS with synthetic local APIs. Not native guest/QGA/backup/thaw, installed recovery, physical-phone or release acceptance."
}
frontend-modern/browser-verification/69011d6c032c31d5.json
+42
View File
@@ -0,0 +1,42 @@
{
"version": 1,
"base_sha": "482a2614e32f875990ee84662dfcbf3010ca7181",
"verified_at": "2026-10-04T23:18:40.484Z",
"result": "passed",
"changed_paths": [
"frontend-modern/src/components/Workloads/GuestDrawer.tsx"
],
"content_sha256": {
"frontend-modern/src/components/Workloads/GuestDrawer.tsx": "a1d197b12aa0bed084664472aeeca1d4b0415911a8aa94c8aa3742e59ce77789"
},
"routes": [
"/proxmox/overview (production drawer and canonical projection in offline synthetic fixture)"
],
"viewports": [
{
"width": 1365,
"height": 900
},
{
"width": 390,
"height": 844
}
],
"states": [
"Combined published saved-URL header action and reviewed guest read/Discovery precaution on exact production bytes. Playwright 1.56.1 Chromium desktop light and WebKit phone dark; eight screenshots inspected.",
"Locked and uncertain retained readings keep saved results visible but live Discovery disabled; healthy clear permits only explicit dispatch. In-flight completion remains attributed when observation becomes deferred.",
"Header Open retains safe href, target _blank, noopener noreferrer, desktop 32px/phone 44px target and expanded drawer. Missing and unsafe URLs expose no link. No horizontal overflow, page errors or external requests."
],
"interactions": [
"Saved header Open retains safe href/target/rel, phone target and expansion; absent/unsafe URLs expose no link, without sending Discovery",
"Native canonical projection, row and real drawer/Discovery: assigned node agent, saved results readable, backup blocks live run with associated wrapping reason",
"All eight read deferrals, retained variants and same-ID tab/snapshot updates block without remount, POST or automatic rescan",
"Explicit healthy keyboard run sends exactly the original target/body once; clearing safety pause alone sends nothing",
"A newly blocked snapshot neither hides nor cancels dispatched work; original completion is shown and no further run is sent",
"Saved header Open retains safe href/target/rel, phone target and expansion; absent/unsafe URLs expose no link, without sending Discovery",
"Native canonical projection, row and real drawer/Discovery: assigned node agent, saved results readable, backup blocks live run with associated wrapping reason",
"All eight read deferrals, retained variants and same-ID tab/snapshot updates block without remount, POST or automatic rescan",
"Explicit healthy keyboard run sends exactly the original target/body once; clearing safety pause alone sends nothing",
"A newly blocked snapshot neither hides nor cancels dispatched work; original completion is shown and no further run is sent"
]
}
frontend-modern/browser-verification/c45de66a92dcfc91.json
+79
View File
@@ -0,0 +1,79 @@
{
"version": 1,
"base_sha": "bc3f4d81d14873caae6f769dfa670d16fa13d518",
"verified_at": "2026-10-04T20:32:45.762Z",
"result": "passed",
"changed_paths": [
"frontend-modern/src/components/Discovery/DiscoveryTab.tsx",
"frontend-modern/src/components/Discovery/useDiscoveryTabState.ts"
],
"content_sha256": {
"frontend-modern/src/components/Discovery/DiscoveryTab.tsx": "2e93209a2b3830f77a0ad94d02bd59ea01509932f8d51e594d60f55d1f5895d0",
"frontend-modern/src/components/Discovery/useDiscoveryTabState.ts": "e600bb993b80e85b6df48a9e070f1bc2db51d7bb8e5ffd6216e3861d610def14"
},
"routes": [
"/browser-tests/guest-discovery-safety.html: production Workloads row, GuestDrawer and DiscoveryTab with synthetic canonical snapshots",
"/browser-tests/discovery-outcome-ownership.html: reused production DiscoveryTab/state, synthetic per-guest APIs and production event-bus progress"
],
"viewports": [
{
"width": 1365,
"height": 900
},
{
"width": 390,
"height": 844
},
{
"width": 320,
"height": 844
}
],
"states": [
"Guest operation lock and all current guest-read deferrals with/without retained data; healthy explicit action",
"Original HTTP result retained after a same-target precaution without cancellation or another scan",
"Background completed-with-error retains an accessible failed outcome and saved evidence",
"Late old-target HTTP result cannot overwrite the current guest or stop its manual request",
"Scanner completion during HTTP does not claim saved success; later terminal event cannot strand earlier success",
"Current explicit failure remains visible without an automatic retry"
],
"interactions": [
"Keyboard and touch drawer expansion, Discovery tabs, explicit enabled runs and associated disabled reasons",
"No POST on blocked/unlock/tab/snapshot change and exactly one target/body POST per explicit healthy action",
"Keyboard-labelled error dismissal with saved content preserved",
"Change target during an in-flight request, deliver the old and current responses independently and inspect original ownership",
"Render desktop light/dark and WebKit 390px/320px, inspect wrapping and all 24 captures"
],
"script": "frontend-modern/browser-tests/guest-discovery-completion.cjs",
"browsers": [
{
"name": "desktop-light",
"browser": "141.0.7390.37",
"width": 1365,
"dark": false
},
{
"name": "desktop-dark",
"browser": "141.0.7390.37",
"width": 1365,
"dark": true
},
{
"name": "phone-dark",
"browser": "26.0",
"width": 390,
"dark": true
},
{
"name": "narrow-light",
"browser": "26.0",
"width": 320,
"dark": false
}
],
"limitations": [
"Synthetic inputs and API responses, not native QGA, backup or automatic poller acceptance.",
"Vite-transformed production components. Exact production build is checked separately in the source-proof VM.",
"No original freeze cause, independent thaw, covered-filesystem writes, installed use, physical phone or containing release is established."
]
}
frontend-modern/public/docs/TRUENAS.md
+94 -28
View File
@@ -101,48 +101,114 @@ All endpoints require admin authentication.
### Testing and adding a connection (API)
The **Settings → Infrastructure → Platform connections** flow is the simplest option. For an API client,
prepare the private Pulse header file as described in the
[API authentication guide](API.md). The Pulse API token authenticates to
Pulse; the separate TrueNAS API key belongs in the connection's JSON file.
Do not put either credential in shell arguments, URLs or a public thread.
Prefer **Settings → Infrastructure → Platform connections** for setup. These
API examples are for an authorised administrator using curl 7.76 or later,
not a way to diagnose stale polling. Testing makes a live system-information
request to TrueNAS; saving creates a connection and starts ordinary polling.
Do not use either during a backup, freeze/thaw or an unresponsive-host incident.
Create or protect the connection file before opening it in a trusted editor:
First prepare a private Pulse header file using the
[API token file procedure](API.md#api-token-recommended), with the admin access
needed for connection management. The Pulse token authenticates to Pulse;
the separate TrueNAS API key belongs in the connection's JSON file. Enter
credentials only in your local editor, never in arguments, URLs or a thread.
On a trusted machine, as the account running curl, prepare the connection
file below. Stop if either file-preparation step fails. This preserves an
existing file, restricts the directory and refuses symlinked credential paths
or a non-regular connection file before opening the editor:
```bash
umask 077
mkdir -p "$HOME/.config/pulse"
touch "$HOME/.config/pulse/truenas-connection.json"
chmod 600 "$HOME/.config/pulse/truenas-connection.json"
vi "$HOME/.config/pulse/truenas-connection.json"
(
set -eu
umask 077
config_dir="$HOME/.config/pulse"
connection_file="$config_dir/truenas-connection.json"
if [ -L "$HOME/.config" ] || [ -L "$config_dir" ] || [ -L "$connection_file" ]; then
printf 'Refusing a symlinked credential path.\n' >&2
exit 1
fi
if [ -e "$connection_file" ] && [ ! -f "$connection_file" ]; then
printf 'Connection file must be a regular file.\n' >&2
exit 1
fi
mkdir -p "$config_dir"
chmod 700 "$config_dir"
touch "$connection_file"
chmod 600 "$connection_file"
vi "$connection_file"
)
```
In that file, enter a JSON object with `name`, `host` (the TrueNAS HTTPS URL),
`username` (the key owner) and `apiKey`. Keep both credential files private;
do not paste their contents into a terminal command or upload them.
In the editor, save a JSON object with `name`, `host` (the TrueNAS HTTPS URL),
`username` (the key owner) and `apiKey`. Keep both credential files private and
outside repositories and diagnostics. Do not upload them.
Test the connection before saving it:
The following examples run **on the Pulse host**, using its direct loopback
address. They ignore curl configuration, do not follow redirects or retry,
and retain each response in a new private local file. Responses and errors can
contain identifying details or echoed credentials; review them locally, not in
a public terminal recording or thread. Keep `--disable` first so local curl
configuration cannot enable tracing or change the request.
Test once without saving:
```bash
curl --fail-with-body --silent --show-error \
--header "@$HOME/.config/pulse/api-header" \
--header 'Content-Type: application/json' \
--data-binary "@$HOME/.config/pulse/truenas-connection.json" \
http://127.0.0.1:7655/api/truenas/connections/test
(
set -eu
umask 077
result_file=$(mktemp "$HOME/.config/pulse/truenas-test.XXXXXX")
printf 'Private response file: %s\n' "$result_file"
status=$(curl --disable --fail-with-body --silent --show-error \
--connect-timeout 5 --max-time 20 --proto '=http' --noproxy 127.0.0.1 \
--header "@$HOME/.config/pulse/api-header" \
--header 'Content-Type: application/json' \
--data-binary "@$HOME/.config/pulse/truenas-connection.json" \
--output "$result_file" --write-out '%{http_code}' \
http://127.0.0.1:7655/api/truenas/connections/test)
printf 'HTTP %s\n' "$status"
[ "$status" = 200 ]
)
```
After checking the test response, save the connection with:
Require exit 0, HTTP **200** and `success: true` in that private test response.
A redirect, HTTP error or `000` is not a successful test. Test success is not
inventory, metric or guest-thaw acceptance.
Only after reviewing the test response, create the connection once:
```bash
curl --fail-with-body --silent --show-error \
--header "@$HOME/.config/pulse/api-header" \
--header 'Content-Type: application/json' \
--data-binary "@$HOME/.config/pulse/truenas-connection.json" \
http://127.0.0.1:7655/api/truenas/connections
(
set -eu
umask 077
result_file=$(mktemp "$HOME/.config/pulse/truenas-save.XXXXXX")
printf 'Private response file: %s\n' "$result_file"
status=$(curl --disable --fail-with-body --silent --show-error \
--connect-timeout 5 --max-time 20 --proto '=http' --noproxy 127.0.0.1 \
--header "@$HOME/.config/pulse/api-header" \
--header 'Content-Type: application/json' \
--data-binary "@$HOME/.config/pulse/truenas-connection.json" \
--output "$result_file" --write-out '%{http_code}' \
http://127.0.0.1:7655/api/truenas/connections)
printf 'HTTP %s\n' "$status"
[ "$status" = 201 ]
)
```
These Pulse URLs are loopback-only examples. For a remote Pulse server, use
its trusted HTTPS origin without disabling certificate verification.
A successful save returns HTTP **201** and the new connection's `id`, not
proof of a complete poll. If saving times out or loses its response, it may
already have created the connection. Check **Platform connections** or
`/api/truenas/connections` in your existing signed-in Pulse browser before
trying again; do not blindly repeat POST and create duplicates. Continue with
the [ordinary polling checks](#stale-truenas-data) for collection acceptance.
For remote Pulse access, replace the loopback origin in the relevant command
with your trusted **HTTPS** origin and change `--proto '=http'` to
`--proto '=https'`. Keep certificate verification enabled. For a private CA,
add `--cacert /path/to/trusted-ca.pem` using a CA obtained independently from
your administrator, not from the failing connection. Do not add `--insecure`,
`--location`, verbose/trace flags or automatic retries. Share only the relevant
HTTP status and a manually redacted error, never the full response or headers.
## Troubleshooting
frontend-modern/src/components/Discovery/DiscoveryTab.tsx
+32 -3
View File
@@ -1,4 +1,4 @@
import { Component, For, Show, createMemo } from 'solid-js';
import { Component, For, Show, createMemo, createUniqueId } from 'solid-js';
import ExternalLinkIcon from 'lucide-solid/icons/external-link';
import TriangleAlertIcon from 'lucide-solid/icons/triangle-alert';
import type { ResourceType } from '../../types/discovery';
@@ -33,6 +33,7 @@ import {
import { ButtonLink, CopyValueButton } from '@/components/shared/Button';
import { CopyableCodeRow } from '@/components/shared/CopyableCodeRow';
import { InfoCardFrame, InfoCardKeyValueRow } from '@/components/shared/InfoCardFrame';
import { InlineNotice } from '@/components/shared/InlineNotice';
import { useDiscoveryTabState } from './useDiscoveryTabState';
import { orderFactsByActionability } from './factOrdering';
import { deriveCliCommand } from './cliCommand';
@@ -54,6 +55,8 @@ interface DiscoveryTabProps {
commandsEnabled?: boolean;
/** Show the primary run action at the top of embedded drawer contexts. */
showManualRunAction?: boolean;
/** Safety pause for live manual scans, independent of command permissions. */
runBlockReason?: string | null;
}
export const DiscoveryTab: Component<DiscoveryTabProps> = (props) => {
@@ -75,6 +78,7 @@ export const DiscoveryTab: Component<DiscoveryTabProps> = (props) => {
liveElapsedSeconds,
notesText,
mutateDiscovery,
runBlockReason,
saveError,
scanError,
scanProgress,
@@ -90,6 +94,7 @@ export const DiscoveryTab: Component<DiscoveryTabProps> = (props) => {
startEditingNotes,
validDiscovery,
} = useDiscoveryTabState(props);
const runBlockReasonId = `discovery-run-block-${createUniqueId()}`;
const suggestedURLReasonText = createMemo(() => {
const current = discovery();
@@ -225,6 +230,19 @@ export const DiscoveryTab: Component<DiscoveryTabProps> = (props) => {
</div>
</Show>
<Show when={runBlockReason()}>
{(reason) => (
<InlineNotice
id={runBlockReasonId}
tone="warning"
role="status"
data-testid="discovery-run-block"
>
{reason()}
</InlineNotice>
)}
</Show>
<Show when={showManualRunAction()}>
<InfoCardFrame>
<div class="flex flex-col gap-3 sm:flex-row sm:items-center sm:justify-between">
@@ -236,6 +254,7 @@ export const DiscoveryTab: Component<DiscoveryTabProps> = (props) => {
type="button"
onClick={() => handleTriggerDiscovery(true)}
disabled={isScanning() || !canTriggerDiscovery()}
aria-describedby={runBlockReason() ? runBlockReasonId : undefined}
class="inline-flex min-h-9 items-center justify-center gap-1.5 rounded-sm border border-border bg-surface px-3 py-1.5 text-xs font-medium text-base-content transition-colors hover:bg-surface-hover disabled:cursor-not-allowed disabled:opacity-50"
>
<Show
@@ -455,7 +474,10 @@ export const DiscoveryTab: Component<DiscoveryTabProps> = (props) => {
{/* Scan Error */}
<Show when={scanError()}>
<div class="mb-4 rounded-md border border-red-200 bg-red-50 p-4 dark:border-red-800 dark:bg-red-900/25">
<div
role="alert"
class="mb-4 rounded-md border border-red-200 bg-red-50 p-4 dark:border-red-800 dark:bg-red-900/25"
>
<div class="flex items-start gap-3">
<svg
class="w-5 h-5 text-red-500 dark:text-red-400 shrink-0 mt-0.5"
@@ -471,10 +493,14 @@ export const DiscoveryTab: Component<DiscoveryTabProps> = (props) => {
/>
</svg>
<div>
<p class="text-sm font-medium text-red-800 dark:text-red-200">Discovery Failed</p>
<p class="text-sm font-medium text-red-800 dark:text-red-200">
Discovery did not complete
</p>
<p class="text-sm text-red-700 dark:text-red-300 mt-1">{scanError()}</p>
</div>
<button
type="button"
aria-label="Dismiss Discovery error"
onClick={() => setScanError(null)}
class="ml-auto text-red-500 hover:text-red-700 dark:text-red-400 dark:hover:text-red-300"
>
@@ -633,6 +659,7 @@ export const DiscoveryTab: Component<DiscoveryTabProps> = (props) => {
<button
onClick={() => handleTriggerDiscovery(true)}
disabled={isScanning() || !canTriggerDiscovery()}
aria-describedby={runBlockReason() ? runBlockReasonId : undefined}
class="px-4 py-2 bg-blue-600 text-white text-sm rounded-md hover:bg-blue-700 disabled:opacity-50 disabled:cursor-not-allowed transition-colors"
>
{isScanning() ? (
@@ -708,6 +735,7 @@ export const DiscoveryTab: Component<DiscoveryTabProps> = (props) => {
<button
onClick={() => handleTriggerDiscovery(true)}
disabled={isScanning() || !canTriggerDiscovery()}
aria-describedby={runBlockReason() ? runBlockReasonId : undefined}
class="px-4 py-2 bg-blue-600 text-white text-sm rounded-md hover:bg-blue-700 disabled:opacity-50 disabled:cursor-not-allowed transition-colors"
>
{isScanning() ? (
@@ -1192,6 +1220,7 @@ export const DiscoveryTab: Component<DiscoveryTabProps> = (props) => {
<button
onClick={() => handleTriggerDiscovery(true)}
disabled={isScanning() || !canTriggerDiscovery()}
aria-describedby={runBlockReason() ? runBlockReasonId : undefined}
class="px-3 py-1.5 bg-surface-hover text-base-content text-xs rounded-sm hover:bg-surface-hover disabled:opacity-50 disabled:cursor-not-allowed transition-colors flex items-center gap-1.5"
>
<Show
frontend-modern/src/components/Discovery/__tests__/DiscoveryRun.lifecycle.test.tsx
+341
View File
@@ -0,0 +1,341 @@
import { createSignal } from 'solid-js';
import { cleanup, render, waitFor } from '@solidjs/testing-library';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import type { ResourceDiscovery } from '@/types/discovery';
import { eventBus } from '@/stores/events';
import { resetAIRuntimeState, syncAIRuntimeSettings } from '@/stores/aiRuntimeState';
import * as api from '@/api/discovery';
import { useDiscoveryTabState } from '../useDiscoveryTabState';
vi.mock('@/api/discovery', () => ({
getDiscovery: vi.fn(async () => null),
getDiscoveryInfo: vi.fn(async () => ({
ai_provider: { provider: 'test', model: 'test', is_local: true, label: 'Local' },
})),
getConnectedAgents: vi.fn(async () => ({ count: 1, agents: [{ agent_id: 'node-agent' }] })),
triggerDiscovery: vi.fn(),
updateDiscoveryNotes: vi.fn(async () => undefined),
}));
function deferred<T>() {
let resolve!: (value: T) => void;
let reject!: (reason: Error) => void;
const promise = new Promise<T>((yes, no) => {
resolve = yes;
reject = no;
});
return { promise, resolve, reject };
}
const saved = (id = '100'): ResourceDiscovery => ({
id: `vm:node-agent:${id}`,
resource_type: 'vm',
resource_id: id,
target_id: 'node-agent',
hostname: `guest-${id}`,
service_type: 'home-assistant',
service_name: `Saved service ${id}`,
service_version: 'fixture',
category: 'home_automation',
cli_access: '',
user_secrets: {},
ai_reasoning: 'Synthetic saved evidence',
scan_duration: 0,
facts: [],
config_paths: [],
data_paths: [],
log_paths: [],
ports: [],
user_notes: '',
discovered_at: '2026-10-03T12:00:00Z',
updated_at: '2026-10-03T12:00:00Z',
confidence: 0.9,
});
async function mount(initialAgent = 'node-agent') {
const [id, setId] = createSignal('100');
const [agent, setAgent] = createSignal(initialAgent);
let state!: ReturnType<typeof useDiscoveryTabState>;
const view = render(() => {
state = useDiscoveryTabState({
resourceType: 'vm',
get agentId() {
return agent();
},
get resourceId() {
return id();
},
get hostname() {
return `guest-${id()}`;
},
});
return <span>Discovery lifecycle</span>;
});
await waitFor(() => expect(state.discoveryInfo.loading).toBe(false));
await waitFor(() => expect(state.discovery.loading).toBe(false));
return { state, setId, setAgent, dispose: view.unmount };
}
beforeEach(() => {
vi.resetAllMocks();
resetAIRuntimeState();
syncAIRuntimeSettings({ discovery_enabled: true } as Parameters<typeof syncAIRuntimeSettings>[0]);
vi.mocked(api.getDiscovery).mockImplementation(async (_type, _agent, id) => saved(id));
vi.mocked(api.getDiscoveryInfo).mockResolvedValue({
ai_provider: { provider: 'test', model: 'test', is_local: true, label: 'Local' },
});
vi.mocked(api.getConnectedAgents).mockResolvedValue({ count: 0, agents: [] });
vi.mocked(api.triggerDiscovery).mockResolvedValue(saved());
});
afterEach(() => {
cleanup();
resetAIRuntimeState();
vi.useRealTimers();
});
describe('Discovery outcome ownership', () => {
it.each(['success', 'failure'] as const)(
'does not apply an old target HTTP %s to the new guest or its in-flight run',
async (outcome) => {
const old = deferred<ResourceDiscovery>();
const current = deferred<ResourceDiscovery>();
vi.mocked(api.triggerDiscovery)
.mockReturnValueOnce(old.promise)
.mockReturnValueOnce(current.promise);
const { state, setId } = await mount();
const first = state.handleTriggerDiscovery(true);
setId('101');
await waitFor(() => expect(state.discovery()?.resource_id).toBe('101'));
const second = state.handleTriggerDiscovery(true);
if (outcome === 'success') old.resolve(saved());
else old.reject(new Error('Old guest execution paused'));
await first;
expect(state.discovery()?.resource_id).toBe('101');
expect(state.isScanning()).toBe(true);
expect(state.scanError()).toBeNull();
expect(state.scanSuccess()).toBe(false);
current.resolve(saved('101'));
await second;
expect(state.scanSuccess()).toBe(true);
expect(api.triggerDiscovery).toHaveBeenNthCalledWith(2, 'vm', 'node-agent', '101', {
force: true,
hostname: 'guest-101',
});
},
);
it('invalidates an old run even when the operator leaves and returns to the same target', async () => {
const old = deferred<ResourceDiscovery>();
vi.mocked(api.triggerDiscovery).mockReturnValueOnce(old.promise);
const { state, setId } = await mount();
const first = state.handleTriggerDiscovery(true);
setId('101');
await waitFor(() => expect(state.discovery()?.resource_id).toBe('101'));
setId('100');
await waitFor(() => expect(state.discovery()?.resource_id).toBe('100'));
old.resolve({ ...saved(), service_name: 'Old run result' });
await first;
expect(state.discovery()?.service_name).toBe('Saved service 100');
expect(state.scanSuccess()).toBe(false);
});
it('does not resurrect a run after Discovery was disabled and re-enabled', async () => {
const old = deferred<ResourceDiscovery>();
vi.mocked(api.triggerDiscovery).mockReturnValueOnce(old.promise);
const { state } = await mount();
const first = state.handleTriggerDiscovery(true);
syncAIRuntimeSettings({ discovery_enabled: false } as Parameters<
typeof syncAIRuntimeSettings
>[0]);
await waitFor(() => expect(state.canTriggerDiscovery()).toBe(false));
syncAIRuntimeSettings({ discovery_enabled: true } as Parameters<
typeof syncAIRuntimeSettings
>[0]);
await waitFor(() => expect(state.canTriggerDiscovery()).toBe(true));
old.resolve({ ...saved(), service_name: 'Disabled run result' });
await first;
expect(state.discovery()?.service_name).toBe('Saved service 100');
expect(state.scanSuccess()).toBe(false);
});
it('ignores HTTP completion after disposal without scheduling another success timer', async () => {
const old = deferred<ResourceDiscovery>();
vi.mocked(api.triggerDiscovery).mockReturnValueOnce(old.promise);
const { state, dispose } = await mount();
vi.useFakeTimers();
const first = state.handleTriggerDiscovery(true);
dispose();
old.resolve(saved());
await first;
expect(state.scanSuccess()).toBe(false);
expect(vi.getTimerCount()).toBe(0);
});
it('does not leave a spinner running when a retained handler has no target agent', async () => {
const { state } = await mount('');
await state.handleTriggerDiscovery(true);
expect(api.triggerDiscovery).not.toHaveBeenCalled();
expect(state.scanError()).toBe('Agent identifier unavailable for discovery');
expect(state.isScanning()).toBe(false);
expect(state.scanSuccess()).toBe(false);
});
it('does not let an earlier success timer clear the success of a later explicit run', async () => {
const { state } = await mount();
vi.useFakeTimers();
await state.handleTriggerDiscovery(true);
await vi.advanceTimersByTimeAsync(1500);
await state.handleTriggerDiscovery(true);
await vi.advanceTimersByTimeAsync(600);
expect(state.scanSuccess()).toBe(true);
await vi.advanceTimersByTimeAsync(1400);
expect(state.scanSuccess()).toBe(false);
});
it('does not leave the previous HTTP success visible forever after a later completion event', async () => {
const { state } = await mount();
vi.useFakeTimers();
await state.handleTriggerDiscovery(true);
eventBus.emit('ai_discovery_progress', {
resource_id: 'vm:node-agent:100',
status: 'completed',
});
await vi.advanceTimersByTimeAsync(2500);
expect(state.scanSuccess()).toBe(false);
expect(state.discovery()?.resource_id).toBe('100');
});
it.each(['failed', 'completed'] as const)(
'retains a background %s-with-error outcome and the saved evidence, not silent success',
async (status) => {
const { state } = await mount();
const before = vi.mocked(api.getDiscovery).mock.calls.length;
vi.useFakeTimers();
eventBus.emit('ai_discovery_progress', {
resource_id: 'vm:node-agent:100',
status,
error: 'Guest execution paused: backup lock observed',
current_step: 'Guest execution paused',
});
await vi.advanceTimersByTimeAsync(600);
expect(state.scanError()).toBe('Guest execution paused: backup lock observed');
expect(state.scanSuccess()).toBe(false);
expect(state.discovery()?.service_name).toBe('Saved service 100');
expect(api.getDiscovery).toHaveBeenCalledTimes(before);
expect(api.triggerDiscovery).not.toHaveBeenCalled();
},
);
it('shows background progress without claiming completion of a manual HTTP request', async () => {
const http = deferred<ResourceDiscovery>();
vi.mocked(api.triggerDiscovery).mockReturnValueOnce(http.promise);
const { state } = await mount();
eventBus.emit('ai_discovery_progress', {
resource_id: 'vm:node-agent:100',
status: 'running',
current_step: 'Collecting guest evidence',
});
expect(state.isScanning()).toBe(true);
eventBus.emit('ai_discovery_progress', {
resource_id: 'vm:node-agent:100',
status: 'completed',
});
expect(state.isScanning()).toBe(false);
const first = state.handleTriggerDiscovery(true);
eventBus.emit('ai_discovery_progress', {
resource_id: 'vm:node-agent:100',
status: 'completed',
});
expect(state.isScanning()).toBe(true);
expect(state.scanSuccess()).toBe(false);
http.resolve(saved());
await first;
expect(state.scanSuccess()).toBe(true);
});
it('drops a delayed background refresh when its target changed before dispatch', async () => {
const { state, setId } = await mount();
vi.useFakeTimers();
eventBus.emit('ai_discovery_progress', {
resource_id: 'vm:node-agent:100',
status: 'completed',
});
setId('101');
await Promise.resolve();
const before = vi.mocked(api.getDiscovery).mock.calls.length;
await vi.advanceTimersByTimeAsync(600);
expect(api.getDiscovery).toHaveBeenCalledTimes(before);
expect(state.discovery()?.resource_id).toBe('101');
expect(api.triggerDiscovery).not.toHaveBeenCalled();
});
it('does not apply a delayed background read to a newer manual result on the same guest', async () => {
const refresh = deferred<ResourceDiscovery | null>();
const { state } = await mount();
vi.mocked(api.getDiscovery).mockReturnValueOnce(refresh.promise);
vi.useFakeTimers();
eventBus.emit('ai_discovery_progress', {
resource_id: 'vm:node-agent:100',
status: 'completed',
});
await vi.advanceTimersByTimeAsync(500);
vi.mocked(api.triggerDiscovery).mockResolvedValueOnce({
...saved(),
service_name: 'New manual result',
});
await state.handleTriggerDiscovery(true);
refresh.resolve({ ...saved(), service_name: 'Old background result' });
await Promise.resolve();
expect(state.discovery()?.service_name).toBe('New manual result');
expect(state.scanSuccess()).toBe(true);
});
it.each([401, 403])(
'withdraws saved evidence when a completion refresh is denied with %s',
async (status) => {
const { state } = await mount();
vi.mocked(api.getDiscovery).mockRejectedValueOnce(
Object.assign(new Error('Discovery details unavailable.'), { status }),
);
vi.useFakeTimers();
eventBus.emit('ai_discovery_progress', {
resource_id: 'vm:node-agent:100',
status: 'completed',
});
await vi.advanceTimersByTimeAsync(500);
expect(state.discovery()).toBeNull();
expect(state.scanSuccess()).toBe(false);
expect(api.triggerDiscovery).not.toHaveBeenCalled();
},
);
it('keeps saved evidence without success when HTTP supplies no result', async () => {
vi.mocked(api.triggerDiscovery).mockResolvedValueOnce(null as unknown as ResourceDiscovery);
const { state } = await mount();
await state.handleTriggerDiscovery(true);
expect(state.scanError()).toBe('Discovery returned no saved result.');
expect(state.scanSuccess()).toBe(false);
expect(state.discovery()?.service_name).toBe('Saved service 100');
expect(state.isScanning()).toBe(false);
});
it('does not close another guest’s notes editor when an old save finishes', async () => {
const save = deferred<ResourceDiscovery>();
vi.mocked(api.updateDiscoveryNotes).mockReturnValueOnce(save.promise);
const { state, setId } = await mount();
state.startEditingNotes();
state.setNotesText('Original guest notes');
const first = state.handleSaveNotes();
setId('101');
await waitFor(() => expect(state.discovery()?.resource_id).toBe('101'));
state.startEditingNotes();
state.setNotesText('New guest draft');
save.resolve(saved());
await first;
expect(state.editingNotes()).toBe(true);
expect(state.notesText()).toBe('New guest draft');
expect(api.updateDiscoveryNotes).toHaveBeenCalledWith('vm', 'node-agent', '100', {
user_notes: 'Original guest notes',
});
});
});
frontend-modern/src/components/Discovery/__tests__/DiscoveryRun.safety.test.tsx
+184
View File
@@ -0,0 +1,184 @@
import { createSignal, Suspense } from 'solid-js';
import { cleanup, fireEvent, render, screen, waitFor } from '@solidjs/testing-library';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import type { ResourceDiscovery } from '@/types/discovery';
import { DiscoveryTab } from '../DiscoveryTab';
import { useDiscoveryTabState } from '../useDiscoveryTabState';
import { resetAIRuntimeState, syncAIRuntimeSettings } from '@/stores/aiRuntimeState';
import * as discoveryApi from '@/api/discovery';
vi.mock('@/api/discovery', async (importOriginal) => ({
...(await importOriginal<typeof import('@/api/discovery')>()),
getDiscovery: vi.fn(async () => null),
getDiscoveryInfo: vi.fn(async () => ({
ai_provider: { provider: 'test', model: 'test', is_local: true, label: 'Local' },
commands: [],
command_categories: [],
})),
getConnectedAgents: vi.fn(async () => ({ count: 1, agents: [{ agent_id: 'node-agent' }] })),
triggerDiscovery: vi.fn(async () => null),
}));
const reason = 'Discovery paused for this guest. Saved results remain available.';
const target = {
resourceType: 'vm' as const,
agentId: 'node-agent',
resourceId: '100',
hostname: 'guest',
};
const saved: ResourceDiscovery = {
id: 'vm:node-agent:100',
resource_type: 'vm',
resource_id: '100',
target_id: 'node-agent',
hostname: 'guest',
service_type: 'home-assistant',
service_name: 'Saved service',
category: 'home_automation',
service_version: 'fixture',
cli_access: '',
user_secrets: {},
ai_reasoning: 'Synthetic saved evidence',
scan_duration: 0,
facts: [],
config_paths: [],
data_paths: [],
log_paths: [],
ports: [],
user_notes: '',
discovered_at: '2026-10-03T12:00:00Z',
updated_at: '2026-10-03T12:00:00Z',
confidence: 0.9,
};
beforeEach(() => {
resetAIRuntimeState();
syncAIRuntimeSettings({ discovery_enabled: true } as Parameters<typeof syncAIRuntimeSettings>[0]);
vi.mocked(discoveryApi.getDiscovery).mockResolvedValue(null);
vi.mocked(discoveryApi.triggerDiscovery).mockResolvedValue(saved);
});
afterEach(() => {
cleanup();
resetAIRuntimeState();
vi.clearAllMocks();
});
describe('manual discovery safety', () => {
it.each([
['loading', (): Promise<ResourceDiscovery | null> => new Promise(() => undefined)],
['absent', async (): Promise<ResourceDiscovery | null> => null],
[
'empty',
async (): Promise<ResourceDiscovery | null> => ({
...saved,
service_type: '',
service_name: '',
confidence: 0,
}),
],
['saved', async (): Promise<ResourceDiscovery | null> => saved],
] as const)(
'blocks all run affordances with %s discovery while retaining passive reads',
async (_name, lookup) => {
vi.mocked(discoveryApi.getDiscovery).mockImplementation(lookup);
const props = { ...target, runBlockReason: reason };
render(() => <DiscoveryTab {...props} />);
await waitFor(() => expect(discoveryApi.getDiscoveryInfo).toHaveBeenCalled());
await waitFor(() => {
const buttons = screen.getAllByRole('button', { name: /^(Run|Re-scan|Update) Discovery/ });
for (const button of buttons) expect(button).toBeDisabled();
});
expect(screen.getByTestId('discovery-run-block')).toHaveTextContent(reason);
expect(discoveryApi.getDiscovery).toHaveBeenCalledWith('vm', 'node-agent', '100');
expect(discoveryApi.triggerDiscovery).not.toHaveBeenCalled();
},
);
it('keeps saved evidence mounted and restores only the explicit run action, not an automatic scan', async () => {
vi.mocked(discoveryApi.getDiscovery).mockResolvedValue(saved);
const [block, setBlock] = createSignal<string | null>(reason);
const props = {
...target,
get runBlockReason() {
return block();
},
showManualRunAction: true,
};
render(() => (
<Suspense>
<DiscoveryTab {...props} />
</Suspense>
));
const button = await screen.findByRole('button', { name: 'Run Discovery' });
await waitFor(() => expect(button).toBeDisabled());
const evidence = await screen.findByText('Saved service');
expect(evidence).toBeVisible();
setBlock(null);
await waitFor(() => expect(button).toBeEnabled());
expect(screen.getByText('Saved service')).toBe(evidence);
expect(screen.queryByTestId('discovery-run-block')).toBeNull();
expect(discoveryApi.triggerDiscovery).not.toHaveBeenCalled();
fireEvent.click(button);
await waitFor(() => expect(discoveryApi.triggerDiscovery).toHaveBeenCalledTimes(1));
});
it('checks the current block inside the handler even when a caller retained the old enabled verdict', async () => {
const [block, setBlock] = createSignal<string | null>(null);
const props = {
...target,
get runBlockReason() {
return block();
},
};
let state!: ReturnType<typeof useDiscoveryTabState>;
render(() => {
state = useDiscoveryTabState(props);
return <span>state</span>;
});
await waitFor(() => expect(state.canTriggerDiscovery()).toBe(true));
setBlock(reason);
await state.handleTriggerDiscovery(true);
expect(discoveryApi.triggerDiscovery).not.toHaveBeenCalled();
expect(state.isScanning()).toBe(false);
expect(state.scanSuccess()).toBe(false);
});
it('does not cancel or hide an in-flight scan when a new snapshot blocks further runs', async () => {
const finishes: Array<(value: ResourceDiscovery) => void> = [];
vi.mocked(discoveryApi.triggerDiscovery).mockImplementation(
() =>
new Promise((resolve) => {
finishes.push(resolve);
}),
);
const [block, setBlock] = createSignal<string | null>(null);
const props = {
...target,
get runBlockReason() {
return block();
},
};
let state!: ReturnType<typeof useDiscoveryTabState>;
render(() => {
state = useDiscoveryTabState(props);
return <span>state</span>;
});
await waitFor(() => expect(state.canTriggerDiscovery()).toBe(true));
const original = state.handleTriggerDiscovery(true);
expect(state.isScanning()).toBe(true);
setBlock(reason);
const second = state.handleTriggerDiscovery(true);
try {
expect(discoveryApi.triggerDiscovery).toHaveBeenCalledTimes(1);
expect(state.isScanning()).toBe(true);
} finally {
for (const finish of finishes) finish(saved);
await Promise.all([original, second]);
}
expect(state.isScanning()).toBe(false);
expect(state.discovery()).toBe(saved);
expect(state.scanSuccess()).toBe(true);
setBlock(null);
expect(discoveryApi.triggerDiscovery).toHaveBeenCalledTimes(1);
});
});
frontend-modern/src/components/Discovery/useDiscoveryTabState.ts
+138 -54
View File
@@ -24,6 +24,8 @@ export interface DiscoveryTabStateProps {
resourceId: string;
hostname: string;
commandsEnabled?: boolean;
/** Snapshot-based safety pause for manual runs; saved reads remain available. */
runBlockReason?: string | null;
}
const makeResourceId = (type: ResourceType, agentId: string, resourceId: string) =>
@@ -45,6 +47,18 @@ export function useDiscoveryTabState(props: DiscoveryTabStateProps) {
const [httpScanInProgress, setHttpScanInProgress] = createSignal(false);
const [copiedDiscoveryValue, setCopiedDiscoveryValue] = createSignal('');
let copyFeedbackTimer: ReturnType<typeof setTimeout> | undefined;
let successTimer: ReturnType<typeof setTimeout> | undefined;
let completionTimer: ReturnType<typeof setTimeout> | undefined;
let contextVersion = 0;
let runVersion = 0;
let disposed = false;
const clearScanTimers = () => {
clearTimeout(successTimer);
clearTimeout(completionTimer);
successTimer = undefined;
completionTimer = undefined;
};
const targetAgentId = createMemo(() => props.agentId || '');
const discoverySourceKey = createMemo(
@@ -57,6 +71,23 @@ export function useDiscoveryTabState(props: DiscoveryTabStateProps) {
const { discoveryFeatureEnabled, discoveryFeatureKnownDisabled } =
useDiscoveryFeatureAvailability();
// A target key alone is insufficient: leaving and returning to the same
// guest must not admit a response from the previous visit. A precaution on
// the same target does not change this ownership or cancel dispatched work.
const captureContext = () => ({
version: contextVersion,
key: discoverySourceKey(),
type: props.resourceType,
agentId: targetAgentId(),
resourceId: props.resourceId,
hostname: props.hostname,
});
const isCurrentContext = (context: ReturnType<typeof captureContext>) =>
!disposed &&
context.version === contextVersion &&
context.key === discoverySourceKey() &&
discoveryFeatureEnabled();
const [discoveryInfo] = createResource(
() => (discoveryFeatureEnabled() ? props.resourceType : null),
async (type) => {
@@ -120,8 +151,13 @@ export function useDiscoveryTabState(props: DiscoveryTabStateProps) {
// the tab-wide banner already tells the user to configure one before
// scanning. (Command/connectivity gaps are surfaced separately, not blocked
// here, since their backend semantics are murkier.)
const runBlockReason = createMemo(() => props.runBlockReason?.trim() || null);
const canTriggerDiscovery = createMemo(
() => discoveryFeatureEnabled() && aiProviderConfigured() && Boolean(targetAgentId()),
() =>
discoveryFeatureEnabled() &&
aiProviderConfigured() &&
Boolean(targetAgentId()) &&
!runBlockReason(),
);
const [discovery, { refetch, mutate }] = createResource(
@@ -143,6 +179,9 @@ export function useDiscoveryTabState(props: DiscoveryTabStateProps) {
createEffect(() => {
void discoveryFeatureEnabled();
void discoverySourceKey();
contextVersion++;
runVersion++;
clearScanTimers();
setIsScanning(false);
setHttpScanInProgress(false);
setScanProgress(null);
@@ -185,11 +224,23 @@ export function useDiscoveryTabState(props: DiscoveryTabStateProps) {
});
const handleTriggerDiscovery = async (force = false) => {
// Check the current snapshot at the dispatch boundary as well as disabling
// buttons. Do not cancel a dispatched scan when a precaution appears.
if (disposed || runBlockReason() || isScanning()) return;
if (!discoveryFeatureEnabled()) {
setScanError('Service context is disabled in Settings -> Pulse Intelligence -> Assistant.');
return;
}
const context = captureContext();
if (!context.agentId) {
setScanError('Agent identifier unavailable for discovery');
return;
}
if (!aiProviderConfigured()) return;
clearScanTimers();
const version = ++runVersion;
const ownsRun = () => isCurrentContext(context) && version === runVersion;
setIsScanning(true);
setHttpScanInProgress(true);
setScanProgress(null);
@@ -199,59 +250,58 @@ export function useDiscoveryTabState(props: DiscoveryTabStateProps) {
setLiveElapsedSeconds(0);
try {
const agentId = targetAgentId();
if (!agentId) {
setScanError('Agent identifier unavailable for discovery');
return;
}
const result = await triggerDiscovery(props.resourceType, agentId, props.resourceId, {
const result = await triggerDiscovery(context.type, context.agentId, context.resourceId, {
force,
hostname: props.hostname,
hostname: context.hostname,
});
if (result) {
mutate(result);
}
setHttpScanInProgress(false);
setIsScanning(false);
setScanProgress(null);
setScanStartTime(null);
if (!ownsRun()) return;
if (!result) throw new Error('Discovery returned no saved result.');
mutate(result);
setScanError(null);
setScanSuccess(true);
setTimeout(() => setScanSuccess(false), 2000);
successTimer = setTimeout(() => {
successTimer = undefined;
if (ownsRun()) setScanSuccess(false);
}, 2000);
} catch (err) {
if (!ownsRun()) return;
console.error('Discovery failed:', err);
const message = err instanceof Error ? err.message : 'Discovery scan failed';
if (message.includes('no connected agent')) {
setScanError(getDiscoveryNoConnectedAgentMessage(props.commandsEnabled));
} else {
setScanError(message);
setScanError(
message.includes('no connected agent')
? getDiscoveryNoConnectedAgentMessage(props.commandsEnabled)
: message,
);
} finally {
if (ownsRun()) {
setHttpScanInProgress(false);
setIsScanning(false);
setScanProgress(null);
setScanStartTime(null);
}
setHttpScanInProgress(false);
setIsScanning(false);
setScanProgress(null);
setScanStartTime(null);
}
};
const handleSaveNotes = async () => {
if (disposed || !discoveryFeatureEnabled()) return;
setSaveError(null);
const agentId = targetAgentId();
if (!agentId) {
const context = captureContext();
if (!context.agentId) {
setSaveError('Agent identifier unavailable for discovery');
return;
}
try {
await updateDiscoveryNotes(props.resourceType, agentId, props.resourceId, {
await updateDiscoveryNotes(context.type, context.agentId, context.resourceId, {
user_notes: notesText(),
});
if (!isCurrentContext(context)) return;
setEditingNotes(false);
await refetch();
} catch (err) {
setSaveError(err instanceof Error ? err.message : 'Failed to save notes');
if (isCurrentContext(context)) {
setSaveError(err instanceof Error ? err.message : 'Failed to save notes');
}
}
};
@@ -267,14 +317,19 @@ export function useDiscoveryTabState(props: DiscoveryTabStateProps) {
};
onCleanup(() => {
disposed = true;
contextVersion++;
runVersion++;
clearScanTimers();
clearCopyFeedbackTimer();
});
const handleCopyDiscoveryValue = async (value?: string | null) => {
const text = (value || '').trim();
if (!text) return;
const context = captureContext();
const copied = await copyToClipboard(text);
if (!copied) return;
if (!copied || !isCurrentContext(context)) return;
clearCopyFeedbackTimer();
setCopiedDiscoveryValue(text);
@@ -291,28 +346,56 @@ export function useDiscoveryTabState(props: DiscoveryTabStateProps) {
if (!progress || progress.resource_id !== resourceId()) return;
setScanProgress(progress);
if (
(progress.status === 'completed' || progress.status === 'failed') &&
!httpScanInProgress()
) {
setIsScanning(false);
setTimeout(async () => {
const agentId = targetAgentId();
if (!agentId) return;
try {
const result = await getDiscovery(props.resourceType, agentId, props.resourceId);
if (result) {
mutate(result);
}
} catch (err) {
console.error('Failed to fetch discovery after completion:', err);
}
setScanProgress(null);
}, 500);
if (progress.status === 'running' || progress.status === 'pending') {
if (!httpScanInProgress() && !isScanning()) {
clearScanTimers();
runVersion++;
setScanError(null);
setScanSuccess(false);
const startedAt = Date.parse(progress.started_at || '');
setScanStartTime(Number.isFinite(startedAt) ? startedAt : Date.now());
}
setIsScanning(true);
return;
}
if (progress.status !== 'completed' && progress.status !== 'failed') return;
// The scanner can emit completed-with-error when QGA is paused or no
// command evidence was collected. It is not successful saved Discovery.
// Keep that reason visible even when this tab did not initiate the scan.
const error =
progress.error?.trim() || (progress.status === 'failed' ? 'Discovery scan failed.' : null);
if (error) {
setScanError(error);
setScanSuccess(false);
}
// Scanner completion can precede analysis/persistence. A manual request
// owns its final result until HTTP settles, not an intermediate event.
if (httpScanInProgress()) return;
clearScanTimers();
const version = ++runVersion;
setScanSuccess(false);
setIsScanning(false);
setScanStartTime(null);
setScanProgress(null);
if (error) return;
const context = captureContext();
const ownsRefresh = () => isCurrentContext(context) && version === runVersion;
completionTimer = setTimeout(async () => {
completionTimer = undefined;
if (!context.agentId || !ownsRefresh()) return;
try {
const result = await getDiscovery(context.type, context.agentId, context.resourceId);
if (ownsRefresh()) mutate(result);
} catch (err) {
if (!ownsRefresh()) return;
const status = (err as { status?: number } | null)?.status;
if (status === 401 || status === 403) mutate(null);
console.error('Failed to fetch discovery after completion:', err);
}
}, 500);
});
onCleanup(() => {
@@ -347,6 +430,7 @@ export function useDiscoveryTabState(props: DiscoveryTabStateProps) {
notesText,
mutateDiscovery: mutate,
refetchDiscovery: refetch,
runBlockReason,
saveError,
scanError,
scanProgress,
frontend-modern/src/components/Workloads/GuestDrawer.test.tsx
+55
View File
@@ -220,6 +220,61 @@ afterEach(() => {
// ── Tests ──────────────────────────────────────────────────────────────
describe('GuestDrawer', () => {
it('keeps stored History gaps through live guest recovery without another read', async () => {
const points = makeHistoryPoints(10);
const payload = {
resourceType: 'vm',
resourceId: 'inst1:node1:100',
range: '24h',
start: 1,
end: 3,
source: 'store',
metrics: { cpu: points, memory: [points[0], points[2]] },
};
chartsApiMocks.getMetricsHistory.mockResolvedValue(payload);
const [guest, setGuest] = createSignal(
makeGuest({ lock: 'backup', diskStatusReason: 'prev-vm-locked' }),
);
render(() => <GuestDrawer guest={guest()} onClose={vi.fn()} />);
fireEvent.click(screen.getByRole('tab', { name: 'History' }));
await screen.findByRole('slider', { name: 'Inspect Utilization history' });
const chart = screen.getAllByTestId('guest-history-group-chart')[0];
expect(chart.querySelectorAll('path')).toHaveLength(1);
expect(chart.querySelectorAll('[data-history-observation="memory"]')).toHaveLength(2);
expect(chart.querySelector('[data-history-gaps]')).toHaveTextContent(
'Missing observations: Memory.',
);
setGuest({
...guest(),
lock: '',
diskStatusReason: '',
memory: {
...guest().memory,
usage: 99,
observation: { state: 'current', source: 'status-mem', observedAt: '2026-10-04T12:00:00Z' },
},
});
expect(screen.getAllByTestId('guest-history-group-chart')[0]).toBe(chart);
expect(chart.querySelectorAll('[data-history-observation="memory"]')).toHaveLength(2);
expect(chart).not.toHaveTextContent('99.0%');
expect(chartsApiMocks.getMetricsHistory).toHaveBeenCalledTimes(1);
expect(chartsApiMocks.getMetricsHistory.mock.calls[0][0]).toMatchObject({
resourceType: 'vm',
resourceId: 'inst1:node1:100',
range: '24h',
maxPoints: 240,
});
chartsApiMocks.getMetricsHistory.mockResolvedValue({
...payload,
metrics: { cpu: points, memory: points },
});
fireEvent.click(screen.getByRole('button', { name: 'Refresh history' }));
await waitFor(() => expect(chart.querySelector('[data-history-gaps]')).toBeNull());
expect(chart.querySelectorAll('path')).toHaveLength(2);
expect(chart.querySelector('[data-history-observation="memory"]')).toBeNull();
expect(chartsApiMocks.getMetricsHistory).toHaveBeenCalledTimes(2);
});
it('keeps the backup precaution visible across drawer tabs without starting a guest check', async () => {
const [guest, setGuest] = createSignal(
makeGuest({
frontend-modern/src/components/Workloads/GuestDrawer.tsx
+5
View File
@@ -215,6 +215,11 @@ const GuestDrawerContent: Component<GuestDrawerProps> = (props) => {
hostname={props.guest.name}
canonicalResourceId={props.guest.id}
showManualRunAction
runBlockReason={
guestReadPrecaution()
? 'Discovery is paused during a backup or guest-read deferral. Saved results remain available. Clearing this pause does not prove thaw.'
: null
}
/>
</Suspense>
</div>
frontend-modern/src/components/Workloads/GuestDrawerHistory.tsx
+33 -3
View File
@@ -38,6 +38,7 @@ import {
buildGuestDrawerHistoryPath,
getGuestDrawerHistoryRangeBounds,
getGuestDrawerHistoryScale,
getGuestDrawerHistorySegments,
getGuestDrawerHistoryValueLabel,
normalizeGuestDrawerHistoryPoints,
type GuestDrawerHistoryGroupConfig,
@@ -277,8 +278,30 @@ const GuestDrawerHistoryGroupChart: Component<GuestDrawerHistoryGroupChartProps>
if (count === 0) return inspectionValueText();
return `${count} stored observation ${count === 1 ? 'time' : 'times'}. ${inspectionValueText()}`;
});
const drawableSeries = createMemo(() => series().filter((item) => item.points.length >= 2));
const singlePointSeries = createMemo(() => series().filter((item) => item.points.length === 1));
const segmentedSeries = createMemo(() =>
series().map((item) => ({
...item,
segments: getGuestDrawerHistorySegments(item.points, observationTimes()),
})),
);
const gapMessage = createMemo(() => {
const labels = segmentedSeries()
.filter((item) => item.segments.length > 1)
.map((item) => item.label);
return labels.length > 0
? `Missing observations: ${labels.join(', ')}. Lines stop where a series has no reading at another stored time in this panel. This does not identify the cause or duration of a monitoring outage.`
: null;
});
const drawableSeries = createMemo(() =>
segmentedSeries().flatMap((item) =>
item.segments.filter((points) => points.length >= 2).map((points) => ({ ...item, points })),
),
);
const singlePointSeries = createMemo(() =>
segmentedSeries().flatMap((item) =>
item.segments.filter((points) => points.length === 1).map((points) => ({ ...item, points })),
),
);
const singleObservation = createMemo(() =>
observationTimes().length === 1 ? { timestamp: observationTimes()[0] } : null,
);
@@ -443,7 +466,7 @@ const GuestDrawerHistoryGroupChart: Component<GuestDrawerHistoryGroupChartProps>
</For>
<svg
aria-label={`${props.group.label} history`}
aria-describedby={`${inspectionId}-description`}
aria-describedby={`${inspectionId}-description${gapMessage() ? ` ${inspectionId}-gaps` : ''}`}
class="absolute inset-0 h-full w-full cursor-crosshair"
data-testid="guest-history-plot"
onMouseMove={handleHoverMove}
@@ -585,6 +608,13 @@ const GuestDrawerHistoryGroupChart: Component<GuestDrawerHistoryGroupChartProps>
<p id={`${inspectionId}-description`} class="sr-only">
{chartDescription()}
</p>
<Show when={gapMessage()}>
{(message) => (
<p id={`${inspectionId}-gaps`} class="mt-2 text-xs text-muted" data-history-gaps>
{message()}
</p>
)}
</Show>
<Show when={!props.loading && singleObservation()}>
{(observation) => (
<p class="mt-2 text-[10px] text-muted">
frontend-modern/src/components/Workloads/GuestDrawerOverview.tsx
+15 -5
View File
@@ -138,12 +138,22 @@ export function GuestDrawerOverview(props: GuestDrawerOverviewProps) {
props.guest.vmware?.datacenterName ||
props.guest.vmware?.clusterName,
);
const diskReason = isGuestDrawerVM(props.guest) ? props.guest.diskStatusReason : undefined;
const diskReadState =
props.guest.telemetryAvailability?.disk === false
? 'unavailable'
: diskReason
? diskReason.startsWith('prev-')
? 'last-known'
: 'unavailable'
: 'current';
const diskRows = (props.guest.disks ?? []).map((disk, index) =>
buildWorkloadsDiskPresentation(disk, index, props.diskThresholds),
buildWorkloadsDiskPresentation(disk, index, props.diskThresholds, diskReadState),
);
const diskStatusMessage =
isGuestDrawerVM(props.guest) && props.guest.diskStatusReason
? getWorkloadGuestDiskStatusMessage(props.guest.diskStatusReason)
const diskStatusMessage = diskReason
? getWorkloadGuestDiskStatusMessage(diskReason)
: diskReadState === 'unavailable'
? 'Filesystem usage is unavailable.'
: null;
return compactDetailSections([
@@ -217,6 +227,7 @@ export function GuestDrawerOverview(props: GuestDrawerOverviewProps) {
? {
label: 'Filesystems',
rows: compactDetailRows([
makeDetailRow('Status', diskStatusMessage, { layout: 'stacked', wrap: true }),
...diskRows.map((disk) =>
makeDetailRow(
disk.label,
@@ -238,7 +249,6 @@ export function GuestDrawerOverview(props: GuestDrawerOverviewProps) {
},
),
),
makeDetailRow('Status', diskStatusMessage, { layout: 'stacked', wrap: true }),
]),
}
: null,
frontend-modern/src/components/Workloads/__tests__/DiskList.test.tsx
+32
View File
@@ -28,6 +28,38 @@ afterEach(() => {
cleanup();
});
describe('filesystem presentation read admission', () => {
it.each(['last-known', 'unavailable'] as const)(
'keeps %s readings out of threshold-coloured utilization',
(state) => {
const disk = makeDisk();
const reading = buildWorkloadsDiskPresentation(disk, 0, undefined, state);
expect(reading.progressValue).toBeNull();
expect(reading.progressWidth).toBe('0%');
expect(reading.progressClass).toBe('bg-surface-hover');
expect(reading.label).toBe('/');
expect(reading.typeLabel).toBe('EXT4');
expect(reading.usagePercentLabel).toBe(
state === 'last-known' ? 'Last known 50%' : 'Usage unavailable',
);
expect(reading.usageText).toBe(state === 'last-known' ? '50.0 GB/100 GB' : '?/100 GB');
expect(disk.used).toBe(53687091200);
},
);
it('does not infer a retained value from capacity or free bytes', () => {
const reading = buildWorkloadsDiskPresentation(
makeDisk({ used: undefined }),
0,
undefined,
'last-known',
);
expect(reading.usagePercentLabel).toBe('Usage unavailable');
expect(reading.usageText).toBe('?/100 GB');
expect(reading.progressValue).toBeNull();
});
});
describe('DiskList', () => {
describe('fallback when no disks', () => {
it('renders fallback "-" when disks array is empty', () => {
frontend-modern/src/components/Workloads/__tests__/GuestDrawer.discovery-safety.test.tsx
+161
View File
@@ -0,0 +1,161 @@
import { createSignal, Suspense } from 'solid-js';
import { cleanup, fireEvent, render, screen, waitFor } from '@solidjs/testing-library';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import type { WorkloadGuest } from '@/types/workloads';
import { GuestDrawer } from '../GuestDrawer';
import { guestDiskDeferrals } from '../__fixtures__/guestDiskDeferrals';
import { resetAIRuntimeState, syncAIRuntimeSettings } from '@/stores/aiRuntimeState';
import { resetCreateNonSuspendingQueryCacheForTest } from '@/hooks/createNonSuspendingQuery';
import * as discoveryApi from '@/api/discovery';
vi.mock('@/api/discovery', async (importOriginal) => ({
...(await importOriginal<typeof import('@/api/discovery')>()),
getDiscovery: vi.fn(async () => null),
getDiscoveryInfo: vi.fn(async () => ({
ai_provider: { provider: 'test', model: 'test', is_local: true, label: 'Local' },
commands: [],
command_categories: [],
})),
getConnectedAgents: vi.fn(async () => ({ count: 1, agents: [{ agent_id: 'node-agent' }] })),
triggerDiscovery: vi.fn(async () => null),
}));
vi.mock('@/api/charts', async (importOriginal) => ({
...(await importOriginal<typeof import('@/api/charts')>()),
ChartsAPI: {
getMetricsHistory: vi.fn(async () => ({ metrics: {}, start: 1, end: 2, source: 'store' })),
},
}));
vi.mock('@/stores/license', () => ({
isRangeLocked: () => false,
loadRuntimeCapabilities: vi.fn(),
maxHistoryDays: () => 90,
}));
const guest = (patch: Partial<WorkloadGuest> = {}): WorkloadGuest =>
({
id: 'inst:node:100',
instance: 'inst',
node: 'node',
vmid: 100,
name: 'Backup guest',
type: 'qemu',
status: 'running',
cpu: 0.25,
cpus: 2,
memory: { total: 1024, used: 512, free: 512, usage: 50 },
disk: { total: 1024, used: 512, free: 512, usage: 50 },
networkIn: 1,
networkOut: 2,
diskRead: 3,
diskWrite: 4,
uptime: 100,
template: false,
lastBackup: 0,
tags: [],
lock: '',
lastSeen: '2026-10-04T12:00:00Z',
guestAgentStatus: 'available',
discoveryTarget: { resourceType: 'vm', agentId: 'node-agent', resourceId: '100' },
...patch,
}) as WorkloadGuest;
beforeEach(() => {
resetAIRuntimeState();
syncAIRuntimeSettings({ discovery_enabled: true } as Parameters<typeof syncAIRuntimeSettings>[0]);
resetCreateNonSuspendingQueryCacheForTest();
});
afterEach(() => {
cleanup();
resetAIRuntimeState();
vi.clearAllMocks();
});
describe('guest drawer and real Discovery safety wiring', () => {
it.each([...guestDiskDeferrals.flatMap(([reason]) => [reason, `prev-${reason}`])])(
'does not offer a live scan for %s, even with an assigned node agent and available metadata',
async (reason) => {
render(() => (
<Suspense>
<GuestDrawer guest={guest({ diskStatusReason: reason })} onClose={vi.fn()} />
</Suspense>
));
fireEvent.click(await screen.findByRole('tab', { name: 'Discovery' }));
const button = await screen.findByRole('button', { name: 'Run Discovery' });
await waitFor(() => expect(button).toBeDisabled());
fireEvent.click(button);
expect(discoveryApi.triggerDiscovery).not.toHaveBeenCalled();
expect(screen.getByTestId('guest-read-precaution')).toBeVisible();
},
);
it.each([{ lock: 'backup' }, { backupInProgress: true }, { guestAgentStatus: 'deferred' }])(
'guards the native operation signal %j without fabricating a disk read reason',
async (patch) => {
render(() => (
<Suspense>
<GuestDrawer guest={guest(patch)} onClose={vi.fn()} />
</Suspense>
));
fireEvent.click(await screen.findByRole('tab', { name: 'Discovery' }));
await waitFor(() =>
expect(screen.getByRole('button', { name: 'Run Discovery' })).toBeDisabled(),
);
expect(discoveryApi.triggerDiscovery).not.toHaveBeenCalled();
},
);
it('updates the existing same-guest button without remounting the tab or automatically scanning', async () => {
const [value, setValue] = createSignal(guest({ lock: 'backup' }));
render(() => (
<Suspense>
<GuestDrawer guest={value()} onClose={vi.fn()} />
</Suspense>
));
fireEvent.click(await screen.findByRole('tab', { name: 'Discovery' }));
const button = await screen.findByRole('button', { name: 'Run Discovery' });
await waitFor(() => expect(button).toBeDisabled());
setValue(guest());
await waitFor(() => expect(button).toBeEnabled());
expect(screen.getByRole('button', { name: 'Run Discovery' })).toBe(button);
expect(screen.getByRole('tab', { name: 'Discovery' })).toHaveAttribute('aria-selected', 'true');
expect(discoveryApi.triggerDiscovery).not.toHaveBeenCalled();
setValue(guest({ diskStatusReason: 'prev-agent-timeout' }));
await waitFor(() => expect(button).toBeDisabled());
expect(discoveryApi.triggerDiscovery).not.toHaveBeenCalled();
});
it.each([
{ type: 'qemu' },
{
type: 'lxc',
lock: 'backup',
diskStatusReason: 'prev-agent-timeout',
discoveryTarget: {
resourceType: 'system-container',
agentId: 'node-agent',
resourceId: '100',
},
},
{
type: 'vm',
platformType: 'vmware',
guestAgentStatus: '',
lock: '',
discoveryTarget: { resourceType: 'vm', agentId: 'node-agent', resourceId: '100' },
},
] as const)(
'preserves ordinary manual scans outside the Proxmox guest safety condition %j',
async (patch) => {
render(() => (
<Suspense>
<GuestDrawer guest={guest(patch)} onClose={vi.fn()} />
</Suspense>
));
fireEvent.click(await screen.findByRole('tab', { name: 'Discovery' }));
const button = await screen.findByRole('button', { name: 'Run Discovery' });
await waitFor(() => expect(button).toBeEnabled());
fireEvent.click(button);
await waitFor(() => expect(discoveryApi.triggerDiscovery).toHaveBeenCalledTimes(1));
},
);
});
frontend-modern/src/components/Workloads/__tests__/GuestDrawerHistory.gaps.test.tsx
+245
View File
@@ -0,0 +1,245 @@
import { cleanup, fireEvent, render, screen, waitFor, within } from '@solidjs/testing-library';
import { createSignal } from 'solid-js';
import { afterEach, describe, expect, it, vi } from 'vitest';
import { ChartsAPI, type AllMetricsHistoryResponse } from '@/api/charts';
import { resetCreateNonSuspendingQueryCacheForTest } from '@/hooks/createNonSuspendingQuery';
import { GuestDrawerHistory } from '../GuestDrawerHistory';
vi.mock('@/stores/license', () => ({
loadRuntimeCapabilities: vi.fn(async () => undefined),
maxHistoryDays: () => 90,
isRangeLocked: () => false,
}));
afterEach(() => {
cleanup();
resetCreateNonSuspendingQueryCacheForTest();
vi.restoreAllMocks();
});
const start = Date.UTC(2026, 9, 4, 12);
const target = { resourceType: 'vm' as const, resourceId: 'fixture:pve:101' };
const point = (minute: number, value: number) => ({
timestamp: start + minute * 60_000,
value,
min: value,
max: value,
});
const response = (
metrics: AllMetricsHistoryResponse['metrics'] = {
cpu: [0, 1, 2, 3, 4].map((minute) => point(minute, 10)),
memory: [point(0, 20), point(1, 25), point(4, 30)],
disk: [point(0, 50), point(4, 50)],
},
): AllMetricsHistoryResponse => ({
...target,
range: '1h',
start,
end: start + 60 * 60_000,
source: 'store',
metrics,
});
const group = (id = 'utilization') =>
screen.getAllByTestId('guest-history-group-chart').find((el) => el.dataset.historyGroup === id)!;
const paths = (colour: string, id = 'utilization') => [
...group(id).querySelectorAll(`path[stroke="${colour}"]`),
];
const marks = (metric: string, id = 'utilization') =>
group(id).querySelectorAll(`[data-history-observation="${metric}"]`);
const note = (id = 'utilization') => group(id).querySelector('[data-history-gaps]');
const mount = () => render(() => <GuestDrawerHistory target={target} range="1h" />);
const refresh = () => fireEvent.click(screen.getByRole('button', { name: 'Refresh history' }));
describe('drawer History does not join missing stored observations', () => {
it('breaks only the affected metric and keeps isolated recovery observations visible', async () => {
const fetch = vi.spyOn(ChartsAPI, 'getMetricsHistory').mockResolvedValue(response());
mount();
await screen.findByRole('slider', { name: 'Inspect Utilization history' });
expect(paths('#8b5cf6')[0].getAttribute('d')?.match(/L/g)).toHaveLength(4);
expect(paths('#f59e0b')).toHaveLength(1);
expect(paths('#f59e0b')[0].getAttribute('d')?.match(/L/g)).toHaveLength(1);
expect(marks('memory')).toHaveLength(1);
expect(paths('#10b981')).toHaveLength(0);
expect(marks('disk')).toHaveLength(2);
expect(note()).toHaveTextContent('Missing observations: Memory, Disk.');
expect(note()).toHaveTextContent(
'Lines stop where a series has no reading at another stored time in this panel.',
);
expect(fetch).toHaveBeenCalledTimes(1);
});
it('retains isolated measured zero, not an invented zero during a gap', async () => {
vi.spyOn(ChartsAPI, 'getMetricsHistory').mockResolvedValue(
response({
cpu: [point(0, 0), point(1, 1), point(2, 2)],
memory: [point(0, 0), point(2, 0)],
}),
);
mount();
await screen.findByRole('slider', { name: 'Inspect Utilization history' });
expect(paths('#f59e0b')).toHaveLength(0);
expect(marks('memory')).toHaveLength(2);
const slider = within(group()).getByRole('slider');
fireEvent.input(slider, { target: { value: '1' } });
expect(slider).toHaveAttribute(
'aria-valuetext',
expect.stringContaining('Memory no observation.'),
);
expect(group()).toHaveTextContent('Memory-');
fireEvent.input(slider, { target: { value: '2' } });
expect(group()).toHaveTextContent('Memory0.0%');
});
it('does not fabricate a trend for interleaved, independently sampled series', async () => {
vi.spyOn(ChartsAPI, 'getMetricsHistory').mockResolvedValue(
response({
cpu: [point(0, 10), point(2, 20)],
memory: [point(1, 30), point(3, 40)],
}),
);
mount();
await screen.findByRole('slider', { name: 'Inspect Utilization history' });
expect(group().querySelectorAll('path')).toHaveLength(0);
expect(marks('cpu')).toHaveLength(2);
expect(marks('memory')).toHaveLength(2);
expect(note()).toHaveTextContent('Missing observations: CPU, Memory.');
});
it('preserves multiple continuous segments without dropping either side of the gap', async () => {
vi.spyOn(ChartsAPI, 'getMetricsHistory').mockResolvedValue(
response({
cpu: [0, 1, 2, 3, 4].map((minute) => point(minute, 10)),
memory: [0, 1, 3, 4].map((minute) => point(minute, 20)),
}),
);
mount();
await screen.findByRole('slider', { name: 'Inspect Utilization history' });
expect(paths('#f59e0b')).toHaveLength(2);
for (const path of paths('#f59e0b'))
expect(path.getAttribute('d')?.match(/L/g)).toHaveLength(1);
expect(marks('memory')).toHaveLength(0);
expect(within(group()).getByRole('img')).toHaveAccessibleDescription(
expect.stringContaining('Missing observations: Memory.'),
);
});
it('applies the same rule to network and disk I/O, retaining valid zero rates', async () => {
vi.spyOn(ChartsAPI, 'getMetricsHistory').mockResolvedValue(
response({
netin: [point(0, 0), point(1, 100), point(2, 200)],
netout: [point(0, 0), point(2, 0)],
diskread: [point(0, 0), point(2, 0)],
diskwrite: [point(0, 0), point(1, 50), point(2, 100)],
}),
);
mount();
await screen.findByRole('slider', { name: 'Inspect Network I/O history' });
expect(paths('#10b981', 'network')).toHaveLength(1);
expect(paths('#fb923c', 'network')).toHaveLength(0);
expect(marks('netout', 'network')).toHaveLength(2);
expect(note('network')).toHaveTextContent('Missing observations: Out.');
expect(paths('#3b82f6', 'disk-io')).toHaveLength(0);
expect(marks('diskread', 'disk-io')).toHaveLength(2);
expect(paths('#f59e0b', 'disk-io')).toHaveLength(1);
expect(note('disk-io')).toHaveTextContent('Missing observations: Read.');
});
it('does not guess missed polls from elapsed time or timestamps in other panels', async () => {
vi.spyOn(ChartsAPI, 'getMetricsHistory').mockResolvedValue(
response({
cpu: [point(0, 10), point(59, 20)],
memory: [point(0, 30), point(59, 40)],
netin: [point(20, 100)],
unconfigured: [point(30, 100)],
}),
);
mount();
await screen.findByRole('slider', { name: 'Inspect Utilization history' });
expect(paths('#8b5cf6')).toHaveLength(1);
expect(paths('#f59e0b')).toHaveLength(1);
expect(note()).toBeNull();
});
it('does not create a gap from invalid samples or an entirely absent series', async () => {
vi.spyOn(ChartsAPI, 'getMetricsHistory').mockResolvedValue(
response({
cpu: [point(0, 10), point(1, NaN), point(2, 20)],
memory: [point(0, 30), point(2, 40)],
}),
);
mount();
await screen.findByRole('slider', { name: 'Inspect Utilization history' });
expect(paths('#8b5cf6')).toHaveLength(1);
expect(paths('#f59e0b')).toHaveLength(1);
expect(note()).toBeNull();
expect(marks('disk')).toHaveLength(0);
});
it('cannot close a stored gap with live recovery or retained numeric carriers', async () => {
const fetch = vi.spyOn(ChartsAPI, 'getMetricsHistory').mockResolvedValue(response());
const [live, setLive] = createSignal(false);
render(() => (
<GuestDrawerHistory
target={target}
range="1h"
currentMetrics={live() ? { memory: 99, disk: 99 } : {}}
deferredMetrics={live() ? {} : { memory: { lastKnownValue: 25, message: 'Last known.' } }}
/>
));
await screen.findByRole('slider', { name: 'Inspect Utilization history' });
const chart = group();
expect(marks('memory')).toHaveLength(1);
setLive(true);
expect(group()).toBe(chart);
expect(marks('memory')).toHaveLength(1);
expect(paths('#10b981')).toHaveLength(0);
expect(note()).toHaveTextContent('Memory, Disk');
expect(group()).not.toHaveTextContent('99.0%');
expect(fetch).toHaveBeenCalledTimes(1);
});
it('closes a gap only with matching stored observations and preserves dated inspection', async () => {
const fetch = vi
.spyOn(ChartsAPI, 'getMetricsHistory')
.mockResolvedValueOnce(response())
.mockResolvedValueOnce(
response({
cpu: [0, 1, 2, 3, 4].map((minute) => point(minute, 10)),
memory: [0, 1, 2, 3, 4].map((minute) => point(minute, 20)),
}),
);
mount();
await screen.findByRole('slider', { name: 'Inspect Utilization history' });
const chart = group();
const slider = within(chart).getByRole('slider');
fireEvent.input(slider, { target: { value: '2' } });
expect(slider).toHaveAttribute(
'aria-valuetext',
expect.stringContaining('Memory no observation.'),
);
refresh();
await waitFor(() => expect(note()).toBeNull());
expect(group()).toBe(chart);
expect(paths('#f59e0b')).toHaveLength(1);
expect(paths('#f59e0b')[0].getAttribute('d')?.match(/L/g)).toHaveLength(4);
expect(slider).toHaveValue('2');
expect(slider).toHaveAttribute('aria-valuetext', expect.stringContaining('Memory 20.0%.'));
expect(fetch).toHaveBeenCalledTimes(2);
});
it('retains gaps on a transient refresh failure, but withdraws them on access denial', async () => {
vi.spyOn(ChartsAPI, 'getMetricsHistory')
.mockResolvedValueOnce(response())
.mockRejectedValueOnce(Object.assign(new Error('private transient detail'), { status: 503 }))
.mockRejectedValueOnce(Object.assign(new Error('private denial'), { status: 403 }));
mount();
await screen.findByRole('slider', { name: 'Inspect Utilization history' });
refresh();
await screen.findByRole('button', { name: 'Retry history' });
expect(note()).toHaveTextContent('Memory, Disk');
expect(paths('#10b981')).toHaveLength(0);
fireEvent.click(screen.getByRole('button', { name: 'Retry history' }));
await waitFor(() => expect(screen.queryByTestId('guest-history-group-chart')).toBeNull());
expect(document.querySelector('[data-history-gaps]')).toBeNull();
expect(document.body).not.toHaveTextContent(/private transient detail|private denial/);
});
});
frontend-modern/src/components/Workloads/__tests__/GuestDrawerHistory.pointer.test.tsx
+3 -2
View File
@@ -84,7 +84,7 @@ describe('GuestDrawerHistory common-time pointer inspection', () => {
expect(
within(utilization()).getByRole('img', { name: 'Utilization history' }),
).toHaveAccessibleDescription(
`${new Date(time).toLocaleString()}. CPU 11.0%. Memory no observation. Disk no observation.`,
`${new Date(time).toLocaleString()}. CPU 11.0%. Memory no observation. Disk no observation. Missing observations: CPU, Memory. Lines stop where a series has no reading at another stored time in this panel. This does not identify the cause or duration of a monitoring outage.`,
);
fireEvent.pointerLeave(plot);
expect(utilization()).toHaveTextContent('CPU43.0%');
@@ -101,7 +101,8 @@ describe('GuestDrawerHistory common-time pointer inspection', () => {
expect(utilization()).toHaveTextContent('Memory-');
expect(utilization()).toHaveTextContent('Disk0.0%');
expect(utilization().querySelectorAll('circle[r="3"]')).toHaveLength(1);
expect(utilization().querySelectorAll('path')).toHaveLength(2);
expect(utilization().querySelectorAll('path')).toHaveLength(1);
expect(utilization().querySelectorAll('[data-history-observation]')).toHaveLength(4);
});
it('renders all metrics that really share a timestamp at the same cursor position', async () => {
frontend-modern/src/components/Workloads/__tests__/GuestDrawerHistory.sparse.test.tsx
+6 -8
View File
@@ -165,7 +165,7 @@ describe('GuestDrawerHistory sparse observations and current provenance', () =>
},
);
it('draws a lone series alongside a trend and labels only the live fallback', async () => {
it('preserves interleaved lone observations and labels only the live fallback', async () => {
vi.spyOn(ChartsAPI, 'getMetricsHistory').mockResolvedValue(
response({
cpu: [point(start, 10), point(end, 20)],
@@ -173,12 +173,10 @@ describe('GuestDrawerHistory sparse observations and current provenance', () =>
}),
);
mount({ cpu: 99, memory: 98, disk: 88 });
await waitFor(() => expect(dots()).toHaveLength(1));
expect(group().querySelectorAll('path')).toHaveLength(1);
expect(group().querySelector('[data-history-observation]')).toHaveAttribute(
'data-history-observation',
'memory',
);
await waitFor(() => expect(dots()).toHaveLength(3));
expect(group().querySelectorAll('path')).toHaveLength(0);
expect(group().querySelectorAll('[data-history-observation="memory"]')).toHaveLength(1);
expect(group().querySelectorAll('[data-history-observation="cpu"]')).toHaveLength(2);
expect(group().querySelectorAll('[data-history-current]')).toHaveLength(1);
expect(group().querySelector('[data-history-current="disk"]')).toHaveTextContent(
'Disk88.0%current',
@@ -191,7 +189,7 @@ describe('GuestDrawerHistory sparse observations and current provenance', () =>
expect(group()).toHaveTextContent('Memory0.0%');
expect(group()).toHaveTextContent('Disk-');
expect(group().querySelector('[data-history-current]')).toBeNull();
expect(dots()).toHaveLength(1);
expect(dots()).toHaveLength(3);
});
it('does not connect single observations from different series or borrow their times', async () => {
frontend-modern/src/components/Workloads/__tests__/GuestDrawerOverview.filesystem-provenance.test.tsx
+172
View File
@@ -0,0 +1,172 @@
import { cleanup, render, screen, within } from '@solidjs/testing-library';
import { createSignal, type ComponentProps } from 'solid-js';
import { afterEach, describe, expect, it, vi } from 'vitest';
import { GuestDrawerOverview } from '../GuestDrawerOverview';
import { guestDiskDeferrals } from '../__fixtures__/guestDiskDeferrals';
import type { WorkloadGuest } from '@/types/workloads';
vi.mock('../GuestPhysicalDisks', () => ({ GuestPhysicalDisks: () => null }));
const path = '/srv/backups/covered-filesystem';
const unavailableDisk = {
cpu: true,
memory: true,
disk: false,
networkIO: true,
diskIO: true,
uptime: true,
} satisfies NonNullable<WorkloadGuest['telemetryAvailability']>;
const disk = {
mountpoint: path,
device: '/dev/vda1',
type: 'ext4',
total: 10 * 1024 ** 3,
used: 9 * 1024 ** 3,
usage: 90,
};
const guest = (overrides: Partial<WorkloadGuest> = {}): WorkloadGuest => ({
id: 'lab:pve-a:101',
vmid: 101,
name: 'backup-guest',
node: 'pve-a',
instance: 'lab',
status: 'running',
type: 'qemu',
cpu: 0.1,
cpus: 2,
memory: { total: 1024 ** 3, used: 256 * 1024 ** 2, free: 768 * 1024 ** 2, usage: 25 },
disk,
disks: [disk],
networkIn: 0,
networkOut: 0,
diskRead: 0,
diskWrite: 0,
uptime: 3600,
template: false,
tags: [],
lock: '',
lastBackup: 0,
lastSeen: '2026-10-04T21:27:00Z',
...overrides,
});
const props = (value: WorkloadGuest): ComponentProps<typeof GuestDrawerOverview> => ({
guest: value,
guestOsSummary: 'Linux',
agentHeading: 'Guest agent',
agentLabel: 'QEMU 9.2',
agentTitle: 'Previously observed agent version',
hasAgentInfo: true,
hasFilesystemDetails: true,
hasNetworkInterfaces: false,
hasOsInfo: true,
hasWorkloadActionAgent: false,
showInGuestAgentInstallCue: false,
ipAddresses: [],
networkInterfaces: [],
normalizedTags: [],
backupPresentation: null,
workloadActionAgentTitle: '',
});
const filesystemRow = () => screen.getByText(path).closest('tr')!;
afterEach(cleanup);
describe('filesystem details do not promote retained usage to current health', () => {
it.each(guestDiskDeferrals)(
'keeps prev-%s values labelled, without a current utilization bar',
(reason) => {
render(() => (
<GuestDrawerOverview {...props(guest({ diskStatusReason: `prev-${reason}` }))} />
));
const row = within(filesystemRow());
expect(row.getByText('Last known 90% · 9.00 GB/10.0 GB · EXT4')).toBeVisible();
expect(row.queryByRole('progressbar')).not.toBeInTheDocument();
expect(screen.getByText(/^Using last known disk stats\./)).toBeVisible();
},
);
it.each([
...guestDiskDeferrals.map(([reason]) => reason),
'permission-denied',
'future-private-reason',
])('declines unqualified numeric usage for %s', (reason) => {
render(() => <GuestDrawerOverview {...props(guest({ diskStatusReason: reason }))} />);
const row = within(filesystemRow());
expect(row.getByText('Usage unavailable · ?/10.0 GB · EXT4')).toBeVisible();
expect(row.queryByRole('progressbar')).not.toBeInTheDocument();
expect(row.queryByText(/90%|9\.00 GB/)).not.toBeInTheDocument();
expect(screen.queryByText('future-private-reason')).not.toBeInTheDocument();
});
it('does not fabricate retained usage from capacity, negative or missing bytes', () => {
const [value, setValue] = createSignal(
guest({ diskStatusReason: 'prev-vm-locked', disks: [{ ...disk, used: undefined }] }),
);
render(() => <GuestDrawerOverview {...props(value())} />);
for (const input of [undefined, -1, NaN]) {
setValue({ ...value(), disks: [{ ...disk, used: input }] });
expect(
within(filesystemRow()).getByText('Usage unavailable · ?/10.0 GB · EXT4'),
).toBeVisible();
expect(within(filesystemRow()).queryByRole('progressbar')).not.toBeInTheDocument();
}
});
it('withdraws a numeric bar on explicit unavailability even without a reason', () => {
render(() => (
<GuestDrawerOverview {...props(guest({ telemetryAvailability: unavailableDisk }))} />
));
expect(within(filesystemRow()).getByText('Usage unavailable · ?/10.0 GB · EXT4')).toBeVisible();
expect(within(filesystemRow()).queryByRole('progressbar')).not.toBeInTheDocument();
});
it('explains explicit unavailability even before any filesystem sample', () => {
render(() => (
<GuestDrawerOverview
{...props(guest({ disks: [], telemetryAvailability: unavailableDisk }))}
hasFilesystemDetails={false}
/>
));
expect(screen.getByText('Filesystems')).toBeVisible();
expect(screen.getByText('Filesystem usage is unavailable.')).toBeVisible();
expect(screen.queryByRole('progressbar')).not.toBeInTheDocument();
});
it('keeps retained values until the reason clears, then resumes the same drawer', () => {
const [value, setValue] = createSignal(
guest({ diskStatusReason: 'prev-vm-locked', lock: 'backup', backupInProgress: true }),
);
render(() => <GuestDrawerOverview {...props(value())} />);
const initialDrawer = screen.getByTestId('guest-technical-details');
setValue({ ...value(), lock: '', backupInProgress: false });
expect(within(filesystemRow()).getByText(/^Last known 90%/)).toBeVisible();
expect(within(filesystemRow()).queryByRole('progressbar')).not.toBeInTheDocument();
setValue({
...value(),
diskStatusReason: '',
disks: [{ ...disk, used: 3 * 1024 ** 3, usage: 30 }],
});
expect(screen.getByTestId('guest-technical-details')).toBe(initialDrawer);
expect(within(filesystemRow()).getByText('30% · 3.00 GB/10.0 GB · EXT4')).toBeVisible();
expect(
within(filesystemRow()).getByRole('progressbar', { name: `Filesystem ${path} utilization` }),
).toHaveAttribute('aria-valuenow', '30');
expect(screen.queryByText(/Using last known/)).not.toBeInTheDocument();
});
it.each(['qemu', 'lxc'])('preserves fresh %s evidence with no deferral', (type) => {
render(() => <GuestDrawerOverview {...props(guest({ type }))} />);
expect(within(filesystemRow()).getByText('90% · 9.00 GB/10.0 GB · EXT4')).toBeVisible();
expect(within(filesystemRow()).getByRole('progressbar')).toHaveAttribute('aria-valuenow', '90');
expect(
within(filesystemRow()).queryByText(/Last known|Usage unavailable/),
).not.toBeInTheDocument();
});
it('does not attach VM deferral semantics to an independent LXC reading', () => {
render(() => (
<GuestDrawerOverview {...props(guest({ type: 'lxc', diskStatusReason: 'prev-vm-locked' }))} />
));
expect(within(filesystemRow()).getByText('90% · 9.00 GB/10.0 GB · EXT4')).toBeVisible();
expect(within(filesystemRow()).getByRole('progressbar')).toHaveAttribute('aria-valuenow', '90');
});
});
frontend-modern/src/components/Workloads/__tests__/GuestDrawerOverview.filesystems.test.tsx
+3
View File
@@ -85,6 +85,9 @@ describe('GuestDrawerOverview filesystem labels', () => {
).toHaveAttribute('colspan', '2');
expect(screen.queryByText('prev-vm-locked')).not.toBeInTheDocument();
for (const path of paths) expect(screen.getByText(path)).toBeInTheDocument();
expect(screen.getByText('Last known 50% · 5.00 GB/10.0 GB · MP0')).toBeVisible();
expect(screen.getByText('Usage unavailable · ?/10.0 GB · MP1')).toBeVisible();
expect(screen.queryByRole('progressbar')).not.toBeInTheDocument();
setGuest({
...guest(),
diskStatusReason: '',
frontend-modern/src/components/Workloads/__tests__/guestHistorySegments.test.ts
+45
View File
@@ -0,0 +1,45 @@
import { describe, expect, it } from 'vitest';
import { getGuestDrawerHistorySegments } from '../guestDrawerModel';
const point = (timestamp: number, value = 0) => ({ timestamp, value, min: value, max: value });
describe('stored History segments', () => {
it('retains real points and both sides of each missing in-panel time', () => {
const points = Object.freeze([point(1), point(2), point(4), point(6), point(7)]);
const times = Object.freeze([1, 2, 3, 4, 5, 6, 7]);
const segments = getGuestDrawerHistorySegments(points, times);
expect(segments).toEqual([[points[0], points[1]], [points[2]], [points[3], points[4]]]);
expect(segments.flat()).toEqual(points);
expect(segments[1][0]).toBe(points[2]);
});
it('keeps duplicate-time readings in the same segment, without creating a new time', () => {
const points = [point(1, 0), point(1, 10), point(2, 20), point(4, 30)];
expect(getGuestDrawerHistorySegments(points, [1, 2, 3, 4])).toEqual([
points.slice(0, 3),
[points[3]],
]);
});
it('cannot guess gaps from elapsed time when all stored times match', () => {
const points = [point(1), point(1_000_000)];
expect(getGuestDrawerHistorySegments(points, [1, 1_000_000])).toEqual([points]);
});
it('keeps an empty series empty and never accepts a point outside the group times', () => {
expect(getGuestDrawerHistorySegments([], [1, 2, 3])).toEqual([]);
expect(getGuestDrawerHistorySegments([point(1)], [])).toEqual([]);
expect(getGuestDrawerHistorySegments([point(1), point(2), point(3)], [1, 3])).toEqual([
[point(1), point(3)],
]);
});
it('preserves every point at the existing request bound', () => {
const times = Array.from({ length: 240 }, (_, index) => index);
const points = times.filter((time) => time % 2 === 0).map((time) => point(time));
const segments = getGuestDrawerHistorySegments(points, times);
expect(segments).toHaveLength(120);
expect(segments.every((segment) => segment.length === 1)).toBe(true);
expect(segments.flat()).toEqual(points);
});
});
frontend-modern/src/components/Workloads/diskListModel.ts
+18 -5
View File
@@ -92,18 +92,31 @@ export const buildWorkloadsDiskPresentation = (
disk: Disk,
index: number,
thresholds?: MetricDisplayThresholds | null,
readState: 'current' | 'last-known' | 'unavailable' = 'current',
): WorkloadsDiskPresentation => {
const label = getWorkloadsDiskLabel(disk);
// Retention preserves identity and capacity, not a current utilization
// measurement. Only an explicitly retained read may display old used bytes.
const reading = readState === 'unavailable' ? { ...disk, used: undefined } : disk;
const percentLabel = getWorkloadsDiskUsagePercentLabel(reading);
return {
key: `${disk.mountpoint ?? ''}:${disk.device ?? ''}:${index}`,
label,
labelTitle: getWorkloadsDiskLabelTitle(label),
progressClass: getWorkloadsDiskProgressClass(disk, thresholds),
progressValue: getWorkloadsDiskProgressValue(disk),
progressWidth: getWorkloadsDiskProgressWidth(disk),
progressClass:
readState === 'current'
? getWorkloadsDiskProgressClass(reading, thresholds)
: 'bg-surface-hover',
progressValue: readState === 'current' ? getWorkloadsDiskProgressValue(reading) : null,
progressWidth: readState === 'current' ? getWorkloadsDiskProgressWidth(reading) : '0%',
typeLabel: getWorkloadsDiskTypeLabel(disk),
usageText: getWorkloadsDiskUsageText(disk),
usagePercentLabel: getWorkloadsDiskUsagePercentLabel(disk),
usageText: getWorkloadsDiskUsageText(reading),
usagePercentLabel:
readState === 'current'
? percentLabel
: percentLabel === '—'
? 'Usage unavailable'
: `Last known ${percentLabel}`,
};
};
frontend-modern/src/components/Workloads/guestDrawerModel.ts
+23
View File
@@ -355,6 +355,29 @@ export const getGuestDrawerHistoryScale = (
return { minValue: 0, maxValue: Math.max(1, maxValue * 1.15) };
};
// Both inputs are normalized stored observations, ordered by timestamp. A
// series cannot span a time where another series in its panel has a reading
// and this one does not. Do not infer failed polls or cadence from elapsed
// time: the API supplies neither. Keep singleton segments as real evidence.
export const getGuestDrawerHistorySegments = (
points: readonly AggregatedMetricPoint[],
observationTimes: readonly number[],
): AggregatedMetricPoint[][] => {
const timeIndices = new Map(observationTimes.map((timestamp, index) => [timestamp, index]));
const segments: AggregatedMetricPoint[][] = [];
let previousIndex: number | undefined;
for (const point of points) {
const index = timeIndices.get(point.timestamp);
if (index === undefined) continue;
if (previousIndex === undefined || (index !== previousIndex && index !== previousIndex + 1)) {
segments.push([]);
}
segments[segments.length - 1].push(point);
previousIndex = index;
}
return segments;
};
export const buildGuestDrawerHistoryPath = (
points: readonly AggregatedMetricPoint[],
scale: GuestDrawerHistoryScale,
internal/alerts/filter_evaluation.go
+20 -12
View File
@@ -19,18 +19,20 @@ func (m *Manager) evaluateFilterCondition(guest any, condition FilterCondition)
// guestMetrics holds common metrics for filter evaluation
type guestMetrics struct {
CPU float64 // CPU usage as percentage (0-100)
MemUsage float64 // Memory usage percentage
DiskUsage float64 // Disk usage percentage
DiskRead int64 // Bytes/s
DiskWrite int64 // Bytes/s
NetworkIn int64 // Bytes/s
NetworkOut int64 // Bytes/s
IORateValidity models.IORateValidity
Name string
Node string
ID string
Status string
CPU float64 // CPU usage as percentage (0-100)
MemUsage float64 // Memory usage percentage
DiskUsage float64 // Disk usage percentage
MemoryUnavailable bool
DiskUnavailable bool
DiskRead int64 // Bytes/s
DiskWrite int64 // Bytes/s
NetworkIn int64 // Bytes/s
NetworkOut int64 // Bytes/s
IORateValidity models.IORateValidity
Name string
Node string
ID string
Status string
}
// extractGuestMetrics extracts common metrics from a VM or Container
@@ -57,8 +59,14 @@ func evaluateGuestCondition(metrics guestMetrics, condition FilterCondition) boo
case "cpu":
value = metrics.CPU
case "memory":
if metrics.MemoryUnavailable {
return false
}
value = metrics.MemUsage
case "disk":
if metrics.DiskUnavailable {
return false
}
value = metrics.DiskUsage
case "diskread":
if !validity.DiskRead {
internal/alerts/guest.go
+25 -6
View File
@@ -259,8 +259,11 @@ func (m *Manager) CheckGuest(guest any, instanceName string) {
memoryMetric = &UnifiedResourceMetric{Percent: memUsage}
}
diskReadMetric, diskWriteMetric, networkInMetric, networkOutMetric := guestIORateMetrics(snapshot)
diskMetric := &UnifiedResourceMetric{Percent: diskUsage}
if len(disks) > 0 {
var diskMetric *UnifiedResourceMetric
if !snapshot.DiskUnavailable {
diskMetric = &UnifiedResourceMetric{Percent: diskUsage}
}
if !snapshot.DiskUnavailable && len(disks) > 0 {
seenAggregateDiskKeys := make(map[string]struct{})
for idx, disk := range disks {
if disk.Total <= 0 || disk.Usage < 0 {
@@ -295,6 +298,12 @@ func (m *Manager) CheckGuest(guest any, instanceName string) {
NetworkOut: networkOutMetric,
}, thresholds, evalOpts)
// Retained/failed filesystem evidence is useful to display, but neither
// its values nor an empty result can establish a breach, recovery or disk
// removal. CPU/I/O and independent current memory were evaluated above.
// Still apply explicit per-filesystem disablement to retained identities;
// this is operator policy, not recovery inferred from an old value.
if len(disks) > 0 {
seenDiskKeys := make(map[string]struct{})
seenDiskResources := make(map[string]struct{})
@@ -331,6 +340,12 @@ func (m *Manager) CheckGuest(guest any, instanceName string) {
effectiveDiskThreshold = ensureHysteresisThreshold(diskOverride.Disk)
}
}
if snapshot.DiskUnavailable {
if effectiveDiskThreshold == nil || effectiveDiskThreshold.Trigger <= 0 {
m.clearAlert(canonicalMetricStateID(perDiskResourceID, "disk"))
}
continue
}
log.Debug().
Str("guest", name).
@@ -376,11 +391,15 @@ func (m *Manager) CheckGuest(guest any, instanceName string) {
MonitorOnly: monitorOnly,
})
}
if cleared := m.cleanupGuestDiskAlerts(guestID, seenDiskResources); cleared > 0 {
m.saveActiveAlertsAsync("guest-disk-set-changed")
if !snapshot.DiskUnavailable {
if cleared := m.cleanupGuestDiskAlerts(guestID, seenDiskResources); cleared > 0 {
m.saveActiveAlertsAsync("guest-disk-set-changed")
}
}
} else if !snapshot.DiskUnavailable {
if cleared := m.cleanupGuestDiskAlerts(guestID, nil); cleared > 0 {
m.saveActiveAlertsAsync("guest-disk-alerts-cleared")
}
} else if cleared := m.cleanupGuestDiskAlerts(guestID, nil); cleared > 0 {
m.saveActiveAlertsAsync("guest-disk-alerts-cleared")
}
}
internal/alerts/guest_observation_test.go
+220
View File
@@ -0,0 +1,220 @@
package alerts
import (
"bytes"
"encoding/json"
"sort"
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
)
func guestObservationManager(t *testing.T) *Manager {
t.Helper()
m := newEventLogManager(t)
cfg := characterizationBaseConfig()
cfg.GuestDefaults.DiskWrite = &HysteresisThreshold{Trigger: 5, Clear: 4}
m.UpdateConfig(cfg)
disableTestTimeThresholds(m)
return m
}
func guestObservationVM() models.VM {
return models.VM{
ID: "site:node:105", VMID: 105, Name: "database", Node: "node", Instance: "site", Type: "qemu", Status: "running",
Memory: models.Memory{Total: 1000, Used: 960, Free: 40, Usage: 96,
Observation: models.MemoryObservation{State: "current", Source: "guest-agent-meminfo", ObservedAt: time.Now().Add(-time.Second)}},
Disk: models.Disk{Total: 1000, Used: 980, Free: 20, Usage: 98},
Disks: []models.Disk{{Total: 1000, Used: 980, Free: 20, Usage: 98, Mountpoint: "/", Device: "/dev/vda"},
{Total: 1000, Used: 980, Free: 20, Usage: 98, Mountpoint: "/data", Device: "/dev/vdb"}},
IORateValidity: models.IORateValidity{Explicit: true},
}
}
func guestObservationBytes(t *testing.T, m *Manager) []byte {
t.Helper()
active := m.GetActiveAlerts()
sort.Slice(active, func(i, j int) bool { return active[i].ID < active[j].ID })
data, err := json.Marshal(active)
if err != nil {
t.Fatal(err)
}
return data
}
func TestGuestAlertRetainedObservationsCannotFire(t *testing.T) {
for _, state := range []string{"last-known", "unavailable", "unknown-version"} {
for _, kind := range []string{"vm", "vm-pointer", "lxc", "lxc-pointer"} {
t.Run(kind+"/"+state, func(t *testing.T) {
m := guestObservationManager(t)
vm := guestObservationVM()
vm.Memory.Observation.State = state
vm.DiskStatusReason, vm.GuestAgentStatus = "prev-vm-locked", "deferred"
var guest any = vm
switch kind {
case "vm-pointer":
guest = &vm
case "lxc", "lxc-pointer":
ct := models.Container{ID: vm.ID, VMID: vm.VMID, Name: vm.Name, Node: vm.Node, Instance: vm.Instance, Status: vm.Status, Memory: vm.Memory,
Disk: models.Disk{Usage: 10}, IORateValidity: vm.IORateValidity}
guest = ct
if kind == "lxc-pointer" {
guest = &ct
}
}
for poll := 0; poll < 4; poll++ {
m.CheckGuest(guest, "site")
}
if got := m.GetActiveAlerts(); len(got) != 0 {
t.Fatalf("retained observations opened alerts: %+v", got)
}
if len(queryAlertEvents(t, m, eventlog.Filter{})) != 0 {
t.Fatal("retained observations emitted lifecycle/delivery evidence")
}
})
}
}
}
func TestGuestAlertUnavailableDiskInventoryIsNotRecovery(t *testing.T) {
for _, reason := range []string{"prev-vm-locked", "prev-lock-unverified", "agent-timeout", "agent-not-running", "permission-denied", "agent-error", "no-filesystems"} {
t.Run(reason, func(t *testing.T) {
m := guestObservationManager(t)
vm := guestObservationVM()
m.CheckGuest(vm, "site")
active := m.GetActiveAlerts()
if len(active) != 3 {
t.Fatalf("current memory and two filesystems must fire: %+v", active)
}
for _, a := range active {
if err := m.AcknowledgeAlert(a.ID, "operator"); err != nil {
t.Fatal(err)
}
}
before := guestObservationBytes(t, m)
events := len(queryAlertEvents(t, m, eventlog.Filter{}))
vm.Memory.Usage, vm.Memory.Used, vm.Memory.Free = 10, 100, 900
vm.Memory.Observation.State = "last-known"
vm.Disk.Usage, vm.DiskStatusReason = 10, reason
for i := range vm.Disks {
vm.Disks[i].Usage = 10
}
m.CheckGuest(vm, "site")
vm.Disks = nil // Failed/expired retained inventory is not removal.
vm.Disk.Usage = -1
for poll := 0; poll < 4; poll++ {
m.CheckGuest(vm, "site")
}
if after := guestObservationBytes(t, m); !bytes.Equal(before, after) {
t.Fatalf("unknown observation changed incident values/identity/ack/age: before=%s after=%s", before, after)
}
if len(queryAlertEvents(t, m, eventlog.Filter{})) != events {
t.Fatal("unknown observation emitted lifecycle/delivery evidence")
}
vm.Memory.Observation.State = "current"
vm.Disk.Usage, vm.DiskStatusReason = 10, ""
vm.Disks = []models.Disk{{Total: 1000, Used: 100, Free: 900, Usage: 10, Mountpoint: "/", Device: "/dev/vda"}}
m.CheckGuest(vm, "site") // Fresh recovery plus authoritative removal of /data.
m.CheckGuest(vm, "site")
if len(m.GetActiveAlerts()) != 0 || len(queryAlertEvents(t, m, eventlog.Filter{Types: []string{eventlog.TypeResolved}})) != 3 {
t.Fatal("fresh recovery/removal must resolve each original incident exactly once")
}
})
}
}
func TestGuestAlertRetainedAggregateAndPendingCannotAdvance(t *testing.T) {
m := guestObservationManager(t)
vm := guestObservationVM()
vm.Disks = nil
vm.Memory.Usage = 90
m.mu.Lock()
m.config.MetricTimeThresholds = map[string]map[string]int{"guest": {"memory": 60, "disk": 60}}
m.mu.Unlock()
m.CheckGuest(vm, "site")
if len(m.GetActiveAlerts()) != 0 {
t.Fatal("fresh warning observations should begin their delay, not fire")
}
m.mu.Lock()
m.core.ShiftPending(-2 * time.Minute)
m.mu.Unlock()
vm.Memory.Observation.State, vm.DiskStatusReason = "last-known", "prev-agent-cooldown"
for poll := 0; poll < 4; poll++ {
m.CheckGuest(vm, "site")
}
if len(m.GetActiveAlerts()) != 0 {
t.Fatal("retained memory/disk promoted a delayed observation to an incident")
}
vm.Memory.Observation.State, vm.DiskStatusReason = "current", ""
m.CheckGuest(vm, "site")
if len(m.GetActiveAlerts()) != 2 {
t.Fatal("fresh evidence must resume the existing delay policy")
}
}
func TestGuestAlertFreshIndependentMetricsAndDisablement(t *testing.T) {
for _, state := range []string{"", "current"} {
t.Run("legacy-or-current/"+state, func(t *testing.T) {
m := guestObservationManager(t)
vm := guestObservationVM()
vm.Memory.Observation.State = state
// Guest-agent state alone cannot suppress independent current data.
vm.GuestAgentStatus, vm.Lock = "deferred", "backup"
vm.Memory.Observation.Source = "agent"
m.CheckGuest(vm, "site")
if len(m.GetActiveAlerts()) != 3 {
t.Fatal("legacy/current independent memory and filesystems were suppressed")
}
vm.Memory.Observation.State, vm.DiskStatusReason = "last-known", "prev-vm-locked"
cfg := m.GetConfig()
cfg.GuestDefaults.Memory = &HysteresisThreshold{Trigger: -1}
cfg.GuestDefaults.Disk = &HysteresisThreshold{Trigger: -1}
m.UpdateConfig(cfg)
m.CheckGuest(vm, "site")
if len(m.GetActiveAlerts()) != 0 {
t.Fatal("explicit rule disablement must remain independent of telemetry")
}
})
}
m := guestObservationManager(t)
vm := guestObservationVM()
vm.Memory.Observation.State, vm.DiskStatusReason = "last-known", "prev-vm-locked"
vm.CPU, vm.DiskWrite = .9, 10*1024*1024
vm.IORateValidity.DiskWrite = true
m.CheckGuest(vm, "site")
active := m.GetActiveAlerts()
if len(active) != 2 {
t.Fatalf("current CPU/I/O should still fire, retained memory/disk should not: %+v", active)
}
for _, a := range active {
if a.Type != "cpu" && a.Type != "diskWrite" {
t.Fatalf("unexpected retained metric alert: %+v", a)
}
}
}
func TestGuestFiltersDoNotUseRetainedObservations(t *testing.T) {
m := guestObservationManager(t)
vm := guestObservationVM()
for _, field := range []string{"memory", "disk"} {
for _, op := range []string{">", "<", "=="} {
condition := FilterCondition{Type: "metric", Field: field, Operator: op, Value: 96.0}
vm.Memory.Observation.State, vm.DiskStatusReason = "last-known", "prev-vm-locked"
if m.evaluateVMCondition(vm, condition) || m.evaluateFilterCondition(&vm, condition) {
t.Fatalf("retained %s matched %s", field, op)
}
}
}
vm.Memory.Observation.State, vm.DiskStatusReason = "current", ""
for _, field := range []string{"memory", "disk"} {
if !m.evaluateVMCondition(vm, FilterCondition{Type: "metric", Field: field, Operator: ">", Value: 90.0}) {
t.Fatalf("fresh %s could not match", field)
}
}
vm.Memory = models.UnavailableMemory(1000)
if m.evaluateVMCondition(vm, FilterCondition{Type: "metric", Field: "memory", Operator: "<", Value: 90.0}) {
t.Fatal("unavailable memory was treated as a measured zero in a filter")
}
}
internal/alerts/guest_snapshot.go
+28 -16
View File
@@ -28,9 +28,10 @@ type guestSnapshot struct {
CPUPercent float64
MemUsage float64
// MemoryUnavailable prevents a missing cache-aware sample from clearing a
// real alert or starting a false one.
// Unavailable observations include retained display values. They cannot
// supply a new breach, recovery, filter match or filesystem removal.
MemoryUnavailable bool
DiskUnavailable bool
DiskUsage float64
DiskRead int64
DiskWrite int64
@@ -81,18 +82,20 @@ func (g guestSnapshot) resourceType() string {
func (g guestSnapshot) metrics() guestMetrics {
return guestMetrics{
CPU: g.CPUPercent,
MemUsage: g.MemUsage,
DiskUsage: g.DiskUsage,
DiskRead: g.DiskRead,
DiskWrite: g.DiskWrite,
NetworkIn: g.NetworkIn,
NetworkOut: g.NetworkOut,
IORateValidity: g.IORateValidity,
Name: g.Name,
Node: g.Node,
ID: g.ID,
Status: g.Status,
CPU: g.CPUPercent,
MemUsage: g.MemUsage,
DiskUsage: g.DiskUsage,
MemoryUnavailable: g.MemoryUnavailable,
DiskUnavailable: g.DiskUnavailable,
DiskRead: g.DiskRead,
DiskWrite: g.DiskWrite,
NetworkIn: g.NetworkIn,
NetworkOut: g.NetworkOut,
IORateValidity: g.IORateValidity,
Name: g.Name,
Node: g.Node,
ID: g.ID,
Status: g.Status,
}
}
@@ -108,7 +111,8 @@ func guestSnapshotFromVM(vm models.VM) guestSnapshot {
Lock: vm.Lock,
CPUPercent: unifiedresources.ProxmoxGuestCPUPercent(vm.CPU),
MemUsage: vm.Memory.Usage,
MemoryUnavailable: vm.Memory.UsageUnavailable,
MemoryUnavailable: guestMemoryUnavailableForAlerts(vm.Memory),
DiskUnavailable: vm.DiskStatusReason != "" || vm.Disk.Usage < 0,
DiskUsage: vm.Disk.Usage,
DiskRead: vm.DiskRead,
DiskWrite: vm.DiskWrite,
@@ -133,7 +137,8 @@ func guestSnapshotFromContainer(container models.Container) guestSnapshot {
Lock: container.Lock,
CPUPercent: unifiedresources.ProxmoxGuestCPUPercent(container.CPU),
MemUsage: container.Memory.Usage,
MemoryUnavailable: container.Memory.UsageUnavailable,
MemoryUnavailable: guestMemoryUnavailableForAlerts(container.Memory),
DiskUnavailable: container.Disk.Usage < 0,
DiskUsage: container.Disk.Usage,
DiskRead: container.DiskRead,
DiskWrite: container.DiskWrite,
@@ -146,6 +151,13 @@ func guestSnapshotFromContainer(container models.Container) guestSnapshot {
}.normalizeCollections()
}
func guestMemoryUnavailableForAlerts(memory models.Memory) bool {
// The producer owns freshness and source selection. Legacy values without
// an observation retain their existing behavior; do not infer freshness
// from disk/QGA state, since PVE or a linked Pulse agent can be independent.
return memory.UsageUnavailable || (memory.Observation.State != "" && memory.Observation.State != "current")
}
func guestKindFromType(guestType string) guestKind {
switch strings.ToLower(strings.TrimSpace(guestType)) {
case "qemu", "vm":
internal/alerts/guest_snapshot_test.go
+66
View File
@@ -19,6 +19,72 @@ func TestGuestSnapshotResourceTypeUsesCanonicalSystemContainer(t *testing.T) {
}
}
func TestGuestSnapshotObservationAvailability(t *testing.T) {
for _, test := range []struct {
name string
memory models.Memory
disk models.Disk
reason string
wantMemory bool
wantDisk bool
}{
{name: "legacy measured zero", memory: models.Memory{}, disk: models.Disk{}},
{name: "current independent read", memory: models.Memory{Usage: 96, Observation: models.MemoryObservation{State: "current", Source: "agent"}}, disk: models.Disk{Usage: 98}},
{name: "retained memory only", memory: models.Memory{Usage: 96, Observation: models.MemoryObservation{State: "last-known"}}, disk: models.Disk{Usage: 98}, wantMemory: true},
{name: "unavailable numeric placeholder", memory: models.Memory{Usage: 0, Observation: models.MemoryObservation{State: "unavailable"}}, wantMemory: true},
{name: "unknown observation version", memory: models.Memory{Usage: 96, Observation: models.MemoryObservation{State: "future-version"}}, wantMemory: true},
{name: "explicit unavailable overrides current", memory: models.Memory{Usage: 96, UsageUnavailable: true, Observation: models.MemoryObservation{State: "current"}}, wantMemory: true},
{name: "retained filesystem only", memory: models.Memory{Usage: 96}, disk: models.Disk{Usage: 98}, reason: "prev-vm-locked", wantDisk: true},
{name: "failed empty filesystem inventory", memory: models.Memory{Usage: 96}, disk: models.Disk{Usage: -1}, reason: "agent-error", wantDisk: true},
{name: "legacy unavailable aggregate", memory: models.Memory{Usage: 96}, disk: models.Disk{Usage: -1}, wantDisk: true},
} {
t.Run(test.name, func(t *testing.T) {
// A deferred QGA session does not by itself make independent
// selected memory or Pulse-agent filesystems unavailable.
vm := models.VM{Memory: test.memory, Disk: test.disk, DiskStatusReason: test.reason, GuestAgentStatus: "deferred"}
for _, guest := range []any{vm, &vm} {
snapshot, ok := extractGuestSnapshot(guest)
if !ok || snapshot.MemoryUnavailable != test.wantMemory || snapshot.DiskUnavailable != test.wantDisk {
t.Fatalf("observation flags lost: %+v", snapshot)
}
metrics := snapshot.metrics()
if metrics.MemoryUnavailable != test.wantMemory || metrics.DiskUnavailable != test.wantDisk || metrics.MemUsage != test.memory.Usage || metrics.DiskUsage != test.disk.Usage {
t.Fatal("filter projection lost availability or rewrote retained values")
}
}
ct := models.Container{Memory: test.memory, Disk: test.disk}
for _, guest := range []any{ct, &ct} {
snapshot, _ := extractGuestSnapshot(guest)
if snapshot.MemoryUnavailable != test.wantMemory || snapshot.DiskUnavailable != (test.disk.Usage < 0) {
t.Fatal("LXC observation availability changed with an unrelated VM disk failure")
}
}
})
}
}
func TestGuestAlertRetainedDiskStillHonoursExplicitOverride(t *testing.T) {
for _, override := range []ThresholdConfig{{Disabled: true}, {Disk: &HysteresisThreshold{Trigger: -1}}} {
m := guestObservationManager(t)
vm := guestObservationVM()
vm.Memory.Usage = 10
m.CheckGuest(vm, "site")
if len(m.GetActiveAlerts()) != 2 {
t.Fatal("expected two current filesystem incidents")
}
_, keySource, key := guestDiskIdentity(vm.Disks[0], 0)
cfg := m.GetConfig()
cfg.Overrides[guestDiskOverrideKey(vm, vm.ID, keySource)] = override
m.UpdateConfig(cfg)
vm.DiskStatusReason = "prev-vm-locked"
m.CheckGuest(vm, "site")
active := m.GetActiveAlerts()
if len(active) != 1 || active[0].ResourceID == vm.ID+"-disk-"+key {
t.Fatalf("explicit filesystem disablement was lost or crossed disk identity: %+v", active)
}
}
}
func TestGuestSnapshotUsesCanonicalProxmoxCPUPercent(t *testing.T) {
for _, tc := range []struct {
name string
internal/hostagent/guest_exec_discovery_integration_test.go
+37 -1
View File
@@ -6,6 +6,7 @@ import (
"net/http"
"net/http/httptest"
"os/exec"
"runtime"
"strings"
"sync/atomic"
"testing"
@@ -39,13 +40,27 @@ func (e guestSafetyDiscoveryExecutor) IsAgentConnected(id string) bool {
// Real server/session/registration/agent/Discovery paths, with a fake non-QGA
// config reader and fake provider process. This is NOT native PVE acceptance.
func testGuestExecRealServerAgentDiscoveryAdmission(t *testing.T) {
platforms := []string{runtime.GOOS, "linux", "darwin", "windows"}
seen := make(map[string]bool)
for _, platform := range platforms {
if seen[platform] {
continue
}
seen[platform] = true
t.Run(platform, func(t *testing.T) {
testGuestExecRealServerAgentDiscoveryPlatform(t, platform)
})
}
}
func testGuestExecRealServerAgentDiscoveryPlatform(t *testing.T, platform string) {
s := agentexec.NewServer(func(token, agent, host string) bool {
return token == "fixture" && agent == "node-agent" && host == "node"
})
ts := httptest.NewServer(http.HandlerFunc(s.HandleWebSocket))
defer ts.Close()
logger := zerolog.Nop()
c := NewCommandClient(Config{PulseURL: ts.URL, APIToken: "fixture", StateDir: t.TempDir(), Logger: &logger}, "node-agent", "node", "linux", "6")
c := NewCommandClient(Config{PulseURL: ts.URL, APIToken: "fixture", StateDir: t.TempDir(), Logger: &logger}, "node-agent", "node", platform, "6")
var locked, lockDuringHandoff, failedGuest atomic.Bool
var qgaCalls, configReads atomic.Int32
locked.Store(true)
@@ -109,6 +124,27 @@ func testGuestExecRealServerAgentDiscoveryAdmission(t *testing.T) {
}
}
locked.Store(false)
if runtime.GOOS != "linux" || platform != "linux" {
// PVE admission belongs to a Linux host agent. A platform label must
// not make a non-Linux executable advertise that capability, and a
// Linux executable labelled as another platform must still be refused.
// Exercise the real registration and server gate, not a skipped test
// or an artificial capability override just to reach the happy path.
for _, rt := range []servicediscovery.ResourceType{servicediscovery.ResourceTypeVM, servicediscovery.ResourceTypeDockerVM} {
id := "105"
if rt == servicediscovery.ResourceTypeDockerVM {
id = "105:app"
}
got, err := scanner.Scan(context.Background(), servicediscovery.DiscoveryRequest{ResourceType: rt, ResourceID: id, TargetID: "node-agent"})
if err == nil || err.Error() != agentexec.GuestExecDeferred(agentexec.GuestExecGuardUnavailable).Error() || got == nil || len(got.CommandOutputs) != 0 || qgaCalls.Load() != 0 || configReads.Load() != 0 {
t.Fatalf("unsupported platform reached guest admission: %+v %v calls=%d reads=%d", got, err, qgaCalls.Load(), configReads.Load())
}
}
if !s.IsAgentConnected("node-agent") {
t.Fatal("platform deferral dropped session liveness")
}
return
}
got, err := scanner.ScanVM(context.Background(), "node-agent", "node", "105")
want := int32(len(servicediscovery.GetCommandsForResource(servicediscovery.ResourceTypeVM)))
if err != nil || got == nil || int32(len(got.CommandOutputs)) != want || qgaCalls.Load() != want {
internal/monitoring/guest_alert_observation_integration_test.go
+140
View File
@@ -0,0 +1,140 @@
package monitoring
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
"github.com/rcourtman/pulse-go-rewrite/internal/config"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"github.com/rcourtman/pulse-go-rewrite/pkg/proxmox"
)
// Exercise actual HTTP client, builders, saved snapshots, unified views, History
// and the production alert call. No native QGA or backup is performed.
func TestGuestBackupDeferralAlertObservationLifecycle(t *testing.T) {
const mib = uint64(1024 * 1024)
var phase, guestCalls atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
switch {
case strings.HasSuffix(r.URL.Path, "/config"):
if phase.Load() == 1 {
fmt.Fprint(w, `{"data":{"lock":"backup"}}`)
} else {
fmt.Fprint(w, `{"data":{}}`)
}
case strings.HasSuffix(r.URL.Path, "/status/current"):
fmt.Fprintf(w, `{"data":{"status":"running","agent":1,"maxmem":%d,"mem":%d}}`, 1000*mib, 1000*mib)
case strings.Contains(r.URL.Path, "/agent/"):
guestCalls.Add(1)
switch {
case strings.HasSuffix(r.URL.Path, "file-read"):
available := 40 * 1024
if phase.Load() == 2 {
available = 800 * 1024
}
json.NewEncoder(w).Encode(map[string]any{"data": map[string]any{"content": fmt.Sprintf("MemTotal: 1024000 kB\nMemFree: 10240 kB\nMemAvailable: %d kB\n", available)}})
case strings.HasSuffix(r.URL.Path, "get-fsinfo"):
used := 980 * mib
if phase.Load() == 2 {
used = 200 * mib
}
fmt.Fprintf(w, `{"data":{"result":[{"mountpoint":"/","type":"ext4","total-bytes":%d,"used-bytes":%d}]}}`, 1000*mib, used)
case strings.HasSuffix(r.URL.Path, "network-get-interfaces"):
fmt.Fprint(w, `{"data":{"result":[]}}`)
case strings.HasSuffix(r.URL.Path, "get-osinfo"):
fmt.Fprint(w, `{"data":{"name":"Linux","version":"fixture"}}`)
default:
fmt.Fprint(w, `{"data":{"result":{"version":"1.0"}}}`)
}
default:
http.NotFound(w, r)
}
}))
defer server.Close()
client, err := proxmox.NewClient(proxmox.ClientConfig{Host: server.URL, TokenName: "fixture@pve!pulse", TokenValue: "fixture", Timeout: time.Second})
if err != nil {
t.Fatal(err)
}
am := alerts.NewManagerWithDataDir(t.TempDir(), alerts.WithoutPersistedAlertRestore())
defer am.Stop()
cfg := am.GetConfig()
cfg.Enabled = true
cfg.GuestDefaults.Memory = &alerts.HysteresisThreshold{Trigger: 85, Clear: 80}
cfg.GuestDefaults.Disk = &alerts.HysteresisThreshold{Trigger: 90, Clear: 85}
cfg.GuestDefaults.CPU = &alerts.HysteresisThreshold{Trigger: 80, Clear: 75}
cfg.MetricTimeThresholds = map[string]map[string]int{"guest": {"memory": 0, "disk": 0, "cpu": 0}}
am.UpdateConfig(cfg)
registry := unifiedresources.NewRegistry(nil)
m := &Monitor{config: &config.Config{}, alertManager: am, rateTracker: NewRateTracker(), metricsHistory: NewMetricsHistory(32, time.Hour), guestMetadataLimiter: make(map[string]time.Time)}
res := proxmox.ClusterResource{Type: "qemu", Node: "node", Name: "guest", VMID: 105, Status: "running", MaxMem: 1000 * mib, Mem: 1000 * mib, MaxDisk: 1000 * mib}
identity := makeGuestID("alert-observation", "node", 105)
var previous *models.VM
build := func() models.VM {
t.Helper()
vm, raw, source, notes, at, ok := m.buildVMFromClusterResource(context.Background(), "alert-observation", res, client, identity, nil, previous)
if !ok {
t.Fatal("VM disappeared")
}
m.recordGuestSnapshot("alert-observation", "qemu", "node", 105, GuestMemorySnapshot{Status: vm.Status, RetrievedAt: at, MemorySource: source, Memory: vm.Memory, Raw: raw, Notes: notes})
m.recordGuestMetrics([]models.VM{vm}, nil, time.Now().Add(-time.Second))
registry.IngestSnapshot(models.StateSnapshot{VMs: []models.VM{vm}})
next := previousVMFromView(registry.VMs()[0])
previous = &next
return vm
}
initial := build()
// The observation was made before alert policy became enabled for this VM.
// A retained high value must not create an incident upon the next poll.
phase.Store(1)
before := guestCalls.Load()
locked := build()
if locked.Memory.Observation.State != "last-known" || !strings.HasPrefix(locked.DiskStatusReason, "prev-") {
t.Fatalf("fixture did not produce real last-known evidence: %+v", locked)
}
m.checkGuestAlertsForVM("alert-observation", locked)
if len(am.GetActiveAlerts()) != 0 {
t.Fatal("production alert call treated retained backup-lock evidence as current")
}
// The original current evidence can establish an alert. Subsequent lock
// polls must neither refresh its observation nor resolve it on expiry.
m.checkGuestAlertsForVM("alert-observation", initial)
if len(am.GetActiveAlerts()) != 2 {
t.Fatal("current memory/filesystem evidence did not establish two alerts")
}
for poll := 0; poll < 4; poll++ {
m.checkGuestAlertsForVM("alert-observation", build())
}
if guestCalls.Load() != before || len(am.GetActiveAlerts()) != 2 {
t.Fatal("backup deferral queried QGA or lost the original alerts")
}
for _, a := range am.GetActiveAlerts() {
if a.Value != 96 && a.Value != 98 {
t.Fatal("retained poll changed the last trusted alert value")
}
}
if got := len(m.metricsHistory.GetGuestMetrics(identity, "memory", time.Hour)); got != 1 {
t.Fatalf("retained memory refreshed History: %d points", got)
}
if got := len(m.metricsHistory.GetGuestMetrics(identity, "disk", time.Hour)); got != 1 {
t.Fatalf("retained filesystem refreshed History: %d points", got)
}
phase.Store(2)
entry := m.vmAgentMemCache[guestMemoryCacheKey("alert-observation", "node", 105)]
entry.fetchedAt = time.Now().Add(-2 * vmAgentMemCacheTTL)
m.vmAgentMemCache[guestMemoryCacheKey("alert-observation", "node", 105)] = entry
resumed := build()
m.checkGuestAlertsForVM("alert-observation", resumed)
if guestCalls.Load() <= before || resumed.Memory.Observation.State != "current" || resumed.DiskStatusReason != "" || len(am.GetActiveAlerts()) != 0 {
t.Fatal("fresh unlocked memory/filesystem evidence did not recover the original incidents")
}
}
internal/monitoring/guest_history_observation_test.go
+294
View File
@@ -0,0 +1,294 @@
package monitoring
import (
"context"
"fmt"
"math"
"net/http"
"net/http/httptest"
"reflect"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/mock"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"github.com/rcourtman/pulse-go-rewrite/pkg/metrics"
"github.com/rcourtman/pulse-go-rewrite/pkg/proxmox"
)
func guestHistoryObservationMonitor(t *testing.T) *Monitor {
t.Helper()
wasMock := mock.IsMockEnabled()
mustSetMockEnabled(t, false)
t.Cleanup(func() { mustSetMockEnabled(t, wasMock) })
cfg := metrics.DefaultConfig(t.TempDir())
cfg.WriteBufferSize = 512
cfg.FlushInterval = time.Hour
store, err := metrics.NewStore(cfg)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = store.Close() })
if err := store.WaitForMaintenance(5 * time.Second); err != nil {
t.Fatal(err)
}
return &Monitor{metricsHistory: NewMetricsHistory(128, 24*time.Hour), metricsStore: store}
}
func guestHistoryStoredPoints(t *testing.T, m *Monitor, kind, id, metric string) []metrics.MetricPoint {
t.Helper()
m.metricsStore.Flush()
now := time.Now()
points, err := m.metricsStore.Query(kind, id, metric, now.Add(-time.Minute), now.Add(time.Minute), 0)
if err != nil {
t.Fatal(err)
}
return points
}
func TestGuestHistoryMemoryObservationAdmission(t *testing.T) {
for _, tc := range []struct {
name string
memory models.Memory
want bool
}{
{name: "legacy", memory: models.Memory{Total: 100, Used: 40, Free: 60, Usage: 40}, want: true},
{name: "current", memory: models.Memory{Total: 100, Used: 40, Free: 60, Usage: 40, Observation: models.MemoryObservation{State: "current"}}, want: true},
{name: "current measured zero", memory: models.Memory{Total: 100, Free: 100, Observation: models.MemoryObservation{State: "current"}}, want: true},
{name: "last-known", memory: models.Memory{Total: 100, Used: 40, Free: 60, Usage: 40, Observation: models.MemoryObservation{State: "last-known"}}},
{name: "unavailable observation with retained numbers", memory: models.Memory{Total: 100, Used: 40, Free: 60, Usage: 40, Observation: models.MemoryObservation{State: "unavailable"}}},
{name: "unrecognised observation", memory: models.Memory{Total: 100, Used: 40, Free: 60, Usage: 40, Observation: models.MemoryObservation{State: "future-state"}}},
{name: "missing usage", memory: models.UnavailableMemory(100)},
{name: "NaN", memory: models.Memory{Total: 100, Used: 40, Free: 60, Usage: math.NaN()}},
{name: "contradictory bytes", memory: models.Memory{Total: 100, Used: 140, Free: 60, Usage: 40}},
} {
t.Run(tc.name, func(t *testing.T) {
gotUsage, gotUsed := historyMemoryUsage(tc.memory), historyMemoryUsed(tc.memory)
if tc.want {
if gotUsage != tc.memory.Usage || gotUsed != float64(tc.memory.Used) {
t.Errorf("current/legacy observation lost percentage or byte history: %v / %v", gotUsage, gotUsed)
}
} else if gotUsage != -1 || gotUsed != -1 {
t.Errorf("non-current observation became percentage/byte History: %v / %v", gotUsage, gotUsed)
}
})
}
}
func TestGuestHistoryObservationContinuity(t *testing.T) {
m := guestHistoryObservationMonitor(t)
now := time.Now()
vm := models.VM{ID: "history:node:105", Instance: "history", Node: "node", VMID: 105, Type: "qemu", Status: "running", CPU: .1, LastSeen: now,
Memory: models.Memory{Total: 100, Used: 40, Free: 60, Usage: 40, Observation: models.MemoryObservation{State: "current", Source: "available-field", ObservedAt: now}},
Disk: models.Disk{Total: 100, Used: 50, Free: 50, Usage: 50}, DiskWrite: 10,
IORateValidity: models.IORateValidity{Explicit: true, DiskWrite: true}}
ct := models.Container{ID: "history:node:106", Status: "running", LastSeen: now, CPU: .2, Memory: vm.Memory, Disk: vm.Disk}
record := func() {
t.Helper()
vm.LastSeen, ct.LastSeen = time.Now(), time.Now()
m.recordGuestMetrics([]models.VM{vm}, []models.Container{ct}, time.Now().Add(-time.Second))
}
record()
type series struct{ kind, id, metric string }
seriesToPreserve := []series{{"vm", vm.ID, "memory"}, {"vm", vm.ID, "memoryused"}, {"vm", vm.ID, "disk"}, {"container", ct.ID, "memory"}, {"container", ct.ID, "memoryused"}}
originalMemory := make(map[series][]MetricPoint)
originalStored := make(map[series][]metrics.MetricPoint)
for _, s := range seriesToPreserve {
originalMemory[s] = m.metricsHistory.GetGuestMetrics(s.id, s.metric, time.Hour)
originalStored[s] = guestHistoryStoredPoints(t, m, s.kind, s.id, s.metric)
if len(originalMemory[s]) != 1 || len(originalStored[s]) != 1 {
t.Fatalf("fresh %s %s failed to establish both histories", s.kind, s.metric)
}
}
// Persisted points use second precision. Cross a second before the failed
// polls so a falsely renewed point cannot hide behind the original key.
time.Sleep(1100 * time.Millisecond)
for i, state := range []string{"last-known", "unavailable", "future-state"} {
vm.Memory.Observation.State, ct.Memory.Observation.State = state, state
vm.GuestAgentStatus = "available" // Not just backup/cooldown deferral.
vm.DiskStatusReason = []string{"no-status", "agent-error", "prev-agent-timeout"}[i]
record()
}
for _, s := range seriesToPreserve {
if got := m.metricsHistory.GetGuestMetrics(s.id, s.metric, time.Hour); !reflect.DeepEqual(got, originalMemory[s]) {
t.Errorf("%s %s renewed or changed its last trusted in-memory observations", s.kind, s.metric)
}
if got := guestHistoryStoredPoints(t, m, s.kind, s.id, s.metric); !reflect.DeepEqual(got, originalStored[s]) {
t.Errorf("%s %s renewed or changed its last trusted persisted observations", s.kind, s.metric)
}
}
for _, metric := range []string{"cpu", "diskwrite"} {
if got := m.metricsHistory.GetGuestMetrics(vm.ID, metric, time.Hour); len(got) != 4 {
t.Errorf("independently current %s was suppressed: %d points", metric, len(got))
}
}
if got := m.metricsHistory.GetGuestMetrics(ct.ID, "disk", time.Hour); len(got) != 4 {
t.Error("independently current container filesystem was suppressed by memory state")
}
// Current measured zero is recovery evidence, not missing usage. The
// original points are kept; this repair never deletes existing History.
time.Sleep(1100 * time.Millisecond)
vm.Memory = models.Memory{Total: 100, Free: 100, Observation: models.MemoryObservation{State: "current", Source: "available-field", ObservedAt: time.Now()}}
ct.Memory = vm.Memory
vm.Disk = models.Disk{Total: 100, Free: 100}
vm.DiskStatusReason = ""
record()
for _, s := range seriesToPreserve {
points := m.metricsHistory.GetGuestMetrics(s.id, s.metric, time.Hour)
if len(points) != 2 || points[1].Value != 0 || !reflect.DeepEqual(points[:1], originalMemory[s]) {
t.Errorf("%s %s did not resume with current zero while preserving old evidence", s.kind, s.metric)
}
stored := guestHistoryStoredPoints(t, m, s.kind, s.id, s.metric)
if len(stored) != 2 || stored[1].Value != 0 || !reflect.DeepEqual(stored[:1], originalStored[s]) {
t.Errorf("%s %s persistent recovery lost current zero or original evidence", s.kind, s.metric)
}
for _, duration := range []time.Duration{time.Minute, 4 * time.Hour} {
chart := m.GetGuestMetricsForChart(s.id, s.kind, s.id, duration)[s.metric]
if len(chart) != 2 || chart[0].Value != originalMemory[s][0].Value || chart[1].Value != 0 {
t.Errorf("%s %s chart (%v) presented retained polls as new readings: %+v", s.kind, s.metric, duration, chart)
}
}
}
}
func TestGuestHistoryPreservesIndependentCurrentMemory(t *testing.T) {
for _, source := range []string{"available-field", "agent", "guest-agent-meminfo"} {
for _, snapshotSource := range []string{"", "previous-snapshot", "guest-agent-meminfo"} {
t.Run(source+"/diagnostic-"+snapshotSource, func(t *testing.T) {
m := guestHistoryObservationMonitor(t)
now := time.Now()
vm := models.VM{ID: "independent:node:105", Instance: "independent", Node: "node", VMID: 105, Status: "running", LastSeen: now, GuestAgentStatus: "deferred",
Memory: models.Memory{Total: 100, Used: 25, Free: 75, Usage: 25, Observation: models.MemoryObservation{State: "current", Source: source, ObservedAt: now.Add(-time.Second)}}}
if snapshotSource != "" {
m.recordGuestSnapshot(vm.Instance, "qemu", vm.Node, vm.VMID, GuestMemorySnapshot{Status: "running", RetrievedAt: now, MemorySource: snapshotSource})
}
m.recordGuestMetrics([]models.VM{vm}, nil, now.Add(-time.Second))
for _, metric := range []string{"memory", "memoryused"} {
if got := m.metricsHistory.GetGuestMetrics(vm.ID, metric, time.Hour); len(got) != 1 || got[0].Value != 25 {
t.Errorf("current selected %s/%s lost History to unrelated deferred disk or diagnostic source", source, metric)
}
if got := guestHistoryStoredPoints(t, m, "vm", vm.ID, metric); len(got) != 1 || got[0].Value != 25 {
t.Errorf("current selected %s/%s lost persistent History", source, metric)
}
}
})
}
}
// Preserve the historical guard for an unannotated deferred guest. New
// producer observations, not a change to numeric/display availability,
// are what let independently current memory bypass that legacy heuristic.
m := guestHistoryObservationMonitor(t)
vm := models.VM{ID: "legacy-deferred", Status: "running", LastSeen: time.Now(), GuestAgentStatus: "deferred", Memory: models.Memory{Total: 100, Used: 25, Free: 75, Usage: 25}}
m.recordGuestMetrics([]models.VM{vm}, nil, time.Now().Add(-time.Second))
for _, metric := range []string{"memory", "memoryused"} {
if got := m.metricsHistory.GetGuestMetrics(vm.ID, metric, time.Hour); len(got) != 0 {
t.Errorf("unannotated deferral lost its historical guard: %s", metric)
}
}
}
func TestGuestHistoryPollFailureAndRecovery(t *testing.T) {
const mib = uint64(1024 * 1024)
var phase atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
switch {
case strings.HasSuffix(r.URL.Path, "/config"):
fmt.Fprint(w, `{"data":{}}`)
case strings.HasSuffix(r.URL.Path, "/status/current"):
if strings.Contains(r.URL.Path, "/106/") {
http.Error(w, "permission denied", http.StatusForbidden)
return
}
meminfo := ""
if phase.Load() != 1 {
available := 800 * mib
if phase.Load() == 2 {
available = 600 * mib
}
meminfo = fmt.Sprintf(`,"meminfo":{"total":%d,"available":%d}`, 1000*mib, available)
}
fmt.Fprintf(w, `{"data":{"status":"running","agent":1,"maxmem":%d,"mem":%d%s}}`, 1000*mib, 1000*mib, meminfo)
case strings.HasSuffix(r.URL.Path, "file-read"):
http.Error(w, "permission denied", http.StatusForbidden)
case strings.HasSuffix(r.URL.Path, "get-fsinfo"):
if phase.Load() == 1 {
http.Error(w, "permission denied", http.StatusForbidden)
return
}
fmt.Fprintf(w, `{"data":{"result":[{"mountpoint":"/","type":"ext4","total-bytes":%d,"used-bytes":%d}]}}`, 1000*mib, 300*mib)
case strings.HasSuffix(r.URL.Path, "network-get-interfaces"):
fmt.Fprint(w, `{"data":{"result":[]}}`)
case strings.HasSuffix(r.URL.Path, "get-osinfo"):
fmt.Fprint(w, `{"data":{"name":"Linux","version":"fixture"}}`)
case strings.HasSuffix(r.URL.Path, "info"):
fmt.Fprint(w, `{"data":{"result":{"version":"1.0"}}}`)
default:
http.NotFound(w, r)
}
}))
defer server.Close()
client, err := proxmox.NewClient(proxmox.ClientConfig{Host: server.URL, TokenName: "fixture@pve!pulse", TokenValue: "fixture", Timeout: time.Second})
if err != nil {
t.Fatal(err)
}
m := guestHistoryObservationMonitor(t)
m.rateTracker = NewRateTracker()
m.guestMetadataLimiter = make(map[string]time.Time)
registry := unifiedresources.NewRegistry(nil)
res := proxmox.ClusterResource{Type: "qemu", Node: "node", Name: "guest", VMID: 105, Status: "running", MaxMem: 1000 * mib, Mem: 1000 * mib, MaxDisk: 1000 * mib, Disk: 400 * mib, CPU: .1}
id := makeGuestID("poll-history", res.Node, res.VMID)
var previous *models.VM
build := func() models.VM {
t.Helper()
vm, raw, source, notes, at, ok := m.buildVMFromClusterResource(context.Background(), "poll-history", res, client, id, nil, previous)
if !ok {
t.Fatal("guest disappeared")
}
m.recordGuestSnapshot("poll-history", "qemu", res.Node, res.VMID, GuestMemorySnapshot{Status: vm.Status, RetrievedAt: at, MemorySource: source, Memory: vm.Memory, Raw: raw, Notes: notes})
m.recordGuestMetrics([]models.VM{vm}, nil, time.Now().Add(-time.Second))
registry.IngestSnapshot(models.StateSnapshot{VMs: []models.VM{vm}})
next := previousVMFromView(registry.VMs()[0])
previous = &next
return vm
}
initial := build()
if initial.Memory.Observation.State != "current" || initial.Memory.Usage != 20 || initial.DiskStatusReason != "" {
t.Fatalf("initial poll did not supply current native observations: %+v", initial)
}
phase.Store(1)
failed := build()
if failed.Memory.Observation.State != "last-known" || failed.Memory.Usage != 20 || failed.GuestAgentStatus == "deferred" || !strings.HasPrefix(failed.DiskStatusReason, "prev-") {
t.Fatalf("fixture failed to reproduce ordinary failed-read retention: %+v", failed)
}
for _, metric := range []string{"memory", "memoryused", "disk"} {
if got := m.metricsHistory.GetGuestMetrics(id, metric, time.Hour); len(got) != 1 {
t.Errorf("real client/poll ordinary failure added %s History: %d points", metric, len(got))
}
}
// A new guest with no status has only a cluster disk allocation. Its
// positive number is not filesystem evidence, even without prev- prefix.
other := res
other.VMID = 106
newVM, _, _, _, _, ok := m.buildVMFromClusterResource(context.Background(), "poll-history", other, client, makeGuestID("poll-history", "node", 106), nil, nil)
if !ok || newVM.DiskStatusReason != "no-status" || newVM.Disk.Usage <= 0 {
t.Fatalf("fixture did not retain the unverified positive allocation: %+v", newVM)
}
m.recordGuestMetrics([]models.VM{newVM}, nil, time.Now().Add(-time.Second))
if got := m.metricsHistory.GetGuestMetrics(newVM.ID, "disk", time.Hour); len(got) != 0 {
t.Error("unverified cluster disk allocation became a filesystem History sample")
}
phase.Store(2)
resumed := build()
if resumed.Memory.Observation.State != "current" || resumed.Memory.Usage != 40 || resumed.DiskStatusReason != "" {
t.Fatalf("recovery did not supply current independent PVE and filesystem observations: %+v", resumed)
}
for _, metric := range []string{"memory", "memoryused", "disk"} {
if got := m.metricsHistory.GetGuestMetrics(id, metric, time.Hour); len(got) != 2 {
t.Errorf("real current recovery did not resume %s History: %d points", metric, len(got))
}
}
}
internal/monitoring/guest_metadata.go
+20 -5
View File
@@ -43,9 +43,9 @@ type guestMetadataCacheEntry struct {
osName string
osVersion string
agentVersion string
fetchedAt time.Time
osInfoFailureCount int // Track consecutive OS info failures
osInfoSkip bool // Skip OS info calls after repeated failures (refs #692)
fetchedAt time.Time // Last accepted useful metadata, not the last attempt.
osInfoFailureCount int // Track consecutive OS info failures
osInfoSkip bool // Skip OS info calls after repeated failures (refs #692)
}
func guestMetadataCacheHasUsefulData(entry guestMetadataCacheEntry) bool {
@@ -267,6 +267,7 @@ func (m *Monitor) fetchGuestAgentMetadata(ctx context.Context, client PVEClientI
osName := cached.osName
osVersion := cached.osVersion
agentVersion := cached.agentVersion
metadataObserved := false
if reserved {
if !m.acquireGuestMetadataSlot(ctx) {
@@ -295,6 +296,7 @@ func (m *Monitor) fetchGuestAgentMetadata(ctx context.Context, client PVEClientI
processedIPs, processedIfaces := processGuestNetworkInterfaces(ifaces)
if len(processedIPs) > 0 || len(processedIfaces) > 0 {
ipAddresses, networkIfaces = processedIPs, processedIfaces
metadataObserved = true
} else if len(cached.ipAddresses) == 0 && len(cached.networkInterfaces) == 0 {
ipAddresses = nil
networkIfaces = nil
@@ -359,6 +361,7 @@ func (m *Monitor) fetchGuestAgentMetadata(ctx context.Context, client PVEClientI
extractedOSName, extractedOSVersion := extractGuestOSInfo(agentInfo)
if extractedOSName != "" || extractedOSVersion != "" {
osName, osVersion = extractedOSName, extractedOSVersion
metadataObserved = true
}
osInfoFailureCount = 0 // Reset on success
osInfoSkip = false
@@ -392,17 +395,25 @@ func (m *Monitor) fetchGuestAgentMetadata(ctx context.Context, client PVEClientI
Msg("Guest agent version unavailable")
} else if version, ok := versionRaw.(string); ok && version != "" {
agentVersion = version
metadataObserved = true
} else if cached.agentVersion == "" {
agentVersion = ""
}
// Errors and empty/unusable replies may retain display identity, but cannot
// make that identity new evidence that the guest agent is still responsive.
// Retry timing is tracked separately by the limiter, not by renewing its age.
fetchedAt := cached.fetchedAt
if metadataObserved {
fetchedAt = time.Now()
}
entry := guestMetadataCacheEntry{
ipAddresses: cloneStringSlice(ipAddresses),
networkInterfaces: cloneGuestNetworkInterfaces(networkIfaces),
osName: osName,
osVersion: osVersion,
agentVersion: agentVersion,
fetchedAt: time.Now(),
fetchedAt: fetchedAt,
osInfoFailureCount: osInfoFailureCount,
osInfoSkip: osInfoSkip,
}
@@ -414,7 +425,11 @@ func (m *Monitor) fetchGuestAgentMetadata(ctx context.Context, client PVEClientI
m.guestMetadataCache[key] = entry
m.guestMetadataMu.Unlock()
if reserved {
m.scheduleGuestMetadataFetchForEntry(key, time.Now(), entry)
if metadataObserved {
m.scheduleGuestMetadataFetchForEntry(key, time.Now(), entry)
} else {
m.deferGuestMetadataRetry(key, time.Now())
}
}
return ipAddresses, networkIfaces, osName, osVersion, agentVersion
internal/monitoring/guest_metadata_observation_test.go
+248
View File
@@ -0,0 +1,248 @@
package monitoring
import (
"context"
"errors"
"fmt"
"net/http"
"net/http/httptest"
"reflect"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/config"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/pkg/proxmox"
)
type metadataObservationClient struct {
mockPVEClient
err error
network []proxmox.VMNetworkInterface
os map[string]interface{}
version string
calls int
}
func (c *metadataObservationClient) GetVMNetworkInterfaces(context.Context, string, int) ([]proxmox.VMNetworkInterface, error) {
c.calls++
return c.network, c.err
}
func (c *metadataObservationClient) GetVMAgentInfo(context.Context, string, int) (map[string]interface{}, error) {
c.calls++
return c.os, c.err
}
func (c *metadataObservationClient) GetVMAgentVersion(context.Context, string, int) (string, error) {
c.calls++
return c.version, c.err
}
func metadataObservationFixture(at time.Time) guestMetadataCacheEntry {
return guestMetadataCacheEntry{
ipAddresses: []string{"192.0.2.10"},
networkInterfaces: []models.GuestNetworkInterface{{Name: "eth0", MAC: "02:00:00:00:00:01", Addresses: []string{"192.0.2.10"}}},
osName: "Linux",
osVersion: "fixture",
agentVersion: "1.0",
fetchedAt: at,
}
}
func TestGuestMetadataFailedRefreshPreservesObservation(t *testing.T) {
for _, tc := range []struct {
name string
client metadataObservationClient
}{
{"ordinary-error", metadataObservationClient{err: errors.New("metadata unavailable")}},
{"empty-success", metadataObservationClient{}},
{"filtered-network", metadataObservationClient{network: []proxmox.VMNetworkInterface{{Name: "lo", IPAddresses: []proxmox.VMIPAddress{{Address: "127.0.0.1"}}}}}},
{"unusable-os", metadataObservationClient{os: map[string]interface{}{"result": map[string]interface{}{"unrelated": "fixture"}}}},
} {
t.Run(tc.name, func(t *testing.T) {
key := guestMetadataCacheKey("metadata", "node", 105)
original := metadataObservationFixture(time.Now().Add(-2 * guestMetadataCacheTTL))
m := &Monitor{guestMetadataCache: map[string]guestMetadataCacheEntry{key: original}, guestMetadataLimiter: make(map[string]time.Time), guestMetadataRetryBackoff: 45 * time.Second}
status := &proxmox.VMStatus{Agent: proxmox.VMAgentField{Value: 1}}
client := tc.client
for attempt := 0; attempt < 2; attempt++ {
m.guestMetadataLimiter[key] = time.Now().Add(-time.Second)
start := time.Now()
ips, ifaces, name, version, agent := m.fetchGuestAgentMetadata(context.Background(), &client, "metadata", "node", "guest", 105, status, false)
end := time.Now()
if !reflect.DeepEqual(ips, original.ipAddresses) || !reflect.DeepEqual(ifaces, original.networkInterfaces) || name != original.osName || version != original.osVersion || agent != original.agentVersion {
t.Error("unsuccessful refresh lost last-known identity")
}
entry := m.guestMetadataCache[key]
if !entry.fetchedAt.Equal(original.fetchedAt) || m.hasRecentGuestMetadataEvidence("metadata", "node", 105, end) {
t.Error("unsuccessful refresh renewed old guest-agent evidence")
}
if next := m.guestMetadataLimiter[key]; next.Before(start.Add(45*time.Second)) || next.After(end.Add(45*time.Second)) {
t.Error("unsuccessful refresh did not retain the configured retry backoff")
}
// Caller-owned slices cannot change the retained cache.
ips[0] = "192.0.2.99"
ifaces[0].Addresses[0] = "192.0.2.99"
if !reflect.DeepEqual(m.guestMetadataCache[key].networkInterfaces, original.networkInterfaces) || !reflect.DeepEqual(m.guestMetadataCache[key].ipAddresses, original.ipAddresses) {
t.Error("retained identity aliases returned values")
}
calls := client.calls
m.fetchGuestAgentMetadata(context.Background(), &client, "metadata", "node", "guest", 105, status, false)
if client.calls != calls {
t.Error("backoff queued another metadata command")
}
}
if client.calls != 6 {
t.Errorf("metadata calls = %d, want exactly two three-command attempts", client.calls)
}
})
}
}
func TestGuestMetadataUsefulRefreshRenewsOnlyObservedEvidence(t *testing.T) {
for _, tc := range []struct {
name string
client metadataObservationClient
}{
{"network", metadataObservationClient{network: []proxmox.VMNetworkInterface{{Name: "eth1", IPAddresses: []proxmox.VMIPAddress{{Address: "192.0.2.11"}}}}}},
{"os", metadataObservationClient{os: map[string]interface{}{"name": "NewOS", "version": "2"}}},
{"agent-version", metadataObservationClient{version: "2.0"}},
{"unchanged-but-observed", metadataObservationClient{version: "1.0"}},
} {
t.Run(tc.name, func(t *testing.T) {
key := guestMetadataCacheKey("metadata", "node", 105)
original := metadataObservationFixture(time.Now().Add(-2 * guestMetadataCacheTTL))
m := &Monitor{guestMetadataCache: map[string]guestMetadataCacheEntry{key: original}, guestMetadataLimiter: make(map[string]time.Time)}
client := tc.client
start := time.Now()
m.fetchGuestAgentMetadata(context.Background(), &client, "metadata", "node", "guest", 105, &proxmox.VMStatus{Agent: proxmox.VMAgentField{Value: 1}}, false)
end := time.Now()
entry := m.guestMetadataCache[key]
if entry.fetchedAt.Before(start) || entry.fetchedAt.After(end) || !m.hasRecentGuestMetadataEvidence("metadata", "node", 105, end) {
t.Error("accepted useful metadata did not establish current evidence")
}
if tc.client.version != "" && entry.agentVersion != tc.client.version {
t.Error("current agent version was lost")
}
if tc.client.os != nil && (entry.osName != "NewOS" || entry.osVersion != "2") {
t.Error("current OS identity was lost")
}
if len(tc.client.network) > 0 && (len(entry.networkInterfaces) != 1 || entry.networkInterfaces[0].Name != "eth1") {
t.Error("current network metadata was lost")
}
if len(tc.client.network) == 0 && !reflect.DeepEqual(entry.networkInterfaces, original.networkInterfaces) {
t.Error("an independent useful field cleared retained network identity")
}
})
}
}
// The real HTTP/client/metadata/builder path must not turn a complete failed or
// empty metadata refresh into recent guest-agent evidence. This is not native
// QGA execution, backup overlap or workload recovery.
func TestGuestMetadataObservationHTTPPollLifecycle(t *testing.T) {
for _, mode := range []string{"refused", "empty", "unusable"} {
t.Run(mode, func(t *testing.T) {
const mib = uint64(1024 * 1024)
var phase, calls atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
p := phase.Load()
switch {
case strings.HasSuffix(r.URL.Path, "/config"):
fmt.Fprint(w, `{"data":{}}`)
case strings.HasSuffix(r.URL.Path, "/status/current"):
agent := `1`
if p == 2 {
agent = `{"enabled":1,"available":0}`
}
fmt.Fprintf(w, `{"data":{"status":"running","agent":%s,"maxmem":%d,"mem":%d,"meminfo":{"available":%d},"cpu":0.2}}`, agent, 8*mib, 3*mib, 5*mib)
case strings.Contains(r.URL.Path, "/agent/"):
calls.Add(1)
if p == 1 {
if mode == "refused" {
http.Error(w, "permission denied", http.StatusForbidden)
return
}
switch {
case strings.HasSuffix(r.URL.Path, "network-get-interfaces"):
fmt.Fprint(w, `{"data":{"result":[]}}`)
case strings.HasSuffix(r.URL.Path, "get-osinfo") && mode == "unusable":
fmt.Fprint(w, `{"data":{"unrelated":"fixture"}}`)
case strings.HasSuffix(r.URL.Path, "get-osinfo"):
fmt.Fprint(w, `{"data":{}}`)
default:
fmt.Fprint(w, `{"data":{"result":{}}}`)
}
return
}
ip, version := "192.0.2.10", "1.0"
if p == 3 {
ip, version = "192.0.2.11", "2.0"
}
switch {
case strings.HasSuffix(r.URL.Path, "network-get-interfaces"):
fmt.Fprintf(w, `{"data":{"result":[{"name":"eth0","ip-addresses":[{"ip-address":%q,"ip-address-type":"ipv4","prefix":24}]}]}}`, ip)
case strings.HasSuffix(r.URL.Path, "get-osinfo"):
fmt.Fprint(w, `{"data":{"name":"Linux","version":"fixture"}}`)
case strings.HasSuffix(r.URL.Path, "info"):
fmt.Fprintf(w, `{"data":{"result":{"version":%q}}}`, version)
case strings.HasSuffix(r.URL.Path, "get-fsinfo"):
fmt.Fprintf(w, `{"data":{"result":[{"mountpoint":"/","type":"ext4","total-bytes":%d,"used-bytes":%d}]}}`, 1000*mib, 300*mib)
default:
http.NotFound(w, r)
}
default:
http.NotFound(w, r)
}
}))
defer server.Close()
client, err := proxmox.NewClient(proxmox.ClientConfig{Host: server.URL, TokenName: "fixture@pve!pulse", TokenValue: "fixture", Timeout: time.Second})
if err != nil {
t.Fatal(err)
}
m := &Monitor{config: &config.Config{}, rateTracker: NewRateTracker(), metricsHistory: NewMetricsHistory(16, time.Hour), guestMetadataLimiter: make(map[string]time.Time)}
res := proxmox.ClusterResource{Type: "qemu", Node: "node", Name: "guest", VMID: 105, Status: "running", MaxMem: 8 * mib, Mem: 3 * mib, MaxDisk: 1000 * mib, CPU: 0.2}
build := func() models.VM {
vm, _, _, _, _, ok := m.buildVMFromClusterResource(context.Background(), "metadata", res, client, "metadata:node:105", nil, nil)
if !ok {
t.Fatal("guest disappeared")
}
return vm
}
first := build()
if first.GuestAgentStatus != "available" || first.AgentVersion != "1.0" || calls.Load() != 4 {
t.Fatalf("healthy fixture absent: status=%s version=%s calls=%d", first.GuestAgentStatus, first.AgentVersion, calls.Load())
}
key := guestMetadataCacheKey("metadata", "node", 105)
original := m.guestMetadataCache[key]
original.fetchedAt = time.Now().Add(-2 * guestMetadataCacheTTL)
m.guestMetadataCache[key] = original
m.guestMetadataLimiter[key] = time.Now().Add(-time.Second)
phase.Store(1)
ips, ifaces, name, version, agent := m.fetchGuestAgentMetadata(context.Background(), client, "metadata", "node", "guest", 105, &proxmox.VMStatus{Agent: proxmox.VMAgentField{Value: 1}}, false)
if !reflect.DeepEqual(ips, first.IPAddresses) || !reflect.DeepEqual(ifaces, first.NetworkInterfaces) || name != first.OSName || version != first.OSVersion || agent != first.AgentVersion {
t.Error("failed HTTP refresh lost last-known metadata")
}
if !m.guestMetadataCache[key].fetchedAt.Equal(original.fetchedAt) || m.hasRecentGuestMetadataEvidence("metadata", "node", 105, time.Now()) {
t.Error("failed HTTP refresh became new evidence")
}
phase.Store(2)
unavailable := build()
if unavailable.GuestAgentStatus != "not-running" || unavailable.GuestAgentExpected {
t.Errorf("retained identity changed guest-agent runtime classification: %s/%t", unavailable.GuestAgentStatus, unavailable.GuestAgentExpected)
}
if unavailable.CPU != first.CPU || unavailable.Memory.Used != first.Memory.Used || unavailable.ID != first.ID || calls.Load() != 7 {
t.Error("independent status/identity was lost or unavailable agent was queried")
}
phase.Store(3)
recovered := build()
if recovered.GuestAgentStatus != "available" || recovered.AgentVersion != "2.0" || len(recovered.IPAddresses) != 1 || recovered.IPAddresses[0] != "192.0.2.11" || !m.hasRecentGuestMetadataEvidence("metadata", "node", 105, time.Now()) || calls.Load() != 11 {
t.Error("healthy poll did not recover current metadata and evidence")
}
})
}
}
internal/monitoring/guest_metadata_test.go
+2 -2
View File
@@ -1035,8 +1035,8 @@ func TestFetchGuestAgentMetadataPreservesCachedValuesOnEmptyResponses(t *testing
if len(updated.networkInterfaces) != 1 || updated.networkInterfaces[0].Name != "eth0" {
t.Fatalf("expected cache interfaces to remain populated, got %#v", updated.networkInterfaces)
}
if updated.fetchedAt.Before(cachedFetchedAt) {
t.Fatalf("expected cache timestamp to be refreshed, old=%v new=%v", cachedFetchedAt, updated.fetchedAt)
if !updated.fetchedAt.Equal(cachedFetchedAt) {
t.Fatalf("empty responses renewed retained metadata, old=%v new=%v", cachedFetchedAt, updated.fetchedAt)
}
}
internal/monitoring/monitor_pve_guest_helpers.go
+12 -4
View File
@@ -136,10 +136,15 @@ func (m *Monitor) recordGuestMetrics(allVMs []models.VM, allContainers []models.
if vm.Status == "running" {
diskRead, diskWrite, networkIn, networkOut := guestHistoryRates(vm.DiskRead, vm.DiskWrite, vm.NetworkIn, vm.NetworkOut, vm.IORateValidity)
memoryUsage, memoryUsed, diskUsage := historyMemoryUsage(vm.Memory), historyMemoryUsed(vm.Memory), vm.Disk.Usage
if strings.HasPrefix(vm.DiskStatusReason, "prev-") {
// A failed filesystem read can leave a positive cluster allocation,
// not just a retained prev- reading. Neither is current usage.
if vm.DiskStatusReason != "" {
diskUsage = -1
}
if vm.GuestAgentStatus == "deferred" {
// Only legacy unannotated memory needs the diagnostic heuristic.
// An explicit observation belongs to the selected source: deferred
// filesystem/QGA work cannot disqualify independent current memory.
if vm.GuestAgentStatus == "deferred" && vm.Memory.Observation.State == "" {
snapshot := m.previousGuestSnapshot(vm.Instance, "qemu", vm.Node, vm.VMID)
if snapshot == nil || snapshot.MemorySource == "previous-snapshot" || strings.HasPrefix(CanonicalMemorySource(snapshot.MemorySource), "guest-agent-meminfo") {
memoryUsage, memoryUsed = -1, -1
@@ -176,15 +181,18 @@ func guestHistoryRates(diskRead, diskWrite, networkIn, networkOut int64, validit
return diskRead, diskWrite, networkIn, networkOut
}
// Retained numbers remain useful for display, but a newly refreshed guest row
// cannot turn them into a new percentage or byte observation. The producer owns
// freshness; ordinary current TTL-cache hits and legacy readings remain valid.
func historyMemoryUsage(memory models.Memory) float64 {
if !memory.HasKnownUsage() {
if !memory.HasKnownUsage() || (memory.Observation.State != "" && memory.Observation.State != "current") {
return -1
}
return memory.Usage
}
func historyMemoryUsed(memory models.Memory) float64 {
if !memory.HasKnownUsage() {
if !memory.HasKnownUsage() || (memory.Observation.State != "" && memory.Observation.State != "current") {
return -1
}
return float64(memory.Used)
scripts/installtests/truenas_api_guidance_test.go
+366
View File
@@ -0,0 +1,366 @@
package installtests
import (
"context"
"encoding/pem"
"fmt"
"io"
"log"
"net/http"
"net/http/httptest"
"os"
"os/exec"
"path/filepath"
"regexp"
"strings"
"testing"
"time"
)
func trueNASAPIBlocks(doc string) ([]string, error) {
_, section, found := strings.Cut(doc, "### Testing and adding a connection (API)\n")
if !found {
return nil, fmt.Errorf("connection API guidance is missing")
}
section, _, _ = strings.Cut(section, "\n## Troubleshooting")
matches := regexp.MustCompile("(?s)```bash\n(.*?)```").FindAllStringSubmatch(section, -1)
if len(matches) != 3 {
return nil, fmt.Errorf("want separate preparation/test/save blocks, got %d", len(matches))
}
blocks := make([]string, 3)
for i, match := range matches {
blocks[i] = match[1]
}
return blocks, nil
}
func trueNASAPISafety(blocks []string) error {
for _, required := range []string{"set -eu", "umask 077", `[ -L "$HOME/.config" ]`,
`[ -L "$config_dir" ]`, `[ -L "$connection_file" ]`, `[ ! -f "$connection_file" ]`,
`chmod 700 "$config_dir"`, `chmod 600 "$connection_file"`, `vi "$connection_file"`} {
if !strings.Contains(blocks[0], required) {
return fmt.Errorf("unsafe file preparation: missing %s", required)
}
}
for i, block := range blocks[1:] {
for _, required := range []string{"set -eu", "umask 077", `result_file=$(mktemp "$HOME/.config/pulse/truenas-`,
"status=$(curl --disable --fail-with-body --silent --show-error", "--connect-timeout 5 --max-time 20",
"--proto '=http' --noproxy 127.0.0.1", `--header "@$HOME/.config/pulse/api-header"`,
`--data-binary "@$HOME/.config/pulse/truenas-connection.json"`,
`--output "$result_file" --write-out '%{http_code}'`,
fmt.Sprintf(`[ "$status" = %d ]`, 200+i)} {
if !strings.Contains(block, required) {
return fmt.Errorf("unsafe request %d: missing %s", i+1, required)
}
}
for _, forbidden := range []string{"--insecure", "--location", "--trace", "--verbose", "--config", "--retry", "X-API-Token:", "Authorization:"} {
if strings.Contains(block, forbidden) {
return fmt.Errorf("unsafe request: %s", forbidden)
}
}
if strings.Count(block, "curl ") != 1 || strings.Contains(block, " | ") {
return fmt.Errorf("each copied request must retain its own result and make only one POST")
}
}
return nil
}
func trueNASAPIGuidance(t *testing.T) []string {
t.Helper()
doc, err := os.ReadFile(repoFile("docs", "TRUENAS.md"))
if err != nil {
t.Fatal(err)
}
blocks, err := trueNASAPIBlocks(string(doc))
if err != nil {
t.Fatal(err)
}
if err := trueNASAPISafety(blocks); err != nil {
t.Fatal(err)
}
mirror, err := os.ReadFile(repoFile("frontend-modern", "public", "docs", "TRUENAS.md"))
if err != nil || string(mirror) != string(doc) {
t.Fatal("shipped TrueNAS guide must be identical to the verified recipe")
}
return blocks
}
func runTrueNASRecipe(t *testing.T, recipe, home, bin string, extraEnv ...string) (string, int) {
t.Helper()
ctx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, "bash", "-c", recipe)
cmd.Env = append(os.Environ(), "HOME="+home, "PATH="+bin+":"+os.Getenv("PATH"),
"http_proxy=", "HTTP_PROXY=", "https_proxy=", "HTTPS_PROXY=", "ALL_PROXY=", "all_proxy=", "NO_PROXY=", "no_proxy=")
cmd.Env = append(cmd.Env, extraEnv...)
output, err := cmd.CombinedOutput()
if ctx.Err() != nil {
t.Fatal("copied recipe exceeded its bounded test deadline")
}
if err == nil {
return string(output), 0
}
failure, ok := err.(*exec.ExitError)
if !ok {
t.Fatal(err)
}
return string(output), failure.ExitCode()
}
func TestTrueNASAPIGuidancePreparationIsPrivateAndFailClosed(t *testing.T) {
for _, mode := range []string{"new", "existing", "config symlink", "directory symlink", "file symlink", "non-regular", "mkdir failure", "chmod failure"} {
t.Run(mode, func(t *testing.T) {
home := t.TempDir()
bin := filepath.Join(home, "bin")
if err := os.Mkdir(bin, 0700); err != nil {
t.Fatal(err)
}
edited := filepath.Join(home, "editor-started")
if err := os.WriteFile(filepath.Join(bin, "vi"), []byte("#!/bin/sh\nprintf 'started' > \"$EDITOR_STARTED\"\n"), 0700); err != nil {
t.Fatal(err)
}
dir := filepath.Join(home, ".config", "pulse")
file := filepath.Join(dir, "truenas-connection.json")
if mode == "existing" || mode == "file symlink" || mode == "non-regular" {
if err := os.MkdirAll(dir, 0755); err != nil {
t.Fatal(err)
}
}
target := filepath.Join(home, "untouched")
if strings.Contains(mode, "symlink") {
link := file
if mode == "file symlink" {
if err := os.WriteFile(target, []byte("private-existing-key"), 0644); err != nil {
t.Fatal(err)
}
} else {
if err := os.Mkdir(target, 0755); err != nil {
t.Fatal(err)
}
link = filepath.Join(home, ".config")
if mode == "directory symlink" {
if err := os.Mkdir(link, 0755); err != nil {
t.Fatal(err)
}
link = dir
}
}
if err := os.Symlink(target, link); err != nil {
t.Fatal(err)
}
}
if mode == "existing" {
if err := os.WriteFile(file, []byte("private-existing-key"), 0644); err != nil {
t.Fatal(err)
}
}
if mode == "non-regular" {
if err := os.Mkdir(file, 0700); err != nil {
t.Fatal(err)
}
}
if strings.HasSuffix(mode, " failure") {
if err := os.WriteFile(filepath.Join(bin, strings.TrimSuffix(mode, " failure")), []byte("#!/bin/sh\nexit 1\n"), 0700); err != nil {
t.Fatal(err)
}
}
output, exit := runTrueNASRecipe(t, trueNASAPIGuidance(t)[0], home, bin, "EDITOR_STARTED="+edited)
_, editorErr := os.Stat(edited)
if mode != "new" && mode != "existing" {
if exit == 0 || !os.IsNotExist(editorErr) {
t.Fatal("unsafe or failed preparation reached the editor")
}
if strings.Contains(mode, "symlink") {
info, err := os.Stat(target)
if err != nil || info.Mode().Perm() != map[bool]os.FileMode{true: 0644, false: 0755}[mode == "file symlink"] {
t.Fatal("refused path modified its target permissions")
}
}
} else {
if exit != 0 || editorErr != nil {
t.Fatalf("safe preparation failed: exit %d, %s", exit, output)
}
for path, permissions := range map[string]os.FileMode{dir: 0700, file: 0600} {
info, err := os.Stat(path)
if err != nil || info.Mode().Perm() != permissions {
t.Fatal("credential path has incorrect permissions")
}
}
content, err := os.ReadFile(file)
if err != nil || (mode == "existing" && string(content) != "private-existing-key") {
t.Fatal("preparation did not preserve the existing credential file")
}
}
if strings.Contains(output, "private-existing-key") {
t.Fatal("preparation exposed private contents")
}
})
}
}
func TestTrueNASAPIGuidanceRejectsUnsafeRecipesAndExactParent(t *testing.T) {
blocks := trueNASAPIGuidance(t)
for name, edit := range map[string][2]string{
"config": {"curl --disable ", "curl "}, "body": {`--output "$result_file" `, ""},
"deadline": {"--max-time 20 ", ""}, "retry": {"curl --disable ", "curl --disable --retry 2 "},
"redirect": {"curl --disable ", "curl --disable --location "},
"inline token": {`"@$HOME/.config/pulse/api-header"`, `'X-API-Token: synthetic-secret'`},
"fail-open": {"set -eu", ""}, "wrong save status": {`[ "$status" = 201 ]`, `[ "$status" = 200 ]`},
} {
t.Run(name, func(t *testing.T) {
changed := make([]string, len(blocks))
for i, block := range blocks {
changed[i] = strings.ReplaceAll(block, edit[0], edit[1])
}
if trueNASAPISafety(changed) == nil {
t.Fatal("unsafe mutation was accepted")
}
})
}
if inputs := os.Getenv("PULSE_PROOF_INPUTS"); inputs != "" {
parent, err := os.ReadFile(filepath.Join(inputs, "truenas-parent.md"))
if os.IsNotExist(err) {
// Other source proofs can supply unrelated inputs. The mutation
// controls above still apply without this optional parent receipt.
return
}
if err != nil {
t.Fatal(err)
}
old, err := trueNASAPIBlocks(string(parent))
if err != nil {
t.Fatal(err)
}
if err := trueNASAPISafety(old); err == nil {
t.Fatal("unchanged exact parent guide passed new safety controls")
} else {
t.Logf("exact parent rejected: %s", err)
}
}
}
// Synthetic peers exercise the actual copied commands and deliberately echo
// both fixture secrets. This is credential/transport safety, not native
// TrueNAS collection, Pulse authorisation or a customer's setup acceptance.
func TestTrueNASAPIGuidanceCopiedRequestsKeepSecretsPrivate(t *testing.T) {
for _, scenario := range []string{"success", "HTTP error", "redirect", "wrong success status", "trusted HTTPS", "untrusted HTTPS"} {
for index, recipe := range trueNASAPIGuidance(t)[1:] {
t.Run(fmt.Sprintf("%s/%d", scenario, index+1), func(t *testing.T) {
realCurl, err := exec.LookPath("curl")
if err != nil {
t.Fatal("curl is required to verify copied TrueNAS guidance")
}
home := t.TempDir()
bin := filepath.Join(home, "bin")
dir := filepath.Join(home, ".config", "pulse")
for _, path := range []string{bin, dir} {
if err := os.MkdirAll(path, 0700); err != nil {
t.Fatal(err)
}
}
const token = "synthetic-pulse-secret"
const key = "synthetic-truenas-secret"
const body = `{"host":"https://synthetic.invalid","apiKey":"` + key + `"}`
for file, content := range map[string]string{"api-header": "X-API-Token: " + token, "truenas-connection.json": body} {
if err := os.WriteFile(filepath.Join(dir, file), []byte(content), 0600); err != nil {
t.Fatal(err)
}
}
capture := filepath.Join(home, "argv")
trace := filepath.Join(home, "must-not-trace")
wrapper := "#!/bin/sh\nprintf '%s\\0' \"$@\" > \"$TN_ARGV\"\nexec \"$TN_CURL\" \"$@\"\n"
if err := os.WriteFile(filepath.Join(bin, "curl"), []byte(wrapper), 0700); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(home, ".curlrc"), []byte("insecure\nlocation\ntrace = "+trace+"\n"), 0600); err != nil {
t.Fatal(err)
}
requests := make(chan string, 4)
server := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
payload, _ := io.ReadAll(r.Body)
requests <- strings.Join([]string{r.Method, r.URL.Path, r.Header.Get("X-API-Token"), r.Header.Get("Content-Type"), string(payload)}, "\n")
code := 200 + index
switch scenario {
case "HTTP error":
code = 400
case "redirect":
w.Header().Set("Location", "/must-not-follow")
code = 307
case "wrong success status":
code = 201 - index
}
w.WriteHeader(code)
fmt.Fprint(w, token+" "+key+" private-response")
}))
server.Config.ErrorLog = log.New(io.Discard, "", 0)
if strings.Contains(scenario, "HTTPS") {
server.StartTLS()
recipe = strings.Replace(recipe, "--proto '=http'", "--proto '=https'", 1)
if scenario == "trusted HTTPS" {
ca := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: server.Certificate().Raw})
if err := os.WriteFile(filepath.Join(home, "ca.pem"), ca, 0600); err != nil {
t.Fatal(err)
}
recipe = strings.Replace(recipe, "curl --disable ", `curl --disable --cacert "$HOME/ca.pem" `, 1)
}
} else {
server.Start()
}
t.Cleanup(server.Close)
recipe = strings.Replace(recipe, "http://127.0.0.1:7655", server.URL, 1)
output, exit := runTrueNASRecipe(t, recipe, home, bin, "TN_ARGV="+capture, "TN_CURL="+realCurl)
wantExit := 0
if scenario == "HTTP error" {
wantExit = 22
} else if scenario == "untrusted HTTPS" {
wantExit = 60
} else if scenario == "redirect" || scenario == "wrong success status" {
wantExit = 1
}
if exit != wantExit {
t.Fatalf("copied request exit %d, want %d: %s", exit, wantExit, output)
}
argv, err := os.ReadFile(capture)
if err != nil {
t.Fatal(err)
}
for _, secret := range []string{token, key, "private-response"} {
if strings.Contains(string(argv), secret) || strings.Contains(output, secret) {
t.Fatal("copied request exposed a fixture credential or response")
}
}
if _, err := os.Stat(trace); !os.IsNotExist(err) {
t.Fatal("curl configuration enabled tracing")
}
responses, err := filepath.Glob(filepath.Join(dir, "truenas-*.??????"))
if err != nil || len(responses) != 1 {
t.Fatal("request must retain one new private response file")
}
info, err := os.Stat(responses[0])
if err != nil || info.Mode().Perm() != 0600 {
t.Fatal("response file is not private")
}
if scenario == "untrusted HTTPS" {
if len(requests) != 0 {
t.Fatal("untrusted TLS peer received credentials")
}
} else {
if len(requests) != 1 {
t.Fatal("request was missing, redirected or retried")
}
path := "/api/truenas/connections"
if index == 0 {
path += "/test"
}
if got := <-requests; got != strings.Join([]string{"POST", path, token, "application/json", body}, "\n") {
t.Fatal("copied POST differs from the documented endpoint, header or JSON file")
}
response, err := os.ReadFile(responses[0])
if err != nil || string(response) != token+" "+key+" private-response" {
t.Fatal("complete sensitive response was not retained locally")
}
}
})
}
}
}
scripts/tests/test_truenas_docs.py
+13 -6
View File
@@ -27,10 +27,12 @@ PAYLOAD = {"name": "test-nas", "host": "https://nas.example.invalid",
def commands():
blocks = re.findall(r"```bash\n(.*?)```", DOC.read_text(), re.DOTALL)
preparation = next(block for block in blocks if "umask 077" in block)
requests = [block for block in blocks if block.startswith("curl ")]
return preparation, requests
section = DOC.read_text().split("### Testing and adding a connection (API)\n", 1)[1]
section = section.split("\n## Troubleshooting", 1)[0]
blocks = re.findall(r"```bash\n(.*?)```", section, re.DOTALL)
if len(blocks) != 3:
raise AssertionError("expected separate preparation, test and save recipes")
return blocks[0], blocks[1:]
@contextmanager
@@ -41,7 +43,7 @@ def server(status=200):
def do_POST(self):
body = self.rfile.read(int(self.headers["Content-Length"]))
requests.append((self.path, dict(self.headers), json.loads(body)))
self.send_response(status)
self.send_response(status if status != 200 or self.path.endswith("/test") else 201)
self.end_headers()
self.wfile.write(b'{"fixture":true}\n')
@@ -192,6 +194,11 @@ class TrueNASDocsTest(unittest.TestCase):
for secret in (TOKEN, NAS_KEY):
self.assertNotIn(secret, " ".join(argv))
self.assertNotIn(secret.encode(), result.stdout + result.stderr)
output = Path(argv[argv.index("--output") + 1])
self.assertEqual(output.parent, private)
self.assertEqual(stat.S_IMODE(output.stat().st_mode), 0o600)
self.assertEqual(output.read_bytes(), b'{"fixture":true}\n')
self.assertNotIn(b'{"fixture":true}', result.stdout + result.stderr)
return result
def test_test_and_save_send_header_and_json_from_private_files(self):
@@ -215,7 +222,7 @@ class TrueNASDocsTest(unittest.TestCase):
with server(status) as (port, _):
result = self.request(Path(temporary), command, port)
self.assertEqual(result.returncode, 22, result.stderr.decode())
self.assertIn(b'{"fixture":true}', result.stdout)
self.assertNotIn(b'{"fixture":true}', result.stdout + result.stderr)
if __name__ == "__main__":