resourceFromDockerContainer only encoded the container name in the
canonical identity, and ingestDockerContainer passed the raw docker
container ID as the source key. Both chooseNewID and the bySource
mapping then hashed only source + sourceID, so any path that briefly
produced the same source ID for two containers across different
docker hosts (a docker ps that returns an empty ID and falls back to
the container name in parseDockerInventoryContainerLine, a future
short-ID truncation, daemon-side identifier reset across recreate
cycles) collapsed both containers into one registry resource. The
second ingest hit mergeInto and overwrote the surviving Docker
payload — including HostSourceID and ParentID — with the colliding
host's data, surfacing on the Docker page as the "frigate@141" host
re-attribution flicker where a container briefly appears under the
wrong docker host before snapping back.
ingestDockerContainer now keys bySource[SourceDocker] by
host.ID/container/ct.ID via dockerContainerSourceID, and
seedSourceIDForResourceLocked produces the same shape for records-
path ingests so both paths resolve to the same resources. The
hashed candidate ID derived from this composite key gives each
(host, container) pair a unique registry entry.
BuildMetricsTarget for ResourceTypeAppContainer prefers
resource.Docker.ContainerID over the bySource SourceID so the
metrics target the API hands out continues to match what the
metrics store writes for container metrics (which still key by
the raw docker container ID).
Add a regression test that ingests two docker hosts each reporting
a container with the same colliding source ID and verifies both
survive as distinct resources with their original HostSourceIDs.
The test fails on the prior implementation (one resource instead
of two).
Audit pass on the Proxmox Mail sub-tab + PBS rows.
PBS and PMG adapters both had `instance.Uptime` /
`maxPMGUptime(instance.Nodes)` available but never projected to the
top-level `Resource.Uptime` field the canonical table reads. Same
projection pattern as the prior commits in this chain. Live mock now
reports `uptime: 2592016` on PBS rows and `uptime: 1555214` on PMG
rows.
The Proxmox > Mail sub-tab was rendering PMG instances via the local
`ProxmoxInfrastructureTable` helper (UnifiedResourceTable-backed) so
operators saw dashes for Disk I/O / Temperature / CPU% and got none
of the mail-flow context that matters for a Mail Gateway. Replace
with a new `ProxmoxMailGatewayTable` that reuses canonical shared
primitives (Card, Table, SearchInput, FilterButtonGroup, StatusDot)
and surfaces PMG-native columns: instance, version, nodes, uptime,
mail in, spam, virus, quarantine, queue, deferred. The page now
deletes the local `ProxmoxInfrastructureTable` helper entirely.
`ResourcePMGMeta` added to the frontend Resource type — the backend
already emits these fields on `pmg` rows; the frontend type just
hadn't surfaced them.
Browser verification (Playwright, chromium, live mock-mode):
- 9 tests pass; every-sub-tab operator-controls audit still green.
Contract-neutral bypass: PULSE_ALLOW_CONTRACT_NEUTRAL_COMMIT set.
Continues the audit chain (c7bdd11e0 → 5b94724bf → 0fcf9944b →
72dc2930b → 1e03c77f6). New Uptime projection on PBS/PMG adapters
is strictly additive; the new bespoke table lives in
features/proxmox/ and reuses canonical primitives only;
ResourcePMGMeta is a type surfacing of existing canonical payload
data.
Audit pass on Docker Hosts and vSphere Hosts.
Docker hosts had `docker.uptimeSeconds` and `docker.temperature` in
the payload but the canonical adapter never projected them to the
top-level `Resource.Uptime` / `Resource.Temperature` fields the table
reads, so both columns showed dashes. Same projection pattern as the
prior commits in this chain (c7bdd11e0 for host/node, 5b94724bf for
TrueNAS). Live mock now reports `uptime: 781089, temperature: 44.3°C`
on the Docker host row.
vSphere ESXi hosts had a richer column-fit problem: the generic
infrastructure table omits the vCenter inventory context that
matters most for the operator — datacenter, cluster, power state,
datastore count, VM count, vCenter — and shows dashes for Uptime /
Temperature because the upstream VMware inventory model does not
expose host bootTime today (canonical-model gap, not a projection
bug). New `VsphereHostsTable` reuses canonical shared primitives
(Card, Table, SearchInput, FilterButtonGroup, StatusDot) and
surfaces the vCenter-native columns; per-host VM count is computed
from the page scope client-side (no extra API calls). Mounted on
`/vmware/overview` in place of the generic infrastructure table.
New `DockerHostsTable` (mounted on `/docker/overview`) does the
parallel job for Docker / Podman hosts: runtime (Docker vs Podman),
runtime version, container count, Swarm role, plus CPU / Memory /
Uptime / Temp from the canonical metrics + new projections.
Browser verification (Playwright, chromium, live mock-mode):
- 9 tests pass; every-sub-tab operator-controls audit still finds
the canonical search input on /docker/overview and /vmware/overview
(now from each bespoke table's toolbar).
Targeted Go: `go test ./internal/unifiedresources/...` green.
Contract-neutral bypass: PULSE_ALLOW_CONTRACT_NEUTRAL_COMMIT set.
Continues the column-fit audit chain (c7bdd11e0 → 5b94724bf →
69f70a3fc). New Uptime/Temperature projection is strictly additive
on existing AgentData/DockerData payloads; the two new bespoke
tables live inside features/ and reuse canonical primitives only.
Follow-up to the K8s deployments fix (69f70a3fc), done as the full
audit pass the user asked for instead of one-page-at-a-time. Three
distinct column-fit issues remained across the platform pages:
1. **Top-level Uptime/Temperature were always dashes on agent-backed
tables (Docker Hosts, K8s Nodes, vSphere Hosts, TrueNAS Systems).**
The backend had the data in `agent.uptimeSeconds`,
`agent.temperature`, `proxmox.uptime`, and the max-sensor
`proxmox.temperature` projection, but `unifiedresources.Resource`
never surfaced them at the top level that the canonical table
reads. Add `Resource.Uptime` and `Resource.Temperature` and
populate them from `resourceFromHost` (for Pulse Agents) and
`resourceFromNode` (for Proxmox nodes). Resource types that have
no native uptime/temperature concept (k8s-deployment,
docker-service, k8s-cluster aggregates) leave them unset so
bespoke tables can hide the columns instead of rendering dashes.
Live sample: agent rows now expose
`uptime: 2592000, temperature: 78.95`.
2. **Docker Swarm services had no metrics at all.** Services are
cluster-scoped declarations, not running processes — they don't
have their own CPU/Memory/Disk/Disk I/O/Uptime/Temperature. New
`DockerServicesTable` reuses canonical shared primitives (Card,
Table, SearchInput, FilterButtonGroup, StatusDot) and surfaces
the operator columns the data actually backs: image, mode,
desired/running tasks, ports, host. Mounted on `/docker/services`
in place of the generic infrastructure table.
3. **K8s Clusters tab was just metric bars.** Clusters are
control-plane aggregates. Operator-meaningful columns are name +
context + version + counts of nodes/pods/deployments alongside
the aggregated CPU/Memory utilisation. New
`KubernetesClustersTable` renders those, computing per-cluster
counts client-side from the same resource scope already fetched
by the page (no additional API calls).
`ResourceKubernetesMeta` gains `version` and `server` fields (the
backend already emits them; the frontend type just hadn't surfaced
them). `ResourceDockerMeta` is introduced as the typed projection of
`resource.docker` for Swarm service rows (image, mode, replicas,
endpointPorts, swarm identity).
Browser verification (Playwright, chromium, live mock-mode dev runtime):
- 9 tests pass. The every-sub-tab operator-controls audit still finds
the canonical search input on /docker/services and
/kubernetes/overview (now provided by each bespoke table's
toolbar).
Targeted tests:
- `tsc --noEmit` clean
- `go test ./internal/unifiedresources/... ./internal/mock/...
./internal/monitoring/...` all green
Contract:
- `unified-resources.md` Extension Points: documents the top-level
`Resource.Uptime`/`Resource.Temperature` projections, the adapter
responsibility to populate them from nested AgentData/ProxmoxData,
and the convention that resource types without a native uptime or
temperature concept (k8s-deployment, docker-service, k8s-cluster
aggregates) leave them unset so bespoke platform-page tables can
hide the column instead of rendering dashes.
Contract-neutral bypass: PULSE_ALLOW_CONTRACT_NEUTRAL_COMMIT set
because this completes the column-fit audit started in 69f70a3fc —
no public contract shape changes (the new top-level fields are
additive optional projections of existing nested data; the two new
bespoke tables live inside features/ and reuse canonical primitives
only).
The follow-up to 8fb141c0a: deployment rows still showed dashes for
the Disk I/O columns because `metricsFromKubernetesDeployment`
populated NetIn/NetOut but not DiskRead/DiskWrite. Extend the
synthetic generator to scale realistic read/write rates with the
deployment's availability and replica count (read leads write
slightly, both modulated by availability) so the canonical Disk I/O
columns render alongside CPU / Memory / Disk / Network on every
deployment row under mock mode. Real-mode behaviour stays unchanged
(nil metrics until pod-metric aggregation lands).
Live verification (`/api/resources?type=k8s-deployment`): metrics now
carry cpu / memory / disk / netIn / netOut / diskRead / diskWrite for
every deployment row.
Tests: `go test ./internal/unifiedresources/...` green; existing
`TestMetricsFromKubernetesDeployment_*` and
`TestResourceFromKubernetesDeployment_*` continue to pass.
Contract-neutral bypass: PULSE_ALLOW_CONTRACT_NEUTRAL_COMMIT set
because this extends the existing synthetic K8s deployment metrics
helper with two additional rate fields without changing any public
contract shape, ownership, or boundary; the canonical helper
position is already recorded in unified-resources.md from 8fb141c0a.
The Kubernetes Deployments tab was rendering CPU / Memory / Disk as
"—" for every row because `resourceFromKubernetesDeployment` never set
the Resource.Metrics payload. Upstream Deployments do not natively
expose CPU/memory metrics (they are scheduling abstractions over their
controlled pods), so the unified adapter would have to aggregate pod
metrics into the owning deployment to back the table with real data.
That aggregation is a longer thread because mock pods today are tagged
with random `OwnerName` values that do not necessarily resolve to the
generated deployment names.
For the immediate platform-page quality fix, add
`metricsFromKubernetesDeployment(cluster, deployment)` to the
canonical metrics layer:
- Real-mode: returns nil (preserves current behaviour for live
clusters; deployments still show dashes until owner-based pod-metric
aggregation lands).
- Mock-mode: synthesizes deployment-stable CPU / Memory / Disk /
NetIn / NetOut values via `syntheticKubernetesDeploymentMetrics`,
scaled by the deployment's ready/desired/available replica state so
degraded deployments (ready < desired) read as elevated pressure on
surviving replicas, and by replica count so larger deployments show
proportionally higher absolute network throughput.
`resourceFromKubernetesDeployment` now calls the helper inline.
Live mock verification: `/api/resources?type=k8s-deployment` rows now
carry populated `metrics.cpu/memory/disk` percentages varied per
deployment, so `/kubernetes/deployments` renders bars and values
instead of dashes.
Tests:
- `go test ./internal/unifiedresources/... ./internal/mock/...
./internal/monitoring/...` all green.
- New `TestResourceFromKubernetesDeployment_PopulatesMetricsUnderMockMode`,
`TestResourceFromKubernetesDeployment_NilMetricsOutsideMockMode`,
`TestMetricsFromKubernetesDeployment_NilOutsideMockMode`, and
`TestMetricsFromKubernetesDeployment_SyntheticUnderMockMode` lock
the contract: real-mode returns nil; mock-mode synthesises and
degraded deployments read as elevated CPU vs healthy.
- Playwright (9/9) green against the live mock-mode dev runtime.
Contract updated: `unified-resources.md` Extension Points records the
canonical position of `metricsFromKubernetesDeployment` (real-mode nil
fallback today, mock-mode synthetic) and the future expansion path
(real pod-metric aggregation).
ActionAuditRecord gains a VerificationOutcome{status, evidenceSummary}
field with a closed enum (unknown/verified/unverified/failed). Existing
records read back as unknown by default via the normalizer and a new
SQLite column verification_outcome_json. The redaction pass scrubs the
evidence summary alongside other operator-authored text.
A new agentexec/verifier_postconditions.go registers postconditions for
qm.start, pct.start, docker.restart, systemctl.restart, and
kubectl.rollout, each parsed by verifier_postconditions_test.go.
Three pre-existing action JSON snapshot tests
(TestContract_ActionDecisionJSONSnapshot,
TestContract_ActionExecutionJSONSnapshot,
TestContract_UnifiedActionAuditsJSONSnapshot) now include the new
verificationOutcome field. The two flagged failing contract tests on
this branch
(TestContract_ActionDryRunOnlyExecutionErrorJSONSnapshot,
TestContract_RouterBridgesVerificationOntoActionCompleted) are
unrelated to this change and were left alone per lane D-002 scope.
When a maintenance window ends on a resource, the sentinel runs
deterministic checks (active alerts, Patrol findings, failed actions
since window start, basic post-window metric recovery) and writes a
durable LoopReport. Operators can list reports per resource, mark them
reviewed, or rerun verification immediately. UI surfaces the section in
the resource detail drawer; scoped Patrol runs and Assistant deep-link
are deferred until those entry points stabilise.
Pre-existing bug exposed by slice 33: the contract test had two
filepath.Join(".", "actions.go") entries in its requiredSnippets map.
Go silently drops duplicate map keys (last entry wins), so the first
list — including the canonical ErrActionPlanDrift pin and the new
ErrResourceRemediationLocked pin from slice 33 — was never actually
running. The test reported PASS while pinning nothing it claimed to.
Merges both lists into a single actions.go entry so every snippet
runs. Also corrects ErrActionPlanDrift's expected alignment to the
single-space form gofmt produced when the var block grew the new
sentinel.
Third compounding slice on per-resource operator state, completing the
suppression triangle: maintenance window (slice 31), intentionally
offline (slice 32), and now NeverAutoRemediate at the action-broker
boundary. When the operator has set NeverAutoRemediate=true on a
resource, executeCommandWithAudit refuses the dispatch even with a
valid approval and matching plan hash — per-resource operator intent
outranks per-action approval.
The check sits next to the drift refusal in the same audit hot path:
fail-open on store-lookup error (logged but doesn't block), persist a
Failed audit record with `resource_remediation_locked:` prefix on the
ErrorMessage so audit-UI filters and alert rules can branch on the
stable token. Returns ErrResourceRemediationLocked sentinel for
caller-side branching.
Reuses the existing actionAuditStore handle (which is the unified
ResourceStore from slice 29), so no new provider plumbing needed —
the broker already has the data path it needs.
Foundational wedge for the operator-set per-resource state feature
(intentionally offline, never auto-remediate, maintenance windows,
criticality hint). Defines the canonical schema, validation, and
persistence contract; subsequent slices wire in the API surface and the
finding-suppression / action-broker integrations.
ResourceOperatorState carries four narrow operator-intent fields plus
attribution metadata. The shape is intentionally fixed (not a freeform
metadata bag) so consumers have a stable contract to honor.
ValidateResourceOperatorState rejects malformed records with a stable
sentinel (partial maintenance window, end <= start, unknown criticality);
NormalizeResourceOperatorState handles whitespace and case.
Adds GetResourceOperatorState / SetResourceOperatorState /
ClearResourceOperatorState to the ResourceStore interface, implements
both on SQLite (new resource_operator_state table) and Memory. Set is
upsert by canonical_id; Clear is idempotent. Tests cover IsEmpty,
IsInMaintenanceAt window semantics (half-open interval, partial windows
treated as no window), validation rejections, and round-trip
persistence on both stores.
Closes the read-after-write piece of the user's act-with-control loop:
"Act with control: ask for approval, execute through Pulse tools, then
verify." Until now Pulse authored verification narrative copy at
approval time but never actually executed the check.
Adds ActionVerificationResult to internal/unifiedresources/actions.go
and embeds it as ExecutionResult.Verification so the existing
result_json column persists the outcome without a schema migration.
The struct records: did the check run, what command did Pulse send,
what came back, did it succeed, when, and any failure note.
Adds VerificationCommandForCommand in tools_control.go that derives
the per-class read-after-write check used by the broker:
- service-restart / service-start / service-reload / service-stop
→ systemctl is-active <unit>
Container classes (container-restart, container-stop) are
intentionally deferred to pulse_docker's existing tool-level
docker inspect verification — adding a broker-level dispatch would
double-run the same check.
executeCommandWithAudit now runs the derived verification command via
the same agent path immediately after a successful dispatch, captures
output and exit code, and writes the result onto
ExecutionResult.Verification before recordActionExecutionResult
persists. If the check returns non-zero, Verification.Success=false
with a Note explaining the exit code so the audit history honestly
shows that Pulse ran the action but couldn't confirm it took.
Verification is best-effort and class-scoped: unknown command shapes
leave Verification nil rather than fabricating a verified=true entry,
matching the no-fabrication boundary from Impact / preflight authoring.
Tests cover:
- VerificationCommandForCommand derives the right check per class,
shell-escapes single quotes in unit names, and returns "", false
for docker (deferred) and unknown commands.
- TestExecuteCommandWithAuditRunsClassDerivedVerificationAfterDispatch
- TestExecuteCommandWithAuditMarksVerificationFailedWhenReadbackDoesNotConfirm
- TestExecuteCommandWithAuditSkipsVerificationForUnclassifiedCommands
Updates ai-runtime contract with the read-after-write rule and
container-class deferral. Adds Completion Obligation #21 to
unified-resources pinning ExecutionResult.Verification shape and the
no-fabrication boundary. Extends TestActionExecutionContractStaysAPIOwned
to pin ActionVerificationResult struct and the Verification field on
ExecutionResult.
The action audit log is plaintext SQL. Operators sometimes paste
secrets into a natural-language `reason` field ("rotate the key
sk-abc123 because it leaked"), and command output sometimes echoes
tokens. Both flows previously persisted unredacted, leaving the
audit history as a leaked-credential surface.
Adds RedactAuditText / RedactAuditRecord in a new
internal/unifiedresources/audit_redaction.go with a curated regex set
that targets the credential shapes most likely to be real secrets:
- URL with embedded basic-auth credentials (https://user:pass@host)
- Authorization: Bearer <token> and x-api-key: <token> headers
- Query-string secret params (?api_key=, &token=, &access_token=)
- JSON-style secret fields ("api_key": "...", "password": "...")
- Env-style or CLI-style secret assignments (PASSWORD=, api_key=)
- OpenAI/Anthropic-style API keys (sk-...)
The set is intentionally narrower than the patrol-failure redactor in
internal/ai/patrol_runtime_failure.go: it does NOT strip arbitrary
URLs, because operators legitimately reference runbooks, ticket
links, and GitHub issues in audit reasons. Only patterns very likely
to be real secrets are touched.
Wired at the persistence boundary (top of each store method) so all
record paths get redaction uniformly:
- SQLiteResourceStore.RecordActionAudit
- SQLiteResourceStore.RecordActionExecutionStart
- SQLiteResourceStore.RecordActionExecutionResult
- MemoryStore.RecordActionAudit
- MemoryStore.RecordActionExecutionStart
- MemoryStore.RecordActionExecutionResult
Plan, Approvals, and identity fields are left alone — they are
produced by Pulse, not operators or external command output, so they
do not need redaction (and changing them would break PlanHash drift
detection).
Verification artifacts:
- audit_redaction_test.go: pattern-by-pattern coverage plus public-URL
passthrough and empty-string passthrough; full RedactAuditRecord
shape test (Reason + Params string values + Result output redacted;
non-string Params and Plan untouched).
- code_standards_test.go: pins RedactAuditRecord call sites in
store.go and helper signatures in audit_redaction.go so future
refactors cannot silently bypass redaction.
- registry_test.go: cross-cutting integration test that exercises the
registry-store boundary (MemoryStore.RecordActionAudit) and asserts
redaction is applied while Plan fields stay untouched.
Adds new Completion Obligation #20 to unified-resources contract
pinning the redaction-at-persistence-boundary rule and the
no-touch-Plan-fields invariant.
Slice 13 added the drift refusal path but only logged at WARN level —
the audit history showed nothing, so an operator reviewing the action
trail could not see "Pulse caught this drift attempt." Now the drift
branch writes a Failed audit record with Result.ErrorMessage prefixed
"plan_drift:" and dispatches a Failed lifecycle event before returning
ErrActionPlanDrift.
The record carries the same Request, Plan, and Approvals snapshots that
a normal audit record would, so the operator-facing audit row shows
exactly what was attempted and what was approved. The "plan_drift:"
prefix is a stable token for downstream surfaces (audit UI filters,
alert rules) to distinguish drift refusals from generic execution
failures.
Extends TestExecuteCommandWithAuditRefusesPayloadDriftAgainstApprovedPlan
to assert the audit record exists with State=Failed and the plan_drift
error message after refusal. Updates the code-standards snippet check
for ErrActionPlanDrift to match gofmt's actual alignment in actions.go.
ai-runtime contract pinned with the audit-record-on-drift rule.
PlanHash existed on ActionPlan as the contract for "the operator
approved exactly this (command, target, reason) combination" but the
broker never validated it: at execute time the freshly-recomputed hash
was overwritten by the approved plan's hash via mergeApprovedActionPlan
without comparison. A drifted payload (LLM re-emits with different
command, agent ID changes between approval and execute, malicious
injection) would run under a stale approval.
Fix: at the dispatch boundary in executeCommandWithAudit, recompute the
approval-equivalent hash from the actual payload using approvalPlanHash
(same function used at approval-creation time, so direct comparison is
meaningful), compare to plan.PlanHash, and refuse with a new
ErrActionPlanDrift error when they differ. The drift refusal also logs
at WARN level with action_id, approval_id, and both hashes so audit
review can see when drift was caught.
When approvedHash is empty (older approval records or contract paths
that did not author one), validation is skipped and existing behavior
is preserved.
Adds two tests:
- TestExecuteCommandWithAuditRefusesPayloadDriftAgainstApprovedPlan:
approval is for "systemctl restart workload"; payload at execute
time is "rm -rf /var/log/pulse"; expects ErrActionPlanDrift and no
agent dispatch.
- TestExecuteCommandWithAuditAllowsMatchingPlanHash: same payload
matches the approved hash; expects normal dispatch.
Updates two pre-existing tests that used stub PlanHash strings:
- TestExecuteCommandWithDeniedApprovalDoesNotDispatch: was testing the
denial path with PlanHash:"sha256:test". Now uses the real approval-
equivalent hash so denial fires (not drift) and the test still
isolates the denial behavior.
- ControlledConsumesApprovedCommandWithResolvedRoutingTarget: same fix
pattern.
Extends TestActionExecutionContractStaysAPIOwned in code_standards
test to pin ErrActionPlanDrift's existence in actions.go so future
refactors cannot silently downgrade drift into a generic error kind.
Native action path (executeNativeActionWithAudit) is left for a
follow-up: it has the same drift gap but uses actionPlanHashForParams
which is shaped differently from approvalPlanHash, so a sound fix
needs a coherent canonical hash function rather than just adding the
check. Contracts pinned in ai-runtime and unified-resources (new
Completion Obligation #20).
Parse the /proc/mdstat operation keyword for mdadm arrays and propagate it through host reports, models, unified resources, monitoring views, alert metadata, and AI storage summaries.
Treat recovery and reshape as rebuild signals while silencing routine check and resync maintenance, with fallback rebuild detection only when no mdstat operation is available.
Tests cover mdstat operation parsing plus recovery, check, and resync alert behavior.
Fixes#1446
rcourtman