The installer compares the agent binary it downloaded against the server that
served it, stripping a leading "v" so "v6.0.4" and "6.0.4" match. It did not
strip semver build metadata, so a server built from a working tree reporting
"6.2.0-rc.8+git.46.g98a638e00.dirty" never matched the "v6.2.0-rc.8" agent it
had just served, and the mismatch warning fired on every correct development
install.
This is the warning's whole job, so a false positive is expensive. It is the
only client-side signal that a stale agent was downloaded, and because it
always fired it read as background noise. That is exactly how a genuinely
stale v6.0.5 agent was installed on a live host earlier today: the warning
was there, above the install output, and looked like the one that always
appears.
Strip build metadata from both sides before comparing, keeping the prerelease
suffix because 6.2.0-rc.8 and 6.2.0 are genuinely different releases. This is
the same release-identity reduction the server applies when deciding whether a
local agent artifact is fresh enough to serve; the contracts now state that
one definition governs both ends rather than leaving each side to invent its
own.
Guarded by a test that pins both normalisation steps and exercises the
comparison across the cases that matter: the dev-server shape that used to
warn wrongly, the stale-download shape that must still warn, and a prerelease
against its release. Verified to fail when either strip is removed.
Completes the wrapper-teardown rule across the remaining branches. The QNAP
install and both uninstall paths still used a bare pkill -f
"start-pulse-agent.sh" and still stopped the agent before its wrapper.
The bare pattern is narrower than it looks and wider than it should be. It
does NOT match a co-installed agent's supervisor, so the sibling case was
already safe; what it does match is anything where the unescaped dot stands in
for another character and the unbounded tail keeps going, including a .bak
copy of the wrapper and an editor session holding it open. Escaping the dot
and bounding the far end removes both without narrowing the intended match.
Ordering is the more consequential half. A wrapper is a watchdog, so stopping
the agent while its wrapper still loops only races the respawn. QNAP and the
uninstall paths now stop the supervisor first, which is what the contracts
already required of every branch that writes and launches a wrapper.
Uninstall keeps a deliberately broader match than install, with no leading
path separator, so it still reaches a wrapper invoked by a relative path or
stranded at a superseded location. Both contracts now carry that distinction
and the teardown ordering rule, which each had stated only for install.
Guarded by two tests that pin every wrapper kill in the file rather than one
branch: one requires the escaped dot and the bounded tail everywhere, the
other walks each stop block and fails if an agent kill precedes its wrapper
kill. Both were confirmed to fail against the pre-fix QNAP block.
The Unraid install path killed the running agent but never the wrapper
supervising it, then appended a second wrapper at the end of the install. The
survivor and the newcomer both loop trying to own the same agent id, and
because the old wrapper is a watchdog it respawns the agent mid-install with
the previous binary and arguments. Observed on a live Unraid host: a
supervisor from a July install was still running beside the one the reinstall
had just started.
Nothing reports this as a failure. It presents later as an agent that
restarts on its own or reverts to superseded arguments.
Stop the wrapper first, then the agent: killing a supervised agent while its
wrapper still loops only races the respawn. The pattern matches the trailing
path segment so a wrapper left at an older storage location is caught too,
with the dot escaped and the far end bounded so a co-installed agent's
supervisor (start-pulse-agent-prod.sh) is not.
The QNAP branch already stopped its wrapper, which is what made the Unraid
omission visible; the contracts now require every wrapper-writing branch to
own the same teardown.
Guarded by two tests: one pins that the Unraid branch stops the wrapper and
does so before the agent, the other pins that the wrapper pattern spares a
sibling supervisor, with a premise check that the loose pattern really does
match so neither assertion can pass vacuously.
pkill -f matches the whole command line and "^" only anchors the start, so
"^/usr/local/bin/pulse-agent" also matches "/usr/local/bin/pulse-agent-prod".
On a host running a second agent whose binary name shares the prefix, every
install, every upgrade, and every restart of the generated Unraid wrapper
silently killed the other agent too. Confirmed on a live dual-agent Unraid
box: the old pattern matched both the dev agent and the production dogfood
agent, the bounded pattern matches only its own.
The wrapper is the worst of the three because restarting through it is the
documented runbook step, so the collateral kill repeats every time an
operator follows it.
Bound the far end of each binary-anchored pattern with ([[:space:]]|$), and
swap the bare pkill -9 -f "pulse-agent" for -x on the exact process name,
which keeps that site's deliberate path-agnostic intent while excluding the
sibling. The pkill -x sites were already safe and are unchanged.
Guarded by two tests: one pins that no binary-anchored pkill in the installer
is left unbounded, the other exercises POSIX ERE semantics against the two
command lines a dual-agent host presents, including a premise check that the
unbounded pattern really does match the sibling so the assertion cannot pass
vacuously.
pct exec runs with PATH=/sbin:/bin:/usr/sbin:/usr/bin, which does not
include /usr/local/bin where the pulse binary is linked. Every first-run
instruction that invoked a bare `pulse` through pct exec therefore failed
with exit 127, including the command the installer itself prints at the
end of an LXC install.
Reproduced on Proxmox VE 9.1.9 and 8.3.0:
pct exec <ctid> -- pulse --version
-> exit 127, Failed to exec "pulse"
pct exec <ctid> -- /usr/local/bin/pulse --version
-> Pulse v6.2.0-rc.8
The installer now prints $BINARY_LINK_PATH and $UPDATE_HELPER_PATH rather
than a bare name, so both stay correct when a custom service name moves
them. The update helper was only affected for custom service names, since
the default /bin/update already sits on the pct exec PATH.
Docs updated in INSTALL.md, FAQ.md and TROUBLESHOOTING.md plus their
frontend-modern/public/docs mirrors. The setup wizard prints the same
broken command and is fixed separately.
The agent_runtime verify-only build pointed its registry buildcache at
ghcr.io/<owner>/pulse-agent, a package no workflow publishes. A cache ref
creates the package it points at, so every release recreated it and the
repository's Packages sidebar advertised an empty pulse-agent entry that
reads like a pullable agent image sitting beside the real pulse,
pulse-control-plane and pulse-chart/pulse packages. The only release tag it
ever carried is a stray v6.0.0-rc.3.
Move the cache to ghcr.io/<owner>/pulse:agent-buildcache, alongside the
runtime stage's existing pulse:buildcache tag, so it lives in the package
that actually ships and shares backend-builder layers with the runtime
build.
TestAgentBuildCacheDoesNotResurrectPulseAgentPackage locks this in. It scans
every workflow for registry-qualified refs to the pulse-agent package,
collapsing ${{ }} expressions first so an interpolated owner cannot hide one
behind its spaces, and asserts both the new cache target and the chart's
published agent image default. Verified red on a restored buildcache ref and
red again on a ref hidden behind ${{ github.repository_owner }}.
Two stale claims corrected while here. The chart values comment said the old
agent.image default was never published, when it did carry v6.0.0-rc.3. The
deployment-installability contract called rcourtman/pulse the only image
publish-docker.yml pushes, when the same workflow also pushes the MSP
pulse-control-plane image. The contract now also forbids buildcache
references to the pulse-agent package, which is what let this regress.
Deleting the orphaned package itself remains a manual GHCR step.
Advance the governed install, Docker, and Helm metadata together; add the RC9 release packet for the post-RC8 notification, lifecycle, and resource-read fixes; and record the existing mobile-candidate compatibility decision.
The mock:on and mock:off scripts used GNU 'sed -i' syntax, which fails on
the default macOS BSD sed. The '||' fallback then appended a fresh
PULSE_MOCK_MODE line on every invocation, so a local .env accumulated three
contradictory entries.
They also targeted the repo-root .env, which hot-dev.sh does not consult
when choosing the data directory. hot-dev.sh reads the canonical flag from
tmp/dev-config/.env, written authoritatively by toggle-mock.sh. The npm
wrappers were therefore a no-op that corrupted .env as a side effect, while
leaving the operator believing mock mode had switched.
Point them at toggle-mock.sh, which handles BSD sed, writes the canonical
file, syncs the runtime env and restarts the managed runtime. Add
mock:status and mock:edit; the hot-dev startup banner already advertised
'npm run mock:edit', which did not exist.
Extend the deployment-installability contract to cover the mock wrappers
alongside the existing repo-root dev entry rules, and add a regression test
asserting the wrappers delegate rather than rewriting the flag inline, and
that every mock command named in the hot-dev banner exists.
d00249fd7 added frontend-modern/src/pages/__tests__/Alerts.readOnly.test.tsx
without adding it to the pinned proof list in the canonical completion
guard, so the guard's expected list no longer matched what the registry
derives. canonical-governance has been red on main since that commit, and
the pre-commit hook has been refusing every commit locally for the same
reason. Six commits landed in that window, including the v6.2.0-rc.7 release
preparation.
The delta is one file added and none removed, so this restores the snapshot
rather than lowering the bar. A removal would have meant proof coverage was
dropped, which is a regression to investigate rather than a pin to update.
Contract-Neutral: governance pin restored to match the registry
The evaluation mode added earlier today was hollow. An unlicensed control
plane starts, but release-build client runtimes only trust entitlement
leases chained to a Pulse-signed licence, so its client workspaces ran
without the capabilities the provider was evaluating. Standing the stack
up proved the portal and the isolation boundary and nothing else.
setup.sh now requests a capped evaluation licence from the licence server
when no licence path is set, sending only the public half of the key it
generated locally. It degrades rather than blocks: a missing key, an
unreachable server, or a licence-free response leaves the install
unlicensed with an explicit warning, an existing licence on disk is
reused, and PULSE_PROVIDER_MSP_SKIP_EVAL_LICENSE skips it for air-gapped
hosts. Guarded with an if-test rather than a trailing true inside the
command substitution, because the derive helper calls die and exit in a
subshell is not a status that can be caught, so setup.sh aborted under
set -e.
Second fix. The lease capability ceiling was selected by licence
presence, so an unlicensed provider control plane fell through to the
Pulse-hosted branch and minted leases claiming relay, mobile and push,
which a provider deployment cannot serve and which previously caused
repeating relay registration failures in client runtimes. The ceiling now
follows hosting via SetProviderHosted. providerChained keeps its narrower
meaning of having a licence available to embed.
Third. Corrects a sentence I wrote into cloud-paid.md this afternoon
claiming msp_eval carries the same MSP capabilities. A plan version
selects the workspace cap; it does not entitle a workspace.
The regression test was negative-tested by reverting the ceiling selector
and confirming it catches relay. All three setup.sh degradation paths
were exercised directly. Licence server side is pulse-pro 7f6a319 and is
not live until the next deploy-license-server run.
Two follow-ups from auditing this morning's triage commits.
2a1bf2839 placed the config-directory guard after safe_systemctl enable, so
a failure to create the directory would have left the update timer enabled
while the installer reported automatic updates as disabled and never wrote
system.json. Move the guard above both the asset install and the timer
enable so a failure leaves nothing half-configured, and pin the ordering
with a test.
419385ccf regenerated the whole bundle-size baseline when only the FilterBar
chunk was over budget. That raised the recorded baseline for 22 other chunks
that were still passing, absorbing about 15 kB of accumulated drift and
resetting their headroom. Restore every other chunk and both totals to their
previous values and bump FilterBar alone, so the drift those chunks have
already accrued stays visible to the next change.
Contract-Neutral: scope corrections to earlier triage commits
Refs #1663
Two mandatory round-trips stood between an interested MSP and their first
screen, and neither was technical.
setup.sh required four image digests shipped as literal <pin>
placeholders, so the only way to get them was to ask. All four images are
publicly readable, so there was never anything to hand out. setup.sh now
resolves each blank pin to an immutable digest from its published tag via
buildx imagetools and writes it back to .env; hand-set values are left
alone.
setup.sh then died outright without a licence file, so nobody could start
the stack, create a workspace, or see the portal until a human minted a
licence for them. The control plane already ran unlicensed via
ProviderMSPPlanSourceEnvFallback; only the installer refused. A licence
path that is set but missing is still a hard failure, since that is a
misconfiguration rather than a choice.
Unlicensed now means evaluation rather than the cheapest paid tier. The
env fallback defaulted to msp_starter, handing every unlicensed
deployment the full 5-client Starter allowance and leaving no boundary
between evaluating and buying. Adds msp_eval at 2 workspaces: same
capabilities, smaller cap, not purchasable, not on the public ladder.
An isolation guarantee is the one claim a provider cannot evaluate from a
screenshot, and both MSP leads this year went quiet at exactly this step.
Contracts: cloud-paid records the unlicensed plan rule and the
strictly-below-paid invariant; deployment-installability records
credential-free, correspondence-free installability.
Verification: TestMSPEvalCapStaysBelowCheapestPaidTier,
TestCanonicalizePlanVersion_MSPEval,
TestProviderMSPSetupScriptSupportsUnlicensedEvaluation. The last was
negative-tested by reintroducing a <pin> placeholder and confirming it
fails. ensure_image_pins exercised against the live registries.
licensing, cloudcp, control-plane and installtests all green.
Reinstalling after removing /etc/pulse reaches setup_auto_updates before
setup_directories has recreated the config directory. The system.json
write then failed with "No such file or directory" while the run still
printed that automatic updates were enabled, so the installer reported a
state it had not reached.
mkdir -p the config directory first, and fall back to disabling
auto-updates when it cannot be created.
Contract-Neutral: installer behavioural fix, no contract delta
Refs #1663
Every unattended update created another full config snapshot under
config-backups (or next to the config dir) and nothing ever pruned old
ones, so small root filesystems filled up within days (#1646, reported
on the hardened-unit fallback path where snapshots land under the
install dir). backup_existing now keeps the five newest snapshots and
removes the rest after each successful copy.
Contract-Neutral: installer config backup rotation; shell-only fix, no runtime contract
install.ps1 set ServicePointManager's certificate callback to a
PowerShell scriptblock. ServicePointManager can invoke that callback on
a worker thread that has no runspace, where a scriptblock delegate
fails closed, so agent downloads died with a TLS error even though the
certificate policy would have accepted them (hit by an rc.6 tester with
Skip TLS verification enabled). The copied install command already
migrated to a compiled validator for exactly this reason in rc.6; the
downloaded script now does the same. Validation policy is unchanged,
with fingerprint pinning first, then insecure, then clean chains, then
the custom CA. Verified with pwsh covering all four policy branches
plus the callback running on a runspace-free worker thread.
Contract-Neutral: installer TLS callback implementation moved to a compiled type; same validation policy, no payload or endpoint change
The paid-runtime R2 prefix was derived from the wall-clock date, and the
enterprise build was always dispatched as a fresh rebuild. Rerunning the
failed publish job after a transient promotion failure therefore hit the
enterprise R2 overwrite guard (same run id, existing prefix) and the
Definitive Release Verdict could never go green by rerun, as on
v6.2.0-rc.6. The prefix date now comes from the run's createdAt, which
is stable across rerun attempts, and the build dispatch passes
reuse_existing_packet=true so pulse-enterprise validates the packet the
earlier attempt uploaded, skips the rebuild, and only the promotion
re-executes against it. The deployment-installability contract now
records rerun recoverability as part of the promotion boundary.
A box with the binary still at /opt/pulse/bin/pulse but /etc/pulse, the
systemd unit and the /usr/local/bin symlink deleted by hand takes the
update path ("Reinstalling version ..."), which assumed the previous
install's environment still existed. With auto-updates enabled the run
died at setup_auto_updates writing system.json into the missing config
dir; without them it printed a success completion while systemctl
enable/start had failed with "Unit pulse.service could not be found",
softened into the unprivileged-container note (#1663).
Both the --version and menu update flows now run setup_directories and
recreate the unit file when it is missing (an existing unit stays
untouched so user customizations survive normal updates), and
start_pulse refuses to report success when the unit does not exist at
all. Documented as a deployment-installability contract invariant.
Fourth candidate on the v6.2.0 line, superseding v6.2.0-rc.3 and still following stable v6.1.2. This hardening cut includes action recovery, kiosk containment, alert and notification accounting, node-restricted Proxmox storage, container-runtime selection, and host CPU sampling fixes that landed after rc.3. Version pins move to 6.2.0-rc.4 across the repo root, Docker bootstrap defaults, and Helm metadata; stable install pointers remain on v6.1.2 until governed promotion.
The Build and Test workflow ran everything in one serial job — frontend
lint/tests/build, the full go test -race sweep, and benchmarks — for a
37-55 minute wall clock on every push. The job now splits into parallel
jobs: frontend (lint, unit tests, type-check, bundle build and size gate),
backend tests sharded three ways with internal/api on its own shard since
it alone takes ~10m under -race, script smoke tests with the backend
build, and benchmarks with the existing baseline cache flow. Backend jobs
satisfy the frontend embed with a stub index.html, which unit tests do
not assert against; the real bundle is still built and size-gated by the
frontend job and by release builds. Expected wall clock is roughly the
slowest single job, 10-15 minutes.
A new changes job skips the build/test jobs when a push touches only
docs/ and Markdown, mirroring the path filters Core E2E already has. The
gitleaks secret scan deliberately stays unconditional since this workflow
is the only per-push scan and docs can leak secrets too.
scripts/dev-prepush.sh runs the checks that most often turn main red,
scoped to the outgoing commits: the canonical completion guard in CI
mode, the registry snapshot tests when registry.json changed, the
mutation registry audits, build plus tests for touched Go packages, and
the frontend type-check. It exists because CI verdicts arrive well after
the push; today's unclassified force-fail route would have failed it
locally in about a minute.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The installer's discover_rootless_container_runtime only globbed
/run/user/* sockets and never consulted the system Docker daemon, so a
socket-activated rootless Podman API socket (alive only for root's login
session on Debian/OMV) won discovery over a healthy rootful Docker and
pinned PULSE_DOCKER_RUNTIME=podman plus CONTAINER_HOST/PODMAN_HOST/
XDG_RUNTIME_DIR into the agent unit. The env-application block also ran
for explicit --enable-docker installs. Rootless discovery now defers to
system_docker_runtime_is_active (docker info with DOCKER_HOST stripped,
or a live /var/run/docker.sock probe) before touching rootless sockets.
On the agent, detectRuntime short-circuited to podman whenever the
preference said podman, mislabeling connections that actually fell
through to the Docker socket and disabling Swarm collection. The
preference is now an ordering hint: a podman-preferred connection landing
on a docker endpoint reports docker, while unlabeled endpoints with no
runtime signals still honor the pin. When the bound socket disappears
mid-run the agent re-runs runtime discovery after three consecutive
daemon-unavailable collects, swapping the connection behind a
swappableDockerClient so concurrent goroutines keep a stable handle.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second candidate on the v6.2.0 line, superseding v6.2.0-rc.1 and still
following stable v6.1.2. This is a hardening and bugfix candidate:
auto-update reliability including in-place migration of already-deployed
update units, PBS backup attribution across multiple Proxmox clusters,
Proxmox installer registration including per-canonical-type bootstrap
grants on combined PVE+PBS hosts, Patrol readiness streaming transport
plus a completed-verdict readiness gate, discovery-policy DNS and SSH
retry churn, explicit per-metric threshold off toggles, request-derived
SSO callback URLs, and the new Entra ID SSO guide. Version pins move to
6.2.0-rc.2 across the repo root, Docker bootstrap defaults, and Helm
metadata per the deployment-installability contract; stable install
pointers remain on v6.1.2 until governed promotion.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Richard Courtman