The internal/api race suite now routinely exceeds the old 20-minute package timeout on hosted runners while passing. Set a governed 30-minute package timeout and 40-minute release job ceiling, pin the relationship with contract tests, and refresh the rc.4 packet with the fixes landed since preparation.
The installer staged the download in /tmp and installed the runtime
binary to /usr/local/bin, both on the small RAM-backed QTS/QuTS hero
root, and the boot wrapper copied 34MiB back onto that root at every
boot. Roots without ~50MiB of headroom could not install at all, and
setting TMPDIR only moved the staging half of the requirement.
QNAP's own QPKG packages execute from the data volume, so do the same:
relocate the install dir to the data volume's state dir before the
preflight and download, default TMPDIR there too, skip the boot-time
self-copy when the stored and runtime binaries are one file, and remove
a pre-relocation runtime copy from /usr/local/bin to give that space
back. Split layouts with an operator-supplied state dir keep the copy
semantics. The rendered wrapper is exercised in both layouts by the
installer tests.
Refs #1617
Contract-Neutral: Refs #1617: QNAP installer layout fix with its deployment-installability contract clause staged in this commit; residual proof policies for unrelated boundaries do not apply to this shell-only change
Live proof on a real systemd host (Debian 12 container, full
install/report/update/uninstall cycle) caught what no unit test did:
NoNewPrivileges=true blocks sudo outright, so the --grant-smart and
--grant-pct helpers failed inside the service and SMART/pct silently
disappeared while the install reported success. A unit with an active
grant now sets NoNewPrivileges=false, keeping the rest of the hardening;
a grantless least-privilege install keeps NNP enabled. Docs and contracts
state the trade-off: each grant is a scoped, auditable widening.
Also proven in the same live cycle: the flag-conflict refusals, user and
sudoers provisioning, wrapper execution as the service user, the healthy
privilege report ({runningAsRoot:false, serviceUser:pulse-agent,
smartctlHelper:true}), --update preserving the profile without repeated
flags, and uninstall removing the sudoers file and helpers.
The unified agent's Linux installer only offered the root profile, and the
docs called non-root unsupported. That default is the most-cited reason
security-conscious evaluators reject Pulse without trying it. install.sh
gains --least-privilege: the service runs as a dedicated nologin
pulse-agent system user with every existing hardening directive, no
LXC-attach ambient capabilities, docker-group membership for socket reads,
and refusal (not silent root fallback) on appliance platforms, non-systemd
init systems, and --enable-commands. Optional --grant-smart and --grant-pct
restore the only two collectors that need elevation through
visudo-validated exact-command sudoers rules and root-owned wrappers the
agent reaches via new absolute-path-only PULSE_SMARTCTL_PATH /
PULSE_PCT_PATH overrides; the pct grant covers pct list and pct df only
and can never widen into pct exec. --update preserves the profile and its
grants by reading the installed unit.
The agent now authors a privilege block in its report (effective root,
service user, active helpers), carried through models into the fleet
doctor as a descriptive field: Agent Doctor shows the profile and its
helpers instead of presenting intentionally absent collectors as a fault,
and a least-privilege agent can never be marked unhealthy on that evidence
alone.
lxc-attach into an unprivileged guest writes /proc/<pid>/uid_map, which
needs CAP_SETUID in the parent user namespace. NoNewPrivileges drops
CAP_SETUID from the effective set and also stops lxc-attach falling back
to the setuid newuidmap/newgidmap helpers, so the socket probe dies with
"write_id_mapping: 61 Operation not permitted".
install.sh already relaxed NoNewPrivileges for this, but only when the
agent was installed with --enable-commands. Command execution is also
togglable from the server afterwards: applyRemoteConfig starts the
command client without rewriting the unit. An agent installed without
the flag and switched on later therefore ends up able to run commands
and unable to attach to unprivileged guests, so Docker inside every
unprivileged LXC disappears from the Proxmox page. The probe failure is
logged at debug level and retried on every poll, so the surface looks
empty rather than broken while the agent re-probes the whole guest list.
Grant CAP_SETUID/CAP_SETGID to any PVE agent rather than gating on the
install-time flag, so the later toggle lands on a unit that can attach.
Ambient capabilities restore exactly the privilege lxc-attach needs and
leave the rest of the sandbox intact; the existing install-time
relaxation is unchanged.
Verified on a live PVE node. With the hardened unit the probe succeeded
only on the three privileged guests and failed on every unprivileged
one. After the ambient grant CapEff regained CAP_SETUID and both
unprivileged Docker guests were discovered, taking that node from one
Docker LXC to three.
Production SignPath signing requests require manual approval in the
SignPath UI, so the previous single-job flow (submit with
wait-for-completion inside a 40-minute window) let approval latency fail
the Windows build, and any re-run rebuilt the binaries and submitted a
second request needing a second approval.
The Windows lane is now two jobs: sign-windows-agent builds the unsigned
executables, submits the SignPath request without waiting, and uploads a
7-day signing-request record; collect-windows-signing absorbs approval
latency by polling the recorded request, downloads the signed artifact
by request id, and keeps the existing verification and evidence steps.
If approval outlasts the 115-minute polling window, the collection job
fails with re-run guidance and "Re-run failed jobs" collects the same
recorded request - no rebuild, no resubmission. The legacy PFX
break-glass backend rides the same two-job shape via an artifact
hand-off. Workflow output wiring, artifact names, and evidence content
are unchanged for downstream consumers.
The shape test now pins the async invariants (no wait-for-completion:
true in the candidate workflow), and the code signing policy plus the
deployment-installability contract describe the two-phase flow.
Traefik took the whole operator .env via env_file, so the internet-facing
container's environment carried CP_ADMIN_KEY and the entitlement signing
private key, neither of which it needs. It now receives only ACME/DNS
material: the CF token by explicit passthrough plus an optional
dns-credentials.env that setup.sh creates 0600.
The same wiring removes the silent Cloudflare-only narrowing: the resolver
provider is now ACME_DNS_PROVIDER (default cloudflare) via Traefik's env
override, the CF token is required only for the default provider, and any
other Traefik dnsChallenge provider supplies its credential variables through
dns-credentials.env. Pinned by
TestProviderMSPTraefikEnvIsMinimalAndDNSProviderOverridable; contract delta
in deployment-installability.
The installer compares the agent binary it downloaded against the server that
served it, stripping a leading "v" so "v6.0.4" and "6.0.4" match. It did not
strip semver build metadata, so a server built from a working tree reporting
"6.2.0-rc.8+git.46.g98a638e00.dirty" never matched the "v6.2.0-rc.8" agent it
had just served, and the mismatch warning fired on every correct development
install.
This is the warning's whole job, so a false positive is expensive. It is the
only client-side signal that a stale agent was downloaded, and because it
always fired it read as background noise. That is exactly how a genuinely
stale v6.0.5 agent was installed on a live host earlier today: the warning
was there, above the install output, and looked like the one that always
appears.
Strip build metadata from both sides before comparing, keeping the prerelease
suffix because 6.2.0-rc.8 and 6.2.0 are genuinely different releases. This is
the same release-identity reduction the server applies when deciding whether a
local agent artifact is fresh enough to serve; the contracts now state that
one definition governs both ends rather than leaving each side to invent its
own.
Guarded by a test that pins both normalisation steps and exercises the
comparison across the cases that matter: the dev-server shape that used to
warn wrongly, the stale-download shape that must still warn, and a prerelease
against its release. Verified to fail when either strip is removed.
Completes the wrapper-teardown rule across the remaining branches. The QNAP
install and both uninstall paths still used a bare pkill -f
"start-pulse-agent.sh" and still stopped the agent before its wrapper.
The bare pattern is narrower than it looks and wider than it should be. It
does NOT match a co-installed agent's supervisor, so the sibling case was
already safe; what it does match is anything where the unescaped dot stands in
for another character and the unbounded tail keeps going, including a .bak
copy of the wrapper and an editor session holding it open. Escaping the dot
and bounding the far end removes both without narrowing the intended match.
Ordering is the more consequential half. A wrapper is a watchdog, so stopping
the agent while its wrapper still loops only races the respawn. QNAP and the
uninstall paths now stop the supervisor first, which is what the contracts
already required of every branch that writes and launches a wrapper.
Uninstall keeps a deliberately broader match than install, with no leading
path separator, so it still reaches a wrapper invoked by a relative path or
stranded at a superseded location. Both contracts now carry that distinction
and the teardown ordering rule, which each had stated only for install.
Guarded by two tests that pin every wrapper kill in the file rather than one
branch: one requires the escaped dot and the bounded tail everywhere, the
other walks each stop block and fails if an agent kill precedes its wrapper
kill. Both were confirmed to fail against the pre-fix QNAP block.
The Unraid install path killed the running agent but never the wrapper
supervising it, then appended a second wrapper at the end of the install. The
survivor and the newcomer both loop trying to own the same agent id, and
because the old wrapper is a watchdog it respawns the agent mid-install with
the previous binary and arguments. Observed on a live Unraid host: a
supervisor from a July install was still running beside the one the reinstall
had just started.
Nothing reports this as a failure. It presents later as an agent that
restarts on its own or reverts to superseded arguments.
Stop the wrapper first, then the agent: killing a supervised agent while its
wrapper still loops only races the respawn. The pattern matches the trailing
path segment so a wrapper left at an older storage location is caught too,
with the dot escaped and the far end bounded so a co-installed agent's
supervisor (start-pulse-agent-prod.sh) is not.
The QNAP branch already stopped its wrapper, which is what made the Unraid
omission visible; the contracts now require every wrapper-writing branch to
own the same teardown.
Guarded by two tests: one pins that the Unraid branch stops the wrapper and
does so before the agent, the other pins that the wrapper pattern spares a
sibling supervisor, with a premise check that the loose pattern really does
match so neither assertion can pass vacuously.
pkill -f matches the whole command line and "^" only anchors the start, so
"^/usr/local/bin/pulse-agent" also matches "/usr/local/bin/pulse-agent-prod".
On a host running a second agent whose binary name shares the prefix, every
install, every upgrade, and every restart of the generated Unraid wrapper
silently killed the other agent too. Confirmed on a live dual-agent Unraid
box: the old pattern matched both the dev agent and the production dogfood
agent, the bounded pattern matches only its own.
The wrapper is the worst of the three because restarting through it is the
documented runbook step, so the collateral kill repeats every time an
operator follows it.
Bound the far end of each binary-anchored pattern with ([[:space:]]|$), and
swap the bare pkill -9 -f "pulse-agent" for -x on the exact process name,
which keeps that site's deliberate path-agnostic intent while excluding the
sibling. The pkill -x sites were already safe and are unchanged.
Guarded by two tests: one pins that no binary-anchored pkill in the installer
is left unbounded, the other exercises POSIX ERE semantics against the two
command lines a dual-agent host presents, including a premise check that the
unbounded pattern really does match the sibling so the assertion cannot pass
vacuously.
pct exec runs with PATH=/sbin:/bin:/usr/sbin:/usr/bin, which does not
include /usr/local/bin where the pulse binary is linked. Every first-run
instruction that invoked a bare `pulse` through pct exec therefore failed
with exit 127, including the command the installer itself prints at the
end of an LXC install.
Reproduced on Proxmox VE 9.1.9 and 8.3.0:
pct exec <ctid> -- pulse --version
-> exit 127, Failed to exec "pulse"
pct exec <ctid> -- /usr/local/bin/pulse --version
-> Pulse v6.2.0-rc.8
The installer now prints $BINARY_LINK_PATH and $UPDATE_HELPER_PATH rather
than a bare name, so both stay correct when a custom service name moves
them. The update helper was only affected for custom service names, since
the default /bin/update already sits on the pct exec PATH.
Docs updated in INSTALL.md, FAQ.md and TROUBLESHOOTING.md plus their
frontend-modern/public/docs mirrors. The setup wizard prints the same
broken command and is fixed separately.
The agent_runtime verify-only build pointed its registry buildcache at
ghcr.io/<owner>/pulse-agent, a package no workflow publishes. A cache ref
creates the package it points at, so every release recreated it and the
repository's Packages sidebar advertised an empty pulse-agent entry that
reads like a pullable agent image sitting beside the real pulse,
pulse-control-plane and pulse-chart/pulse packages. The only release tag it
ever carried is a stray v6.0.0-rc.3.
Move the cache to ghcr.io/<owner>/pulse:agent-buildcache, alongside the
runtime stage's existing pulse:buildcache tag, so it lives in the package
that actually ships and shares backend-builder layers with the runtime
build.
TestAgentBuildCacheDoesNotResurrectPulseAgentPackage locks this in. It scans
every workflow for registry-qualified refs to the pulse-agent package,
collapsing ${{ }} expressions first so an interpolated owner cannot hide one
behind its spaces, and asserts both the new cache target and the chart's
published agent image default. Verified red on a restored buildcache ref and
red again on a ref hidden behind ${{ github.repository_owner }}.
Two stale claims corrected while here. The chart values comment said the old
agent.image default was never published, when it did carry v6.0.0-rc.3. The
deployment-installability contract called rcourtman/pulse the only image
publish-docker.yml pushes, when the same workflow also pushes the MSP
pulse-control-plane image. The contract now also forbids buildcache
references to the pulse-agent package, which is what let this regress.
Deleting the orphaned package itself remains a manual GHCR step.
Advance the governed install, Docker, and Helm metadata together; add the RC9 release packet for the post-RC8 notification, lifecycle, and resource-read fixes; and record the existing mobile-candidate compatibility decision.
rcourtman