Device removal, execution stagger and a live run dashboard #20

Merged
gsadmin merged 20 commits from dev into main 2026-11-01 01:40:43 +00:00
Owner

Device removal with short-lived agent credentials, execution stagger, and a live run dashboard.

Device removal

  • Remove Agent… on devices, selections, groups and folders (nested groups included). Needs the new devices.remove permission, which only Admin has by default.
  • What happens:
    • The device leaves the system at once, and a removal record stands in for it.
    • An online agent uninstalls now, through a detached watchdog runner: Windows Installer in process for MSI installs, a helper outside the folder for portable ones. Credentials and data are wiped.
    • An offline agent is chased indefinitely when it returns. Stop Trying ends that; the device stays blocked and keeps its agent.
  • Short-lived credentials: agents on 1.1.0 or later get 24-hour NATS credentials, renewed over HTTPS at start and at half life. A removed device's renewal answers 410, and the agent uninstalls itself. A 410 is ignored over plain HTTP or with TlsSkipVerify.
  • Re-enrollment: tokens created at or before the removal are refused; a newer token re-enrolls. Old commands are cleared first so nothing replays.
  • The list stays clean: removals show dimmed in the Devices list for 7 days after their last change, then only under the Removals status filter.

Execution stagger

  • Spread Over and Devices at a Time on Run Script and schedules. Device order is shuffled per run.
  • Durable: each job has a stored start time, sent by a new dispatcher, so it survives restarts.
  • Consistent: every device runs the script as it was at the start of the run (snapshot protected with Data Protection). Secrets and URLs are prepared at send time. Only Devices Online Now is re-checked at each device's slot.
  • While watching: Next Start In and Last Start In countdowns corrected for clock skew, a countdown on each device, and Cancel Waiting.

Live run dashboard

  • Clickable tiles, an outcome ring, exit codes and completions over time.
  • Errors grouped by exception type and normalized message (GET /api/executions/{runId}/breakdown). Tiles, exit codes and errors all filter the device table.
  • Job events now carry their run, so the page refreshes only for its own run.

Fixes

  • A failed publish no longer leaves a job queued forever, and the dispatch race with a fast agent's first event is closed.
  • Removed LiveDeployment, which never rendered.

Behaviour changes

  • PowerShell results: exit 5 now fails the job with code 5 (it reported Succeeded/0). A native command's non-zero exit no longer fails a job by itself, and an error silenced with -ErrorAction SilentlyContinue no longer fails it.
  • Unsent jobs fail after 24 hours.
  • Version floor: agents on 1.1.0 or later can't be moved below 1.1.0, because older agents can't renew. The server and the agent both refuse it.
  • Agents below 1.1.0 can't take a remove command and keep non-expiring credentials. Their removal shows as needing a manual uninstall or an upgrade first, and the server ignores what they report, but they can still connect to NATS.
  • SQL Server migration: shortens Jobs.State to nvarchar(450) so it can be indexed. EF flags this as possible data loss; the values are short state names.

Testing

  • Automated: 292 backend, 211 agent, 110 watchdog and 173 shared tests pass. The console typecheck and lint are clean, and all three providers report no pending model changes.
  • Real on Windows: MSI uninstall through the detached runner, and a portable removal with real services.
  • Browser: bulk removal, the Removals filter, Stop Trying, a 60-device staggered run, the countdown, Cancel Waiting, the dashboard filters and the schedule editor. Finished jobs in that check were written directly to the database.
  • Migrations: applied cleanly to a copy of an older SQLite database.
  • Not tested:
    • Anything live over NATS with a real agent: command delivery, renewal and expiry, clearing a removed device's commands.
    • The Linux and macOS uninstall paths, which are compiled and unit-tested only.
    • The Postgres and SQL Server migrations, which were generated but not applied.

🤖 Generated with Claude Code

Device removal with short-lived agent credentials, execution stagger, and a live run dashboard. ## Device removal - **Remove Agent…** on devices, selections, groups and folders (nested groups included). Needs the new `devices.remove` permission, which only Admin has by default. - **What happens:** - The device leaves the system at once, and a removal record stands in for it. - An online agent uninstalls now, through a detached watchdog runner: Windows Installer in process for MSI installs, a helper outside the folder for portable ones. Credentials and data are wiped. - An offline agent is chased indefinitely when it returns. **Stop Trying** ends that; the device stays blocked and keeps its agent. - **Short-lived credentials:** agents on 1.1.0 or later get 24-hour NATS credentials, renewed over HTTPS at start and at half life. A removed device's renewal answers 410, and the agent uninstalls itself. A 410 is ignored over plain HTTP or with `TlsSkipVerify`. - **Re-enrollment:** tokens created at or before the removal are refused; a newer token re-enrolls. Old commands are cleared first so nothing replays. - **The list stays clean:** removals show dimmed in the Devices list for 7 days after their last change, then only under the **Removals** status filter. ## Execution stagger - **Spread Over** and **Devices at a Time** on Run Script and schedules. Device order is shuffled per run. - **Durable:** each job has a stored start time, sent by a new dispatcher, so it survives restarts. - **Consistent:** every device runs the script as it was at the start of the run (snapshot protected with Data Protection). Secrets and URLs are prepared at send time. **Only Devices Online Now** is re-checked at each device's slot. - **While watching:** Next Start In and Last Start In countdowns corrected for clock skew, a countdown on each device, and **Cancel Waiting**. ## Live run dashboard - Clickable tiles, an outcome ring, exit codes and completions over time. - Errors grouped by exception type and normalized message (`GET /api/executions/{runId}/breakdown`). Tiles, exit codes and errors all filter the device table. - Job events now carry their run, so the page refreshes only for its own run. ## Fixes - A failed publish no longer leaves a job queued forever, and the dispatch race with a fast agent's first event is closed. - Removed `LiveDeployment`, which never rendered. ## Behaviour changes - **PowerShell results:** `exit 5` now fails the job with code 5 (it reported Succeeded/0). A native command's non-zero exit no longer fails a job by itself, and an error silenced with `-ErrorAction SilentlyContinue` no longer fails it. - **Unsent jobs** fail after 24 hours. - **Version floor:** agents on 1.1.0 or later can't be moved below 1.1.0, because older agents can't renew. The server and the agent both refuse it. - **Agents below 1.1.0** can't take a remove command and keep non-expiring credentials. Their removal shows as needing a manual uninstall or an upgrade first, and the server ignores what they report, but they can still connect to NATS. - **SQL Server migration:** shortens `Jobs.State` to `nvarchar(450)` so it can be indexed. EF flags this as possible data loss; the values are short state names. ## Testing - **Automated:** 292 backend, 211 agent, 110 watchdog and 173 shared tests pass. The console typecheck and lint are clean, and all three providers report no pending model changes. - **Real on Windows:** MSI uninstall through the detached runner, and a portable removal with real services. - **Browser:** bulk removal, the Removals filter, Stop Trying, a 60-device staggered run, the countdown, Cancel Waiting, the dashboard filters and the schedule editor. Finished jobs in that check were written directly to the database. - **Migrations:** applied cleanly to a copy of an older SQLite database. - **Not tested:** - Anything live over NATS with a real agent: command delivery, renewal and expiry, clearing a removed device's commands. - The Linux and macOS uninstall paths, which are compiled and unit-tested only. - The Postgres and SQL Server migrations, which were generated but not applied. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
gsadmin added 20 commits 2026-11-01 01:05:10 +00:00
The remove command and its events, removal states and reasons, the console
API, how long a removal shows in the Devices list before only its filter does,
the agent credential's lifetime and the 410 answer for a removed device, and
the removal the watchdog carries out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /api/device-removals takes devices and groups (nested included) under a
new devices.remove permission. Each device's row goes in one save with its group
memberships and trigger runs; its own schedules are disabled and unstarted jobs
cancelled. A DeviceRemovals row stands in for it and stays as the history, so old
runs keep their names and a removed device keeps its trail. The groups it was in
are kept there too, which is how a scoped role keeps seeing its removals.

A scheduler sends the remove command when the device is seen online, marks agents
too old to take it, retries, and closes a removal after silence following
"uninstalling", clearing the durable consumer and command subjects first so a
device enrolled again never replays them. Sightings from a client id with no row
update the removal instead of being dropped, and never bring the row back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agents from the minimum version get 24 h credentials, renewed over HTTPS; older
ones keep a non-expiring credential since they could never renew. A renewal for a
removed device proves the enrolled key first, then answers 410 (open and closed
removals) or a plain 401 (stopped: Stop Trying means the agent stays installed).
Tokens created at or before a removal no longer enroll that device; a newer one
closes the removal and clears old commands before creating a fresh row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An agent that renews holds a credential that expires; an older version could not
renew it and would strand within a day. Upgrades, roll backs and the scheduler
leave such a device out as credential-downgrade. Exactly the minimum is fine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Remove Agent(s) on the device menus and Remove Agents on groups, with a confirm
dialog and a See Why result. Recent removals sit dimmed in the plain list with
their state and Stop Trying; a Removals status filter shows them all. A removal's
row opens the history the device left behind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On removal.json the watchdog starts a runner copied outside the install and data folders. It uninstalls through MsiConfigureProductExW or removes the services through the service manager, then verifies and finishes the purge. Three attempts, restoring the services after each; failed is left in removal.json for the agent to report. Neither a new upgrade nor the supervisor touches a removal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A background service renews at every start and at half life, NATS connected or not; NATS reads the current credential at each connect. Nothing is replaced while an upgrade could roll back, and a 410 from an unverified server uninstalls nothing. The remove command checks its enrollment and refuses foreground runs and upgrades in progress; a device with no watchdog runs the same runner from a copy of the agent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A refused renewal now names its removal, and an event from an agent that
doesn't know the id (a removal it started without one) finds its device's open
removal instead of being dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Behaviour change: the in-process PowerShell engine now reports what the
script asked for, the way pwsh -File does.

- `exit 5` reports exit 5 and Failed. It used to report Succeeded, 0: an
  AddScript exit reaches only the host's SetShouldExit, which nothing read,
  so a small host now captures it. `exit 0` succeeds, even after errors.
- A native command's non-zero $LASTEXITCODE no longer fails a script unless
  the script exits with it, and neither do errors it silenced (HadErrors is
  set by both, so the error stream decides now).
- Write-Error still fails a script that never calls exit, with exit 1.

JobStatePayload gains optional ErrorType, ErrorId and ErrorMessage: the first
error's exception type, FullyQualifiedErrorId and message; for external
engines the first stderr line. Older agents omit them. It also gains RunId,
for the backend to set when relaying to browsers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
StaggerSeconds and StaggerBatchSize spread a run's starts: slots =
ceil(targets / batch), slot i at i × spread ÷ (slots − 1), the first at once.
The order is shuffled by a hash of (run, device). Each job gets a fixed
NotBeforeUtc and stays Queued ("waiting") until a new JobDispatcher on the
SCHEDULE_WORKER node sends it. All jobs are written in one save, so a run
starts at once however many devices it has. A run with one slot or no spread
takes the old path.

The run keeps a Data Protection sealed snapshot (script, prelude, parameters,
arguments, bindings), so every device runs what was started; secrets and
presigned URLs are prepared afresh for each pass and never stored. The
snapshot is cleared once nothing is left to send. OnlineOnly is checked again
at the slot: a device offline by then becomes skipped, and its job row goes.

- POST /api/executions/{runId}/cancel-waiting, jobs.run and scoped, audited.
- Counts gain Waiting and TimedOut; targets gain NotBeforeUtc and the detail
  ServerTimeUtc. A run with only skips left can still be opened and listed.
- Schedules store the same two settings and apply them per firing.
- A publish that fails no longer leaves a job Queued forever: the dispatcher
  retries it, for 24 hours. Prepare failures are retried for 5 minutes.
- Dispatched is recorded only from Queued, so it cannot overwrite a fast
  agent's Running or a cancel.
- JobStatePayload's error details land in new job columns; job.state relays
  to browsers carry the RunId.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GET /api/executions/{runId}/breakdown (jobs.view, scoped): exit-code counts,
failed jobs grouped by exception type and first line (position text and paths
cut, up to five sample devices, every job id) and finished jobs by outcome in
buckets sized to the run. Jobs from older agents fall back to their summary,
then their first Error-stream line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Stagger switch with Spread Over (minutes or hours, presets 5 to 60 minutes)
and Devices at a Time, on the Run Script Review tab's Delivery row and beside
a schedule's cadence, with a live "12 batches, one every 1 min 22 s" signpost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An outage with jobs due would otherwise retry and log every 2 seconds. It now
waits 15 seconds after a refused publish.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
While devices wait, a strip counts down to the next and last start, corrected
with the server's clock, with Cancel Waiting; waiting devices get a chip and
their own countdown. Above the device table: tiles for each state (Waiting and
Timed Out are new), the progress bar, an outcome ring, exit codes, grouped
errors with sample devices and completions over time. Tiles, exit codes and
errors filter the table.

The page now re-reads only on its own run's events (job.state carries the
RunId), not on every job in the fleet. TimelineChart's columns become a general
StackedColumns; BarList rows can be selectable.

Removes sections/dashboard/LiveDeployment.tsx: it counted queued and dispatched
job.state events, which are never relayed, so its total stayed 0 and it never
rendered. The dashboard's own poll and activity refresh already keep it current.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs: stagger, watching a run, and the changed exit code reporting
CI / build-test (push) Failing after 26m20s
CI / images (push) Has been skipped
CI / build-test (pull_request) Successful in 7m23s
CI / images (pull_request) Has been skipped
355c967292
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gsadmin merged commit 631c6f9705 into main 2026-11-01 01:40:43 +00:00
Sign in to join this conversation.