Device removal, execution stagger and a live run dashboard #20

Merged
gsadmin merged 20 commits from dev into main 2026-11-01 01:40:43 +00:00

20 Commits

Author SHA256 Message Date
Alphaeus Mote 355c967292 docs: stagger, watching a run, and the changed exit code reporting
CI / build-test (push) Failing after 26m20s
CI / images (push) Has been skipped
CI / build-test (pull_request) Successful in 7m23s
CI / images (pull_request) Has been skipped
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:19:40 -04:00
Alphaeus Mote 90195dc6a7 feat(console): countdown, Cancel Waiting and a live dashboard on a run
While devices wait, a strip counts down to the next and last start, corrected
with the server's clock, with Cancel Waiting; waiting devices get a chip and
their own countdown. Above the device table: tiles for each state (Waiting and
Timed Out are new), the progress bar, an outcome ring, exit codes, grouped
errors with sample devices and completions over time. Tiles, exit codes and
errors filter the table.

The page now re-reads only on its own run's events (job.state carries the
RunId), not on every job in the fleet. TimelineChart's columns become a general
StackedColumns; BarList rows can be selectable.

Removes sections/dashboard/LiveDeployment.tsx: it counted queued and dispatched
job.state events, which are never relayed, so its total stayed 0 and it never
rendered. The dashboard's own poll and activity refresh already keep it current.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:19:40 -04:00
Alphaeus Mote 4266976f63 fix(jobs): pause the dispatcher when messaging refuses a publish
An outage with jobs due would otherwise retry and log every 2 seconds. It now
waits 15 seconds after a refused publish.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:19:40 -04:00
Alphaeus Mote 1b4b20266d feat(console): a Stagger setting on Run Script and schedules
A Stagger switch with Spread Over (minutes or hours, presets 5 to 60 minutes)
and Devices at a Time, on the Run Script Review tab's Delivery row and beside
a schedule's cadence, with a live "12 batches, one every 1 min 22 s" signpost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:19:39 -04:00
Alphaeus Mote df62b38a00 feat(jobs): a run's breakdown of exit codes, errors and completions
GET /api/executions/{runId}/breakdown (jobs.view, scoped): exit-code counts,
failed jobs grouped by exception type and first line (position text and paths
cut, up to five sample devices, every job id) and finished jobs by outcome in
buckets sized to the run. Jobs from older agents fall back to their summary,
then their first Error-stream line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:19:39 -04:00
Alphaeus Mote 846020e116 test: stagger slots, dispatcher passes, restart, cancel, online at slot, schedules
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:19:02 -04:00
Alphaeus Mote 48019c6e87 feat(jobs): stagger a run's starts, durably, from run script and schedules
StaggerSeconds and StaggerBatchSize spread a run's starts: slots =
ceil(targets / batch), slot i at i × spread ÷ (slots − 1), the first at once.
The order is shuffled by a hash of (run, device). Each job gets a fixed
NotBeforeUtc and stays Queued ("waiting") until a new JobDispatcher on the
SCHEDULE_WORKER node sends it. All jobs are written in one save, so a run
starts at once however many devices it has. A run with one slot or no spread
takes the old path.

The run keeps a Data Protection sealed snapshot (script, prelude, parameters,
arguments, bindings), so every device runs what was started; secrets and
presigned URLs are prepared afresh for each pass and never stored. The
snapshot is cleared once nothing is left to send. OnlineOnly is checked again
at the slot: a device offline by then becomes skipped, and its job row goes.

- POST /api/executions/{runId}/cancel-waiting, jobs.run and scoped, audited.
- Counts gain Waiting and TimedOut; targets gain NotBeforeUtc and the detail
  ServerTimeUtc. A run with only skips left can still be opened and listed.
- Schedules store the same two settings and apply them per firing.
- A publish that fails no longer leaves a job Queued forever: the dispatcher
  retries it, for 24 hours. Prepare failures are retried for 5 minutes.
- Dispatched is recorded only from Queued, so it cannot overwrite a fast
  agent's Running or a cancel.
- JobStatePayload's error details land in new job columns; job.state relays
  to browsers carry the RunId.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:19:02 -04:00
Alphaeus Mote b35c956865 fix(agent): report a script's real exit code and first error
Behaviour change: the in-process PowerShell engine now reports what the
script asked for, the way pwsh -File does.

- `exit 5` reports exit 5 and Failed. It used to report Succeeded, 0: an
  AddScript exit reaches only the host's SetShouldExit, which nothing read,
  so a small host now captures it. `exit 0` succeeds, even after errors.
- A native command's non-zero $LASTEXITCODE no longer fails a script unless
  the script exits with it, and neither do errors it silenced (HadErrors is
  set by both, so the error stream decides now).
- Write-Error still fails a script that never calls exit, with exit 1.

JobStatePayload gains optional ErrorType, ErrorId and ErrorMessage: the first
error's exception type, FullyQualifiedErrorId and message; for external
engines the first stderr line. Older agents omit them. It also gains RunId,
for the backend to set when relaying to browsers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:18:50 -04:00
Alphaeus Mote 3183b32c61 fix(removal): the agent's and server's sides agree on which removal an event means
A refused renewal now names its removal, and an event from an agent that
doesn't know the id (a removal it started without one) finds its device's open
removal instead of being dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:15:26 -04:00
Alphaeus Mote 2969b759b5 docs: what removing a device does on the device
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:10:13 -04:00
Alphaeus Mote a4463e88e6 test: credential renewal, remove command, runner phases and upgrade exclusion
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:09:50 -04:00
Alphaeus Mote 1fc4569910 feat(agent): renew the credential over HTTPS and remove the agent on command or 410
A background service renews at every start and at half life, NATS connected or not; NATS reads the current credential at each connect. Nothing is replaced while an upgrade could roll back, and a 410 from an unverified server uninstalls nothing. The remove command checks its enrollment and refuses foreground runs and upgrades in progress; a device with no watchdog runs the same runner from a copy of the agent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:09:50 -04:00
Alphaeus Mote c2b6a9baed feat(watchdog): the detached removal runner, purge and upgrade exclusion
On removal.json the watchdog starts a runner copied outside the install and data folders. It uninstalls through MsiConfigureProductExW or removes the services through the service manager, then verifies and finishes the purge. Three attempts, restoring the services after each; failed is left in removal.json for the agent to report. Neither a new upgrade nor the supervisor touches a removal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:09:50 -04:00
Alphaeus Mote 2f424feba8 docs: removing devices
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:09:49 -04:00
Alphaeus Mote 70a3931361 feat(console): remove agents, and show removals in the Devices list
Remove Agent(s) on the device menus and Remove Agents on groups, with a confirm
dialog and a See Why result. Recent removals sit dimmed in the plain list with
their state and Stop Trying; a Removals status filter shows them all. A removal's
row opens the history the device left behind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:09:49 -04:00
Alphaeus Mote e8326cb1da test(removal): removal requests, scheduler, enrollment and the upgrade guard
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:09:49 -04:00
Alphaeus Mote 8ff23db6c9 feat(removal): never move an agent below the first version that renews
An agent that renews holds a credential that expires; an older version could not
renew it and would strand within a day. Upgrades, roll backs and the scheduler
leave such a device out as credential-downgrade. Exactly the minimum is fine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:09:49 -04:00
Alphaeus Mote f8abdda4a6 feat(removal): short-lived agent credentials, and enrollment answers a removal
Agents from the minimum version get 24 h credentials, renewed over HTTPS; older
ones keep a non-expiring credential since they could never renew. A renewal for a
removed device proves the enrolled key first, then answers 410 (open and closed
removals) or a plain 401 (stopped: Stop Trying means the agent stays installed).
Tokens created at or before a removal no longer enroll that device; a newer one
closes the removal and clears old commands before creating a fresh row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:09:49 -04:00
Alphaeus Mote d1525f8b3d feat(removal): remove devices from the system, and chase their agents
POST /api/device-removals takes devices and groups (nested included) under a
new devices.remove permission. Each device's row goes in one save with its group
memberships and trigger runs; its own schedules are disabled and unstarted jobs
cancelled. A DeviceRemovals row stands in for it and stays as the history, so old
runs keep their names and a removed device keeps its trail. The groups it was in
are kept there too, which is how a scoped role keeps seeing its removals.

A scheduler sends the remove command when the device is seen online, marks agents
too old to take it, retries, and closes a removal after silence following
"uninstalling", clearing the durable consumer and command subjects first so a
device enrolled again never replays them. Sightings from a client id with no row
update the removal instead of being dropped, and never bring the row back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:09:49 -04:00
Alphaeus Mote 5d7d021af1 feat(removal): contracts for removing devices and short-lived agent credentials
The remove command and its events, removal states and reasons, the console
API, how long a removal shows in the Devices list before only its filter does,
the agent credential's lifetime and the 410 answer for a removed device, and
the removal the watchdog carries out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 19:30:08 -04:00