Device removal, execution stagger and a live run dashboard #20

Merged
gsadmin merged 20 commits from dev into main 2026-11-01 01:40:43 +00:00
122 changed files with 18776 additions and 628 deletions
+213 -6
View File
@@ -53,12 +53,14 @@ updates with **no SignalR, SSE, or polling**.
25. [Database providers](#database-providers)
26. [Logging](#logging)
27. [Device lifecycle: reconnect or orphan](#device-lifecycle-reconnect-or-orphan)
28. [Agent upgrades from the server](#agent-upgrades-from-the-server)
29. [S3 file delivery](#s3-file-delivery)
30. [CI/CD and GITEA_* variables](#cicd-and-gitea_-variables)
31. [Versioning](#versioning)
32. [Troubleshooting](#troubleshooting)
33. [Licence](#licence)
28. [Removing devices](#removing-devices)
29. [Removing devices](#removing-devices)
30. [Agent upgrades from the server](#agent-upgrades-from-the-server)
31. [S3 file delivery](#s3-file-delivery)
32. [CI/CD and GITEA_* variables](#cicd-and-gitea_-variables)
33. [Versioning](#versioning)
34. [Troubleshooting](#troubleshooting)
35. [Licence](#licence)
---
@@ -721,6 +723,7 @@ still holds it.
| `upgrade/msi/` | the MSI being installed, the one to go back to (`rollback/`) and the runner's copy of the watchdog (`runner/`) |
| `<install>\Logs\agent-msi-<version>-<id>.log`, `ExecuteR-Watchdog-msi.log` | Windows Installer's verbose log of each MSI run, and the runner's own log |
| `%ProgramData%\ExecuteR\packages\<sha256>\` | agent MSIs kept for rollback (the newest three) |
| `upgrade/removal.json` | a removal under way, or one that failed until the agent has reported it ([On the device](#on-the-device)) |
If the watchdog itself stops part way (power loss, a kill), its next start
finishes or undoes the run from the journal: `stopping` starts over (the agent
@@ -1101,6 +1104,24 @@ an object's properties as columns) rather than a type name such as
`System.Collections.Hashtable`, still streamed line by line and without
`…` truncation. Strings pass through unchanged.
**Exit codes.** A PowerShell 7 script reports what `pwsh -File` would: `exit N`
is exit code N and Failed when N is not 0, `exit 0` succeeds even after errors,
and a terminating error is exit 1. A native command's non-zero `$LASTEXITCODE`
only counts if the script exits with it (`exit $LASTEXITCODE`), and errors the
script silenced do not fail it. One difference, kept on purpose: anything on the
error stream (`Write-Error`, a failing cmdlet) fails a script that never calls
`exit`, with exit code 1.
> **Changed behaviour:** agents before this version reported `exit 5` as Succeeded
> with exit code 0, and failed any script whose native command ended non-zero or
> that silenced an error with `-ErrorAction SilentlyContinue`. A script relying on
> either now reports differently.
A failed job also records its first error's exception type, error id and message
(for Windows PowerShell and Bash, the first stderr line). For jobs from older
agents the run's error groups use the summary or the first error line of the
output.
**Downloads.** A job's output drawer downloads its complete stored output as
`COMPUTERNAME.JobID.yyyy.MM.dd.HHmmss.log` (optionally with timestamps), and
a run (Jobs › run, or a script's Executions tab) downloads every target's
@@ -1317,6 +1338,38 @@ of one run shares one execution, which the result links to.
call, answering with the job), `clientIds`, `groupId`, `groupIds` and
`onlineOnly`, deduplicated and scope-filtered into one `runId`.
**Stagger** spreads a run's starts so a large fleet does not hit one share or
service at once: **Spread Over** (`staggerSeconds`, up to 24 hours) and
**Devices at a Time** (`staggerBatchSize`, default 1), on the Review tab and on
schedules. Devices fall into `ceil(targets ÷ batch)` batches evenly spaced over
the spread: the first starts at once, the last when the spread ends, in an order
shuffled per run so the same device is not always last. With one batch or no
spread the run dispatches as it always has. What it guarantees:
- It is durable. Each job keeps its fixed start time and a worker on the
`SCHEDULE_WORKER` node sends it when it is due, so a backend restart loses
nothing and starting a run returns at once for any number of devices.
- Every device runs the script **as it was when the run started**, with its
arguments and prelude, from a snapshot sealed with Data Protection and cleared
when nothing is left to send. Secrets and presigned URLs are resolved when each
batch is sent and never stored.
- **Only Devices Online Now** is checked again at a device's start: offline by
then, it becomes skipped.
- A deleted script fails the jobs still waiting. **Cancel Waiting**
(`POST /api/executions/{runId}/cancel-waiting`, Run Jobs, audited, within the
caller's device scope) cancels what has not been sent.
- A job whose publish fails is retried by the same worker, with or without a
stagger, for up to 24 hours before it fails; a vault or S3 failure at send time
is retried for 5 minutes.
**Watching a run** (Jobs › a run): while devices wait, a strip counts down to the
next and the last start, corrected for the browser's clock, and each waiting
device counts down in its row. Above the devices, updating live: a tile per
state, an outcome ring, exit codes, errors grouped by exception type and first
line with sample devices, and completions over time; tiles, exit codes and errors
filter the table. `GET /api/executions/{runId}/breakdown` serves the charts,
limited to the caller's device scope.
**Schedules** (the Schedules page, plus a Schedules tab on scripts, devices
and groups) are scheduled triggers: *when* (one-shot or recurring), *target*
(a device, a group whose members resolve at fire time, or all devices), an
@@ -1342,6 +1395,8 @@ Semantics worth knowing:
- The worker sweeps regularly; recurrence advances from *now* after each
firing, so server downtime causes **at most one late firing**, never a storm.
- One-shot schedules disable themselves after firing.
- A schedule's **Stagger** applies to each firing; event triggers run on one
device at a time and have none.
- Each schedule records its **last result** (jobs queued, or the error — e.g.
a vault failure) and the execution it created; scheduled jobs are attributed
to `schedule`, run-now firings to the operator.
@@ -1460,6 +1515,158 @@ the device keeps its history, groups and tags.
---
## Removing devices
**Remove Agent…** on a device (row menu, bulk Actions, device page) or **Remove
Agents…** on a group or folder (nested groups included) takes the device out of
the system at once and uninstalls its agent now, or whenever it next comes back.
It needs the `devices.remove` permission, which Admin has and Operator and Viewer
do not. Scoped roles reach only the devices in their scope.
The device row goes in the same transaction as its group memberships and trigger
runs; schedules aimed at that device alone are disabled and jobs that never
started are cancelled. Finished jobs, their output and the device's history stay,
and old runs keep its name. What stands in for the row is a **removal**, kept
forever as the history.
```mermaid
flowchart TD
R["Remove Agent: device row deleted, removal waiting"] --> S{"Seen online?"}
S -- "No, however long" --> S
S -- "Yes, agent 1.1.0 or later" --> C["Remove command sent (at most every 5 min)"]
S -- "Yes, older agent" --> O["Agent too old: stays open, stays blocked"]
C --> U["Agent reports uninstalling"]
C -- "It failed" --> F["Couldn't Uninstall: asked again"]
C -- "No answer for 30 min" --> S
F --> S
U -- "Silent for 15 min" --> D["Removed: consumer deleted, commands purged"]
U -- "Still reporting after 15 min" --> S
D -- "Reports again" --> S
```
**Online.** The next scheduler pass (every 15 seconds, on the node with
`SCHEDULE_WORKER` on) sends the command. Its enrollment id must match what the
agent holds, so a stale command can never uninstall a device enrolled again since.
After a successful uninstall nobody is left to report success, so 15 minutes of
silence after "uninstalling" counts as gone.
**Offline.** A removal waits for as long as the device stays away. Presence,
inventory and ready notices from a client id with no device row only update the
removal; they never bring the row back.
**Credentials.** Agents from 1.1.0 on hold a NATS credential that lasts 24 hours,
renewed over HTTPS at half life and at every start. A renewal for a removed device
answers `410` with problem type `urn:problem:device-removed`, and the agent
uninstalls itself, so a device that slips past the command is still cut off within
a day. Only the holder of the enrolled key learns this: a bad signature gets the
same plain `401` as an id nobody knows. Older agents never renew and could not
survive an expiring credential, so enrollment still gives them one that never
expires. To keep that true an agent on 1.1.0 or later is never upgraded or rolled
back to anything older (it is left out as `credential-downgrade`); exactly 1.1.0
is fine.
**Tokens.** A bootstrap token created at or before the removal no longer enrolls
that device, whatever its other limits say, so the install it has and a GPO still
pushing an old token cannot bring it back. The response is the plain `401` of any
refused token, with the reason in the audit trail. A token created after the
removal enrolls it on a fresh device row: an open removal closes as removed, and
its old command consumer and commands are cleared first, so nothing old replays.
**In the Devices list.** A removal shows in the list for 7 days after its last
change: dimmed, with its state, not selectable and not counted in the totals. After
that the list stays uncluttered, and the **Removals** option of the Status filter
shows every removal whatever its age, with who asked, when, and what happened.
Clicking a removal opens the device's history. `GET /api/device-removals` takes
`state`, `clientId` and `limit` (default 200). Scoped roles see the removals of
devices that were in one of their scope groups when they were removed; a device
that was in none is visible to unscoped roles only.
**Stop Trying.** `POST /api/device-removals/{id}/stop` on an open removal ends the
chase: the device stays blocked, its pending command is purged and its agent stays
installed, cut off when its credential expires (a renewal gets a plain `401`, not
`410`, because `410` would uninstall it after all). Enrolling it again needs a new
token.
**Agents too old.** An agent below 1.1.0 cannot take the command or renew. Its
removal stays open marked *Uninstall It by Hand or Upgrade It First*, and since
its credential never expires the server cannot cut it off from NATS: it can keep
connecting, but the server ignores everything it reports. Once such an agent reports
1.1.0 or later the command goes out as usual.
The audit trail keeps `device-removal-requested`, `device-removed` and
`device-removal-stopped`. The project wiki is updated separately.
### On the device
**Credentials.** The agent's NATS credential lives 24 hours. A small background
service renews it over HTTPS (`POST /api/enrollment/credential`, signed with the
enrolled NKey, TLS as for enrollment) at every start and at half life, whether
or not NATS is connected, so an expired credential recovers by itself as soon as
the server can be reached, and a credential from before 1.1 that never expires
is replaced at the first start. The old credential stays until the new one is
stored; the NATS connection takes the new one at its next reconnect, at the
latest when the server ends the connection as the old one expires.
| The server answers | The agent |
|---|---|
| a new credential | stores it, atomically, and looks again at half its life |
| network error, 5xx, anything unexpected | keeps what it has and tries again after 30 s, doubling to 15 minutes |
| 401 (orphaned, or the clock is off) | keeps what it has and tries again hourly; a configured bootstrap token enrolls it again ([Device lifecycle](#device-lifecycle-reconnect-or-orphan)) |
| 410 `urn:problem:device-removed` | uninstalls itself, as below. Not with `TlsSkipVerify` on or over plain http: an unverified server must not be able to uninstall an agent, so that only logs a warning |
While an upgrade could still roll back (`handover.json`, `journal.json`,
`msi.json`), the stored credential is left alone: a version before 1.1 cannot
renew and would strand on a 24-hour one. It renews when the upgrade is kept. The
one exception is a credential within an hour of expiring, which would strand
either way and would keep the new version from reporting healthy. For the same
reason an agent on 1.1 or later refuses an upgrade to a version before 1.1
(**failed / unsupported**).
**The remove command.** `command.agent.remove` may be old, since it waits in its
stream, so the agent acts only when its enrollment ID is the one the agent holds
(else **failed / stale-enrollment**). A foreground or development run answers
**failed / unsupported**, an upgrade in progress **failed / busy** (the server
asks again). Otherwise the agent reports **uninstalling**, writes
`upgrade/removal.json` and leaves the rest to the watchdog. A device where no
watchdog can take it (none installed, or it has not taken it within 2 minutes)
runs the same runner from a copy of the agent's own binary. The watchdog starts
no upgrade while `removal.json` exists and a removal waits for an upgrade under
way; it does not supervise the agent during a removal.
**The runner.** The watchdog copies itself out of both folders it is about to
wipe, into `%SystemRoot%\Temp\ExecuteR-<id>-<random>\` (closed to SYSTEM and
Administrators), or `/var/lib/executer-removal-<random>/` on Linux and macOS, and
starts the copy apart from its service: detached on Windows, as a transient
`systemd-run` unit on Linux, as a one-shot launchd daemon on macOS, since the
service manager would otherwise end it with the service. The runner:
1. stops the agent;
2. uninstalls: an MSI install in process through Windows Installer
(`MsiConfigureProductExW`, no UI, `REBOOT=ReallySuppress`, never msiexec),
which removes both services; 1618 is waited out for up to 10 minutes. A
portable install loses both services through the service manager (the runner
refuses to wipe a folder that holds no agent binary, or a shared one such as
`C:\Program Files`);
3. verifies and finishes the purge, whatever the MSI's own purge managed.
Three attempts, 5 minutes apart, 30 minutes at most each. A failed attempt starts
both services again. After the third, `removal.json` says **failed** with the
reason, the agent reports **failed / msi** or **failed / uninstall**, deletes the
file and carries on; the server asks again. After a success nobody is left to
report it.
| Wiped | Left |
|---|---|
| `credentials.dat` and the whole data folder (`%ProgramData%\ExecuteR`, `/var/lib/executer`) | the policy key `HKLM\SOFTWARE\Policies\ExecuteR` (and the policy section of `agent.json`): it is group policy's |
| `HKLM\SOFTWARE\ExecuteR` with `Runtime`, `ServerURL`, the token, proxy and TLS values (`/etc/executer/agent.json`, the `/usr/local/bin` link) | a server CA the agent did not itself add to the trust store. That includes every CA installed before 1.1, which kept no record of it |
| the install folder, `Logs` included | the runner's folder on Windows, until the next restart |
| both services; the server CA the agent added (`managed-ca.txt`) | |
A file still locked after a minute is deleted at the next restart. A failed
removal keeps its logs as `<install>\Logs\agent-removal-<id>-*.log`.
---
## Agent installer delivery
**The installer ships inside the backend image — no manual seeding.** The CI
+9
View File
@@ -25,6 +25,15 @@ public sealed class AgentSupervisor(WatchdogOptions options, IAgentService agent
public async Task CheckAsync(CancellationToken cancellationToken)
{
// A removal stops the agent on purpose; only one that failed hands it back.
if (WatchdogFileStore.Read(options.RemovalPath, WatchdogJsonContext.Default.WatchdogRemoval) is { Phase: not Shared.Contracts.Models.WatchdogRemovalPhases.Failed })
{
runningSince = null;
failedSince = null;
Note("A removal is under way; the agent is left to it");
return;
}
var status = await agent.QueryAsync(cancellationToken);
var now = clock.UtcNow;
@@ -17,6 +17,12 @@ public interface IWindowsInstaller
/// <summary><c>MsiInstallProductW</c>: installs the package and returns Windows Installer's own code (what msiexec would exit with).</summary>
int InstallProduct(string package, string commandLine);
/// <summary><c>MsiConfigureProductExW</c> with <c>INSTALLSTATE_ABSENT</c>: uninstalls the product and returns Windows Installer's own code.</summary>
int UninstallProduct(string productCode, string commandLine);
/// <summary><c>MsiEnumRelatedProductsW</c>: the product code installed under the UpgradeCode, or null.</summary>
string? RelatedProduct(string upgradeCode);
}
/// <summary>Windows Installer's answers an MSI run acts on.</summary>
@@ -37,6 +43,8 @@ public static class MsiExitCodes
public const int PackageInvalid = 1620;
/// <summary>ERROR_PRODUCT_VERSION: a newer version is installed (a downgrade without ALLOWDOWNGRADE=1).</summary>
public const int ProductVersion = 1638;
/// <summary>ERROR_UNKNOWN_PRODUCT: the product is not installed (an uninstall that has nothing to do).</summary>
public const int UnknownProduct = 1605;
/// <summary>Done, done with a restart pending, or done with a restart started.</summary>
public static bool Succeeded(int code) => code is Success or SuccessRebootRequired or SuccessRebootInitiated;
@@ -51,6 +59,7 @@ public static class MsiExitCodes
InstallFailure => "a fatal error during installation",
PackageInvalid => "the package could not be opened",
ProductVersion => "another version of the product is already installed",
UnknownProduct => "the product is not installed",
1619 => "the package could not be opened",
1625 => "the installation is forbidden by system policy",
1633 => "the package is not supported on this platform",
@@ -64,6 +73,12 @@ public sealed partial class WindowsInstallerApi : IWindowsInstaller
/// <summary>INSTALLUILEVEL_NONE.</summary>
private const int InstallUiLevelNone = 2;
/// <summary>INSTALLLEVEL_DEFAULT.</summary>
private const int InstallLevelDefault = 0;
/// <summary>INSTALLSTATE_ABSENT: Windows Installer sets REMOVE=ALL, as for msiexec's <c>/x</c>.</summary>
private const int InstallStateAbsent = 2;
/// <summary>
/// msiexec's <c>/l*v</c> (msi.h INSTALLLOGMODE_*): FATALEXIT 0x1, ERROR 0x2,
/// WARNING 0x4, USER 0x8, INFO 0x10, RESOLVESOURCE 0x40, OUTOFDISKSPACE 0x80,
@@ -89,6 +104,21 @@ public sealed partial class WindowsInstallerApi : IWindowsInstaller
public int InstallProduct(string package, string commandLine) => (int)MsiInstallProductW(package, commandLine);
public int UninstallProduct(string productCode, string commandLine)
=> (int)MsiConfigureProductExW(productCode, InstallLevelDefault, InstallStateAbsent, commandLine);
public unsafe string? RelatedProduct(string upgradeCode)
{
var buffer = stackalloc char[39];
return MsiEnumRelatedProductsW(upgradeCode, 0, 0, buffer) == 0 ? new string(buffer, 0, 38) : null;
}
[LibraryImport("msi.dll", StringMarshalling = StringMarshalling.Utf16)]
private static partial uint MsiConfigureProductExW(string product, int installLevel, int installState, string commandLine);
[LibraryImport("msi.dll", StringMarshalling = StringMarshalling.Utf16)]
private static unsafe partial uint MsiEnumRelatedProductsW(string upgradeCode, uint reserved, uint index, char* productCode);
[LibraryImport("msi.dll")]
private static partial int MsiSetInternalUI(int level, nint window);
@@ -12,7 +12,20 @@ internal static partial class ServiceInterop
internal const uint SC_MANAGER_CONNECT = 0x1;
internal const uint SC_MANAGER_ALL_ACCESS = 0xF003F;
internal const uint SERVICE_QUERY_STATUS = 0x4;
internal const uint SERVICE_START = 0x10;
internal const uint SERVICE_STOP = 0x20;
internal const uint DELETE = 0x10000;
internal const uint SERVICE_ALL_ACCESS = 0xF01FF;
internal const uint SERVICE_CONTROL_STOP = 0x1;
internal const uint SERVICE_STOPPED = 0x1;
internal const uint SERVICE_START_PENDING = 0x2;
internal const uint SERVICE_STOP_PENDING = 0x3;
internal const uint SERVICE_RUNNING = 0x4;
internal const int ERROR_SERVICE_ALREADY_RUNNING = 1056;
internal const int ERROR_SERVICE_DOES_NOT_EXIST = 1060;
internal const int ERROR_SERVICE_CANNOT_ACCEPT_CTRL = 1061;
internal const int ERROR_SERVICE_NOT_ACTIVE = 1062;
internal const int ERROR_SERVICE_MARKED_FOR_DELETE = 1072;
internal const uint SERVICE_WIN32_OWN_PROCESS = 0x10;
internal const uint SERVICE_AUTO_START = 0x2;
internal const uint SERVICE_ERROR_NORMAL = 0x1;
@@ -62,6 +75,18 @@ internal static partial class ServiceInterop
public uint dwServiceFlags;
}
[StructLayout(LayoutKind.Sequential)]
internal struct SERVICE_STATUS
{
public uint dwServiceType;
public uint dwCurrentState;
public uint dwControlsAccepted;
public uint dwWin32ExitCode;
public uint dwServiceSpecificExitCode;
public uint dwCheckPoint;
public uint dwWaitHint;
}
[LibraryImport("advapi32.dll", EntryPoint = "OpenSCManagerW", StringMarshalling = StringMarshalling.Utf16, SetLastError = true)]
internal static partial IntPtr OpenSCManager(string? machineName, string? databaseName, uint access);
@@ -119,6 +144,14 @@ internal static partial class ServiceInterop
[return: MarshalAs(UnmanagedType.Bool)]
internal static partial bool CloseServiceHandle(IntPtr handle);
[LibraryImport("advapi32.dll", EntryPoint = "StartServiceW", StringMarshalling = StringMarshalling.Utf16, SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
internal static partial bool StartService(IntPtr service, int argumentCount, IntPtr arguments);
[LibraryImport("advapi32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
internal static partial bool ControlService(IntPtr service, uint control, out SERVICE_STATUS status);
internal static Win32Exception LastError(string operation)
=> new(Marshal.GetLastPInvokeError(), $"{operation} failed");
@@ -0,0 +1,124 @@
using Shared.Contracts.Models;
using Shared.Contracts.Serialization;
namespace Agent.Watchdog.Removal;
/// <summary>
/// The watchdog's side of a removal: on <c>removal.json</c> it starts the detached
/// runner and leaves it to it. The runner stops and removes this watchdog, so a
/// runner that died can only be noticed if the removal failed and the watchdog is
/// still (or again) here; it is then started again, at most <see cref="MaxLaunches"/>
/// times. An upgrade under way finishes first; the agent refuses a removal
/// during one, so that takes a command and a hand-over crossing.
/// </summary>
public sealed class RemovalCoordinator(WatchdogOptions options, IRemovalRunnerLauncher? launcher, IAgentService agent, IWatchdogClock clock, WatchdogLog log)
{
public const int MaxLaunches = 5;
/// <summary>A runner this watchdog just started is given this long to take its lock.</summary>
public static readonly TimeSpan LaunchGrace = TimeSpan.FromMinutes(1);
private DateTimeOffset launchedUtc = DateTimeOffset.MinValue;
private int launches;
private bool waitingNoted;
/// <summary>True while the removal owns this pass: no upgrade step and no supervision.</summary>
public async Task<bool> PassAsync(CancellationToken cancellationToken)
{
if (!File.Exists(options.RemovalPath))
{
launches = 0;
return false;
}
var removal = WatchdogFileStore.Read(options.RemovalPath, WatchdogJsonContext.Default.WatchdogRemoval);
if (removal is null)
{
log.Error($"{options.RemovalPath} does not read; it is removed and nothing was changed");
WatchdogFileStore.TryDelete(options.RemovalPath);
return false;
}
if (removal.Phase == WatchdogRemovalPhases.Failed)
{
// Over; the agent reports it and deletes the file.
return false;
}
if (File.Exists(options.JournalPath) || File.Exists(options.MsiJournalPath))
{
if (!waitingNoted)
{
log.Warn($"Removal {removal.RemovalId} waits for the upgrade under way to end");
waitingNoted = true;
}
return false;
}
waitingNoted = false;
if (removal.RunnerDirectory is { } directory
&& (RemovalRunLock.IsHeld(Path.Combine(directory, WatchdogRemovalFiles.RunnerLock)) || clock.UtcNow - launchedUtc < LaunchGrace))
{
return true;
}
if (launcher is null)
{
await GiveUpAsync(removal, "This watchdog runs in its development mode and cannot remove the agent.", cancellationToken);
return false;
}
if (removal.Attempts >= WatchdogRemovalFiles.MaxAttempts || launches >= MaxLaunches)
{
await GiveUpAsync(removal, removal.LastError ?? $"The removal runner stopped before the removal ended, {launches} times.", cancellationToken);
return false;
}
if (removal.RunnerDirectory is { } previous)
{
SwapPaths.TryDeleteDirectory(previous);
}
log.BeginRun();
log.Info($"Removal {removal.RemovalId} handed over ({removal.InstallKind} install); starting the runner");
var (runnerDirectory, problem) = launcher.Prepare(removal.RemovalId);
if (problem is null)
{
WatchdogFileStore.Write(options.RemovalPath,
removal with { Phase = WatchdogRemovalPhases.Uninstalling, RunnerDirectory = runnerDirectory },
WatchdogJsonContext.Default.WatchdogRemoval);
problem = launcher.Start(removal.RemovalId, runnerDirectory!);
}
launches++;
if (problem is not null)
{
await GiveUpAsync(removal, $"The removal runner did not start: {problem}.", cancellationToken);
return false;
}
launchedUtc = clock.UtcNow;
return true;
}
private async Task GiveUpAsync(WatchdogRemoval removal, string why, CancellationToken cancellationToken)
{
log.Error($"Removal {removal.RemovalId} gave up: {why}");
WatchdogFileStore.Write(options.RemovalPath,
removal with { Phase = WatchdogRemovalPhases.Failed, LastError = why },
WatchdogJsonContext.Default.WatchdogRemoval);
try
{
var status = await agent.QueryAsync(cancellationToken);
if (status.State != AgentRunState.NotInstalled && !status.IsRunning)
{
await agent.StartAsync(cancellationToken);
}
}
catch (Exception exception) when (exception is not OperationCanceledException || !cancellationToken.IsCancellationRequested)
{
log.Warn("The agent could not be started", exception);
}
}
}
@@ -0,0 +1,484 @@
using System.ComponentModel;
using System.Diagnostics;
using System.Runtime.InteropServices;
using System.Runtime.Versioning;
using System.Security.Cryptography.X509Certificates;
using System.Text.Json.Nodes;
using Agent.Watchdog.Msi;
using Agent.Watchdog.Platform;
using Microsoft.Win32;
using Shared.Contracts.Models;
using Shared.Infrastructure.Constants;
using static Agent.Watchdog.Platform.ServiceInterop;
namespace Agent.Watchdog.Removal;
/// <summary>The services a portable install's removal takes out (an MSI's are the MSI's to remove); replaced in tests.</summary>
public interface IRemovalServices
{
Task StopWatchdogAsync(CancellationToken cancellationToken);
Task StartWatchdogAsync(CancellationToken cancellationToken);
/// <summary>Removes the agent's and the watchdog's registrations; nothing to do for one already gone.</summary>
Task RemoveAsync(CancellationToken cancellationToken);
}
/// <summary>
/// Hidden entry point of the runner, for both binaries that have it: the runner
/// uses no agent code and no watchdog-only code, so the agent can fall back on it
/// when no watchdog can take a removal. Logs to its own folder, since the agent's
/// log folder goes with the install.
/// </summary>
public static class RemovalHost
{
public static async Task<int> RunAsync(Guid removalId, string[] args)
{
string? Value(string name)
{
for (var i = 0; i < args.Length - 1; i++)
{
if (string.Equals(args[i], name, StringComparison.OrdinalIgnoreCase))
{
return args[i + 1];
}
}
return null;
}
if (Value("--install-dir") is not { } install || Value("--data-dir") is not { } data || Value("--runner-dir") is not { } runnerDirectory)
{
Console.Error.WriteLine("Usage: --install-dir <path> --data-dir <path> --runner-dir <path>");
return 1;
}
var options = new WatchdogOptions
{
InstallDirectory = Path.GetFullPath(install),
DataDirectory = Path.GetFullPath(data),
SelfPath = null,
};
var log = new WatchdogLog(runnerDirectory, console: false, name: RemovalRunner.LogFileName);
var agentPath = Path.Combine(options.InstallDirectory, options.AgentExecutable);
IAgentService agent;
IRemovalServices services;
IRemovalMachine machine;
IWindowsInstaller? installer = null;
if (OperatingSystem.IsWindows())
{
agent = new ScmService(Branding.AgentServiceName, agentPath, log);
services = new WindowsRemovalServices(
(ScmService)agent,
new ScmService(Branding.WatchdogServiceName, Path.Combine(options.InstallDirectory, Branding.WatchdogExecutableName), log));
machine = new WindowsRemovalMachine(log);
installer = new WindowsInstallerApi();
}
else if (OperatingSystem.IsMacOS())
{
agent = new LaunchdAgentService(Branding.AgentLaunchdLabel);
services = new UnixRemovalServices();
machine = new UnixRemovalMachine(options, log);
}
else
{
agent = new SystemdAgentService(Branding.AgentSystemdUnit);
services = new UnixRemovalServices();
machine = new UnixRemovalMachine(options, log);
}
var clock = SystemClock.Instance;
var purge = new RemovalPurge(options, machine, clock, log, runnerDirectory);
var runner = new RemovalRunner(options, agent, services, installer, purge, clock, log, runnerDirectory);
return await runner.RunAsync(removalId, CancellationToken.None);
}
}
/// <summary>A Windows service through the SCM's own APIs (never sc.exe, and none of ServiceController, which the agent's linux/macOS build lacks).</summary>
[SupportedOSPlatform("windows")]
public sealed class ScmService(string serviceName, string executablePath, WatchdogLog log) : IAgentService
{
private static readonly TimeSpan StateChangeTimeout = TimeSpan.FromSeconds(60);
public Task<AgentServiceStatus> QueryAsync(CancellationToken cancellationToken)
{
if (QueryStatus(serviceName) is not { } status)
{
return Task.FromResult(AgentServiceStatus.NotInstalled);
}
var state = status.dwCurrentState switch
{
SERVICE_STOPPED => AgentRunState.Stopped,
SERVICE_START_PENDING => AgentRunState.Starting,
SERVICE_STOP_PENDING => AgentRunState.Stopping,
_ => AgentRunState.Running,
};
return Task.FromResult(new AgentServiceStatus(state, true, false));
}
public async Task StartAsync(CancellationToken cancellationToken)
{
WithService(SERVICE_START, service =>
{
if (!StartService(service, 0, IntPtr.Zero) && Marshal.GetLastPInvokeError() != ERROR_SERVICE_ALREADY_RUNNING)
{
throw LastError($"StartService({serviceName})");
}
});
await WaitForAsync(SERVICE_RUNNING, cancellationToken);
}
public async Task StopAsync(CancellationToken cancellationToken)
{
WithService(SERVICE_STOP, service =>
{
if (!ControlService(service, SERVICE_CONTROL_STOP, out _)
&& Marshal.GetLastPInvokeError() is not (ERROR_SERVICE_NOT_ACTIVE or ERROR_SERVICE_CANNOT_ACCEPT_CTRL))
{
throw LastError($"ControlService({serviceName})");
}
});
await WaitForAsync(SERVICE_STOPPED, cancellationToken);
await StopLeftoversAsync(cancellationToken);
}
/// <summary>Removes the registration: gone once the last handle on it closes.</summary>
public void Delete() => WithService(DELETE, service =>
{
if (!DeleteService(service) && Marshal.GetLastPInvokeError() != ERROR_SERVICE_MARKED_FOR_DELETE)
{
throw LastError($"DeleteService({serviceName})");
}
log.Info($"Service {serviceName} removed");
});
/// <summary>Runs <paramref name="action"/> on the open service; a service that is not there is nothing to do.</summary>
private void WithService(uint access, Action<IntPtr> action)
{
var scm = OpenSCManager(null, null, SC_MANAGER_CONNECT);
if (scm == IntPtr.Zero)
{
throw LastError("OpenSCManager");
}
try
{
var service = OpenService(scm, serviceName, access);
if (service == IntPtr.Zero)
{
if (Marshal.GetLastPInvokeError() == ERROR_SERVICE_DOES_NOT_EXIST)
{
return;
}
throw LastError($"OpenService({serviceName})");
}
try
{
action(service);
}
finally
{
CloseServiceHandle(service);
}
}
finally
{
CloseServiceHandle(scm);
}
}
private async Task WaitForAsync(uint state, CancellationToken cancellationToken)
{
var deadline = DateTime.UtcNow + StateChangeTimeout;
while (QueryStatus(serviceName) is { } status && status.dwCurrentState != state)
{
if (DateTime.UtcNow >= deadline)
{
throw new System.TimeoutException($"{serviceName} did not reach state {state} within {StateChangeTimeout.TotalSeconds:0} s");
}
await Task.Delay(250, cancellationToken);
}
}
/// <summary>The service process can outlive its stop report, and the agent's own children hold its binary: wait for them, then end what is left.</summary>
private async Task StopLeftoversAsync(CancellationToken cancellationToken)
{
var name = Path.GetFileNameWithoutExtension(executablePath);
var deadline = DateTime.UtcNow.AddSeconds(10);
while (true)
{
var leftovers = Process.GetProcessesByName(name).Where(IsThisBinary).ToList();
if (leftovers.Count == 0)
{
return;
}
var late = DateTime.UtcNow >= deadline;
foreach (var process in leftovers)
{
try
{
if (late)
{
log.Warn($"Ending process {process.Id} of {name} that outlived its service");
process.Kill(entireProcessTree: true);
}
}
catch (Exception exception) when (exception is InvalidOperationException or Win32Exception)
{
}
finally
{
process.Dispose();
}
}
if (late)
{
return;
}
await Task.Delay(500, cancellationToken);
}
}
private bool IsThisBinary(Process process)
{
try
{
return string.Equals(process.MainModule?.FileName, executablePath, StringComparison.OrdinalIgnoreCase);
}
catch (Exception exception) when (exception is InvalidOperationException or Win32Exception)
{
return false;
}
}
}
[SupportedOSPlatform("windows")]
public sealed class WindowsRemovalServices(ScmService agent, ScmService watchdog) : IRemovalServices
{
public Task StopWatchdogAsync(CancellationToken cancellationToken) => watchdog.StopAsync(cancellationToken);
public async Task StartWatchdogAsync(CancellationToken cancellationToken)
{
if ((await watchdog.QueryAsync(cancellationToken)) is { IsRunning: false, State: not AgentRunState.NotInstalled })
{
await watchdog.StartAsync(cancellationToken);
}
}
public Task RemoveAsync(CancellationToken cancellationToken)
{
agent.Delete();
watchdog.Delete();
return Task.CompletedTask;
}
}
/// <summary>The normal registry key, the Local Machine certificate stores, and restart-time deletes.</summary>
[SupportedOSPlatform("windows")]
public sealed partial class WindowsRemovalMachine(WatchdogLog log) : IRemovalMachine
{
private const uint MoveFileDelayUntilReboot = 0x4;
public void RemoveConfiguration() => Registry.LocalMachine.DeleteSubKeyTree(Branding.AgentRegistryKey, throwOnMissingSubKey: false);
public bool ConfigurationRemains
{
get
{
using var key = Registry.LocalMachine.OpenSubKey(Branding.AgentRegistryKey);
return key is not null;
}
}
public void RemoveManagedCa(IReadOnlyCollection<string> thumbprints)
{
foreach (var name in new[] { StoreName.Root, StoreName.CertificateAuthority })
{
using var store = new X509Store(name, StoreLocation.LocalMachine);
store.Open(OpenFlags.ReadWrite);
foreach (var thumbprint in thumbprints)
{
foreach (var certificate in store.Certificates.Find(X509FindType.FindByThumbprint, thumbprint, validOnly: false))
{
store.Remove(certificate);
log.Info($"Removed the server CA {certificate.Subject} from LocalMachine\\{name}");
}
}
}
}
public void DeleteAtRestart(string path)
{
if (!MoveFileExW(path, null, MoveFileDelayUntilReboot))
{
log.Warn($"{path} could not be scheduled for deletion at the next restart (error {Marshal.GetLastPInvokeError()})");
}
}
/// <summary>Held by the running binary until it exits; its log stays for support until the restart that deletes it.</summary>
public void RemoveRunnerDirectory(string directory)
{
try
{
foreach (var file in Directory.EnumerateFiles(directory, "*", SearchOption.AllDirectories))
{
DeleteAtRestart(file);
}
DeleteAtRestart(directory);
}
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
{
log.Warn($"{directory} could not be scheduled for deletion at the next restart", exception);
}
}
[LibraryImport("kernel32.dll", StringMarshalling = StringMarshalling.Utf16, SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
private static partial bool MoveFileExW(string existingFileName, string? newFileName, uint flags);
}
/// <summary>Both units or daemons, run from a runner that is outside either's cgroup or process group.</summary>
public sealed class UnixRemovalServices : IRemovalServices
{
public Task StopWatchdogAsync(CancellationToken cancellationToken) => Task.Run(UnixWatchdogRegistration.Stop, cancellationToken);
public Task StartWatchdogAsync(CancellationToken cancellationToken) => Task.Run(UnixWatchdogRegistration.Start, cancellationToken);
public Task RemoveAsync(CancellationToken cancellationToken) => Task.Run(() =>
{
if (OperatingSystem.IsMacOS())
{
Command.Run("launchctl", "bootout", $"system/{Branding.AgentLaunchdLabel}");
DeleteIfThere($"/Library/LaunchDaemons/{Branding.AgentLaunchdLabel}.plist");
}
else
{
Command.Run("systemctl", "disable", "--now", Branding.AgentSystemdUnit);
DeleteIfThere($"/etc/systemd/system/{Branding.AgentSystemdUnit}.service");
}
UnixWatchdogRegistration.Remove();
}, cancellationToken);
private static void DeleteIfThere(string path)
{
if (File.Exists(path))
{
File.Delete(path);
}
}
}
/// <summary><c>agent.json</c> (its policy section stays), the PATH link, and the trust anchors ServerTrust wrote.</summary>
public sealed class UnixRemovalMachine(WatchdogOptions options, WatchdogLog log) : IRemovalMachine
{
private const string PathLink = "/usr/local/bin/" + Branding.AgentUnixExecutableName;
private const string AnchorName = "executer-server-ca.crt";
private const string DebianAnchors = "/usr/local/share/ca-certificates/";
private const string RedHatAnchors = "/etc/pki/ca-trust/source/anchors/";
private static string PolicySection => Branding.AgentPolicyRegistryKey.Replace('\\', '/');
public void RemoveConfiguration()
{
var path = Branding.AgentUnixConfigFile;
if (new FileInfo(PathLink).LinkTarget is { } target
&& (SwapPaths.IsInside(target, options.InstallDirectory) || target.Contains("/executer/", StringComparison.Ordinal)))
{
File.Delete(PathLink);
}
if (File.Exists(path))
{
var kept = new JsonObject();
try
{
if (JsonNode.Parse(File.ReadAllText(path)) is JsonObject root
&& root.FirstOrDefault(p => string.Equals(p.Key, PolicySection, StringComparison.OrdinalIgnoreCase)) is { Value: JsonObject policy } section)
{
kept[section.Key] = policy.DeepClone();
}
}
catch (System.Text.Json.JsonException)
{
log.Warn($"{path} does not read; it goes whole");
}
if (kept.Count > 0)
{
File.WriteAllText(path, kept.ToJsonString());
return;
}
File.Delete(path);
}
if (Path.GetDirectoryName(path) is { } folder)
{
SwapPaths.TryDeleteIfEmpty(folder);
}
}
public bool ConfigurationRemains
{
get
{
if (!File.Exists(Branding.AgentUnixConfigFile))
{
return false;
}
try
{
// Only the policy section may stay.
return JsonNode.Parse(File.ReadAllText(Branding.AgentUnixConfigFile)) is not JsonObject root
|| root.Any(p => !string.Equals(p.Key, PolicySection, StringComparison.OrdinalIgnoreCase));
}
catch (System.Text.Json.JsonException)
{
return true;
}
}
}
public void RemoveManagedCa(IReadOnlyCollection<string> thumbprints)
{
if (OperatingSystem.IsMacOS())
{
foreach (var thumbprint in thumbprints)
{
Command.Run("security", "delete-certificate", "-Z", thumbprint, "/Library/Keychains/System.keychain");
}
return;
}
foreach (var (folder, refresh) in new[] { (DebianAnchors, "update-ca-certificates"), (RedHatAnchors, "update-ca-trust") })
{
var anchor = folder + AnchorName;
if (File.Exists(anchor))
{
File.Delete(anchor);
Command.Run(refresh);
log.Info($"Removed the server CA {anchor}");
}
}
}
public void DeleteAtRestart(string path)
{
// Unlinking works here even for a file in use; a sweep already tried.
}
public void RemoveRunnerDirectory(string directory) => SwapPaths.TryDeleteDirectory(directory);
}
+247
View File
@@ -0,0 +1,247 @@
using Shared.Contracts.Models;
using Shared.Infrastructure.Constants;
namespace Agent.Watchdog.Removal;
/// <summary>What a removal wipes outside the install and data folders; replaced in tests.</summary>
public interface IRemovalMachine
{
/// <summary>
/// The agent's configuration and runtime state: the normal registry key tree
/// (<c>agent.json</c> and the PATH link on linux/macOS). The policy key is
/// group policy's and stays.
/// </summary>
void RemoveConfiguration();
bool ConfigurationRemains { get; }
/// <summary>The server CA certificates the agent installed itself (<see cref="WatchdogFiles.ManagedCa"/>).</summary>
void RemoveManagedCa(IReadOnlyCollection<string> thumbprints);
/// <summary>Deletes the file or empty folder at the next restart (nothing to do on linux/macOS, where nothing is locked).</summary>
void DeleteAtRestart(string path);
/// <summary>The runner's folder, which holds the running binary: now where that works, otherwise at the next restart.</summary>
void RemoveRunnerDirectory(string directory);
}
/// <summary>
/// Verifies and finishes the wipe, whichever way the product went. The MSI's
/// <c>purge-credentials</c> custom action runs with <c>Return="ignore"</c>, so
/// its success proves nothing: <c>credentials.dat</c>, the configuration, the
/// data and install folders (<c>Logs</c> with them) are each checked and removed
/// here; what is still locked after a minute goes at the next restart.
/// </summary>
public sealed class RemovalPurge(WatchdogOptions options, IRemovalMachine machine, IWatchdogClock clock, WatchdogLog log, string runnerDirectory)
{
public const int Sweeps = 7;
public static readonly TimeSpan SweepDelay = TimeSpan.FromSeconds(10);
private readonly List<string> thumbprints = [];
private string CredentialPath => Path.Combine(options.DataDirectory, Branding.CredentialFileName);
private string ManagedCaPath => Path.Combine(options.DataDirectory, WatchdogFiles.ManagedCa);
private string ManagedCaCopy => Path.Combine(runnerDirectory, WatchdogFiles.ManagedCa);
/// <summary>Takes what the purge needs later from the data folder, into the runner's folder (a resumed runner finds it there).</summary>
public void Snapshot()
{
try
{
if (File.Exists(ManagedCaPath))
{
File.Copy(ManagedCaPath, ManagedCaCopy, overwrite: true);
}
thumbprints.Clear();
if (File.Exists(ManagedCaCopy))
{
thumbprints.AddRange(File.ReadAllLines(ManagedCaCopy).Select(l => l.Trim()).Where(l => l.Length > 0));
}
}
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
{
log.Warn("The list of installed server CAs could not be read; they stay", exception);
}
}
/// <summary>Why the install folder must not be wiped; null when it may. Checked before a portable install's services go.</summary>
public string? InstallFolderRefusal()
{
if (IsProtectedFolder(options.InstallDirectory))
{
return $"The agent runs from {options.InstallDirectory}, which is not a folder of its own; it was left alone.";
}
return File.Exists(Path.Combine(options.InstallDirectory, options.AgentExecutable))
? null
: $"{options.InstallDirectory} holds no {options.AgentExecutable}; it was left alone.";
}
/// <summary>Removes what is left; what stays after every sweep is listed and goes at the next restart.</summary>
public async Task<IReadOnlyList<string>> FinishAsync(CancellationToken cancellationToken)
{
var left = new List<string>();
for (var sweep = 1; sweep <= Sweeps; sweep++)
{
left = Sweep();
if (left.Count == 0)
{
return left;
}
if (sweep < Sweeps)
{
log.Info($"Still there (sweep {sweep} of {Sweeps}): {string.Join("; ", left)}");
await clock.DelayAsync(SweepDelay, cancellationToken);
}
}
foreach (var folder in new[] { options.DataDirectory, options.InstallDirectory })
{
if (!IsProtectedFolder(folder))
{
ScheduleAtRestart(folder);
}
}
return left;
}
public void RemoveRunnerDirectory(string directory) => machine.RemoveRunnerDirectory(directory);
private List<string> Sweep()
{
var left = new List<string>();
// First and on its own: nothing else on the device can authenticate as it.
Try("credentials", () =>
{
if (File.Exists(CredentialPath))
{
File.SetAttributes(CredentialPath, FileAttributes.Normal);
File.Delete(CredentialPath);
}
}, left);
Try("configuration", machine.RemoveConfiguration, left);
if (thumbprints.Count > 0)
{
Try("server CA", () => machine.RemoveManagedCa(thumbprints), left);
}
Try(options.DataDirectory, () => DeleteFolder(options.DataDirectory), left);
Try(options.InstallDirectory, () => DeleteFolder(options.InstallDirectory), left);
if (Path.GetDirectoryName(Path.TrimEndingDirectorySeparator(options.InstallDirectory)) is { } parent
&& !IsProtectedFolder(parent)
&& string.Equals(Path.GetFileName(parent), Branding.ProductKey, StringComparison.OrdinalIgnoreCase))
{
// %ProgramFiles%\ExecuteR, /opt/executer: only ever ours, when nothing else is in it.
SwapPaths.TryDeleteIfEmpty(parent);
}
if (File.Exists(CredentialPath) && !left.Contains("credentials"))
{
left.Add("credentials");
}
if (machine.ConfigurationRemains && !left.Contains("configuration"))
{
left.Add("configuration");
}
return left;
}
private void Try(string what, Action action, List<string> left)
{
try
{
action();
}
catch (Exception exception) when (exception is not OperationCanceledException)
{
log.Warn($"{what} could not be removed", exception);
left.Add(what);
}
}
private static void DeleteFolder(string folder)
{
if (!Directory.Exists(folder))
{
return;
}
if (IsProtectedFolder(folder))
{
throw new InvalidOperationException($"{folder} is not a folder of the agent's own");
}
foreach (var file in Directory.EnumerateFiles(folder, "*", SearchOption.AllDirectories))
{
// Directory.Delete stops at a read-only file on Windows.
if ((File.GetAttributes(file) & FileAttributes.ReadOnly) != 0)
{
File.SetAttributes(file, FileAttributes.Normal);
}
}
Directory.Delete(folder, recursive: true);
}
private void ScheduleAtRestart(string folder)
{
try
{
if (!Directory.Exists(folder))
{
return;
}
foreach (var file in Directory.EnumerateFiles(folder, "*", SearchOption.AllDirectories))
{
machine.DeleteAtRestart(file);
}
// Deepest first: a folder goes only once it is empty.
foreach (var directory in Directory.EnumerateDirectories(folder, "*", SearchOption.AllDirectories).OrderByDescending(d => d.Length))
{
machine.DeleteAtRestart(directory);
}
machine.DeleteAtRestart(folder);
}
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
{
log.Warn($"{folder} could not be scheduled for the next restart", exception);
}
}
/// <summary>A root, or a folder the system or other software shares: a portable agent unpacked straight into one never takes it along.</summary>
public static bool IsProtectedFolder(string folder)
{
var full = Path.TrimEndingDirectorySeparator(Path.GetFullPath(folder));
if (string.IsNullOrEmpty(Path.GetFileName(full)))
{
return true;
}
IEnumerable<string> shared = OperatingSystem.IsWindows()
?
[
Environment.GetFolderPath(Environment.SpecialFolder.Windows),
Environment.GetFolderPath(Environment.SpecialFolder.System),
Environment.GetFolderPath(Environment.SpecialFolder.ProgramFiles),
Environment.GetFolderPath(Environment.SpecialFolder.ProgramFilesX86),
Environment.GetFolderPath(Environment.SpecialFolder.CommonProgramFiles),
Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData),
Environment.GetFolderPath(Environment.SpecialFolder.UserProfile),
Path.Combine(Path.GetPathRoot(full) ?? "", "Users"),
]
: ["/bin", "/boot", "/dev", "/etc", "/home", "/lib", "/lib64", "/opt", "/proc", "/root", "/run", "/sbin", "/sys", "/tmp", "/usr", "/usr/bin",
"/usr/lib", "/usr/local", "/usr/local/bin", "/usr/sbin", "/var", "/var/lib", "/var/tmp", "/Applications", "/Library", "/System", "/Users"];
return shared.Any(path => path.Length > 0 && SwapPaths.Comparer.Equals(Path.TrimEndingDirectorySeparator(path), full));
}
}
+344
View File
@@ -0,0 +1,344 @@
using Agent.Watchdog.Msi;
using Shared.Contracts.Models;
using Shared.Contracts.Serialization;
using Shared.Infrastructure.Constants;
namespace Agent.Watchdog.Removal;
/// <summary>Timings of a removal; shortened in tests.</summary>
public sealed record RemovalRunnerSettings
{
/// <summary>Between tries while Windows Installer is busy with another installation (1618).</summary>
public TimeSpan BusyRetryDelay { get; init; } = TimeSpan.FromSeconds(30);
/// <summary>How long a busy Windows Installer is waited out within one attempt.</summary>
public TimeSpan BusyRetryFor { get; init; } = TimeSpan.FromMinutes(10);
/// <summary>The longest one attempt's uninstall may take (real time: Windows Installer's call cannot be cut short).</summary>
public TimeSpan AttemptTimeout { get; init; } = WatchdogFiles.MsiInstallTimeout;
}
/// <summary>
/// The detached removal runner (<c>ExecuteR-Watchdog.exe remove-run &lt;id&gt;</c>, or
/// <c>ExecuteR-Agent.exe remove-self &lt;id&gt;</c> where no watchdog can take it),
/// started from a copy of its binary outside the install and data folders, since
/// it wipes both. It owns the removal in <c>removal.json</c> while it holds
/// <c>removal.lock</c> in its own folder:
/// <list type="number">
/// <item>stop the agent;</item>
/// <item>MSI install: Windows Installer in process, by product code
/// (<c>REBOOT=ReallySuppress</c>, no UI, verbose log in the runner's folder);
/// 0, 3010, 1641 and 1605 (already gone) are success, 1618 is tried again every
/// 30 s for up to 10 minutes. Portable install: the watchdog and agent services
/// go through the service manager;</item>
/// <item>the purge is verified and finished (<see cref="RemovalPurge"/>): whatever
/// the MSI's own purge left is removed, and what is locked goes at the next restart.</item>
/// </list>
/// A failed attempt starts the services again and the next one follows 5 minutes
/// later, three in all; then the phase says failed, with the reason, for the
/// agent to report, and the device carries on. Unlike <see cref="MsiRunner"/> this
/// runner writes no outcome and verifies no health: on success nobody is left.
/// </summary>
public sealed class RemovalRunner(
WatchdogOptions options,
IAgentService agent,
IRemovalServices services,
IWindowsInstaller? installer,
RemovalPurge purge,
IWatchdogClock clock,
WatchdogLog log,
string runnerDirectory,
RemovalRunnerSettings? settings = null)
{
public const string UninstallProperties = "REBOOT=ReallySuppress";
public const string LogFileName = "removal";
private readonly RemovalRunnerSettings timings = settings ?? new RemovalRunnerSettings();
private string LockPath => Path.Combine(runnerDirectory, WatchdogRemovalFiles.RunnerLock);
/// <summary>Runs (or resumes) the removal; 0 unless <c>removal.json</c> does not name it.</summary>
public async Task<int> RunAsync(Guid removalId, CancellationToken cancellationToken)
{
bool gone;
using (var held = RemovalRunLock.TryHold(LockPath))
{
if (held is null)
{
log.Info($"Another runner owns removal {removalId}; this one exits");
return 0;
}
var removal = WatchdogFileStore.Read(options.RemovalPath, WatchdogJsonContext.Default.WatchdogRemoval);
if (removal is null || removal.RemovalId != removalId)
{
log.Warn($"Removal {removalId} is not what {options.RemovalPath} names; nothing to do");
return removal is null ? 1 : 0;
}
log.BeginRun();
log.Info($"Runner {Environment.ProcessId} owns removal {removalId} ({removal.InstallKind} install, {removal.Attempts} attempts so far)");
// The MSI's own purge takes the data folder this reads.
purge.Snapshot();
gone = await AttemptAllAsync(removal, cancellationToken);
if (gone)
{
var left = await purge.FinishAsync(cancellationToken);
log.Info(left.Count == 0
? $"Removal {removalId} is complete; nothing is left"
: $"Removal {removalId} is complete, except what goes at the next restart: {string.Join("; ", left)}");
}
}
purge.RemoveRunnerDirectory(runnerDirectory);
return 0;
}
/// <summary>True once the product and its services are gone; false when every attempt failed.</summary>
private async Task<bool> AttemptAllAsync(WatchdogRemoval removal, CancellationToken cancellationToken)
{
while (true)
{
if (removal.Attempts >= WatchdogRemovalFiles.MaxAttempts)
{
Finish(removal with { LastError = removal.LastError ?? "The removal runner stopped before it ended." });
return false;
}
removal = removal with { Phase = WatchdogRemovalPhases.Uninstalling, Attempts = removal.Attempts + 1 };
Write(removal);
log.Info($"Removing the agent (attempt {removal.Attempts} of {WatchdogRemovalFiles.MaxAttempts})");
string? problem;
try
{
problem = await AttemptAsync(removal, cancellationToken);
}
catch (Exception exception) when (exception is not OperationCanceledException)
{
problem = $"The removal failed ({exception.GetBaseException().Message}).";
}
if (problem is null)
{
return true;
}
log.Error($"Attempt {removal.Attempts} failed: {problem}");
await RestoreAsync(cancellationToken);
removal = removal with { LastError = problem };
if (removal.Attempts >= WatchdogRemovalFiles.MaxAttempts)
{
Finish(removal);
return false;
}
Write(removal);
log.Info($"The next attempt follows in {WatchdogRemovalFiles.AttemptDelay.TotalMinutes:0} minutes; the agent runs meanwhile");
await clock.DelayAsync(WatchdogRemovalFiles.AttemptDelay, cancellationToken);
}
}
/// <summary>One attempt; null when the agent is gone, otherwise why not.</summary>
private async Task<string?> AttemptAsync(WatchdogRemoval removal, CancellationToken cancellationToken)
{
try
{
await agent.StopAsync(cancellationToken);
}
catch (Exception exception) when (exception is not OperationCanceledException)
{
// Windows Installer and the service manager both stop it themselves.
log.Warn("The agent did not stop", exception);
}
var product = removal.InstallKind == AgentInstallKinds.Msi && installer is not null
? installer.RelatedProduct(Branding.AgentMsiUpgradeCode)
: null;
return product is not null
? await UninstallMsiAsync(removal, product)
: await RemovePortableAsync(removal, cancellationToken);
}
private async Task<string?> UninstallMsiAsync(WatchdogRemoval removal, string product)
{
var logPath = MsiLogPath(removal.RemovalId);
log.Info($"Uninstalling {product} through Windows Installer ({UninstallProperties}); its log is {logPath}");
var giveUp = clock.UtcNow + timings.BusyRetryFor;
while (true)
{
var answer = await UninstallOnceAsync(product, logPath);
if (answer is not { } code)
{
return $"Windows Installer did not finish within {timings.AttemptTimeout.TotalMinutes:0} minutes; its log is {logPath}.";
}
if (code == MsiExitCodes.InstallerBusy)
{
if (clock.UtcNow + timings.BusyRetryDelay > giveUp)
{
return $"Another installation kept Windows Installer busy (1618) for {timings.BusyRetryFor.TotalMinutes:0} minutes.";
}
log.Warn($"Windows Installer is busy with another installation (1618); trying again in {timings.BusyRetryDelay.TotalSeconds:0} s");
await clock.DelayAsync(timings.BusyRetryDelay, CancellationToken.None);
continue;
}
if (!MsiExitCodes.Succeeded(code) && code != MsiExitCodes.UnknownProduct)
{
return $"Windows Installer ended with {code} ({MsiExitCodes.Describe(code)}); its log is {logPath}.";
}
log.Info($"Windows Installer ended with {code} ({MsiExitCodes.Describe(code)})");
if (installer!.RelatedProduct(Branding.AgentMsiUpgradeCode) is { } still)
{
return $"Windows Installer ended with {code}, but {still} is still installed.";
}
return null;
}
}
/// <summary>
/// Windows Installer's answer, or null when it gave none within the timeout.
/// The three calls run on one background thread of their own; one cut off by
/// the timeout ends with the process.
/// </summary>
private async Task<int?> UninstallOnceAsync(string product, string logPath)
{
var uninstall = Task.Factory.StartNew(() =>
{
installer!.SetSilentUI();
installer.EnableLog(logPath);
return installer.UninstallProduct(product, UninstallProperties);
}, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default);
try
{
return await uninstall.WaitAsync(timings.AttemptTimeout);
}
catch (TimeoutException)
{
return null;
}
}
private async Task<string?> RemovePortableAsync(WatchdogRemoval removal, CancellationToken cancellationToken)
{
if (purge.InstallFolderRefusal() is { } refusal)
{
return refusal;
}
log.Info($"Removing the services of the install in {options.InstallDirectory}");
await services.StopWatchdogAsync(cancellationToken);
await services.RemoveAsync(cancellationToken);
log.Info(removal.InstallKind == AgentInstallKinds.Msi
? "Windows Installer no longer knows the product; its services were removed directly"
: "The agent and watchdog services are removed");
return null;
}
/// <summary>Whatever the failed attempt stopped runs again, the watchdog first.</summary>
private async Task RestoreAsync(CancellationToken cancellationToken)
{
try
{
await services.StartWatchdogAsync(cancellationToken);
}
catch (Exception exception) when (exception is not OperationCanceledException)
{
log.Warn("The watchdog service could not be started", exception);
}
try
{
var status = await agent.QueryAsync(cancellationToken);
if (status.State != AgentRunState.NotInstalled && !status.IsRunning)
{
await agent.StartAsync(cancellationToken);
}
}
catch (Exception exception) when (exception is not OperationCanceledException)
{
log.Warn("The agent could not be started", exception);
}
}
/// <summary>Failed for good: the agent reports it from <c>removal.json</c>; the logs stay where an operator looks.</summary>
private void Finish(WatchdogRemoval removal)
{
log.Error($"Removal {removal.RemovalId} gave up after {removal.Attempts} attempts: {removal.LastError}");
Write(removal with { Phase = WatchdogRemovalPhases.Failed });
foreach (var file in new[] { Path.Combine(runnerDirectory, LogFileName + ".log"), MsiLogPath(removal.RemovalId) })
{
try
{
if (File.Exists(file))
{
Directory.CreateDirectory(options.LogDirectory);
File.Copy(file, Path.Combine(options.LogDirectory, $"agent-removal-{removal.RemovalId:N}-{Path.GetFileName(file)}"), overwrite: true);
}
}
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
{
log.Warn($"{file} could not be kept", exception);
}
}
}
private string MsiLogPath(Guid removalId) => Path.Combine(runnerDirectory, $"agent-msi-uninstall-{removalId:N}.log");
private void Write(WatchdogRemoval removal)
=> WatchdogFileStore.Write(options.RemovalPath, removal, WatchdogJsonContext.Default.WatchdogRemoval);
}
/// <summary>
/// <c>removal.lock</c> in the runner's own folder, held open with no sharing for
/// as long as it lives: released when the process ends however it ends, so a lock
/// nobody holds is a runner that is gone.
/// </summary>
public static class RemovalRunLock
{
public static FileStream? TryHold(string path)
{
try
{
return new FileStream(path, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None);
}
catch (IOException)
{
return null;
}
}
public static bool IsHeld(string path)
{
if (!File.Exists(path))
{
return false;
}
try
{
using var probe = new FileStream(path, FileMode.Open, FileAccess.ReadWrite, FileShare.None);
return false;
}
catch (FileNotFoundException)
{
return false;
}
catch (IOException)
{
return true;
}
catch (UnauthorizedAccessException)
{
return false;
}
}
}
@@ -0,0 +1,211 @@
using System.ComponentModel;
using System.Globalization;
using System.Runtime.Versioning;
using System.Security.AccessControl;
using System.Security.Principal;
using Agent.Watchdog.Msi;
using Agent.Watchdog.Platform;
using Shared.Contracts.Models;
using Shared.Infrastructure.Constants;
namespace Agent.Watchdog.Removal;
/// <summary>Starts the removal runner apart from the service that asks for it (the runner removes that service); replaced in tests.</summary>
public interface IRemovalRunnerLauncher
{
/// <summary>A new folder outside the install and data folders, with a copy of this binary in it; or why not.</summary>
(string? RunnerDirectory, string? Problem) Prepare(Guid removalId);
/// <summary>Starts the copy; null when it started, otherwise why not.</summary>
string? Start(Guid removalId, string runnerDirectory);
}
/// <summary>The launcher for this platform; shared with the agent, which has it as its fallback when no watchdog can take a removal.</summary>
public static class RemovalRunnerLauncher
{
public static IRemovalRunnerLauncher Create(WatchdogOptions options, WatchdogLog log, string verb)
{
if (OperatingSystem.IsWindows())
{
return new WindowsRemovalRunnerLauncher(options, log, verb);
}
return new UnixRemovalRunnerLauncher(options, log, verb);
}
internal static string[] Arguments(string verb, Guid removalId, WatchdogOptions options, string runnerDirectory) =>
[
verb, removalId.ToString("D"),
"--install-dir", options.InstallDirectory,
"--data-dir", options.DataDirectory,
"--runner-dir", runnerDirectory,
];
internal static string CopySelf(WatchdogOptions options, string runnerDirectory)
{
var self = options.SelfPath is { } path && File.Exists(path)
? path
: throw new FileNotFoundException("this process has no binary of its own to run from");
var copy = Path.Combine(runnerDirectory, Path.GetFileName(self));
File.Copy(self, copy, overwrite: true);
SwapPaths.MakeExecutable(copy);
return copy;
}
}
/// <summary>
/// <c>%SystemRoot%\Temp\ExecuteR-&lt;id&gt;-&lt;random&gt;\</c>: anyone may add to a folder
/// created under that Temp by default, and this one runs as SYSTEM and loads
/// system DLLs from its own folder, so it is created closed (SYSTEM and
/// Administrators only), under a name no one can have prepared, and its owner is
/// checked before the binary goes in. Started detached: stopping both services
/// never takes it down.
/// </summary>
[SupportedOSPlatform("windows")]
public sealed class WindowsRemovalRunnerLauncher(WatchdogOptions options, WatchdogLog log, string verb) : IRemovalRunnerLauncher
{
public (string? RunnerDirectory, string? Problem) Prepare(Guid removalId)
{
var directory = Path.Combine(
Environment.GetFolderPath(Environment.SpecialFolder.Windows), "Temp",
$"{Branding.ProductKey}-{removalId:N}-{Random.Shared.Next():x8}");
try
{
var folder = new DirectoryInfo(directory);
if (folder.Exists)
{
return (null, $"{directory} already exists");
}
folder.Create(ClosedFolder());
var owner = folder.GetAccessControl().GetOwner(typeof(SecurityIdentifier));
using var identity = WindowsIdentity.GetCurrent();
if (owner is not SecurityIdentifier sid
|| !(sid.IsWellKnown(WellKnownSidType.LocalSystemSid) || sid.IsWellKnown(WellKnownSidType.BuiltinAdministratorsSid) || sid == identity.User))
{
return (null, $"{directory} belongs to someone else");
}
RemovalRunnerLauncher.CopySelf(options, directory);
return (directory, null);
}
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
{
return (null, $"its binary could not be copied to {directory} ({exception.Message})");
}
}
public string? Start(Guid removalId, string runnerDirectory)
{
var copy = Path.Combine(runnerDirectory, Path.GetFileName(options.SelfPath!));
try
{
var processId = DetachedProcess.Start(copy, RemovalRunnerLauncher.Arguments(verb, removalId, options, runnerDirectory), runnerDirectory);
log.Info($"Removal runner started (process {processId}) from {copy}");
return null;
}
catch (Win32Exception exception)
{
return exception.Message;
}
}
private static DirectorySecurity ClosedFolder()
{
var security = new DirectorySecurity();
security.SetAccessRuleProtection(isProtected: true, preserveInheritance: false);
foreach (var account in new[] { WellKnownSidType.LocalSystemSid, WellKnownSidType.BuiltinAdministratorsSid })
{
security.AddAccessRule(new FileSystemAccessRule(
new SecurityIdentifier(account, null),
FileSystemRights.FullControl,
InheritanceFlags.ContainerInherit | InheritanceFlags.ObjectInherit,
PropagationFlags.None,
AccessControlType.Allow));
}
return security;
}
}
/// <summary>
/// <c>&lt;parent of data&gt;/executer-removal-&lt;random&gt;/</c> (mode 700; <c>/var/lib</c>, which
/// unlike a temp folder is not mounted noexec), started through the service
/// manager, since a plain child dies with the service that started it: systemd
/// kills the service's whole cgroup, launchd its process group. Linux: a transient
/// unit (<c>systemd-run</c>). macOS: a one-shot daemon from a plist in that folder.
/// </summary>
public sealed class UnixRemovalRunnerLauncher(WatchdogOptions options, WatchdogLog log, string verb) : IRemovalRunnerLauncher
{
public (string? RunnerDirectory, string? Problem) Prepare(Guid removalId)
{
if (OperatingSystem.IsWindows())
{
return (null, "not on this platform");
}
var parent = Path.GetDirectoryName(Path.TrimEndingDirectorySeparator(options.DataDirectory));
var directory = Path.Combine(parent ?? "/var/lib", $"executer-removal-{Random.Shared.Next():x8}");
try
{
Directory.CreateDirectory(directory, UnixFileMode.UserRead | UnixFileMode.UserWrite | UnixFileMode.UserExecute);
RemovalRunnerLauncher.CopySelf(options, directory);
return (directory, null);
}
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
{
return (null, $"its binary could not be copied to {directory} ({exception.Message})");
}
}
public string? Start(Guid removalId, string runnerDirectory)
{
var copy = Path.Combine(runnerDirectory, Path.GetFileName(options.SelfPath!));
var arguments = RemovalRunnerLauncher.Arguments(verb, removalId, options, runnerDirectory);
try
{
var (code, output) = OperatingSystem.IsMacOS() ? StartLaunchd(copy, arguments, removalId, runnerDirectory) : StartSystemd(copy, arguments, removalId);
if (code != 0)
{
return output.Trim();
}
log.Info($"Removal runner started from {copy}");
return null;
}
catch (Exception exception) when (exception is Win32Exception or InvalidOperationException or IOException)
{
return exception.Message;
}
}
private static (int Code, string Output) StartSystemd(string copy, string[] arguments, Guid removalId)
=> Command.Run("systemd-run", [$"--unit={Branding.AgentSystemdUnit}-removal-{removalId:N}", "--collect", copy, .. arguments]);
private static (int Code, string Output) StartLaunchd(string copy, string[] arguments, Guid removalId, string runnerDirectory)
{
// Booted from outside /Library/LaunchDaemons: gone at the next restart, with nothing to clean up.
var label = string.Create(CultureInfo.InvariantCulture, $"{Branding.AgentLaunchdLabel}.removal.{removalId:N}");
var strings = string.Concat(new[] { copy }.Concat(arguments).Select(a => $"<string>{System.Security.SecurityElement.Escape(a)}</string>"));
var plist = Path.Combine(runnerDirectory, label + ".plist");
File.WriteAllText(plist, $"""
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key><string>{label}</string>
<key>ProgramArguments</key>
<array>{strings}</array>
<key>RunAtLoad</key><true/>
<key>AbandonProcessGroup</key><true/>
</dict>
</plist>
""");
if (!OperatingSystem.IsWindows())
{
File.SetUnixFileMode(plist, UnixFileMode.UserRead | UnixFileMode.UserWrite | UnixFileMode.GroupRead | UnixFileMode.OtherRead);
}
return Command.Run("launchctl", "bootstrap", "system", plist);
}
}
+13
View File
@@ -309,6 +309,19 @@ public sealed class UpgradeApplier(
}
var handover = WatchdogFileStore.Read(options.HandoverPath, WatchdogJsonContext.Default.WatchdogHandover);
if (handover is not null && File.Exists(options.RemovalPath))
{
// The device is being removed: an upgrade that crossed it never starts.
log.Warn($"Upgrade {handover.UpgradeId} is set aside: the agent is being removed");
WatchdogFileStore.TryDelete(options.HandoverPath);
WatchdogFileStore.Write(options.OutcomePath,
new WatchdogOutcome(handover.UpgradeId, AgentUpgradeStates.Failed, AgentUpgradeReasons.Busy, handover.Version, handover.FromVersion, [],
clock.UtcNow, "The agent is being removed from this device; nothing was changed."),
WatchdogJsonContext.Default.WatchdogOutcome);
finished.Add(handover.UpgradeId);
return null;
}
if (handover is null)
{
var id = WatchdogFileStore.ReadUpgradeId(options.HandoverPath);
+6 -1
View File
@@ -60,6 +60,9 @@ public static class WatchdogCli
&& Guid.TryParse(args[1], out var upgradeId):
return await RunMsiAsync(upgradeId, args[2..]);
case WatchdogRemovalFiles.RunVerb when args.Length > 1 && Guid.TryParse(args[1], out var removalId):
return await Removal.RemovalHost.RunAsync(removalId, args[2..]);
case null or "help" or "-h" or "--help" or "/?":
PrintUsage();
return command is null ? 1 : 0;
@@ -140,7 +143,9 @@ public static class WatchdogCli
agent, clock, log);
var applier = new UpgradeApplier(options, agent, clock, log, new ProcessSmokeTest(), restart, msi);
var supervisor = new AgentSupervisor(options, agent, clock, log);
var service = new WatchdogService(options, applier, supervisor, clock, log);
var removal = new Removal.RemovalCoordinator(options,
processMode ? null : Removal.RemovalRunnerLauncher.Create(options, log, WatchdogRemovalFiles.RunVerb), agent, clock, log);
var service = new WatchdogService(options, applier, supervisor, clock, log, removal);
if (processMode)
{
+3
View File
@@ -63,6 +63,9 @@ public sealed record WatchdogOptions
public string MsiRunnerLockPath => Path.Combine(UpgradeDirectory, WatchdogMsiFiles.RunnerLock);
/// <summary>A removal the agent handed over: while it is there no new upgrade starts, and until it failed the agent is not supervised.</summary>
public string RemovalPath => Path.Combine(UpgradeDirectory, WatchdogFiles.Removal);
/// <summary>Windows Installer's verbose log of one MSI run, beside the agent's own logs.</summary>
public string MsiLogPath(string version, Guid upgradeId)
=> Path.Combine(LogDirectory, $"agent-msi-{version.Split('+')[0]}-{upgradeId:N}.log");
+13 -1
View File
@@ -5,7 +5,13 @@ namespace Agent.Watchdog;
/// left half done and tidies the install folder; then, every poll, it carries an
/// upgrade on when one is waiting and otherwise supervises the agent.
/// </summary>
public sealed class WatchdogService(WatchdogOptions options, UpgradeApplier applier, AgentSupervisor supervisor, IWatchdogClock clock, WatchdogLog log)
public sealed class WatchdogService(
WatchdogOptions options,
UpgradeApplier applier,
AgentSupervisor supervisor,
IWatchdogClock clock,
WatchdogLog log,
Removal.RemovalCoordinator? removal = null)
{
/// <summary>Runs until cancelled, or until the watchdog replaced itself (true: exit so the new binary takes over).</summary>
public async Task<bool> RunAsync(CancellationToken cancellationToken)
@@ -29,6 +35,12 @@ public sealed class WatchdogService(WatchdogOptions options, UpgradeApplier appl
{
try
{
if (removal is not null && await removal.PassAsync(cancellationToken))
{
await clock.DelayAsync(options.PollInterval, cancellationToken);
continue;
}
switch (await applier.PassAsync(cancellationToken))
{
case PassResult.RestartRequested:
+16
View File
@@ -60,6 +60,22 @@
<ProjectReference Include="..\Shared.Infrastructure\Shared.Infrastructure.csproj" />
</ItemGroup>
<ItemGroup>
<!-- The removal runner, linked from the watchdog: a device where no watchdog
can take a removal runs the same code from a copy of this binary
(remove-self). Nothing here needs ServiceController, which the linux/macOS
build has not got. -->
<Compile Include="..\Agent.Watchdog\Removal\*.cs" LinkBase="Upgrade\RemovalRunner" />
<Compile Include="..\Agent.Watchdog\Abstractions.cs" LinkBase="Upgrade\RemovalRunner" />
<Compile Include="..\Agent.Watchdog\SwapPaths.cs" LinkBase="Upgrade\RemovalRunner" />
<Compile Include="..\Agent.Watchdog\WatchdogLog.cs" LinkBase="Upgrade\RemovalRunner" />
<Compile Include="..\Agent.Watchdog\WatchdogOptions.cs" LinkBase="Upgrade\RemovalRunner" />
<Compile Include="..\Agent.Watchdog\Msi\WindowsInstaller.cs" LinkBase="Upgrade\RemovalRunner" />
<Compile Include="..\Agent.Watchdog\Msi\MsiRunnerLauncher.cs" LinkBase="Upgrade\RemovalRunner" />
<Compile Include="..\Agent.Watchdog\Platform\ServiceInterop.cs" LinkBase="Upgrade\RemovalRunner" />
<Compile Include="..\Agent.Watchdog\Platform\UnixServices.cs" LinkBase="Upgrade\RemovalRunner" />
</ItemGroup>
<!--
Trim the agent payload before single-file bundling. The PowerShell SDK
ships only the core engine modules (Management, Utility, Security, Host,
+5
View File
@@ -110,6 +110,11 @@ public static class AgentCli
case "purge-credentials":
return PurgeCredentials();
// Hidden: the removal runner, from a copy of this binary, on a
// device where no watchdog can take a removal.
case Shared.Contracts.Models.WatchdogRemovalFiles.AgentRunVerb when args.Length > 1 && Guid.TryParse(args[1], out var removalId):
return await Agent.Watchdog.Removal.RemovalHost.RunAsync(removalId, args[2..]);
default:
Console.Error.WriteLine($"Unknown command '{command}'.");
PrintUsage();
+7
View File
@@ -28,6 +28,7 @@ public sealed class CommandProcessor(
JobHistoryStore jobHistory,
AgentConfiguration configuration,
AgentUpgradeHandler upgrades,
AgentRemoveCommandHandler removals,
AgentHealthState health,
ILogger logger)
{
@@ -125,6 +126,12 @@ public sealed class CommandProcessor(
await HandleUpgradeCommandAsync(envelope, message, cancellationToken);
break;
case MessageTypes.CommandAgentRemove:
// Acknowledged once removal.json holds it: a redelivery finds it there.
await removals.HandleAsync(envelope.GetPayload<AgentRemoveCommandPayload>(), cancellationToken);
await message.AckAsync(cancellationToken: cancellationToken);
break;
default:
logger.LogWarning("Unknown command type {MessageType}; terminating delivery", envelope.MessageType);
await message.AckTerminateAsync(cancellationToken: cancellationToken);
+404
View File
@@ -0,0 +1,404 @@
using System.Buffers.Text;
using System.Net;
using System.Net.Http.Json;
using System.Text;
using System.Text.Json;
using Agent.Configuration;
using Agent.Storage;
using Agent.Upgrade;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using NATS.NKeys;
using Shared.Contracts.Api;
using Shared.Contracts.Models;
using Shared.Contracts.Serialization;
namespace Agent.Enrollment;
/// <summary>What a NATS user JWT says about its own lifetime (its signature is the server's to check).</summary>
public static class AgentJwt
{
/// <summary><c>iat</c> and <c>exp</c>; either is null when absent, both when it is not a readable JWT.</summary>
public static (DateTimeOffset? IssuedUtc, DateTimeOffset? ExpiresUtc) ReadLifetime(string jwt)
{
var parts = jwt.Split('.');
if (parts.Length != 3)
{
return (null, null);
}
try
{
using var claims = JsonDocument.Parse(Base64Url.DecodeFromChars(parts[1]));
if (claims.RootElement.ValueKind != JsonValueKind.Object)
{
return (null, null);
}
return (Seconds(claims.RootElement, "iat"), Seconds(claims.RootElement, "exp"));
}
catch (Exception exception) when (exception is FormatException or JsonException)
{
return (null, null);
}
}
private static DateTimeOffset? Seconds(JsonElement claims, string name)
=> claims.TryGetProperty(name, out var value) && value.TryGetInt64(out var seconds) && seconds > 0
? DateTimeOffset.FromUnixTimeSeconds(seconds)
: null;
}
/// <summary>What credential renewal works with; every seam a test needs is here.</summary>
public sealed class CredentialRenewalContext
{
public required ICredentialStore Store { get; init; }
public required RuntimeStateStore RuntimeState { get; init; }
public required AgentConfigurationLoader Configuration { get; init; }
public required UpgradeFolders Folders { get; init; }
/// <summary>The server refused the credential for good: a removed device (the id, when the answer names one).</summary>
public required Func<Guid, CancellationToken, Task> OnRemoved { get; init; }
/// <summary>A 401 while a bootstrap token is configured: enrolling again is how a device comes back.</summary>
public required Func<AgentConfiguration, CancellationToken, Task> Reenroll { get; init; }
/// <summary>The HTTP handler: OS trust (the managed CA was installed there at enrollment), or accept-any when configured.</summary>
public Func<bool, HttpMessageHandler>? CreateHttpHandler { get; init; }
public Func<DateTimeOffset> UtcNow { get; init; } = () => DateTimeOffset.UtcNow;
public Func<TimeSpan, CancellationToken, Task> Delay { get; init; } = Task.Delay;
}
/// <summary>
/// Keeps the agent's NATS credential current over HTTPS, at every start and at
/// half life, whether or not NATS is connected: an expired credential recovers by
/// itself as soon as the server can be reached, since the renewal is signed with
/// the enrolled NKey and not with the credential. The old one stays in the store
/// until a new one replaces it, and the connection takes the new one at its next
/// reconnect (<see cref="Messaging.AgentNatsOptionsFactory"/>), at the latest when
/// the server ends it as the old one expires.
/// <list type="bullet">
/// <item><b>Upgrades:</b> while an upgrade could still roll back, the stored
/// credential stays: a version before <see cref="DeviceRemovalDefaults.MinimumAgentVersion"/>
/// cannot renew and would strand on a short-lived one. Only one about to expire
/// is replaced then: it would strand either way, and the new version could not
/// report healthy.</item>
/// <item><b>410 (removed):</b> the agent uninstalls itself, unless the server
/// could not be verified (<c>TlsSkipVerify</c>, or plain http): an unverified
/// server must not be able to uninstall an agent.</item>
/// <item><b>401 (orphaned):</b> tried again hourly, and a configured bootstrap
/// token enrolls the device again.</item>
/// <item><b>Network or server errors:</b> tried again with back-off.</item>
/// </list>
/// </summary>
public sealed class CredentialRenewal(CredentialRenewalContext context, ILogger logger)
{
public static readonly TimeSpan RequestTimeout = TimeSpan.FromSeconds(30);
public static readonly TimeSpan RefusedRetry = TimeSpan.FromHours(1);
public static readonly TimeSpan BackoffMin = TimeSpan.FromSeconds(30);
public static readonly TimeSpan BackoffMax = TimeSpan.FromMinutes(15);
/// <summary>The longest one wait: the clock may jump (a sleeping laptop), so it is looked at again.</summary>
public static readonly TimeSpan MaxWait = TimeSpan.FromMinutes(5);
/// <summary>How often a state that blocks renewal (no credential, an upgrade, a removal) is looked at again.</summary>
public static readonly TimeSpan BlockedRecheck = TimeSpan.FromMinutes(1);
/// <summary>A credential this close to its end is replaced even during an upgrade.</summary>
public static readonly TimeSpan ExpiringSoon = TimeSpan.FromHours(1);
private static readonly TimeSpan HalfLifetime = AgentCredentialDefaults.Lifetime / 2;
private bool answeredSinceStart;
private bool upgradeNoted;
private int failures;
private DateTimeOffset notBefore = DateTimeOffset.MinValue;
private DateTimeOffset? renewedUtc;
public async Task RunAsync(CancellationToken cancellationToken)
{
while (!cancellationToken.IsCancellationRequested)
{
TimeSpan wait;
try
{
wait = await PassAsync(cancellationToken);
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
return;
}
catch (Exception exception)
{
logger.LogWarning(exception, "Credential renewal failed; it is tried again");
wait = Backoff();
}
try
{
await context.Delay(wait, cancellationToken);
}
catch (OperationCanceledException)
{
return;
}
}
}
/// <summary>Renews when it is time to; how long until the next pass.</summary>
public async Task<TimeSpan> PassAsync(CancellationToken cancellationToken)
{
var now = context.UtcNow();
if (now < notBefore)
{
return Capped(notBefore - now);
}
var configuration = context.Configuration.Load();
var stored = context.Store.Load();
if (stored is null || configuration.ServerUrl is null)
{
return BlockedRecheck;
}
if (WatchdogFileStore.Read(context.Folders.Removal, WatchdogJsonContext.Default.WatchdogRemoval) is { Phase: not WatchdogRemovalPhases.Failed })
{
return BlockedRecheck;
}
var expires = ExpiryOf(stored);
if (answeredSinceStart && DueAt(stored, expires) is { } due && due > now)
{
return Capped(due - now);
}
if (context.Folders.UpgradeUnderWay && (expires is null || expires - now > ExpiringSoon))
{
if (!upgradeNoted)
{
logger.LogInformation("The credential is renewed once the upgrade under way is kept; a rollback could not renew it");
upgradeNoted = true;
}
return BlockedRecheck;
}
upgradeNoted = false;
return await RenewAsync(configuration, stored, cancellationToken);
}
/// <summary>The JWT's own <c>exp</c>, else what the server said when it issued it; null for a credential that never expires.</summary>
public static DateTimeOffset? ExpiryOf(AgentCredentials credentials)
=> AgentJwt.ReadLifetime(credentials.UserJwt).ExpiresUtc ?? credentials.ExpiresUtc;
/// <summary>
/// Half way between <c>iat</c> and <c>exp</c> (<c>iat</c> taken as a full lifetime
/// back when absent). One that does not expire, however the server answers, is
/// asked again every half lifetime.
/// </summary>
private DateTimeOffset? DueAt(AgentCredentials credentials, DateTimeOffset? expires)
{
if (expires is null)
{
return renewedUtc + HalfLifetime;
}
var issued = AgentJwt.ReadLifetime(credentials.UserJwt).IssuedUtc ?? expires.Value - AgentCredentialDefaults.Lifetime;
return issued + (expires.Value - issued) / 2;
}
private async Task<TimeSpan> RenewAsync(AgentConfiguration configuration, AgentCredentials stored, CancellationToken cancellationToken)
{
var skipVerify = configuration.TlsSkipVerify.Value;
var endpoint = configuration.ServerUrl!.ApiEndpoint(ApiRoutes.RefreshCredential);
HttpResponseMessage response;
try
{
using var handler = CreateHandler(skipVerify);
using var http = new HttpClient(handler, disposeHandler: false) { Timeout = RequestTimeout };
response = await http.PostAsJsonAsync(endpoint, Sign(stored), JsonDefaults.Options, cancellationToken);
}
catch (Exception exception) when (exception is HttpRequestException or TaskCanceledException && !cancellationToken.IsCancellationRequested)
{
var wait = Backoff();
logger.LogWarning(exception, "The server could not be reached to renew the credential; trying again in {Wait}", wait);
return Capped(wait);
}
using (response)
{
switch (response.StatusCode)
{
case HttpStatusCode.OK:
return await StoreAsync(stored, response, cancellationToken);
case HttpStatusCode.Gone when await ReadRemovedAsync(response, cancellationToken) is { } removalId:
answeredSinceStart = true;
return await RemovedAsync(removalId, endpoint, skipVerify, cancellationToken);
case HttpStatusCode.Unauthorized:
answeredSinceStart = true;
logger.LogWarning("The server refused to renew the credential (401; orphaned, or the clock is off); trying again in {Wait}", RefusedRetry);
Refused();
if (configuration.HasBootstrapToken)
{
await context.Reenroll(configuration, cancellationToken);
}
return Capped(RefusedRetry);
default:
var wait = Backoff();
logger.LogWarning("The server answered {Status} to renewing the credential; trying again in {Wait}", (int)response.StatusCode, wait);
return Capped(wait);
}
}
}
private async Task<TimeSpan> StoreAsync(AgentCredentials stored, HttpResponseMessage response, CancellationToken cancellationToken)
{
var result = await response.Content.ReadFromJsonAsync<CredentialRefreshResponse>(JsonDefaults.Options, cancellationToken);
if (result is null || string.IsNullOrWhiteSpace(result.UserJwt))
{
var wait = Backoff();
logger.LogWarning("The server's new credential was empty; trying again in {Wait}", wait);
return Capped(wait);
}
// Enrolled again meanwhile (a new key): this answer is for the old one.
if (context.Store.Load() is not { } current || current.NkeySeed != stored.NkeySeed)
{
logger.LogInformation("The agent enrolled again during the renewal; its answer is dropped");
return TimeSpan.Zero;
}
var expires = AgentJwt.ReadLifetime(result.UserJwt).ExpiresUtc ?? result.CredentialExpiresUtc;
var renewed = current with { UserJwt = result.UserJwt, ExpiresUtc = expires };
context.Store.Save(renewed);
context.RuntimeState.CredentialExpiresUtc = expires;
answeredSinceStart = true;
failures = 0;
notBefore = DateTimeOffset.MinValue;
renewedUtc = context.UtcNow();
logger.LogInformation("Credential renewed (expires {Expires})", expires?.ToString("O") ?? "never");
var due = DueAt(renewed, expires);
return due is { } at && at > context.UtcNow() ? Capped(at - context.UtcNow()) : BlockedRecheck;
}
private async Task<TimeSpan> RemovedAsync(Guid removalId, Uri endpoint, bool skipVerify, CancellationToken cancellationToken)
{
Refused();
if (skipVerify || endpoint.Scheme != Uri.UriSchemeHttps)
{
logger.LogWarning(
"The server at {Endpoint} says this device was removed, but it is not verified (TlsSkipVerify, or plain http), so the agent stays; remove it by hand if that is right",
endpoint);
return Capped(RefusedRetry);
}
logger.LogWarning("The server says this device was removed");
await context.OnRemoved(removalId, cancellationToken);
return Capped(RefusedRetry);
}
private void Refused()
{
failures = 0;
notBefore = context.UtcNow() + RefusedRetry;
}
/// <summary>The removal id of a real removal's 410 (empty when the answer names none); null for any other 410, say from a proxy.</summary>
private static async Task<Guid?> ReadRemovedAsync(HttpResponseMessage response, CancellationToken cancellationToken)
{
try
{
using var problem = JsonDocument.Parse(await response.Content.ReadAsStringAsync(cancellationToken));
var root = problem.RootElement;
if (root.ValueKind != JsonValueKind.Object
|| !root.TryGetProperty("type", out var type)
|| type.ValueKind != JsonValueKind.String
|| type.GetString() != AgentCredentialDefaults.RemovedProblemType)
{
return null;
}
return root.TryGetProperty("removalId", out var id) && id.ValueKind == JsonValueKind.String && Guid.TryParse(id.GetString(), out var removalId)
? removalId
: Guid.Empty;
}
catch (JsonException)
{
return null;
}
}
/// <summary>Proof of the enrolled key: <c>{clientId}|{timestampUtc:O}</c>, signed with the seed.</summary>
private CredentialRefreshRequest Sign(AgentCredentials stored)
{
using var key = KeyPair.FromSeed(stored.NkeySeed.AsSpan());
var timestamp = context.UtcNow().ToUniversalTime();
var signature = new byte[64];
key.Sign(Encoding.UTF8.GetBytes($"{stored.ClientId}|{timestamp:O}"), signature);
return new CredentialRefreshRequest(stored.ClientId, key.GetPublicKey(), timestamp, Convert.ToBase64String(signature));
}
private HttpMessageHandler CreateHandler(bool skipVerify)
{
if (context.CreateHttpHandler is { } create)
{
return create(skipVerify);
}
var handler = new HttpClientHandler();
if (skipVerify)
{
handler.ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator;
}
return handler;
}
/// <summary>30 s doubling to 15 minutes; the same for every failure until an answer comes.</summary>
private TimeSpan Backoff()
{
var wait = TimeSpan.FromTicks(Math.Min(BackoffMin.Ticks << Math.Min(failures, 10), BackoffMax.Ticks));
failures++;
notBefore = context.UtcNow() + wait;
return wait;
}
private static TimeSpan Capped(TimeSpan wait) => wait < MaxWait ? wait : MaxWait;
}
/// <summary>Runs <see cref="CredentialRenewal"/> for as long as the agent runs, NATS connected or not.</summary>
public sealed class CredentialRenewalService(CredentialRenewal renewal) : BackgroundService
{
protected override Task ExecuteAsync(CancellationToken stoppingToken) => renewal.RunAsync(stoppingToken);
}
/// <summary>Takes a 410 from the renewal to the removal.</summary>
public static class CredentialRemoval
{
/// <summary>Starts the removal the server's refusal asks for; what cannot start now is asked again by the next refusal.</summary>
public static async Task HandleAsync(AgentRemoval removal, Guid removalId, ILogger logger, CancellationToken cancellationToken)
{
if (removal.UnderWay)
{
return;
}
if (removal.Refusal(strict: false) is { } refusal)
{
logger.LogWarning("This device was removed from the server, but the agent cannot remove itself now ({Reason})", refusal);
return;
}
if (await removal.HandOverAsync(removalId, cancellationToken) is { } problem)
{
logger.LogError("This device was removed from the server, but the removal did not start: {Problem}", problem);
}
}
}
+40 -7
View File
@@ -88,16 +88,32 @@ public sealed class CredentialStore(ILogger<CredentialStore> logger) : ICredenti
var payload = plain;
#endif
// A temporary name and a rename: renewal replaces this file for the life
// of the agent, and half a file would be an agent that is no longer enrolled.
var path = Path.Combine(directory, Branding.CredentialFileName);
File.WriteAllBytes(path, payload);
#if WINDOWS_AGENT
ApplyRestrictedAcl(new FileInfo(path));
#else
if (!OperatingSystem.IsWindows())
var temporary = $"{path}.{Guid.NewGuid():N}.tmp";
try
{
File.SetUnixFileMode(path, UnixFileMode.UserRead | UnixFileMode.UserWrite);
}
File.WriteAllBytes(temporary, payload);
#if WINDOWS_AGENT
ApplyRestrictedAcl(new FileInfo(temporary));
#else
if (!OperatingSystem.IsWindows())
{
File.SetUnixFileMode(temporary, UnixFileMode.UserRead | UnixFileMode.UserWrite);
}
#endif
MoveOver(temporary, path);
}
catch
{
if (File.Exists(temporary))
{
File.Delete(temporary);
}
throw;
}
logger.LogInformation("Credentials persisted for client {ClientId}", credentials.ClientId);
}
@@ -110,6 +126,23 @@ public sealed class CredentialStore(ILogger<CredentialStore> logger) : ICredenti
}
}
/// <summary>Windows refuses a rename over a file another thread is reading for a moment (a reconnect reads it).</summary>
private static void MoveOver(string temporary, string path)
{
for (var attempt = 1; ; attempt++)
{
try
{
File.Move(temporary, path, overwrite: true);
return;
}
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException && attempt < 40)
{
Thread.Sleep(25);
}
}
}
private static string EnsureProtectedDirectory()
{
var directory = new DirectoryInfo(Branding.AgentProgramDataDirectory);
+21 -3
View File
@@ -105,7 +105,7 @@ public static class ServerTrust
}
else if (OperatingSystem.IsMacOS())
{
InstallMac(pem, logger);
InstallMac(pem, chain, logger);
}
else
{
@@ -168,7 +168,7 @@ public static class ServerTrust
}
else if (OperatingSystem.IsMacOS())
{
InstallMac(pem, logger);
InstallMac(pem, chain, logger);
}
else
{
@@ -201,6 +201,7 @@ public static class ServerTrust
if (store.Certificates.Find(X509FindType.FindByThumbprint, cert.Thumbprint, validOnly: false).Count == 0)
{
store.Add(cert);
RememberInstalled([cert.Thumbprint], logger);
logger.LogInformation("Installed server CA into LocalMachine\\{Store}: {Subject}",
selfIssued ? "Root" : "CA", cert.Subject);
}
@@ -212,6 +213,22 @@ public static class ServerTrust
}
}
/// <summary>Notes what the agent itself added to the trust store, the only certificates a removal takes back out.</summary>
private static void RememberInstalled(IEnumerable<string> thumbprints, ILogger logger)
{
try
{
var path = Path.Combine(Shared.Infrastructure.Constants.Branding.AgentProgramDataDirectory, Shared.Contracts.Models.WatchdogFiles.ManagedCa);
Directory.CreateDirectory(Path.GetDirectoryName(path)!);
var known = File.Exists(path) ? File.ReadAllLines(path) : [];
File.WriteAllLines(path, known.Union(thumbprints, StringComparer.OrdinalIgnoreCase));
}
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
{
logger.LogWarning(ex, "Could not note the installed server CA; a removal will leave it in the trust store");
}
}
private static void InstallLinux(string pem, ILogger logger)
{
try
@@ -238,7 +255,7 @@ public static class ServerTrust
}
}
private static void InstallMac(string pem, ILogger logger)
private static void InstallMac(string pem, X509Certificate2Collection chain, ILogger logger)
{
try
{
@@ -246,6 +263,7 @@ public static class ServerTrust
File.WriteAllText(path, pem);
RunBestEffort("security", logger,
"add-trusted-cert", "-d", "-r", "trustRoot", "-k", "/Library/Keychains/System.keychain", path);
RememberInstalled(chain.Cast<X509Certificate2>().Select(c => c.Thumbprint), logger);
File.Delete(path);
}
catch (Exception ex)
@@ -17,7 +17,8 @@ public static class AgentNatsOptionsFactory
ServerUrl serverUrl,
AgentCredentials credentials,
AgentConfiguration configuration,
ILoggerFactory loggerFactory)
ILoggerFactory loggerFactory,
Func<AgentCredentials?>? current = null)
{
var options = NatsOpts.Default with
{
@@ -25,8 +26,13 @@ public static class AgentNatsOptionsFactory
Name = $"agent-{credentials.ClientId}",
AuthOpts = NatsAuthOpts.Default with
{
Jwt = credentials.UserJwt,
Seed = credentials.NkeySeed,
// Asked at every connect, so a reconnect takes the credential
// renewed meanwhile, and an expired one recovers by itself.
AuthCredCallback = (_, _) =>
{
var latest = current?.Invoke() ?? credentials;
return ValueTask.FromResult(NatsAuthCred.FromJwt(latest.UserJwt, latest.NkeySeed));
},
},
ReconnectWaitMin = configuration.ReconnectMinDelay.Value,
ReconnectWaitMax = configuration.ReconnectMaxDelay.Value,
+47
View File
@@ -0,0 +1,47 @@
using System.Globalization;
using System.Management.Automation.Host;
namespace Agent.PowerShell;
/// <summary>
/// The in-process engine's host: behaves like the engine's own default host
/// (no UI, so Write-Host stays on the information stream) and remembers what
/// <c>SetShouldExit</c> was given. <c>exit N</c> in a script run through
/// AddScript reaches the host this way and nowhere else: the engine sets
/// neither <c>$LASTEXITCODE</c> nor HadErrors for it.
/// </summary>
internal sealed class ExitCapturingHost : PSHost
{
private readonly CultureInfo culture = CultureInfo.CurrentCulture;
private readonly CultureInfo uiCulture = CultureInfo.CurrentUICulture;
private readonly Guid instanceId = Guid.NewGuid();
/// <summary>The last exit code the script asked for; null when it never called exit.</summary>
public int? ExitCode { get; private set; }
public override CultureInfo CurrentCulture => culture;
public override CultureInfo CurrentUICulture => uiCulture;
public override Guid InstanceId => instanceId;
public override string Name => "Default Host";
public override PSHostUserInterface? UI => null;
public override Version Version => new(1, 0);
public override void EnterNestedPrompt() => throw new NotSupportedException();
public override void ExitNestedPrompt() => throw new NotSupportedException();
public override void NotifyBeginApplication()
{
}
public override void NotifyEndApplication()
{
}
public override void SetShouldExit(int exitCode) => ExitCode = exitCode;
}
@@ -3,8 +3,33 @@ using System.Management.Automation.Runspaces;
namespace Agent.PowerShell;
/// <summary>Result of one in-process PowerShell execution.</summary>
public sealed record PowerShellExecutionResult(bool HadErrors, bool WasCancelled, string? FailureMessage);
/// <summary>The first error a script hit: its exception type, fully qualified error id and message.</summary>
public sealed record ScriptErrorInfo(string? Type, string? Id, string? Message)
{
public const int MaxTextLength = 2000;
public const int MaxNameLength = 300;
public static ScriptErrorInfo From(ErrorRecord record) => new(
Cut(record.Exception?.GetType().FullName, MaxNameLength),
Cut(record.FullyQualifiedErrorId, MaxNameLength),
Cut(record.Exception?.Message ?? record.ToString(), MaxTextLength));
private static string? Cut(string? value, int length)
=> value is { Length: > 0 } ? (value.Length > length ? value[..length] : value) : null;
}
/// <summary>
/// Result of one in-process PowerShell execution. <paramref name="ExitCode"/>
/// is what the script's own <c>exit N</c> asked for (null when it never called
/// exit); it decides the outcome over <paramref name="HadErrors"/>, as
/// <c>pwsh -File</c> does.
/// </summary>
public sealed record PowerShellExecutionResult(
bool HadErrors,
bool WasCancelled,
string? FailureMessage,
int? ExitCode = null,
ScriptErrorInfo? Error = null);
/// <summary>
/// Executes scripts in-process through System.Management.Automation:
@@ -40,7 +65,8 @@ public sealed class PowerShellExecutionService
IReadOnlyDictionary<string, Shared.Contracts.Models.ValueExpansion>? expansions = null,
IReadOnlyDictionary<string, object?>? sessionVariables = null)
{
using var runspace = PowerShellRunspaceFactory.Create();
var host = new ExitCapturingHost();
using var runspace = PowerShellRunspaceFactory.Create(host);
using var powerShell = System.Management.Automation.PowerShell.Create();
powerShell.Runspace = runspace;
@@ -78,8 +104,13 @@ public sealed class PowerShellExecutionService
var output = new PSDataCollection<PSObject>();
output.DataAdded += (_, e) => onRecord(PowerShellOutputNormalizer.FromOutput(output[e.Index]));
ScriptErrorInfo? firstError = null;
powerShell.Streams.Error.DataAdded += (_, e) =>
onRecord(PowerShellOutputNormalizer.FromError(powerShell.Streams.Error[e.Index]));
{
var record = powerShell.Streams.Error[e.Index];
firstError ??= ScriptErrorInfo.From(record);
onRecord(PowerShellOutputNormalizer.FromError(record));
};
powerShell.Streams.Warning.DataAdded += (_, e) =>
onRecord(PowerShellOutputNormalizer.FromWarning(powerShell.Streams.Warning[e.Index]));
powerShell.Streams.Verbose.DataAdded += (_, e) =>
@@ -118,13 +149,18 @@ public sealed class PowerShellExecutionService
{
// Terminating error: surface as an error record + failed result.
failureMessage = ex.Message;
firstError ??= ScriptErrorInfo.From(ex.ErrorRecord);
onRecord(new NormalizedOutputRecord(Shared.Contracts.Models.OutputStreamKind.Error, ex.Message));
}
return new PowerShellExecutionResult(
HadErrors: powerShell.HadErrors || failureMessage is not null,
// Not powerShell.HadErrors: that is also set by a native command's non-zero
// exit and by errors the script silenced, neither of which fails a script.
HadErrors: firstError is not null || failureMessage is not null,
WasCancelled: wasCancelled,
FailureMessage: failureMessage);
FailureMessage: failureMessage,
ExitCode: host.ExitCode,
Error: firstError);
}
/// <summary>
+32 -14
View File
@@ -68,6 +68,7 @@ public sealed class PowerShellJobHandler(
JobState finalState;
int exitCode;
string? message = null;
ScriptErrorInfo? error = null;
// S3 file deliveries land before the script starts. Destination
// paths expand on the device (%VAR% natively; $env:/$(…) through a
@@ -114,6 +115,8 @@ public sealed class PowerShellJobHandler(
else
{
var externalParameters = ExpandForExternal(boundParameters, expansions);
// An external engine reports no exception: its first stderr line stands in.
string? firstStderr = null;
var external = await ExternalScriptRunner.RunAsync(
command.Engine,
command.Content,
@@ -121,7 +124,15 @@ public sealed class PowerShellJobHandler(
externalParameters,
command.Secrets,
string.IsNullOrWhiteSpace(command.SecretsVariableName) ? "Secrets" : command.SecretsVariableName,
record => pending.Add(PublishOutputAsync(record, Interlocked.Increment(ref sequence))),
record =>
{
if (record.Stream == OutputStreamKind.Error && !string.IsNullOrWhiteSpace(record.Text))
{
Interlocked.CompareExchange(ref firstStderr, record.Text, null);
}
pending.Add(PublishOutputAsync(record, Interlocked.Increment(ref sequence)));
},
timeoutCts.Token,
files: filesDict);
@@ -142,6 +153,12 @@ public sealed class PowerShellJobHandler(
finalState = external.ExitCode == 0 ? JobState.Succeeded : JobState.Failed;
exitCode = external.ExitCode;
message = external.ExitCode == 0 ? null : $"Process exited with {external.ExitCode}.";
if (external.ExitCode != 0 && firstStderr is not null)
{
error = new ScriptErrorInfo(null, null, firstStderr.Length > ScriptErrorInfo.MaxTextLength
? firstStderr[..ScriptErrorInfo.MaxTextLength]
: firstStderr);
}
}
}
@@ -173,29 +190,28 @@ public sealed class PowerShellJobHandler(
exitCode = -1;
message = $"Execution exceeded the {timeout.TotalSeconds:F0}s timeout.";
}
else if (result.HadErrors)
{
finalState = JobState.Failed;
exitCode = 1;
message = result.FailureMessage;
}
else
{
finalState = JobState.Succeeded;
exitCode = 0;
(finalState, exitCode, message) = ScriptOutcome.From(result);
if (finalState == JobState.Failed)
{
error = result.Error;
}
}
}
jobHistory.MarkCompleted(jobId, finalState.ToString());
await PublishStateAsync(finalState, exitCode, message);
await PublishStateAsync(finalState, exitCode, message, error);
logger.LogInformation("Job {JobId} finished: {State}", jobId, finalState);
logger.LogInformation("Job {JobId} finished: {State} ({ExitCode})", jobId, finalState, exitCode);
}
catch (Exception ex)
{
logger.LogError(ex, "Job {JobId} crashed", jobId);
jobHistory.MarkCompleted(jobId, JobState.Failed.ToString());
await PublishStateAsync(JobState.Failed, exitCode: 1, message: ex.Message);
await PublishStateAsync(
JobState.Failed, exitCode: 1, message: ex.Message,
new ScriptErrorInfo(ex.GetType().FullName, null, ex.Message));
}
finally
{
@@ -293,12 +309,14 @@ public sealed class PowerShellJobHandler(
}
}
async Task PublishStateAsync(JobState state, int? exitCode, string? message)
async Task PublishStateAsync(JobState state, int? exitCode, string? message, ScriptErrorInfo? error = null)
{
await publisher.PublishAsync(
NatsSubjects.JobState(jobId),
MessageTypes.JobState,
new JobStatePayload(jobId, publisher.ClientId, state, exitCode, DateTimeOffset.UtcNow, message),
new JobStatePayload(
jobId, publisher.ClientId, state, exitCode, DateTimeOffset.UtcNow, message,
ErrorType: error?.Type, ErrorId: error?.Id, ErrorMessage: error?.Message),
jobTarget,
correlationId,
causationId);
@@ -7,14 +7,17 @@ public static class PowerShellRunspaceFactory
{
private static bool modulesPathEnsured;
public static Runspace Create()
/// <summary>A new opened runspace; <paramref name="host"/> replaces the engine's default host.</summary>
public static Runspace Create(System.Management.Automation.Host.PSHost? host = null)
{
EnsureBundledModulesOnPath();
var initialSessionState = InitialSessionState.CreateDefault();
initialSessionState.ExecutionPolicy = Microsoft.PowerShell.ExecutionPolicy.Bypass;
var runspace = RunspaceFactory.CreateRunspace(initialSessionState);
var runspace = host is null
? RunspaceFactory.CreateRunspace(initialSessionState)
: RunspaceFactory.CreateRunspace(host, initialSessionState);
runspace.Open();
return runspace;
}
+24
View File
@@ -0,0 +1,24 @@
using Shared.Contracts.Models;
namespace Agent.PowerShell;
/// <summary>How a finished in-process script reports: its state, exit code and summary.</summary>
public sealed record ScriptOutcome(JobState State, int ExitCode, string? Message)
{
/// <summary>
/// Mirrors <c>pwsh -File</c>: the script's own <c>exit N</c> decides (N≠0 is
/// a failure, even with no errors; 0 a success, even after errors), a
/// terminating error is exit 1, and a native command's failing
/// <c>$LASTEXITCODE</c> counts only when the script exits with it. One
/// difference, kept on purpose: anything on the error stream (Write-Error, a
/// failing cmdlet) fails a script that never calls exit, with exit code 1.
/// </summary>
public static ScriptOutcome From(PowerShellExecutionResult result) => result switch
{
{ ExitCode: 0 } => new ScriptOutcome(JobState.Succeeded, 0, null),
{ ExitCode: { } code } => new ScriptOutcome(
JobState.Failed, code, result.FailureMessage ?? result.Error?.Message ?? $"The script exited with {code}."),
{ HadErrors: true } => new ScriptOutcome(JobState.Failed, 1, result.FailureMessage),
_ => new ScriptOutcome(JobState.Succeeded, 0, null),
};
}
+45
View File
@@ -6,6 +6,7 @@ using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Serilog;
using Shared.Contracts.Models;
using Shared.Infrastructure.Constants;
using Shared.Infrastructure.Logging;
@@ -74,7 +75,30 @@ public static class AgentHost
// service mode (and never override GPO in any mode).
builder.Services.AddSingleton(new AgentConfigurationLoader(registry, allowEnvironment: !runAsService));
builder.Services.AddSingleton(sp => CreateRemoval(
sp.GetRequiredService<Upgrade.AgentRuntimeInfo>(),
sp.GetRequiredService<Upgrade.UpgradeFolders>(),
sp.GetRequiredService<ILoggerFactory>()));
builder.Services.AddSingleton(sp =>
{
var removal = sp.GetRequiredService<Upgrade.AgentRemoval>();
var enrollment = sp.GetRequiredService<EnrollmentService>();
var logger = sp.GetRequiredService<ILoggerFactory>().CreateLogger<CredentialRenewal>();
return new CredentialRenewal(
new CredentialRenewalContext
{
Store = sp.GetRequiredService<ICredentialStore>(),
RuntimeState = sp.GetRequiredService<RuntimeStateStore>(),
Configuration = sp.GetRequiredService<AgentConfigurationLoader>(),
Folders = sp.GetRequiredService<Upgrade.UpgradeFolders>(),
OnRemoved = (removalId, cancellationToken) => CredentialRemoval.HandleAsync(removal, removalId, logger, cancellationToken),
Reenroll = (configuration, cancellationToken) => enrollment.EnsureEnrolledAsync(configuration, cancellationToken),
},
logger);
});
builder.Services.AddHostedService<AgentWorker>();
builder.Services.AddHostedService<CredentialRenewalService>();
// Keeps a self-signed server's CA current in the OS trust store across
// renewals (no-op for a real/edge certificate).
builder.Services.AddHostedService<CaRefreshService>();
@@ -84,8 +108,29 @@ public static class AgentHost
if (runAsService)
{
builder.Services.AddHostedService<Upgrade.HealthBeaconService>();
builder.Services.AddHostedService<Upgrade.RemovalPickupService>();
}
return builder.Build();
}
private static Upgrade.AgentRemoval CreateRemoval(Upgrade.AgentRuntimeInfo runtime, Upgrade.UpgradeFolders folders, ILoggerFactory loggers)
{
var logger = loggers.CreateLogger<Upgrade.AgentRemoval>();
var options = new Agent.Watchdog.WatchdogOptions
{
InstallDirectory = Path.GetFullPath(folders.InstallDirectory),
DataDirectory = Path.GetFullPath(folders.DataDirectory),
};
return new Upgrade.AgentRemoval(
new Upgrade.AgentRemovalContext
{
Folders = folders,
CanRemove = runtime.CanSelfUpgrade,
Watchdog = new Upgrade.WatchdogInstaller(folders.InstallDirectory, logger),
SelfRunner = Agent.Watchdog.Removal.RemovalRunnerLauncher.Create(
options, new Agent.Watchdog.WatchdogLog(directory: null, console: false), WatchdogRemovalFiles.AgentRunVerb),
},
logger);
}
}
+14 -3
View File
@@ -32,10 +32,12 @@ public sealed class AgentSessionFactory(
JobHistoryStore jobHistory,
AgentRuntimeInfo runtime,
AgentHealthState health,
AgentRemoval removal,
ICredentialStore credentialStore,
ILoggerFactory loggerFactory) : IAgentSessionFactory
{
public IAgentSession Create(NatsConnection connection, AgentCredentials credentials, AgentConfiguration configuration)
=> new AgentSession(connection, credentials, configuration, hardwareIdentity, jobHistory, runtime, health, loggerFactory);
=> new AgentSession(connection, credentials, configuration, hardwareIdentity, jobHistory, runtime, health, removal, credentialStore, loggerFactory);
}
/// <summary>
@@ -50,6 +52,7 @@ public sealed class AgentSession : IAgentSession
private readonly CommandProcessor commands;
private readonly ShellCommandListener shell;
private readonly UpgradeOutcomeReporter upgradeOutcomes;
private readonly RemovalOutcomeReporter removalOutcomes;
private readonly NatsConnection connection;
private readonly AgentHealthState health;
private readonly ILogger<AgentSession> logger;
@@ -64,6 +67,8 @@ public sealed class AgentSession : IAgentSession
JobHistoryStore jobHistory,
AgentRuntimeInfo runtime,
AgentHealthState health,
AgentRemoval removal,
ICredentialStore credentialStore,
ILoggerFactory loggerFactory)
{
this.connection = connection;
@@ -96,13 +101,18 @@ public sealed class AgentSession : IAgentSession
loggerFactory.CreateLogger<AgentUpgradeHandler>());
upgradeOutcomes = new UpgradeOutcomeReporter(
folders, upgradeEvents, () => health.Connected, credentials.ClientId, loggerFactory.CreateLogger<UpgradeOutcomeReporter>());
var removalEvents = new NatsRemovalEvents(publisher);
var removals = new AgentRemoveCommandHandler(
removal, () => credentialStore.Load()?.EnrollmentId, removalEvents, credentials.ClientId, loggerFactory.CreateLogger<AgentRemoveCommandHandler>());
removalOutcomes = new RemovalOutcomeReporter(
folders, removalEvents, () => health.Connected, credentials.ClientId, loggerFactory.CreateLogger<RemovalOutcomeReporter>());
presence = new PresencePublisher(
publisher, () => jobHandler.ActiveJobs, loggerFactory.CreateLogger<PresencePublisher>());
inventory = new InventoryPublisher(
collector, publisher, configuration.InventoryInterval.Value, loggerFactory.CreateLogger<InventoryPublisher>());
commands = new CommandProcessor(
connection, publisher, inventory, jobHandler, fileHandler, jobHistory, configuration, upgrades, health,
connection, publisher, inventory, jobHandler, fileHandler, jobHistory, configuration, upgrades, removals, health,
loggerFactory.CreateLogger<CommandProcessor>());
shell = new ShellCommandListener(
connection, publisher, configuration, loggerFactory.CreateLogger<ShellCommandListener>());
@@ -126,7 +136,8 @@ public sealed class AgentSession : IAgentSession
inventory.RunAsync(cancellationToken),
commands.RunAsync(cancellationToken),
shell.RunAsync(cancellationToken),
upgradeOutcomes.RunAsync(cancellationToken));
upgradeOutcomes.RunAsync(cancellationToken),
removalOutcomes.RunAsync(cancellationToken));
}
finally
{
+2 -1
View File
@@ -17,6 +17,7 @@ namespace Agent.Service;
public sealed class AgentWorker(
AgentConfigurationLoader configurationLoader,
EnrollmentService enrollment,
ICredentialStore credentialStore,
RuntimeStateStore runtimeState,
IAgentSessionFactory sessionFactory,
ILoggerFactory loggerFactory,
@@ -51,7 +52,7 @@ public sealed class AgentWorker(
continue;
}
var opts = AgentNatsOptionsFactory.Build(configuration.ServerUrl, credentials, configuration, loggerFactory);
var opts = AgentNatsOptionsFactory.Build(configuration.ServerUrl, credentials, configuration, loggerFactory, credentialStore.Load);
await using var nats = new NatsConnection(opts);
+295
View File
@@ -0,0 +1,295 @@
using Agent.Messaging;
using Agent.Watchdog.Removal;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Shared.Contracts.Messages;
using Shared.Contracts.Models;
using Shared.Contracts.Serialization;
using Shared.Infrastructure.Nats;
namespace Agent.Upgrade;
/// <summary>Where removal progress goes: the agent event subject in production, a list in tests.</summary>
public interface IRemovalEvents
{
Task PublishAsync(AgentRemovalEventPayload payload, CancellationToken cancellationToken);
}
public sealed class NatsRemovalEvents(AgentPublisher publisher) : IRemovalEvents
{
public async Task PublishAsync(AgentRemovalEventPayload payload, CancellationToken cancellationToken)
=> await publisher.PublishAsync(
NatsSubjects.AgentEvent(publisher.ClientId), MessageTypes.AgentRemoval, payload, cancellationToken: cancellationToken);
}
/// <summary>What the agent's side of a removal works with; every seam a test needs is here.</summary>
public sealed class AgentRemovalContext
{
public required UpgradeFolders Folders { get; init; }
/// <summary>Only the installed service removes itself, as only it upgrades itself.</summary>
public required bool CanRemove { get; init; }
public Func<string> InstallKind { get; init; } = () => AgentInstall.Current;
public required IWatchdogInstaller Watchdog { get; init; }
/// <summary>Copies this binary out and starts its <c>remove-self</c> verb, when no watchdog takes the removal.</summary>
public required IRemovalRunnerLauncher SelfRunner { get; init; }
public Func<DateTimeOffset> UtcNow { get; init; } = () => DateTimeOffset.UtcNow;
public TimeSpan PickupDeadline { get; init; } = WatchdogRemovalFiles.PickupDeadline;
}
/// <summary>
/// Hands a removal over: writes <c>removal.json</c> and makes sure the watchdog
/// runs to take it. A device with no watchdog that can (never had one, its binary
/// gone, its service disabled), or one that has not taken it within the pickup
/// deadline, runs the same runner from a copy of the agent's own binary.
/// Nothing here needs NATS: a removal found through a refused credential renewal
/// comes when the connection may be down for good.
/// </summary>
public sealed class AgentRemoval(AgentRemovalContext context, ILogger logger)
{
private UpgradeFolders Folders => context.Folders;
/// <summary>A removal waits for or has a runner.</summary>
public bool UnderWay => Read() is { Phase: not WatchdogRemovalPhases.Failed };
/// <summary>
/// Why this removal cannot start now (<see cref="DeviceRemovalReasons"/>), or
/// null. <paramref name="strict"/> also counts an unreported upgrade outcome as
/// busy: fine for a command, which means a connection that will report it, but
/// never for a renewal that was refused, which may have none.
/// </summary>
public string? Refusal(bool strict)
{
if (!context.CanRemove)
{
return DeviceRemovalReasons.Unsupported;
}
return (strict ? Folders.UpgradePending : Folders.UpgradeUnderWay) ? DeviceRemovalReasons.Busy : null;
}
/// <summary>Writes the removal down and hands it over; null when <c>removal.json</c> now carries it, otherwise why it could not be written.</summary>
public async Task<string?> HandOverAsync(Guid removalId, CancellationToken cancellationToken)
{
var removal = new WatchdogRemoval(removalId, context.InstallKind(), WatchdogRemovalPhases.Requested, 0, null, context.UtcNow());
try
{
WatchdogFileStore.Write(Folders.Removal, removal, WatchdogJsonContext.Default.WatchdogRemoval);
}
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
{
// Nothing was handed over, and nothing can carry the reason; the server asks again.
logger.LogError(exception, "Removal {RemovalId} could not be written to {Path}", removalId, Folders.Removal);
return $"The removal could not be written ({exception.Message}).";
}
logger.LogWarning("Removal {RemovalId}: this device was removed from the server; the agent uninstalls itself", removalId);
var problem = await context.Watchdog.EnsureRunningAsync(Folders.InstallDirectory, cancellationToken);
if (problem is not null)
{
logger.LogWarning("Removal {RemovalId}: no watchdog can take it ({Problem}); the agent starts the runner itself", removalId, problem);
StartSelfRunner(removal);
}
return null;
}
/// <summary>The fallback for a removal the watchdog has not taken in time; true when it acted.</summary>
public bool CheckPickup()
{
if (Read() is not { Phase: WatchdogRemovalPhases.Requested } removal || context.UtcNow() - removal.RequestedUtc < context.PickupDeadline)
{
return false;
}
logger.LogWarning("Removal {RemovalId}: the watchdog has not taken it; the agent starts the runner itself", removal.RemovalId);
StartSelfRunner(removal);
return true;
}
/// <summary>A runner that does not start leaves the reason in <c>removal.json</c>: the reporter tells the server, and the agent carries on.</summary>
private void StartSelfRunner(WatchdogRemoval removal)
{
try
{
var (directory, problem) = context.SelfRunner.Prepare(removal.RemovalId);
if (problem is null)
{
WatchdogFileStore.Write(Folders.Removal,
removal with { Phase = WatchdogRemovalPhases.Uninstalling, RunnerDirectory = directory },
WatchdogJsonContext.Default.WatchdogRemoval);
problem = context.SelfRunner.Start(removal.RemovalId, directory!);
}
if (problem is not null)
{
var why = $"The removal runner did not start: {problem}.";
logger.LogError("Removal {RemovalId}: {Why}", removal.RemovalId, why);
WatchdogFileStore.Write(Folders.Removal,
removal with { Phase = WatchdogRemovalPhases.Failed, LastError = why },
WatchdogJsonContext.Default.WatchdogRemoval);
}
}
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
{
logger.LogError(exception, "Removal {RemovalId}: {Path} could not be written", removal.RemovalId, Folders.Removal);
}
}
private WatchdogRemoval? Read() => WatchdogFileStore.Read(Folders.Removal, WatchdogJsonContext.Default.WatchdogRemoval);
}
/// <summary>
/// Handles <see cref="MessageTypes.CommandAgentRemove"/>. Idempotent by
/// <c>removal.json</c>, and deliberately not by removal id: a refused command is
/// asked again under the same id. The command may be old (it waits in its stream),
/// so it acts only on the enrollment this agent holds.
/// </summary>
public sealed class AgentRemoveCommandHandler(
AgentRemoval removal, Func<Guid?> currentEnrollment, IRemovalEvents events, string clientId, ILogger logger)
{
public async Task HandleAsync(AgentRemoveCommandPayload? command, CancellationToken cancellationToken)
{
if (command is null || command.RemovalId == Guid.Empty)
{
logger.LogWarning("Remove command without a removal id; ignored");
return;
}
if (currentEnrollment() != command.EnrollmentId)
{
logger.LogWarning("Removal {RemovalId} is for enrollment {EnrollmentId}, which this agent no longer holds; ignored", command.RemovalId, command.EnrollmentId);
await PublishAsync(command.RemovalId, DeviceRemovalStates.Failed, DeviceRemovalReasons.StaleEnrollment,
"The command is for an enrollment this agent no longer holds; nothing was changed.", cancellationToken);
return;
}
if (removal.UnderWay)
{
logger.LogInformation("Removal {RemovalId} asked again while one is under way", command.RemovalId);
await PublishAsync(command.RemovalId, DeviceRemovalStates.Uninstalling, null, null, cancellationToken);
return;
}
if (removal.Refusal(strict: true) is { } refusal)
{
logger.LogWarning("Removal {RemovalId} refused: {Reason}", command.RemovalId, refusal);
await PublishAsync(command.RemovalId, DeviceRemovalStates.Failed, refusal, refusal == DeviceRemovalReasons.Busy
? "An agent upgrade is in progress on this device."
: "The agent runs in the foreground or a development host; only the installed service removes itself.", cancellationToken);
return;
}
await PublishAsync(command.RemovalId, DeviceRemovalStates.Uninstalling, null, null, cancellationToken);
if (await removal.HandOverAsync(command.RemovalId, cancellationToken) is { } problem)
{
await PublishAsync(command.RemovalId, DeviceRemovalStates.Failed, DeviceRemovalReasons.Uninstall, problem, cancellationToken);
}
}
private async Task PublishAsync(Guid removalId, string state, string? reason, string? detail, CancellationToken cancellationToken)
{
try
{
await events.PublishAsync(new AgentRemovalEventPayload(removalId, clientId, state, reason, detail, DateTimeOffset.UtcNow), cancellationToken);
}
catch (Exception exception) when (exception is not OperationCanceledException || !cancellationToken.IsCancellationRequested)
{
// The removal goes on regardless; the server asks again if it heard nothing.
logger.LogWarning(exception, "Removal {RemovalId}: {State} could not be published", removalId, state);
}
}
}
/// <summary>
/// Reports a removal that failed once the agent is connected (the runner started
/// the agent again and left the reason in <c>removal.json</c>), then deletes the
/// file: the server asks again, and the device carries on meanwhile.
/// </summary>
public sealed class RemovalOutcomeReporter(UpgradeFolders folders, IRemovalEvents events, Func<bool> connected, string clientId, ILogger logger)
{
public static readonly TimeSpan CheckInterval = TimeSpan.FromSeconds(5);
public async Task RunAsync(CancellationToken cancellationToken)
{
while (!cancellationToken.IsCancellationRequested)
{
if (connected())
{
await ReportPendingAsync(cancellationToken);
}
try
{
await Task.Delay(CheckInterval, cancellationToken);
}
catch (OperationCanceledException)
{
break;
}
}
}
/// <summary>Publishes and deletes a failed removal; true when one was reported.</summary>
public async Task<bool> ReportPendingAsync(CancellationToken cancellationToken)
{
if (WatchdogFileStore.Read(folders.Removal, WatchdogJsonContext.Default.WatchdogRemoval) is not { Phase: WatchdogRemovalPhases.Failed } removal)
{
return false;
}
// An MSI that uninstalled leaves no agent to say anything else: a failure
// on an MSI install is always Windows Installer's.
var reason = removal.InstallKind == AgentInstallKinds.Msi ? DeviceRemovalReasons.Msi : DeviceRemovalReasons.Uninstall;
try
{
await events.PublishAsync(
new AgentRemovalEventPayload(removal.RemovalId, clientId, DeviceRemovalStates.Failed, reason, removal.LastError, DateTimeOffset.UtcNow),
cancellationToken);
}
catch (Exception exception) when (exception is not OperationCanceledException || !cancellationToken.IsCancellationRequested)
{
// Kept for the next check.
logger.LogWarning(exception, "Removal {RemovalId}: its failure could not be published; retrying", removal.RemovalId);
return false;
}
logger.LogWarning("Removal {RemovalId} failed and was reported: {Reason} {Detail}", removal.RemovalId, reason, removal.LastError);
WatchdogFileStore.TryDelete(folders.Removal);
return true;
}
}
/// <summary>Watches for a removal the watchdog does not take; independent of any connection, like the renewal that may have started it.</summary>
public sealed class RemovalPickupService(AgentRemoval removal, ILogger<RemovalPickupService> logger) : BackgroundService
{
private static readonly TimeSpan CheckInterval = TimeSpan.FromSeconds(5);
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
using var timer = new PeriodicTimer(CheckInterval);
try
{
while (await timer.WaitForNextTickAsync(stoppingToken))
{
try
{
removal.CheckPickup();
}
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
{
logger.LogWarning(exception, "The removal's hand-over could not be checked");
}
}
}
catch (OperationCanceledException)
{
// Shutdown.
}
}
}
+10 -1
View File
@@ -161,7 +161,16 @@ public sealed class AgentUpgradeHandler(AgentUpgradeContext context, IUpgradeEve
return;
}
if (Folders.UpgradePending || !context.Gate.TryEnter())
// This version renewed its credential to a short-lived one; the target could not.
if (AgentVersions.Compare(command.Version, DeviceRemovalDefaults.MinimumAgentVersion) < 0
&& AgentVersions.Compare(context.CurrentVersion, DeviceRemovalDefaults.MinimumAgentVersion) >= 0)
{
await FailAsync(command, AgentUpgradeReasons.Unsupported,
$"{command.Version} cannot renew its credential (that takes {DeviceRemovalDefaults.MinimumAgentVersion}) and would lose the server within a day.", cancellationToken);
return;
}
if (Folders.UpgradePending || File.Exists(Folders.Removal) || !context.Gate.TryEnter())
{
await FailAsync(command, AgentUpgradeReasons.Busy, "Another upgrade is under way on this device.", cancellationToken);
return;
+7
View File
@@ -6,6 +6,8 @@ namespace Agent.Upgrade;
/// <summary>Where the agent and the watchdog meet (<c>&lt;data&gt;/upgrade/</c>) and where the agent is installed.</summary>
public sealed class UpgradeFolders(string dataDirectory, string installDirectory)
{
public string DataDirectory { get; } = dataDirectory;
/// <summary>The installed layout: the data folder and the running binary's own folder.</summary>
public static UpgradeFolders Installed { get; } = new(Branding.AgentProgramDataDirectory, AppContext.BaseDirectory);
@@ -35,6 +37,11 @@ public sealed class UpgradeFolders(string dataDirectory, string installDirectory
public string Outcome => Path.Combine(Root, WatchdogFiles.Outcome);
public string Removal => Path.Combine(Root, WatchdogFiles.Removal);
/// <summary>A hand-over, a swap or an MSI run: what runs now may still be rolled back.</summary>
public bool UpgradeUnderWay => File.Exists(Handover) || File.Exists(Journal) || File.Exists(MsiJournal);
/// <summary>A hand-over, a swap, an MSI run or an outcome not yet reported: another upgrade must wait.</summary>
public bool UpgradePending => File.Exists(Handover) || File.Exists(Journal) || File.Exists(MsiJournal) || File.Exists(Outcome);
+19 -3
View File
@@ -98,7 +98,11 @@ public static class DeviceEndpoints
await groups.GetGroupsForDeviceAsync(clientId, cancellationToken)));
}
/// <summary>The device's phase trail (enrolled, renamed, OS changed), newest first.</summary>
/// <summary>
/// The device's phase trail (enrolled, renamed, OS changed), newest first. A
/// removed device has no row but keeps its trail, within the scope its removal
/// recorded.
/// </summary>
private static async Task<IResult> GetHistoryAsync(
string clientId, int? limit, AppDbContext db, DeviceScopeService scope,
HttpContext context, CancellationToken cancellationToken)
@@ -108,8 +112,10 @@ public static class DeviceEndpoints
return Results.BadRequest();
}
if (!await scope.IsAllowedAsync(context.User, clientId, cancellationToken)
|| !await db.Devices.AnyAsync(d => d.ClientId == clientId, cancellationToken))
var reachable = await db.Devices.AnyAsync(d => d.ClientId == clientId, cancellationToken)
? await scope.IsAllowedAsync(context.User, clientId, cancellationToken)
: await IsRemovedWithinScopeAsync(clientId, db, scope, context, cancellationToken);
if (!reachable)
{
return Results.NotFound();
}
@@ -125,6 +131,16 @@ public static class DeviceEndpoints
return Results.Ok(events);
}
private static async Task<bool> IsRemovedWithinScopeAsync(
string clientId, AppDbContext db, DeviceScopeService scope, HttpContext context, CancellationToken cancellationToken)
{
var scopeGroupIds = await scope.GetScopeGroupIdsAsync(context.User, cancellationToken);
var removals = await db.DeviceRemovals.AsNoTracking()
.Where(r => r.ClientId == clientId)
.ToListAsync(cancellationToken);
return removals.Any(r => Removals.DeviceRemovalService.IsVisibleTo(r, scopeGroupIds));
}
private static async Task<IResult> RefreshInventoryAsync(
string clientId,
AppDbContext db,
+20
View File
@@ -23,6 +23,13 @@ public static class DeviceHistory
/// <summary>An upgrade failed and the agent stayed where it was (detail carries the reason).</summary>
public const string AgentUpgradeFailed = "agent-upgrade-failed";
/// <summary>An operator removed the device; its agent uninstalls now or when it returns.</summary>
public const string RemovalRequested = "device-removal-requested";
/// <summary>The agent has gone, or the device enrolled again and the removal closed.</summary>
public const string Removed = "device-removed";
/// <summary>An operator stopped trying: the device stays blocked, its agent stays installed.</summary>
public const string RemovalStopped = "device-removal-stopped";
/// <summary>
/// Applies a reported hostname and (when <paramref name="reportsOs"/>) OS
/// to the device, recording a rename / OS change against the previous
@@ -77,6 +84,19 @@ public static class DeviceHistory
public static void RecordAgentUpgrade(AppDbContext db, string clientId, string action, string detail)
=> Record(db, clientId, action, detail, DateTimeOffset.UtcNow);
/// <summary>Records a removal request, with who asked and their note.</summary>
public static void RecordRemovalRequested(AppDbContext db, string clientId, string? requestedBy, string? note)
=> Record(db, clientId, RemovalRequested,
$"By {requestedBy ?? "unknown"}{(string.IsNullOrWhiteSpace(note) ? string.Empty : $" · {note}")}", DateTimeOffset.UtcNow);
/// <summary>Records the end of a removal.</summary>
public static void RecordRemoved(AppDbContext db, string clientId, string detail)
=> Record(db, clientId, Removed, detail, DateTimeOffset.UtcNow);
/// <summary>Records an operator stopping a removal.</summary>
public static void RecordRemovalStopped(AppDbContext db, string clientId, string? stoppedBy)
=> Record(db, clientId, RemovalStopped, $"By {stoppedBy ?? "unknown"}", DateTimeOffset.UtcNow);
/// <summary>"Windows PE 10.0.26100"; just the caption when the version is unknown.</summary>
public static string Describe(string? caption, string? version)
{
+8
View File
@@ -46,6 +46,14 @@ public sealed class DeviceScopeService(
return allowed = clientIds;
}
/// <summary>
/// The groups the caller's roles are limited to; null when unrestricted. A
/// removed device is in no group any more, so its removal is matched on these.
/// </summary>
public async Task<IReadOnlyList<Guid>?> GetScopeGroupIdsAsync(
ClaimsPrincipal user, CancellationToken cancellationToken)
=> (await permissions.ResolveAccessAsync(user, cancellationToken)).ScopeGroupIds;
/// <summary>True when the caller may act on the device.</summary>
public async Task<bool> IsAllowedAsync(
ClaimsPrincipal user, string clientId, CancellationToken cancellationToken)
+2
View File
@@ -14,6 +14,7 @@ public static class Permissions
public const string DevicesView = "devices.view";
public const string DevicesManage = "devices.manage";
public const string DevicesRemove = "devices.remove";
public const string GroupsView = "groups.view";
public const string GroupsManage = "groups.manage";
public const string ScriptsView = "scripts.view";
@@ -38,6 +39,7 @@ public static class Permissions
[
new(DevicesView, "View devices", "See the device list, details and inventory."),
new(DevicesManage, "Manage devices", "Refresh inventory and edit device tags."),
new(DevicesRemove, "Remove devices", "Remove devices from the system and uninstall their agents."),
]),
new("Groups",
[
+31 -1
View File
@@ -33,6 +33,7 @@ public class AppDbContext(DbContextOptions<AppDbContext> options) : DbContext(op
public DbSet<SecretsProviderEntity> SecretsProviders => Set<SecretsProviderEntity>();
public DbSet<AgentPackageEntity> AgentPackages => Set<AgentPackageEntity>();
public DbSet<AgentUpgradeEntity> AgentUpgrades => Set<AgentUpgradeEntity>();
public DbSet<DeviceRemovalEntity> DeviceRemovals => Set<DeviceRemovalEntity>();
/// <summary>
/// SQLite cannot ORDER BY / compare DateTimeOffset natively, so it stores
@@ -66,7 +67,7 @@ public class AppDbContext(DbContextOptions<AppDbContext> options) : DbContext(op
}
/// <summary>
/// Every write to a trigger or an agent upgrade gets a fresh stamp; the
/// Every write to a trigger, an agent upgrade or a removal gets a fresh stamp; the
/// update is conditioned on the stamp it was read with, so a concurrent
/// write surfaces as DbUpdateConcurrencyException instead of silently winning.
/// </summary>
@@ -87,6 +88,14 @@ public class AppDbContext(DbContextOptions<AppDbContext> options) : DbContext(op
entry.Entity.ConcurrencyStamp = Guid.NewGuid();
}
}
foreach (var entry in ChangeTracker.Entries<DeviceRemovalEntity>())
{
if (entry.State is EntityState.Added or EntityState.Modified)
{
entry.Entity.ConcurrencyStamp = Guid.NewGuid();
}
}
}
protected override void OnModelCreating(ModelBuilder modelBuilder)
@@ -220,7 +229,12 @@ public class AppDbContext(DbContextOptions<AppDbContext> options) : DbContext(op
entity.HasIndex(e => e.CreatedUtc);
entity.HasIndex(e => new { e.ScriptId, e.RunId });
entity.HasIndex(e => e.RunId);
// The dispatcher's every-tick query: queued jobs whose time has come.
entity.HasIndex(e => new { e.State, e.NotBeforeUtc });
entity.Property(e => e.State).HasConversion<string>();
entity.Property(e => e.ErrorType).HasMaxLength(300);
entity.Property(e => e.ErrorId).HasMaxLength(300);
entity.Property(e => e.ErrorMessage).HasMaxLength(2000);
});
modelBuilder.Entity<RunEntity>(entity =>
@@ -287,6 +301,7 @@ public class AppDbContext(DbContextOptions<AppDbContext> options) : DbContext(op
// The schedule sweep queries due scheduled triggers by next-run.
entity.HasIndex(e => e.NextRunUtc);
entity.Property(e => e.TriggerType).HasConversion<string>();
entity.Property(e => e.StaggerBatchSize).HasDefaultValue(1);
entity.Property(e => e.ConcurrencyStamp).IsConcurrencyToken();
});
@@ -345,5 +360,20 @@ public class AppDbContext(DbContextOptions<AppDbContext> options) : DbContext(op
entity.Property(e => e.ConcurrencyStamp).IsConcurrencyToken();
});
modelBuilder.Entity<DeviceRemovalEntity>(entity =>
{
entity.ToTable("DeviceRemovals");
entity.HasKey(e => e.Id);
entity.Property(e => e.State).HasMaxLength(16);
entity.Property(e => e.Reason).HasMaxLength(32);
entity.Property(e => e.Detail).HasMaxLength(2000);
entity.Property(e => e.RequestedBy).HasMaxLength(256);
entity.Property(e => e.Note).HasMaxLength(500);
// Enrollment, renewal and sightings look up by client id; the scheduler sweeps by state.
entity.HasIndex(e => e.ClientId);
entity.HasIndex(e => e.State);
entity.HasIndex(e => e.RequestedUtc);
entity.Property(e => e.ConcurrencyStamp).IsConcurrencyToken();
});
}
}
+88
View File
@@ -289,6 +289,12 @@ public sealed class ScriptTriggerEntity
/// <summary>Execution (job RunId) the last firing created, for drill-in.</summary>
public Guid? LastRunId { get; set; }
/// <summary>Spreads each firing's starts over this many seconds; 0 starts them all at once.</summary>
public int StaggerSeconds { get; set; }
/// <summary>Devices started together in each batch of a stagger.</summary>
public int StaggerBatchSize { get; set; } = 1;
/// <summary>
/// Optimistic-concurrency token, renewed on every save (AppDbContext), so
/// a schedule edited while it is firing is not overwritten by the sweep.
@@ -412,6 +418,12 @@ public sealed class RunEntity
/// <summary>JSON array of RunSkip: devices targeted but given no job.</summary>
public string SkippedJson { get; set; } = "[]";
/// <summary>
/// The sealed RunSnapshot the dispatcher sends this run's jobs from
/// (arguments can be sensitive); set only while it has jobs left to send.
/// </summary>
public string? SnapshotProtected { get; set; }
}
/// <summary>A targeted device that got no job, and why (orphaned, offline).</summary>
@@ -451,6 +463,21 @@ public sealed class JobEntity
public int? ExitCode { get; set; }
public string? RequestedBy { get; set; }
public string? OutputSummary { get; set; }
/// <summary>
/// Set while the dispatcher owns sending this job: a Queued job waits until
/// this time (its stagger slot), then is sent. Null on a job sent directly.
/// </summary>
public DateTimeOffset? NotBeforeUtc { get; set; }
/// <summary>The first error's exception type, as the agent reported it; null from older agents and external engines.</summary>
public string? ErrorType { get; set; }
/// <summary>The first error's FullyQualifiedErrorId, as the agent reported it.</summary>
public string? ErrorId { get; set; }
/// <summary>The first error's message, or an external engine's first stderr line.</summary>
public string? ErrorMessage { get; set; }
}
public sealed class JobOutputEntity
@@ -561,6 +588,67 @@ public sealed class AgentUpgradeEntity
public Guid ConcurrencyStamp { get; set; }
}
/// <summary>
/// One device's removal. The device row is gone from the moment it is requested;
/// this stands in for it until the agent has gone, and stays afterwards as the
/// history (old runs still find the name here, and bootstrap the date of removal).
/// A client id has at most one open removal: one is only added as the device row
/// is deleted, and enrolling again closes it as the row comes back.
/// </summary>
public sealed class DeviceRemovalEntity
{
public Guid Id { get; set; }
public required string ClientId { get; set; }
/// <summary>The enrolled key: a credential renewal must still prove possession of it.</summary>
public required string NkeyPublic { get; set; }
/// <summary>Carried by the remove command; the agent ignores one for another enrollment.</summary>
public Guid EnrollmentId { get; set; }
public string? Hostname { get; set; }
public string? OsCaption { get; set; }
/// <summary>Running version: as removed, then from presence.</summary>
public string? AgentVersion { get; set; }
/// <summary>A DeviceRemovalStates value.</summary>
public required string State { get; set; }
/// <summary>A DeviceRemovalReasons value.</summary>
public string? Reason { get; set; }
public string? Detail { get; set; }
/// <summary>Times the command was handed to JetStream.</summary>
public int Attempts { get; set; }
public string? RequestedBy { get; set; }
public string? Note { get; set; }
public DateTimeOffset RequestedUtc { get; set; }
/// <summary>Last change of state; sightings move only <see cref="LastSeenUtc"/>.</summary>
public DateTimeOffset UpdatedUtc { get; set; }
public DateTimeOffset? SentUtc { get; set; }
public DateTimeOffset? LastSeenUtc { get; set; }
public DateTimeOffset? FinishedUtc { get; set; }
/// <summary>
/// JSON array of the groups the device was in, nesting included, when it was
/// removed: a scoped role's device scope is its groups, and it keeps seeing
/// the removals of the devices that were in them.
/// </summary>
public string? GroupIdsJson { get; set; }
/// <summary>
/// Renewed on every save (AppDbContext): the scheduler, the event consumer,
/// credential renewal and an operator's stop may touch the same row on
/// different nodes, and the later write is refused instead of silently winning.
/// </summary>
public Guid ConcurrencyStamp { get; set; }
}
public sealed class ShellSessionEntity
{
public Guid Id { get; set; }
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,67 @@
using System;
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace Backend.Data.Migrations.Postgres
{
/// <inheritdoc />
public partial class DeviceRemovals : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.CreateTable(
name: "DeviceRemovals",
columns: table => new
{
Id = table.Column<Guid>(type: "uuid", nullable: false),
ClientId = table.Column<string>(type: "text", nullable: false),
NkeyPublic = table.Column<string>(type: "text", nullable: false),
EnrollmentId = table.Column<Guid>(type: "uuid", nullable: false),
Hostname = table.Column<string>(type: "text", nullable: true),
OsCaption = table.Column<string>(type: "text", nullable: true),
AgentVersion = table.Column<string>(type: "text", nullable: true),
State = table.Column<string>(type: "character varying(16)", maxLength: 16, nullable: false),
Reason = table.Column<string>(type: "character varying(32)", maxLength: 32, nullable: true),
Detail = table.Column<string>(type: "character varying(2000)", maxLength: 2000, nullable: true),
Attempts = table.Column<int>(type: "integer", nullable: false),
RequestedBy = table.Column<string>(type: "character varying(256)", maxLength: 256, nullable: true),
Note = table.Column<string>(type: "character varying(500)", maxLength: 500, nullable: true),
RequestedUtc = table.Column<DateTimeOffset>(type: "timestamp with time zone", nullable: false),
UpdatedUtc = table.Column<DateTimeOffset>(type: "timestamp with time zone", nullable: false),
SentUtc = table.Column<DateTimeOffset>(type: "timestamp with time zone", nullable: true),
LastSeenUtc = table.Column<DateTimeOffset>(type: "timestamp with time zone", nullable: true),
FinishedUtc = table.Column<DateTimeOffset>(type: "timestamp with time zone", nullable: true),
GroupIdsJson = table.Column<string>(type: "text", nullable: true),
ConcurrencyStamp = table.Column<Guid>(type: "uuid", nullable: false)
},
constraints: table =>
{
table.PrimaryKey("PK_DeviceRemovals", x => x.Id);
});
migrationBuilder.CreateIndex(
name: "IX_DeviceRemovals_ClientId",
table: "DeviceRemovals",
column: "ClientId");
migrationBuilder.CreateIndex(
name: "IX_DeviceRemovals_RequestedUtc",
table: "DeviceRemovals",
column: "RequestedUtc");
migrationBuilder.CreateIndex(
name: "IX_DeviceRemovals_State",
table: "DeviceRemovals",
column: "State");
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropTable(
name: "DeviceRemovals");
}
}
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,103 @@
using System;
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace Backend.Data.Migrations.Postgres
{
/// <inheritdoc />
public partial class JobStagger : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.AddColumn<int>(
name: "StaggerBatchSize",
table: "ScriptTriggers",
type: "integer",
nullable: false,
defaultValue: 1);
migrationBuilder.AddColumn<int>(
name: "StaggerSeconds",
table: "ScriptTriggers",
type: "integer",
nullable: false,
defaultValue: 0);
migrationBuilder.AddColumn<string>(
name: "SnapshotProtected",
table: "Runs",
type: "text",
nullable: true);
migrationBuilder.AddColumn<string>(
name: "ErrorId",
table: "Jobs",
type: "character varying(300)",
maxLength: 300,
nullable: true);
migrationBuilder.AddColumn<string>(
name: "ErrorMessage",
table: "Jobs",
type: "character varying(2000)",
maxLength: 2000,
nullable: true);
migrationBuilder.AddColumn<string>(
name: "ErrorType",
table: "Jobs",
type: "character varying(300)",
maxLength: 300,
nullable: true);
migrationBuilder.AddColumn<DateTimeOffset>(
name: "NotBeforeUtc",
table: "Jobs",
type: "timestamp with time zone",
nullable: true);
migrationBuilder.CreateIndex(
name: "IX_Jobs_State_NotBeforeUtc",
table: "Jobs",
columns: new[] { "State", "NotBeforeUtc" });
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropIndex(
name: "IX_Jobs_State_NotBeforeUtc",
table: "Jobs");
migrationBuilder.DropColumn(
name: "StaggerBatchSize",
table: "ScriptTriggers");
migrationBuilder.DropColumn(
name: "StaggerSeconds",
table: "ScriptTriggers");
migrationBuilder.DropColumn(
name: "SnapshotProtected",
table: "Runs");
migrationBuilder.DropColumn(
name: "ErrorId",
table: "Jobs");
migrationBuilder.DropColumn(
name: "ErrorMessage",
table: "Jobs");
migrationBuilder.DropColumn(
name: "ErrorType",
table: "Jobs");
migrationBuilder.DropColumn(
name: "NotBeforeUtc",
table: "Jobs");
}
}
}
@@ -417,6 +417,89 @@ namespace Backend.Data.Migrations.Postgres
b.ToTable("DeviceInventory", (string)null);
});
modelBuilder.Entity("Backend.Data.DeviceRemovalEntity", b =>
{
b.Property<Guid>("Id")
.ValueGeneratedOnAdd()
.HasColumnType("uuid");
b.Property<string>("AgentVersion")
.HasColumnType("text");
b.Property<int>("Attempts")
.HasColumnType("integer");
b.Property<string>("ClientId")
.IsRequired()
.HasColumnType("text");
b.Property<Guid>("ConcurrencyStamp")
.IsConcurrencyToken()
.HasColumnType("uuid");
b.Property<string>("Detail")
.HasMaxLength(2000)
.HasColumnType("character varying(2000)");
b.Property<Guid>("EnrollmentId")
.HasColumnType("uuid");
b.Property<DateTimeOffset?>("FinishedUtc")
.HasColumnType("timestamp with time zone");
b.Property<string>("GroupIdsJson")
.HasColumnType("text");
b.Property<string>("Hostname")
.HasColumnType("text");
b.Property<DateTimeOffset?>("LastSeenUtc")
.HasColumnType("timestamp with time zone");
b.Property<string>("NkeyPublic")
.IsRequired()
.HasColumnType("text");
b.Property<string>("Note")
.HasMaxLength(500)
.HasColumnType("character varying(500)");
b.Property<string>("OsCaption")
.HasColumnType("text");
b.Property<string>("Reason")
.HasMaxLength(32)
.HasColumnType("character varying(32)");
b.Property<string>("RequestedBy")
.HasMaxLength(256)
.HasColumnType("character varying(256)");
b.Property<DateTimeOffset>("RequestedUtc")
.HasColumnType("timestamp with time zone");
b.Property<DateTimeOffset?>("SentUtc")
.HasColumnType("timestamp with time zone");
b.Property<string>("State")
.IsRequired()
.HasMaxLength(16)
.HasColumnType("character varying(16)");
b.Property<DateTimeOffset>("UpdatedUtc")
.HasColumnType("timestamp with time zone");
b.HasKey("Id");
b.HasIndex("ClientId");
b.HasIndex("RequestedUtc");
b.HasIndex("State");
b.ToTable("DeviceRemovals", (string)null);
});
modelBuilder.Entity("Backend.Data.FunctionModuleEntity", b =>
{
b.Property<Guid>("Id")
@@ -466,9 +549,24 @@ namespace Backend.Data.Migrations.Postgres
b.Property<DateTimeOffset>("CreatedUtc")
.HasColumnType("timestamp with time zone");
b.Property<string>("ErrorId")
.HasMaxLength(300)
.HasColumnType("character varying(300)");
b.Property<string>("ErrorMessage")
.HasMaxLength(2000)
.HasColumnType("character varying(2000)");
b.Property<string>("ErrorType")
.HasMaxLength(300)
.HasColumnType("character varying(300)");
b.Property<int?>("ExitCode")
.HasColumnType("integer");
b.Property<DateTimeOffset?>("NotBeforeUtc")
.HasColumnType("timestamp with time zone");
b.Property<string>("OutputSummary")
.HasColumnType("text");
@@ -502,6 +600,8 @@ namespace Backend.Data.Migrations.Postgres
b.HasIndex("ScriptId", "RunId");
b.HasIndex("State", "NotBeforeUtc");
b.ToTable("Jobs", (string)null);
});
@@ -688,6 +788,9 @@ namespace Backend.Data.Migrations.Postgres
.IsRequired()
.HasColumnType("text");
b.Property<string>("SnapshotProtected")
.HasColumnType("text");
b.Property<string>("Source")
.IsRequired()
.HasMaxLength(32)
@@ -819,6 +922,14 @@ namespace Backend.Data.Migrations.Postgres
b.Property<Guid>("ScriptId")
.HasColumnType("uuid");
b.Property<int>("StaggerBatchSize")
.ValueGeneratedOnAdd()
.HasColumnType("integer")
.HasDefaultValue(1);
b.Property<int>("StaggerSeconds")
.HasColumnType("integer");
b.Property<string>("TimeZoneId")
.IsRequired()
.HasColumnType("text");
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,67 @@
using System;
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace Backend.Data.Migrations.SqlServer
{
/// <inheritdoc />
public partial class DeviceRemovals : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.CreateTable(
name: "DeviceRemovals",
columns: table => new
{
Id = table.Column<Guid>(type: "uniqueidentifier", nullable: false),
ClientId = table.Column<string>(type: "nvarchar(450)", nullable: false),
NkeyPublic = table.Column<string>(type: "nvarchar(max)", nullable: false),
EnrollmentId = table.Column<Guid>(type: "uniqueidentifier", nullable: false),
Hostname = table.Column<string>(type: "nvarchar(max)", nullable: true),
OsCaption = table.Column<string>(type: "nvarchar(max)", nullable: true),
AgentVersion = table.Column<string>(type: "nvarchar(max)", nullable: true),
State = table.Column<string>(type: "nvarchar(16)", maxLength: 16, nullable: false),
Reason = table.Column<string>(type: "nvarchar(32)", maxLength: 32, nullable: true),
Detail = table.Column<string>(type: "nvarchar(2000)", maxLength: 2000, nullable: true),
Attempts = table.Column<int>(type: "int", nullable: false),
RequestedBy = table.Column<string>(type: "nvarchar(256)", maxLength: 256, nullable: true),
Note = table.Column<string>(type: "nvarchar(500)", maxLength: 500, nullable: true),
RequestedUtc = table.Column<DateTimeOffset>(type: "datetimeoffset", nullable: false),
UpdatedUtc = table.Column<DateTimeOffset>(type: "datetimeoffset", nullable: false),
SentUtc = table.Column<DateTimeOffset>(type: "datetimeoffset", nullable: true),
LastSeenUtc = table.Column<DateTimeOffset>(type: "datetimeoffset", nullable: true),
FinishedUtc = table.Column<DateTimeOffset>(type: "datetimeoffset", nullable: true),
GroupIdsJson = table.Column<string>(type: "nvarchar(max)", nullable: true),
ConcurrencyStamp = table.Column<Guid>(type: "uniqueidentifier", nullable: false)
},
constraints: table =>
{
table.PrimaryKey("PK_DeviceRemovals", x => x.Id);
});
migrationBuilder.CreateIndex(
name: "IX_DeviceRemovals_ClientId",
table: "DeviceRemovals",
column: "ClientId");
migrationBuilder.CreateIndex(
name: "IX_DeviceRemovals_RequestedUtc",
table: "DeviceRemovals",
column: "RequestedUtc");
migrationBuilder.CreateIndex(
name: "IX_DeviceRemovals_State",
table: "DeviceRemovals",
column: "State");
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropTable(
name: "DeviceRemovals");
}
}
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,119 @@
using System;
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace Backend.Data.Migrations.SqlServer
{
/// <inheritdoc />
public partial class JobStagger : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.AddColumn<int>(
name: "StaggerBatchSize",
table: "ScriptTriggers",
type: "int",
nullable: false,
defaultValue: 1);
migrationBuilder.AddColumn<int>(
name: "StaggerSeconds",
table: "ScriptTriggers",
type: "int",
nullable: false,
defaultValue: 0);
migrationBuilder.AddColumn<string>(
name: "SnapshotProtected",
table: "Runs",
type: "nvarchar(max)",
nullable: true);
migrationBuilder.AlterColumn<string>(
name: "State",
table: "Jobs",
type: "nvarchar(450)",
nullable: false,
oldClrType: typeof(string),
oldType: "nvarchar(max)");
migrationBuilder.AddColumn<string>(
name: "ErrorId",
table: "Jobs",
type: "nvarchar(300)",
maxLength: 300,
nullable: true);
migrationBuilder.AddColumn<string>(
name: "ErrorMessage",
table: "Jobs",
type: "nvarchar(2000)",
maxLength: 2000,
nullable: true);
migrationBuilder.AddColumn<string>(
name: "ErrorType",
table: "Jobs",
type: "nvarchar(300)",
maxLength: 300,
nullable: true);
migrationBuilder.AddColumn<DateTimeOffset>(
name: "NotBeforeUtc",
table: "Jobs",
type: "datetimeoffset",
nullable: true);
migrationBuilder.CreateIndex(
name: "IX_Jobs_State_NotBeforeUtc",
table: "Jobs",
columns: new[] { "State", "NotBeforeUtc" });
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropIndex(
name: "IX_Jobs_State_NotBeforeUtc",
table: "Jobs");
migrationBuilder.DropColumn(
name: "StaggerBatchSize",
table: "ScriptTriggers");
migrationBuilder.DropColumn(
name: "StaggerSeconds",
table: "ScriptTriggers");
migrationBuilder.DropColumn(
name: "SnapshotProtected",
table: "Runs");
migrationBuilder.DropColumn(
name: "ErrorId",
table: "Jobs");
migrationBuilder.DropColumn(
name: "ErrorMessage",
table: "Jobs");
migrationBuilder.DropColumn(
name: "ErrorType",
table: "Jobs");
migrationBuilder.DropColumn(
name: "NotBeforeUtc",
table: "Jobs");
migrationBuilder.AlterColumn<string>(
name: "State",
table: "Jobs",
type: "nvarchar(max)",
nullable: false,
oldClrType: typeof(string),
oldType: "nvarchar(450)");
}
}
}
@@ -417,6 +417,89 @@ namespace Backend.Data.Migrations.SqlServer
b.ToTable("DeviceInventory", (string)null);
});
modelBuilder.Entity("Backend.Data.DeviceRemovalEntity", b =>
{
b.Property<Guid>("Id")
.ValueGeneratedOnAdd()
.HasColumnType("uniqueidentifier");
b.Property<string>("AgentVersion")
.HasColumnType("nvarchar(max)");
b.Property<int>("Attempts")
.HasColumnType("int");
b.Property<string>("ClientId")
.IsRequired()
.HasColumnType("nvarchar(450)");
b.Property<Guid>("ConcurrencyStamp")
.IsConcurrencyToken()
.HasColumnType("uniqueidentifier");
b.Property<string>("Detail")
.HasMaxLength(2000)
.HasColumnType("nvarchar(2000)");
b.Property<Guid>("EnrollmentId")
.HasColumnType("uniqueidentifier");
b.Property<DateTimeOffset?>("FinishedUtc")
.HasColumnType("datetimeoffset");
b.Property<string>("GroupIdsJson")
.HasColumnType("nvarchar(max)");
b.Property<string>("Hostname")
.HasColumnType("nvarchar(max)");
b.Property<DateTimeOffset?>("LastSeenUtc")
.HasColumnType("datetimeoffset");
b.Property<string>("NkeyPublic")
.IsRequired()
.HasColumnType("nvarchar(max)");
b.Property<string>("Note")
.HasMaxLength(500)
.HasColumnType("nvarchar(500)");
b.Property<string>("OsCaption")
.HasColumnType("nvarchar(max)");
b.Property<string>("Reason")
.HasMaxLength(32)
.HasColumnType("nvarchar(32)");
b.Property<string>("RequestedBy")
.HasMaxLength(256)
.HasColumnType("nvarchar(256)");
b.Property<DateTimeOffset>("RequestedUtc")
.HasColumnType("datetimeoffset");
b.Property<DateTimeOffset?>("SentUtc")
.HasColumnType("datetimeoffset");
b.Property<string>("State")
.IsRequired()
.HasMaxLength(16)
.HasColumnType("nvarchar(16)");
b.Property<DateTimeOffset>("UpdatedUtc")
.HasColumnType("datetimeoffset");
b.HasKey("Id");
b.HasIndex("ClientId");
b.HasIndex("RequestedUtc");
b.HasIndex("State");
b.ToTable("DeviceRemovals", (string)null);
});
modelBuilder.Entity("Backend.Data.FunctionModuleEntity", b =>
{
b.Property<Guid>("Id")
@@ -466,9 +549,24 @@ namespace Backend.Data.Migrations.SqlServer
b.Property<DateTimeOffset>("CreatedUtc")
.HasColumnType("datetimeoffset");
b.Property<string>("ErrorId")
.HasMaxLength(300)
.HasColumnType("nvarchar(300)");
b.Property<string>("ErrorMessage")
.HasMaxLength(2000)
.HasColumnType("nvarchar(2000)");
b.Property<string>("ErrorType")
.HasMaxLength(300)
.HasColumnType("nvarchar(300)");
b.Property<int?>("ExitCode")
.HasColumnType("int");
b.Property<DateTimeOffset?>("NotBeforeUtc")
.HasColumnType("datetimeoffset");
b.Property<string>("OutputSummary")
.HasColumnType("nvarchar(max)");
@@ -490,7 +588,7 @@ namespace Backend.Data.Migrations.SqlServer
b.Property<string>("State")
.IsRequired()
.HasColumnType("nvarchar(max)");
.HasColumnType("nvarchar(450)");
b.HasKey("JobId");
@@ -502,6 +600,8 @@ namespace Backend.Data.Migrations.SqlServer
b.HasIndex("ScriptId", "RunId");
b.HasIndex("State", "NotBeforeUtc");
b.ToTable("Jobs", (string)null);
});
@@ -688,6 +788,9 @@ namespace Backend.Data.Migrations.SqlServer
.IsRequired()
.HasColumnType("nvarchar(max)");
b.Property<string>("SnapshotProtected")
.HasColumnType("nvarchar(max)");
b.Property<string>("Source")
.IsRequired()
.HasMaxLength(32)
@@ -819,6 +922,14 @@ namespace Backend.Data.Migrations.SqlServer
b.Property<Guid>("ScriptId")
.HasColumnType("uniqueidentifier");
b.Property<int>("StaggerBatchSize")
.ValueGeneratedOnAdd()
.HasColumnType("int")
.HasDefaultValue(1);
b.Property<int>("StaggerSeconds")
.HasColumnType("int");
b.Property<string>("TimeZoneId")
.IsRequired()
.HasColumnType("nvarchar(max)");
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,67 @@
using System;
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace Backend.Data.Migrations.Sqlite
{
/// <inheritdoc />
public partial class DeviceRemovals : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.CreateTable(
name: "DeviceRemovals",
columns: table => new
{
Id = table.Column<Guid>(type: "TEXT", nullable: false),
ClientId = table.Column<string>(type: "TEXT", nullable: false),
NkeyPublic = table.Column<string>(type: "TEXT", nullable: false),
EnrollmentId = table.Column<Guid>(type: "TEXT", nullable: false),
Hostname = table.Column<string>(type: "TEXT", nullable: true),
OsCaption = table.Column<string>(type: "TEXT", nullable: true),
AgentVersion = table.Column<string>(type: "TEXT", nullable: true),
State = table.Column<string>(type: "TEXT", maxLength: 16, nullable: false),
Reason = table.Column<string>(type: "TEXT", maxLength: 32, nullable: true),
Detail = table.Column<string>(type: "TEXT", maxLength: 2000, nullable: true),
Attempts = table.Column<int>(type: "INTEGER", nullable: false),
RequestedBy = table.Column<string>(type: "TEXT", maxLength: 256, nullable: true),
Note = table.Column<string>(type: "TEXT", maxLength: 500, nullable: true),
RequestedUtc = table.Column<long>(type: "INTEGER", nullable: false),
UpdatedUtc = table.Column<long>(type: "INTEGER", nullable: false),
SentUtc = table.Column<long>(type: "INTEGER", nullable: true),
LastSeenUtc = table.Column<long>(type: "INTEGER", nullable: true),
FinishedUtc = table.Column<long>(type: "INTEGER", nullable: true),
GroupIdsJson = table.Column<string>(type: "TEXT", nullable: true),
ConcurrencyStamp = table.Column<Guid>(type: "TEXT", nullable: false)
},
constraints: table =>
{
table.PrimaryKey("PK_DeviceRemovals", x => x.Id);
});
migrationBuilder.CreateIndex(
name: "IX_DeviceRemovals_ClientId",
table: "DeviceRemovals",
column: "ClientId");
migrationBuilder.CreateIndex(
name: "IX_DeviceRemovals_RequestedUtc",
table: "DeviceRemovals",
column: "RequestedUtc");
migrationBuilder.CreateIndex(
name: "IX_DeviceRemovals_State",
table: "DeviceRemovals",
column: "State");
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropTable(
name: "DeviceRemovals");
}
}
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,102 @@
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace Backend.Data.Migrations.Sqlite
{
/// <inheritdoc />
public partial class JobStagger : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.AddColumn<int>(
name: "StaggerBatchSize",
table: "ScriptTriggers",
type: "INTEGER",
nullable: false,
defaultValue: 1);
migrationBuilder.AddColumn<int>(
name: "StaggerSeconds",
table: "ScriptTriggers",
type: "INTEGER",
nullable: false,
defaultValue: 0);
migrationBuilder.AddColumn<string>(
name: "SnapshotProtected",
table: "Runs",
type: "TEXT",
nullable: true);
migrationBuilder.AddColumn<string>(
name: "ErrorId",
table: "Jobs",
type: "TEXT",
maxLength: 300,
nullable: true);
migrationBuilder.AddColumn<string>(
name: "ErrorMessage",
table: "Jobs",
type: "TEXT",
maxLength: 2000,
nullable: true);
migrationBuilder.AddColumn<string>(
name: "ErrorType",
table: "Jobs",
type: "TEXT",
maxLength: 300,
nullable: true);
migrationBuilder.AddColumn<long>(
name: "NotBeforeUtc",
table: "Jobs",
type: "INTEGER",
nullable: true);
migrationBuilder.CreateIndex(
name: "IX_Jobs_State_NotBeforeUtc",
table: "Jobs",
columns: new[] { "State", "NotBeforeUtc" });
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropIndex(
name: "IX_Jobs_State_NotBeforeUtc",
table: "Jobs");
migrationBuilder.DropColumn(
name: "StaggerBatchSize",
table: "ScriptTriggers");
migrationBuilder.DropColumn(
name: "StaggerSeconds",
table: "ScriptTriggers");
migrationBuilder.DropColumn(
name: "SnapshotProtected",
table: "Runs");
migrationBuilder.DropColumn(
name: "ErrorId",
table: "Jobs");
migrationBuilder.DropColumn(
name: "ErrorMessage",
table: "Jobs");
migrationBuilder.DropColumn(
name: "ErrorType",
table: "Jobs");
migrationBuilder.DropColumn(
name: "NotBeforeUtc",
table: "Jobs");
}
}
}
@@ -408,6 +408,89 @@ namespace Backend.Data.Migrations.Sqlite
b.ToTable("DeviceInventory", (string)null);
});
modelBuilder.Entity("Backend.Data.DeviceRemovalEntity", b =>
{
b.Property<Guid>("Id")
.ValueGeneratedOnAdd()
.HasColumnType("TEXT");
b.Property<string>("AgentVersion")
.HasColumnType("TEXT");
b.Property<int>("Attempts")
.HasColumnType("INTEGER");
b.Property<string>("ClientId")
.IsRequired()
.HasColumnType("TEXT");
b.Property<Guid>("ConcurrencyStamp")
.IsConcurrencyToken()
.HasColumnType("TEXT");
b.Property<string>("Detail")
.HasMaxLength(2000)
.HasColumnType("TEXT");
b.Property<Guid>("EnrollmentId")
.HasColumnType("TEXT");
b.Property<long?>("FinishedUtc")
.HasColumnType("INTEGER");
b.Property<string>("GroupIdsJson")
.HasColumnType("TEXT");
b.Property<string>("Hostname")
.HasColumnType("TEXT");
b.Property<long?>("LastSeenUtc")
.HasColumnType("INTEGER");
b.Property<string>("NkeyPublic")
.IsRequired()
.HasColumnType("TEXT");
b.Property<string>("Note")
.HasMaxLength(500)
.HasColumnType("TEXT");
b.Property<string>("OsCaption")
.HasColumnType("TEXT");
b.Property<string>("Reason")
.HasMaxLength(32)
.HasColumnType("TEXT");
b.Property<string>("RequestedBy")
.HasMaxLength(256)
.HasColumnType("TEXT");
b.Property<long>("RequestedUtc")
.HasColumnType("INTEGER");
b.Property<long?>("SentUtc")
.HasColumnType("INTEGER");
b.Property<string>("State")
.IsRequired()
.HasMaxLength(16)
.HasColumnType("TEXT");
b.Property<long>("UpdatedUtc")
.HasColumnType("INTEGER");
b.HasKey("Id");
b.HasIndex("ClientId");
b.HasIndex("RequestedUtc");
b.HasIndex("State");
b.ToTable("DeviceRemovals", (string)null);
});
modelBuilder.Entity("Backend.Data.FunctionModuleEntity", b =>
{
b.Property<Guid>("Id")
@@ -457,9 +540,24 @@ namespace Backend.Data.Migrations.Sqlite
b.Property<long>("CreatedUtc")
.HasColumnType("INTEGER");
b.Property<string>("ErrorId")
.HasMaxLength(300)
.HasColumnType("TEXT");
b.Property<string>("ErrorMessage")
.HasMaxLength(2000)
.HasColumnType("TEXT");
b.Property<string>("ErrorType")
.HasMaxLength(300)
.HasColumnType("TEXT");
b.Property<int?>("ExitCode")
.HasColumnType("INTEGER");
b.Property<long?>("NotBeforeUtc")
.HasColumnType("INTEGER");
b.Property<string>("OutputSummary")
.HasColumnType("TEXT");
@@ -493,6 +591,8 @@ namespace Backend.Data.Migrations.Sqlite
b.HasIndex("ScriptId", "RunId");
b.HasIndex("State", "NotBeforeUtc");
b.ToTable("Jobs", (string)null);
});
@@ -677,6 +777,9 @@ namespace Backend.Data.Migrations.Sqlite
.IsRequired()
.HasColumnType("TEXT");
b.Property<string>("SnapshotProtected")
.HasColumnType("TEXT");
b.Property<string>("Source")
.IsRequired()
.HasMaxLength(32)
@@ -808,6 +911,14 @@ namespace Backend.Data.Migrations.Sqlite
b.Property<Guid>("ScriptId")
.HasColumnType("TEXT");
b.Property<int>("StaggerBatchSize")
.ValueGeneratedOnAdd()
.HasColumnType("INTEGER")
.HasDefaultValue(1);
b.Property<int>("StaggerSeconds")
.HasColumnType("INTEGER");
b.Property<string>("TimeZoneId")
.IsRequired()
.HasColumnType("TEXT");
+158 -29
View File
@@ -15,7 +15,9 @@ namespace Backend.Enrollment;
/// Agent bootstrap and credential refresh. Bootstrap validates and consumes a
/// one-time token, binds the agent's public NKey to the device and returns a
/// scoped NATS user JWT. Refresh proves possession of the enrolled NKey by
/// verifying a signature over <c>{clientId}|{timestampUtc:O}</c>.
/// verifying a signature over <c>{clientId}|{timestampUtc:O}</c>. Both answer
/// to a device's removal: tokens from before it no longer enroll the device, and
/// a renewal is told the device has gone.
/// </summary>
public static class EnrollmentEndpoints
{
@@ -39,6 +41,7 @@ public static class EnrollmentEndpoints
JetStreamProvisioner provisioner,
Messaging.BrowserEventBroadcaster broadcaster,
Groups.ScriptTriggerService triggers,
Removals.IDeviceRemovalSender removalSender,
AuditService audit,
HttpContext context,
ILoggerFactory loggerFactory,
@@ -88,6 +91,45 @@ public static class EnrollmentEndpoints
return Results.Unauthorized();
}
// A removed device comes back only on a token made after its removal, so
// the install it already has, or a GPO still pushing an old one, cannot.
var removal = await db.DeviceRemovals
.Where(r => r.ClientId == request.ClientId)
.OrderByDescending(r => r.RequestedUtc)
.FirstOrDefaultAsync(cancellationToken);
if (removal is not null && token.CreatedUtc <= removal.RequestedUtc)
{
await audit.LogAsync(AuditCategories.Enrollment, "bootstrap-rejected",
clientId: request.ClientId, detail: $"token predates the device's removal {removal.Id:D}", remoteIp: remoteIp,
cancellationToken: cancellationToken);
logger.LogWarning("Bootstrap rejected for removed client {ClientId} from {RemoteIp}", request.ClientId, remoteIp);
return Results.Unauthorized();
}
// Enrolling again: nothing queued for the old enrollment may replay, and
// the agent retries when the messaging system could not make sure of that.
if (removal is not null && device is null)
{
try
{
await removalSender.ClearCommandsAsync(request.ClientId, cancellationToken);
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
logger.LogWarning(ex, "Could not clear old commands of re-enrolling client {ClientId}", request.ClientId);
return Results.Problem(statusCode: StatusCodes.Status503ServiceUnavailable, title: "Messaging unavailable.");
}
if (Shared.Contracts.Models.DeviceRemovalStates.Open.Contains(removal.State))
{
removal.State = Shared.Contracts.Models.DeviceRemovalStates.Removed;
removal.Detail = "The device enrolled again.";
removal.FinishedUtc = now;
removal.UpdatedUtc = now;
Api.DeviceHistory.RecordRemoved(db, request.ClientId, "The device enrolled again on a new token before its agent uninstalled.");
}
}
// Idempotent enrollment by client id: re-enrollment rebinds the NKey.
var enrollmentId = Guid.NewGuid();
@@ -146,7 +188,12 @@ public static class EnrollmentEndpoints
device.BootstrapTokenId = token.Id;
var userJwt = issuer.IssueAgentCredential(request.ClientId, request.NkeyPublic);
// An agent too old to renew keeps a credential that never expires: one
// that expired would strand it, and it cannot take a remove command either.
DateTimeOffset? credentialExpires = RenewsItsCredential(request.AgentVersion)
? now + Shared.Contracts.Models.AgentCredentialDefaults.Lifetime
: null;
var userJwt = issuer.IssueAgentCredential(request.ClientId, request.NkeyPublic, credentialExpires);
await db.SaveChangesAsync(cancellationToken);
await audit.LogAsync(AuditCategories.Enrollment, "bootstrap-succeeded",
@@ -179,9 +226,12 @@ public static class EnrollmentEndpoints
await triggers.FireAsync(
Shared.Contracts.Models.ScriptTriggerType.OnRegistration, request.ClientId, cancellationToken);
return Results.Ok(new BootstrapResponse(enrollmentId, request.ClientId, userJwt, CredentialExpiresUtc: null));
return Results.Ok(new BootstrapResponse(enrollmentId, request.ClientId, userJwt, credentialExpires));
}
private static bool RenewsItsCredential(string? agentVersion)
=> Shared.Contracts.Models.AgentVersions.Compare(agentVersion, Shared.Contracts.Models.DeviceRemovalDefaults.MinimumAgentVersion) >= 0;
private static bool IsValidUserPublicKey(string candidate)
{
try
@@ -199,6 +249,7 @@ public static class EnrollmentEndpoints
CredentialRefreshRequest request,
AppDbContext db,
NatsCredentialIssuer issuer,
Removals.DeviceRemovalService removals,
AuditService audit,
HttpContext context,
CancellationToken cancellationToken)
@@ -209,8 +260,12 @@ public static class EnrollmentEndpoints
}
var device = await db.Devices.SingleOrDefaultAsync(d => d.ClientId == request.ClientId, cancellationToken);
if (device is null
|| !string.Equals(device.NkeyPublic, request.NkeyPublic, StringComparison.Ordinal)
if (device is null)
{
return await RefuseRemovedAsync(request, db, removals, audit, context, cancellationToken);
}
if (!string.Equals(device.NkeyPublic, request.NkeyPublic, StringComparison.Ordinal)
|| !issuer.IsConfigured)
{
return Results.Unauthorized();
@@ -226,32 +281,11 @@ public static class EnrollmentEndpoints
return Results.Unauthorized();
}
var skew = DateTimeOffset.UtcNow - request.TimestampUtc;
if (skew.Duration() > RefreshTimestampTolerance)
if (!await ProvesKeyAsync(request, device.NkeyPublic, audit, context, cancellationToken))
{
return Results.Unauthorized();
}
byte[] signature;
try
{
signature = Convert.FromBase64String(request.SignatureBase64);
}
catch (FormatException)
{
return Results.Unauthorized();
}
var payload = Encoding.UTF8.GetBytes($"{request.ClientId}|{request.TimestampUtc:O}");
using var verifier = KeyPair.FromPublicKey(device.NkeyPublic);
if (!verifier.Verify(payload, signature))
{
await audit.LogAsync(AuditCategories.Enrollment, "credential-refresh-rejected",
clientId: request.ClientId, detail: "signature verification failed",
remoteIp: ClientIp.EffectiveText(context), cancellationToken: cancellationToken);
return Results.Unauthorized();
}
// A verified refresh is the freshest sighting of the agent's public IP.
var publicIp = ClientIp.EffectiveText(context);
if (!string.Equals(device.PublicIp, publicIp, StringComparison.Ordinal))
@@ -260,7 +294,102 @@ public static class EnrollmentEndpoints
await db.SaveChangesAsync(cancellationToken);
}
var userJwt = issuer.IssueAgentCredential(request.ClientId, request.NkeyPublic);
return Results.Ok(new CredentialRefreshResponse(userJwt, CredentialExpiresUtc: null));
// Only agents that renew call this, so the new credential always expires.
var expires = DateTimeOffset.UtcNow + Shared.Contracts.Models.AgentCredentialDefaults.Lifetime;
var userJwt = issuer.IssueAgentCredential(request.ClientId, request.NkeyPublic, expires);
return Results.Ok(new CredentialRefreshResponse(userJwt, expires));
}
/// <summary>True when the request is fresh and signed with the enrolled key.</summary>
private static async Task<bool> ProvesKeyAsync(
CredentialRefreshRequest request, string nkeyPublic, AuditService audit, HttpContext context,
CancellationToken cancellationToken)
{
var skew = DateTimeOffset.UtcNow - request.TimestampUtc;
if (skew.Duration() > RefreshTimestampTolerance)
{
return false;
}
byte[] signature;
try
{
signature = Convert.FromBase64String(request.SignatureBase64);
}
catch (FormatException)
{
return false;
}
var payload = Encoding.UTF8.GetBytes($"{request.ClientId}|{request.TimestampUtc:O}");
using var verifier = KeyPair.FromPublicKey(nkeyPublic);
if (verifier.Verify(payload, signature))
{
return true;
}
await audit.LogAsync(AuditCategories.Enrollment, "credential-refresh-rejected",
clientId: request.ClientId, detail: "signature verification failed",
remoteIp: ClientIp.EffectiveText(context), cancellationToken: cancellationToken);
return false;
}
/// <summary>
/// A renewal for a client id with no device row. Only the agent holding the
/// enrolled key learns anything: everyone else gets the plain 401 an unknown
/// id gets. An open removal answers 410 and the agent uninstalls itself, which
/// also reaches a device that came back too briefly for a command. So does a
/// closed one, as an agent that returns from the dead should still go. A
/// stopped removal gets the plain 401: Stop Trying means its agent stays
/// installed and stays blocked, and 410 would uninstall it after all.
/// </summary>
private static async Task<IResult> RefuseRemovedAsync(
CredentialRefreshRequest request, AppDbContext db, Removals.DeviceRemovalService removals, AuditService audit,
HttpContext context, CancellationToken cancellationToken)
{
var removal = await removals.LatestAsync(request.ClientId, cancellationToken);
if (removal is null
|| !string.Equals(removal.NkeyPublic, request.NkeyPublic, StringComparison.Ordinal)
|| !await ProvesKeyAsync(request, removal.NkeyPublic, audit, context, cancellationToken))
{
return Results.Unauthorized();
}
var remoteIp = ClientIp.EffectiveText(context);
if (removal.State == Shared.Contracts.Models.DeviceRemovalStates.Stopped)
{
await audit.LogAsync(AuditCategories.Enrollment, "credential-refresh-refused",
clientId: request.ClientId, detail: "removal stopped: the device stays blocked", remoteIp: remoteIp,
cancellationToken: cancellationToken);
return Results.Unauthorized();
}
if (Shared.Contracts.Models.DeviceRemovalStates.Open.Contains(removal.State))
{
var now = DateTimeOffset.UtcNow;
removal.State = Shared.Contracts.Models.DeviceRemovalStates.Uninstalling;
removal.Reason = null;
removal.UpdatedUtc = now;
removal.LastSeenUtc = now;
try
{
await db.SaveChangesAsync(cancellationToken);
await removals.AnnounceAsync(removal, cancellationToken);
}
catch (DbUpdateConcurrencyException)
{
// Changed on another node meanwhile; the answer is the same, and the event or next renewal says so again.
db.ChangeTracker.Clear();
}
}
await audit.LogAsync(AuditCategories.Enrollment, "credential-refresh-refused",
clientId: request.ClientId, detail: $"device removed ({removal.Id:D})", remoteIp: remoteIp,
cancellationToken: cancellationToken);
return Results.Problem(
statusCode: StatusCodes.Status410Gone,
type: Shared.Contracts.Models.AgentCredentialDefaults.RemovedProblemType,
title: "This device was removed.",
extensions: new Dictionary<string, object?> { ["removalId"] = removal.Id });
}
}
+7 -2
View File
@@ -103,10 +103,15 @@ public sealed class ScriptTriggerService(
continue;
}
await runner.DispatchAsync(
prepared, job, TriggerExecutor.ParseArguments(trigger.ArgumentsJson),
var arguments = TriggerExecutor.ParseArguments(trigger.ArgumentsJson);
var sent = await runner.DispatchAsync(
prepared, job, arguments,
$"Trigger ({triggerType}) queued “{script.Name}” for {device.Hostname}",
cancellationToken);
if (!sent)
{
await runner.LeaveForDispatcherAsync(runId, prepared, arguments, [job], cancellationToken);
}
logger.LogInformation(
"Trigger {TriggerId} ({TriggerType}) fired for {ClientId} as job {JobId}",
+13 -1
View File
@@ -48,7 +48,9 @@ public static class TriggerEndpoints
Name: t.Name,
LastRunId: t.LastRunId,
Arguments: TriggerExecutor.ParseArguments(t.ArgumentsJson),
CreatedBy: t.CreatedBy)).ToList());
CreatedBy: t.CreatedBy,
StaggerSeconds: t.StaggerSeconds,
StaggerBatchSize: t.StaggerBatchSize)).ToList());
}
private static async Task<IResult> CreateAsync(
@@ -348,6 +350,11 @@ public static class TriggerEndpoints
return "Name must be 200 characters or fewer.";
}
if (JobStagger.Validate(request.StaggerSeconds, request.StaggerBatchSize) is { } badStagger)
{
return badStagger;
}
var timeZone = string.IsNullOrWhiteSpace(request.TimeZoneId) ? "UTC" : request.TimeZoneId.Trim();
string? cron = null;
DateTimeOffset? runOnce = null;
@@ -409,6 +416,11 @@ public static class TriggerEndpoints
entity.RunOnceUtc = runOnce;
entity.NextRunUtc = nextRun;
entity.ArgumentsJson = request.Arguments is { Count: > 0 } ? JsonDefaults.Serialize(request.Arguments) : null;
// Event triggers fire on one device at a time: nothing to spread.
var scheduled = request.TriggerType == ScriptTriggerType.Scheduled;
entity.StaggerSeconds = scheduled ? request.StaggerSeconds : 0;
entity.StaggerBatchSize = scheduled ? request.StaggerBatchSize : 1;
return null;
}
+19 -1
View File
@@ -11,11 +11,17 @@ public interface IAgentUpgradeSender
Task<bool> DispatchAgentUpgradeAsync(string clientId, AgentUpgradeCommandPayload command, CancellationToken cancellationToken);
}
/// <summary>Sends script commands (the job runner's seam; tests record instead).</summary>
public interface IScriptCommandSender
{
Task<bool> DispatchScriptAsync(string clientId, ScriptCommandPayload command, CancellationToken cancellationToken);
}
/// <summary>
/// Publishes durable agent commands through JetStream (awaiting stream acks)
/// so script dispatch survives agent disconnects and backend restarts.
/// </summary>
public sealed class CommandDispatcher(NatsHub hub, ILogger<CommandDispatcher> logger) : IAgentUpgradeSender
public sealed class CommandDispatcher(NatsHub hub, ILogger<CommandDispatcher> logger) : IAgentUpgradeSender, IScriptCommandSender
{
private static readonly EndpointRef BackendSource = new(EndpointKinds.Backend, "backend");
@@ -31,6 +37,18 @@ public sealed class CommandDispatcher(NatsHub hub, ILogger<CommandDispatcher> lo
return await PublishAsync(NatsSubjects.AgentCommandUpgrade(clientId), envelope, cancellationToken);
}
public async Task<bool> DispatchAgentRemoveAsync(
string clientId, AgentRemoveCommandPayload command, CancellationToken cancellationToken)
{
var envelope = MessageEnvelope.Create(
MessageTypes.CommandAgentRemove,
BackendSource,
command,
new EndpointRef(EndpointKinds.Agent, clientId));
return await PublishAsync(NatsSubjects.AgentCommandRemove(clientId), envelope, cancellationToken);
}
public async Task<bool> DispatchScriptAsync(
string clientId, ScriptCommandPayload command, CancellationToken cancellationToken)
{
+240
View File
@@ -0,0 +1,240 @@
using Backend.Data;
using Backend.Secrets;
using Microsoft.EntityFrameworkCore;
using Shared.Contracts.Models;
using Shared.Contracts.Serialization;
using Shared.Infrastructure.Constants;
namespace Backend.Jobs;
/// <summary>What one dispatcher pass did (for tests and logs).</summary>
public sealed record DispatchPass(int Sent, int Skipped, int Failed, bool More, bool Refused = false);
/// <summary>
/// Sends the jobs whose time has come: staggered jobs at their NotBeforeUtc,
/// and any job whose publish failed. Everything it needs is in the database
/// (the job's start time, the run's sealed snapshot), so a restart loses
/// nothing. Secrets and presigned URLs are prepared afresh for each pass, since
/// a run can outlast both.
///
/// Like the schedule worker it runs on the node with SCHEDULE_WORKER on. A job
/// is published first and then marked, only if still queued: a second send
/// after a crash is dropped by the agent, which dedupes on JobId.
/// </summary>
public sealed class JobDispatcher(IServiceScopeFactory scopeFactory, ILogger<JobDispatcher> logger) : BackgroundService
{
public static readonly TimeSpan Tick = TimeSpan.FromSeconds(2);
public const int PassLimit = 500;
/// <summary>The pause after messaging refused a publish, so an outage is not retried and logged every tick.</summary>
public static readonly TimeSpan RefusedDelay = TimeSpan.FromSeconds(15);
/// <summary>A vault or S3 failure is retried this long before the job fails.</summary>
public static readonly TimeSpan PrepareRetryWindow = TimeSpan.FromMinutes(5);
/// <summary>A job that could not be sent for this long fails, rather than running long after anyone expects it.</summary>
public static readonly TimeSpan SendExpiry = TimeSpan.FromHours(24);
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
var more = false;
var delay = Tick;
try
{
var pass = await RunPassAsync(DateTimeOffset.UtcNow, stoppingToken);
more = pass.More;
delay = pass.Refused ? RefusedDelay : Tick;
}
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
{
return;
}
catch (Exception ex)
{
logger.LogError(ex, "Job dispatch pass failed");
}
if (more)
{
continue;
}
try
{
await Task.Delay(delay, stoppingToken);
}
catch (OperationCanceledException)
{
return;
}
}
}
internal async Task<DispatchPass> RunPassAsync(DateTimeOffset now, CancellationToken cancellationToken)
{
using var scope = scopeFactory.CreateScope();
var services = scope.ServiceProvider;
var db = services.GetRequiredService<AppDbContext>();
var runner = services.GetRequiredService<JobRunner>();
var snapshots = services.GetRequiredService<RunSnapshots>();
var due = await db.Jobs
.Where(j => j.State == JobState.Queued && j.NotBeforeUtc != null && j.NotBeforeUtc <= now)
.OrderBy(j => j.NotBeforeUtc)
.Take(PassLimit)
.ToListAsync(cancellationToken);
var tally = new Tally();
var delivering = true;
foreach (var group in due.GroupBy(j => j.RunId))
{
if (group.Key is not { } runId)
{
continue;
}
delivering = await DispatchRunAsync(db, runner, snapshots, runId, group.ToList(), now, delivering, tally, cancellationToken);
}
await runner.ClearFinishedSnapshotsAsync(cancellationToken);
if (tally.Sent + tally.Skipped + tally.Failed > 0)
{
logger.LogDebug("Dispatch pass: {Sent} sent, {Skipped} skipped, {Failed} failed", tally.Sent, tally.Skipped, tally.Failed);
}
// Messaging being down is no reason to spin: the next pass tries again, after a pause.
return new DispatchPass(tally.Sent, tally.Skipped, tally.Failed, More: delivering && due.Count == PassLimit, Refused: !delivering);
}
/// <summary>One run's due jobs. Returns false once a publish was refused, so the rest of the pass sends nothing.</summary>
private async Task<bool> DispatchRunAsync(
AppDbContext db, JobRunner runner, RunSnapshots snapshots, Guid runId, List<JobEntity> jobs, DateTimeOffset now,
bool delivering, Tally tally, CancellationToken cancellationToken)
{
var run = await db.Runs.SingleOrDefaultAsync(r => r.RunId == runId, cancellationToken);
var snapshot = snapshots.Open(run?.SnapshotProtected);
if (run is null || snapshot is null)
{
await FailAllAsync(runner, jobs, "The run's saved script could not be read.", tally, cancellationToken);
return delivering;
}
if (!await db.Scripts.AnyAsync(s => s.Id == snapshot.ScriptId, cancellationToken))
{
await FailAllAsync(runner, jobs, "The script was deleted before this device's start.", tally, cancellationToken);
return delivering;
}
var clientIds = jobs.Select(j => j.ClientId).ToList();
var devices = await db.Devices.AsNoTracking()
.Where(d => clientIds.Contains(d.ClientId))
.ToDictionaryAsync(d => d.ClientId, StringComparer.Ordinal, cancellationToken);
var skips = new List<RunSkip>();
var sendable = new List<(JobEntity Job, DeviceEntity Device)>();
foreach (var job in jobs)
{
if (now - job.NotBeforeUtc > SendExpiry)
{
await FailAsync(runner, job, $"Could not be sent within {SendExpiry.TotalHours:0} hours.", tally, cancellationToken);
}
else if (!devices.TryGetValue(job.ClientId, out var device))
{
await FailAsync(runner, job, "The device was removed before its start.", tally, cancellationToken);
}
else if (device.IsOrphaned)
{
skips.Add(new RunSkip(device.ClientId, device.Hostname, SkipReasons.Orphaned));
await RemoveAsync(db, job, cancellationToken);
}
else if (run.OnlineOnly && ProtocolDefaults.CalculateStatus(device.LastSeenUtc, now) != DeviceStatus.Online)
{
skips.Add(new RunSkip(device.ClientId, device.Hostname, SkipReasons.Offline));
await RemoveAsync(db, job, cancellationToken);
}
else
{
sendable.Add((job, device));
}
}
if (skips.Count > 0)
{
var skipped = JsonDefaults.Deserialize<List<RunSkip>>(run.SkippedJson) ?? [];
skipped.AddRange(skips);
run.SkippedJson = JsonDefaults.Serialize(skipped);
await db.SaveChangesAsync(cancellationToken);
tally.Skipped += skips.Count;
}
if (sendable.Count == 0 || !delivering)
{
return delivering;
}
PreparedScript prepared;
try
{
prepared = await runner.PrepareAsync(snapshot, cancellationToken);
}
catch (Exception ex) when (ex is SecretResolutionException or Backend.Files.FileResolutionException)
{
foreach (var (job, _) in sendable.Where(s => now - s.Job.NotBeforeUtc > PrepareRetryWindow))
{
await FailAsync(runner, job, ex.Message, tally, cancellationToken);
}
logger.LogWarning("Run {RunId}: could not prepare “{Script}”: {Message}", runId, snapshot.Name, ex.Message);
return delivering;
}
foreach (var (job, device) in sendable)
{
var sent = await runner.DispatchAsync(
prepared, job, snapshot.Arguments, $"Job “{snapshot.Name}” started for {device.Hostname}", cancellationToken);
if (!sent)
{
logger.LogWarning("Run {RunId}: job {JobId} could not be sent; retrying next pass", runId, job.JobId);
return false;
}
tally.Sent++;
await runner.PruneHistoryAsync(job.ClientId, cancellationToken);
}
return delivering;
}
private static async Task FailAllAsync(
JobRunner runner, List<JobEntity> jobs, string message, Tally tally, CancellationToken cancellationToken)
{
foreach (var job in jobs)
{
await FailAsync(runner, job, message, tally, cancellationToken);
}
}
private static async Task FailAsync(
JobRunner runner, JobEntity job, string message, Tally tally, CancellationToken cancellationToken)
{
await runner.MarkFailedAsync(job, message, cancellationToken);
tally.Failed++;
}
/// <summary>A skipped device has no job: a run counts its targets as jobs plus skips.</summary>
private static Task RemoveAsync(AppDbContext db, JobEntity job, CancellationToken cancellationToken)
{
db.Entry(job).State = EntityState.Detached;
return db.Jobs.Where(j => j.JobId == job.JobId && j.State == JobState.Queued).ExecuteDeleteAsync(cancellationToken);
}
private sealed class Tally
{
public int Sent;
public int Skipped;
public int Failed;
}
}
+136 -25
View File
@@ -27,6 +27,70 @@ public static class JobEndpoints
app.MapGet(ApiRoutes.JobOutputDownload, DownloadJobOutputAsync).RequirePermission(Permissions.JobsView);
app.MapGet(ApiRoutes.ExecutionOutputDownload, DownloadExecutionOutputAsync).RequirePermission(Permissions.JobsView);
app.MapPost(ApiRoutes.ExecutionOutputArchive, DownloadExecutionArchiveAsync).RequirePermission(Permissions.JobsView);
app.MapPost(ApiRoutes.ExecutionCancelWaiting, CancelWaitingAsync).RequirePermission(Permissions.JobsRun);
app.MapGet(ApiRoutes.ExecutionBreakdown, GetBreakdownAsync).RequirePermission(Permissions.JobsView);
}
/// <summary>
/// A run's exit codes, grouped errors and completions over time, for the
/// jobs in the caller's scope. Reads only the columns it needs; a job's
/// output is read only when nothing else says why it failed.
/// </summary>
private static async Task<IResult> GetBreakdownAsync(
Guid runId, AppDbContext db, DeviceScopeService scope, HttpContext context, CancellationToken cancellationToken)
{
var jobs = await (await ScopedJobsAsync(db, scope, context, cancellationToken))
.Where(j => j.RunId == runId || j.JobId == runId)
.Select(j => new RunBreakdownJob(
j.JobId, j.ClientId, j.State, j.ExitCode, j.CreatedUtc, j.CompletedUtc, j.ErrorType, j.ErrorMessage, j.OutputSummary))
.ToListAsync(cancellationToken);
if (jobs.Count == 0)
{
var run = await db.Runs.AsNoTracking().SingleOrDefaultAsync(r => r.RunId == runId, cancellationToken);
var allowed = await scope.GetAllowedClientIdsAsync(context.User, cancellationToken);
return SkipsOf(run, allowed).Count == 0 ? Results.NotFound() : Results.Ok(RunBreakdown.Build([], new Dictionary<Guid, string>(), new Dictionary<string, string>()));
}
// The first Error-stream line of each job that has nothing better (agents before error reporting).
var silent = RunBreakdown.NeedingOutput(jobs);
var firstLines = silent.Count == 0
? []
: await db.JobOutput.AsNoTracking()
.Where(o => silent.Contains(o.JobId) && o.Stream == OutputStreamKind.Error
&& !db.JobOutput.Any(p => p.JobId == o.JobId && p.Stream == OutputStreamKind.Error && p.Sequence < o.Sequence))
.ToDictionaryAsync(o => o.JobId, o => o.Text, cancellationToken);
var hostnames = await HostnamesAsync(db, jobs.Select(j => j.ClientId).Distinct().ToList(), cancellationToken);
return Results.Ok(RunBreakdown.Build(jobs, firstLines, hostnames));
}
/// <summary>
/// Cancels the run's jobs not yet sent (those the dispatcher owns), within
/// the caller's device scope; a job already on its way is left alone.
/// </summary>
private static async Task<IResult> CancelWaitingAsync(
Guid runId, AppDbContext db, DeviceScopeService scope, JobRunner runner, AuditService audit, HttpContext context,
CancellationToken cancellationToken)
{
var jobs = (await ScopedJobsAsync(db, scope, context, cancellationToken)).Where(j => j.RunId == runId);
if (!await jobs.AnyAsync(cancellationToken))
{
return Results.NotFound();
}
var now = DateTimeOffset.UtcNow;
var cancelled = await jobs
.Where(j => j.State == JobState.Queued && j.NotBeforeUtc != null)
.ExecuteUpdateAsync(
s => s.SetProperty(j => j.State, JobState.Cancelled)
.SetProperty(j => j.CompletedUtc, now)
.SetProperty(j => j.OutputSummary, "Cancelled before its start."),
cancellationToken);
await runner.ClearFinishedSnapshotsAsync(cancellationToken);
await audit.LogAsync(AuditCategories.Jobs, "cancel-waiting", context.User.Identity?.Name,
detail: $"run={runId} cancelled={cancelled}", cancellationToken: cancellationToken);
return Results.Ok(new CancelWaitingResultDto(cancelled));
}
/// <summary>One job's complete stored output as a text file.</summary>
@@ -255,7 +319,7 @@ public static class JobEndpoints
var allowed = await scope.GetAllowedClientIdsAsync(context.User, cancellationToken);
var online = await OnlineAsync(db, groups.SelectMany(g => g).Select(j => j.ClientId), cancellationToken);
return groups.Select(g =>
var summaries = groups.Select(g =>
{
var run = runs.GetValueOrDefault(g.Key);
var counts = Count(g.ToList(), SkipsOf(run, allowed), online);
@@ -268,12 +332,33 @@ public static class JobEndpoints
g.Count(),
counts.Succeeded,
counts.Failed,
counts.Pending + counts.Running,
counts.Waiting + counts.Pending + counts.Running,
run?.ScriptName ?? first.ScriptName,
run?.Source,
run?.TargetsLabel,
counts);
}).ToList();
// A staggered run whose devices were all offline at their start has only skips left.
var emptied = await db.Runs.AsNoTracking()
.Where(r => (scriptId == null || r.ScriptId == scriptId)
&& r.SkippedJson != "[]"
&& !db.Jobs.Any(j => j.RunId == r.RunId))
.OrderByDescending(r => r.CreatedUtc)
.Take(limit)
.ToListAsync(cancellationToken);
foreach (var run in emptied)
{
var skips = SkipsOf(run, allowed);
if (skips.Count > 0)
{
summaries.Add(new ExecutionSummaryDto(
run.RunId, run.ScriptId, run.CreatedUtc, run.RequestedBy, 0, 0, 0, 0, run.ScriptName, run.Source,
run.TargetsLabel, Count([], skips, online)));
}
}
return summaries.OrderByDescending(s => s.StartedUtc).Take(limit).ToList();
}
/// <summary>A single execution: every target with its state, the skipped devices and the counts.</summary>
@@ -284,27 +369,32 @@ public static class JobEndpoints
.Where(j => j.RunId == runId || j.JobId == runId)
.OrderBy(j => j.CreatedUtc)
.ToListAsync(cancellationToken);
if (jobs.Count == 0)
{
return Results.NotFound();
}
var run = await db.Runs.AsNoTracking().SingleOrDefaultAsync(r => r.RunId == runId, cancellationToken);
var allowed = await scope.GetAllowedClientIdsAsync(context.User, cancellationToken);
var skipped = SkipsOf(run, allowed);
// No jobs but visible skips: a staggered run whose devices were all offline at their start.
if (jobs.Count == 0 && skipped.Count == 0)
{
return Results.NotFound();
}
var hostnames = await HostnamesAsync(db, jobs, cancellationToken);
var online = await OnlineAsync(db, jobs.Select(j => j.ClientId), cancellationToken);
var targets = jobs.Select(j => new ExecutionTargetDto(
j.JobId, j.ClientId, hostnames.GetValueOrDefault(j.ClientId), j.State, j.ExitCode,
j.CreatedUtc, j.StartedUtc, j.CompletedUtc, online.Contains(j.ClientId), j.OutputSummary)).ToList();
j.CreatedUtc, j.StartedUtc, j.CompletedUtc, online.Contains(j.ClientId), j.OutputSummary ?? j.ErrorMessage,
j.NotBeforeUtc)).ToList();
var first = jobs[0];
var first = jobs.FirstOrDefault();
return Results.Ok(new ExecutionDetailDto(
runId, first.ScriptId, run?.ScriptName ?? first.ScriptName, jobs.Min(j => j.CreatedUtc),
runId, first?.ScriptId ?? run!.ScriptId, run?.ScriptName ?? first!.ScriptName, first?.CreatedUtc ?? run!.CreatedUtc,
run?.RequestedBy ?? jobs.Select(j => j.RequestedBy).FirstOrDefault(r => r is not null), targets,
run?.Source, run?.TargetsLabel, run?.OnlineOnly ?? false,
skipped.Select(s => new ExecutionSkipDto(s.ClientId, s.Hostname, s.Reason)).ToList(),
Count(jobs, skipped, online)));
Count(jobs, skipped, online),
DateTimeOffset.UtcNow));
}
/// <summary>The run's skipped devices a scoped role may see.</summary>
@@ -336,15 +426,21 @@ public static class JobEndpoints
}
private static RunCountsDto Count(IReadOnlyCollection<JobEntity> jobs, IReadOnlyCollection<RunSkip> skipped, HashSet<string> online)
=> new(
{
var now = DateTimeOffset.UtcNow;
var waiting = jobs.Count(j => j.State == JobState.Queued && j.NotBeforeUtc > now);
return new(
Targets: jobs.Count + skipped.Count,
Pending: jobs.Count(j => j.State is JobState.Queued or JobState.Dispatched),
Pending: jobs.Count(j => j.State is JobState.Queued or JobState.Dispatched) - waiting,
Running: jobs.Count(j => j.State == JobState.Running),
Succeeded: jobs.Count(j => j.State == JobState.Succeeded),
Failed: jobs.Count(j => j.State is JobState.Failed or JobState.TimedOut),
Cancelled: jobs.Count(j => j.State == JobState.Cancelled),
Skipped: skipped.Count,
Online: jobs.Select(j => j.ClientId).Distinct().Count(online.Contains));
Online: jobs.Select(j => j.ClientId).Distinct().Count(online.Contains),
Waiting: waiting,
TimedOut: jobs.Count(j => j.State == JobState.TimedOut));
}
private static async Task<IResult> ListAsync(
AppDbContext db, DeviceScopeService scope, HttpContext context, string? clientId, Guid? scriptId, int? limit,
@@ -409,11 +505,23 @@ public static class JobEndpoints
private static Task<Dictionary<string, string>> HostnamesAsync(
AppDbContext db, IReadOnlyCollection<JobEntity> jobs, CancellationToken cancellationToken)
=> HostnamesAsync(db, jobs.Select(j => j.ClientId).Distinct().ToList(), cancellationToken);
private static async Task<Dictionary<string, string>> HostnamesAsync(
AppDbContext db, List<string> clientIds, CancellationToken cancellationToken)
{
var clientIds = jobs.Select(j => j.ClientId).Distinct().ToList();
return db.Devices.AsNoTracking()
var hostnames = await db.Devices.AsNoTracking()
.Where(d => clientIds.Contains(d.ClientId))
.ToDictionaryAsync(d => d.ClientId, d => d.Hostname, cancellationToken);
// Old runs of a removed device keep its name.
foreach (var (clientId, hostname) in await Removals.DeviceRemovalService.RemovedHostnamesAsync(
db, clientIds.Where(id => !hostnames.ContainsKey(id)), cancellationToken))
{
hostnames[clientId] = hostname;
}
return hostnames;
}
private static async Task<IResult> CreateAsync(
@@ -444,6 +552,11 @@ public static class JobEndpoints
return Results.BadRequest(new { error = "Invalid client id." });
}
if (JobStagger.Validate(request.StaggerSeconds, request.StaggerBatchSize) is { } badStagger)
{
return Results.BadRequest(new { error = badStagger });
}
var script = await db.Scripts.AsNoTracking()
.SingleOrDefaultAsync(s => s.Id == request.ScriptId, cancellationToken);
if (script is null)
@@ -502,7 +615,6 @@ public static class JobEndpoints
.ToListAsync(cancellationToken);
var now = DateTimeOffset.UtcNow;
var jobs = new List<JobDto>();
var runId = Guid.NewGuid();
var via = selected.Count == 1 && clientIds.Count == 0 ? $" (group “{selected[0].Name}”)" : string.Empty;
@@ -535,14 +647,12 @@ public static class JobEndpoints
cancellationToken, request.OnlineOnly);
}
foreach (var device in eligible)
{
var job = await runner.CreateJobAsync(script, device.ClientId, actor, cancellationToken, runId);
await runner.DispatchAsync(
prepared, job, request.Arguments,
$"Job “{script.Name}” queued for {device.Hostname}{via}", cancellationToken);
jobs.Add(ToDto(job, device.Hostname));
}
var hostnames = eligible.ToDictionary(d => d.ClientId, d => d.Hostname, StringComparer.Ordinal);
var jobs = (await runner.StartRunAsync(
runId, prepared, request.Arguments, eligible, actor, request.StaggerSeconds, request.StaggerBatchSize,
device => $"Job “{script.Name}” queued for {device.Hostname}{via}", cancellationToken))
.Select(job => ToDto(job, hostnames.GetValueOrDefault(job.ClientId)))
.ToList();
if (singleDevice && jobs.Count == 0)
{
@@ -561,7 +671,8 @@ public static class JobEndpoints
await audit.LogAsync(AuditCategories.Jobs, "group-run", actor,
detail: $"script={script.Name} run={runId} groups={selected.Count} devices={clientIds.Count} " +
$"jobs={jobs.Count} skippedOrphaned={skippedOrphaned.Count} skippedOffline={skippedOffline.Count}",
$"jobs={jobs.Count} skippedOrphaned={skippedOrphaned.Count} skippedOffline={skippedOffline.Count}" +
(request.StaggerSeconds > 0 ? $" staggerSeconds={request.StaggerSeconds} staggerBatchSize={request.StaggerBatchSize}" : string.Empty),
cancellationToken: cancellationToken);
return Results.Ok(new GroupRunDto(
+157 -10
View File
@@ -30,10 +30,11 @@ public sealed record PreparedScript(
/// </summary>
public sealed class JobRunner(
AppDbContext db,
CommandDispatcher dispatcher,
IScriptCommandSender dispatcher,
BrowserEventBroadcaster broadcaster,
SecretsProviderService secretsService,
Backend.Files.S3PresignService s3,
RunSnapshots snapshots,
IConfiguration configuration)
{
/// <summary>
@@ -43,9 +44,18 @@ public sealed class JobRunner(
/// </summary>
public async Task<PreparedScript> PrepareAsync(ScriptEntity script, CancellationToken cancellationToken)
{
var parameters = Backend.Scripts.ScriptEndpoints.DeserializeParameters(script.ParametersJson);
var moduleIds = Backend.Scripts.ScriptEndpoints.DeserializeModuleIds(script.FunctionModuleIdsJson);
var prelude = await ComposePreludeAsync(moduleIds, cancellationToken);
return await PrepareAsync(script, prelude, cancellationToken);
}
/// <summary>The same from a run's snapshot: secrets and presigned URLs are new, the script and prelude as they were.</summary>
public Task<PreparedScript> PrepareAsync(RunSnapshot snapshot, CancellationToken cancellationToken)
=> PrepareAsync(snapshot.ToScript(), snapshot.Prelude, cancellationToken);
private async Task<PreparedScript> PrepareAsync(ScriptEntity script, string? prelude, CancellationToken cancellationToken)
{
var parameters = Backend.Scripts.ScriptEndpoints.DeserializeParameters(script.ParametersJson);
IReadOnlyList<ScriptSecret>? secrets = null;
var bindings = Backend.Scripts.ScriptEndpoints.DeserializeSecretBindings(script.SecretBindingsJson);
@@ -173,12 +183,113 @@ public sealed class JobRunner(
return job;
}
/// <summary>
/// Starts a recorded run on its devices. Without a stagger, each device's
/// job is created and sent here; a publish that fails is left for the
/// dispatcher to retry. With one, every job is written at once with its
/// start time and the dispatcher sends them, so this returns immediately
/// however many devices there are.
/// </summary>
public async Task<IReadOnlyList<JobEntity>> StartRunAsync(
Guid runId,
PreparedScript prepared,
IReadOnlyDictionary<string, JsonElement>? arguments,
IReadOnlyList<DeviceEntity> devices,
string? requestedBy,
int staggerSeconds,
int staggerBatchSize,
Func<DeviceEntity, string> activityText,
CancellationToken cancellationToken)
{
if (JobStagger.Applies(staggerSeconds, staggerBatchSize, devices.Count))
{
return await ScheduleStaggeredAsync(
runId, prepared, arguments, devices, requestedBy, staggerSeconds, staggerBatchSize, cancellationToken);
}
var jobs = new List<JobEntity>(devices.Count);
var unsent = new List<JobEntity>();
foreach (var device in devices)
{
var job = await CreateJobAsync(prepared.Script, device.ClientId, requestedBy, cancellationToken, runId);
jobs.Add(job);
if (!await DispatchAsync(prepared, job, arguments, activityText(device), cancellationToken))
{
unsent.Add(job);
}
}
if (unsent.Count > 0)
{
await LeaveForDispatcherAsync(runId, prepared, arguments, unsent, cancellationToken);
}
return jobs;
}
private async Task<IReadOnlyList<JobEntity>> ScheduleStaggeredAsync(
Guid runId, PreparedScript prepared, IReadOnlyDictionary<string, JsonElement>? arguments,
IReadOnlyList<DeviceEntity> devices, string? requestedBy, int staggerSeconds, int staggerBatchSize,
CancellationToken cancellationToken)
{
var script = prepared.Script;
var start = DateTimeOffset.UtcNow;
var jobs = JobStagger
.Plan(runId, devices.Select(d => d.ClientId), staggerSeconds, staggerBatchSize)
.Select(slot => new JobEntity
{
JobId = Guid.NewGuid(),
RunId = runId,
ClientId = slot.ClientId,
ScriptId = script.Id,
ScriptName = script.Name,
State = JobState.Queued,
CreatedUtc = start,
RequestedBy = requestedBy,
NotBeforeUtc = start + slot.Offset,
})
.ToList();
db.Jobs.AddRange(jobs);
// One save, so the dispatcher never sees a job without its snapshot.
var run = await db.Runs.SingleAsync(r => r.RunId == runId, cancellationToken);
run.SnapshotProtected = snapshots.Seal(RunSnapshot.Of(prepared, arguments));
await db.SaveChangesAsync(cancellationToken);
await broadcaster.BroadcastActivityAsync(new ActivityEventPayload(
"job-created", $"Job “{script.Name}” staggered across {jobs.Count} devices",
null, null, DateTimeOffset.UtcNow), cancellationToken);
return jobs;
}
/// <summary>Hands jobs whose publish failed to the dispatcher, with what it needs to send them.</summary>
public async Task LeaveForDispatcherAsync(
Guid runId, PreparedScript prepared, IReadOnlyDictionary<string, JsonElement>? arguments,
IReadOnlyList<JobEntity> jobs, CancellationToken cancellationToken)
{
var now = DateTimeOffset.UtcNow;
foreach (var job in jobs)
{
job.NotBeforeUtc = now;
}
var run = await db.Runs.SingleAsync(r => r.RunId == runId, cancellationToken);
run.SnapshotProtected = snapshots.Seal(RunSnapshot.Of(prepared, arguments));
await db.SaveChangesAsync(cancellationToken);
}
/// <summary>Drops the snapshots of runs with nothing left to send: arguments stay at rest no longer than needed.</summary>
public Task<int> ClearFinishedSnapshotsAsync(CancellationToken cancellationToken)
=> db.Runs
.Where(r => r.SnapshotProtected != null && !db.Jobs.Any(j => j.RunId == r.RunId && j.State == JobState.Queued))
.ExecuteUpdateAsync(s => s.SetProperty(r => r.SnapshotProtected, (string?)null), cancellationToken);
/// <summary>
/// Execution history retention: keeps the newest JOB_HISTORY_LIMIT jobs
/// per device (default 200, 0 disables pruning) and drops the stored
/// output of anything older.
/// </summary>
private async Task PruneHistoryAsync(string clientId, CancellationToken cancellationToken)
public async Task PruneHistoryAsync(string clientId, CancellationToken cancellationToken)
{
var limit = int.TryParse(configuration["JOB_HISTORY_LIMIT"], out var configured) && configured >= 0
? configured
@@ -228,8 +339,12 @@ public sealed class JobRunner(
if (dispatched)
{
job.State = JobState.Dispatched;
await db.SaveChangesAsync(cancellationToken);
// Only from Queued: a fast agent's Running may already be stored, and
// an operator may have cancelled the job meanwhile.
var moved = await db.Jobs
.Where(j => j.JobId == job.JobId && j.State == JobState.Queued)
.ExecuteUpdateAsync(s => s.SetProperty(j => j.State, JobState.Dispatched), cancellationToken);
await SyncAsync(job, moved > 0, j => j.State = JobState.Dispatched, cancellationToken);
}
await broadcaster.BroadcastActivityAsync(new ActivityEventPayload(
@@ -238,13 +353,45 @@ public sealed class JobRunner(
return dispatched;
}
/// <summary>Terminal failure before dispatch (e.g. the vault was unreachable).</summary>
/// <summary>Terminal failure before dispatch (e.g. the vault was unreachable). Only a job still queued changes.</summary>
public async Task MarkFailedAsync(JobEntity job, string message, CancellationToken cancellationToken)
{
job.State = JobState.Failed;
job.CompletedUtc = DateTimeOffset.UtcNow;
job.OutputSummary = message;
await db.SaveChangesAsync(cancellationToken);
var now = DateTimeOffset.UtcNow;
var moved = await db.Jobs
.Where(j => j.JobId == job.JobId && j.State == JobState.Queued)
.ExecuteUpdateAsync(
s => s.SetProperty(j => j.State, JobState.Failed)
.SetProperty(j => j.CompletedUtc, now)
.SetProperty(j => j.OutputSummary, message),
cancellationToken);
await SyncAsync(job, moved > 0, j =>
{
j.State = JobState.Failed;
j.CompletedUtc = now;
j.OutputSummary = message;
}, cancellationToken);
}
/// <summary>Brings a tracked job in line with a direct update (or, when another writer won, with the database).</summary>
private async Task SyncAsync(JobEntity job, bool moved, Action<JobEntity> apply, CancellationToken cancellationToken)
{
var entry = db.Entry(job);
if (!moved)
{
if (entry.State != EntityState.Detached)
{
await entry.ReloadAsync(cancellationToken);
}
return;
}
apply(job);
if (entry.State != EntityState.Detached)
{
entry.OriginalValues.SetValues(entry.CurrentValues);
entry.State = EntityState.Unchanged;
}
}
private async Task<string?> ComposePreludeAsync(
+62
View File
@@ -0,0 +1,62 @@
using System.Security.Cryptography;
using System.Text;
namespace Backend.Jobs;
/// <summary>
/// When each device of a staggered run starts. The devices fall into batches
/// (slots) of StaggerBatchSize, evenly spaced over StaggerSeconds: the first
/// starts at once, the last when the spread ends. The order is shuffled by a
/// hash of (run, device), so it is repeatable for a run but the same device is
/// not always last.
/// </summary>
public static class JobStagger
{
public const int MaxSeconds = 86_400;
public const int MaxBatchSize = 10_000;
/// <summary>The problem with these settings, or null.</summary>
public static string? Validate(int seconds, int batchSize)
{
if (seconds is < 0 or > MaxSeconds)
{
return $"Spread Over must be between 0 and {MaxSeconds} seconds.";
}
return batchSize is < 1 or > MaxBatchSize
? $"Devices at a Time must be between 1 and {MaxBatchSize}."
: null;
}
public static int Slots(int targets, int batchSize) =>
targets <= 0 ? 0 : (targets + Math.Max(batchSize, 1) - 1) / Math.Max(batchSize, 1);
/// <summary>False when everything would start at once anyway, and the run dispatches as usual.</summary>
public static bool Applies(int seconds, int batchSize, int targets) => seconds > 0 && Slots(targets, batchSize) > 1;
/// <summary>How long after the run's start a slot begins: slot × spread ÷ (slots − 1).</summary>
public static TimeSpan SlotOffset(int slot, int slots, int seconds) =>
slots <= 1 ? TimeSpan.Zero : TimeSpan.FromTicks(slot * (seconds * TimeSpan.TicksPerSecond) / (slots - 1));
/// <summary>Each client id with how long after the run's start it begins, earliest first.</summary>
public static IReadOnlyList<(string ClientId, TimeSpan Offset)> Plan(
Guid runId, IEnumerable<string> clientIds, int seconds, int batchSize)
{
var ordered = clientIds
.Select(id => (Id: id, Rank: Rank(runId, id)))
.OrderBy(d => d.Rank)
.ThenBy(d => d.Id, StringComparer.Ordinal)
.Select(d => d.Id)
.ToList();
var size = Math.Max(batchSize, 1);
var slots = Slots(ordered.Count, size);
return ordered.Select((id, index) => (id, SlotOffset(index / size, slots, seconds))).ToList();
}
private static ulong Rank(Guid runId, string clientId)
{
Span<byte> hash = stackalloc byte[SHA256.HashSizeInBytes];
SHA256.HashData([.. runId.ToByteArray(), .. Encoding.UTF8.GetBytes(clientId)], hash);
return BitConverter.ToUInt64(hash);
}
}
+143
View File
@@ -0,0 +1,143 @@
using System.Text.RegularExpressions;
using Shared.Contracts.Api;
using Shared.Contracts.Models;
namespace Backend.Jobs;
/// <summary>What the breakdown needs of one job, so thousands of them stay cheap to read.</summary>
public sealed record RunBreakdownJob(
Guid JobId,
string ClientId,
JobState State,
int? ExitCode,
DateTimeOffset CreatedUtc,
DateTimeOffset? CompletedUtc,
string? ErrorType,
string? ErrorMessage,
string? OutputSummary);
/// <summary>Turns a run's jobs into exit-code counts, grouped errors and completions over time.</summary>
public static partial class RunBreakdown
{
public const int MaxSamples = 5;
public const int MaxMessageLength = 300;
public const string NoMessage = "No Error Reported";
private const int MaxBuckets = 60;
private static readonly int[] BucketSeconds = [5, 10, 30, 60, 300, 900, 1800, 3600, 7200, 14400, 43200, 86400];
/// <summary>The failed jobs nothing says why about: their first Error-stream line stands in (the caller reads those).</summary>
public static List<Guid> NeedingOutput(IEnumerable<RunBreakdownJob> jobs) => jobs
.Where(j => Failed(j) && string.IsNullOrWhiteSpace(j.ErrorMessage) && string.IsNullOrWhiteSpace(j.OutputSummary))
.Select(j => j.JobId)
.ToList();
public static ExecutionBreakdownDto Build(
IReadOnlyCollection<RunBreakdownJob> jobs,
IReadOnlyDictionary<Guid, string> firstErrorLines,
IReadOnlyDictionary<string, string> hostnames)
{
var exitCodes = jobs
.Where(j => j.CompletedUtc is not null && j.ExitCode is not null)
.GroupBy(j => j.ExitCode!.Value)
.Select(g => new ExitCodeCountDto(g.Key, g.Count()))
.OrderByDescending(c => c.Count)
.ThenBy(c => c.ExitCode)
.ToList();
var errors = jobs
.Where(Failed)
.Select(j => (Job: j, Message: Normalize(j.ErrorMessage ?? j.OutputSummary ?? firstErrorLines.GetValueOrDefault(j.JobId))))
.GroupBy(e => (e.Job.ErrorType, e.Message))
.Select(g => new RunErrorGroupDto(
g.Key.ErrorType,
g.Key.Message,
g.Count(),
g.Take(MaxSamples)
.Select(e => new RunErrorSampleDto(e.Job.JobId, e.Job.ClientId, hostnames.GetValueOrDefault(e.Job.ClientId)))
.ToList(),
g.Select(e => e.Job.JobId).ToList()))
.OrderByDescending(g => g.Count)
.ThenBy(g => g.Message, StringComparer.Ordinal)
.ToList();
var (bucketSeconds, completions) = Completions(jobs);
return new ExecutionBreakdownDto(exitCodes, errors, bucketSeconds, completions);
}
/// <summary>
/// An error's first line, without what differs from device to device or
/// run to run: "At line:1 char:1" position text and file system paths.
/// </summary>
public static string Normalize(string? text)
{
var line = text?.Split('\n', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries).FirstOrDefault();
if (string.IsNullOrEmpty(line))
{
return NoMessage;
}
line = PositionText().Replace(line, string.Empty);
line = QuotedPath().Replace(line, m => $"{m.Groups[1].Value}…{m.Groups[1].Value}");
line = BarePath().Replace(line, "…");
line = Whitespace().Replace(line, " ").Trim();
if (line.Length == 0)
{
return NoMessage;
}
return line.Length > MaxMessageLength ? line[..(MaxMessageLength - 1)] + "…" : line;
}
private static bool Failed(RunBreakdownJob job) => job.State is JobState.Failed or JobState.TimedOut;
private static (int Seconds, List<RunCompletionBucketDto> Buckets) Completions(IReadOnlyCollection<RunBreakdownJob> jobs)
{
var finished = jobs.Where(j => j.CompletedUtc is not null).ToList();
if (finished.Count == 0)
{
return (BucketSeconds[0], []);
}
var start = jobs.Min(j => j.CreatedUtc);
var span = Math.Max(0, (finished.Max(j => j.CompletedUtc!.Value) - start).TotalSeconds);
var seconds = BucketSeconds.FirstOrDefault(b => span / b < MaxBuckets);
if (seconds == 0)
{
seconds = (int)Math.Ceiling(span / MaxBuckets);
}
var count = (int)(span / seconds) + 1;
var tally = new (int Succeeded, int Failed, int TimedOut, int Cancelled)[count];
foreach (var job in finished)
{
// An agent's clock can put a completion before the run's start.
var index = Math.Clamp((int)((job.CompletedUtc!.Value - start).TotalSeconds / seconds), 0, count - 1);
switch (job.State)
{
case JobState.Succeeded: tally[index].Succeeded++; break;
case JobState.Failed: tally[index].Failed++; break;
case JobState.TimedOut: tally[index].TimedOut++; break;
case JobState.Cancelled: tally[index].Cancelled++; break;
}
}
return (seconds, tally
.Select((t, i) => new RunCompletionBucketDto(start.AddSeconds(i * seconds), t.Succeeded, t.Failed, t.TimedOut, t.Cancelled))
.ToList());
}
[GeneratedRegex(@"\s*\bAt (?:line:\d+|\S*:\d+) char:\d+.*$", RegexOptions.IgnoreCase)]
private static partial Regex PositionText();
/// <summary>A quoted path, spaces included: 'C:\Program Files\x', "\\server\share\y", '/var/log/z'.</summary>
[GeneratedRegex("""(['"])(?:[A-Za-z]:[\\/]|\\\\|/)[^'"]*\1""")]
private static partial Regex QuotedPath();
/// <summary>An unquoted path: C:\Temp\x.log, \\server\share, /var/log/z (two segments at least, so "and/or" stays).</summary>
[GeneratedRegex(@"(?:(?<![\w])[A-Za-z]:[\\/][^\s'"",;]*|\\\\[^\s'"",;]+|(?<![\w/:.])/(?:[\w.\-]+/)+[\w.\-]*)")]
private static partial Regex BarePath();
[GeneratedRegex(@"\s+")]
private static partial Regex Whitespace();
}
+77
View File
@@ -0,0 +1,77 @@
using System.Security.Cryptography;
using System.Text.Json;
using Backend.Data;
using Microsoft.AspNetCore.DataProtection;
using Shared.Contracts.Serialization;
namespace Backend.Jobs;
/// <summary>
/// A script as it was when its run started, with the run's arguments: every
/// device of a run the dispatcher sends later gets this, whatever the script
/// has become since. It holds secret and file <em>bindings</em>; their values
/// and presigned URLs are resolved anew at each dispatch and never stored.
/// </summary>
public sealed record RunSnapshot(
Guid ScriptId,
string Name,
string Language,
string Content,
string? ParametersJson,
string? Prelude,
string? SecretBindingsJson,
string? SecretsVariableName,
string? FileBindingsJson,
IReadOnlyDictionary<string, JsonElement>? Arguments)
{
public static RunSnapshot Of(PreparedScript prepared, IReadOnlyDictionary<string, JsonElement>? arguments) => new(
prepared.Script.Id,
prepared.Script.Name,
prepared.Script.Language,
prepared.Script.Content,
prepared.Script.ParametersJson,
prepared.Prelude,
prepared.Script.SecretBindingsJson,
prepared.Script.SecretsVariableName,
prepared.Script.FileBindingsJson,
arguments);
/// <summary>A detached script entity to prepare and dispatch from.</summary>
public ScriptEntity ToScript() => new()
{
Id = ScriptId,
Name = Name,
Language = Language,
Content = Content,
ParametersJson = ParametersJson,
SecretBindingsJson = SecretBindingsJson,
SecretsVariableName = SecretsVariableName,
FileBindingsJson = FileBindingsJson,
};
}
/// <summary>Seals run snapshots with Data Protection: arguments can be sensitive, so the database never holds them in the clear.</summary>
public sealed class RunSnapshots(IDataProtectionProvider provider)
{
private readonly IDataProtector protector = provider.CreateProtector("Backend.Jobs.RunSnapshot");
public string Seal(RunSnapshot snapshot) => protector.Protect(JsonDefaults.Serialize(snapshot));
/// <summary>The snapshot, or null when it can no longer be opened (the key ring was replaced).</summary>
public RunSnapshot? Open(string? sealedValue)
{
if (sealedValue is null)
{
return null;
}
try
{
return JsonDefaults.Deserialize<RunSnapshot>(protector.Unprotect(sealedValue));
}
catch (Exception ex) when (ex is CryptographicException or JsonException)
{
return null;
}
}
}
+4 -8
View File
@@ -62,14 +62,10 @@ public sealed class TriggerExecutor(
.ToList();
await runner.RecordRunAsync(
runId, script, requestedBy, RunSources.Schedule, targetsLabel, skipped, cancellationToken, triggerId: trigger.Id);
foreach (var device in devices)
{
var job = await runner.CreateJobAsync(script, device.ClientId, requestedBy, cancellationToken, runId);
await runner.DispatchAsync(
prepared, job, arguments,
$"{label} queued “{script.Name}” for {device.Hostname}",
cancellationToken);
}
await runner.StartRunAsync(
runId, prepared, arguments, devices, requestedBy, trigger.StaggerSeconds, trigger.StaggerBatchSize,
device => $"{label} queued “{script.Name}” for {device.Hostname}",
cancellationToken);
logger.LogInformation(
"Trigger {TriggerId} queued {Count} job(s) of “{Script}”", trigger.Id, devices.Count, script.Name);
+30 -5
View File
@@ -92,7 +92,9 @@ public sealed class MessagingWorkers(
var device = await db.Devices.SingleOrDefaultAsync(d => d.ClientId == payload.ClientId, cancellationToken);
if (device is null)
{
continue; // Presence from a device we have no enrollment for.
// No enrollment: a removal may be waiting for this device; otherwise dropped.
await Removals.DeviceRemovalEvents.NoteSightingAsync(scope.ServiceProvider, msg.Subject, payload.ClientId, payload.AgentVersion, cancellationToken);
continue;
}
if (device.IsOrphaned)
@@ -161,6 +163,7 @@ public sealed class MessagingWorkers(
var device = await db.Devices.SingleOrDefaultAsync(d => d.ClientId == clientId, cancellationToken);
if (device is null)
{
await Removals.DeviceRemovalEvents.NoteSightingAsync(scope.ServiceProvider, msg.Subject, clientId, null, cancellationToken);
continue;
}
@@ -208,8 +211,8 @@ public sealed class MessagingWorkers(
/// <summary>
/// Agent lifecycle events. A "ready" notice means the agent is connected
/// and consuming job commands, which fires OnAgentReady script triggers.
/// Agent upgrade progress and outcomes share the subject, told apart by
/// the envelope's message type.
/// Agent upgrade and removal progress share the subject, told apart by the
/// envelope's message type.
/// </summary>
private async Task ConsumeAgentEventsAsync(CancellationToken cancellationToken)
{
@@ -230,6 +233,20 @@ public sealed class MessagingWorkers(
continue;
}
if (envelope?.MessageType == MessageTypes.AgentRemoval)
{
try
{
await Removals.DeviceRemovalEvents.HandleAsync(scopeFactory, msg.Subject, envelope, cancellationToken);
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
logger.LogError(ex, "Agent removal event from {Subject} failed", msg.Subject);
}
continue;
}
AgentEventPayload? payload = null;
try
{
@@ -253,6 +270,7 @@ public sealed class MessagingWorkers(
var device = await db.Devices.SingleOrDefaultAsync(d => d.ClientId == payload.ClientId, cancellationToken);
if (device is null)
{
await Removals.DeviceRemovalEvents.NoteSightingAsync(scope.ServiceProvider, msg.Subject, payload.ClientId, null, cancellationToken);
continue;
}
@@ -350,7 +368,9 @@ public sealed class MessagingWorkers(
ApplyJobState(job, payload);
await db.SaveChangesAsync(cancellationToken);
await broadcaster.BroadcastJobEventAsync(payload.JobId, MessageTypes.JobState, payload, cancellationToken);
// Browsers get the run too (agents do not know it), so a run page can pick out its own jobs.
await broadcaster.BroadcastJobEventAsync(
payload.JobId, MessageTypes.JobState, payload with { RunId = job.RunId ?? job.JobId }, cancellationToken);
if (payload.State is JobState.Running)
{
@@ -370,7 +390,7 @@ public sealed class MessagingWorkers(
}
}
private static void ApplyJobState(JobEntity job, JobStatePayload payload)
internal static void ApplyJobState(JobEntity job, JobStatePayload payload)
{
// Never regress a terminal state (redelivery safety).
if (job.State is JobState.Succeeded or JobState.Failed or JobState.Cancelled or JobState.TimedOut)
@@ -388,8 +408,13 @@ public sealed class MessagingWorkers(
job.CompletedUtc = payload.TimestampUtc;
job.ExitCode = payload.ExitCode;
job.OutputSummary = payload.Message;
job.ErrorType = Cut(payload.ErrorType, 300);
job.ErrorId = Cut(payload.ErrorId, 300);
job.ErrorMessage = Cut(payload.ErrorMessage, 2000);
break;
}
static string? Cut(string? value, int length) => value is { } text && text.Length > length ? text[..length] : value;
}
private async Task SweepStatusesAsync(CancellationToken cancellationToken)
+30
View File
@@ -74,4 +74,34 @@ public sealed class JetStreamProvisioner(NatsHub hub, ILogger<JetStreamProvision
logger.LogInformation("Durable command consumer ensured for {ClientId}", clientId);
}
/// <summary>
/// Deletes one agent's durable command consumer and purges its command
/// subjects. The stream keeps acked messages until they age out, and a new
/// consumer delivers everything still there, so without this a device enrolled
/// again would replay its old commands.
/// </summary>
public async Task RemoveAgentCommandsAsync(string clientId, CancellationToken cancellationToken)
{
if (!hub.IsConfigured)
{
return;
}
var js = new NatsJSContext(hub.Require());
try
{
await js.DeleteConsumerAsync(
JetStreamDefaults.CommandStreamName, JetStreamDefaults.AgentCommandConsumerName(clientId), cancellationToken);
}
catch (NatsJSApiException ex) when (ex.Error.Code == 404)
{
// Already gone.
}
var stream = await js.GetStreamAsync(JetStreamDefaults.CommandStreamName, cancellationToken: cancellationToken);
await stream.PurgeAsync(new StreamPurgeRequest { Filter = NatsSubjects.AgentCommandWildcard(clientId) }, cancellationToken);
logger.LogInformation("Durable command consumer and commands cleared for {ClientId}", clientId);
}
}
+11 -4
View File
@@ -7,9 +7,10 @@ namespace Backend.Nats;
/// <summary>
/// Issues scoped NATS user JWTs signed with the account signing key
/// (NATS_ACCOUNT_SIGNING_SEED). Agents get long-lived credentials restricted
/// to their own client id; browsers get short-lived credentials restricted to
/// their session subjects. The signing seed never appears in logs or responses.
/// (NATS_ACCOUNT_SIGNING_SEED). Agents get credentials restricted to their own
/// client id, expiring within a day for agents that renew them; browsers get
/// short-lived credentials restricted to their session subjects. The signing
/// seed never appears in logs or responses.
/// </summary>
public sealed class NatsCredentialIssuer : IDisposable
{
@@ -30,12 +31,18 @@ public sealed class NatsCredentialIssuer : IDisposable
public bool IsConfigured => signingKey is not null;
public string IssueAgentCredential(string clientId, string userPublicKey)
/// <param name="expiresUtc">Null only for agents too old to renew a credential.</param>
public string IssueAgentCredential(string clientId, string userPublicKey, DateTimeOffset? expiresUtc)
{
var (key, account) = RequireConfiguration();
var claims = NatsJwt.NewUserClaims(userPublicKey);
claims.Name = $"agent-{clientId}";
if (expiresUtc is { } expires)
{
claims.Expires = expires;
}
claims.User.IssuerAccount = account;
claims.User.Pub.Allow = [.. NatsSubjects.AgentPublishPermissions(clientId)];
claims.User.Sub.Allow = [.. NatsSubjects.AgentSubscribePermissions(clientId)];
+11 -3
View File
@@ -164,6 +164,11 @@ builder.Services.AddSingleton<Backend.Upgrades.AgentUpgradeSettingsStore>();
builder.Services.AddScoped<Backend.Upgrades.AgentCatalogService>();
builder.Services.AddScoped<Backend.Upgrades.AgentUpgradeService>();
builder.Services.AddSingleton<IAgentUpgradeSender>(sp => sp.GetRequiredService<CommandDispatcher>());
// Device removals: the tombstones and the sender their scheduler (with the schedules) uses.
builder.Services.AddScoped<Backend.Removals.DeviceRemovalService>();
builder.Services.AddSingleton<Backend.Removals.IDeviceRemovalSender, Backend.Removals.DeviceRemovalSender>();
builder.Services.AddSingleton<IScriptCommandSender>(sp => sp.GetRequiredService<CommandDispatcher>());
builder.Services.AddSingleton<RunSnapshots>();
if (builder.Configuration["AGENT_CATALOG_SYNC"] != "0")
{
builder.Services.AddHostedService<Backend.Upgrades.AgentCatalogWorker>();
@@ -182,13 +187,15 @@ builder.Services.AddSingleton<Backend.Secrets.ISecretsProviderDriver, Backend.Se
builder.Services.AddScoped<Backend.Secrets.SecretsProviderService>();
builder.Services.AddSingleton<ShellBridge>();
builder.Services.AddHostedService<MessagingWorkers>();
// SCHEDULE_WORKER=0 keeps the background sweeps (schedules, agent upgrades)
// out of test hosts, where they race the tests' deterministic manual sweeps,
// and off all but the one worker replica in a scaled deployment.
// SCHEDULE_WORKER=0 keeps the background sweeps (schedules, staggered jobs,
// agent upgrades) out of test hosts, where they race the tests' deterministic
// manual sweeps, and off all but the one worker replica in a scaled deployment.
if (builder.Configuration["SCHEDULE_WORKER"] != "0")
{
builder.Services.AddHostedService<ScheduleWorker>();
builder.Services.AddHostedService<JobDispatcher>();
builder.Services.AddHostedService<Backend.Upgrades.AgentUpgradeScheduler>();
builder.Services.AddHostedService<Backend.Removals.DeviceRemovalScheduler>();
}
builder.Services.AddHostedService(sp => sp.GetRequiredService<ShellBridge>());
@@ -403,6 +410,7 @@ GroupEndpoints.Map(app);
TriggerEndpoints.Map(app);
SettingsEndpoints.Map(app);
Backend.Upgrades.AgentUpgradeEndpoints.Map(app);
Backend.Removals.DeviceRemovalEndpoints.Map(app);
Backend.Secrets.SecretsEndpoints.Map(app);
if (staticUiPath is not null)
@@ -0,0 +1,189 @@
using Backend.Api;
using Backend.Auth;
using Backend.Data;
using Backend.Groups;
using Backend.Messaging;
using Backend.Security;
using Microsoft.EntityFrameworkCore;
using Shared.Contracts.Api;
using Shared.Contracts.Messages;
using Shared.Contracts.Models;
using Shared.Infrastructure.Nats;
namespace Backend.Removals;
/// <summary>Removing devices: the request, the list behind the Devices page, and Stop Trying.</summary>
public static class DeviceRemovalEndpoints
{
public const int DefaultListLimit = 200;
public static void Map(WebApplication app)
{
app.MapGet(ApiRoutes.DeviceRemovals, ListAsync).RequirePermission(Permissions.DevicesView);
app.MapPost(ApiRoutes.DeviceRemovals, RemoveAsync).RequirePermission(Permissions.DevicesRemove);
app.MapPost(ApiRoutes.DeviceRemovalStop, StopAsync).RequirePermission(Permissions.DevicesRemove);
}
private static async Task<IResult> ListAsync(
string? state, string? clientId, int? limit, AppDbContext db, DeviceScopeService scope, HttpContext context,
CancellationToken cancellationToken)
{
var query = db.DeviceRemovals.AsNoTracking();
if (!string.IsNullOrWhiteSpace(state))
{
query = query.Where(r => r.State == state);
}
if (!string.IsNullOrWhiteSpace(clientId))
{
query = query.Where(r => r.ClientId == clientId);
}
var take = Math.Clamp(limit ?? DefaultListLimit, 1, 5000);
query = query.OrderByDescending(r => r.RequestedUtc);
// A scoped role sees removals by the groups kept on each row: they are
// JSON, so the page is cut after this filter for such a role.
var scopeGroupIds = await scope.GetScopeGroupIdsAsync(context.User, cancellationToken);
var rows = scopeGroupIds is null
? await query.Take(take).ToListAsync(cancellationToken)
: (await query.ToListAsync(cancellationToken))
.Where(r => DeviceRemovalService.IsVisibleTo(r, scopeGroupIds))
.Take(take)
.ToList();
var now = DateTimeOffset.UtcNow;
return Results.Ok(rows.Select(r => DeviceRemovalService.ToDto(r, now)).ToList());
}
private static async Task<IResult> RemoveAsync(
RemoveDevicesRequest request, AppDbContext db, DeviceRemovalService removals, DeviceGroupService groups,
DeviceScopeService scope, AuditService audit, BrowserEventBroadcaster broadcaster, HttpContext context,
CancellationToken cancellationToken)
{
var named = (request.ClientIds ?? []).Distinct(StringComparer.Ordinal).ToList();
var groupIds = (request.GroupIds ?? []).Distinct().ToList();
if (named.Count == 0 && groupIds.Count == 0)
{
return Results.BadRequest(new { error = "Choose at least one device or group." });
}
if (named.Any(id => !SubjectIdentifier.IsValid(id)))
{
return Results.BadRequest(new { error = "Invalid client id." });
}
var selected = await db.DeviceGroups.AsNoTracking()
.Where(g => groupIds.Contains(g.Id))
.ToListAsync(cancellationToken);
if (selected.Count < groupIds.Count)
{
return Results.NotFound(new { error = groupIds.Count == 1 ? "Group not found." : "Some groups no longer exist." });
}
var fromGroups = new HashSet<string>(StringComparer.Ordinal);
foreach (var group in selected)
{
fromGroups.UnionWith(await groups.GetMembersAsync(group, cancellationToken));
}
// A scoped role only reaches the devices inside its own scope. A device it
// named outside that reads the same whether it exists or not; one a group
// brought in is not its to know about, as with agent upgrades.
var allowed = await scope.GetAllowedClientIdsAsync(context.User, cancellationToken);
var skipped = new List<DeviceRemovalSkipDto>();
var targets = new HashSet<string>(StringComparer.Ordinal);
foreach (var clientId in named)
{
if (allowed is null || allowed.Contains(clientId))
{
targets.Add(clientId);
}
else
{
skipped.Add(new DeviceRemovalSkipDto(clientId, null, DeviceRemovalService.SkipOutOfScope));
}
}
targets.UnionWith(allowed is null ? fromGroups : fromGroups.Where(allowed.Contains));
var note = string.IsNullOrWhiteSpace(request.Reason) ? null : request.Reason.Trim();
if (note is { Length: > DeviceRemovalService.NoteMaxLength })
{
note = note[..DeviceRemovalService.NoteMaxLength];
}
var actor = context.User.Identity?.Name;
var (removed, unknown) = await removals.RemoveAsync(targets.Order(StringComparer.Ordinal).ToList(), note, actor, cancellationToken);
skipped.AddRange(unknown);
var left = string.Join(' ', skipped.GroupBy(s => s.Reason).Select(g => $"{g.Key}={g.Count()}"));
await audit.LogAsync(AuditCategories.Admin, "device-removal-requested", actor,
clientId: removed.Count == 1 ? removed[0].ClientId : null,
detail: $"removed={removed.Count}{(left.Length > 0 ? " skipped: " + left : string.Empty)}",
cancellationToken: cancellationToken);
if (removed.Count > 0)
{
await broadcaster.BroadcastActivityAsync(new ActivityEventPayload(
DeviceRemovalService.ActivityKind,
removed.Count == 1 ? $"{removed[0].Hostname} removal requested" : $"Removal requested for {removed.Count} devices",
removed.Count == 1 ? removed[0].ClientId : null, null, DateTimeOffset.UtcNow), cancellationToken);
}
var now = DateTimeOffset.UtcNow;
return Results.Ok(new DeviceRemovalBatchDto(
removed.OrderBy(r => r.Hostname, StringComparer.OrdinalIgnoreCase).Select(r => DeviceRemovalService.ToDto(r, now)).ToList(),
skipped));
}
/// <summary>
/// Stops chasing a device: it stays blocked and its agent stays installed.
/// A remove command still waiting for it is purged so it cannot arrive later;
/// that is best effort, since a command left behind expires with the stream.
/// </summary>
private static async Task<IResult> StopAsync(
Guid id, AppDbContext db, DeviceRemovalService removals, IDeviceRemovalSender sender, DeviceScopeService scope,
AuditService audit, HttpContext context, ILogger<DeviceRemovalService> logger, CancellationToken cancellationToken)
{
var row = await db.DeviceRemovals.SingleOrDefaultAsync(r => r.Id == id, cancellationToken);
if (row is null || !DeviceRemovalService.IsVisibleTo(row, await scope.GetScopeGroupIdsAsync(context.User, cancellationToken)))
{
return Results.NotFound();
}
if (!DeviceRemovalStates.Open.Contains(row.State))
{
return Results.Conflict(new { error = $"This removal has already finished; it is {row.State}." });
}
var now = DateTimeOffset.UtcNow;
row.State = DeviceRemovalStates.Stopped;
row.FinishedUtc = now;
row.UpdatedUtc = now;
var actor = context.User.Identity?.Name;
DeviceHistory.RecordRemovalStopped(db, row.ClientId, actor);
try
{
await db.SaveChangesAsync(cancellationToken);
}
catch (DbUpdateConcurrencyException)
{
return Results.Conflict(new { error = "The removal changed meanwhile. Try again." });
}
try
{
await sender.ClearCommandsAsync(row.ClientId, cancellationToken);
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
logger.LogWarning(ex, "Could not clear the commands of device {ClientId} after its removal stopped", row.ClientId);
}
await audit.LogAsync(AuditCategories.Admin, "device-removal-stopped", actor, clientId: row.ClientId,
detail: $"removal={row.Id}", cancellationToken: cancellationToken);
await removals.AnnounceAsync(row, cancellationToken);
return Results.Ok(DeviceRemovalService.ToDto(row, now));
}
}
@@ -0,0 +1,288 @@
using Backend.Api;
using Backend.Data;
using Backend.Jobs;
using Backend.Nats;
using Microsoft.EntityFrameworkCore;
using Shared.Contracts.Messages;
using Shared.Contracts.Models;
using Shared.Infrastructure.Constants;
namespace Backend.Removals;
/// <summary>Reaches an agent for its removal (the seam the tests record through).</summary>
public interface IDeviceRemovalSender
{
/// <summary>Publishes the remove command; false when messaging is down.</summary>
Task<bool> SendRemoveAsync(string clientId, AgentRemoveCommandPayload command, CancellationToken cancellationToken);
/// <summary>
/// Deletes the agent's durable consumer and purges its command subjects, so
/// a device enrolled again never replays old commands. Throws when messaging
/// is down and something may be left behind.
/// </summary>
Task ClearCommandsAsync(string clientId, CancellationToken cancellationToken);
}
public sealed class DeviceRemovalSender(CommandDispatcher dispatcher, JetStreamProvisioner provisioner, ILogger<DeviceRemovalSender> logger)
: IDeviceRemovalSender
{
public async Task<bool> SendRemoveAsync(string clientId, AgentRemoveCommandPayload command, CancellationToken cancellationToken)
{
try
{
// A removal reopened after it closed has lost its consumer.
await provisioner.EnsureAgentConsumerAsync(clientId, cancellationToken);
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
logger.LogWarning(ex, "Could not ensure the command consumer for {ClientId}", clientId);
return false;
}
return await dispatcher.DispatchAgentRemoveAsync(clientId, command, cancellationToken);
}
public Task ClearCommandsAsync(string clientId, CancellationToken cancellationToken)
=> provisioner.RemoveAgentCommandsAsync(clientId, cancellationToken);
}
/// <summary>What one scheduler pass did (for tests and logs).</summary>
public sealed record RemovalPass(int Sent, int Finished);
/// <summary>
/// Drives open removals every 15 seconds:
/// <list type="bullet">
/// <item>Sends the remove command to a device seen online, not more often than
/// <see cref="ResendAfter"/>. An agent too old to take it is marked and stays
/// open: the server keeps blocking it.</item>
/// <item>A command nobody answered goes back to waiting after
/// <see cref="ReportDeadline"/>, for when the device next returns.</item>
/// <item>An agent that said it was uninstalling and then went silent counts as
/// gone, since nobody is left to report success: its consumer and commands are
/// cleared and the removal closes. One that keeps talking was not uninstalled,
/// and goes back to waiting.</item>
/// </list>
/// Like the schedule worker it runs on the node with SCHEDULE_WORKER on; each
/// row's write is guarded by its concurrency stamp, so a second scheduler, an
/// event on another node or an operator's stop can never double-send a row.
/// </summary>
public sealed class DeviceRemovalScheduler(
IServiceScopeFactory scopeFactory,
IDeviceRemovalSender sender,
ILogger<DeviceRemovalScheduler> logger) : BackgroundService
{
public static readonly TimeSpan Tick = TimeSpan.FromSeconds(15);
public static readonly TimeSpan ResendAfter = TimeSpan.FromMinutes(5);
public static readonly TimeSpan ReportDeadline = TimeSpan.FromMinutes(30);
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
try
{
await RunPassAsync(DateTimeOffset.UtcNow, stoppingToken);
}
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
{
return;
}
catch (Exception ex)
{
logger.LogError(ex, "Device removal pass failed");
}
try
{
await Task.Delay(Tick, stoppingToken);
}
catch (OperationCanceledException)
{
return;
}
}
}
internal async Task<RemovalPass> RunPassAsync(DateTimeOffset now, CancellationToken cancellationToken)
{
using var scope = scopeFactory.CreateScope();
var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
var removals = scope.ServiceProvider.GetRequiredService<DeviceRemovalService>();
var open = DeviceRemovalStates.Open.ToList();
var rows = await db.DeviceRemovals.Where(r => open.Contains(r.State)).ToListAsync(cancellationToken);
var sent = 0;
var finished = 0;
var messagingDown = false;
foreach (var row in rows)
{
switch (row.State)
{
case DeviceRemovalStates.Uninstalling:
finished += await AdvanceUninstallingAsync(db, removals, row, now, cancellationToken) ? 1 : 0;
break;
case DeviceRemovalStates.Sent:
if (now - (row.SentUtc ?? row.UpdatedUtc) > ReportDeadline)
{
row.State = DeviceRemovalStates.Waiting;
row.UpdatedUtc = now;
if (await TrySaveAsync(db, row, cancellationToken))
{
await removals.AnnounceAsync(row, cancellationToken);
}
}
break;
case DeviceRemovalStates.Waiting or DeviceRemovalStates.Failed when !messagingDown:
switch (await TrySendAsync(db, removals, row, now, cancellationToken))
{
case SendResult.Sent:
sent++;
break;
case SendResult.MessagingDown:
messagingDown = true;
logger.LogWarning("Removal {RemovalId} could not be sent; retrying next pass", row.Id);
break;
}
break;
}
}
return new RemovalPass(sent, finished);
}
private enum SendResult
{
NotDue,
Sent,
MessagingDown,
}
private async Task<SendResult> TrySendAsync(
AppDbContext db, DeviceRemovalService removals, DeviceRemovalEntity row, DateTimeOffset now, CancellationToken cancellationToken)
{
// Presence brings the running version, so an agent upgraded by hand gets past this.
var tooOld = !string.IsNullOrWhiteSpace(row.AgentVersion)
&& AgentVersions.Compare(row.AgentVersion, DeviceRemovalDefaults.MinimumAgentVersion) < 0;
if (tooOld)
{
if (row.Reason != DeviceRemovalReasons.AgentTooOld)
{
row.Reason = DeviceRemovalReasons.AgentTooOld;
row.Detail = $"The agent runs {row.AgentVersion}; removing it needs {DeviceRemovalDefaults.MinimumAgentVersion} or later.";
row.UpdatedUtc = now;
if (await TrySaveAsync(db, row, cancellationToken))
{
await removals.AnnounceAsync(row, cancellationToken);
}
}
return SendResult.NotDue;
}
var online = ProtocolDefaults.CalculateStatus(row.LastSeenUtc, now) == DeviceStatus.Online;
if (!online
|| string.IsNullOrWhiteSpace(row.AgentVersion)
|| (row.SentUtc is { } lastSent && now - lastSent < ResendAfter))
{
return SendResult.NotDue;
}
// Mark it sent first: a row is never handed to JetStream twice.
var (state, reason, detail, sentUtc, updatedUtc) = (row.State, row.Reason, row.Detail, row.SentUtc, row.UpdatedUtc);
row.State = DeviceRemovalStates.Sent;
row.Reason = null;
row.Detail = null;
row.SentUtc = now;
row.UpdatedUtc = now;
row.Attempts++;
if (!await TrySaveAsync(db, row, cancellationToken))
{
return SendResult.NotDue; // Stopped or taken meanwhile.
}
if (!await sender.SendRemoveAsync(row.ClientId, new AgentRemoveCommandPayload(row.Id, row.EnrollmentId), cancellationToken))
{
// Messaging is down: as it was, the next pass tries again.
(row.State, row.Reason, row.Detail, row.SentUtc, row.UpdatedUtc) = (state, reason, detail, sentUtc, updatedUtc);
row.Attempts--;
await TrySaveAsync(db, row, cancellationToken);
return SendResult.MessagingDown;
}
await removals.AnnounceAsync(row, cancellationToken);
return SendResult.Sent;
}
/// <summary>True when the removal closed.</summary>
private async Task<bool> AdvanceUninstallingAsync(
AppDbContext db, DeviceRemovalService removals, DeviceRemovalEntity row, DateTimeOffset now, CancellationToken cancellationToken)
{
// UpdatedUtc is when it said so: presence moves only LastSeenUtc.
if (row.LastSeenUtc is { } seen && seen - row.UpdatedUtc > DeviceRemovalDefaults.UninstallSilence)
{
row.State = DeviceRemovalStates.Waiting;
row.Detail = "The agent kept reporting after it said it was uninstalling.";
row.UpdatedUtc = now;
if (await TrySaveAsync(db, row, cancellationToken))
{
await removals.AnnounceAsync(row, cancellationToken);
}
return false;
}
var lastSign = row.LastSeenUtc is { } last && last > row.UpdatedUtc ? last : row.UpdatedUtc;
if (now - lastSign <= DeviceRemovalDefaults.UninstallSilence)
{
return false;
}
try
{
await sender.ClearCommandsAsync(row.ClientId, cancellationToken);
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
// Close it only once nothing can replay; the next pass tries again.
logger.LogWarning(ex, "Could not clear the commands of removed device {ClientId}", row.ClientId);
return false;
}
row.State = DeviceRemovalStates.Removed;
row.FinishedUtc = now;
row.UpdatedUtc = now;
DeviceHistory.RecordRemoved(db, row.ClientId, "The agent said it was uninstalling, then went silent.");
if (!await TrySaveAsync(db, row, cancellationToken))
{
return false;
}
await removals.AnnounceAsync(row, cancellationToken);
return true;
}
/// <summary>Saves one row's change; false (the row re-read, its history dropped) when it changed elsewhere first.</summary>
private static async Task<bool> TrySaveAsync(AppDbContext db, DeviceRemovalEntity row, CancellationToken cancellationToken)
{
try
{
await db.SaveChangesAsync(cancellationToken);
return true;
}
catch (DbUpdateConcurrencyException)
{
foreach (var added in db.ChangeTracker.Entries<AuditEventEntity>().Where(e => e.State == EntityState.Added).ToList())
{
added.State = EntityState.Detached;
}
await db.Entry(row).ReloadAsync(cancellationToken);
return false;
}
}
}
@@ -0,0 +1,400 @@
using Backend.Api;
using Backend.Data;
using Backend.Groups;
using Backend.Messaging;
using Backend.Security;
using Backend.Upgrades;
using Microsoft.EntityFrameworkCore;
using Shared.Contracts.Api;
using Shared.Contracts.Messages;
using Shared.Contracts.Models;
using Shared.Contracts.Serialization;
using Shared.Infrastructure.Nats;
namespace Backend.Removals;
/// <summary>
/// Removing devices: the tombstone that stands in for a deleted device row, the
/// agent's sightings and events, and the device-history trail. The scheduler
/// sends what is open here; nothing is sent from this class.
/// </summary>
public sealed class DeviceRemovalService(
AppDbContext db,
DeviceGroupService groups,
BrowserEventBroadcaster broadcaster,
ILogger<DeviceRemovalService> logger)
{
/// <summary>Activity kind consoles refresh devices and removals on.</summary>
public const string ActivityKind = "device-removal";
public const string SkipOutOfScope = "out-of-scope";
public const string SkipUnknown = "unknown";
public const int NoteMaxLength = 500;
private const int DetailMaxLength = 2000;
private const int ReasonMaxLength = 32;
/// <summary>States an agent may report.</summary>
private static readonly IReadOnlySet<string> AgentStates = new HashSet<string>(StringComparer.Ordinal)
{
DeviceRemovalStates.Uninstalling, DeviceRemovalStates.Failed,
};
public static DeviceRemovalDto ToDto(DeviceRemovalEntity row, DateTimeOffset now) => new(
row.Id, row.ClientId, row.Hostname, row.OsCaption, row.AgentVersion, row.State, row.Reason, row.Detail,
row.Attempts, row.RequestedBy, row.Note, row.RequestedUtc, row.UpdatedUtc, row.SentUtc, row.LastSeenUtc,
row.FinishedUtc, ShowInDeviceList(row, now));
public static bool ShowInDeviceList(DeviceRemovalEntity row, DateTimeOffset now)
=> now - row.UpdatedUtc < DeviceRemovalDefaults.VisibleInDeviceList;
/// <summary>
/// For each device: its tombstone in, its row and everything that would
/// reattach to a returning client id out, in one save. A device that is
/// already gone (a second request racing this one) is skipped as unknown.
/// </summary>
public async Task<(List<DeviceRemovalEntity> Removed, List<DeviceRemovalSkipDto> Skipped)> RemoveAsync(
IReadOnlyList<string> clientIds, string? note, string? requestedBy, CancellationToken cancellationToken)
{
var removed = new List<DeviceRemovalEntity>();
var skipped = new List<DeviceRemovalSkipDto>();
if (clientIds.Count == 0)
{
return (removed, skipped);
}
// Read while the rows exist: a dynamic group's rules need them. Group ids
// are what lets a scoped role keep seeing the removals of its devices.
var groupsOf = await EffectiveGroupsAsync(clientIds, cancellationToken);
foreach (var clientId in clientIds)
{
if (await TryRemoveOneAsync(clientId, groupsOf.GetValueOrDefault(clientId), note, requestedBy, cancellationToken) is { } row)
{
removed.Add(row);
}
else
{
skipped.Add(new DeviceRemovalSkipDto(clientId, null, SkipUnknown));
}
}
return (removed, skipped);
}
private async Task<Dictionary<string, List<Guid>>> EffectiveGroupsAsync(
IReadOnlyCollection<string> clientIds, CancellationToken cancellationToken)
{
var result = new Dictionary<string, List<Guid>>(StringComparer.Ordinal);
if (!await db.DeviceGroups.AnyAsync(cancellationToken))
{
return result;
}
var wanted = clientIds.ToHashSet(StringComparer.Ordinal);
var membership = await groups.GetMembershipAsync(cancellationToken);
foreach (var group in membership.Groups)
{
foreach (var clientId in membership.Effective(group.Id).Where(wanted.Contains))
{
if (!result.TryGetValue(clientId, out var list))
{
result[clientId] = list = [];
}
list.Add(group.Id);
}
}
return result;
}
private async Task<DeviceRemovalEntity?> TryRemoveOneAsync(
string clientId, List<Guid>? groupIds, string? note, string? requestedBy, CancellationToken cancellationToken)
{
for (var attempt = 0; attempt < 3; attempt++)
{
var device = await db.Devices.SingleOrDefaultAsync(d => d.ClientId == clientId, cancellationToken);
if (device is null)
{
return null;
}
var now = DateTimeOffset.UtcNow;
var row = new DeviceRemovalEntity
{
Id = Guid.NewGuid(),
ClientId = clientId,
NkeyPublic = device.NkeyPublic,
EnrollmentId = device.EnrollmentId,
Hostname = device.Hostname,
OsCaption = device.OsCaption,
AgentVersion = device.AgentVersion,
State = DeviceRemovalStates.Waiting,
RequestedBy = requestedBy,
Note = note,
RequestedUtc = now,
UpdatedUtc = now,
LastSeenUtc = device.LastSeenUtc,
GroupIdsJson = groupIds is { Count: > 0 } ? JsonDefaults.Serialize(groupIds) : null,
};
db.DeviceRemovals.Add(row);
await StageDependentsAsync(clientId, now, cancellationToken);
db.Devices.Remove(device);
DeviceHistory.RecordRemovalRequested(db, clientId, requestedBy, note);
try
{
await db.SaveChangesAsync(cancellationToken);
return row;
}
catch (DbUpdateConcurrencyException)
{
// A schedule fired or the device went meanwhile: read it all again.
db.ChangeTracker.Clear();
}
}
logger.LogWarning("Device {ClientId} kept changing; its removal was left out", clientId);
return null;
}
/// <summary>
/// What references the client id without a foreign key. Group memberships
/// would attach to the device again if the id returns; device-specific
/// schedules would keep firing (disabled, as when their group is deleted);
/// trigger runs would stop on-registration triggers firing after a legitimate
/// re-enrollment. Jobs that never started would sit queued forever. Finished
/// jobs, their output and the audit trail stay as history.
/// </summary>
private async Task StageDependentsAsync(string clientId, DateTimeOffset now, CancellationToken cancellationToken)
{
db.DeviceGroupMembers.RemoveRange(
await db.DeviceGroupMembers.Where(m => m.ClientId == clientId).ToListAsync(cancellationToken));
db.ScriptTriggerRuns.RemoveRange(
await db.ScriptTriggerRuns.Where(r => r.ClientId == clientId).ToListAsync(cancellationToken));
foreach (var trigger in await db.ScriptTriggers.Where(t => t.ClientId == clientId && t.Enabled).ToListAsync(cancellationToken))
{
trigger.Enabled = false;
trigger.LastResult = "Disabled: its target device was removed.";
}
var unstarted = await db.Jobs
.Where(j => j.ClientId == clientId && (j.State == JobState.Queued || j.State == JobState.Dispatched))
.ToListAsync(cancellationToken);
foreach (var job in unstarted)
{
job.State = JobState.Cancelled;
job.CompletedUtc = now;
job.OutputSummary = "Cancelled: the device was removed.";
}
}
/// <summary>The newest removal of a client id, tracked; older ones are history only.</summary>
public Task<DeviceRemovalEntity?> LatestAsync(string clientId, CancellationToken cancellationToken)
=> db.DeviceRemovals
.Where(r => r.ClientId == clientId)
.OrderByDescending(r => r.RequestedUtc)
.FirstOrDefaultAsync(cancellationToken);
/// <summary>
/// Presence, inventory or a ready notice from a client id with no device
/// row. An open removal notes the sighting (the scheduler sends on it), so
/// the device row is never resurrected. An agent still talking after its
/// removal closed as removed was not gone after all: back to waiting, with
/// its command consumer restored when the next command is sent. A stopped
/// removal stays stopped. True when a removal took the sighting.
/// </summary>
public async Task<bool> NoteSightingAsync(
string? subjectClientId, string clientId, string? agentVersion, CancellationToken cancellationToken)
{
if (subjectClientId is not null && !string.Equals(subjectClientId, clientId, StringComparison.Ordinal))
{
return false;
}
var row = await LatestAsync(clientId, cancellationToken);
if (row is null || row.State == DeviceRemovalStates.Stopped)
{
return false;
}
var now = DateTimeOffset.UtcNow;
var reopened = row.State == DeviceRemovalStates.Removed;
row.LastSeenUtc = now;
row.AgentVersion = string.IsNullOrWhiteSpace(agentVersion) ? row.AgentVersion : agentVersion;
if (reopened)
{
row.State = DeviceRemovalStates.Waiting;
row.FinishedUtc = null;
row.UpdatedUtc = now;
row.Detail = "The device came back after it was marked as removed.";
}
try
{
await db.SaveChangesAsync(cancellationToken);
}
catch (DbUpdateConcurrencyException)
{
// The scheduler or an event got there first; the next sighting is seconds away.
db.ChangeTracker.Clear();
return true;
}
if (reopened)
{
await AnnounceAsync(row, cancellationToken);
}
return true;
}
/// <summary>
/// Applies an agent's removal event. Ignored: unknown ids, another device's
/// removal, states only the server sets, and anything after the removal
/// closed. True when the row changed.
/// </summary>
public async Task<bool> ApplyEventAsync(
string? subjectClientId, AgentRemovalEventPayload payload, CancellationToken cancellationToken)
{
if (!SubjectIdentifier.IsValid(payload.ClientId)
|| (subjectClientId is not null && !string.Equals(subjectClientId, payload.ClientId, StringComparison.Ordinal))
|| !AgentStates.Contains(payload.State))
{
return false;
}
var openStates = DeviceRemovalStates.Open.ToList();
for (var attempt = 0; attempt < 3; attempt++)
{
// A removal the agent started from a refused renewal may not know its id; its device names it.
var row = payload.RemovalId != Guid.Empty
? await db.DeviceRemovals.SingleOrDefaultAsync(r => r.Id == payload.RemovalId, cancellationToken)
: await db.DeviceRemovals
.Where(r => r.ClientId == payload.ClientId && openStates.Contains(r.State))
.OrderByDescending(r => r.RequestedUtc)
.FirstOrDefaultAsync(cancellationToken);
if (row is null
|| !string.Equals(row.ClientId, payload.ClientId, StringComparison.Ordinal)
|| !DeviceRemovalStates.Open.Contains(row.State))
{
return false;
}
var now = DateTimeOffset.UtcNow;
row.State = payload.State;
row.Reason = Truncate(payload.Reason, ReasonMaxLength);
row.Detail = Truncate(payload.Detail, DetailMaxLength);
row.UpdatedUtc = now;
row.LastSeenUtc = now;
try
{
await db.SaveChangesAsync(cancellationToken);
}
catch (DbUpdateConcurrencyException)
{
db.ChangeTracker.Clear();
continue;
}
await AnnounceAsync(row, cancellationToken);
return true;
}
logger.LogWarning("Removal {RemovalId} kept changing; event {State} dropped", payload.RemovalId, payload.State);
return false;
}
/// <summary>Tells consoles a removal changed, with a line for the activity feed.</summary>
public ValueTask AnnounceAsync(DeviceRemovalEntity row, CancellationToken cancellationToken)
=> broadcaster.BroadcastActivityAsync(new ActivityEventPayload(
ActivityKind, Describe(row), row.ClientId, null, DateTimeOffset.UtcNow), cancellationToken);
private static string Describe(DeviceRemovalEntity row)
{
var name = row.Hostname ?? row.ClientId;
return row.State switch
{
DeviceRemovalStates.Sent => $"{name} remove command sent",
DeviceRemovalStates.Uninstalling => $"{name} agent uninstalling",
DeviceRemovalStates.Removed => $"{name} agent removed",
DeviceRemovalStates.Failed => $"{name} agent could not uninstall ({row.Reason ?? "unknown"})",
DeviceRemovalStates.Stopped => $"{name} removal stopped",
_ => $"{name} removal waiting for the device",
};
}
private static string? Truncate(string? value, int length)
=> value is not null && value.Length > length ? value[..length] : value;
/// <summary>Names of removed devices, for what outlives them (old runs, old upgrades); the newest removal wins.</summary>
public static async Task<Dictionary<string, string>> RemovedHostnamesAsync(
AppDbContext db, IEnumerable<string> clientIds, CancellationToken cancellationToken)
{
var ids = clientIds.Distinct(StringComparer.Ordinal).ToList();
if (ids.Count == 0)
{
return new Dictionary<string, string>(StringComparer.Ordinal);
}
var rows = await db.DeviceRemovals.AsNoTracking()
.Where(r => ids.Contains(r.ClientId) && r.Hostname != null)
.OrderBy(r => r.RequestedUtc)
.Select(r => new { r.ClientId, r.Hostname })
.ToListAsync(cancellationToken);
var names = new Dictionary<string, string>(StringComparer.Ordinal);
foreach (var row in rows)
{
names[row.ClientId] = row.Hostname!;
}
return names;
}
/// <summary>
/// True when the caller's roles may see this removal: an unscoped role sees
/// all, a scoped role those of devices that were in one of its scope groups
/// when they were removed.
/// </summary>
public static bool IsVisibleTo(DeviceRemovalEntity row, IReadOnlyList<Guid>? scopeGroupIds)
{
if (scopeGroupIds is null)
{
return true;
}
if (string.IsNullOrEmpty(row.GroupIdsJson))
{
return false;
}
return (JsonDefaults.Deserialize<List<Guid>>(row.GroupIdsJson) ?? []).Intersect(scopeGroupIds).Any();
}
}
/// <summary>Removal events on the agent event subject (MessageTypes.AgentRemoval).</summary>
public static class DeviceRemovalEvents
{
public static async Task HandleAsync(
IServiceScopeFactory scopeFactory, string? subject, MessageEnvelope envelope, CancellationToken cancellationToken)
{
var payload = envelope.GetPayload<AgentRemovalEventPayload>();
if (payload is null)
{
return;
}
using var scope = scopeFactory.CreateScope();
await scope.ServiceProvider.GetRequiredService<DeviceRemovalService>()
.ApplyEventAsync(AgentUpgradeEvents.SubjectClientId(subject), payload, cancellationToken);
}
/// <summary>A sighting from a client id with no device row; <paramref name="agentVersion"/> comes with presence.</summary>
public static Task NoteSightingAsync(
IServiceProvider services, string? subject, string clientId, string? agentVersion, CancellationToken cancellationToken)
=> services.GetRequiredService<DeviceRemovalService>()
.NoteSightingAsync(AgentUpgradeEvents.SubjectClientId(subject), clientId, agentVersion, cancellationToken);
}
+11 -2
View File
@@ -240,13 +240,22 @@ public static class AgentUpgradeEndpoints
return Results.Ok(AgentUpgradeService.ToDto(row, hostname));
}
private static Task<Dictionary<string, string>> HostnamesAsync(
private static async Task<Dictionary<string, string>> HostnamesAsync(
AppDbContext db, IEnumerable<string> clientIds, CancellationToken cancellationToken)
{
var ids = clientIds.Distinct(StringComparer.Ordinal).ToList();
return db.Devices.AsNoTracking()
var hostnames = await db.Devices.AsNoTracking()
.Where(d => ids.Contains(d.ClientId))
.ToDictionaryAsync(d => d.ClientId, d => d.Hostname, cancellationToken);
// Old upgrades of a removed device keep its name.
foreach (var (clientId, hostname) in await Removals.DeviceRemovalService.RemovedHostnamesAsync(
db, ids.Where(id => !hostnames.ContainsKey(id)), cancellationToken))
{
hostnames[clientId] = hostname;
}
return hostnames;
}
private static async Task<IResult> GetSettingsAsync(
@@ -237,6 +237,16 @@ public sealed class AgentUpgradeScheduler(
continue;
}
// Queued before the device moved past the minimum version, or by a
// request that raced it: an older agent could not renew its credential.
if (AgentUpgradeService.IsCredentialDowngrade(runningVersion(row.ClientId), row.TargetVersion))
{
upgrades.Finish(row, AgentUpgradeStates.Cancelled, AgentUpgradeService.SkipCredentialDowngrade,
$"The device runs {runningVersion(row.ClientId)}, past the oldest version that renews its credential.", now);
await TrySaveRowAsync(db, row, cancellationToken);
continue;
}
// Mark it sent first: a row is never handed to JetStream twice.
row.State = AgentUpgradeStates.Sent;
row.SentUtc = now;
@@ -26,6 +26,20 @@ public sealed class AgentUpgradeService(
public const string SkipOrphaned = SkipReasons.Orphaned;
public const string SkipNoPrevious = "no-previous";
/// <summary>
/// The target is older than the first agent that renews its credential, and
/// the device already holds a credential that expires: it would strand within a day.
/// </summary>
public const string SkipCredentialDowngrade = "credential-downgrade";
/// <summary>
/// True when a device on <paramref name="running"/> must not go to
/// <paramref name="target"/>: exactly the minimum version still renews.
/// </summary>
public static bool IsCredentialDowngrade(string? running, string? target)
=> AgentVersions.Compare(running, DeviceRemovalDefaults.MinimumAgentVersion) >= 0
&& AgentVersions.Compare(target, DeviceRemovalDefaults.MinimumAgentVersion) < 0;
/// <summary>Detail of a row the scheduler closed because presence already showed the target version.</summary>
public const string PresenceConfirmed = "Confirmed by presence: the device reports the target version.";
@@ -181,6 +195,10 @@ public sealed class AgentUpgradeService(
// Already there (or, without an explicit version, ahead of the newest held).
skip = AgentUpgradeReasons.Already;
}
else if (IsCredentialDowngrade(device.AgentVersion, package.Version))
{
skip = SkipCredentialDowngrade;
}
}
if (skip is not null)
@@ -231,6 +249,10 @@ public sealed class AgentUpgradeService(
{
skip = AgentUpgradeReasons.Already;
}
else if (IsCredentialDowngrade(device.AgentVersion, from))
{
skip = SkipCredentialDowngrade;
}
else if ((package = await catalog.FindAsync(from, types[device.ClientId]!, cancellationToken)) is null)
{
skip = SkipNoPrevious;
+12
View File
@@ -83,6 +83,11 @@ public static class ApiRoutes
/// <summary>POST: cancel an upgrade that has not been sent yet.</summary>
public const string AgentUpgradeCancel = "/api/agent-upgrades/{id}/cancel";
public const string AgentUpgradeSettings = "/api/settings/agent-upgrades";
/// <summary>GET (?state=, ?clientId=, ?limit=): removals, newest first. POST: remove devices.</summary>
public const string DeviceRemovals = "/api/device-removals";
/// <summary>POST: stop trying to uninstall; the device stays blocked.</summary>
public const string DeviceRemovalStop = "/api/device-removals/{id}/stop";
/// <summary>GET (anonymous, ranges): one catalog package by SHA-256; what an upgrade command's DownloadUrl points at.</summary>
public const string AgentPackageBySha = "/api/agent-packages/{sha256}";
@@ -99,6 +104,10 @@ public static class ApiRoutes
public const string ExecutionOutputDownload = "/api/executions/{runId}/output/download";
/// <summary>POST: a zip with one log per chosen job of the run (all when none are named).</summary>
public const string ExecutionOutputArchive = "/api/executions/{runId}/output/archive";
/// <summary>POST: cancels the run's jobs that have not been sent yet (in the caller's scope).</summary>
public const string ExecutionCancelWaiting = "/api/executions/{runId}/cancel-waiting";
/// <summary>Exit codes, grouped errors and completions over time of the run's jobs in the caller's scope.</summary>
public const string ExecutionBreakdown = "/api/executions/{runId}/breakdown";
public const string Dashboard = "/api/dashboard";
@@ -128,6 +137,7 @@ public static class ApiRoutes
public static string Device(string clientId) => $"/api/devices/{Uri.EscapeDataString(clientId)}";
public static string DeviceInventory(string clientId) => $"/api/devices/{Uri.EscapeDataString(clientId)}/inventory";
public static string DeviceHistory(string clientId) => $"/api/devices/{Uri.EscapeDataString(clientId)}/history";
public static string DeviceRemovalStop(Guid id) => $"/api/device-removals/{id:D}/stop";
public static string AgentVersion(string version) => $"/api/agent-versions/{Uri.EscapeDataString(version)}";
public static string AgentUpgradeCancel(Guid id) => $"/api/agent-upgrades/{id:D}/cancel";
public static string AgentPackageBySha(string sha256) => $"/api/agent-packages/{Uri.EscapeDataString(sha256)}";
@@ -150,6 +160,8 @@ public static class ApiRoutes
public static string Execution(Guid runId) => $"/api/executions/{runId:D}";
public static string ExecutionOutputDownload(Guid runId) => $"/api/executions/{runId:D}/output/download";
public static string ExecutionOutputArchive(Guid runId) => $"/api/executions/{runId:D}/output/archive";
public static string ExecutionCancelWaiting(Guid runId) => $"/api/executions/{runId:D}/cancel-waiting";
public static string ExecutionBreakdown(Guid runId) => $"/api/executions/{runId:D}/breakdown";
public static string AdminBootstrapToken(Guid id) => $"/api/admin/bootstrap-tokens/{id:D}";
public static string AdminRole(Guid id) => $"/api/admin/roles/{id:D}";
public static string AdminUser(Guid id) => $"/api/admin/users/{id:D}";
@@ -0,0 +1,43 @@
namespace Shared.Contracts.Api;
/// <summary>
/// POST /api/device-removals: remove these devices and every device in these
/// groups (nested groups and folders included). Each leaves the system at once;
/// its agent is uninstalled now if online, or when it next returns.
/// </summary>
public sealed record RemoveDevicesRequest(
IReadOnlyList<string>? ClientIds = null,
IReadOnlyList<Guid>? GroupIds = null,
string? Reason = null);
/// <summary>
/// One device's removal. <c>ShowInDeviceList</c> is true for the first
/// <c>DeviceRemovalDefaults.VisibleInDeviceList</c> after <c>UpdatedUtc</c>; after
/// that only the Removals filter shows it.
/// </summary>
public sealed record DeviceRemovalDto(
Guid Id,
string ClientId,
string? Hostname,
string? OsCaption,
string? AgentVersion,
string State,
string? Reason,
string? Detail,
int Attempts,
string? RequestedBy,
string? Note,
DateTimeOffset RequestedUtc,
DateTimeOffset UpdatedUtc,
DateTimeOffset? SentUtc,
DateTimeOffset? LastSeenUtc,
DateTimeOffset? FinishedUtc,
bool ShowInDeviceList);
/// <summary>A device a removal request left out, and why.</summary>
public sealed record DeviceRemovalSkipDto(string ClientId, string? Hostname, string Reason);
/// <summary>What a removal request did.</summary>
public sealed record DeviceRemovalBatchDto(
IReadOnlyList<DeviceRemovalDto> Removals,
IReadOnlyList<DeviceRemovalSkipDto> Skipped);
+64 -11
View File
@@ -85,13 +85,17 @@ public sealed record ScriptTriggerDto(
string? Name = null,
Guid? LastRunId = null,
IReadOnlyDictionary<string, System.Text.Json.JsonElement>? Arguments = null,
string? CreatedBy = null);
string? CreatedBy = null,
int StaggerSeconds = 0,
int StaggerBatchSize = 1);
/// <summary>
/// POST /api/triggers and PUT /api/triggers/{id}. Target: exactly one of
/// ClientId / GroupId, or neither for all devices. For TriggerType=Scheduled,
/// supply CronExpression (recurring) or RunOnceUtc (one-shot) plus an optional
/// TimeZoneId and Arguments.
/// TimeZoneId and Arguments. <c>StaggerSeconds</c> and <c>StaggerBatchSize</c>
/// spread each firing as on <see cref="CreateJobRequest"/>; event triggers fire
/// on one device and ignore them.
/// </summary>
public sealed record SaveScriptTriggerRequest(
Guid ScriptId,
@@ -103,7 +107,9 @@ public sealed record SaveScriptTriggerRequest(
DateTimeOffset? RunOnceUtc = null,
string? TimeZoneId = null,
IReadOnlyDictionary<string, System.Text.Json.JsonElement>? Arguments = null,
string? Name = null);
string? Name = null,
int StaggerSeconds = 0,
int StaggerBatchSize = 1);
/// <summary>POST /api/triggers/preview — upcoming fire times of a cadence, evaluated server-side.</summary>
public sealed record SchedulePreviewRequest(string? CronExpression, string? TimeZoneId, int Count = 5);
@@ -118,8 +124,10 @@ public sealed record TriggerRunResultDto(Guid? RunId, int Queued, string Message
public sealed record SetTriggerEnabledRequest(bool Enabled);
/// <summary>
/// Where every target of a run stands. Targets = jobs + skipped. Pending is
/// queued or sent and not started; Online counts job targets online now.
/// Where every target of a run stands. Targets = jobs + skipped. Waiting is
/// held back by a stagger until its start time; Pending is queued or sent and
/// not started; Failed includes the TimedOut ones; Online counts job targets
/// online now.
/// </summary>
public sealed record RunCountsDto(
int Targets,
@@ -129,7 +137,9 @@ public sealed record RunCountsDto(
int Failed,
int Cancelled,
int Skipped,
int Online);
int Online,
int Waiting = 0,
int TimedOut = 0);
/// <summary>One execution (run) of a script — the per-device jobs it created, summarized.</summary>
public sealed record ExecutionSummaryDto(
@@ -157,12 +167,17 @@ public sealed record ExecutionTargetDto(
DateTimeOffset? StartedUtc,
DateTimeOffset? CompletedUtc,
bool Online = false,
string? Summary = null);
string? Summary = null,
DateTimeOffset? NotBeforeUtc = null);
/// <summary>A targeted device that was given no job: orphaned, or offline with Only Devices Online Now.</summary>
public sealed record ExecutionSkipDto(string ClientId, string Hostname, string Reason);
/// <summary>GET /api/executions/{runId} — a run and the targets it ran on.</summary>
/// <summary>
/// GET /api/executions/{runId} — a run and the targets it ran on.
/// <c>ServerTimeUtc</c> lets a console count down to each target's
/// <c>NotBeforeUtc</c> whatever its own clock says.
/// </summary>
public sealed record ExecutionDetailDto(
Guid RunId,
Guid ScriptId,
@@ -174,7 +189,39 @@ public sealed record ExecutionDetailDto(
string? TargetsLabel = null,
bool OnlineOnly = false,
IReadOnlyList<ExecutionSkipDto>? Skipped = null,
RunCountsDto? Counts = null);
RunCountsDto? Counts = null,
DateTimeOffset? ServerTimeUtc = null);
/// <summary>POST /api/executions/{runId}/cancel-waiting — how many waiting jobs were cancelled.</summary>
public sealed record CancelWaitingResultDto(int Cancelled);
/// <summary>How many of a run's finished jobs reported this exit code.</summary>
public sealed record ExitCodeCountDto(int ExitCode, int Count);
/// <summary>A device one error group happened on.</summary>
public sealed record RunErrorSampleDto(Guid JobId, string ClientId, string? Hostname);
/// <summary>
/// Failed jobs that stopped on the same error: its exception type (when the
/// agent reported one) and its first line, with position text and paths cut.
/// <c>Samples</c> holds up to five devices, <c>JobIds</c> every job in the group.
/// </summary>
public sealed record RunErrorGroupDto(
string? Type,
string Message,
int Count,
IReadOnlyList<RunErrorSampleDto> Samples,
IReadOnlyList<Guid> JobIds);
/// <summary>Jobs that finished during one bucket of a run.</summary>
public sealed record RunCompletionBucketDto(DateTimeOffset StartUtc, int Succeeded, int Failed, int TimedOut, int Cancelled);
/// <summary>GET /api/executions/{runId}/breakdown — a run's exit codes, grouped errors and completions over time.</summary>
public sealed record ExecutionBreakdownDto(
IReadOnlyList<ExitCodeCountDto> ExitCodes,
IReadOnlyList<RunErrorGroupDto> Errors,
int BucketSeconds,
IReadOnlyList<RunCompletionBucketDto> Completions);
/// <summary>Device detail for GET /api/devices/{clientId}.</summary>
public sealed record DeviceDetailDto(
@@ -296,7 +343,11 @@ public sealed record JobDto(
/// (<c>GroupId</c>/<c>GroupIds</c>), deduplicated into one execution. A lone
/// <c>ClientId</c> answers with the single <see cref="JobDto"/>; anything else
/// with a <see cref="GroupRunDto"/>. <c>OnlineOnly</c> skips devices that are
/// not online now instead of queueing for their return.
/// not online now instead of queueing for their return. <c>StaggerSeconds</c>
/// (0–86400; 0 = all at once) spreads the starts over that long, in batches of
/// <c>StaggerBatchSize</c> devices (1–10000): the first batch starts now and
/// the last when the spread ends. With OnlineOnly, a device is checked again
/// when its batch comes up.
/// </summary>
public sealed record CreateJobRequest(
string? ClientId,
@@ -305,7 +356,9 @@ public sealed record CreateJobRequest(
Guid? GroupId = null,
IReadOnlyList<Guid>? GroupIds = null,
IReadOnlyList<string>? ClientIds = null,
bool OnlineOnly = false);
bool OnlineOnly = false,
int StaggerSeconds = 0,
int StaggerBatchSize = 1);
/// <summary>
/// POST /api/executions/{runId}/output/archive request body: the run's jobs to
@@ -0,0 +1,23 @@
namespace Shared.Contracts.Messages;
/// <summary>
/// Removes the agent from the device; durable command on
/// <c>agent.&lt;clientId&gt;.command.remove</c>. The agent acts only when
/// <paramref name="EnrollmentId"/> is the enrollment it holds, so a stale
/// command can never uninstall a device that was enrolled again since.
/// </summary>
public sealed record AgentRemoveCommandPayload(Guid RemovalId, Guid EnrollmentId);
/// <summary>
/// Progress of a removal, published by the agent on
/// <c>agent.&lt;clientId&gt;.event</c> with message type <see cref="MessageTypes.AgentRemoval"/>.
/// <paramref name="State"/> is uninstalling or failed (<c>DeviceRemovalStates</c>):
/// after a successful uninstall nobody is left to say so.
/// </summary>
public sealed record AgentRemovalEventPayload(
Guid RemovalId,
string ClientId,
string State,
string? Reason,
string? Detail,
DateTimeOffset TimestampUtc);
@@ -11,11 +11,14 @@ public static class MessageTypes
public const string AgentEvent = "agent.event";
/// <summary>Upgrade progress or outcome on the agent event subject (<see cref="AgentUpgradeEventPayload"/>).</summary>
public const string AgentUpgrade = "agent.upgrade";
/// <summary>Removal progress on the agent event subject (<see cref="AgentRemovalEventPayload"/>).</summary>
public const string AgentRemoval = "agent.removal";
public const string CommandInventory = "command.inventory";
public const string CommandScript = "command.script";
public const string CommandFileDownload = "command.file.download";
public const string CommandAgentUpgrade = "command.agent.upgrade";
public const string CommandAgentRemove = "command.agent.remove";
public const string CommandShellOpen = "command.shell.open";
public const string CommandShellInput = "command.shell.input";
public const string CommandShellResize = "command.shell.resize";
+12 -2
View File
@@ -127,14 +127,24 @@ public sealed record JobProgressPayload(
int? SecondsRemaining,
DateTimeOffset TimestampUtc);
/// <summary>Job state transition on <c>job.&lt;jobId&gt;.state</c>.</summary>
/// <summary>
/// Job state transition on <c>job.&lt;jobId&gt;.state</c>. A failed job says
/// which error stopped it: the exception's full type name, the error record's
/// fully qualified id and the message (older agents omit all three). The
/// backend adds <paramref name="RunId"/> when relaying this to browsers, so a
/// run page can pick out its own jobs; agents never send it.
/// </summary>
public sealed record JobStatePayload(
Guid JobId,
string ClientId,
JobState State,
int? ExitCode,
DateTimeOffset TimestampUtc,
string? Message);
string? Message,
string? ErrorType = null,
string? ErrorId = null,
string? ErrorMessage = null,
Guid? RunId = null);
/// <summary>
/// Opens a remote shell; sent on <c>agent.&lt;clientId&gt;.command.shell.open</c>.
@@ -0,0 +1,108 @@
namespace Shared.Contracts.Models;
/// <summary>
/// States of one device's removal. The device row is gone from the first moment;
/// this record stands in for it until the agent has gone, and stays afterwards
/// as the history. A removal waits for as long as the device stays away.
/// </summary>
public static class DeviceRemovalStates
{
/// <summary>Waiting for the device to come back (or for the next send).</summary>
public const string Waiting = "waiting";
/// <summary>The remove command was sent; no answer yet.</summary>
public const string Sent = "sent";
/// <summary>The agent accepted and is uninstalling (it told us, or its credential renewal was refused).</summary>
public const string Uninstalling = "uninstalling";
/// <summary>The agent has gone: it uninstalled, or went silent for good after saying it was uninstalling.</summary>
public const string Removed = "removed";
/// <summary>The uninstall failed on the device; the agent says why and the server asks again when it next returns.</summary>
public const string Failed = "failed";
/// <summary>An operator stopped trying. The device stays blocked; its agent stays installed.</summary>
public const string Stopped = "stopped";
/// <summary>Not finished: the scheduler still acts on these.</summary>
public static readonly IReadOnlySet<string> Open = new HashSet<string>(StringComparer.Ordinal)
{ Waiting, Sent, Uninstalling, Failed };
}
/// <summary>Why a removal failed, or what the agent could not do.</summary>
public static class DeviceRemovalReasons
{
/// <summary>The agent is too old to take a remove command: uninstall it by hand, or upgrade it first.</summary>
public const string AgentTooOld = "agent-too-old";
/// <summary>The agent cannot remove itself here (foreground or development run, container).</summary>
public const string Unsupported = "unsupported";
/// <summary>An agent upgrade is in progress; the server asks again.</summary>
public const string Busy = "busy";
/// <summary>Windows Installer failed to uninstall the agent.</summary>
public const string Msi = "msi";
/// <summary>Files or services could not be removed.</summary>
public const string Uninstall = "uninstall";
/// <summary>The command's enrollment is not the one this agent holds (a stale command); it was ignored.</summary>
public const string StaleEnrollment = "stale-enrollment";
}
/// <summary>Timings shared by the server's removal scheduler and the console.</summary>
public static class DeviceRemovalDefaults
{
/// <summary>
/// How long a removal shows in the default Devices list after its last
/// change. After that only the Removals status filter shows it, so the main
/// list stays uncluttered while nothing is lost.
/// </summary>
public static readonly TimeSpan VisibleInDeviceList = TimeSpan.FromDays(7);
/// <summary>Silence after "uninstalling" that counts as gone: nobody is left to report success.</summary>
public static readonly TimeSpan UninstallSilence = TimeSpan.FromMinutes(15);
/// <summary>The first agent release that understands the remove command and renews its credential.</summary>
public const string MinimumAgentVersion = "1.1.0";
}
/// <summary>
/// The agent's lifetime on its NATS credential. It renews over HTTPS at half
/// life and at every start (older non-expiring credentials are replaced then),
/// so a removed device is cut off within a day: renewal answers 410 Gone.
/// </summary>
public static class AgentCredentialDefaults
{
public static readonly TimeSpan Lifetime = TimeSpan.FromHours(24);
/// <summary>Problem type of the 410 answer: the agent then uninstalls itself.</summary>
public const string RemovedProblemType = "urn:problem:device-removed";
}
/// <summary>
/// A removal handed from the agent to the watchdog, in
/// <c>&lt;data&gt;/upgrade/removal.json</c>. The watchdog runs it from a detached
/// copy of itself outside the install and data folders, since it wipes both.
/// </summary>
public sealed record WatchdogRemoval(
Guid RemovalId,
string InstallKind,
string Phase,
int Attempts,
string? LastError,
DateTimeOffset RequestedUtc,
string? RunnerDirectory = null);
/// <summary>How a removal runs on the device, shared by the agent, the watchdog and the runner.</summary>
public static class WatchdogRemovalFiles
{
/// <summary>Hidden watchdog verb of the detached runner: <c>remove-run &lt;id&gt; --install-dir … --data-dir … --runner-dir …</c>.</summary>
public const string RunVerb = "remove-run";
/// <summary>The same runner in the agent's binary, for a device where no watchdog can take the removal.</summary>
public const string AgentRunVerb = "remove-self";
/// <summary>In <see cref="WatchdogRemoval.RunnerDirectory"/>: held open (no sharing) by the runner for as long as it runs.</summary>
public const string RunnerLock = "removal.lock";
/// <summary>Attempts at uninstalling before the removal ends failed.</summary>
public const int MaxAttempts = 3;
public static readonly TimeSpan AttemptDelay = TimeSpan.FromMinutes(5);
/// <summary>A removal the watchdog has not taken by then: the agent starts the runner itself.</summary>
public static readonly TimeSpan PickupDeadline = TimeSpan.FromMinutes(2);
}
/// <summary>Phases of <see cref="WatchdogRemoval"/>.</summary>
public static class WatchdogRemovalPhases
{
public const string Requested = "requested";
public const string Uninstalling = "uninstalling";
public const string Failed = "failed";
}
@@ -29,6 +29,8 @@ public static class WatchdogFiles
/// <summary>MSI upgrades: the runner's journal (<see cref="WatchdogMsiJournal"/>), beside handover.json.</summary>
public const string MsiJournal = "msi.json";
/// <summary>A removal handed to the watchdog (<see cref="WatchdogRemoval"/>), beside handover.json.</summary>
public const string Removal = "removal.json";
/// <summary>MSI upgrades: <c>upgrade/msi/</c> holds the MSIs being installed and the runner's copy of the watchdog binary.</summary>
public const string MsiFolder = "msi";
/// <summary>
@@ -38,6 +40,12 @@ public static class WatchdogFiles
/// later upgrades brought; the newest 3 are kept.
/// </summary>
public const string KeptPackages = "packages";
/// <summary>
/// Beside <c>credentials.dat</c>, one thumbprint per line: the server CA
/// certificates the agent itself added to the machine's trust store, the only
/// ones a removal takes out (the same CA may be there for the server's own sake).
/// </summary>
public const string ManagedCa = "managed-ca.txt";
public const int KeptPackagesMax = 3;
/// <summary>The longest an MSI install (or the reinstall of the previous one) may run.</summary>
public static readonly TimeSpan MsiInstallTimeout = TimeSpan.FromMinutes(30);
@@ -20,4 +20,5 @@ namespace Shared.Contracts.Serialization;
[JsonSerializable(typeof(WatchdogJournal))]
[JsonSerializable(typeof(WatchdogOutcome))]
[JsonSerializable(typeof(WatchdogMsiJournal))]
[JsonSerializable(typeof(WatchdogRemoval))]
public sealed partial class WatchdogJsonContext : JsonSerializerContext;
@@ -15,6 +15,7 @@ public static class JetStreamDefaults
"agent.*.command.inventory",
"agent.*.command.file",
"agent.*.command.upgrade",
"agent.*.command.remove",
];
public const string JobEventsStreamName = "JOB_EVENTS";
@@ -18,6 +18,7 @@ public static class NatsSubjects
public static string AgentCommandScript(string clientId) => $"agent.{SubjectIdentifier.Require(clientId)}.command.script";
public static string AgentCommandFile(string clientId) => $"agent.{SubjectIdentifier.Require(clientId)}.command.file";
public static string AgentCommandUpgrade(string clientId) => $"agent.{SubjectIdentifier.Require(clientId)}.command.upgrade";
public static string AgentCommandRemove(string clientId) => $"agent.{SubjectIdentifier.Require(clientId)}.command.remove";
public static string AgentCommandShellOpen(string clientId) => $"agent.{SubjectIdentifier.Require(clientId)}.command.shell.open";
public static string AgentCommandShellInput(string clientId) => $"agent.{SubjectIdentifier.Require(clientId)}.command.shell.input";
public static string AgentCommandShellResize(string clientId) => $"agent.{SubjectIdentifier.Require(clientId)}.command.shell.resize";
+11
View File
@@ -10,6 +10,8 @@ import type {
DashboardDto,
DeviceDetailDto,
DeviceDto,
DeviceRemovalBatchDto,
DeviceRemovalDto,
DeviceGroupDto,
ExecutionDetailDto,
ExecutionSummaryDto,
@@ -37,6 +39,7 @@ import type {
SecretsProviderDto,
SecretsProvidersResponse,
SecretsTestResultDto,
RemoveDevicesBody,
ServerInfoDto,
StartAgentUpgradeBody,
TriggerRunResultDto
@@ -206,6 +209,9 @@ const enc = encodeURIComponent;
const upgradeQuery = ({ clientId, open, limit }: AgentUpgradeQuery) => qs({ clientId, open: open ? 'true' : undefined, limit });
/** Every removal the page may show (the API's upper limit): the Devices list and its Removals filter share this read. */
export const removalsKey = '/api/device-removals?limit=5000';
/** Every REST call the console makes. Reads also have SWR keys (see `keys`). */
export const api = {
// Devices
@@ -227,6 +233,8 @@ export const api = {
rollBackAgent: (body: RollBackAgentBody) => request<AgentUpgradeBatchDto>('/api/agent-upgrades/rollback', json('POST', body)),
/** Only an upgrade still queued can be cancelled. */
cancelAgentUpgrade: (id: string) => request<void>(`/api/agent-upgrades/${enc(id)}/cancel`, { method: 'POST' }),
removeDevices: (body: RemoveDevicesBody) => request<DeviceRemovalBatchDto>('/api/device-removals', json('POST', body)),
stopDeviceRemoval: (id: string) => request<DeviceRemovalDto>(`/api/device-removals/${enc(id)}/stop`, { method: 'POST' }),
agentUpgradeSettings: () => request<AgentUpgradeSettingsDto>('/api/settings/agent-upgrades'),
saveAgentUpgradeSettings: (body: AgentUpgradeSettingsDto) =>
request<AgentUpgradeSettingsDto | undefined>('/api/settings/agent-upgrades', json('PUT', body)),
@@ -257,6 +265,8 @@ export const api = {
executions: (limit = 50) => request<ExecutionSummaryDto[]>(`/api/executions?limit=${limit}`),
/** Runs a script on any mix of devices and groups as one execution. */
runScript: (body: RunScriptBody) => request<GroupRunDto>('/api/jobs', json('POST', body)),
/** Cancels the run's jobs that have not been sent yet. */
cancelWaiting: (runId: string) => request<{ cancelled: number }>(`/api/executions/${runId}/cancel-waiting`, json('POST', {})),
// Groups
groups: () => request<DeviceGroupDto[]>('/api/groups'),
@@ -382,6 +392,7 @@ export const keys = {
jobs: (params: { clientId?: string; scriptId?: string; limit?: number } = {}) => `/api/jobs${qs(params)}`,
scriptExecutions: (scriptId: string) => `/api/scripts/${scriptId}/executions`,
execution: (runId: string) => `/api/executions/${runId}`,
executionBreakdown: (runId: string) => `/api/executions/${runId}/breakdown`,
executions: (limit = 50) => `/api/executions?limit=${limit}`,
roles: '/api/admin/roles',
permissionCatalog: '/api/admin/permissions'
+97
View File
@@ -230,6 +230,48 @@ export interface AgentUpgradeBatchDto {
skipped: AgentUpgradeSkipDto[];
}
export type DeviceRemovalState = 'waiting' | 'sent' | 'uninstalling' | 'removed' | 'failed' | 'stopped';
/** One device's removal; it stands in for the device row, and stays as the history. */
export interface DeviceRemovalDto {
id: string;
clientId: string;
hostname?: string | null;
osCaption?: string | null;
agentVersion?: string | null;
state: DeviceRemovalState;
reason?: string | null;
detail?: string | null;
attempts: number;
requestedBy?: string | null;
note?: string | null;
requestedUtc: string;
updatedUtc: string;
sentUtc?: string | null;
lastSeenUtc?: string | null;
finishedUtc?: string | null;
showInDeviceList: boolean;
}
export interface RemoveDevicesBody {
clientIds?: string[];
groupIds?: string[];
/** The operator's note. */
reason?: string | null;
}
/** A device a request left out: unknown or out-of-scope. */
export interface DeviceRemovalSkipDto {
clientId: string;
hostname?: string | null;
reason: string;
}
export interface DeviceRemovalBatchDto {
removals: DeviceRemovalDto[];
skipped: DeviceRemovalSkipDto[];
}
export interface AgentUpgradeSettingsDto {
automaticUpgrades: boolean;
/** Queued online devices start downloading within this many minutes (1–30), in paced batches. */
@@ -263,6 +305,9 @@ export interface ScriptTriggerDto {
lastRunId?: string;
arguments?: Record<string, unknown>;
createdBy?: string;
/** Spreads each firing's starts over this long; 0 starts them all at once. */
staggerSeconds?: number;
staggerBatchSize?: number;
}
/** POST /api/triggers and PUT /api/triggers/{id}. Target: clientId OR groupId, or neither for all devices. */
@@ -277,6 +322,8 @@ export interface SaveTriggerBody {
runOnceUtc?: string;
timeZoneId?: string;
arguments?: Record<string, unknown>;
staggerSeconds?: number;
staggerBatchSize?: number;
}
export interface SchedulePreviewDto {
@@ -418,6 +465,10 @@ export interface RunScriptBody {
arguments?: Record<string, unknown>;
/** Skip devices that are not online now instead of queueing for their return. */
onlineOnly?: boolean;
/** Spreads the starts over this long (0–86400); 0 starts them all at once. */
staggerSeconds?: number;
/** Devices started together in each batch of a stagger. */
staggerBatchSize?: number;
}
export interface CountPoint {
@@ -566,11 +617,15 @@ export interface RunCountsDto {
pending: number;
running: number;
succeeded: number;
/** Includes the timed out ones. */
failed: number;
cancelled: number;
skipped: number;
/** Job targets online now. */
online: number;
/** Held back by a stagger until their start. */
waiting?: number;
timedOut?: number;
}
export type RunSource = 'manual' | 'schedule' | 'trigger';
@@ -602,6 +657,8 @@ export interface ExecutionTargetDto {
online?: boolean;
/** Failure or completion summary from the agent. */
summary?: string;
/** When a stagger starts it, while it is queued. */
notBeforeUtc?: string;
}
/** A targeted device given no job. */
@@ -623,6 +680,41 @@ export interface ExecutionDetailDto {
onlineOnly?: boolean;
skipped?: ExecutionSkipDto[];
counts?: RunCountsDto;
/** The server's clock at this answer, to count down whatever the browser's says. */
serverTimeUtc?: string;
}
export interface RunErrorSampleDto {
jobId: string;
clientId: string;
hostname?: string;
}
/** Failed jobs that stopped on the same error. */
export interface RunErrorGroupDto {
/** The exception's full type name, when the agent reported one. */
type?: string;
message: string;
count: number;
/** Up to five devices. */
samples: RunErrorSampleDto[];
jobIds: string[];
}
export interface RunCompletionBucketDto {
startUtc: string;
succeeded: number;
failed: number;
timedOut: number;
cancelled: number;
}
/** GET /api/executions/{runId}/breakdown */
export interface ExecutionBreakdownDto {
exitCodes: { exitCode: number; count: number }[];
errors: RunErrorGroupDto[];
bucketSeconds: number;
completions: RunCompletionBucketDto[];
}
export interface RoleDto {
@@ -728,6 +820,11 @@ export interface JobStatePayload {
exitCode?: number;
timestampUtc: string;
message?: string;
errorType?: string;
errorId?: string;
errorMessage?: string;
/** The run this job belongs to, added by the server. */
runId?: string;
}
export interface ShellOutputPayload {
+4 -1
View File
@@ -60,6 +60,8 @@ interface Props<T> {
rowLabel?: (row: T) => string;
/** Marks the row currently open in a side panel. */
activeRowId?: string;
/** Dims rows that only stand in for something (not selectable, fewer actions). */
dimmedRow?: (row: T) => boolean;
minWidth?: number;
}
@@ -93,6 +95,7 @@ export default function DataTable<T>({
dense = true,
rowLabel,
activeRowId,
dimmedRow,
minWidth = 720
}: Props<T>) {
const [sort, setSort] = useState(defaultSort);
@@ -207,7 +210,7 @@ export default function DataTable<T>({
hover
selected={isChosen || activeRowId === id}
onClick={onRowClick ? () => onRowClick(row) : undefined}
sx={{ cursor: onRowClick ? 'pointer' : 'default' }}
sx={{ cursor: onRowClick ? 'pointer' : 'default', ...(dimmedRow?.(row) && { '& > td': { opacity: 0.65 } }) }}
>
{selection && (
<TableCell padding="checkbox" sx={{ pl: 2 }} onClick={(e) => e.stopPropagation()}>
+1
View File
@@ -15,6 +15,7 @@ const TONES: Record<string, { tone: Tone; label: string }> = {
offline: { tone: 'secondary', label: 'Offline' },
orphaned: { tone: 'error', label: 'Orphaned' },
queued: { tone: 'secondary', label: 'Queued' },
waiting: { tone: 'info', label: 'Waiting' },
dispatched: { tone: 'info', label: 'Dispatched' },
running: { tone: 'primary', label: 'Running' },
succeeded: { tone: 'success', label: 'Succeeded' },
+34 -3
View File
@@ -1,3 +1,5 @@
import { KeyboardEvent } from 'react';
// material-ui
import Box from '@mui/material/Box';
import Stack from '@mui/material/Stack';
@@ -19,12 +21,27 @@ interface Props {
label: string;
/** Unit for the tooltip ("jobs"). */
unit?: string;
/** Makes each row a toggle, e.g. to filter a table. */
onSelect?: (item: CountPoint) => void;
/** The label of the row that is on. */
selected?: string | null;
/** A hue per row, when rows are outcomes rather than one series. */
colorOf?: (item: CountPoint) => string;
}
// ==============================|| CHART - BAR LIST ||============================== //
/** Horizontal bars for ranked counts: label, proportional bar, value. */
export default function BarList({ items, color = 'var(--chart-1)', emptyText = 'No data yet.', label, unit = 'jobs' }: Props) {
export default function BarList({
items,
color = 'var(--chart-1)',
emptyText = 'No data yet.',
label,
unit = 'jobs',
onSelect,
selected,
colorOf
}: Props) {
if (items.length === 0) {
return (
<Typography variant="body2" color="text.secondary">
@@ -40,16 +57,28 @@ export default function BarList({ items, color = 'var(--chart-1)', emptyText = '
<Stack component="ul" aria-label={label} sx={{ listStyle: 'none', m: 0, p: 0, gap: 0.5 }}>
{items.map((item) => {
const share = total > 0 ? Math.round((item.count / total) * 100) : 0;
const fill = colorOf?.(item) ?? color;
return (
<Tooltip
key={item.label}
placement="top"
title={<ChartTooltip heading={item.label} rows={[{ label: `${unit} · ${share}%`, value: item.count, color }]} />}
title={<ChartTooltip heading={item.label} rows={[{ label: `${unit} · ${share}%`, value: item.count, color: fill }]} />}
>
<Box
component="li"
tabIndex={0}
aria-label={`${item.label}: ${item.count}`}
{...(onSelect && {
role: 'button',
'aria-pressed': selected === item.label,
onClick: () => onSelect(item),
onKeyDown: (event: KeyboardEvent) => {
if (event.key === 'Enter' || event.key === ' ') {
event.preventDefault();
onSelect(item);
}
}
})}
sx={(theme) => ({
display: 'grid',
gridTemplateColumns: 'minmax(0, 9rem) 1fr auto',
@@ -59,6 +88,8 @@ export default function BarList({ items, color = 'var(--chart-1)', emptyText = '
px: 0.5,
borderRadius: 1,
outline: 'none',
cursor: onSelect ? 'pointer' : undefined,
bgcolor: selected === item.label ? theme.vars.palette.action.selected : undefined,
'& .bar': { transition: 'opacity 0.15s' },
'&:hover, &:focus-visible': { bgcolor: theme.vars.palette.action.hover },
'&:hover .bar, &:focus-visible .bar': { opacity: 0.8 }
@@ -69,7 +100,7 @@ export default function BarList({ items, color = 'var(--chart-1)', emptyText = '
</Typography>
<Box component="svg" aria-hidden sx={{ width: 1, height: 8, display: 'block' }}>
<Box component="rect" width="100%" height={8} rx={4} sx={(theme) => ({ fill: theme.vars.palette.divider })} />
<rect className="bar" width={`${Math.max((item.count / max) * 100, 1)}%`} height={8} rx={4} style={{ fill: color }} />
<rect className="bar" width={`${Math.max((item.count / max) * 100, 1)}%`} height={8} rx={4} style={{ fill }} />
</Box>
<Typography variant="caption" sx={{ fontWeight: 600, fontVariantNumeric: 'tabular-nums', textAlign: 'right' }}>
{item.count.toLocaleString()}
@@ -0,0 +1,245 @@
import { useEffect, useRef, useState } from 'react';
// material-ui
import { useTheme } from '@mui/material/styles';
import Box from '@mui/material/Box';
import Stack from '@mui/material/Stack';
import Tooltip from '@mui/material/Tooltip';
import Typography from '@mui/material/Typography';
// project imports
import ChartTooltip from './ChartTooltip';
export interface ColumnSeries {
key: string;
label: string;
color: string;
}
export interface ColumnPoint {
/** Stable and unique among the points. */
key: string;
/** Under the column, where there is room. */
label: string;
/** Tooltip heading; the label when not given. */
heading?: string;
values: Record<string, number>;
}
interface Props {
points: ColumnPoint[];
series: ColumnSeries[];
height?: number;
/** Accessible name ("Completions over time"). */
label: string;
emptyText?: string;
/** Column heading of the screen-reader table ("Day"). */
pointHeading?: string;
/** Puts each series' total beside its name in the legend. */
legendTotals?: boolean;
}
const AXIS_HEIGHT = 20;
const TOP_PAD = 14;
/** Surface gap between stacked segments. */
const GAP = 2;
const RADIUS = 4;
/** Off-screen but still read by assistive technology. */
const visuallyHidden = {
position: 'absolute',
width: '1px',
height: '1px',
overflow: 'hidden',
clip: 'rect(0 0 0 0)',
whiteSpace: 'nowrap',
border: 0,
p: 0,
m: '-1px'
} as const;
/** A rect with only its top corners rounded (the data end; the baseline end stays square). */
function topRoundedRect(x: number, y: number, w: number, h: number, r: number) {
const radius = Math.min(r, h, w / 2);
return `M${x},${y + h} V${y + radius} Q${x},${y} ${x + radius},${y} H${x + w - radius} Q${x + w},${y} ${x + w},${y + radius} V${y + h} Z`;
}
/** Tracks an element's content width. */
function useWidth<T extends HTMLElement>() {
const ref = useRef<T>(null);
const [width, setWidth] = useState(0);
useEffect(() => {
const element = ref.current;
if (!element) return;
const observer = new ResizeObserver(([entry]) => setWidth(entry.contentRect.width));
observer.observe(element);
return () => observer.disconnect();
}, []);
return [ref, width] as const;
}
// ==============================|| CHART - STACKED COLUMNS ||============================== //
/** Stacked columns over a sequence (days, minutes of a run), with a per-column hover readout. */
export default function StackedColumns({
points,
series,
height = 150,
label,
emptyText = 'No data yet.',
pointHeading = 'Time',
legendTotals
}: Props) {
const theme = useTheme();
const [ref, width] = useWidth<HTMLDivElement>();
const [hovered, setHovered] = useState<number | null>(null);
const sumOf = (point: ColumnPoint) => series.reduce((sum, s) => sum + (point.values[s.key] ?? 0), 0);
const max = Math.max(1, ...points.map(sumOf));
const total = points.reduce((sum, point) => sum + sumOf(point), 0);
const plotHeight = height - AXIS_HEIGHT - TOP_PAD;
const baseline = TOP_PAD + plotHeight;
const slot = points.length > 0 ? width / points.length : 0;
const barWidth = Math.max(Math.min(slot * 0.62, 28), 2);
const labelStep = Math.max(1, Math.ceil((points.length * 52) / Math.max(width, 1)));
return (
<Stack sx={{ gap: 1 }}>
<Box ref={ref} sx={{ width: 1, position: 'relative' }}>
{total === 0 ? (
<Typography variant="body2" color="text.secondary">
{emptyText}
</Typography>
) : (
width > 0 && (
<svg width={width} height={height} role="img" aria-label={`${label}: ${total} in total`} style={{ display: 'block' }}>
<title>{`${label}: ${total} in total`}</title>
{/* recessive grid: top line labelled with the scale maximum, and the baseline */}
<line
x1={0}
x2={width}
y1={TOP_PAD}
y2={TOP_PAD}
style={{ stroke: theme.vars.palette.divider, strokeDasharray: '3 3', strokeWidth: 1 }}
/>
<text x={width} y={TOP_PAD - 4} textAnchor="end" style={{ fill: theme.vars.palette.text.secondary, fontSize: 10 }}>
{max}
</text>
<line x1={0} x2={width} y1={baseline} y2={baseline} style={{ stroke: theme.vars.palette.divider, strokeWidth: 1 }} />
{points.map((point, i) => {
const x = i * slot + (slot - barWidth) / 2;
let y = baseline;
const stacked = series.filter((s) => (point.values[s.key] ?? 0) > 0);
const segments = stacked.map((s, index) => {
const h = (point.values[s.key] / max) * plotHeight;
const top = y - h;
// Every segment above the first gives up GAP px at its bottom.
const drawn = index > 0 ? Math.max(h - GAP, 0.5) : Math.max(h, 1);
y = top;
return { ...s, top, drawn, last: index === stacked.length - 1 };
});
const readout = (
<ChartTooltip
heading={point.heading ?? point.label}
rows={[
...series.map((s) => ({ label: s.label, value: point.values[s.key] ?? 0, color: s.color })),
{ label: 'Total', value: sumOf(point) }
]}
/>
);
return (
<g key={point.key} style={{ opacity: hovered !== null && hovered !== i ? 0.6 : 1, transition: 'opacity 0.15s' }}>
{segments.map((s) =>
s.last ? (
<path key={s.key} d={topRoundedRect(x, s.top, barWidth, s.drawn, RADIUS)} style={{ fill: s.color }} />
) : (
<rect key={s.key} x={x} y={s.top} width={barWidth} height={s.drawn} style={{ fill: s.color }} />
)
)}
{i % labelStep === 0 && (
<text
// Labels at the edges hang inward instead of being cut off.
x={i * slot + slot / 2 < 26 ? 0 : i * slot + slot / 2 > width - 26 ? width : i * slot + slot / 2}
y={height - 4}
textAnchor={i * slot + slot / 2 < 26 ? 'start' : i * slot + slot / 2 > width - 26 ? 'end' : 'middle'}
style={{ fill: theme.vars.palette.text.secondary, fontSize: 10 }}
>
{point.label}
</text>
)}
{/* the whole column is the hit target */}
<Tooltip title={readout} placement="top" followCursor>
<rect
x={i * slot}
y={0}
width={slot}
height={baseline}
tabIndex={0}
aria-label={`${point.heading ?? point.label}: ${series.map((s) => `${point.values[s.key] ?? 0} ${s.label.toLowerCase()}`).join(', ')}`}
onMouseEnter={() => setHovered(i)}
onMouseLeave={() => setHovered(null)}
onFocus={() => setHovered(i)}
onBlur={() => setHovered(null)}
style={{ fill: 'transparent', outline: 'none' }}
/>
</Tooltip>
</g>
);
})}
</svg>
)
)}
</Box>
{total > 0 && (
<>
<Stack direction="row" sx={{ gap: 2, flexWrap: 'wrap' }}>
{series.map((s) => (
<Stack key={s.key} direction="row" sx={{ alignItems: 'center', gap: 0.75 }}>
<Box sx={{ width: 10, height: 10, borderRadius: '3px', bgcolor: s.color }} />
<Typography variant="caption" color="text.secondary">
{s.label}
</Typography>
{legendTotals && (
<Typography variant="caption" sx={{ fontWeight: 600, fontVariantNumeric: 'tabular-nums' }}>
{points.reduce((sum, point) => sum + (point.values[s.key] ?? 0), 0).toLocaleString()}
</Typography>
)}
</Stack>
))}
</Stack>
{/* the same numbers as a table, for screen readers */}
<Box component="table" sx={visuallyHidden}>
<caption>{label}</caption>
<thead>
<tr>
<th scope="col">{pointHeading}</th>
{series.map((s) => (
<th key={s.key} scope="col">
{s.label}
</th>
))}
</tr>
</thead>
<tbody>
{points.map((point) => (
<tr key={point.key}>
<th scope="row">{point.heading ?? point.label}</th>
{series.map((s) => (
<td key={s.key}>{point.values[s.key] ?? 0}</td>
))}
</tr>
))}
</tbody>
</Box>
</>
)}
</Stack>
);
}
+18 -197
View File
@@ -1,14 +1,8 @@
import { useEffect, useRef, useState } from 'react';
// material-ui
import { useTheme } from '@mui/material/styles';
import Box from '@mui/material/Box';
import Stack from '@mui/material/Stack';
import Tooltip from '@mui/material/Tooltip';
import Typography from '@mui/material/Typography';
// project imports
import ChartTooltip from './ChartTooltip';
import StackedColumns from './StackedColumns';
// types
import type { DayPoint } from 'api/types';
@@ -21,47 +15,6 @@ interface Props {
emptyText?: string;
}
type SeriesKey = 'succeeded' | 'failed' | 'other';
const AXIS_HEIGHT = 20;
const TOP_PAD = 14;
/** Surface gap between stacked segments. */
const GAP = 2;
const RADIUS = 4;
/** Off-screen but still read by assistive technology. */
const visuallyHidden = {
position: 'absolute',
width: '1px',
height: '1px',
overflow: 'hidden',
clip: 'rect(0 0 0 0)',
whiteSpace: 'nowrap',
border: 0,
p: 0,
m: '-1px'
} as const;
/** A rect with only its top corners rounded (the data end; the baseline end stays square). */
function topRoundedRect(x: number, y: number, w: number, h: number, r: number) {
const radius = Math.min(r, h, w / 2);
return `M${x},${y + h} V${y + radius} Q${x},${y} ${x + radius},${y} H${x + w - radius} Q${x + w},${y} ${x + w},${y + radius} V${y + h} Z`;
}
/** Tracks an element's content width. */
function useWidth<T extends HTMLElement>() {
const ref = useRef<T>(null);
const [width, setWidth] = useState(0);
useEffect(() => {
const element = ref.current;
if (!element) return;
const observer = new ResizeObserver(([entry]) => setWidth(entry.contentRect.width));
observer.observe(element);
return () => observer.disconnect();
}, []);
return [ref, width] as const;
}
// ==============================|| CHART - EXECUTIONS TIMELINE ||============================== //
/** Stacked columns of executions per day (succeeded / failed / other) with a per-day hover readout. */
@@ -72,156 +25,24 @@ export default function TimelineChart({
emptyText = 'No executions in the last 14 days.'
}: Props) {
const theme = useTheme();
const [ref, width] = useWidth<HTMLDivElement>();
const [hovered, setHovered] = useState<number | null>(null);
// Status colors: these segments are outcomes, so they wear the status palette.
const series: { key: SeriesKey; label: string; color: string }[] = [
{ key: 'succeeded', label: 'Succeeded', color: theme.vars.palette.success.main },
{ key: 'failed', label: 'Failed', color: theme.vars.palette.error.main },
{ key: 'other', label: 'Other', color: 'var(--chart-6)' }
];
const sumOf = (d: DayPoint) => d.succeeded + d.failed + d.other;
const max = Math.max(1, ...days.map(sumOf));
const total = days.reduce((sum, d) => sum + sumOf(d), 0);
const plotHeight = height - AXIS_HEIGHT - TOP_PAD;
const baseline = TOP_PAD + plotHeight;
const slot = days.length > 0 ? width / days.length : 0;
const barWidth = Math.max(Math.min(slot * 0.62, 28), 2);
const labelStep = Math.max(1, Math.ceil((days.length * 52) / Math.max(width, 1)));
return (
<Stack sx={{ gap: 1 }}>
<Box ref={ref} sx={{ width: 1, position: 'relative' }}>
{total === 0 ? (
<Typography variant="body2" color="text.secondary">
{emptyText}
</Typography>
) : (
width > 0 && (
<svg width={width} height={height} role="img" aria-label={`${label}: ${total} in total`} style={{ display: 'block' }}>
<title>{`${label}: ${total} in total`}</title>
{/* recessive grid: top line labelled with the scale maximum, and the baseline */}
<line
x1={0}
x2={width}
y1={TOP_PAD}
y2={TOP_PAD}
style={{ stroke: theme.vars.palette.divider, strokeDasharray: '3 3', strokeWidth: 1 }}
/>
<text x={width} y={TOP_PAD - 4} textAnchor="end" style={{ fill: theme.vars.palette.text.secondary, fontSize: 10 }}>
{max}
</text>
<line x1={0} x2={width} y1={baseline} y2={baseline} style={{ stroke: theme.vars.palette.divider, strokeWidth: 1 }} />
{days.map((day, i) => {
const x = i * slot + (slot - barWidth) / 2;
let y = baseline;
const stacked = series.filter((s) => day[s.key] > 0);
const segments = stacked.map((s, index) => {
const h = (day[s.key] / max) * plotHeight;
const top = y - h;
// Every segment above the first gives up GAP px at its bottom.
const drawn = index > 0 ? Math.max(h - GAP, 0.5) : Math.max(h, 1);
y = top;
return { ...s, top, drawn, last: index === stacked.length - 1 };
});
const readout = (
<ChartTooltip
heading={day.date}
rows={[
...series.map((s) => ({ label: s.label, value: day[s.key], color: s.color })),
{ label: 'Total', value: sumOf(day) }
]}
/>
);
return (
<g key={day.date} style={{ opacity: hovered !== null && hovered !== i ? 0.6 : 1, transition: 'opacity 0.15s' }}>
{segments.map((s) =>
s.last ? (
<path key={s.key} d={topRoundedRect(x, s.top, barWidth, s.drawn, RADIUS)} style={{ fill: s.color }} />
) : (
<rect key={s.key} x={x} y={s.top} width={barWidth} height={s.drawn} style={{ fill: s.color }} />
)
)}
{i % labelStep === 0 && (
<text
x={i * slot + slot / 2}
y={height - 4}
textAnchor="middle"
style={{ fill: theme.vars.palette.text.secondary, fontSize: 10 }}
>
{day.date}
</text>
)}
{/* the whole column is the hit target */}
<Tooltip title={readout} placement="top" followCursor>
<rect
x={i * slot}
y={0}
width={slot}
height={baseline}
tabIndex={0}
aria-label={`${day.date}: ${day.succeeded} succeeded, ${day.failed} failed, ${day.other} other`}
onMouseEnter={() => setHovered(i)}
onMouseLeave={() => setHovered(null)}
onFocus={() => setHovered(i)}
onBlur={() => setHovered(null)}
style={{ fill: 'transparent', outline: 'none' }}
/>
</Tooltip>
</g>
);
})}
</svg>
)
)}
</Box>
{total > 0 && (
<>
<Stack direction="row" sx={{ gap: 2, flexWrap: 'wrap' }}>
{series.map((s) => (
<Stack key={s.key} direction="row" sx={{ alignItems: 'center', gap: 0.75 }}>
<Box sx={{ width: 10, height: 10, borderRadius: '3px', bgcolor: s.color }} />
<Typography variant="caption" color="text.secondary">
{s.label}
</Typography>
</Stack>
))}
</Stack>
{/* the same numbers as a table, for screen readers */}
<Box component="table" sx={visuallyHidden}>
<caption>{label}</caption>
<thead>
<tr>
<th scope="col">Day</th>
{series.map((s) => (
<th key={s.key} scope="col">
{s.label}
</th>
))}
</tr>
</thead>
<tbody>
{days.map((day) => (
<tr key={day.date}>
<th scope="row">{day.date}</th>
{series.map((s) => (
<td key={s.key}>{day[s.key]}</td>
))}
</tr>
))}
</tbody>
</Box>
</>
)}
</Stack>
<StackedColumns
// Status colors: these segments are outcomes, so they wear the status palette.
series={[
{ key: 'succeeded', label: 'Succeeded', color: theme.vars.palette.success.main },
{ key: 'failed', label: 'Failed', color: theme.vars.palette.error.main },
{ key: 'other', label: 'Other', color: 'var(--chart-6)' }
]}
points={days.map((day) => ({
key: day.date,
label: day.date,
values: { succeeded: day.succeeded, failed: day.failed, other: day.other }
}))}
height={height}
label={label}
emptyText={emptyText}
pointHeading="Day"
/>
);
}
-3
View File
@@ -19,7 +19,6 @@ import useApi from 'hooks/useApi';
import useLiveRefresh from 'hooks/useLiveRefresh';
import { useGate } from 'hooks/useSession';
import ActivityFeed from 'sections/dashboard/ActivityFeed';
import LiveDeployment from 'sections/dashboard/LiveDeployment';
import RecentRuns from 'sections/dashboard/RecentRuns';
import { errorMessage, keys } from 'api/client';
import type { CountPoint, DashboardDto } from 'api/types';
@@ -111,8 +110,6 @@ export default function Dashboard() {
{error && !data && <Alert severity="error">{errorMessage(error, 'The dashboard could not be loaded.')}</Alert>}
<LiveDeployment onFinished={() => void mutate()} />
<Grid container spacing={2}>
<Grid size={{ xs: 12, sm: 6, lg: 'grow' }}>
<KpiCard
+1 -1
View File
@@ -71,7 +71,7 @@ export default function DeviceDetailPage() {
() => mutate(),
(event) => event.clientId === clientId
);
const actions = useDeviceActions({ onDone: () => void mutate() });
const actions = useDeviceActions({ onDone: () => void mutate(), onRemoved: () => navigate('/devices') });
const tab = (TABS.find((t) => t.id === params.get('tab'))?.id ?? 'overview') as TabId;
const device = detail?.device;
+215 -65
View File
@@ -5,6 +5,10 @@ import { Link as RouterLink, useNavigate, useSearchParams } from 'react-router-d
import Box from '@mui/material/Box';
import Button from '@mui/material/Button';
import Chip from '@mui/material/Chip';
import Dialog from '@mui/material/Dialog';
import DialogActions from '@mui/material/DialogActions';
import DialogContent from '@mui/material/DialogContent';
import DialogTitle from '@mui/material/DialogTitle';
import FormControlLabel from '@mui/material/FormControlLabel';
import Grid from '@mui/material/Grid';
import InputAdornment from '@mui/material/InputAdornment';
@@ -29,7 +33,11 @@ import OsIcon, { isWinPe, platformOf, WinPeChip } from 'components/OsIcon';
import PageHeader from 'components/PageHeader';
import { DeviceStatusChip } from 'components/StatusChip';
import AddDeviceDialog from 'sections/devices/AddDeviceDialog';
import DeviceHistory from 'sections/devices/DeviceHistory';
import { isOpenRemoval, isRemovalActivity } from 'sections/devices/removalLabels';
import RemovalStateChip from 'sections/devices/RemovalStateChip';
import useDeviceActions from 'sections/devices/useDeviceActions';
import useDeviceRemovals from 'sections/devices/useDeviceRemovals';
import GroupTreePanel, { ALL_NODE, UNGROUPED_NODE } from 'sections/groups/GroupTreePanel';
import useGroupActions from 'sections/groups/useGroupActions';
import { isUpgradeActivity, newestFor, versionSortKey } from 'sections/upgrades/agentVersions';
@@ -38,10 +46,10 @@ import useApi from 'hooks/useApi';
import useAutomation from 'hooks/useAutomation';
import useLiveRefresh from 'hooks/useLiveRefresh';
import useSession, { useGate } from 'hooks/useSession';
import { keys } from 'api/client';
import { keys, removalsKey } from 'api/client';
import { plural } from 'utils/format';
import { ancestors, devicesInGroup, ungroupedDevices } from 'utils/groupTree';
import type { AgentUpgradeDto, AgentVersionDto, DeviceDto, DeviceGroupDto } from 'api/types';
import type { AgentUpgradeDto, AgentVersionDto, DeviceDto, DeviceGroupDto, DeviceRemovalDto } from 'api/types';
// assets
import CalendarOutlined from '@ant-design/icons/CalendarOutlined';
@@ -51,13 +59,33 @@ import PlayCircleOutlined from '@ant-design/icons/PlayCircleOutlined';
import PlusOutlined from '@ant-design/icons/PlusOutlined';
import SearchOutlined from '@ant-design/icons/SearchOutlined';
type StatusFilter = 'all' | 'online' | 'stale' | 'offline' | 'orphaned';
type StatusFilter = 'all' | 'online' | 'stale' | 'offline' | 'orphaned' | 'removals';
/** A row of the table: a device, or a removal standing in for one. */
type ListRow = DeviceDto & { removal?: DeviceRemovalDto };
const removalRow = (removal: DeviceRemovalDto): ListRow => ({
clientId: removal.clientId,
hostname: removal.hostname ?? removal.clientId,
osCaption: removal.osCaption ?? undefined,
agentVersion: removal.agentVersion ?? undefined,
ipAddresses: [],
status: 'offline',
isOrphaned: false,
lastSeenUtc: removal.lastSeenUtc ?? undefined,
enrolledUtc: removal.requestedUtc,
removal
});
const listRowId = (row: ListRow) => (row.removal ? `removal:${row.removal.id}` : row.clientId);
const KIND_LABEL = { static: 'Static Group', dynamic: 'Dynamic Group', folder: 'Folder' } as const;
function matches(device: DeviceDto, query: string): boolean {
function matches(device: ListRow, query: string): boolean {
return [
device.hostname,
device.removal?.requestedBy,
device.removal?.note,
device.clientId,
device.manufacturer,
device.model,
@@ -92,12 +120,14 @@ export default function DevicesPage() {
refreshInterval: 60_000
});
const openByDevice = useMemo(() => new Map((openUpgrades ?? []).map((u) => [u.clientId, u])), [openUpgrades]);
const { data: removals, mutate: refreshRemovals } = useApi<DeviceRemovalDto[]>(removalsKey, { refreshInterval: 60_000 });
// Presence and inventory changes arrive as activity carrying a client id.
useLiveRefresh(
() => Promise.all([refreshDevices(), refreshGroups()]),
(event) => Boolean(event.clientId)
);
useLiveRefresh(() => refreshUpgrades(), isUpgradeActivity);
useLiveRefresh(() => Promise.all([refreshRemovals(), refreshDevices(), refreshGroups()]), isRemovalActivity);
const selectedId = params.get('group') ?? ALL_NODE;
const search = params.get('q') ?? '';
@@ -120,6 +150,7 @@ export default function DevicesPage() {
const [menu, setMenu] = useState<{ anchor: HTMLElement; devices: DeviceDto[] } | null>(null);
const [groupMenu, setGroupMenu] = useState<HTMLElement | null>(null);
const [addOpen, setAddOpen] = useState(false);
const [historyOf, setHistoryOf] = useState<ListRow | null>(null);
const selectedGroup = groups?.find((g) => g.id === selectedId) ?? null;
const groupActions = useGroupActions({ onDeleted: () => setParam('group', null) });
@@ -127,6 +158,9 @@ export default function DevicesPage() {
staticGroup: selectedGroup?.kind === 'static' ? selectedGroup : null,
onDone: () => setChosen(new Set())
});
const removalActions = useDeviceRemovals();
const canRemove = can('devices.remove');
const showingRemovals = status === 'removals';
const inScope = useMemo(() => {
const list = devices ?? [];
@@ -135,33 +169,54 @@ export default function DevicesPage() {
return devicesInGroup(list, groups ?? [], selectedId, nested);
}, [devices, groups, selectedId, nested]);
const rows = useMemo(() => {
const rows = useMemo<ListRow[]>(() => {
const query = search.trim().toLowerCase();
return inScope.filter((d) => {
if (query && !matches(d, query)) return false;
if (status === 'orphaned' ? !d.isOrphaned : status !== 'all' && (d.status !== status || d.isOrphaned)) return false;
if (platform !== 'all' && platformOf(d.osCaption) !== platform) return false;
return true;
});
}, [inScope, search, status, platform]);
const passes = (d: ListRow) => (!query || matches(d, query)) && (platform === 'all' || platformOf(d.osCaption) === platform);
if (showingRemovals) return (removals ?? []).map(removalRow).filter(passes);
const chosenRows = rows.filter((d) => chosen.has(d.clientId));
const listed = inScope.filter((d) => {
if (status === 'orphaned' ? !d.isOrphaned : status !== 'all' && (d.status !== status || d.isOrphaned)) return false;
return passes(d);
});
// Recent removals sit in the plain list; a removed device has no groups, and one enrolled again has its own row.
if (status !== 'all' || selectedId !== ALL_NODE) return listed;
const enrolled = new Set((devices ?? []).map((d) => d.clientId));
return [
...listed,
...(removals ?? [])
.filter((r) => r.showInDeviceList && !enrolled.has(r.clientId))
.map(removalRow)
.filter(passes)
];
}, [devices, inScope, removals, search, showingRemovals, status, platform, selectedId]);
const chosenRows = rows.filter((d) => !d.removal && chosen.has(d.clientId));
const groupName = (id: string) => groups?.find((g) => g.id === id)?.name ?? '…';
const title =
selectedId === ALL_NODE ? 'All Devices' : selectedId === UNGROUPED_NODE ? 'Ungrouped Devices' : (selectedGroup?.name ?? 'Group');
const title = showingRemovals
? 'Removals'
: selectedId === ALL_NODE
? 'All Devices'
: selectedId === UNGROUPED_NODE
? 'Ungrouped Devices'
: (selectedGroup?.name ?? 'Group');
const trail = selectedGroup ? [...ancestors(groups ?? [], selectedGroup.id).reverse(), selectedGroup] : [];
const online = inScope.filter((d) => d.status === 'online').length;
const columns: Column<DeviceDto>[] = [
{
id: 'hostname',
header: 'Name',
sortValue: (d) => d.hostname,
cell: (d) => (
<Stack direction="row" sx={{ alignItems: 'center', gap: 1.25, minWidth: 0 }}>
<OsIcon osCaption={d.osCaption} />
<Stack sx={{ minWidth: 0 }}>
// A removal has no device page; its row opens the history it left behind.
const nameColumn: Column<ListRow> = {
id: 'hostname',
header: 'Name',
sortValue: (d) => d.hostname,
cell: (d) => (
<Stack direction="row" sx={{ alignItems: 'center', gap: 1.25, minWidth: 0 }}>
<OsIcon osCaption={d.osCaption} />
<Stack sx={{ minWidth: 0 }}>
{d.removal ? (
<Typography variant="body2" sx={{ fontWeight: 600 }}>
{d.hostname}
</Typography>
) : (
<Link
component={RouterLink}
to={`/devices/${encodeURIComponent(d.clientId)}`}
@@ -172,18 +227,77 @@ export default function DevicesPage() {
>
{d.hostname}
</Link>
<Typography variant="caption" color="text.secondary" noWrap sx={{ fontFamily: 'var(--font-mono)', maxWidth: 240 }}>
{d.clientId}
</Typography>
</Stack>
)}
<Typography variant="caption" color="text.secondary" noWrap sx={{ fontFamily: 'var(--font-mono)', maxWidth: 240 }}>
{d.clientId}
</Typography>
</Stack>
</Stack>
)
};
const stopTrying = (removal: DeviceRemovalDto) => (
<GatedButton
size="small"
color="secondary"
gate={canRemove ? undefined : 'Needs the Remove Devices permission.'}
onClick={(e: MouseEvent<HTMLButtonElement>) => {
e.stopPropagation();
removalActions.stop(removal);
}}
>
Stop Trying
</GatedButton>
);
const removalColumns: Column<ListRow>[] = [
nameColumn,
{
id: 'state',
header: 'State',
sortValue: (d) => d.removal?.state,
cell: (d) => <RemovalStateChip removal={d.removal!} />
},
{ id: 'requestedBy', header: 'Requested By', sortValue: (d) => d.removal?.requestedBy, cell: (d) => d.removal?.requestedBy ?? '—' },
{
id: 'requested',
header: 'Requested',
nowrap: true,
sortValue: (d) => d.removal?.requestedUtc,
cell: (d) => <RelativeTime value={d.removal?.requestedUtc} />
},
{
id: 'finished',
header: 'Finished',
nowrap: true,
sortValue: (d) => d.removal?.finishedUtc,
cell: (d) => <RelativeTime value={d.removal?.finishedUtc} empty="—" />
},
{
id: 'detail',
header: 'Detail',
cell: (d) => (
<Typography variant="body2" sx={{ wordBreak: 'break-word', maxWidth: 320 }}>
{[d.removal?.note, d.removal?.detail].filter(Boolean).join(' · ') || '—'}
</Typography>
)
},
{
id: 'actions',
header: '',
align: 'right',
nowrap: true,
cell: (d) => (d.removal && isOpenRemoval(d.removal.state) ? stopTrying(d.removal) : null)
}
];
const columns: Column<ListRow>[] = [
nameColumn,
{
id: 'status',
header: 'Status',
sortValue: (d) => (d.isOrphaned ? 'z' : d.status),
cell: (d) => <DeviceStatusChip status={d.status} orphaned={d.isOrphaned} />
sortValue: (d) => d.removal?.state ?? (d.isOrphaned ? 'z' : d.status),
cell: (d) => (d.removal ? <RemovalStateChip removal={d.removal} /> : <DeviceStatusChip status={d.status} orphaned={d.isOrphaned} />)
},
{
id: 'os',
@@ -292,7 +406,7 @@ export default function DevicesPage() {
cell: (d) => (
<AgentVersion
version={d.agentVersion}
newest={newestFor(versions, d.osCaption, d.agentInstall)}
newest={d.removal ? undefined : newestFor(versions, d.osCaption, d.agentInstall)}
open={openByDevice.get(d.clientId)}
install={d.agentInstall}
/>
@@ -310,19 +424,24 @@ export default function DevicesPage() {
header: '',
width: 56,
align: 'right',
cell: (d) => (
<IconButton
size="small"
color="secondary"
aria-label={`Actions for ${d.hostname}`}
onClick={(e: MouseEvent<HTMLButtonElement>) => {
e.stopPropagation();
setMenu({ anchor: e.currentTarget, devices: [d] });
}}
>
<MoreOutlined />
</IconButton>
)
cell: (d) =>
d.removal ? (
isOpenRemoval(d.removal.state) ? (
stopTrying(d.removal)
) : null
) : (
<IconButton
size="small"
color="secondary"
aria-label={`Actions for ${d.hostname}`}
onClick={(e: MouseEvent<HTMLButtonElement>) => {
e.stopPropagation();
setMenu({ anchor: e.currentTarget, devices: [d] });
}}
>
<MoreOutlined />
</IconButton>
)
}
];
@@ -386,6 +505,11 @@ export default function DevicesPage() {
<MenuItem value="stale">Stale</MenuItem>
<MenuItem value="offline">Offline</MenuItem>
<MenuItem value="orphaned">Orphaned</MenuItem>
<MenuItem value="removals">
<Tooltip title="Every removed device, however long ago." placement="right">
<span>Removals</span>
</Tooltip>
</MenuItem>
</TextField>
<TextField
select
@@ -400,7 +524,7 @@ export default function DevicesPage() {
<MenuItem value="linux">Linux</MenuItem>
<MenuItem value="macos">macOS</MenuItem>
</TextField>
{selectedGroup && (
{selectedGroup && !showingRemovals && (
<FormControlLabel
control={<Switch size="small" checked={nested} onChange={(e) => setParam('nested', e.target.checked ? null : '0')} />}
label={
@@ -473,16 +597,23 @@ export default function DevicesPage() {
<Typography variant="h5" noWrap>
{title}
</Typography>
{selectedGroup && <Chip size="small" variant="outlined" label={KIND_LABEL[selectedGroup.kind]} />}
{selectedGroup && !showingRemovals && <Chip size="small" variant="outlined" label={KIND_LABEL[selectedGroup.kind]} />}
</Stack>
<Typography variant="caption" color="text.secondary">
{plural(inScope.length, 'device')}, {online} online
{selectedGroup?.description ? ` · ${selectedGroup.description}` : ''}
{showingRemovals ? (
plural(removals?.length ?? 0, 'removal')
) : (
<>
{plural(inScope.length, 'device')}, {online} online
{selectedGroup?.description ? ` · ${selectedGroup.description}` : ''}
</>
)}
</Typography>
</Stack>
}
secondary={
selectedGroup && (
selectedGroup &&
!showingRemovals && (
<Stack direction="row" sx={{ gap: 1 }}>
<GatedButton
size="small"
@@ -521,31 +652,40 @@ export default function DevicesPage() {
}
>
<DataTable
columns={columns}
key={showingRemovals ? 'removals' : 'devices'}
columns={showingRemovals ? removalColumns : columns}
rows={devices ? rows : undefined}
rowId={(d) => d.clientId}
rowId={listRowId}
loading={loading}
refreshing={validating && !loading}
error={error?.message}
toolbar={toolbar}
defaultSort={{ id: 'hostname', direction: 'asc' }}
defaultSort={showingRemovals ? { id: 'requested', direction: 'desc' } : { id: 'hostname', direction: 'asc' }}
rowLabel={(d) => d.hostname}
onRowClick={(d) => navigate(`/devices/${encodeURIComponent(d.clientId)}`)}
onRowClick={(d) => (d.removal ? setHistoryOf(d) : navigate(`/devices/${encodeURIComponent(d.clientId)}`))}
minWidth={1100}
selection={{
selected: chosen,
onChange: (ids, on) =>
setChosen((current) => {
const next = new Set(current);
for (const id of ids) {
if (on) next.add(id);
else next.delete(id);
dimmedRow={(d) => !showingRemovals && Boolean(d.removal)}
selection={
showingRemovals
? undefined
: {
selected: chosen,
selectable: (d) => !d.removal,
onChange: (ids, on) =>
setChosen((current) => {
const next = new Set(current);
for (const id of ids) {
if (on) next.add(id);
else next.delete(id);
}
return next;
})
}
return next;
})
}}
}
empty={
(devices?.length ?? 0) === 0 ? (
showingRemovals ? (
<EmptyState dense title={(removals?.length ?? 0) === 0 ? 'No Removals Yet' : 'No Matches'} />
) : (devices?.length ?? 0) === 0 ? (
<EmptyState
title="No Devices Yet"
detail="Enrolled devices appear here."
@@ -582,6 +722,16 @@ export default function DevicesPage() {
{groupMenu && <ActionMenu anchorEl={groupMenu} onClose={() => setGroupMenu(null)} entries={groupMenuEntries} />}
{deviceActions.elements}
{groupActions.elements}
{removalActions.elements}
<Dialog open={historyOf !== null} onClose={() => setHistoryOf(null)} maxWidth="md" fullWidth>
<DialogTitle>{historyOf?.hostname}</DialogTitle>
<DialogContent>{historyOf && <DeviceHistory clientId={historyOf.clientId} />}</DialogContent>
<DialogActions sx={{ px: 3, pb: 2 }}>
<Button variant="contained" onClick={() => setHistoryOf(null)}>
Close
</Button>
</DialogActions>
</Dialog>
<Box>
<AddDeviceDialog open={addOpen} onClose={() => setAddOpen(false)} />
</Box>
+12 -11
View File
@@ -10,7 +10,6 @@ import Typography from '@mui/material/Typography';
// project imports
import { RelativeTime } from 'components/DateTime';
import GatedButton from 'components/GatedButton';
import MainCard from 'components/MainCard';
import PageHeader from 'components/PageHeader';
import StatusChip from 'components/StatusChip';
import JobOutputDrawer from 'sections/jobs/JobOutputDrawer';
@@ -29,7 +28,10 @@ import type { ExecutionDetailDto } from 'api/types';
/** Running / Complete / Complete With Failures, for the header. */
function RunStatus({ run }: { run: ExecutionDetailDto }) {
const counts = runCounts(run);
if (!runFinished(counts)) return <StatusChip status="running" label={counts.running ? 'Running' : 'In Progress'} />;
if (!runFinished(counts)) {
if (!counts.running && counts.pending === 0) return <StatusChip status="waiting" />;
return <StatusChip status="running" label={counts.running ? 'Running' : 'In Progress'} />;
}
return counts.failed > 0 ? (
<StatusChip status="failed" label="Complete With Failures" />
) : (
@@ -105,15 +107,14 @@ export default function RunPage() {
{execution.error && !run ? (
<Alert severity="error">{errorMessage(execution.error, 'Could not load the run.')}</Alert>
) : (
<MainCard content={false}>
<RunProgress
run={run}
loading={execution.loading}
refreshing={execution.validating && !execution.loading}
openJobId={openJobId}
onOpenJob={setJob}
/>
</MainCard>
<RunProgress
dashboard
run={run}
loading={execution.loading}
refreshing={execution.validating && !execution.loading}
openJobId={openJobId}
onOpenJob={setJob}
/>
)}
<JobOutputDrawer jobId={openJobId} onClose={() => setJob(null)} />
@@ -1,81 +0,0 @@
import { useEffect, useEffectEvent, useState } from 'react';
// material-ui
import { keyframes } from '@mui/material/styles';
import Box from '@mui/material/Box';
import Button from '@mui/material/Button';
import LinearProgress from '@mui/material/LinearProgress';
import Stack from '@mui/material/Stack';
import Typography from '@mui/material/Typography';
// project imports
import MainCard from 'components/MainCard';
import { TERMINAL_JOB_STATES } from 'components/StatusChip';
import { nats } from 'messaging/nats';
const pulse = keyframes`
0%, 100% { opacity: 1; }
50% { opacity: 0.3; }
`;
// ==============================|| DASHBOARD - LIVE DEPLOYMENT TRACKER ||============================== //
/**
* Rolls live job.state events into a running count, so a group run of N
* endpoints shows progress as results land. Calls `onFinished` whenever a
* job reaches a terminal state.
*/
export default function LiveDeployment({ onFinished }: { onFinished: () => void }) {
const [total, setTotal] = useState(0);
const [finished, setFinished] = useState(0);
const finishedOne = useEffectEvent(onFinished);
useEffect(
() =>
nats.onJobEvent((_jobId, event) => {
if (event.type !== 'job.state') return;
const state = event.payload.state;
if (state === 'queued' || state === 'dispatched') {
setTotal((n) => n + 1);
} else if (TERMINAL_JOB_STATES.includes(state)) {
setFinished((n) => n + 1);
finishedOne();
}
}),
[]
);
if (total === 0 || finished >= total) return null;
const dismiss = () => {
setTotal(0);
setFinished(0);
};
return (
<MainCard sx={(theme) => ({ borderColor: theme.vars.palette.info.main })} contentSX={{ py: 1.5, '&:last-child': { pb: 1.5 } }}>
<Stack sx={{ gap: 1 }}>
<Stack direction="row" sx={{ alignItems: 'center', gap: 1 }}>
<Box
aria-hidden
sx={{ width: 9, height: 9, borderRadius: '50%', bgcolor: 'info.main', animation: `${pulse} 1.2s ease-in-out infinite` }}
/>
<Typography variant="subtitle1">Deploying</Typography>
<Typography variant="body2" color="text.secondary" sx={{ flex: 1 }} aria-live="polite">
{finished} of {total} endpoints finished
</Typography>
<Button size="small" color="secondary" onClick={dismiss}>
Dismiss
</Button>
</Stack>
<LinearProgress
variant="determinate"
color="info"
value={(finished / total) * 100}
aria-label="Deployment progress"
sx={{ height: 8, borderRadius: 4 }}
/>
</Stack>
</MainCard>
);
}
@@ -20,7 +20,10 @@ const EVENTS: Record<string, { label: string; color: ChipProps['color'] }> = {
'device-os-changed': { label: 'OS Changed', color: 'primary' },
'agent-upgraded': { label: 'Agent Upgraded', color: 'success' },
'agent-rolled-back': { label: 'Agent Rolled Back', color: 'warning' },
'agent-upgrade-failed': { label: 'Agent Upgrade Failed', color: 'error' }
'agent-upgrade-failed': { label: 'Agent Upgrade Failed', color: 'error' },
'device-removal-requested': { label: 'Removal Requested', color: 'warning' },
'device-removed': { label: 'Removed', color: 'secondary' },
'device-removal-stopped': { label: 'Removal Stopped', color: 'warning' }
};
// ==============================|| DEVICE - HISTORY ||============================== //
@@ -65,7 +68,7 @@ export default function DeviceHistory({ clientId }: { clientId: string }) {
title={
<>
Device History
<HoverHelp title="Enrollments, renames, OS changes and agent upgrades, newest first — e.g. Windows PE before imaging, then the installed OS." />
<HoverHelp title="Enrollments, renames, OS changes, agent upgrades and removals, newest first — e.g. Windows PE before imaging, then the installed OS." />
</>
}
>
@@ -0,0 +1,40 @@
// material-ui
import Chip from '@mui/material/Chip';
import Stack from '@mui/material/Stack';
import Tooltip from '@mui/material/Tooltip';
import Typography from '@mui/material/Typography';
// project imports
import Dot from 'components/@extended/Dot';
import { REMOVAL_REASONS, REMOVAL_STATES } from './removalLabels';
import type { DeviceRemovalDto } from 'api/types';
// ==============================|| REMOVAL STATE CHIP ||============================== //
/** A removal's state, explaining itself on hover, and under it what stands in its way when something does. */
export default function RemovalStateChip({ removal }: { removal: DeviceRemovalDto }) {
const known = REMOVAL_STATES[removal.state] ?? { label: removal.state, help: '', tone: 'secondary' as const };
const why = removal.reason ? REMOVAL_REASONS[removal.reason] : undefined;
const chip = (
<Chip
size="small"
variant="light"
color={known.tone}
label={known.label}
icon={<Dot size={6} color={known.tone} sx={{ ml: 0.75 }} />}
sx={{ fontWeight: 500 }}
/>
);
return (
<Stack sx={{ alignItems: 'flex-start', gap: 0.25 }}>
{known.help ? <Tooltip title={known.help}>{chip}</Tooltip> : chip}
{removal.reason && (
<Tooltip title={why?.help ?? ''}>
<Typography variant="caption" color="text.secondary">
{why?.label ?? removal.reason}
</Typography>
</Tooltip>
)}
</Stack>
);
}
@@ -0,0 +1,45 @@
// types
import type { ActivityEventPayload, DeviceRemovalState } from 'api/types';
type Tone = 'success' | 'warning' | 'error' | 'info' | 'primary' | 'secondary';
/** Label, hover text and tone per removal state (one map, so a state always reads the same). */
export const REMOVAL_STATES: Record<DeviceRemovalState, { label: string; help: string; tone: Tone }> = {
waiting: { label: 'Waiting for Device', help: 'The agent uninstalls when the device next comes online.', tone: 'secondary' },
sent: { label: 'Sent', help: 'The remove command was sent, waiting for the agent to answer.', tone: 'info' },
uninstalling: { label: 'Uninstalling', help: 'The agent is uninstalling itself.', tone: 'primary' },
removed: { label: 'Removed', help: 'The agent has gone.', tone: 'success' },
failed: {
label: "Couldn't Uninstall",
help: 'The uninstall failed on the device; the server asks again while it is online.',
tone: 'error'
},
stopped: { label: 'Stopped', help: 'Nobody is trying any more. The device stays blocked and its agent stays installed.', tone: 'warning' }
};
/** States still being worked on: only these can be stopped. */
export const OPEN_REMOVAL_STATES: DeviceRemovalState[] = ['waiting', 'sent', 'uninstalling', 'failed'];
export const isOpenRemoval = (state: DeviceRemovalState) => OPEN_REMOVAL_STATES.includes(state);
/** Why a removal could not go ahead. */
export const REMOVAL_REASONS: Record<string, { label: string; help: string }> = {
'agent-too-old': {
label: 'Uninstall It by Hand or Upgrade It First',
help: 'This agent is too old to take a remove command; the server keeps blocking it.'
},
unsupported: { label: "Can't Remove Itself Here", help: 'A foreground or development run, or a container. Uninstall it by hand.' },
busy: { label: 'Upgrade in Progress', help: 'An agent upgrade is under way; the server asks again.' },
msi: { label: 'Windows Installer Failed', help: 'Windows Installer did not uninstall the agent; the server asks again.' },
uninstall: { label: "Files Couldn't Be Removed", help: 'Files or services could not be removed; the server asks again.' },
'stale-enrollment': { label: 'Another Enrollment', help: 'The agent holds a different enrollment, so it ignored the command.' }
};
/** Why a request left a device out. */
export const REMOVAL_SKIPS: Record<string, { label: string; help: string }> = {
unknown: { label: 'Not Found', help: 'No such device; it may have been removed already.' },
'out-of-scope': { label: 'Outside Your Scope', help: 'Your roles do not reach this device.' }
};
/** The console refreshes devices and removals on this activity. */
export const isRemovalActivity = (event: ActivityEventPayload) => event.kind === 'device-removal';
@@ -15,6 +15,7 @@ import ConfirmDialog from 'components/ConfirmDialog';
import { MenuEntry } from 'components/ActionMenu';
import HoverHelp from 'components/HoverHelp';
import GroupEditor from 'sections/groups/GroupEditor';
import useDeviceRemovals from 'sections/devices/useDeviceRemovals';
import useAgentUpgrades, { UpgradeTarget } from 'sections/upgrades/useAgentUpgrades';
import useApi from 'hooks/useApi';
import useAutomation from 'hooks/useAutomation';
@@ -38,8 +39,11 @@ const upgradeTarget = (devices: DeviceDto[]): UpgradeTarget => ({
/**
* Everything the device "…" and bulk Actions menus can do. Render `elements`
* once; `menuFor(devices)` builds the entries for one device or a selection.
* `onRemoved` runs after a removal (`onDone` when not given).
*/
export default function useDeviceActions(options: { staticGroup?: DeviceGroupDto | null; onDone?: () => void } = {}) {
export default function useDeviceActions(
options: { staticGroup?: DeviceGroupDto | null; onDone?: () => void; onRemoved?: () => void } = {}
) {
const navigate = useNavigate();
const { can } = useSession();
const { openRun, openSchedule } = useAutomation();
@@ -53,6 +57,7 @@ export default function useDeviceActions(options: { staticGroup?: DeviceGroupDto
const [removing, setRemoving] = useState<DeviceDto[] | null>(null);
const upgrades = useAgentUpgrades();
const removals = useDeviceRemovals({ onRemoved: options.onRemoved ?? options.onDone });
const refresh = () => Promise.all([mutate(keys.devices), mutate(keys.groups)]);
const staticGroups = flattenTree(buildGroupTree(groups ?? [])).filter((n) => n.group.kind === 'static');
@@ -89,6 +94,7 @@ export default function useDeviceActions(options: { staticGroup?: DeviceGroupDto
refreshInventory: () => void refreshInventory(devices),
upgradeAgent: () => upgrades.upgrade(upgradeTarget(devices)),
rollBackAgent: () => upgrades.rollBack(upgradeTarget(devices)),
removeAgent: () => removals.remove(upgradeTarget(devices)),
details: () => navigate(`/devices/${encodeURIComponent(devices[0].clientId)}`),
history: () => navigate(`/jobs?clientId=${encodeURIComponent(devices[0].clientId)}`),
copyId: () =>
@@ -104,6 +110,7 @@ export default function useDeviceActions(options: { staticGroup?: DeviceGroupDto
const elements = (
<>
{upgrades.elements}
{removals.elements}
<ConfirmDialog
open={grouping !== null}
title="Add to Group"

Some files were not shown because too many files have changed in this diff Show More