Device removal, execution stagger and a live run dashboard #20
@@ -53,12 +53,14 @@ updates with **no SignalR, SSE, or polling**.
|
||||
25. [Database providers](#database-providers)
|
||||
26. [Logging](#logging)
|
||||
27. [Device lifecycle: reconnect or orphan](#device-lifecycle-reconnect-or-orphan)
|
||||
28. [Agent upgrades from the server](#agent-upgrades-from-the-server)
|
||||
29. [S3 file delivery](#s3-file-delivery)
|
||||
30. [CI/CD and GITEA_* variables](#cicd-and-gitea_-variables)
|
||||
31. [Versioning](#versioning)
|
||||
32. [Troubleshooting](#troubleshooting)
|
||||
33. [Licence](#licence)
|
||||
28. [Removing devices](#removing-devices)
|
||||
29. [Removing devices](#removing-devices)
|
||||
30. [Agent upgrades from the server](#agent-upgrades-from-the-server)
|
||||
31. [S3 file delivery](#s3-file-delivery)
|
||||
32. [CI/CD and GITEA_* variables](#cicd-and-gitea_-variables)
|
||||
33. [Versioning](#versioning)
|
||||
34. [Troubleshooting](#troubleshooting)
|
||||
35. [Licence](#licence)
|
||||
|
||||
---
|
||||
|
||||
@@ -721,6 +723,7 @@ still holds it.
|
||||
| `upgrade/msi/` | the MSI being installed, the one to go back to (`rollback/`) and the runner's copy of the watchdog (`runner/`) |
|
||||
| `<install>\Logs\agent-msi-<version>-<id>.log`, `ExecuteR-Watchdog-msi.log` | Windows Installer's verbose log of each MSI run, and the runner's own log |
|
||||
| `%ProgramData%\ExecuteR\packages\<sha256>\` | agent MSIs kept for rollback (the newest three) |
|
||||
| `upgrade/removal.json` | a removal under way, or one that failed until the agent has reported it ([On the device](#on-the-device)) |
|
||||
|
||||
If the watchdog itself stops part way (power loss, a kill), its next start
|
||||
finishes or undoes the run from the journal: `stopping` starts over (the agent
|
||||
@@ -1101,6 +1104,24 @@ an object's properties as columns) rather than a type name such as
|
||||
`System.Collections.Hashtable`, still streamed line by line and without
|
||||
`…` truncation. Strings pass through unchanged.
|
||||
|
||||
**Exit codes.** A PowerShell 7 script reports what `pwsh -File` would: `exit N`
|
||||
is exit code N and Failed when N is not 0, `exit 0` succeeds even after errors,
|
||||
and a terminating error is exit 1. A native command's non-zero `$LASTEXITCODE`
|
||||
only counts if the script exits with it (`exit $LASTEXITCODE`), and errors the
|
||||
script silenced do not fail it. One difference, kept on purpose: anything on the
|
||||
error stream (`Write-Error`, a failing cmdlet) fails a script that never calls
|
||||
`exit`, with exit code 1.
|
||||
|
||||
> **Changed behaviour:** agents before this version reported `exit 5` as Succeeded
|
||||
> with exit code 0, and failed any script whose native command ended non-zero or
|
||||
> that silenced an error with `-ErrorAction SilentlyContinue`. A script relying on
|
||||
> either now reports differently.
|
||||
|
||||
A failed job also records its first error's exception type, error id and message
|
||||
(for Windows PowerShell and Bash, the first stderr line). For jobs from older
|
||||
agents the run's error groups use the summary or the first error line of the
|
||||
output.
|
||||
|
||||
**Downloads.** A job's output drawer downloads its complete stored output as
|
||||
`COMPUTERNAME.JobID.yyyy.MM.dd.HHmmss.log` (optionally with timestamps), and
|
||||
a run (Jobs › run, or a script's Executions tab) downloads every target's
|
||||
@@ -1317,6 +1338,38 @@ of one run shares one execution, which the result links to.
|
||||
call, answering with the job), `clientIds`, `groupId`, `groupIds` and
|
||||
`onlineOnly`, deduplicated and scope-filtered into one `runId`.
|
||||
|
||||
**Stagger** spreads a run's starts so a large fleet does not hit one share or
|
||||
service at once: **Spread Over** (`staggerSeconds`, up to 24 hours) and
|
||||
**Devices at a Time** (`staggerBatchSize`, default 1), on the Review tab and on
|
||||
schedules. Devices fall into `ceil(targets ÷ batch)` batches evenly spaced over
|
||||
the spread: the first starts at once, the last when the spread ends, in an order
|
||||
shuffled per run so the same device is not always last. With one batch or no
|
||||
spread the run dispatches as it always has. What it guarantees:
|
||||
|
||||
- It is durable. Each job keeps its fixed start time and a worker on the
|
||||
`SCHEDULE_WORKER` node sends it when it is due, so a backend restart loses
|
||||
nothing and starting a run returns at once for any number of devices.
|
||||
- Every device runs the script **as it was when the run started**, with its
|
||||
arguments and prelude, from a snapshot sealed with Data Protection and cleared
|
||||
when nothing is left to send. Secrets and presigned URLs are resolved when each
|
||||
batch is sent and never stored.
|
||||
- **Only Devices Online Now** is checked again at a device's start: offline by
|
||||
then, it becomes skipped.
|
||||
- A deleted script fails the jobs still waiting. **Cancel Waiting**
|
||||
(`POST /api/executions/{runId}/cancel-waiting`, Run Jobs, audited, within the
|
||||
caller's device scope) cancels what has not been sent.
|
||||
- A job whose publish fails is retried by the same worker, with or without a
|
||||
stagger, for up to 24 hours before it fails; a vault or S3 failure at send time
|
||||
is retried for 5 minutes.
|
||||
|
||||
**Watching a run** (Jobs › a run): while devices wait, a strip counts down to the
|
||||
next and the last start, corrected for the browser's clock, and each waiting
|
||||
device counts down in its row. Above the devices, updating live: a tile per
|
||||
state, an outcome ring, exit codes, errors grouped by exception type and first
|
||||
line with sample devices, and completions over time; tiles, exit codes and errors
|
||||
filter the table. `GET /api/executions/{runId}/breakdown` serves the charts,
|
||||
limited to the caller's device scope.
|
||||
|
||||
**Schedules** (the Schedules page, plus a Schedules tab on scripts, devices
|
||||
and groups) are scheduled triggers: *when* (one-shot or recurring), *target*
|
||||
(a device, a group whose members resolve at fire time, or all devices), an
|
||||
@@ -1342,6 +1395,8 @@ Semantics worth knowing:
|
||||
- The worker sweeps regularly; recurrence advances from *now* after each
|
||||
firing, so server downtime causes **at most one late firing**, never a storm.
|
||||
- One-shot schedules disable themselves after firing.
|
||||
- A schedule's **Stagger** applies to each firing; event triggers run on one
|
||||
device at a time and have none.
|
||||
- Each schedule records its **last result** (jobs queued, or the error — e.g.
|
||||
a vault failure) and the execution it created; scheduled jobs are attributed
|
||||
to `schedule`, run-now firings to the operator.
|
||||
@@ -1460,6 +1515,158 @@ the device keeps its history, groups and tags.
|
||||
|
||||
---
|
||||
|
||||
## Removing devices
|
||||
|
||||
**Remove Agent…** on a device (row menu, bulk Actions, device page) or **Remove
|
||||
Agents…** on a group or folder (nested groups included) takes the device out of
|
||||
the system at once and uninstalls its agent now, or whenever it next comes back.
|
||||
It needs the `devices.remove` permission, which Admin has and Operator and Viewer
|
||||
do not. Scoped roles reach only the devices in their scope.
|
||||
|
||||
The device row goes in the same transaction as its group memberships and trigger
|
||||
runs; schedules aimed at that device alone are disabled and jobs that never
|
||||
started are cancelled. Finished jobs, their output and the device's history stay,
|
||||
and old runs keep its name. What stands in for the row is a **removal**, kept
|
||||
forever as the history.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
R["Remove Agent: device row deleted, removal waiting"] --> S{"Seen online?"}
|
||||
S -- "No, however long" --> S
|
||||
S -- "Yes, agent 1.1.0 or later" --> C["Remove command sent (at most every 5 min)"]
|
||||
S -- "Yes, older agent" --> O["Agent too old: stays open, stays blocked"]
|
||||
C --> U["Agent reports uninstalling"]
|
||||
C -- "It failed" --> F["Couldn't Uninstall: asked again"]
|
||||
C -- "No answer for 30 min" --> S
|
||||
F --> S
|
||||
U -- "Silent for 15 min" --> D["Removed: consumer deleted, commands purged"]
|
||||
U -- "Still reporting after 15 min" --> S
|
||||
D -- "Reports again" --> S
|
||||
```
|
||||
|
||||
**Online.** The next scheduler pass (every 15 seconds, on the node with
|
||||
`SCHEDULE_WORKER` on) sends the command. Its enrollment id must match what the
|
||||
agent holds, so a stale command can never uninstall a device enrolled again since.
|
||||
After a successful uninstall nobody is left to report success, so 15 minutes of
|
||||
silence after "uninstalling" counts as gone.
|
||||
|
||||
**Offline.** A removal waits for as long as the device stays away. Presence,
|
||||
inventory and ready notices from a client id with no device row only update the
|
||||
removal; they never bring the row back.
|
||||
|
||||
**Credentials.** Agents from 1.1.0 on hold a NATS credential that lasts 24 hours,
|
||||
renewed over HTTPS at half life and at every start. A renewal for a removed device
|
||||
answers `410` with problem type `urn:problem:device-removed`, and the agent
|
||||
uninstalls itself, so a device that slips past the command is still cut off within
|
||||
a day. Only the holder of the enrolled key learns this: a bad signature gets the
|
||||
same plain `401` as an id nobody knows. Older agents never renew and could not
|
||||
survive an expiring credential, so enrollment still gives them one that never
|
||||
expires. To keep that true an agent on 1.1.0 or later is never upgraded or rolled
|
||||
back to anything older (it is left out as `credential-downgrade`); exactly 1.1.0
|
||||
is fine.
|
||||
|
||||
**Tokens.** A bootstrap token created at or before the removal no longer enrolls
|
||||
that device, whatever its other limits say, so the install it has and a GPO still
|
||||
pushing an old token cannot bring it back. The response is the plain `401` of any
|
||||
refused token, with the reason in the audit trail. A token created after the
|
||||
removal enrolls it on a fresh device row: an open removal closes as removed, and
|
||||
its old command consumer and commands are cleared first, so nothing old replays.
|
||||
|
||||
**In the Devices list.** A removal shows in the list for 7 days after its last
|
||||
change: dimmed, with its state, not selectable and not counted in the totals. After
|
||||
that the list stays uncluttered, and the **Removals** option of the Status filter
|
||||
shows every removal whatever its age, with who asked, when, and what happened.
|
||||
Clicking a removal opens the device's history. `GET /api/device-removals` takes
|
||||
`state`, `clientId` and `limit` (default 200). Scoped roles see the removals of
|
||||
devices that were in one of their scope groups when they were removed; a device
|
||||
that was in none is visible to unscoped roles only.
|
||||
|
||||
**Stop Trying.** `POST /api/device-removals/{id}/stop` on an open removal ends the
|
||||
chase: the device stays blocked, its pending command is purged and its agent stays
|
||||
installed, cut off when its credential expires (a renewal gets a plain `401`, not
|
||||
`410`, because `410` would uninstall it after all). Enrolling it again needs a new
|
||||
token.
|
||||
|
||||
**Agents too old.** An agent below 1.1.0 cannot take the command or renew. Its
|
||||
removal stays open marked *Uninstall It by Hand or Upgrade It First*, and since
|
||||
its credential never expires the server cannot cut it off from NATS: it can keep
|
||||
connecting, but the server ignores everything it reports. Once such an agent reports
|
||||
1.1.0 or later the command goes out as usual.
|
||||
|
||||
The audit trail keeps `device-removal-requested`, `device-removed` and
|
||||
`device-removal-stopped`. The project wiki is updated separately.
|
||||
|
||||
### On the device
|
||||
|
||||
**Credentials.** The agent's NATS credential lives 24 hours. A small background
|
||||
service renews it over HTTPS (`POST /api/enrollment/credential`, signed with the
|
||||
enrolled NKey, TLS as for enrollment) at every start and at half life, whether
|
||||
or not NATS is connected, so an expired credential recovers by itself as soon as
|
||||
the server can be reached, and a credential from before 1.1 that never expires
|
||||
is replaced at the first start. The old credential stays until the new one is
|
||||
stored; the NATS connection takes the new one at its next reconnect, at the
|
||||
latest when the server ends the connection as the old one expires.
|
||||
|
||||
| The server answers | The agent |
|
||||
|---|---|
|
||||
| a new credential | stores it, atomically, and looks again at half its life |
|
||||
| network error, 5xx, anything unexpected | keeps what it has and tries again after 30 s, doubling to 15 minutes |
|
||||
| 401 (orphaned, or the clock is off) | keeps what it has and tries again hourly; a configured bootstrap token enrolls it again ([Device lifecycle](#device-lifecycle-reconnect-or-orphan)) |
|
||||
| 410 `urn:problem:device-removed` | uninstalls itself, as below. Not with `TlsSkipVerify` on or over plain http: an unverified server must not be able to uninstall an agent, so that only logs a warning |
|
||||
|
||||
While an upgrade could still roll back (`handover.json`, `journal.json`,
|
||||
`msi.json`), the stored credential is left alone: a version before 1.1 cannot
|
||||
renew and would strand on a 24-hour one. It renews when the upgrade is kept. The
|
||||
one exception is a credential within an hour of expiring, which would strand
|
||||
either way and would keep the new version from reporting healthy. For the same
|
||||
reason an agent on 1.1 or later refuses an upgrade to a version before 1.1
|
||||
(**failed / unsupported**).
|
||||
|
||||
**The remove command.** `command.agent.remove` may be old, since it waits in its
|
||||
stream, so the agent acts only when its enrollment ID is the one the agent holds
|
||||
(else **failed / stale-enrollment**). A foreground or development run answers
|
||||
**failed / unsupported**, an upgrade in progress **failed / busy** (the server
|
||||
asks again). Otherwise the agent reports **uninstalling**, writes
|
||||
`upgrade/removal.json` and leaves the rest to the watchdog. A device where no
|
||||
watchdog can take it (none installed, or it has not taken it within 2 minutes)
|
||||
runs the same runner from a copy of the agent's own binary. The watchdog starts
|
||||
no upgrade while `removal.json` exists and a removal waits for an upgrade under
|
||||
way; it does not supervise the agent during a removal.
|
||||
|
||||
**The runner.** The watchdog copies itself out of both folders it is about to
|
||||
wipe, into `%SystemRoot%\Temp\ExecuteR-<id>-<random>\` (closed to SYSTEM and
|
||||
Administrators), or `/var/lib/executer-removal-<random>/` on Linux and macOS, and
|
||||
starts the copy apart from its service: detached on Windows, as a transient
|
||||
`systemd-run` unit on Linux, as a one-shot launchd daemon on macOS, since the
|
||||
service manager would otherwise end it with the service. The runner:
|
||||
|
||||
1. stops the agent;
|
||||
2. uninstalls: an MSI install in process through Windows Installer
|
||||
(`MsiConfigureProductExW`, no UI, `REBOOT=ReallySuppress`, never msiexec),
|
||||
which removes both services; 1618 is waited out for up to 10 minutes. A
|
||||
portable install loses both services through the service manager (the runner
|
||||
refuses to wipe a folder that holds no agent binary, or a shared one such as
|
||||
`C:\Program Files`);
|
||||
3. verifies and finishes the purge, whatever the MSI's own purge managed.
|
||||
|
||||
Three attempts, 5 minutes apart, 30 minutes at most each. A failed attempt starts
|
||||
both services again. After the third, `removal.json` says **failed** with the
|
||||
reason, the agent reports **failed / msi** or **failed / uninstall**, deletes the
|
||||
file and carries on; the server asks again. After a success nobody is left to
|
||||
report it.
|
||||
|
||||
| Wiped | Left |
|
||||
|---|---|
|
||||
| `credentials.dat` and the whole data folder (`%ProgramData%\ExecuteR`, `/var/lib/executer`) | the policy key `HKLM\SOFTWARE\Policies\ExecuteR` (and the policy section of `agent.json`): it is group policy's |
|
||||
| `HKLM\SOFTWARE\ExecuteR` with `Runtime`, `ServerURL`, the token, proxy and TLS values (`/etc/executer/agent.json`, the `/usr/local/bin` link) | a server CA the agent did not itself add to the trust store. That includes every CA installed before 1.1, which kept no record of it |
|
||||
| the install folder, `Logs` included | the runner's folder on Windows, until the next restart |
|
||||
| both services; the server CA the agent added (`managed-ca.txt`) | |
|
||||
|
||||
A file still locked after a minute is deleted at the next restart. A failed
|
||||
removal keeps its logs as `<install>\Logs\agent-removal-<id>-*.log`.
|
||||
|
||||
---
|
||||
|
||||
## Agent installer delivery
|
||||
|
||||
**The installer ships inside the backend image — no manual seeding.** The CI
|
||||
|
||||
@@ -25,6 +25,15 @@ public sealed class AgentSupervisor(WatchdogOptions options, IAgentService agent
|
||||
|
||||
public async Task CheckAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
// A removal stops the agent on purpose; only one that failed hands it back.
|
||||
if (WatchdogFileStore.Read(options.RemovalPath, WatchdogJsonContext.Default.WatchdogRemoval) is { Phase: not Shared.Contracts.Models.WatchdogRemovalPhases.Failed })
|
||||
{
|
||||
runningSince = null;
|
||||
failedSince = null;
|
||||
Note("A removal is under way; the agent is left to it");
|
||||
return;
|
||||
}
|
||||
|
||||
var status = await agent.QueryAsync(cancellationToken);
|
||||
var now = clock.UtcNow;
|
||||
|
||||
|
||||
@@ -17,6 +17,12 @@ public interface IWindowsInstaller
|
||||
|
||||
/// <summary><c>MsiInstallProductW</c>: installs the package and returns Windows Installer's own code (what msiexec would exit with).</summary>
|
||||
int InstallProduct(string package, string commandLine);
|
||||
|
||||
/// <summary><c>MsiConfigureProductExW</c> with <c>INSTALLSTATE_ABSENT</c>: uninstalls the product and returns Windows Installer's own code.</summary>
|
||||
int UninstallProduct(string productCode, string commandLine);
|
||||
|
||||
/// <summary><c>MsiEnumRelatedProductsW</c>: the product code installed under the UpgradeCode, or null.</summary>
|
||||
string? RelatedProduct(string upgradeCode);
|
||||
}
|
||||
|
||||
/// <summary>Windows Installer's answers an MSI run acts on.</summary>
|
||||
@@ -37,6 +43,8 @@ public static class MsiExitCodes
|
||||
public const int PackageInvalid = 1620;
|
||||
/// <summary>ERROR_PRODUCT_VERSION: a newer version is installed (a downgrade without ALLOWDOWNGRADE=1).</summary>
|
||||
public const int ProductVersion = 1638;
|
||||
/// <summary>ERROR_UNKNOWN_PRODUCT: the product is not installed (an uninstall that has nothing to do).</summary>
|
||||
public const int UnknownProduct = 1605;
|
||||
|
||||
/// <summary>Done, done with a restart pending, or done with a restart started.</summary>
|
||||
public static bool Succeeded(int code) => code is Success or SuccessRebootRequired or SuccessRebootInitiated;
|
||||
@@ -51,6 +59,7 @@ public static class MsiExitCodes
|
||||
InstallFailure => "a fatal error during installation",
|
||||
PackageInvalid => "the package could not be opened",
|
||||
ProductVersion => "another version of the product is already installed",
|
||||
UnknownProduct => "the product is not installed",
|
||||
1619 => "the package could not be opened",
|
||||
1625 => "the installation is forbidden by system policy",
|
||||
1633 => "the package is not supported on this platform",
|
||||
@@ -64,6 +73,12 @@ public sealed partial class WindowsInstallerApi : IWindowsInstaller
|
||||
/// <summary>INSTALLUILEVEL_NONE.</summary>
|
||||
private const int InstallUiLevelNone = 2;
|
||||
|
||||
/// <summary>INSTALLLEVEL_DEFAULT.</summary>
|
||||
private const int InstallLevelDefault = 0;
|
||||
|
||||
/// <summary>INSTALLSTATE_ABSENT: Windows Installer sets REMOVE=ALL, as for msiexec's <c>/x</c>.</summary>
|
||||
private const int InstallStateAbsent = 2;
|
||||
|
||||
/// <summary>
|
||||
/// msiexec's <c>/l*v</c> (msi.h INSTALLLOGMODE_*): FATALEXIT 0x1, ERROR 0x2,
|
||||
/// WARNING 0x4, USER 0x8, INFO 0x10, RESOLVESOURCE 0x40, OUTOFDISKSPACE 0x80,
|
||||
@@ -89,6 +104,21 @@ public sealed partial class WindowsInstallerApi : IWindowsInstaller
|
||||
|
||||
public int InstallProduct(string package, string commandLine) => (int)MsiInstallProductW(package, commandLine);
|
||||
|
||||
public int UninstallProduct(string productCode, string commandLine)
|
||||
=> (int)MsiConfigureProductExW(productCode, InstallLevelDefault, InstallStateAbsent, commandLine);
|
||||
|
||||
public unsafe string? RelatedProduct(string upgradeCode)
|
||||
{
|
||||
var buffer = stackalloc char[39];
|
||||
return MsiEnumRelatedProductsW(upgradeCode, 0, 0, buffer) == 0 ? new string(buffer, 0, 38) : null;
|
||||
}
|
||||
|
||||
[LibraryImport("msi.dll", StringMarshalling = StringMarshalling.Utf16)]
|
||||
private static partial uint MsiConfigureProductExW(string product, int installLevel, int installState, string commandLine);
|
||||
|
||||
[LibraryImport("msi.dll", StringMarshalling = StringMarshalling.Utf16)]
|
||||
private static unsafe partial uint MsiEnumRelatedProductsW(string upgradeCode, uint reserved, uint index, char* productCode);
|
||||
|
||||
[LibraryImport("msi.dll")]
|
||||
private static partial int MsiSetInternalUI(int level, nint window);
|
||||
|
||||
|
||||
@@ -12,7 +12,20 @@ internal static partial class ServiceInterop
|
||||
internal const uint SC_MANAGER_CONNECT = 0x1;
|
||||
internal const uint SC_MANAGER_ALL_ACCESS = 0xF003F;
|
||||
internal const uint SERVICE_QUERY_STATUS = 0x4;
|
||||
internal const uint SERVICE_START = 0x10;
|
||||
internal const uint SERVICE_STOP = 0x20;
|
||||
internal const uint DELETE = 0x10000;
|
||||
internal const uint SERVICE_ALL_ACCESS = 0xF01FF;
|
||||
internal const uint SERVICE_CONTROL_STOP = 0x1;
|
||||
internal const uint SERVICE_STOPPED = 0x1;
|
||||
internal const uint SERVICE_START_PENDING = 0x2;
|
||||
internal const uint SERVICE_STOP_PENDING = 0x3;
|
||||
internal const uint SERVICE_RUNNING = 0x4;
|
||||
internal const int ERROR_SERVICE_ALREADY_RUNNING = 1056;
|
||||
internal const int ERROR_SERVICE_DOES_NOT_EXIST = 1060;
|
||||
internal const int ERROR_SERVICE_CANNOT_ACCEPT_CTRL = 1061;
|
||||
internal const int ERROR_SERVICE_NOT_ACTIVE = 1062;
|
||||
internal const int ERROR_SERVICE_MARKED_FOR_DELETE = 1072;
|
||||
internal const uint SERVICE_WIN32_OWN_PROCESS = 0x10;
|
||||
internal const uint SERVICE_AUTO_START = 0x2;
|
||||
internal const uint SERVICE_ERROR_NORMAL = 0x1;
|
||||
@@ -62,6 +75,18 @@ internal static partial class ServiceInterop
|
||||
public uint dwServiceFlags;
|
||||
}
|
||||
|
||||
[StructLayout(LayoutKind.Sequential)]
|
||||
internal struct SERVICE_STATUS
|
||||
{
|
||||
public uint dwServiceType;
|
||||
public uint dwCurrentState;
|
||||
public uint dwControlsAccepted;
|
||||
public uint dwWin32ExitCode;
|
||||
public uint dwServiceSpecificExitCode;
|
||||
public uint dwCheckPoint;
|
||||
public uint dwWaitHint;
|
||||
}
|
||||
|
||||
[LibraryImport("advapi32.dll", EntryPoint = "OpenSCManagerW", StringMarshalling = StringMarshalling.Utf16, SetLastError = true)]
|
||||
internal static partial IntPtr OpenSCManager(string? machineName, string? databaseName, uint access);
|
||||
|
||||
@@ -119,6 +144,14 @@ internal static partial class ServiceInterop
|
||||
[return: MarshalAs(UnmanagedType.Bool)]
|
||||
internal static partial bool CloseServiceHandle(IntPtr handle);
|
||||
|
||||
[LibraryImport("advapi32.dll", EntryPoint = "StartServiceW", StringMarshalling = StringMarshalling.Utf16, SetLastError = true)]
|
||||
[return: MarshalAs(UnmanagedType.Bool)]
|
||||
internal static partial bool StartService(IntPtr service, int argumentCount, IntPtr arguments);
|
||||
|
||||
[LibraryImport("advapi32.dll", SetLastError = true)]
|
||||
[return: MarshalAs(UnmanagedType.Bool)]
|
||||
internal static partial bool ControlService(IntPtr service, uint control, out SERVICE_STATUS status);
|
||||
|
||||
internal static Win32Exception LastError(string operation)
|
||||
=> new(Marshal.GetLastPInvokeError(), $"{operation} failed");
|
||||
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
using Shared.Contracts.Models;
|
||||
using Shared.Contracts.Serialization;
|
||||
|
||||
namespace Agent.Watchdog.Removal;
|
||||
|
||||
/// <summary>
|
||||
/// The watchdog's side of a removal: on <c>removal.json</c> it starts the detached
|
||||
/// runner and leaves it to it. The runner stops and removes this watchdog, so a
|
||||
/// runner that died can only be noticed if the removal failed and the watchdog is
|
||||
/// still (or again) here; it is then started again, at most <see cref="MaxLaunches"/>
|
||||
/// times. An upgrade under way finishes first; the agent refuses a removal
|
||||
/// during one, so that takes a command and a hand-over crossing.
|
||||
/// </summary>
|
||||
public sealed class RemovalCoordinator(WatchdogOptions options, IRemovalRunnerLauncher? launcher, IAgentService agent, IWatchdogClock clock, WatchdogLog log)
|
||||
{
|
||||
public const int MaxLaunches = 5;
|
||||
|
||||
/// <summary>A runner this watchdog just started is given this long to take its lock.</summary>
|
||||
public static readonly TimeSpan LaunchGrace = TimeSpan.FromMinutes(1);
|
||||
|
||||
private DateTimeOffset launchedUtc = DateTimeOffset.MinValue;
|
||||
private int launches;
|
||||
private bool waitingNoted;
|
||||
|
||||
/// <summary>True while the removal owns this pass: no upgrade step and no supervision.</summary>
|
||||
public async Task<bool> PassAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
if (!File.Exists(options.RemovalPath))
|
||||
{
|
||||
launches = 0;
|
||||
return false;
|
||||
}
|
||||
|
||||
var removal = WatchdogFileStore.Read(options.RemovalPath, WatchdogJsonContext.Default.WatchdogRemoval);
|
||||
if (removal is null)
|
||||
{
|
||||
log.Error($"{options.RemovalPath} does not read; it is removed and nothing was changed");
|
||||
WatchdogFileStore.TryDelete(options.RemovalPath);
|
||||
return false;
|
||||
}
|
||||
|
||||
if (removal.Phase == WatchdogRemovalPhases.Failed)
|
||||
{
|
||||
// Over; the agent reports it and deletes the file.
|
||||
return false;
|
||||
}
|
||||
|
||||
if (File.Exists(options.JournalPath) || File.Exists(options.MsiJournalPath))
|
||||
{
|
||||
if (!waitingNoted)
|
||||
{
|
||||
log.Warn($"Removal {removal.RemovalId} waits for the upgrade under way to end");
|
||||
waitingNoted = true;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
waitingNoted = false;
|
||||
if (removal.RunnerDirectory is { } directory
|
||||
&& (RemovalRunLock.IsHeld(Path.Combine(directory, WatchdogRemovalFiles.RunnerLock)) || clock.UtcNow - launchedUtc < LaunchGrace))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
if (launcher is null)
|
||||
{
|
||||
await GiveUpAsync(removal, "This watchdog runs in its development mode and cannot remove the agent.", cancellationToken);
|
||||
return false;
|
||||
}
|
||||
|
||||
if (removal.Attempts >= WatchdogRemovalFiles.MaxAttempts || launches >= MaxLaunches)
|
||||
{
|
||||
await GiveUpAsync(removal, removal.LastError ?? $"The removal runner stopped before the removal ended, {launches} times.", cancellationToken);
|
||||
return false;
|
||||
}
|
||||
|
||||
if (removal.RunnerDirectory is { } previous)
|
||||
{
|
||||
SwapPaths.TryDeleteDirectory(previous);
|
||||
}
|
||||
|
||||
log.BeginRun();
|
||||
log.Info($"Removal {removal.RemovalId} handed over ({removal.InstallKind} install); starting the runner");
|
||||
var (runnerDirectory, problem) = launcher.Prepare(removal.RemovalId);
|
||||
if (problem is null)
|
||||
{
|
||||
WatchdogFileStore.Write(options.RemovalPath,
|
||||
removal with { Phase = WatchdogRemovalPhases.Uninstalling, RunnerDirectory = runnerDirectory },
|
||||
WatchdogJsonContext.Default.WatchdogRemoval);
|
||||
problem = launcher.Start(removal.RemovalId, runnerDirectory!);
|
||||
}
|
||||
|
||||
launches++;
|
||||
if (problem is not null)
|
||||
{
|
||||
await GiveUpAsync(removal, $"The removal runner did not start: {problem}.", cancellationToken);
|
||||
return false;
|
||||
}
|
||||
|
||||
launchedUtc = clock.UtcNow;
|
||||
return true;
|
||||
}
|
||||
|
||||
private async Task GiveUpAsync(WatchdogRemoval removal, string why, CancellationToken cancellationToken)
|
||||
{
|
||||
log.Error($"Removal {removal.RemovalId} gave up: {why}");
|
||||
WatchdogFileStore.Write(options.RemovalPath,
|
||||
removal with { Phase = WatchdogRemovalPhases.Failed, LastError = why },
|
||||
WatchdogJsonContext.Default.WatchdogRemoval);
|
||||
try
|
||||
{
|
||||
var status = await agent.QueryAsync(cancellationToken);
|
||||
if (status.State != AgentRunState.NotInstalled && !status.IsRunning)
|
||||
{
|
||||
await agent.StartAsync(cancellationToken);
|
||||
}
|
||||
}
|
||||
catch (Exception exception) when (exception is not OperationCanceledException || !cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
log.Warn("The agent could not be started", exception);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,484 @@
|
||||
using System.ComponentModel;
|
||||
using System.Diagnostics;
|
||||
using System.Runtime.InteropServices;
|
||||
using System.Runtime.Versioning;
|
||||
using System.Security.Cryptography.X509Certificates;
|
||||
using System.Text.Json.Nodes;
|
||||
using Agent.Watchdog.Msi;
|
||||
using Agent.Watchdog.Platform;
|
||||
using Microsoft.Win32;
|
||||
using Shared.Contracts.Models;
|
||||
using Shared.Infrastructure.Constants;
|
||||
using static Agent.Watchdog.Platform.ServiceInterop;
|
||||
|
||||
namespace Agent.Watchdog.Removal;
|
||||
|
||||
/// <summary>The services a portable install's removal takes out (an MSI's are the MSI's to remove); replaced in tests.</summary>
|
||||
public interface IRemovalServices
|
||||
{
|
||||
Task StopWatchdogAsync(CancellationToken cancellationToken);
|
||||
|
||||
Task StartWatchdogAsync(CancellationToken cancellationToken);
|
||||
|
||||
/// <summary>Removes the agent's and the watchdog's registrations; nothing to do for one already gone.</summary>
|
||||
Task RemoveAsync(CancellationToken cancellationToken);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Hidden entry point of the runner, for both binaries that have it: the runner
|
||||
/// uses no agent code and no watchdog-only code, so the agent can fall back on it
|
||||
/// when no watchdog can take a removal. Logs to its own folder, since the agent's
|
||||
/// log folder goes with the install.
|
||||
/// </summary>
|
||||
public static class RemovalHost
|
||||
{
|
||||
public static async Task<int> RunAsync(Guid removalId, string[] args)
|
||||
{
|
||||
string? Value(string name)
|
||||
{
|
||||
for (var i = 0; i < args.Length - 1; i++)
|
||||
{
|
||||
if (string.Equals(args[i], name, StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return args[i + 1];
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
if (Value("--install-dir") is not { } install || Value("--data-dir") is not { } data || Value("--runner-dir") is not { } runnerDirectory)
|
||||
{
|
||||
Console.Error.WriteLine("Usage: --install-dir <path> --data-dir <path> --runner-dir <path>");
|
||||
return 1;
|
||||
}
|
||||
|
||||
var options = new WatchdogOptions
|
||||
{
|
||||
InstallDirectory = Path.GetFullPath(install),
|
||||
DataDirectory = Path.GetFullPath(data),
|
||||
SelfPath = null,
|
||||
};
|
||||
var log = new WatchdogLog(runnerDirectory, console: false, name: RemovalRunner.LogFileName);
|
||||
var agentPath = Path.Combine(options.InstallDirectory, options.AgentExecutable);
|
||||
|
||||
IAgentService agent;
|
||||
IRemovalServices services;
|
||||
IRemovalMachine machine;
|
||||
IWindowsInstaller? installer = null;
|
||||
if (OperatingSystem.IsWindows())
|
||||
{
|
||||
agent = new ScmService(Branding.AgentServiceName, agentPath, log);
|
||||
services = new WindowsRemovalServices(
|
||||
(ScmService)agent,
|
||||
new ScmService(Branding.WatchdogServiceName, Path.Combine(options.InstallDirectory, Branding.WatchdogExecutableName), log));
|
||||
machine = new WindowsRemovalMachine(log);
|
||||
installer = new WindowsInstallerApi();
|
||||
}
|
||||
else if (OperatingSystem.IsMacOS())
|
||||
{
|
||||
agent = new LaunchdAgentService(Branding.AgentLaunchdLabel);
|
||||
services = new UnixRemovalServices();
|
||||
machine = new UnixRemovalMachine(options, log);
|
||||
}
|
||||
else
|
||||
{
|
||||
agent = new SystemdAgentService(Branding.AgentSystemdUnit);
|
||||
services = new UnixRemovalServices();
|
||||
machine = new UnixRemovalMachine(options, log);
|
||||
}
|
||||
|
||||
var clock = SystemClock.Instance;
|
||||
var purge = new RemovalPurge(options, machine, clock, log, runnerDirectory);
|
||||
var runner = new RemovalRunner(options, agent, services, installer, purge, clock, log, runnerDirectory);
|
||||
return await runner.RunAsync(removalId, CancellationToken.None);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>A Windows service through the SCM's own APIs (never sc.exe, and none of ServiceController, which the agent's linux/macOS build lacks).</summary>
|
||||
[SupportedOSPlatform("windows")]
|
||||
public sealed class ScmService(string serviceName, string executablePath, WatchdogLog log) : IAgentService
|
||||
{
|
||||
private static readonly TimeSpan StateChangeTimeout = TimeSpan.FromSeconds(60);
|
||||
|
||||
public Task<AgentServiceStatus> QueryAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
if (QueryStatus(serviceName) is not { } status)
|
||||
{
|
||||
return Task.FromResult(AgentServiceStatus.NotInstalled);
|
||||
}
|
||||
|
||||
var state = status.dwCurrentState switch
|
||||
{
|
||||
SERVICE_STOPPED => AgentRunState.Stopped,
|
||||
SERVICE_START_PENDING => AgentRunState.Starting,
|
||||
SERVICE_STOP_PENDING => AgentRunState.Stopping,
|
||||
_ => AgentRunState.Running,
|
||||
};
|
||||
return Task.FromResult(new AgentServiceStatus(state, true, false));
|
||||
}
|
||||
|
||||
public async Task StartAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
WithService(SERVICE_START, service =>
|
||||
{
|
||||
if (!StartService(service, 0, IntPtr.Zero) && Marshal.GetLastPInvokeError() != ERROR_SERVICE_ALREADY_RUNNING)
|
||||
{
|
||||
throw LastError($"StartService({serviceName})");
|
||||
}
|
||||
});
|
||||
await WaitForAsync(SERVICE_RUNNING, cancellationToken);
|
||||
}
|
||||
|
||||
public async Task StopAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
WithService(SERVICE_STOP, service =>
|
||||
{
|
||||
if (!ControlService(service, SERVICE_CONTROL_STOP, out _)
|
||||
&& Marshal.GetLastPInvokeError() is not (ERROR_SERVICE_NOT_ACTIVE or ERROR_SERVICE_CANNOT_ACCEPT_CTRL))
|
||||
{
|
||||
throw LastError($"ControlService({serviceName})");
|
||||
}
|
||||
});
|
||||
await WaitForAsync(SERVICE_STOPPED, cancellationToken);
|
||||
await StopLeftoversAsync(cancellationToken);
|
||||
}
|
||||
|
||||
/// <summary>Removes the registration: gone once the last handle on it closes.</summary>
|
||||
public void Delete() => WithService(DELETE, service =>
|
||||
{
|
||||
if (!DeleteService(service) && Marshal.GetLastPInvokeError() != ERROR_SERVICE_MARKED_FOR_DELETE)
|
||||
{
|
||||
throw LastError($"DeleteService({serviceName})");
|
||||
}
|
||||
|
||||
log.Info($"Service {serviceName} removed");
|
||||
});
|
||||
|
||||
/// <summary>Runs <paramref name="action"/> on the open service; a service that is not there is nothing to do.</summary>
|
||||
private void WithService(uint access, Action<IntPtr> action)
|
||||
{
|
||||
var scm = OpenSCManager(null, null, SC_MANAGER_CONNECT);
|
||||
if (scm == IntPtr.Zero)
|
||||
{
|
||||
throw LastError("OpenSCManager");
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var service = OpenService(scm, serviceName, access);
|
||||
if (service == IntPtr.Zero)
|
||||
{
|
||||
if (Marshal.GetLastPInvokeError() == ERROR_SERVICE_DOES_NOT_EXIST)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
throw LastError($"OpenService({serviceName})");
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
action(service);
|
||||
}
|
||||
finally
|
||||
{
|
||||
CloseServiceHandle(service);
|
||||
}
|
||||
}
|
||||
finally
|
||||
{
|
||||
CloseServiceHandle(scm);
|
||||
}
|
||||
}
|
||||
|
||||
private async Task WaitForAsync(uint state, CancellationToken cancellationToken)
|
||||
{
|
||||
var deadline = DateTime.UtcNow + StateChangeTimeout;
|
||||
while (QueryStatus(serviceName) is { } status && status.dwCurrentState != state)
|
||||
{
|
||||
if (DateTime.UtcNow >= deadline)
|
||||
{
|
||||
throw new System.TimeoutException($"{serviceName} did not reach state {state} within {StateChangeTimeout.TotalSeconds:0} s");
|
||||
}
|
||||
|
||||
await Task.Delay(250, cancellationToken);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>The service process can outlive its stop report, and the agent's own children hold its binary: wait for them, then end what is left.</summary>
|
||||
private async Task StopLeftoversAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
var name = Path.GetFileNameWithoutExtension(executablePath);
|
||||
var deadline = DateTime.UtcNow.AddSeconds(10);
|
||||
while (true)
|
||||
{
|
||||
var leftovers = Process.GetProcessesByName(name).Where(IsThisBinary).ToList();
|
||||
if (leftovers.Count == 0)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var late = DateTime.UtcNow >= deadline;
|
||||
foreach (var process in leftovers)
|
||||
{
|
||||
try
|
||||
{
|
||||
if (late)
|
||||
{
|
||||
log.Warn($"Ending process {process.Id} of {name} that outlived its service");
|
||||
process.Kill(entireProcessTree: true);
|
||||
}
|
||||
}
|
||||
catch (Exception exception) when (exception is InvalidOperationException or Win32Exception)
|
||||
{
|
||||
}
|
||||
finally
|
||||
{
|
||||
process.Dispose();
|
||||
}
|
||||
}
|
||||
|
||||
if (late)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
await Task.Delay(500, cancellationToken);
|
||||
}
|
||||
}
|
||||
|
||||
private bool IsThisBinary(Process process)
|
||||
{
|
||||
try
|
||||
{
|
||||
return string.Equals(process.MainModule?.FileName, executablePath, StringComparison.OrdinalIgnoreCase);
|
||||
}
|
||||
catch (Exception exception) when (exception is InvalidOperationException or Win32Exception)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
[SupportedOSPlatform("windows")]
|
||||
public sealed class WindowsRemovalServices(ScmService agent, ScmService watchdog) : IRemovalServices
|
||||
{
|
||||
public Task StopWatchdogAsync(CancellationToken cancellationToken) => watchdog.StopAsync(cancellationToken);
|
||||
|
||||
public async Task StartWatchdogAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
if ((await watchdog.QueryAsync(cancellationToken)) is { IsRunning: false, State: not AgentRunState.NotInstalled })
|
||||
{
|
||||
await watchdog.StartAsync(cancellationToken);
|
||||
}
|
||||
}
|
||||
|
||||
public Task RemoveAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
agent.Delete();
|
||||
watchdog.Delete();
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>The normal registry key, the Local Machine certificate stores, and restart-time deletes.</summary>
|
||||
[SupportedOSPlatform("windows")]
|
||||
public sealed partial class WindowsRemovalMachine(WatchdogLog log) : IRemovalMachine
|
||||
{
|
||||
private const uint MoveFileDelayUntilReboot = 0x4;
|
||||
|
||||
public void RemoveConfiguration() => Registry.LocalMachine.DeleteSubKeyTree(Branding.AgentRegistryKey, throwOnMissingSubKey: false);
|
||||
|
||||
public bool ConfigurationRemains
|
||||
{
|
||||
get
|
||||
{
|
||||
using var key = Registry.LocalMachine.OpenSubKey(Branding.AgentRegistryKey);
|
||||
return key is not null;
|
||||
}
|
||||
}
|
||||
|
||||
public void RemoveManagedCa(IReadOnlyCollection<string> thumbprints)
|
||||
{
|
||||
foreach (var name in new[] { StoreName.Root, StoreName.CertificateAuthority })
|
||||
{
|
||||
using var store = new X509Store(name, StoreLocation.LocalMachine);
|
||||
store.Open(OpenFlags.ReadWrite);
|
||||
foreach (var thumbprint in thumbprints)
|
||||
{
|
||||
foreach (var certificate in store.Certificates.Find(X509FindType.FindByThumbprint, thumbprint, validOnly: false))
|
||||
{
|
||||
store.Remove(certificate);
|
||||
log.Info($"Removed the server CA {certificate.Subject} from LocalMachine\\{name}");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public void DeleteAtRestart(string path)
|
||||
{
|
||||
if (!MoveFileExW(path, null, MoveFileDelayUntilReboot))
|
||||
{
|
||||
log.Warn($"{path} could not be scheduled for deletion at the next restart (error {Marshal.GetLastPInvokeError()})");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Held by the running binary until it exits; its log stays for support until the restart that deletes it.</summary>
|
||||
public void RemoveRunnerDirectory(string directory)
|
||||
{
|
||||
try
|
||||
{
|
||||
foreach (var file in Directory.EnumerateFiles(directory, "*", SearchOption.AllDirectories))
|
||||
{
|
||||
DeleteAtRestart(file);
|
||||
}
|
||||
|
||||
DeleteAtRestart(directory);
|
||||
}
|
||||
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
|
||||
{
|
||||
log.Warn($"{directory} could not be scheduled for deletion at the next restart", exception);
|
||||
}
|
||||
}
|
||||
|
||||
[LibraryImport("kernel32.dll", StringMarshalling = StringMarshalling.Utf16, SetLastError = true)]
|
||||
[return: MarshalAs(UnmanagedType.Bool)]
|
||||
private static partial bool MoveFileExW(string existingFileName, string? newFileName, uint flags);
|
||||
}
|
||||
|
||||
/// <summary>Both units or daemons, run from a runner that is outside either's cgroup or process group.</summary>
|
||||
public sealed class UnixRemovalServices : IRemovalServices
|
||||
{
|
||||
public Task StopWatchdogAsync(CancellationToken cancellationToken) => Task.Run(UnixWatchdogRegistration.Stop, cancellationToken);
|
||||
|
||||
public Task StartWatchdogAsync(CancellationToken cancellationToken) => Task.Run(UnixWatchdogRegistration.Start, cancellationToken);
|
||||
|
||||
public Task RemoveAsync(CancellationToken cancellationToken) => Task.Run(() =>
|
||||
{
|
||||
if (OperatingSystem.IsMacOS())
|
||||
{
|
||||
Command.Run("launchctl", "bootout", $"system/{Branding.AgentLaunchdLabel}");
|
||||
DeleteIfThere($"/Library/LaunchDaemons/{Branding.AgentLaunchdLabel}.plist");
|
||||
}
|
||||
else
|
||||
{
|
||||
Command.Run("systemctl", "disable", "--now", Branding.AgentSystemdUnit);
|
||||
DeleteIfThere($"/etc/systemd/system/{Branding.AgentSystemdUnit}.service");
|
||||
}
|
||||
|
||||
UnixWatchdogRegistration.Remove();
|
||||
}, cancellationToken);
|
||||
|
||||
private static void DeleteIfThere(string path)
|
||||
{
|
||||
if (File.Exists(path))
|
||||
{
|
||||
File.Delete(path);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary><c>agent.json</c> (its policy section stays), the PATH link, and the trust anchors ServerTrust wrote.</summary>
|
||||
public sealed class UnixRemovalMachine(WatchdogOptions options, WatchdogLog log) : IRemovalMachine
|
||||
{
|
||||
private const string PathLink = "/usr/local/bin/" + Branding.AgentUnixExecutableName;
|
||||
private const string AnchorName = "executer-server-ca.crt";
|
||||
private const string DebianAnchors = "/usr/local/share/ca-certificates/";
|
||||
private const string RedHatAnchors = "/etc/pki/ca-trust/source/anchors/";
|
||||
|
||||
private static string PolicySection => Branding.AgentPolicyRegistryKey.Replace('\\', '/');
|
||||
|
||||
public void RemoveConfiguration()
|
||||
{
|
||||
var path = Branding.AgentUnixConfigFile;
|
||||
if (new FileInfo(PathLink).LinkTarget is { } target
|
||||
&& (SwapPaths.IsInside(target, options.InstallDirectory) || target.Contains("/executer/", StringComparison.Ordinal)))
|
||||
{
|
||||
File.Delete(PathLink);
|
||||
}
|
||||
|
||||
if (File.Exists(path))
|
||||
{
|
||||
var kept = new JsonObject();
|
||||
try
|
||||
{
|
||||
if (JsonNode.Parse(File.ReadAllText(path)) is JsonObject root
|
||||
&& root.FirstOrDefault(p => string.Equals(p.Key, PolicySection, StringComparison.OrdinalIgnoreCase)) is { Value: JsonObject policy } section)
|
||||
{
|
||||
kept[section.Key] = policy.DeepClone();
|
||||
}
|
||||
}
|
||||
catch (System.Text.Json.JsonException)
|
||||
{
|
||||
log.Warn($"{path} does not read; it goes whole");
|
||||
}
|
||||
|
||||
if (kept.Count > 0)
|
||||
{
|
||||
File.WriteAllText(path, kept.ToJsonString());
|
||||
return;
|
||||
}
|
||||
|
||||
File.Delete(path);
|
||||
}
|
||||
|
||||
if (Path.GetDirectoryName(path) is { } folder)
|
||||
{
|
||||
SwapPaths.TryDeleteIfEmpty(folder);
|
||||
}
|
||||
}
|
||||
|
||||
public bool ConfigurationRemains
|
||||
{
|
||||
get
|
||||
{
|
||||
if (!File.Exists(Branding.AgentUnixConfigFile))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
// Only the policy section may stay.
|
||||
return JsonNode.Parse(File.ReadAllText(Branding.AgentUnixConfigFile)) is not JsonObject root
|
||||
|| root.Any(p => !string.Equals(p.Key, PolicySection, StringComparison.OrdinalIgnoreCase));
|
||||
}
|
||||
catch (System.Text.Json.JsonException)
|
||||
{
|
||||
return true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public void RemoveManagedCa(IReadOnlyCollection<string> thumbprints)
|
||||
{
|
||||
if (OperatingSystem.IsMacOS())
|
||||
{
|
||||
foreach (var thumbprint in thumbprints)
|
||||
{
|
||||
Command.Run("security", "delete-certificate", "-Z", thumbprint, "/Library/Keychains/System.keychain");
|
||||
}
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
foreach (var (folder, refresh) in new[] { (DebianAnchors, "update-ca-certificates"), (RedHatAnchors, "update-ca-trust") })
|
||||
{
|
||||
var anchor = folder + AnchorName;
|
||||
if (File.Exists(anchor))
|
||||
{
|
||||
File.Delete(anchor);
|
||||
Command.Run(refresh);
|
||||
log.Info($"Removed the server CA {anchor}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public void DeleteAtRestart(string path)
|
||||
{
|
||||
// Unlinking works here even for a file in use; a sweep already tried.
|
||||
}
|
||||
|
||||
public void RemoveRunnerDirectory(string directory) => SwapPaths.TryDeleteDirectory(directory);
|
||||
}
|
||||
@@ -0,0 +1,247 @@
|
||||
using Shared.Contracts.Models;
|
||||
using Shared.Infrastructure.Constants;
|
||||
|
||||
namespace Agent.Watchdog.Removal;
|
||||
|
||||
/// <summary>What a removal wipes outside the install and data folders; replaced in tests.</summary>
|
||||
public interface IRemovalMachine
|
||||
{
|
||||
/// <summary>
|
||||
/// The agent's configuration and runtime state: the normal registry key tree
|
||||
/// (<c>agent.json</c> and the PATH link on linux/macOS). The policy key is
|
||||
/// group policy's and stays.
|
||||
/// </summary>
|
||||
void RemoveConfiguration();
|
||||
|
||||
bool ConfigurationRemains { get; }
|
||||
|
||||
/// <summary>The server CA certificates the agent installed itself (<see cref="WatchdogFiles.ManagedCa"/>).</summary>
|
||||
void RemoveManagedCa(IReadOnlyCollection<string> thumbprints);
|
||||
|
||||
/// <summary>Deletes the file or empty folder at the next restart (nothing to do on linux/macOS, where nothing is locked).</summary>
|
||||
void DeleteAtRestart(string path);
|
||||
|
||||
/// <summary>The runner's folder, which holds the running binary: now where that works, otherwise at the next restart.</summary>
|
||||
void RemoveRunnerDirectory(string directory);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Verifies and finishes the wipe, whichever way the product went. The MSI's
|
||||
/// <c>purge-credentials</c> custom action runs with <c>Return="ignore"</c>, so
|
||||
/// its success proves nothing: <c>credentials.dat</c>, the configuration, the
|
||||
/// data and install folders (<c>Logs</c> with them) are each checked and removed
|
||||
/// here; what is still locked after a minute goes at the next restart.
|
||||
/// </summary>
|
||||
public sealed class RemovalPurge(WatchdogOptions options, IRemovalMachine machine, IWatchdogClock clock, WatchdogLog log, string runnerDirectory)
|
||||
{
|
||||
public const int Sweeps = 7;
|
||||
public static readonly TimeSpan SweepDelay = TimeSpan.FromSeconds(10);
|
||||
|
||||
private readonly List<string> thumbprints = [];
|
||||
|
||||
private string CredentialPath => Path.Combine(options.DataDirectory, Branding.CredentialFileName);
|
||||
|
||||
private string ManagedCaPath => Path.Combine(options.DataDirectory, WatchdogFiles.ManagedCa);
|
||||
|
||||
private string ManagedCaCopy => Path.Combine(runnerDirectory, WatchdogFiles.ManagedCa);
|
||||
|
||||
/// <summary>Takes what the purge needs later from the data folder, into the runner's folder (a resumed runner finds it there).</summary>
|
||||
public void Snapshot()
|
||||
{
|
||||
try
|
||||
{
|
||||
if (File.Exists(ManagedCaPath))
|
||||
{
|
||||
File.Copy(ManagedCaPath, ManagedCaCopy, overwrite: true);
|
||||
}
|
||||
|
||||
thumbprints.Clear();
|
||||
if (File.Exists(ManagedCaCopy))
|
||||
{
|
||||
thumbprints.AddRange(File.ReadAllLines(ManagedCaCopy).Select(l => l.Trim()).Where(l => l.Length > 0));
|
||||
}
|
||||
}
|
||||
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
|
||||
{
|
||||
log.Warn("The list of installed server CAs could not be read; they stay", exception);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Why the install folder must not be wiped; null when it may. Checked before a portable install's services go.</summary>
|
||||
public string? InstallFolderRefusal()
|
||||
{
|
||||
if (IsProtectedFolder(options.InstallDirectory))
|
||||
{
|
||||
return $"The agent runs from {options.InstallDirectory}, which is not a folder of its own; it was left alone.";
|
||||
}
|
||||
|
||||
return File.Exists(Path.Combine(options.InstallDirectory, options.AgentExecutable))
|
||||
? null
|
||||
: $"{options.InstallDirectory} holds no {options.AgentExecutable}; it was left alone.";
|
||||
}
|
||||
|
||||
/// <summary>Removes what is left; what stays after every sweep is listed and goes at the next restart.</summary>
|
||||
public async Task<IReadOnlyList<string>> FinishAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
var left = new List<string>();
|
||||
for (var sweep = 1; sweep <= Sweeps; sweep++)
|
||||
{
|
||||
left = Sweep();
|
||||
if (left.Count == 0)
|
||||
{
|
||||
return left;
|
||||
}
|
||||
|
||||
if (sweep < Sweeps)
|
||||
{
|
||||
log.Info($"Still there (sweep {sweep} of {Sweeps}): {string.Join("; ", left)}");
|
||||
await clock.DelayAsync(SweepDelay, cancellationToken);
|
||||
}
|
||||
}
|
||||
|
||||
foreach (var folder in new[] { options.DataDirectory, options.InstallDirectory })
|
||||
{
|
||||
if (!IsProtectedFolder(folder))
|
||||
{
|
||||
ScheduleAtRestart(folder);
|
||||
}
|
||||
}
|
||||
|
||||
return left;
|
||||
}
|
||||
|
||||
public void RemoveRunnerDirectory(string directory) => machine.RemoveRunnerDirectory(directory);
|
||||
|
||||
private List<string> Sweep()
|
||||
{
|
||||
var left = new List<string>();
|
||||
|
||||
// First and on its own: nothing else on the device can authenticate as it.
|
||||
Try("credentials", () =>
|
||||
{
|
||||
if (File.Exists(CredentialPath))
|
||||
{
|
||||
File.SetAttributes(CredentialPath, FileAttributes.Normal);
|
||||
File.Delete(CredentialPath);
|
||||
}
|
||||
}, left);
|
||||
Try("configuration", machine.RemoveConfiguration, left);
|
||||
if (thumbprints.Count > 0)
|
||||
{
|
||||
Try("server CA", () => machine.RemoveManagedCa(thumbprints), left);
|
||||
}
|
||||
|
||||
Try(options.DataDirectory, () => DeleteFolder(options.DataDirectory), left);
|
||||
Try(options.InstallDirectory, () => DeleteFolder(options.InstallDirectory), left);
|
||||
if (Path.GetDirectoryName(Path.TrimEndingDirectorySeparator(options.InstallDirectory)) is { } parent
|
||||
&& !IsProtectedFolder(parent)
|
||||
&& string.Equals(Path.GetFileName(parent), Branding.ProductKey, StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
// %ProgramFiles%\ExecuteR, /opt/executer: only ever ours, when nothing else is in it.
|
||||
SwapPaths.TryDeleteIfEmpty(parent);
|
||||
}
|
||||
|
||||
if (File.Exists(CredentialPath) && !left.Contains("credentials"))
|
||||
{
|
||||
left.Add("credentials");
|
||||
}
|
||||
|
||||
if (machine.ConfigurationRemains && !left.Contains("configuration"))
|
||||
{
|
||||
left.Add("configuration");
|
||||
}
|
||||
|
||||
return left;
|
||||
}
|
||||
|
||||
private void Try(string what, Action action, List<string> left)
|
||||
{
|
||||
try
|
||||
{
|
||||
action();
|
||||
}
|
||||
catch (Exception exception) when (exception is not OperationCanceledException)
|
||||
{
|
||||
log.Warn($"{what} could not be removed", exception);
|
||||
left.Add(what);
|
||||
}
|
||||
}
|
||||
|
||||
private static void DeleteFolder(string folder)
|
||||
{
|
||||
if (!Directory.Exists(folder))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
if (IsProtectedFolder(folder))
|
||||
{
|
||||
throw new InvalidOperationException($"{folder} is not a folder of the agent's own");
|
||||
}
|
||||
|
||||
foreach (var file in Directory.EnumerateFiles(folder, "*", SearchOption.AllDirectories))
|
||||
{
|
||||
// Directory.Delete stops at a read-only file on Windows.
|
||||
if ((File.GetAttributes(file) & FileAttributes.ReadOnly) != 0)
|
||||
{
|
||||
File.SetAttributes(file, FileAttributes.Normal);
|
||||
}
|
||||
}
|
||||
|
||||
Directory.Delete(folder, recursive: true);
|
||||
}
|
||||
|
||||
private void ScheduleAtRestart(string folder)
|
||||
{
|
||||
try
|
||||
{
|
||||
if (!Directory.Exists(folder))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
foreach (var file in Directory.EnumerateFiles(folder, "*", SearchOption.AllDirectories))
|
||||
{
|
||||
machine.DeleteAtRestart(file);
|
||||
}
|
||||
|
||||
// Deepest first: a folder goes only once it is empty.
|
||||
foreach (var directory in Directory.EnumerateDirectories(folder, "*", SearchOption.AllDirectories).OrderByDescending(d => d.Length))
|
||||
{
|
||||
machine.DeleteAtRestart(directory);
|
||||
}
|
||||
|
||||
machine.DeleteAtRestart(folder);
|
||||
}
|
||||
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
|
||||
{
|
||||
log.Warn($"{folder} could not be scheduled for the next restart", exception);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>A root, or a folder the system or other software shares: a portable agent unpacked straight into one never takes it along.</summary>
|
||||
public static bool IsProtectedFolder(string folder)
|
||||
{
|
||||
var full = Path.TrimEndingDirectorySeparator(Path.GetFullPath(folder));
|
||||
if (string.IsNullOrEmpty(Path.GetFileName(full)))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
IEnumerable<string> shared = OperatingSystem.IsWindows()
|
||||
?
|
||||
[
|
||||
Environment.GetFolderPath(Environment.SpecialFolder.Windows),
|
||||
Environment.GetFolderPath(Environment.SpecialFolder.System),
|
||||
Environment.GetFolderPath(Environment.SpecialFolder.ProgramFiles),
|
||||
Environment.GetFolderPath(Environment.SpecialFolder.ProgramFilesX86),
|
||||
Environment.GetFolderPath(Environment.SpecialFolder.CommonProgramFiles),
|
||||
Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData),
|
||||
Environment.GetFolderPath(Environment.SpecialFolder.UserProfile),
|
||||
Path.Combine(Path.GetPathRoot(full) ?? "", "Users"),
|
||||
]
|
||||
: ["/bin", "/boot", "/dev", "/etc", "/home", "/lib", "/lib64", "/opt", "/proc", "/root", "/run", "/sbin", "/sys", "/tmp", "/usr", "/usr/bin",
|
||||
"/usr/lib", "/usr/local", "/usr/local/bin", "/usr/sbin", "/var", "/var/lib", "/var/tmp", "/Applications", "/Library", "/System", "/Users"];
|
||||
return shared.Any(path => path.Length > 0 && SwapPaths.Comparer.Equals(Path.TrimEndingDirectorySeparator(path), full));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,344 @@
|
||||
using Agent.Watchdog.Msi;
|
||||
using Shared.Contracts.Models;
|
||||
using Shared.Contracts.Serialization;
|
||||
using Shared.Infrastructure.Constants;
|
||||
|
||||
namespace Agent.Watchdog.Removal;
|
||||
|
||||
/// <summary>Timings of a removal; shortened in tests.</summary>
|
||||
public sealed record RemovalRunnerSettings
|
||||
{
|
||||
/// <summary>Between tries while Windows Installer is busy with another installation (1618).</summary>
|
||||
public TimeSpan BusyRetryDelay { get; init; } = TimeSpan.FromSeconds(30);
|
||||
|
||||
/// <summary>How long a busy Windows Installer is waited out within one attempt.</summary>
|
||||
public TimeSpan BusyRetryFor { get; init; } = TimeSpan.FromMinutes(10);
|
||||
|
||||
/// <summary>The longest one attempt's uninstall may take (real time: Windows Installer's call cannot be cut short).</summary>
|
||||
public TimeSpan AttemptTimeout { get; init; } = WatchdogFiles.MsiInstallTimeout;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The detached removal runner (<c>ExecuteR-Watchdog.exe remove-run <id></c>, or
|
||||
/// <c>ExecuteR-Agent.exe remove-self <id></c> where no watchdog can take it),
|
||||
/// started from a copy of its binary outside the install and data folders, since
|
||||
/// it wipes both. It owns the removal in <c>removal.json</c> while it holds
|
||||
/// <c>removal.lock</c> in its own folder:
|
||||
/// <list type="number">
|
||||
/// <item>stop the agent;</item>
|
||||
/// <item>MSI install: Windows Installer in process, by product code
|
||||
/// (<c>REBOOT=ReallySuppress</c>, no UI, verbose log in the runner's folder);
|
||||
/// 0, 3010, 1641 and 1605 (already gone) are success, 1618 is tried again every
|
||||
/// 30 s for up to 10 minutes. Portable install: the watchdog and agent services
|
||||
/// go through the service manager;</item>
|
||||
/// <item>the purge is verified and finished (<see cref="RemovalPurge"/>): whatever
|
||||
/// the MSI's own purge left is removed, and what is locked goes at the next restart.</item>
|
||||
/// </list>
|
||||
/// A failed attempt starts the services again and the next one follows 5 minutes
|
||||
/// later, three in all; then the phase says failed, with the reason, for the
|
||||
/// agent to report, and the device carries on. Unlike <see cref="MsiRunner"/> this
|
||||
/// runner writes no outcome and verifies no health: on success nobody is left.
|
||||
/// </summary>
|
||||
public sealed class RemovalRunner(
|
||||
WatchdogOptions options,
|
||||
IAgentService agent,
|
||||
IRemovalServices services,
|
||||
IWindowsInstaller? installer,
|
||||
RemovalPurge purge,
|
||||
IWatchdogClock clock,
|
||||
WatchdogLog log,
|
||||
string runnerDirectory,
|
||||
RemovalRunnerSettings? settings = null)
|
||||
{
|
||||
public const string UninstallProperties = "REBOOT=ReallySuppress";
|
||||
public const string LogFileName = "removal";
|
||||
|
||||
private readonly RemovalRunnerSettings timings = settings ?? new RemovalRunnerSettings();
|
||||
|
||||
private string LockPath => Path.Combine(runnerDirectory, WatchdogRemovalFiles.RunnerLock);
|
||||
|
||||
/// <summary>Runs (or resumes) the removal; 0 unless <c>removal.json</c> does not name it.</summary>
|
||||
public async Task<int> RunAsync(Guid removalId, CancellationToken cancellationToken)
|
||||
{
|
||||
bool gone;
|
||||
using (var held = RemovalRunLock.TryHold(LockPath))
|
||||
{
|
||||
if (held is null)
|
||||
{
|
||||
log.Info($"Another runner owns removal {removalId}; this one exits");
|
||||
return 0;
|
||||
}
|
||||
|
||||
var removal = WatchdogFileStore.Read(options.RemovalPath, WatchdogJsonContext.Default.WatchdogRemoval);
|
||||
if (removal is null || removal.RemovalId != removalId)
|
||||
{
|
||||
log.Warn($"Removal {removalId} is not what {options.RemovalPath} names; nothing to do");
|
||||
return removal is null ? 1 : 0;
|
||||
}
|
||||
|
||||
log.BeginRun();
|
||||
log.Info($"Runner {Environment.ProcessId} owns removal {removalId} ({removal.InstallKind} install, {removal.Attempts} attempts so far)");
|
||||
|
||||
// The MSI's own purge takes the data folder this reads.
|
||||
purge.Snapshot();
|
||||
|
||||
gone = await AttemptAllAsync(removal, cancellationToken);
|
||||
if (gone)
|
||||
{
|
||||
var left = await purge.FinishAsync(cancellationToken);
|
||||
log.Info(left.Count == 0
|
||||
? $"Removal {removalId} is complete; nothing is left"
|
||||
: $"Removal {removalId} is complete, except what goes at the next restart: {string.Join("; ", left)}");
|
||||
}
|
||||
}
|
||||
|
||||
purge.RemoveRunnerDirectory(runnerDirectory);
|
||||
return 0;
|
||||
}
|
||||
|
||||
/// <summary>True once the product and its services are gone; false when every attempt failed.</summary>
|
||||
private async Task<bool> AttemptAllAsync(WatchdogRemoval removal, CancellationToken cancellationToken)
|
||||
{
|
||||
while (true)
|
||||
{
|
||||
if (removal.Attempts >= WatchdogRemovalFiles.MaxAttempts)
|
||||
{
|
||||
Finish(removal with { LastError = removal.LastError ?? "The removal runner stopped before it ended." });
|
||||
return false;
|
||||
}
|
||||
|
||||
removal = removal with { Phase = WatchdogRemovalPhases.Uninstalling, Attempts = removal.Attempts + 1 };
|
||||
Write(removal);
|
||||
log.Info($"Removing the agent (attempt {removal.Attempts} of {WatchdogRemovalFiles.MaxAttempts})");
|
||||
|
||||
string? problem;
|
||||
try
|
||||
{
|
||||
problem = await AttemptAsync(removal, cancellationToken);
|
||||
}
|
||||
catch (Exception exception) when (exception is not OperationCanceledException)
|
||||
{
|
||||
problem = $"The removal failed ({exception.GetBaseException().Message}).";
|
||||
}
|
||||
|
||||
if (problem is null)
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
log.Error($"Attempt {removal.Attempts} failed: {problem}");
|
||||
await RestoreAsync(cancellationToken);
|
||||
|
||||
removal = removal with { LastError = problem };
|
||||
if (removal.Attempts >= WatchdogRemovalFiles.MaxAttempts)
|
||||
{
|
||||
Finish(removal);
|
||||
return false;
|
||||
}
|
||||
|
||||
Write(removal);
|
||||
log.Info($"The next attempt follows in {WatchdogRemovalFiles.AttemptDelay.TotalMinutes:0} minutes; the agent runs meanwhile");
|
||||
await clock.DelayAsync(WatchdogRemovalFiles.AttemptDelay, cancellationToken);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>One attempt; null when the agent is gone, otherwise why not.</summary>
|
||||
private async Task<string?> AttemptAsync(WatchdogRemoval removal, CancellationToken cancellationToken)
|
||||
{
|
||||
try
|
||||
{
|
||||
await agent.StopAsync(cancellationToken);
|
||||
}
|
||||
catch (Exception exception) when (exception is not OperationCanceledException)
|
||||
{
|
||||
// Windows Installer and the service manager both stop it themselves.
|
||||
log.Warn("The agent did not stop", exception);
|
||||
}
|
||||
|
||||
var product = removal.InstallKind == AgentInstallKinds.Msi && installer is not null
|
||||
? installer.RelatedProduct(Branding.AgentMsiUpgradeCode)
|
||||
: null;
|
||||
return product is not null
|
||||
? await UninstallMsiAsync(removal, product)
|
||||
: await RemovePortableAsync(removal, cancellationToken);
|
||||
}
|
||||
|
||||
private async Task<string?> UninstallMsiAsync(WatchdogRemoval removal, string product)
|
||||
{
|
||||
var logPath = MsiLogPath(removal.RemovalId);
|
||||
log.Info($"Uninstalling {product} through Windows Installer ({UninstallProperties}); its log is {logPath}");
|
||||
var giveUp = clock.UtcNow + timings.BusyRetryFor;
|
||||
while (true)
|
||||
{
|
||||
var answer = await UninstallOnceAsync(product, logPath);
|
||||
if (answer is not { } code)
|
||||
{
|
||||
return $"Windows Installer did not finish within {timings.AttemptTimeout.TotalMinutes:0} minutes; its log is {logPath}.";
|
||||
}
|
||||
|
||||
if (code == MsiExitCodes.InstallerBusy)
|
||||
{
|
||||
if (clock.UtcNow + timings.BusyRetryDelay > giveUp)
|
||||
{
|
||||
return $"Another installation kept Windows Installer busy (1618) for {timings.BusyRetryFor.TotalMinutes:0} minutes.";
|
||||
}
|
||||
|
||||
log.Warn($"Windows Installer is busy with another installation (1618); trying again in {timings.BusyRetryDelay.TotalSeconds:0} s");
|
||||
await clock.DelayAsync(timings.BusyRetryDelay, CancellationToken.None);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!MsiExitCodes.Succeeded(code) && code != MsiExitCodes.UnknownProduct)
|
||||
{
|
||||
return $"Windows Installer ended with {code} ({MsiExitCodes.Describe(code)}); its log is {logPath}.";
|
||||
}
|
||||
|
||||
log.Info($"Windows Installer ended with {code} ({MsiExitCodes.Describe(code)})");
|
||||
if (installer!.RelatedProduct(Branding.AgentMsiUpgradeCode) is { } still)
|
||||
{
|
||||
return $"Windows Installer ended with {code}, but {still} is still installed.";
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Windows Installer's answer, or null when it gave none within the timeout.
|
||||
/// The three calls run on one background thread of their own; one cut off by
|
||||
/// the timeout ends with the process.
|
||||
/// </summary>
|
||||
private async Task<int?> UninstallOnceAsync(string product, string logPath)
|
||||
{
|
||||
var uninstall = Task.Factory.StartNew(() =>
|
||||
{
|
||||
installer!.SetSilentUI();
|
||||
installer.EnableLog(logPath);
|
||||
return installer.UninstallProduct(product, UninstallProperties);
|
||||
}, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default);
|
||||
|
||||
try
|
||||
{
|
||||
return await uninstall.WaitAsync(timings.AttemptTimeout);
|
||||
}
|
||||
catch (TimeoutException)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private async Task<string?> RemovePortableAsync(WatchdogRemoval removal, CancellationToken cancellationToken)
|
||||
{
|
||||
if (purge.InstallFolderRefusal() is { } refusal)
|
||||
{
|
||||
return refusal;
|
||||
}
|
||||
|
||||
log.Info($"Removing the services of the install in {options.InstallDirectory}");
|
||||
await services.StopWatchdogAsync(cancellationToken);
|
||||
await services.RemoveAsync(cancellationToken);
|
||||
log.Info(removal.InstallKind == AgentInstallKinds.Msi
|
||||
? "Windows Installer no longer knows the product; its services were removed directly"
|
||||
: "The agent and watchdog services are removed");
|
||||
return null;
|
||||
}
|
||||
|
||||
/// <summary>Whatever the failed attempt stopped runs again, the watchdog first.</summary>
|
||||
private async Task RestoreAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
try
|
||||
{
|
||||
await services.StartWatchdogAsync(cancellationToken);
|
||||
}
|
||||
catch (Exception exception) when (exception is not OperationCanceledException)
|
||||
{
|
||||
log.Warn("The watchdog service could not be started", exception);
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var status = await agent.QueryAsync(cancellationToken);
|
||||
if (status.State != AgentRunState.NotInstalled && !status.IsRunning)
|
||||
{
|
||||
await agent.StartAsync(cancellationToken);
|
||||
}
|
||||
}
|
||||
catch (Exception exception) when (exception is not OperationCanceledException)
|
||||
{
|
||||
log.Warn("The agent could not be started", exception);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Failed for good: the agent reports it from <c>removal.json</c>; the logs stay where an operator looks.</summary>
|
||||
private void Finish(WatchdogRemoval removal)
|
||||
{
|
||||
log.Error($"Removal {removal.RemovalId} gave up after {removal.Attempts} attempts: {removal.LastError}");
|
||||
Write(removal with { Phase = WatchdogRemovalPhases.Failed });
|
||||
foreach (var file in new[] { Path.Combine(runnerDirectory, LogFileName + ".log"), MsiLogPath(removal.RemovalId) })
|
||||
{
|
||||
try
|
||||
{
|
||||
if (File.Exists(file))
|
||||
{
|
||||
Directory.CreateDirectory(options.LogDirectory);
|
||||
File.Copy(file, Path.Combine(options.LogDirectory, $"agent-removal-{removal.RemovalId:N}-{Path.GetFileName(file)}"), overwrite: true);
|
||||
}
|
||||
}
|
||||
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
|
||||
{
|
||||
log.Warn($"{file} could not be kept", exception);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private string MsiLogPath(Guid removalId) => Path.Combine(runnerDirectory, $"agent-msi-uninstall-{removalId:N}.log");
|
||||
|
||||
private void Write(WatchdogRemoval removal)
|
||||
=> WatchdogFileStore.Write(options.RemovalPath, removal, WatchdogJsonContext.Default.WatchdogRemoval);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// <c>removal.lock</c> in the runner's own folder, held open with no sharing for
|
||||
/// as long as it lives: released when the process ends however it ends, so a lock
|
||||
/// nobody holds is a runner that is gone.
|
||||
/// </summary>
|
||||
public static class RemovalRunLock
|
||||
{
|
||||
public static FileStream? TryHold(string path)
|
||||
{
|
||||
try
|
||||
{
|
||||
return new FileStream(path, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None);
|
||||
}
|
||||
catch (IOException)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
public static bool IsHeld(string path)
|
||||
{
|
||||
if (!File.Exists(path))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
using var probe = new FileStream(path, FileMode.Open, FileAccess.ReadWrite, FileShare.None);
|
||||
return false;
|
||||
}
|
||||
catch (FileNotFoundException)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
catch (IOException)
|
||||
{
|
||||
return true;
|
||||
}
|
||||
catch (UnauthorizedAccessException)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,211 @@
|
||||
using System.ComponentModel;
|
||||
using System.Globalization;
|
||||
using System.Runtime.Versioning;
|
||||
using System.Security.AccessControl;
|
||||
using System.Security.Principal;
|
||||
using Agent.Watchdog.Msi;
|
||||
using Agent.Watchdog.Platform;
|
||||
using Shared.Contracts.Models;
|
||||
using Shared.Infrastructure.Constants;
|
||||
|
||||
namespace Agent.Watchdog.Removal;
|
||||
|
||||
/// <summary>Starts the removal runner apart from the service that asks for it (the runner removes that service); replaced in tests.</summary>
|
||||
public interface IRemovalRunnerLauncher
|
||||
{
|
||||
/// <summary>A new folder outside the install and data folders, with a copy of this binary in it; or why not.</summary>
|
||||
(string? RunnerDirectory, string? Problem) Prepare(Guid removalId);
|
||||
|
||||
/// <summary>Starts the copy; null when it started, otherwise why not.</summary>
|
||||
string? Start(Guid removalId, string runnerDirectory);
|
||||
}
|
||||
|
||||
/// <summary>The launcher for this platform; shared with the agent, which has it as its fallback when no watchdog can take a removal.</summary>
|
||||
public static class RemovalRunnerLauncher
|
||||
{
|
||||
public static IRemovalRunnerLauncher Create(WatchdogOptions options, WatchdogLog log, string verb)
|
||||
{
|
||||
if (OperatingSystem.IsWindows())
|
||||
{
|
||||
return new WindowsRemovalRunnerLauncher(options, log, verb);
|
||||
}
|
||||
|
||||
return new UnixRemovalRunnerLauncher(options, log, verb);
|
||||
}
|
||||
|
||||
internal static string[] Arguments(string verb, Guid removalId, WatchdogOptions options, string runnerDirectory) =>
|
||||
[
|
||||
verb, removalId.ToString("D"),
|
||||
"--install-dir", options.InstallDirectory,
|
||||
"--data-dir", options.DataDirectory,
|
||||
"--runner-dir", runnerDirectory,
|
||||
];
|
||||
|
||||
internal static string CopySelf(WatchdogOptions options, string runnerDirectory)
|
||||
{
|
||||
var self = options.SelfPath is { } path && File.Exists(path)
|
||||
? path
|
||||
: throw new FileNotFoundException("this process has no binary of its own to run from");
|
||||
var copy = Path.Combine(runnerDirectory, Path.GetFileName(self));
|
||||
File.Copy(self, copy, overwrite: true);
|
||||
SwapPaths.MakeExecutable(copy);
|
||||
return copy;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// <c>%SystemRoot%\Temp\ExecuteR-<id>-<random>\</c>: anyone may add to a folder
|
||||
/// created under that Temp by default, and this one runs as SYSTEM and loads
|
||||
/// system DLLs from its own folder, so it is created closed (SYSTEM and
|
||||
/// Administrators only), under a name no one can have prepared, and its owner is
|
||||
/// checked before the binary goes in. Started detached: stopping both services
|
||||
/// never takes it down.
|
||||
/// </summary>
|
||||
[SupportedOSPlatform("windows")]
|
||||
public sealed class WindowsRemovalRunnerLauncher(WatchdogOptions options, WatchdogLog log, string verb) : IRemovalRunnerLauncher
|
||||
{
|
||||
public (string? RunnerDirectory, string? Problem) Prepare(Guid removalId)
|
||||
{
|
||||
var directory = Path.Combine(
|
||||
Environment.GetFolderPath(Environment.SpecialFolder.Windows), "Temp",
|
||||
$"{Branding.ProductKey}-{removalId:N}-{Random.Shared.Next():x8}");
|
||||
try
|
||||
{
|
||||
var folder = new DirectoryInfo(directory);
|
||||
if (folder.Exists)
|
||||
{
|
||||
return (null, $"{directory} already exists");
|
||||
}
|
||||
|
||||
folder.Create(ClosedFolder());
|
||||
var owner = folder.GetAccessControl().GetOwner(typeof(SecurityIdentifier));
|
||||
using var identity = WindowsIdentity.GetCurrent();
|
||||
if (owner is not SecurityIdentifier sid
|
||||
|| !(sid.IsWellKnown(WellKnownSidType.LocalSystemSid) || sid.IsWellKnown(WellKnownSidType.BuiltinAdministratorsSid) || sid == identity.User))
|
||||
{
|
||||
return (null, $"{directory} belongs to someone else");
|
||||
}
|
||||
|
||||
RemovalRunnerLauncher.CopySelf(options, directory);
|
||||
return (directory, null);
|
||||
}
|
||||
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
|
||||
{
|
||||
return (null, $"its binary could not be copied to {directory} ({exception.Message})");
|
||||
}
|
||||
}
|
||||
|
||||
public string? Start(Guid removalId, string runnerDirectory)
|
||||
{
|
||||
var copy = Path.Combine(runnerDirectory, Path.GetFileName(options.SelfPath!));
|
||||
try
|
||||
{
|
||||
var processId = DetachedProcess.Start(copy, RemovalRunnerLauncher.Arguments(verb, removalId, options, runnerDirectory), runnerDirectory);
|
||||
log.Info($"Removal runner started (process {processId}) from {copy}");
|
||||
return null;
|
||||
}
|
||||
catch (Win32Exception exception)
|
||||
{
|
||||
return exception.Message;
|
||||
}
|
||||
}
|
||||
|
||||
private static DirectorySecurity ClosedFolder()
|
||||
{
|
||||
var security = new DirectorySecurity();
|
||||
security.SetAccessRuleProtection(isProtected: true, preserveInheritance: false);
|
||||
foreach (var account in new[] { WellKnownSidType.LocalSystemSid, WellKnownSidType.BuiltinAdministratorsSid })
|
||||
{
|
||||
security.AddAccessRule(new FileSystemAccessRule(
|
||||
new SecurityIdentifier(account, null),
|
||||
FileSystemRights.FullControl,
|
||||
InheritanceFlags.ContainerInherit | InheritanceFlags.ObjectInherit,
|
||||
PropagationFlags.None,
|
||||
AccessControlType.Allow));
|
||||
}
|
||||
|
||||
return security;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// <c><parent of data>/executer-removal-<random>/</c> (mode 700; <c>/var/lib</c>, which
|
||||
/// unlike a temp folder is not mounted noexec), started through the service
|
||||
/// manager, since a plain child dies with the service that started it: systemd
|
||||
/// kills the service's whole cgroup, launchd its process group. Linux: a transient
|
||||
/// unit (<c>systemd-run</c>). macOS: a one-shot daemon from a plist in that folder.
|
||||
/// </summary>
|
||||
public sealed class UnixRemovalRunnerLauncher(WatchdogOptions options, WatchdogLog log, string verb) : IRemovalRunnerLauncher
|
||||
{
|
||||
public (string? RunnerDirectory, string? Problem) Prepare(Guid removalId)
|
||||
{
|
||||
if (OperatingSystem.IsWindows())
|
||||
{
|
||||
return (null, "not on this platform");
|
||||
}
|
||||
|
||||
var parent = Path.GetDirectoryName(Path.TrimEndingDirectorySeparator(options.DataDirectory));
|
||||
var directory = Path.Combine(parent ?? "/var/lib", $"executer-removal-{Random.Shared.Next():x8}");
|
||||
try
|
||||
{
|
||||
Directory.CreateDirectory(directory, UnixFileMode.UserRead | UnixFileMode.UserWrite | UnixFileMode.UserExecute);
|
||||
RemovalRunnerLauncher.CopySelf(options, directory);
|
||||
return (directory, null);
|
||||
}
|
||||
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
|
||||
{
|
||||
return (null, $"its binary could not be copied to {directory} ({exception.Message})");
|
||||
}
|
||||
}
|
||||
|
||||
public string? Start(Guid removalId, string runnerDirectory)
|
||||
{
|
||||
var copy = Path.Combine(runnerDirectory, Path.GetFileName(options.SelfPath!));
|
||||
var arguments = RemovalRunnerLauncher.Arguments(verb, removalId, options, runnerDirectory);
|
||||
try
|
||||
{
|
||||
var (code, output) = OperatingSystem.IsMacOS() ? StartLaunchd(copy, arguments, removalId, runnerDirectory) : StartSystemd(copy, arguments, removalId);
|
||||
if (code != 0)
|
||||
{
|
||||
return output.Trim();
|
||||
}
|
||||
|
||||
log.Info($"Removal runner started from {copy}");
|
||||
return null;
|
||||
}
|
||||
catch (Exception exception) when (exception is Win32Exception or InvalidOperationException or IOException)
|
||||
{
|
||||
return exception.Message;
|
||||
}
|
||||
}
|
||||
|
||||
private static (int Code, string Output) StartSystemd(string copy, string[] arguments, Guid removalId)
|
||||
=> Command.Run("systemd-run", [$"--unit={Branding.AgentSystemdUnit}-removal-{removalId:N}", "--collect", copy, .. arguments]);
|
||||
|
||||
private static (int Code, string Output) StartLaunchd(string copy, string[] arguments, Guid removalId, string runnerDirectory)
|
||||
{
|
||||
// Booted from outside /Library/LaunchDaemons: gone at the next restart, with nothing to clean up.
|
||||
var label = string.Create(CultureInfo.InvariantCulture, $"{Branding.AgentLaunchdLabel}.removal.{removalId:N}");
|
||||
var strings = string.Concat(new[] { copy }.Concat(arguments).Select(a => $"<string>{System.Security.SecurityElement.Escape(a)}</string>"));
|
||||
var plist = Path.Combine(runnerDirectory, label + ".plist");
|
||||
File.WriteAllText(plist, $"""
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>Label</key><string>{label}</string>
|
||||
<key>ProgramArguments</key>
|
||||
<array>{strings}</array>
|
||||
<key>RunAtLoad</key><true/>
|
||||
<key>AbandonProcessGroup</key><true/>
|
||||
</dict>
|
||||
</plist>
|
||||
""");
|
||||
if (!OperatingSystem.IsWindows())
|
||||
{
|
||||
File.SetUnixFileMode(plist, UnixFileMode.UserRead | UnixFileMode.UserWrite | UnixFileMode.GroupRead | UnixFileMode.OtherRead);
|
||||
}
|
||||
|
||||
return Command.Run("launchctl", "bootstrap", "system", plist);
|
||||
}
|
||||
}
|
||||
@@ -309,6 +309,19 @@ public sealed class UpgradeApplier(
|
||||
}
|
||||
|
||||
var handover = WatchdogFileStore.Read(options.HandoverPath, WatchdogJsonContext.Default.WatchdogHandover);
|
||||
if (handover is not null && File.Exists(options.RemovalPath))
|
||||
{
|
||||
// The device is being removed: an upgrade that crossed it never starts.
|
||||
log.Warn($"Upgrade {handover.UpgradeId} is set aside: the agent is being removed");
|
||||
WatchdogFileStore.TryDelete(options.HandoverPath);
|
||||
WatchdogFileStore.Write(options.OutcomePath,
|
||||
new WatchdogOutcome(handover.UpgradeId, AgentUpgradeStates.Failed, AgentUpgradeReasons.Busy, handover.Version, handover.FromVersion, [],
|
||||
clock.UtcNow, "The agent is being removed from this device; nothing was changed."),
|
||||
WatchdogJsonContext.Default.WatchdogOutcome);
|
||||
finished.Add(handover.UpgradeId);
|
||||
return null;
|
||||
}
|
||||
|
||||
if (handover is null)
|
||||
{
|
||||
var id = WatchdogFileStore.ReadUpgradeId(options.HandoverPath);
|
||||
|
||||
@@ -60,6 +60,9 @@ public static class WatchdogCli
|
||||
&& Guid.TryParse(args[1], out var upgradeId):
|
||||
return await RunMsiAsync(upgradeId, args[2..]);
|
||||
|
||||
case WatchdogRemovalFiles.RunVerb when args.Length > 1 && Guid.TryParse(args[1], out var removalId):
|
||||
return await Removal.RemovalHost.RunAsync(removalId, args[2..]);
|
||||
|
||||
case null or "help" or "-h" or "--help" or "/?":
|
||||
PrintUsage();
|
||||
return command is null ? 1 : 0;
|
||||
@@ -140,7 +143,9 @@ public static class WatchdogCli
|
||||
agent, clock, log);
|
||||
var applier = new UpgradeApplier(options, agent, clock, log, new ProcessSmokeTest(), restart, msi);
|
||||
var supervisor = new AgentSupervisor(options, agent, clock, log);
|
||||
var service = new WatchdogService(options, applier, supervisor, clock, log);
|
||||
var removal = new Removal.RemovalCoordinator(options,
|
||||
processMode ? null : Removal.RemovalRunnerLauncher.Create(options, log, WatchdogRemovalFiles.RunVerb), agent, clock, log);
|
||||
var service = new WatchdogService(options, applier, supervisor, clock, log, removal);
|
||||
|
||||
if (processMode)
|
||||
{
|
||||
|
||||
@@ -63,6 +63,9 @@ public sealed record WatchdogOptions
|
||||
|
||||
public string MsiRunnerLockPath => Path.Combine(UpgradeDirectory, WatchdogMsiFiles.RunnerLock);
|
||||
|
||||
/// <summary>A removal the agent handed over: while it is there no new upgrade starts, and until it failed the agent is not supervised.</summary>
|
||||
public string RemovalPath => Path.Combine(UpgradeDirectory, WatchdogFiles.Removal);
|
||||
|
||||
/// <summary>Windows Installer's verbose log of one MSI run, beside the agent's own logs.</summary>
|
||||
public string MsiLogPath(string version, Guid upgradeId)
|
||||
=> Path.Combine(LogDirectory, $"agent-msi-{version.Split('+')[0]}-{upgradeId:N}.log");
|
||||
|
||||
@@ -5,7 +5,13 @@ namespace Agent.Watchdog;
|
||||
/// left half done and tidies the install folder; then, every poll, it carries an
|
||||
/// upgrade on when one is waiting and otherwise supervises the agent.
|
||||
/// </summary>
|
||||
public sealed class WatchdogService(WatchdogOptions options, UpgradeApplier applier, AgentSupervisor supervisor, IWatchdogClock clock, WatchdogLog log)
|
||||
public sealed class WatchdogService(
|
||||
WatchdogOptions options,
|
||||
UpgradeApplier applier,
|
||||
AgentSupervisor supervisor,
|
||||
IWatchdogClock clock,
|
||||
WatchdogLog log,
|
||||
Removal.RemovalCoordinator? removal = null)
|
||||
{
|
||||
/// <summary>Runs until cancelled, or until the watchdog replaced itself (true: exit so the new binary takes over).</summary>
|
||||
public async Task<bool> RunAsync(CancellationToken cancellationToken)
|
||||
@@ -29,6 +35,12 @@ public sealed class WatchdogService(WatchdogOptions options, UpgradeApplier appl
|
||||
{
|
||||
try
|
||||
{
|
||||
if (removal is not null && await removal.PassAsync(cancellationToken))
|
||||
{
|
||||
await clock.DelayAsync(options.PollInterval, cancellationToken);
|
||||
continue;
|
||||
}
|
||||
|
||||
switch (await applier.PassAsync(cancellationToken))
|
||||
{
|
||||
case PassResult.RestartRequested:
|
||||
|
||||
@@ -60,6 +60,22 @@
|
||||
<ProjectReference Include="..\Shared.Infrastructure\Shared.Infrastructure.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<!-- The removal runner, linked from the watchdog: a device where no watchdog
|
||||
can take a removal runs the same code from a copy of this binary
|
||||
(remove-self). Nothing here needs ServiceController, which the linux/macOS
|
||||
build has not got. -->
|
||||
<Compile Include="..\Agent.Watchdog\Removal\*.cs" LinkBase="Upgrade\RemovalRunner" />
|
||||
<Compile Include="..\Agent.Watchdog\Abstractions.cs" LinkBase="Upgrade\RemovalRunner" />
|
||||
<Compile Include="..\Agent.Watchdog\SwapPaths.cs" LinkBase="Upgrade\RemovalRunner" />
|
||||
<Compile Include="..\Agent.Watchdog\WatchdogLog.cs" LinkBase="Upgrade\RemovalRunner" />
|
||||
<Compile Include="..\Agent.Watchdog\WatchdogOptions.cs" LinkBase="Upgrade\RemovalRunner" />
|
||||
<Compile Include="..\Agent.Watchdog\Msi\WindowsInstaller.cs" LinkBase="Upgrade\RemovalRunner" />
|
||||
<Compile Include="..\Agent.Watchdog\Msi\MsiRunnerLauncher.cs" LinkBase="Upgrade\RemovalRunner" />
|
||||
<Compile Include="..\Agent.Watchdog\Platform\ServiceInterop.cs" LinkBase="Upgrade\RemovalRunner" />
|
||||
<Compile Include="..\Agent.Watchdog\Platform\UnixServices.cs" LinkBase="Upgrade\RemovalRunner" />
|
||||
</ItemGroup>
|
||||
|
||||
<!--
|
||||
Trim the agent payload before single-file bundling. The PowerShell SDK
|
||||
ships only the core engine modules (Management, Utility, Security, Host,
|
||||
|
||||
@@ -110,6 +110,11 @@ public static class AgentCli
|
||||
case "purge-credentials":
|
||||
return PurgeCredentials();
|
||||
|
||||
// Hidden: the removal runner, from a copy of this binary, on a
|
||||
// device where no watchdog can take a removal.
|
||||
case Shared.Contracts.Models.WatchdogRemovalFiles.AgentRunVerb when args.Length > 1 && Guid.TryParse(args[1], out var removalId):
|
||||
return await Agent.Watchdog.Removal.RemovalHost.RunAsync(removalId, args[2..]);
|
||||
|
||||
default:
|
||||
Console.Error.WriteLine($"Unknown command '{command}'.");
|
||||
PrintUsage();
|
||||
|
||||
@@ -28,6 +28,7 @@ public sealed class CommandProcessor(
|
||||
JobHistoryStore jobHistory,
|
||||
AgentConfiguration configuration,
|
||||
AgentUpgradeHandler upgrades,
|
||||
AgentRemoveCommandHandler removals,
|
||||
AgentHealthState health,
|
||||
ILogger logger)
|
||||
{
|
||||
@@ -125,6 +126,12 @@ public sealed class CommandProcessor(
|
||||
await HandleUpgradeCommandAsync(envelope, message, cancellationToken);
|
||||
break;
|
||||
|
||||
case MessageTypes.CommandAgentRemove:
|
||||
// Acknowledged once removal.json holds it: a redelivery finds it there.
|
||||
await removals.HandleAsync(envelope.GetPayload<AgentRemoveCommandPayload>(), cancellationToken);
|
||||
await message.AckAsync(cancellationToken: cancellationToken);
|
||||
break;
|
||||
|
||||
default:
|
||||
logger.LogWarning("Unknown command type {MessageType}; terminating delivery", envelope.MessageType);
|
||||
await message.AckTerminateAsync(cancellationToken: cancellationToken);
|
||||
|
||||
@@ -0,0 +1,404 @@
|
||||
using System.Buffers.Text;
|
||||
using System.Net;
|
||||
using System.Net.Http.Json;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
using Agent.Configuration;
|
||||
using Agent.Storage;
|
||||
using Agent.Upgrade;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using NATS.NKeys;
|
||||
using Shared.Contracts.Api;
|
||||
using Shared.Contracts.Models;
|
||||
using Shared.Contracts.Serialization;
|
||||
|
||||
namespace Agent.Enrollment;
|
||||
|
||||
/// <summary>What a NATS user JWT says about its own lifetime (its signature is the server's to check).</summary>
|
||||
public static class AgentJwt
|
||||
{
|
||||
/// <summary><c>iat</c> and <c>exp</c>; either is null when absent, both when it is not a readable JWT.</summary>
|
||||
public static (DateTimeOffset? IssuedUtc, DateTimeOffset? ExpiresUtc) ReadLifetime(string jwt)
|
||||
{
|
||||
var parts = jwt.Split('.');
|
||||
if (parts.Length != 3)
|
||||
{
|
||||
return (null, null);
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
using var claims = JsonDocument.Parse(Base64Url.DecodeFromChars(parts[1]));
|
||||
if (claims.RootElement.ValueKind != JsonValueKind.Object)
|
||||
{
|
||||
return (null, null);
|
||||
}
|
||||
|
||||
return (Seconds(claims.RootElement, "iat"), Seconds(claims.RootElement, "exp"));
|
||||
}
|
||||
catch (Exception exception) when (exception is FormatException or JsonException)
|
||||
{
|
||||
return (null, null);
|
||||
}
|
||||
}
|
||||
|
||||
private static DateTimeOffset? Seconds(JsonElement claims, string name)
|
||||
=> claims.TryGetProperty(name, out var value) && value.TryGetInt64(out var seconds) && seconds > 0
|
||||
? DateTimeOffset.FromUnixTimeSeconds(seconds)
|
||||
: null;
|
||||
}
|
||||
|
||||
/// <summary>What credential renewal works with; every seam a test needs is here.</summary>
|
||||
public sealed class CredentialRenewalContext
|
||||
{
|
||||
public required ICredentialStore Store { get; init; }
|
||||
public required RuntimeStateStore RuntimeState { get; init; }
|
||||
public required AgentConfigurationLoader Configuration { get; init; }
|
||||
public required UpgradeFolders Folders { get; init; }
|
||||
|
||||
/// <summary>The server refused the credential for good: a removed device (the id, when the answer names one).</summary>
|
||||
public required Func<Guid, CancellationToken, Task> OnRemoved { get; init; }
|
||||
|
||||
/// <summary>A 401 while a bootstrap token is configured: enrolling again is how a device comes back.</summary>
|
||||
public required Func<AgentConfiguration, CancellationToken, Task> Reenroll { get; init; }
|
||||
|
||||
/// <summary>The HTTP handler: OS trust (the managed CA was installed there at enrollment), or accept-any when configured.</summary>
|
||||
public Func<bool, HttpMessageHandler>? CreateHttpHandler { get; init; }
|
||||
|
||||
public Func<DateTimeOffset> UtcNow { get; init; } = () => DateTimeOffset.UtcNow;
|
||||
|
||||
public Func<TimeSpan, CancellationToken, Task> Delay { get; init; } = Task.Delay;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Keeps the agent's NATS credential current over HTTPS, at every start and at
|
||||
/// half life, whether or not NATS is connected: an expired credential recovers by
|
||||
/// itself as soon as the server can be reached, since the renewal is signed with
|
||||
/// the enrolled NKey and not with the credential. The old one stays in the store
|
||||
/// until a new one replaces it, and the connection takes the new one at its next
|
||||
/// reconnect (<see cref="Messaging.AgentNatsOptionsFactory"/>), at the latest when
|
||||
/// the server ends it as the old one expires.
|
||||
/// <list type="bullet">
|
||||
/// <item><b>Upgrades:</b> while an upgrade could still roll back, the stored
|
||||
/// credential stays: a version before <see cref="DeviceRemovalDefaults.MinimumAgentVersion"/>
|
||||
/// cannot renew and would strand on a short-lived one. Only one about to expire
|
||||
/// is replaced then: it would strand either way, and the new version could not
|
||||
/// report healthy.</item>
|
||||
/// <item><b>410 (removed):</b> the agent uninstalls itself, unless the server
|
||||
/// could not be verified (<c>TlsSkipVerify</c>, or plain http): an unverified
|
||||
/// server must not be able to uninstall an agent.</item>
|
||||
/// <item><b>401 (orphaned):</b> tried again hourly, and a configured bootstrap
|
||||
/// token enrolls the device again.</item>
|
||||
/// <item><b>Network or server errors:</b> tried again with back-off.</item>
|
||||
/// </list>
|
||||
/// </summary>
|
||||
public sealed class CredentialRenewal(CredentialRenewalContext context, ILogger logger)
|
||||
{
|
||||
public static readonly TimeSpan RequestTimeout = TimeSpan.FromSeconds(30);
|
||||
public static readonly TimeSpan RefusedRetry = TimeSpan.FromHours(1);
|
||||
public static readonly TimeSpan BackoffMin = TimeSpan.FromSeconds(30);
|
||||
public static readonly TimeSpan BackoffMax = TimeSpan.FromMinutes(15);
|
||||
|
||||
/// <summary>The longest one wait: the clock may jump (a sleeping laptop), so it is looked at again.</summary>
|
||||
public static readonly TimeSpan MaxWait = TimeSpan.FromMinutes(5);
|
||||
|
||||
/// <summary>How often a state that blocks renewal (no credential, an upgrade, a removal) is looked at again.</summary>
|
||||
public static readonly TimeSpan BlockedRecheck = TimeSpan.FromMinutes(1);
|
||||
|
||||
/// <summary>A credential this close to its end is replaced even during an upgrade.</summary>
|
||||
public static readonly TimeSpan ExpiringSoon = TimeSpan.FromHours(1);
|
||||
|
||||
private static readonly TimeSpan HalfLifetime = AgentCredentialDefaults.Lifetime / 2;
|
||||
|
||||
private bool answeredSinceStart;
|
||||
private bool upgradeNoted;
|
||||
private int failures;
|
||||
private DateTimeOffset notBefore = DateTimeOffset.MinValue;
|
||||
private DateTimeOffset? renewedUtc;
|
||||
|
||||
public async Task RunAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
while (!cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
TimeSpan wait;
|
||||
try
|
||||
{
|
||||
wait = await PassAsync(cancellationToken);
|
||||
}
|
||||
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
return;
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
logger.LogWarning(exception, "Credential renewal failed; it is tried again");
|
||||
wait = Backoff();
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
await context.Delay(wait, cancellationToken);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Renews when it is time to; how long until the next pass.</summary>
|
||||
public async Task<TimeSpan> PassAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
var now = context.UtcNow();
|
||||
if (now < notBefore)
|
||||
{
|
||||
return Capped(notBefore - now);
|
||||
}
|
||||
|
||||
var configuration = context.Configuration.Load();
|
||||
var stored = context.Store.Load();
|
||||
if (stored is null || configuration.ServerUrl is null)
|
||||
{
|
||||
return BlockedRecheck;
|
||||
}
|
||||
|
||||
if (WatchdogFileStore.Read(context.Folders.Removal, WatchdogJsonContext.Default.WatchdogRemoval) is { Phase: not WatchdogRemovalPhases.Failed })
|
||||
{
|
||||
return BlockedRecheck;
|
||||
}
|
||||
|
||||
var expires = ExpiryOf(stored);
|
||||
if (answeredSinceStart && DueAt(stored, expires) is { } due && due > now)
|
||||
{
|
||||
return Capped(due - now);
|
||||
}
|
||||
|
||||
if (context.Folders.UpgradeUnderWay && (expires is null || expires - now > ExpiringSoon))
|
||||
{
|
||||
if (!upgradeNoted)
|
||||
{
|
||||
logger.LogInformation("The credential is renewed once the upgrade under way is kept; a rollback could not renew it");
|
||||
upgradeNoted = true;
|
||||
}
|
||||
|
||||
return BlockedRecheck;
|
||||
}
|
||||
|
||||
upgradeNoted = false;
|
||||
return await RenewAsync(configuration, stored, cancellationToken);
|
||||
}
|
||||
|
||||
/// <summary>The JWT's own <c>exp</c>, else what the server said when it issued it; null for a credential that never expires.</summary>
|
||||
public static DateTimeOffset? ExpiryOf(AgentCredentials credentials)
|
||||
=> AgentJwt.ReadLifetime(credentials.UserJwt).ExpiresUtc ?? credentials.ExpiresUtc;
|
||||
|
||||
/// <summary>
|
||||
/// Half way between <c>iat</c> and <c>exp</c> (<c>iat</c> taken as a full lifetime
|
||||
/// back when absent). One that does not expire, however the server answers, is
|
||||
/// asked again every half lifetime.
|
||||
/// </summary>
|
||||
private DateTimeOffset? DueAt(AgentCredentials credentials, DateTimeOffset? expires)
|
||||
{
|
||||
if (expires is null)
|
||||
{
|
||||
return renewedUtc + HalfLifetime;
|
||||
}
|
||||
|
||||
var issued = AgentJwt.ReadLifetime(credentials.UserJwt).IssuedUtc ?? expires.Value - AgentCredentialDefaults.Lifetime;
|
||||
return issued + (expires.Value - issued) / 2;
|
||||
}
|
||||
|
||||
private async Task<TimeSpan> RenewAsync(AgentConfiguration configuration, AgentCredentials stored, CancellationToken cancellationToken)
|
||||
{
|
||||
var skipVerify = configuration.TlsSkipVerify.Value;
|
||||
var endpoint = configuration.ServerUrl!.ApiEndpoint(ApiRoutes.RefreshCredential);
|
||||
HttpResponseMessage response;
|
||||
try
|
||||
{
|
||||
using var handler = CreateHandler(skipVerify);
|
||||
using var http = new HttpClient(handler, disposeHandler: false) { Timeout = RequestTimeout };
|
||||
response = await http.PostAsJsonAsync(endpoint, Sign(stored), JsonDefaults.Options, cancellationToken);
|
||||
}
|
||||
catch (Exception exception) when (exception is HttpRequestException or TaskCanceledException && !cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
var wait = Backoff();
|
||||
logger.LogWarning(exception, "The server could not be reached to renew the credential; trying again in {Wait}", wait);
|
||||
return Capped(wait);
|
||||
}
|
||||
|
||||
using (response)
|
||||
{
|
||||
switch (response.StatusCode)
|
||||
{
|
||||
case HttpStatusCode.OK:
|
||||
return await StoreAsync(stored, response, cancellationToken);
|
||||
|
||||
case HttpStatusCode.Gone when await ReadRemovedAsync(response, cancellationToken) is { } removalId:
|
||||
answeredSinceStart = true;
|
||||
return await RemovedAsync(removalId, endpoint, skipVerify, cancellationToken);
|
||||
|
||||
case HttpStatusCode.Unauthorized:
|
||||
answeredSinceStart = true;
|
||||
logger.LogWarning("The server refused to renew the credential (401; orphaned, or the clock is off); trying again in {Wait}", RefusedRetry);
|
||||
Refused();
|
||||
if (configuration.HasBootstrapToken)
|
||||
{
|
||||
await context.Reenroll(configuration, cancellationToken);
|
||||
}
|
||||
|
||||
return Capped(RefusedRetry);
|
||||
|
||||
default:
|
||||
var wait = Backoff();
|
||||
logger.LogWarning("The server answered {Status} to renewing the credential; trying again in {Wait}", (int)response.StatusCode, wait);
|
||||
return Capped(wait);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private async Task<TimeSpan> StoreAsync(AgentCredentials stored, HttpResponseMessage response, CancellationToken cancellationToken)
|
||||
{
|
||||
var result = await response.Content.ReadFromJsonAsync<CredentialRefreshResponse>(JsonDefaults.Options, cancellationToken);
|
||||
if (result is null || string.IsNullOrWhiteSpace(result.UserJwt))
|
||||
{
|
||||
var wait = Backoff();
|
||||
logger.LogWarning("The server's new credential was empty; trying again in {Wait}", wait);
|
||||
return Capped(wait);
|
||||
}
|
||||
|
||||
// Enrolled again meanwhile (a new key): this answer is for the old one.
|
||||
if (context.Store.Load() is not { } current || current.NkeySeed != stored.NkeySeed)
|
||||
{
|
||||
logger.LogInformation("The agent enrolled again during the renewal; its answer is dropped");
|
||||
return TimeSpan.Zero;
|
||||
}
|
||||
|
||||
var expires = AgentJwt.ReadLifetime(result.UserJwt).ExpiresUtc ?? result.CredentialExpiresUtc;
|
||||
var renewed = current with { UserJwt = result.UserJwt, ExpiresUtc = expires };
|
||||
context.Store.Save(renewed);
|
||||
context.RuntimeState.CredentialExpiresUtc = expires;
|
||||
answeredSinceStart = true;
|
||||
failures = 0;
|
||||
notBefore = DateTimeOffset.MinValue;
|
||||
renewedUtc = context.UtcNow();
|
||||
logger.LogInformation("Credential renewed (expires {Expires})", expires?.ToString("O") ?? "never");
|
||||
|
||||
var due = DueAt(renewed, expires);
|
||||
return due is { } at && at > context.UtcNow() ? Capped(at - context.UtcNow()) : BlockedRecheck;
|
||||
}
|
||||
|
||||
private async Task<TimeSpan> RemovedAsync(Guid removalId, Uri endpoint, bool skipVerify, CancellationToken cancellationToken)
|
||||
{
|
||||
Refused();
|
||||
if (skipVerify || endpoint.Scheme != Uri.UriSchemeHttps)
|
||||
{
|
||||
logger.LogWarning(
|
||||
"The server at {Endpoint} says this device was removed, but it is not verified (TlsSkipVerify, or plain http), so the agent stays; remove it by hand if that is right",
|
||||
endpoint);
|
||||
return Capped(RefusedRetry);
|
||||
}
|
||||
|
||||
logger.LogWarning("The server says this device was removed");
|
||||
await context.OnRemoved(removalId, cancellationToken);
|
||||
return Capped(RefusedRetry);
|
||||
}
|
||||
|
||||
private void Refused()
|
||||
{
|
||||
failures = 0;
|
||||
notBefore = context.UtcNow() + RefusedRetry;
|
||||
}
|
||||
|
||||
/// <summary>The removal id of a real removal's 410 (empty when the answer names none); null for any other 410, say from a proxy.</summary>
|
||||
private static async Task<Guid?> ReadRemovedAsync(HttpResponseMessage response, CancellationToken cancellationToken)
|
||||
{
|
||||
try
|
||||
{
|
||||
using var problem = JsonDocument.Parse(await response.Content.ReadAsStringAsync(cancellationToken));
|
||||
var root = problem.RootElement;
|
||||
if (root.ValueKind != JsonValueKind.Object
|
||||
|| !root.TryGetProperty("type", out var type)
|
||||
|| type.ValueKind != JsonValueKind.String
|
||||
|| type.GetString() != AgentCredentialDefaults.RemovedProblemType)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
return root.TryGetProperty("removalId", out var id) && id.ValueKind == JsonValueKind.String && Guid.TryParse(id.GetString(), out var removalId)
|
||||
? removalId
|
||||
: Guid.Empty;
|
||||
}
|
||||
catch (JsonException)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Proof of the enrolled key: <c>{clientId}|{timestampUtc:O}</c>, signed with the seed.</summary>
|
||||
private CredentialRefreshRequest Sign(AgentCredentials stored)
|
||||
{
|
||||
using var key = KeyPair.FromSeed(stored.NkeySeed.AsSpan());
|
||||
var timestamp = context.UtcNow().ToUniversalTime();
|
||||
var signature = new byte[64];
|
||||
key.Sign(Encoding.UTF8.GetBytes($"{stored.ClientId}|{timestamp:O}"), signature);
|
||||
return new CredentialRefreshRequest(stored.ClientId, key.GetPublicKey(), timestamp, Convert.ToBase64String(signature));
|
||||
}
|
||||
|
||||
private HttpMessageHandler CreateHandler(bool skipVerify)
|
||||
{
|
||||
if (context.CreateHttpHandler is { } create)
|
||||
{
|
||||
return create(skipVerify);
|
||||
}
|
||||
|
||||
var handler = new HttpClientHandler();
|
||||
if (skipVerify)
|
||||
{
|
||||
handler.ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator;
|
||||
}
|
||||
|
||||
return handler;
|
||||
}
|
||||
|
||||
/// <summary>30 s doubling to 15 minutes; the same for every failure until an answer comes.</summary>
|
||||
private TimeSpan Backoff()
|
||||
{
|
||||
var wait = TimeSpan.FromTicks(Math.Min(BackoffMin.Ticks << Math.Min(failures, 10), BackoffMax.Ticks));
|
||||
failures++;
|
||||
notBefore = context.UtcNow() + wait;
|
||||
return wait;
|
||||
}
|
||||
|
||||
private static TimeSpan Capped(TimeSpan wait) => wait < MaxWait ? wait : MaxWait;
|
||||
}
|
||||
|
||||
/// <summary>Runs <see cref="CredentialRenewal"/> for as long as the agent runs, NATS connected or not.</summary>
|
||||
public sealed class CredentialRenewalService(CredentialRenewal renewal) : BackgroundService
|
||||
{
|
||||
protected override Task ExecuteAsync(CancellationToken stoppingToken) => renewal.RunAsync(stoppingToken);
|
||||
}
|
||||
|
||||
/// <summary>Takes a 410 from the renewal to the removal.</summary>
|
||||
public static class CredentialRemoval
|
||||
{
|
||||
/// <summary>Starts the removal the server's refusal asks for; what cannot start now is asked again by the next refusal.</summary>
|
||||
public static async Task HandleAsync(AgentRemoval removal, Guid removalId, ILogger logger, CancellationToken cancellationToken)
|
||||
{
|
||||
if (removal.UnderWay)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
if (removal.Refusal(strict: false) is { } refusal)
|
||||
{
|
||||
logger.LogWarning("This device was removed from the server, but the agent cannot remove itself now ({Reason})", refusal);
|
||||
return;
|
||||
}
|
||||
|
||||
if (await removal.HandOverAsync(removalId, cancellationToken) is { } problem)
|
||||
{
|
||||
logger.LogError("This device was removed from the server, but the removal did not start: {Problem}", problem);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -88,16 +88,32 @@ public sealed class CredentialStore(ILogger<CredentialStore> logger) : ICredenti
|
||||
var payload = plain;
|
||||
#endif
|
||||
|
||||
// A temporary name and a rename: renewal replaces this file for the life
|
||||
// of the agent, and half a file would be an agent that is no longer enrolled.
|
||||
var path = Path.Combine(directory, Branding.CredentialFileName);
|
||||
File.WriteAllBytes(path, payload);
|
||||
#if WINDOWS_AGENT
|
||||
ApplyRestrictedAcl(new FileInfo(path));
|
||||
#else
|
||||
if (!OperatingSystem.IsWindows())
|
||||
var temporary = $"{path}.{Guid.NewGuid():N}.tmp";
|
||||
try
|
||||
{
|
||||
File.SetUnixFileMode(path, UnixFileMode.UserRead | UnixFileMode.UserWrite);
|
||||
}
|
||||
File.WriteAllBytes(temporary, payload);
|
||||
#if WINDOWS_AGENT
|
||||
ApplyRestrictedAcl(new FileInfo(temporary));
|
||||
#else
|
||||
if (!OperatingSystem.IsWindows())
|
||||
{
|
||||
File.SetUnixFileMode(temporary, UnixFileMode.UserRead | UnixFileMode.UserWrite);
|
||||
}
|
||||
#endif
|
||||
MoveOver(temporary, path);
|
||||
}
|
||||
catch
|
||||
{
|
||||
if (File.Exists(temporary))
|
||||
{
|
||||
File.Delete(temporary);
|
||||
}
|
||||
|
||||
throw;
|
||||
}
|
||||
|
||||
logger.LogInformation("Credentials persisted for client {ClientId}", credentials.ClientId);
|
||||
}
|
||||
@@ -110,6 +126,23 @@ public sealed class CredentialStore(ILogger<CredentialStore> logger) : ICredenti
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Windows refuses a rename over a file another thread is reading for a moment (a reconnect reads it).</summary>
|
||||
private static void MoveOver(string temporary, string path)
|
||||
{
|
||||
for (var attempt = 1; ; attempt++)
|
||||
{
|
||||
try
|
||||
{
|
||||
File.Move(temporary, path, overwrite: true);
|
||||
return;
|
||||
}
|
||||
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException && attempt < 40)
|
||||
{
|
||||
Thread.Sleep(25);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private static string EnsureProtectedDirectory()
|
||||
{
|
||||
var directory = new DirectoryInfo(Branding.AgentProgramDataDirectory);
|
||||
|
||||
@@ -105,7 +105,7 @@ public static class ServerTrust
|
||||
}
|
||||
else if (OperatingSystem.IsMacOS())
|
||||
{
|
||||
InstallMac(pem, logger);
|
||||
InstallMac(pem, chain, logger);
|
||||
}
|
||||
else
|
||||
{
|
||||
@@ -168,7 +168,7 @@ public static class ServerTrust
|
||||
}
|
||||
else if (OperatingSystem.IsMacOS())
|
||||
{
|
||||
InstallMac(pem, logger);
|
||||
InstallMac(pem, chain, logger);
|
||||
}
|
||||
else
|
||||
{
|
||||
@@ -201,6 +201,7 @@ public static class ServerTrust
|
||||
if (store.Certificates.Find(X509FindType.FindByThumbprint, cert.Thumbprint, validOnly: false).Count == 0)
|
||||
{
|
||||
store.Add(cert);
|
||||
RememberInstalled([cert.Thumbprint], logger);
|
||||
logger.LogInformation("Installed server CA into LocalMachine\\{Store}: {Subject}",
|
||||
selfIssued ? "Root" : "CA", cert.Subject);
|
||||
}
|
||||
@@ -212,6 +213,22 @@ public static class ServerTrust
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Notes what the agent itself added to the trust store, the only certificates a removal takes back out.</summary>
|
||||
private static void RememberInstalled(IEnumerable<string> thumbprints, ILogger logger)
|
||||
{
|
||||
try
|
||||
{
|
||||
var path = Path.Combine(Shared.Infrastructure.Constants.Branding.AgentProgramDataDirectory, Shared.Contracts.Models.WatchdogFiles.ManagedCa);
|
||||
Directory.CreateDirectory(Path.GetDirectoryName(path)!);
|
||||
var known = File.Exists(path) ? File.ReadAllLines(path) : [];
|
||||
File.WriteAllLines(path, known.Union(thumbprints, StringComparer.OrdinalIgnoreCase));
|
||||
}
|
||||
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
|
||||
{
|
||||
logger.LogWarning(ex, "Could not note the installed server CA; a removal will leave it in the trust store");
|
||||
}
|
||||
}
|
||||
|
||||
private static void InstallLinux(string pem, ILogger logger)
|
||||
{
|
||||
try
|
||||
@@ -238,7 +255,7 @@ public static class ServerTrust
|
||||
}
|
||||
}
|
||||
|
||||
private static void InstallMac(string pem, ILogger logger)
|
||||
private static void InstallMac(string pem, X509Certificate2Collection chain, ILogger logger)
|
||||
{
|
||||
try
|
||||
{
|
||||
@@ -246,6 +263,7 @@ public static class ServerTrust
|
||||
File.WriteAllText(path, pem);
|
||||
RunBestEffort("security", logger,
|
||||
"add-trusted-cert", "-d", "-r", "trustRoot", "-k", "/Library/Keychains/System.keychain", path);
|
||||
RememberInstalled(chain.Cast<X509Certificate2>().Select(c => c.Thumbprint), logger);
|
||||
File.Delete(path);
|
||||
}
|
||||
catch (Exception ex)
|
||||
|
||||
@@ -17,7 +17,8 @@ public static class AgentNatsOptionsFactory
|
||||
ServerUrl serverUrl,
|
||||
AgentCredentials credentials,
|
||||
AgentConfiguration configuration,
|
||||
ILoggerFactory loggerFactory)
|
||||
ILoggerFactory loggerFactory,
|
||||
Func<AgentCredentials?>? current = null)
|
||||
{
|
||||
var options = NatsOpts.Default with
|
||||
{
|
||||
@@ -25,8 +26,13 @@ public static class AgentNatsOptionsFactory
|
||||
Name = $"agent-{credentials.ClientId}",
|
||||
AuthOpts = NatsAuthOpts.Default with
|
||||
{
|
||||
Jwt = credentials.UserJwt,
|
||||
Seed = credentials.NkeySeed,
|
||||
// Asked at every connect, so a reconnect takes the credential
|
||||
// renewed meanwhile, and an expired one recovers by itself.
|
||||
AuthCredCallback = (_, _) =>
|
||||
{
|
||||
var latest = current?.Invoke() ?? credentials;
|
||||
return ValueTask.FromResult(NatsAuthCred.FromJwt(latest.UserJwt, latest.NkeySeed));
|
||||
},
|
||||
},
|
||||
ReconnectWaitMin = configuration.ReconnectMinDelay.Value,
|
||||
ReconnectWaitMax = configuration.ReconnectMaxDelay.Value,
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
using System.Globalization;
|
||||
using System.Management.Automation.Host;
|
||||
|
||||
namespace Agent.PowerShell;
|
||||
|
||||
/// <summary>
|
||||
/// The in-process engine's host: behaves like the engine's own default host
|
||||
/// (no UI, so Write-Host stays on the information stream) and remembers what
|
||||
/// <c>SetShouldExit</c> was given. <c>exit N</c> in a script run through
|
||||
/// AddScript reaches the host this way and nowhere else: the engine sets
|
||||
/// neither <c>$LASTEXITCODE</c> nor HadErrors for it.
|
||||
/// </summary>
|
||||
internal sealed class ExitCapturingHost : PSHost
|
||||
{
|
||||
private readonly CultureInfo culture = CultureInfo.CurrentCulture;
|
||||
private readonly CultureInfo uiCulture = CultureInfo.CurrentUICulture;
|
||||
private readonly Guid instanceId = Guid.NewGuid();
|
||||
|
||||
/// <summary>The last exit code the script asked for; null when it never called exit.</summary>
|
||||
public int? ExitCode { get; private set; }
|
||||
|
||||
public override CultureInfo CurrentCulture => culture;
|
||||
|
||||
public override CultureInfo CurrentUICulture => uiCulture;
|
||||
|
||||
public override Guid InstanceId => instanceId;
|
||||
|
||||
public override string Name => "Default Host";
|
||||
|
||||
public override PSHostUserInterface? UI => null;
|
||||
|
||||
public override Version Version => new(1, 0);
|
||||
|
||||
public override void EnterNestedPrompt() => throw new NotSupportedException();
|
||||
|
||||
public override void ExitNestedPrompt() => throw new NotSupportedException();
|
||||
|
||||
public override void NotifyBeginApplication()
|
||||
{
|
||||
}
|
||||
|
||||
public override void NotifyEndApplication()
|
||||
{
|
||||
}
|
||||
|
||||
public override void SetShouldExit(int exitCode) => ExitCode = exitCode;
|
||||
}
|
||||
@@ -3,8 +3,33 @@ using System.Management.Automation.Runspaces;
|
||||
|
||||
namespace Agent.PowerShell;
|
||||
|
||||
/// <summary>Result of one in-process PowerShell execution.</summary>
|
||||
public sealed record PowerShellExecutionResult(bool HadErrors, bool WasCancelled, string? FailureMessage);
|
||||
/// <summary>The first error a script hit: its exception type, fully qualified error id and message.</summary>
|
||||
public sealed record ScriptErrorInfo(string? Type, string? Id, string? Message)
|
||||
{
|
||||
public const int MaxTextLength = 2000;
|
||||
public const int MaxNameLength = 300;
|
||||
|
||||
public static ScriptErrorInfo From(ErrorRecord record) => new(
|
||||
Cut(record.Exception?.GetType().FullName, MaxNameLength),
|
||||
Cut(record.FullyQualifiedErrorId, MaxNameLength),
|
||||
Cut(record.Exception?.Message ?? record.ToString(), MaxTextLength));
|
||||
|
||||
private static string? Cut(string? value, int length)
|
||||
=> value is { Length: > 0 } ? (value.Length > length ? value[..length] : value) : null;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Result of one in-process PowerShell execution. <paramref name="ExitCode"/>
|
||||
/// is what the script's own <c>exit N</c> asked for (null when it never called
|
||||
/// exit); it decides the outcome over <paramref name="HadErrors"/>, as
|
||||
/// <c>pwsh -File</c> does.
|
||||
/// </summary>
|
||||
public sealed record PowerShellExecutionResult(
|
||||
bool HadErrors,
|
||||
bool WasCancelled,
|
||||
string? FailureMessage,
|
||||
int? ExitCode = null,
|
||||
ScriptErrorInfo? Error = null);
|
||||
|
||||
/// <summary>
|
||||
/// Executes scripts in-process through System.Management.Automation:
|
||||
@@ -40,7 +65,8 @@ public sealed class PowerShellExecutionService
|
||||
IReadOnlyDictionary<string, Shared.Contracts.Models.ValueExpansion>? expansions = null,
|
||||
IReadOnlyDictionary<string, object?>? sessionVariables = null)
|
||||
{
|
||||
using var runspace = PowerShellRunspaceFactory.Create();
|
||||
var host = new ExitCapturingHost();
|
||||
using var runspace = PowerShellRunspaceFactory.Create(host);
|
||||
using var powerShell = System.Management.Automation.PowerShell.Create();
|
||||
powerShell.Runspace = runspace;
|
||||
|
||||
@@ -78,8 +104,13 @@ public sealed class PowerShellExecutionService
|
||||
var output = new PSDataCollection<PSObject>();
|
||||
output.DataAdded += (_, e) => onRecord(PowerShellOutputNormalizer.FromOutput(output[e.Index]));
|
||||
|
||||
ScriptErrorInfo? firstError = null;
|
||||
powerShell.Streams.Error.DataAdded += (_, e) =>
|
||||
onRecord(PowerShellOutputNormalizer.FromError(powerShell.Streams.Error[e.Index]));
|
||||
{
|
||||
var record = powerShell.Streams.Error[e.Index];
|
||||
firstError ??= ScriptErrorInfo.From(record);
|
||||
onRecord(PowerShellOutputNormalizer.FromError(record));
|
||||
};
|
||||
powerShell.Streams.Warning.DataAdded += (_, e) =>
|
||||
onRecord(PowerShellOutputNormalizer.FromWarning(powerShell.Streams.Warning[e.Index]));
|
||||
powerShell.Streams.Verbose.DataAdded += (_, e) =>
|
||||
@@ -118,13 +149,18 @@ public sealed class PowerShellExecutionService
|
||||
{
|
||||
// Terminating error: surface as an error record + failed result.
|
||||
failureMessage = ex.Message;
|
||||
firstError ??= ScriptErrorInfo.From(ex.ErrorRecord);
|
||||
onRecord(new NormalizedOutputRecord(Shared.Contracts.Models.OutputStreamKind.Error, ex.Message));
|
||||
}
|
||||
|
||||
return new PowerShellExecutionResult(
|
||||
HadErrors: powerShell.HadErrors || failureMessage is not null,
|
||||
// Not powerShell.HadErrors: that is also set by a native command's non-zero
|
||||
// exit and by errors the script silenced, neither of which fails a script.
|
||||
HadErrors: firstError is not null || failureMessage is not null,
|
||||
WasCancelled: wasCancelled,
|
||||
FailureMessage: failureMessage);
|
||||
FailureMessage: failureMessage,
|
||||
ExitCode: host.ExitCode,
|
||||
Error: firstError);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
@@ -68,6 +68,7 @@ public sealed class PowerShellJobHandler(
|
||||
JobState finalState;
|
||||
int exitCode;
|
||||
string? message = null;
|
||||
ScriptErrorInfo? error = null;
|
||||
|
||||
// S3 file deliveries land before the script starts. Destination
|
||||
// paths expand on the device (%VAR% natively; $env:/$(…) through a
|
||||
@@ -114,6 +115,8 @@ public sealed class PowerShellJobHandler(
|
||||
else
|
||||
{
|
||||
var externalParameters = ExpandForExternal(boundParameters, expansions);
|
||||
// An external engine reports no exception: its first stderr line stands in.
|
||||
string? firstStderr = null;
|
||||
var external = await ExternalScriptRunner.RunAsync(
|
||||
command.Engine,
|
||||
command.Content,
|
||||
@@ -121,7 +124,15 @@ public sealed class PowerShellJobHandler(
|
||||
externalParameters,
|
||||
command.Secrets,
|
||||
string.IsNullOrWhiteSpace(command.SecretsVariableName) ? "Secrets" : command.SecretsVariableName,
|
||||
record => pending.Add(PublishOutputAsync(record, Interlocked.Increment(ref sequence))),
|
||||
record =>
|
||||
{
|
||||
if (record.Stream == OutputStreamKind.Error && !string.IsNullOrWhiteSpace(record.Text))
|
||||
{
|
||||
Interlocked.CompareExchange(ref firstStderr, record.Text, null);
|
||||
}
|
||||
|
||||
pending.Add(PublishOutputAsync(record, Interlocked.Increment(ref sequence)));
|
||||
},
|
||||
timeoutCts.Token,
|
||||
files: filesDict);
|
||||
|
||||
@@ -142,6 +153,12 @@ public sealed class PowerShellJobHandler(
|
||||
finalState = external.ExitCode == 0 ? JobState.Succeeded : JobState.Failed;
|
||||
exitCode = external.ExitCode;
|
||||
message = external.ExitCode == 0 ? null : $"Process exited with {external.ExitCode}.";
|
||||
if (external.ExitCode != 0 && firstStderr is not null)
|
||||
{
|
||||
error = new ScriptErrorInfo(null, null, firstStderr.Length > ScriptErrorInfo.MaxTextLength
|
||||
? firstStderr[..ScriptErrorInfo.MaxTextLength]
|
||||
: firstStderr);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -173,29 +190,28 @@ public sealed class PowerShellJobHandler(
|
||||
exitCode = -1;
|
||||
message = $"Execution exceeded the {timeout.TotalSeconds:F0}s timeout.";
|
||||
}
|
||||
else if (result.HadErrors)
|
||||
{
|
||||
finalState = JobState.Failed;
|
||||
exitCode = 1;
|
||||
message = result.FailureMessage;
|
||||
}
|
||||
else
|
||||
{
|
||||
finalState = JobState.Succeeded;
|
||||
exitCode = 0;
|
||||
(finalState, exitCode, message) = ScriptOutcome.From(result);
|
||||
if (finalState == JobState.Failed)
|
||||
{
|
||||
error = result.Error;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
jobHistory.MarkCompleted(jobId, finalState.ToString());
|
||||
await PublishStateAsync(finalState, exitCode, message);
|
||||
await PublishStateAsync(finalState, exitCode, message, error);
|
||||
|
||||
logger.LogInformation("Job {JobId} finished: {State}", jobId, finalState);
|
||||
logger.LogInformation("Job {JobId} finished: {State} ({ExitCode})", jobId, finalState, exitCode);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
logger.LogError(ex, "Job {JobId} crashed", jobId);
|
||||
jobHistory.MarkCompleted(jobId, JobState.Failed.ToString());
|
||||
await PublishStateAsync(JobState.Failed, exitCode: 1, message: ex.Message);
|
||||
await PublishStateAsync(
|
||||
JobState.Failed, exitCode: 1, message: ex.Message,
|
||||
new ScriptErrorInfo(ex.GetType().FullName, null, ex.Message));
|
||||
}
|
||||
finally
|
||||
{
|
||||
@@ -293,12 +309,14 @@ public sealed class PowerShellJobHandler(
|
||||
}
|
||||
}
|
||||
|
||||
async Task PublishStateAsync(JobState state, int? exitCode, string? message)
|
||||
async Task PublishStateAsync(JobState state, int? exitCode, string? message, ScriptErrorInfo? error = null)
|
||||
{
|
||||
await publisher.PublishAsync(
|
||||
NatsSubjects.JobState(jobId),
|
||||
MessageTypes.JobState,
|
||||
new JobStatePayload(jobId, publisher.ClientId, state, exitCode, DateTimeOffset.UtcNow, message),
|
||||
new JobStatePayload(
|
||||
jobId, publisher.ClientId, state, exitCode, DateTimeOffset.UtcNow, message,
|
||||
ErrorType: error?.Type, ErrorId: error?.Id, ErrorMessage: error?.Message),
|
||||
jobTarget,
|
||||
correlationId,
|
||||
causationId);
|
||||
|
||||
@@ -7,14 +7,17 @@ public static class PowerShellRunspaceFactory
|
||||
{
|
||||
private static bool modulesPathEnsured;
|
||||
|
||||
public static Runspace Create()
|
||||
/// <summary>A new opened runspace; <paramref name="host"/> replaces the engine's default host.</summary>
|
||||
public static Runspace Create(System.Management.Automation.Host.PSHost? host = null)
|
||||
{
|
||||
EnsureBundledModulesOnPath();
|
||||
|
||||
var initialSessionState = InitialSessionState.CreateDefault();
|
||||
initialSessionState.ExecutionPolicy = Microsoft.PowerShell.ExecutionPolicy.Bypass;
|
||||
|
||||
var runspace = RunspaceFactory.CreateRunspace(initialSessionState);
|
||||
var runspace = host is null
|
||||
? RunspaceFactory.CreateRunspace(initialSessionState)
|
||||
: RunspaceFactory.CreateRunspace(host, initialSessionState);
|
||||
runspace.Open();
|
||||
return runspace;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
using Shared.Contracts.Models;
|
||||
|
||||
namespace Agent.PowerShell;
|
||||
|
||||
/// <summary>How a finished in-process script reports: its state, exit code and summary.</summary>
|
||||
public sealed record ScriptOutcome(JobState State, int ExitCode, string? Message)
|
||||
{
|
||||
/// <summary>
|
||||
/// Mirrors <c>pwsh -File</c>: the script's own <c>exit N</c> decides (N≠0 is
|
||||
/// a failure, even with no errors; 0 a success, even after errors), a
|
||||
/// terminating error is exit 1, and a native command's failing
|
||||
/// <c>$LASTEXITCODE</c> counts only when the script exits with it. One
|
||||
/// difference, kept on purpose: anything on the error stream (Write-Error, a
|
||||
/// failing cmdlet) fails a script that never calls exit, with exit code 1.
|
||||
/// </summary>
|
||||
public static ScriptOutcome From(PowerShellExecutionResult result) => result switch
|
||||
{
|
||||
{ ExitCode: 0 } => new ScriptOutcome(JobState.Succeeded, 0, null),
|
||||
{ ExitCode: { } code } => new ScriptOutcome(
|
||||
JobState.Failed, code, result.FailureMessage ?? result.Error?.Message ?? $"The script exited with {code}."),
|
||||
{ HadErrors: true } => new ScriptOutcome(JobState.Failed, 1, result.FailureMessage),
|
||||
_ => new ScriptOutcome(JobState.Succeeded, 0, null),
|
||||
};
|
||||
}
|
||||
@@ -6,6 +6,7 @@ using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Serilog;
|
||||
using Shared.Contracts.Models;
|
||||
using Shared.Infrastructure.Constants;
|
||||
using Shared.Infrastructure.Logging;
|
||||
|
||||
@@ -74,7 +75,30 @@ public static class AgentHost
|
||||
// service mode (and never override GPO in any mode).
|
||||
builder.Services.AddSingleton(new AgentConfigurationLoader(registry, allowEnvironment: !runAsService));
|
||||
|
||||
builder.Services.AddSingleton(sp => CreateRemoval(
|
||||
sp.GetRequiredService<Upgrade.AgentRuntimeInfo>(),
|
||||
sp.GetRequiredService<Upgrade.UpgradeFolders>(),
|
||||
sp.GetRequiredService<ILoggerFactory>()));
|
||||
builder.Services.AddSingleton(sp =>
|
||||
{
|
||||
var removal = sp.GetRequiredService<Upgrade.AgentRemoval>();
|
||||
var enrollment = sp.GetRequiredService<EnrollmentService>();
|
||||
var logger = sp.GetRequiredService<ILoggerFactory>().CreateLogger<CredentialRenewal>();
|
||||
return new CredentialRenewal(
|
||||
new CredentialRenewalContext
|
||||
{
|
||||
Store = sp.GetRequiredService<ICredentialStore>(),
|
||||
RuntimeState = sp.GetRequiredService<RuntimeStateStore>(),
|
||||
Configuration = sp.GetRequiredService<AgentConfigurationLoader>(),
|
||||
Folders = sp.GetRequiredService<Upgrade.UpgradeFolders>(),
|
||||
OnRemoved = (removalId, cancellationToken) => CredentialRemoval.HandleAsync(removal, removalId, logger, cancellationToken),
|
||||
Reenroll = (configuration, cancellationToken) => enrollment.EnsureEnrolledAsync(configuration, cancellationToken),
|
||||
},
|
||||
logger);
|
||||
});
|
||||
|
||||
builder.Services.AddHostedService<AgentWorker>();
|
||||
builder.Services.AddHostedService<CredentialRenewalService>();
|
||||
// Keeps a self-signed server's CA current in the OS trust store across
|
||||
// renewals (no-op for a real/edge certificate).
|
||||
builder.Services.AddHostedService<CaRefreshService>();
|
||||
@@ -84,8 +108,29 @@ public static class AgentHost
|
||||
if (runAsService)
|
||||
{
|
||||
builder.Services.AddHostedService<Upgrade.HealthBeaconService>();
|
||||
builder.Services.AddHostedService<Upgrade.RemovalPickupService>();
|
||||
}
|
||||
|
||||
return builder.Build();
|
||||
}
|
||||
|
||||
private static Upgrade.AgentRemoval CreateRemoval(Upgrade.AgentRuntimeInfo runtime, Upgrade.UpgradeFolders folders, ILoggerFactory loggers)
|
||||
{
|
||||
var logger = loggers.CreateLogger<Upgrade.AgentRemoval>();
|
||||
var options = new Agent.Watchdog.WatchdogOptions
|
||||
{
|
||||
InstallDirectory = Path.GetFullPath(folders.InstallDirectory),
|
||||
DataDirectory = Path.GetFullPath(folders.DataDirectory),
|
||||
};
|
||||
return new Upgrade.AgentRemoval(
|
||||
new Upgrade.AgentRemovalContext
|
||||
{
|
||||
Folders = folders,
|
||||
CanRemove = runtime.CanSelfUpgrade,
|
||||
Watchdog = new Upgrade.WatchdogInstaller(folders.InstallDirectory, logger),
|
||||
SelfRunner = Agent.Watchdog.Removal.RemovalRunnerLauncher.Create(
|
||||
options, new Agent.Watchdog.WatchdogLog(directory: null, console: false), WatchdogRemovalFiles.AgentRunVerb),
|
||||
},
|
||||
logger);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -32,10 +32,12 @@ public sealed class AgentSessionFactory(
|
||||
JobHistoryStore jobHistory,
|
||||
AgentRuntimeInfo runtime,
|
||||
AgentHealthState health,
|
||||
AgentRemoval removal,
|
||||
ICredentialStore credentialStore,
|
||||
ILoggerFactory loggerFactory) : IAgentSessionFactory
|
||||
{
|
||||
public IAgentSession Create(NatsConnection connection, AgentCredentials credentials, AgentConfiguration configuration)
|
||||
=> new AgentSession(connection, credentials, configuration, hardwareIdentity, jobHistory, runtime, health, loggerFactory);
|
||||
=> new AgentSession(connection, credentials, configuration, hardwareIdentity, jobHistory, runtime, health, removal, credentialStore, loggerFactory);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -50,6 +52,7 @@ public sealed class AgentSession : IAgentSession
|
||||
private readonly CommandProcessor commands;
|
||||
private readonly ShellCommandListener shell;
|
||||
private readonly UpgradeOutcomeReporter upgradeOutcomes;
|
||||
private readonly RemovalOutcomeReporter removalOutcomes;
|
||||
private readonly NatsConnection connection;
|
||||
private readonly AgentHealthState health;
|
||||
private readonly ILogger<AgentSession> logger;
|
||||
@@ -64,6 +67,8 @@ public sealed class AgentSession : IAgentSession
|
||||
JobHistoryStore jobHistory,
|
||||
AgentRuntimeInfo runtime,
|
||||
AgentHealthState health,
|
||||
AgentRemoval removal,
|
||||
ICredentialStore credentialStore,
|
||||
ILoggerFactory loggerFactory)
|
||||
{
|
||||
this.connection = connection;
|
||||
@@ -96,13 +101,18 @@ public sealed class AgentSession : IAgentSession
|
||||
loggerFactory.CreateLogger<AgentUpgradeHandler>());
|
||||
upgradeOutcomes = new UpgradeOutcomeReporter(
|
||||
folders, upgradeEvents, () => health.Connected, credentials.ClientId, loggerFactory.CreateLogger<UpgradeOutcomeReporter>());
|
||||
var removalEvents = new NatsRemovalEvents(publisher);
|
||||
var removals = new AgentRemoveCommandHandler(
|
||||
removal, () => credentialStore.Load()?.EnrollmentId, removalEvents, credentials.ClientId, loggerFactory.CreateLogger<AgentRemoveCommandHandler>());
|
||||
removalOutcomes = new RemovalOutcomeReporter(
|
||||
folders, removalEvents, () => health.Connected, credentials.ClientId, loggerFactory.CreateLogger<RemovalOutcomeReporter>());
|
||||
|
||||
presence = new PresencePublisher(
|
||||
publisher, () => jobHandler.ActiveJobs, loggerFactory.CreateLogger<PresencePublisher>());
|
||||
inventory = new InventoryPublisher(
|
||||
collector, publisher, configuration.InventoryInterval.Value, loggerFactory.CreateLogger<InventoryPublisher>());
|
||||
commands = new CommandProcessor(
|
||||
connection, publisher, inventory, jobHandler, fileHandler, jobHistory, configuration, upgrades, health,
|
||||
connection, publisher, inventory, jobHandler, fileHandler, jobHistory, configuration, upgrades, removals, health,
|
||||
loggerFactory.CreateLogger<CommandProcessor>());
|
||||
shell = new ShellCommandListener(
|
||||
connection, publisher, configuration, loggerFactory.CreateLogger<ShellCommandListener>());
|
||||
@@ -126,7 +136,8 @@ public sealed class AgentSession : IAgentSession
|
||||
inventory.RunAsync(cancellationToken),
|
||||
commands.RunAsync(cancellationToken),
|
||||
shell.RunAsync(cancellationToken),
|
||||
upgradeOutcomes.RunAsync(cancellationToken));
|
||||
upgradeOutcomes.RunAsync(cancellationToken),
|
||||
removalOutcomes.RunAsync(cancellationToken));
|
||||
}
|
||||
finally
|
||||
{
|
||||
|
||||
@@ -17,6 +17,7 @@ namespace Agent.Service;
|
||||
public sealed class AgentWorker(
|
||||
AgentConfigurationLoader configurationLoader,
|
||||
EnrollmentService enrollment,
|
||||
ICredentialStore credentialStore,
|
||||
RuntimeStateStore runtimeState,
|
||||
IAgentSessionFactory sessionFactory,
|
||||
ILoggerFactory loggerFactory,
|
||||
@@ -51,7 +52,7 @@ public sealed class AgentWorker(
|
||||
continue;
|
||||
}
|
||||
|
||||
var opts = AgentNatsOptionsFactory.Build(configuration.ServerUrl, credentials, configuration, loggerFactory);
|
||||
var opts = AgentNatsOptionsFactory.Build(configuration.ServerUrl, credentials, configuration, loggerFactory, credentialStore.Load);
|
||||
|
||||
await using var nats = new NatsConnection(opts);
|
||||
|
||||
|
||||
@@ -0,0 +1,295 @@
|
||||
using Agent.Messaging;
|
||||
using Agent.Watchdog.Removal;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Shared.Contracts.Messages;
|
||||
using Shared.Contracts.Models;
|
||||
using Shared.Contracts.Serialization;
|
||||
using Shared.Infrastructure.Nats;
|
||||
|
||||
namespace Agent.Upgrade;
|
||||
|
||||
/// <summary>Where removal progress goes: the agent event subject in production, a list in tests.</summary>
|
||||
public interface IRemovalEvents
|
||||
{
|
||||
Task PublishAsync(AgentRemovalEventPayload payload, CancellationToken cancellationToken);
|
||||
}
|
||||
|
||||
public sealed class NatsRemovalEvents(AgentPublisher publisher) : IRemovalEvents
|
||||
{
|
||||
public async Task PublishAsync(AgentRemovalEventPayload payload, CancellationToken cancellationToken)
|
||||
=> await publisher.PublishAsync(
|
||||
NatsSubjects.AgentEvent(publisher.ClientId), MessageTypes.AgentRemoval, payload, cancellationToken: cancellationToken);
|
||||
}
|
||||
|
||||
/// <summary>What the agent's side of a removal works with; every seam a test needs is here.</summary>
|
||||
public sealed class AgentRemovalContext
|
||||
{
|
||||
public required UpgradeFolders Folders { get; init; }
|
||||
|
||||
/// <summary>Only the installed service removes itself, as only it upgrades itself.</summary>
|
||||
public required bool CanRemove { get; init; }
|
||||
|
||||
public Func<string> InstallKind { get; init; } = () => AgentInstall.Current;
|
||||
|
||||
public required IWatchdogInstaller Watchdog { get; init; }
|
||||
|
||||
/// <summary>Copies this binary out and starts its <c>remove-self</c> verb, when no watchdog takes the removal.</summary>
|
||||
public required IRemovalRunnerLauncher SelfRunner { get; init; }
|
||||
|
||||
public Func<DateTimeOffset> UtcNow { get; init; } = () => DateTimeOffset.UtcNow;
|
||||
|
||||
public TimeSpan PickupDeadline { get; init; } = WatchdogRemovalFiles.PickupDeadline;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Hands a removal over: writes <c>removal.json</c> and makes sure the watchdog
|
||||
/// runs to take it. A device with no watchdog that can (never had one, its binary
|
||||
/// gone, its service disabled), or one that has not taken it within the pickup
|
||||
/// deadline, runs the same runner from a copy of the agent's own binary.
|
||||
/// Nothing here needs NATS: a removal found through a refused credential renewal
|
||||
/// comes when the connection may be down for good.
|
||||
/// </summary>
|
||||
public sealed class AgentRemoval(AgentRemovalContext context, ILogger logger)
|
||||
{
|
||||
private UpgradeFolders Folders => context.Folders;
|
||||
|
||||
/// <summary>A removal waits for or has a runner.</summary>
|
||||
public bool UnderWay => Read() is { Phase: not WatchdogRemovalPhases.Failed };
|
||||
|
||||
/// <summary>
|
||||
/// Why this removal cannot start now (<see cref="DeviceRemovalReasons"/>), or
|
||||
/// null. <paramref name="strict"/> also counts an unreported upgrade outcome as
|
||||
/// busy: fine for a command, which means a connection that will report it, but
|
||||
/// never for a renewal that was refused, which may have none.
|
||||
/// </summary>
|
||||
public string? Refusal(bool strict)
|
||||
{
|
||||
if (!context.CanRemove)
|
||||
{
|
||||
return DeviceRemovalReasons.Unsupported;
|
||||
}
|
||||
|
||||
return (strict ? Folders.UpgradePending : Folders.UpgradeUnderWay) ? DeviceRemovalReasons.Busy : null;
|
||||
}
|
||||
|
||||
/// <summary>Writes the removal down and hands it over; null when <c>removal.json</c> now carries it, otherwise why it could not be written.</summary>
|
||||
public async Task<string?> HandOverAsync(Guid removalId, CancellationToken cancellationToken)
|
||||
{
|
||||
var removal = new WatchdogRemoval(removalId, context.InstallKind(), WatchdogRemovalPhases.Requested, 0, null, context.UtcNow());
|
||||
try
|
||||
{
|
||||
WatchdogFileStore.Write(Folders.Removal, removal, WatchdogJsonContext.Default.WatchdogRemoval);
|
||||
}
|
||||
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
|
||||
{
|
||||
// Nothing was handed over, and nothing can carry the reason; the server asks again.
|
||||
logger.LogError(exception, "Removal {RemovalId} could not be written to {Path}", removalId, Folders.Removal);
|
||||
return $"The removal could not be written ({exception.Message}).";
|
||||
}
|
||||
|
||||
logger.LogWarning("Removal {RemovalId}: this device was removed from the server; the agent uninstalls itself", removalId);
|
||||
var problem = await context.Watchdog.EnsureRunningAsync(Folders.InstallDirectory, cancellationToken);
|
||||
if (problem is not null)
|
||||
{
|
||||
logger.LogWarning("Removal {RemovalId}: no watchdog can take it ({Problem}); the agent starts the runner itself", removalId, problem);
|
||||
StartSelfRunner(removal);
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
/// <summary>The fallback for a removal the watchdog has not taken in time; true when it acted.</summary>
|
||||
public bool CheckPickup()
|
||||
{
|
||||
if (Read() is not { Phase: WatchdogRemovalPhases.Requested } removal || context.UtcNow() - removal.RequestedUtc < context.PickupDeadline)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
logger.LogWarning("Removal {RemovalId}: the watchdog has not taken it; the agent starts the runner itself", removal.RemovalId);
|
||||
StartSelfRunner(removal);
|
||||
return true;
|
||||
}
|
||||
|
||||
/// <summary>A runner that does not start leaves the reason in <c>removal.json</c>: the reporter tells the server, and the agent carries on.</summary>
|
||||
private void StartSelfRunner(WatchdogRemoval removal)
|
||||
{
|
||||
try
|
||||
{
|
||||
var (directory, problem) = context.SelfRunner.Prepare(removal.RemovalId);
|
||||
if (problem is null)
|
||||
{
|
||||
WatchdogFileStore.Write(Folders.Removal,
|
||||
removal with { Phase = WatchdogRemovalPhases.Uninstalling, RunnerDirectory = directory },
|
||||
WatchdogJsonContext.Default.WatchdogRemoval);
|
||||
problem = context.SelfRunner.Start(removal.RemovalId, directory!);
|
||||
}
|
||||
|
||||
if (problem is not null)
|
||||
{
|
||||
var why = $"The removal runner did not start: {problem}.";
|
||||
logger.LogError("Removal {RemovalId}: {Why}", removal.RemovalId, why);
|
||||
WatchdogFileStore.Write(Folders.Removal,
|
||||
removal with { Phase = WatchdogRemovalPhases.Failed, LastError = why },
|
||||
WatchdogJsonContext.Default.WatchdogRemoval);
|
||||
}
|
||||
}
|
||||
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
|
||||
{
|
||||
logger.LogError(exception, "Removal {RemovalId}: {Path} could not be written", removal.RemovalId, Folders.Removal);
|
||||
}
|
||||
}
|
||||
|
||||
private WatchdogRemoval? Read() => WatchdogFileStore.Read(Folders.Removal, WatchdogJsonContext.Default.WatchdogRemoval);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Handles <see cref="MessageTypes.CommandAgentRemove"/>. Idempotent by
|
||||
/// <c>removal.json</c>, and deliberately not by removal id: a refused command is
|
||||
/// asked again under the same id. The command may be old (it waits in its stream),
|
||||
/// so it acts only on the enrollment this agent holds.
|
||||
/// </summary>
|
||||
public sealed class AgentRemoveCommandHandler(
|
||||
AgentRemoval removal, Func<Guid?> currentEnrollment, IRemovalEvents events, string clientId, ILogger logger)
|
||||
{
|
||||
public async Task HandleAsync(AgentRemoveCommandPayload? command, CancellationToken cancellationToken)
|
||||
{
|
||||
if (command is null || command.RemovalId == Guid.Empty)
|
||||
{
|
||||
logger.LogWarning("Remove command without a removal id; ignored");
|
||||
return;
|
||||
}
|
||||
|
||||
if (currentEnrollment() != command.EnrollmentId)
|
||||
{
|
||||
logger.LogWarning("Removal {RemovalId} is for enrollment {EnrollmentId}, which this agent no longer holds; ignored", command.RemovalId, command.EnrollmentId);
|
||||
await PublishAsync(command.RemovalId, DeviceRemovalStates.Failed, DeviceRemovalReasons.StaleEnrollment,
|
||||
"The command is for an enrollment this agent no longer holds; nothing was changed.", cancellationToken);
|
||||
return;
|
||||
}
|
||||
|
||||
if (removal.UnderWay)
|
||||
{
|
||||
logger.LogInformation("Removal {RemovalId} asked again while one is under way", command.RemovalId);
|
||||
await PublishAsync(command.RemovalId, DeviceRemovalStates.Uninstalling, null, null, cancellationToken);
|
||||
return;
|
||||
}
|
||||
|
||||
if (removal.Refusal(strict: true) is { } refusal)
|
||||
{
|
||||
logger.LogWarning("Removal {RemovalId} refused: {Reason}", command.RemovalId, refusal);
|
||||
await PublishAsync(command.RemovalId, DeviceRemovalStates.Failed, refusal, refusal == DeviceRemovalReasons.Busy
|
||||
? "An agent upgrade is in progress on this device."
|
||||
: "The agent runs in the foreground or a development host; only the installed service removes itself.", cancellationToken);
|
||||
return;
|
||||
}
|
||||
|
||||
await PublishAsync(command.RemovalId, DeviceRemovalStates.Uninstalling, null, null, cancellationToken);
|
||||
if (await removal.HandOverAsync(command.RemovalId, cancellationToken) is { } problem)
|
||||
{
|
||||
await PublishAsync(command.RemovalId, DeviceRemovalStates.Failed, DeviceRemovalReasons.Uninstall, problem, cancellationToken);
|
||||
}
|
||||
}
|
||||
|
||||
private async Task PublishAsync(Guid removalId, string state, string? reason, string? detail, CancellationToken cancellationToken)
|
||||
{
|
||||
try
|
||||
{
|
||||
await events.PublishAsync(new AgentRemovalEventPayload(removalId, clientId, state, reason, detail, DateTimeOffset.UtcNow), cancellationToken);
|
||||
}
|
||||
catch (Exception exception) when (exception is not OperationCanceledException || !cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
// The removal goes on regardless; the server asks again if it heard nothing.
|
||||
logger.LogWarning(exception, "Removal {RemovalId}: {State} could not be published", removalId, state);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reports a removal that failed once the agent is connected (the runner started
|
||||
/// the agent again and left the reason in <c>removal.json</c>), then deletes the
|
||||
/// file: the server asks again, and the device carries on meanwhile.
|
||||
/// </summary>
|
||||
public sealed class RemovalOutcomeReporter(UpgradeFolders folders, IRemovalEvents events, Func<bool> connected, string clientId, ILogger logger)
|
||||
{
|
||||
public static readonly TimeSpan CheckInterval = TimeSpan.FromSeconds(5);
|
||||
|
||||
public async Task RunAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
while (!cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
if (connected())
|
||||
{
|
||||
await ReportPendingAsync(cancellationToken);
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
await Task.Delay(CheckInterval, cancellationToken);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Publishes and deletes a failed removal; true when one was reported.</summary>
|
||||
public async Task<bool> ReportPendingAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
if (WatchdogFileStore.Read(folders.Removal, WatchdogJsonContext.Default.WatchdogRemoval) is not { Phase: WatchdogRemovalPhases.Failed } removal)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
// An MSI that uninstalled leaves no agent to say anything else: a failure
|
||||
// on an MSI install is always Windows Installer's.
|
||||
var reason = removal.InstallKind == AgentInstallKinds.Msi ? DeviceRemovalReasons.Msi : DeviceRemovalReasons.Uninstall;
|
||||
try
|
||||
{
|
||||
await events.PublishAsync(
|
||||
new AgentRemovalEventPayload(removal.RemovalId, clientId, DeviceRemovalStates.Failed, reason, removal.LastError, DateTimeOffset.UtcNow),
|
||||
cancellationToken);
|
||||
}
|
||||
catch (Exception exception) when (exception is not OperationCanceledException || !cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
// Kept for the next check.
|
||||
logger.LogWarning(exception, "Removal {RemovalId}: its failure could not be published; retrying", removal.RemovalId);
|
||||
return false;
|
||||
}
|
||||
|
||||
logger.LogWarning("Removal {RemovalId} failed and was reported: {Reason} {Detail}", removal.RemovalId, reason, removal.LastError);
|
||||
WatchdogFileStore.TryDelete(folders.Removal);
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Watches for a removal the watchdog does not take; independent of any connection, like the renewal that may have started it.</summary>
|
||||
public sealed class RemovalPickupService(AgentRemoval removal, ILogger<RemovalPickupService> logger) : BackgroundService
|
||||
{
|
||||
private static readonly TimeSpan CheckInterval = TimeSpan.FromSeconds(5);
|
||||
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
using var timer = new PeriodicTimer(CheckInterval);
|
||||
try
|
||||
{
|
||||
while (await timer.WaitForNextTickAsync(stoppingToken))
|
||||
{
|
||||
try
|
||||
{
|
||||
removal.CheckPickup();
|
||||
}
|
||||
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
|
||||
{
|
||||
logger.LogWarning(exception, "The removal's hand-over could not be checked");
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// Shutdown.
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -161,7 +161,16 @@ public sealed class AgentUpgradeHandler(AgentUpgradeContext context, IUpgradeEve
|
||||
return;
|
||||
}
|
||||
|
||||
if (Folders.UpgradePending || !context.Gate.TryEnter())
|
||||
// This version renewed its credential to a short-lived one; the target could not.
|
||||
if (AgentVersions.Compare(command.Version, DeviceRemovalDefaults.MinimumAgentVersion) < 0
|
||||
&& AgentVersions.Compare(context.CurrentVersion, DeviceRemovalDefaults.MinimumAgentVersion) >= 0)
|
||||
{
|
||||
await FailAsync(command, AgentUpgradeReasons.Unsupported,
|
||||
$"{command.Version} cannot renew its credential (that takes {DeviceRemovalDefaults.MinimumAgentVersion}) and would lose the server within a day.", cancellationToken);
|
||||
return;
|
||||
}
|
||||
|
||||
if (Folders.UpgradePending || File.Exists(Folders.Removal) || !context.Gate.TryEnter())
|
||||
{
|
||||
await FailAsync(command, AgentUpgradeReasons.Busy, "Another upgrade is under way on this device.", cancellationToken);
|
||||
return;
|
||||
|
||||
@@ -6,6 +6,8 @@ namespace Agent.Upgrade;
|
||||
/// <summary>Where the agent and the watchdog meet (<c><data>/upgrade/</c>) and where the agent is installed.</summary>
|
||||
public sealed class UpgradeFolders(string dataDirectory, string installDirectory)
|
||||
{
|
||||
public string DataDirectory { get; } = dataDirectory;
|
||||
|
||||
/// <summary>The installed layout: the data folder and the running binary's own folder.</summary>
|
||||
public static UpgradeFolders Installed { get; } = new(Branding.AgentProgramDataDirectory, AppContext.BaseDirectory);
|
||||
|
||||
@@ -35,6 +37,11 @@ public sealed class UpgradeFolders(string dataDirectory, string installDirectory
|
||||
|
||||
public string Outcome => Path.Combine(Root, WatchdogFiles.Outcome);
|
||||
|
||||
public string Removal => Path.Combine(Root, WatchdogFiles.Removal);
|
||||
|
||||
/// <summary>A hand-over, a swap or an MSI run: what runs now may still be rolled back.</summary>
|
||||
public bool UpgradeUnderWay => File.Exists(Handover) || File.Exists(Journal) || File.Exists(MsiJournal);
|
||||
|
||||
/// <summary>A hand-over, a swap, an MSI run or an outcome not yet reported: another upgrade must wait.</summary>
|
||||
public bool UpgradePending => File.Exists(Handover) || File.Exists(Journal) || File.Exists(MsiJournal) || File.Exists(Outcome);
|
||||
|
||||
|
||||
@@ -98,7 +98,11 @@ public static class DeviceEndpoints
|
||||
await groups.GetGroupsForDeviceAsync(clientId, cancellationToken)));
|
||||
}
|
||||
|
||||
/// <summary>The device's phase trail (enrolled, renamed, OS changed), newest first.</summary>
|
||||
/// <summary>
|
||||
/// The device's phase trail (enrolled, renamed, OS changed), newest first. A
|
||||
/// removed device has no row but keeps its trail, within the scope its removal
|
||||
/// recorded.
|
||||
/// </summary>
|
||||
private static async Task<IResult> GetHistoryAsync(
|
||||
string clientId, int? limit, AppDbContext db, DeviceScopeService scope,
|
||||
HttpContext context, CancellationToken cancellationToken)
|
||||
@@ -108,8 +112,10 @@ public static class DeviceEndpoints
|
||||
return Results.BadRequest();
|
||||
}
|
||||
|
||||
if (!await scope.IsAllowedAsync(context.User, clientId, cancellationToken)
|
||||
|| !await db.Devices.AnyAsync(d => d.ClientId == clientId, cancellationToken))
|
||||
var reachable = await db.Devices.AnyAsync(d => d.ClientId == clientId, cancellationToken)
|
||||
? await scope.IsAllowedAsync(context.User, clientId, cancellationToken)
|
||||
: await IsRemovedWithinScopeAsync(clientId, db, scope, context, cancellationToken);
|
||||
if (!reachable)
|
||||
{
|
||||
return Results.NotFound();
|
||||
}
|
||||
@@ -125,6 +131,16 @@ public static class DeviceEndpoints
|
||||
return Results.Ok(events);
|
||||
}
|
||||
|
||||
private static async Task<bool> IsRemovedWithinScopeAsync(
|
||||
string clientId, AppDbContext db, DeviceScopeService scope, HttpContext context, CancellationToken cancellationToken)
|
||||
{
|
||||
var scopeGroupIds = await scope.GetScopeGroupIdsAsync(context.User, cancellationToken);
|
||||
var removals = await db.DeviceRemovals.AsNoTracking()
|
||||
.Where(r => r.ClientId == clientId)
|
||||
.ToListAsync(cancellationToken);
|
||||
return removals.Any(r => Removals.DeviceRemovalService.IsVisibleTo(r, scopeGroupIds));
|
||||
}
|
||||
|
||||
private static async Task<IResult> RefreshInventoryAsync(
|
||||
string clientId,
|
||||
AppDbContext db,
|
||||
|
||||
@@ -23,6 +23,13 @@ public static class DeviceHistory
|
||||
/// <summary>An upgrade failed and the agent stayed where it was (detail carries the reason).</summary>
|
||||
public const string AgentUpgradeFailed = "agent-upgrade-failed";
|
||||
|
||||
/// <summary>An operator removed the device; its agent uninstalls now or when it returns.</summary>
|
||||
public const string RemovalRequested = "device-removal-requested";
|
||||
/// <summary>The agent has gone, or the device enrolled again and the removal closed.</summary>
|
||||
public const string Removed = "device-removed";
|
||||
/// <summary>An operator stopped trying: the device stays blocked, its agent stays installed.</summary>
|
||||
public const string RemovalStopped = "device-removal-stopped";
|
||||
|
||||
/// <summary>
|
||||
/// Applies a reported hostname and (when <paramref name="reportsOs"/>) OS
|
||||
/// to the device, recording a rename / OS change against the previous
|
||||
@@ -77,6 +84,19 @@ public static class DeviceHistory
|
||||
public static void RecordAgentUpgrade(AppDbContext db, string clientId, string action, string detail)
|
||||
=> Record(db, clientId, action, detail, DateTimeOffset.UtcNow);
|
||||
|
||||
/// <summary>Records a removal request, with who asked and their note.</summary>
|
||||
public static void RecordRemovalRequested(AppDbContext db, string clientId, string? requestedBy, string? note)
|
||||
=> Record(db, clientId, RemovalRequested,
|
||||
$"By {requestedBy ?? "unknown"}{(string.IsNullOrWhiteSpace(note) ? string.Empty : $" · {note}")}", DateTimeOffset.UtcNow);
|
||||
|
||||
/// <summary>Records the end of a removal.</summary>
|
||||
public static void RecordRemoved(AppDbContext db, string clientId, string detail)
|
||||
=> Record(db, clientId, Removed, detail, DateTimeOffset.UtcNow);
|
||||
|
||||
/// <summary>Records an operator stopping a removal.</summary>
|
||||
public static void RecordRemovalStopped(AppDbContext db, string clientId, string? stoppedBy)
|
||||
=> Record(db, clientId, RemovalStopped, $"By {stoppedBy ?? "unknown"}", DateTimeOffset.UtcNow);
|
||||
|
||||
/// <summary>"Windows PE 10.0.26100"; just the caption when the version is unknown.</summary>
|
||||
public static string Describe(string? caption, string? version)
|
||||
{
|
||||
|
||||
@@ -46,6 +46,14 @@ public sealed class DeviceScopeService(
|
||||
return allowed = clientIds;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The groups the caller's roles are limited to; null when unrestricted. A
|
||||
/// removed device is in no group any more, so its removal is matched on these.
|
||||
/// </summary>
|
||||
public async Task<IReadOnlyList<Guid>?> GetScopeGroupIdsAsync(
|
||||
ClaimsPrincipal user, CancellationToken cancellationToken)
|
||||
=> (await permissions.ResolveAccessAsync(user, cancellationToken)).ScopeGroupIds;
|
||||
|
||||
/// <summary>True when the caller may act on the device.</summary>
|
||||
public async Task<bool> IsAllowedAsync(
|
||||
ClaimsPrincipal user, string clientId, CancellationToken cancellationToken)
|
||||
|
||||
@@ -14,6 +14,7 @@ public static class Permissions
|
||||
|
||||
public const string DevicesView = "devices.view";
|
||||
public const string DevicesManage = "devices.manage";
|
||||
public const string DevicesRemove = "devices.remove";
|
||||
public const string GroupsView = "groups.view";
|
||||
public const string GroupsManage = "groups.manage";
|
||||
public const string ScriptsView = "scripts.view";
|
||||
@@ -38,6 +39,7 @@ public static class Permissions
|
||||
[
|
||||
new(DevicesView, "View devices", "See the device list, details and inventory."),
|
||||
new(DevicesManage, "Manage devices", "Refresh inventory and edit device tags."),
|
||||
new(DevicesRemove, "Remove devices", "Remove devices from the system and uninstall their agents."),
|
||||
]),
|
||||
new("Groups",
|
||||
[
|
||||
|
||||
@@ -33,6 +33,7 @@ public class AppDbContext(DbContextOptions<AppDbContext> options) : DbContext(op
|
||||
public DbSet<SecretsProviderEntity> SecretsProviders => Set<SecretsProviderEntity>();
|
||||
public DbSet<AgentPackageEntity> AgentPackages => Set<AgentPackageEntity>();
|
||||
public DbSet<AgentUpgradeEntity> AgentUpgrades => Set<AgentUpgradeEntity>();
|
||||
public DbSet<DeviceRemovalEntity> DeviceRemovals => Set<DeviceRemovalEntity>();
|
||||
|
||||
/// <summary>
|
||||
/// SQLite cannot ORDER BY / compare DateTimeOffset natively, so it stores
|
||||
@@ -66,7 +67,7 @@ public class AppDbContext(DbContextOptions<AppDbContext> options) : DbContext(op
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Every write to a trigger or an agent upgrade gets a fresh stamp; the
|
||||
/// Every write to a trigger, an agent upgrade or a removal gets a fresh stamp; the
|
||||
/// update is conditioned on the stamp it was read with, so a concurrent
|
||||
/// write surfaces as DbUpdateConcurrencyException instead of silently winning.
|
||||
/// </summary>
|
||||
@@ -87,6 +88,14 @@ public class AppDbContext(DbContextOptions<AppDbContext> options) : DbContext(op
|
||||
entry.Entity.ConcurrencyStamp = Guid.NewGuid();
|
||||
}
|
||||
}
|
||||
|
||||
foreach (var entry in ChangeTracker.Entries<DeviceRemovalEntity>())
|
||||
{
|
||||
if (entry.State is EntityState.Added or EntityState.Modified)
|
||||
{
|
||||
entry.Entity.ConcurrencyStamp = Guid.NewGuid();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
protected override void OnModelCreating(ModelBuilder modelBuilder)
|
||||
@@ -220,7 +229,12 @@ public class AppDbContext(DbContextOptions<AppDbContext> options) : DbContext(op
|
||||
entity.HasIndex(e => e.CreatedUtc);
|
||||
entity.HasIndex(e => new { e.ScriptId, e.RunId });
|
||||
entity.HasIndex(e => e.RunId);
|
||||
// The dispatcher's every-tick query: queued jobs whose time has come.
|
||||
entity.HasIndex(e => new { e.State, e.NotBeforeUtc });
|
||||
entity.Property(e => e.State).HasConversion<string>();
|
||||
entity.Property(e => e.ErrorType).HasMaxLength(300);
|
||||
entity.Property(e => e.ErrorId).HasMaxLength(300);
|
||||
entity.Property(e => e.ErrorMessage).HasMaxLength(2000);
|
||||
});
|
||||
|
||||
modelBuilder.Entity<RunEntity>(entity =>
|
||||
@@ -287,6 +301,7 @@ public class AppDbContext(DbContextOptions<AppDbContext> options) : DbContext(op
|
||||
// The schedule sweep queries due scheduled triggers by next-run.
|
||||
entity.HasIndex(e => e.NextRunUtc);
|
||||
entity.Property(e => e.TriggerType).HasConversion<string>();
|
||||
entity.Property(e => e.StaggerBatchSize).HasDefaultValue(1);
|
||||
entity.Property(e => e.ConcurrencyStamp).IsConcurrencyToken();
|
||||
});
|
||||
|
||||
@@ -345,5 +360,20 @@ public class AppDbContext(DbContextOptions<AppDbContext> options) : DbContext(op
|
||||
entity.Property(e => e.ConcurrencyStamp).IsConcurrencyToken();
|
||||
});
|
||||
|
||||
modelBuilder.Entity<DeviceRemovalEntity>(entity =>
|
||||
{
|
||||
entity.ToTable("DeviceRemovals");
|
||||
entity.HasKey(e => e.Id);
|
||||
entity.Property(e => e.State).HasMaxLength(16);
|
||||
entity.Property(e => e.Reason).HasMaxLength(32);
|
||||
entity.Property(e => e.Detail).HasMaxLength(2000);
|
||||
entity.Property(e => e.RequestedBy).HasMaxLength(256);
|
||||
entity.Property(e => e.Note).HasMaxLength(500);
|
||||
// Enrollment, renewal and sightings look up by client id; the scheduler sweeps by state.
|
||||
entity.HasIndex(e => e.ClientId);
|
||||
entity.HasIndex(e => e.State);
|
||||
entity.HasIndex(e => e.RequestedUtc);
|
||||
entity.Property(e => e.ConcurrencyStamp).IsConcurrencyToken();
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
@@ -289,6 +289,12 @@ public sealed class ScriptTriggerEntity
|
||||
/// <summary>Execution (job RunId) the last firing created, for drill-in.</summary>
|
||||
public Guid? LastRunId { get; set; }
|
||||
|
||||
/// <summary>Spreads each firing's starts over this many seconds; 0 starts them all at once.</summary>
|
||||
public int StaggerSeconds { get; set; }
|
||||
|
||||
/// <summary>Devices started together in each batch of a stagger.</summary>
|
||||
public int StaggerBatchSize { get; set; } = 1;
|
||||
|
||||
/// <summary>
|
||||
/// Optimistic-concurrency token, renewed on every save (AppDbContext), so
|
||||
/// a schedule edited while it is firing is not overwritten by the sweep.
|
||||
@@ -412,6 +418,12 @@ public sealed class RunEntity
|
||||
|
||||
/// <summary>JSON array of RunSkip: devices targeted but given no job.</summary>
|
||||
public string SkippedJson { get; set; } = "[]";
|
||||
|
||||
/// <summary>
|
||||
/// The sealed RunSnapshot the dispatcher sends this run's jobs from
|
||||
/// (arguments can be sensitive); set only while it has jobs left to send.
|
||||
/// </summary>
|
||||
public string? SnapshotProtected { get; set; }
|
||||
}
|
||||
|
||||
/// <summary>A targeted device that got no job, and why (orphaned, offline).</summary>
|
||||
@@ -451,6 +463,21 @@ public sealed class JobEntity
|
||||
public int? ExitCode { get; set; }
|
||||
public string? RequestedBy { get; set; }
|
||||
public string? OutputSummary { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Set while the dispatcher owns sending this job: a Queued job waits until
|
||||
/// this time (its stagger slot), then is sent. Null on a job sent directly.
|
||||
/// </summary>
|
||||
public DateTimeOffset? NotBeforeUtc { get; set; }
|
||||
|
||||
/// <summary>The first error's exception type, as the agent reported it; null from older agents and external engines.</summary>
|
||||
public string? ErrorType { get; set; }
|
||||
|
||||
/// <summary>The first error's FullyQualifiedErrorId, as the agent reported it.</summary>
|
||||
public string? ErrorId { get; set; }
|
||||
|
||||
/// <summary>The first error's message, or an external engine's first stderr line.</summary>
|
||||
public string? ErrorMessage { get; set; }
|
||||
}
|
||||
|
||||
public sealed class JobOutputEntity
|
||||
@@ -561,6 +588,67 @@ public sealed class AgentUpgradeEntity
|
||||
public Guid ConcurrencyStamp { get; set; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// One device's removal. The device row is gone from the moment it is requested;
|
||||
/// this stands in for it until the agent has gone, and stays afterwards as the
|
||||
/// history (old runs still find the name here, and bootstrap the date of removal).
|
||||
/// A client id has at most one open removal: one is only added as the device row
|
||||
/// is deleted, and enrolling again closes it as the row comes back.
|
||||
/// </summary>
|
||||
public sealed class DeviceRemovalEntity
|
||||
{
|
||||
public Guid Id { get; set; }
|
||||
public required string ClientId { get; set; }
|
||||
|
||||
/// <summary>The enrolled key: a credential renewal must still prove possession of it.</summary>
|
||||
public required string NkeyPublic { get; set; }
|
||||
|
||||
/// <summary>Carried by the remove command; the agent ignores one for another enrollment.</summary>
|
||||
public Guid EnrollmentId { get; set; }
|
||||
|
||||
public string? Hostname { get; set; }
|
||||
public string? OsCaption { get; set; }
|
||||
|
||||
/// <summary>Running version: as removed, then from presence.</summary>
|
||||
public string? AgentVersion { get; set; }
|
||||
|
||||
/// <summary>A DeviceRemovalStates value.</summary>
|
||||
public required string State { get; set; }
|
||||
|
||||
/// <summary>A DeviceRemovalReasons value.</summary>
|
||||
public string? Reason { get; set; }
|
||||
|
||||
public string? Detail { get; set; }
|
||||
|
||||
/// <summary>Times the command was handed to JetStream.</summary>
|
||||
public int Attempts { get; set; }
|
||||
|
||||
public string? RequestedBy { get; set; }
|
||||
public string? Note { get; set; }
|
||||
public DateTimeOffset RequestedUtc { get; set; }
|
||||
|
||||
/// <summary>Last change of state; sightings move only <see cref="LastSeenUtc"/>.</summary>
|
||||
public DateTimeOffset UpdatedUtc { get; set; }
|
||||
|
||||
public DateTimeOffset? SentUtc { get; set; }
|
||||
public DateTimeOffset? LastSeenUtc { get; set; }
|
||||
public DateTimeOffset? FinishedUtc { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// JSON array of the groups the device was in, nesting included, when it was
|
||||
/// removed: a scoped role's device scope is its groups, and it keeps seeing
|
||||
/// the removals of the devices that were in them.
|
||||
/// </summary>
|
||||
public string? GroupIdsJson { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Renewed on every save (AppDbContext): the scheduler, the event consumer,
|
||||
/// credential renewal and an operator's stop may touch the same row on
|
||||
/// different nodes, and the later write is refused instead of silently winning.
|
||||
/// </summary>
|
||||
public Guid ConcurrencyStamp { get; set; }
|
||||
}
|
||||
|
||||
public sealed class ShellSessionEntity
|
||||
{
|
||||
public Guid Id { get; set; }
|
||||
|
||||
+1301
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,67 @@
|
||||
using System;
|
||||
using Microsoft.EntityFrameworkCore.Migrations;
|
||||
|
||||
#nullable disable
|
||||
|
||||
namespace Backend.Data.Migrations.Postgres
|
||||
{
|
||||
/// <inheritdoc />
|
||||
public partial class DeviceRemovals : Migration
|
||||
{
|
||||
/// <inheritdoc />
|
||||
protected override void Up(MigrationBuilder migrationBuilder)
|
||||
{
|
||||
migrationBuilder.CreateTable(
|
||||
name: "DeviceRemovals",
|
||||
columns: table => new
|
||||
{
|
||||
Id = table.Column<Guid>(type: "uuid", nullable: false),
|
||||
ClientId = table.Column<string>(type: "text", nullable: false),
|
||||
NkeyPublic = table.Column<string>(type: "text", nullable: false),
|
||||
EnrollmentId = table.Column<Guid>(type: "uuid", nullable: false),
|
||||
Hostname = table.Column<string>(type: "text", nullable: true),
|
||||
OsCaption = table.Column<string>(type: "text", nullable: true),
|
||||
AgentVersion = table.Column<string>(type: "text", nullable: true),
|
||||
State = table.Column<string>(type: "character varying(16)", maxLength: 16, nullable: false),
|
||||
Reason = table.Column<string>(type: "character varying(32)", maxLength: 32, nullable: true),
|
||||
Detail = table.Column<string>(type: "character varying(2000)", maxLength: 2000, nullable: true),
|
||||
Attempts = table.Column<int>(type: "integer", nullable: false),
|
||||
RequestedBy = table.Column<string>(type: "character varying(256)", maxLength: 256, nullable: true),
|
||||
Note = table.Column<string>(type: "character varying(500)", maxLength: 500, nullable: true),
|
||||
RequestedUtc = table.Column<DateTimeOffset>(type: "timestamp with time zone", nullable: false),
|
||||
UpdatedUtc = table.Column<DateTimeOffset>(type: "timestamp with time zone", nullable: false),
|
||||
SentUtc = table.Column<DateTimeOffset>(type: "timestamp with time zone", nullable: true),
|
||||
LastSeenUtc = table.Column<DateTimeOffset>(type: "timestamp with time zone", nullable: true),
|
||||
FinishedUtc = table.Column<DateTimeOffset>(type: "timestamp with time zone", nullable: true),
|
||||
GroupIdsJson = table.Column<string>(type: "text", nullable: true),
|
||||
ConcurrencyStamp = table.Column<Guid>(type: "uuid", nullable: false)
|
||||
},
|
||||
constraints: table =>
|
||||
{
|
||||
table.PrimaryKey("PK_DeviceRemovals", x => x.Id);
|
||||
});
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_DeviceRemovals_ClientId",
|
||||
table: "DeviceRemovals",
|
||||
column: "ClientId");
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_DeviceRemovals_RequestedUtc",
|
||||
table: "DeviceRemovals",
|
||||
column: "RequestedUtc");
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_DeviceRemovals_State",
|
||||
table: "DeviceRemovals",
|
||||
column: "State");
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void Down(MigrationBuilder migrationBuilder)
|
||||
{
|
||||
migrationBuilder.DropTable(
|
||||
name: "DeviceRemovals");
|
||||
}
|
||||
}
|
||||
}
|
||||
+1246
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,103 @@
|
||||
using System;
|
||||
using Microsoft.EntityFrameworkCore.Migrations;
|
||||
|
||||
#nullable disable
|
||||
|
||||
namespace Backend.Data.Migrations.Postgres
|
||||
{
|
||||
/// <inheritdoc />
|
||||
public partial class JobStagger : Migration
|
||||
{
|
||||
/// <inheritdoc />
|
||||
protected override void Up(MigrationBuilder migrationBuilder)
|
||||
{
|
||||
migrationBuilder.AddColumn<int>(
|
||||
name: "StaggerBatchSize",
|
||||
table: "ScriptTriggers",
|
||||
type: "integer",
|
||||
nullable: false,
|
||||
defaultValue: 1);
|
||||
|
||||
migrationBuilder.AddColumn<int>(
|
||||
name: "StaggerSeconds",
|
||||
table: "ScriptTriggers",
|
||||
type: "integer",
|
||||
nullable: false,
|
||||
defaultValue: 0);
|
||||
|
||||
migrationBuilder.AddColumn<string>(
|
||||
name: "SnapshotProtected",
|
||||
table: "Runs",
|
||||
type: "text",
|
||||
nullable: true);
|
||||
|
||||
migrationBuilder.AddColumn<string>(
|
||||
name: "ErrorId",
|
||||
table: "Jobs",
|
||||
type: "character varying(300)",
|
||||
maxLength: 300,
|
||||
nullable: true);
|
||||
|
||||
migrationBuilder.AddColumn<string>(
|
||||
name: "ErrorMessage",
|
||||
table: "Jobs",
|
||||
type: "character varying(2000)",
|
||||
maxLength: 2000,
|
||||
nullable: true);
|
||||
|
||||
migrationBuilder.AddColumn<string>(
|
||||
name: "ErrorType",
|
||||
table: "Jobs",
|
||||
type: "character varying(300)",
|
||||
maxLength: 300,
|
||||
nullable: true);
|
||||
|
||||
migrationBuilder.AddColumn<DateTimeOffset>(
|
||||
name: "NotBeforeUtc",
|
||||
table: "Jobs",
|
||||
type: "timestamp with time zone",
|
||||
nullable: true);
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_Jobs_State_NotBeforeUtc",
|
||||
table: "Jobs",
|
||||
columns: new[] { "State", "NotBeforeUtc" });
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void Down(MigrationBuilder migrationBuilder)
|
||||
{
|
||||
migrationBuilder.DropIndex(
|
||||
name: "IX_Jobs_State_NotBeforeUtc",
|
||||
table: "Jobs");
|
||||
|
||||
migrationBuilder.DropColumn(
|
||||
name: "StaggerBatchSize",
|
||||
table: "ScriptTriggers");
|
||||
|
||||
migrationBuilder.DropColumn(
|
||||
name: "StaggerSeconds",
|
||||
table: "ScriptTriggers");
|
||||
|
||||
migrationBuilder.DropColumn(
|
||||
name: "SnapshotProtected",
|
||||
table: "Runs");
|
||||
|
||||
migrationBuilder.DropColumn(
|
||||
name: "ErrorId",
|
||||
table: "Jobs");
|
||||
|
||||
migrationBuilder.DropColumn(
|
||||
name: "ErrorMessage",
|
||||
table: "Jobs");
|
||||
|
||||
migrationBuilder.DropColumn(
|
||||
name: "ErrorType",
|
||||
table: "Jobs");
|
||||
|
||||
migrationBuilder.DropColumn(
|
||||
name: "NotBeforeUtc",
|
||||
table: "Jobs");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -417,6 +417,89 @@ namespace Backend.Data.Migrations.Postgres
|
||||
b.ToTable("DeviceInventory", (string)null);
|
||||
});
|
||||
|
||||
modelBuilder.Entity("Backend.Data.DeviceRemovalEntity", b =>
|
||||
{
|
||||
b.Property<Guid>("Id")
|
||||
.ValueGeneratedOnAdd()
|
||||
.HasColumnType("uuid");
|
||||
|
||||
b.Property<string>("AgentVersion")
|
||||
.HasColumnType("text");
|
||||
|
||||
b.Property<int>("Attempts")
|
||||
.HasColumnType("integer");
|
||||
|
||||
b.Property<string>("ClientId")
|
||||
.IsRequired()
|
||||
.HasColumnType("text");
|
||||
|
||||
b.Property<Guid>("ConcurrencyStamp")
|
||||
.IsConcurrencyToken()
|
||||
.HasColumnType("uuid");
|
||||
|
||||
b.Property<string>("Detail")
|
||||
.HasMaxLength(2000)
|
||||
.HasColumnType("character varying(2000)");
|
||||
|
||||
b.Property<Guid>("EnrollmentId")
|
||||
.HasColumnType("uuid");
|
||||
|
||||
b.Property<DateTimeOffset?>("FinishedUtc")
|
||||
.HasColumnType("timestamp with time zone");
|
||||
|
||||
b.Property<string>("GroupIdsJson")
|
||||
.HasColumnType("text");
|
||||
|
||||
b.Property<string>("Hostname")
|
||||
.HasColumnType("text");
|
||||
|
||||
b.Property<DateTimeOffset?>("LastSeenUtc")
|
||||
.HasColumnType("timestamp with time zone");
|
||||
|
||||
b.Property<string>("NkeyPublic")
|
||||
.IsRequired()
|
||||
.HasColumnType("text");
|
||||
|
||||
b.Property<string>("Note")
|
||||
.HasMaxLength(500)
|
||||
.HasColumnType("character varying(500)");
|
||||
|
||||
b.Property<string>("OsCaption")
|
||||
.HasColumnType("text");
|
||||
|
||||
b.Property<string>("Reason")
|
||||
.HasMaxLength(32)
|
||||
.HasColumnType("character varying(32)");
|
||||
|
||||
b.Property<string>("RequestedBy")
|
||||
.HasMaxLength(256)
|
||||
.HasColumnType("character varying(256)");
|
||||
|
||||
b.Property<DateTimeOffset>("RequestedUtc")
|
||||
.HasColumnType("timestamp with time zone");
|
||||
|
||||
b.Property<DateTimeOffset?>("SentUtc")
|
||||
.HasColumnType("timestamp with time zone");
|
||||
|
||||
b.Property<string>("State")
|
||||
.IsRequired()
|
||||
.HasMaxLength(16)
|
||||
.HasColumnType("character varying(16)");
|
||||
|
||||
b.Property<DateTimeOffset>("UpdatedUtc")
|
||||
.HasColumnType("timestamp with time zone");
|
||||
|
||||
b.HasKey("Id");
|
||||
|
||||
b.HasIndex("ClientId");
|
||||
|
||||
b.HasIndex("RequestedUtc");
|
||||
|
||||
b.HasIndex("State");
|
||||
|
||||
b.ToTable("DeviceRemovals", (string)null);
|
||||
});
|
||||
|
||||
modelBuilder.Entity("Backend.Data.FunctionModuleEntity", b =>
|
||||
{
|
||||
b.Property<Guid>("Id")
|
||||
@@ -466,9 +549,24 @@ namespace Backend.Data.Migrations.Postgres
|
||||
b.Property<DateTimeOffset>("CreatedUtc")
|
||||
.HasColumnType("timestamp with time zone");
|
||||
|
||||
b.Property<string>("ErrorId")
|
||||
.HasMaxLength(300)
|
||||
.HasColumnType("character varying(300)");
|
||||
|
||||
b.Property<string>("ErrorMessage")
|
||||
.HasMaxLength(2000)
|
||||
.HasColumnType("character varying(2000)");
|
||||
|
||||
b.Property<string>("ErrorType")
|
||||
.HasMaxLength(300)
|
||||
.HasColumnType("character varying(300)");
|
||||
|
||||
b.Property<int?>("ExitCode")
|
||||
.HasColumnType("integer");
|
||||
|
||||
b.Property<DateTimeOffset?>("NotBeforeUtc")
|
||||
.HasColumnType("timestamp with time zone");
|
||||
|
||||
b.Property<string>("OutputSummary")
|
||||
.HasColumnType("text");
|
||||
|
||||
@@ -502,6 +600,8 @@ namespace Backend.Data.Migrations.Postgres
|
||||
|
||||
b.HasIndex("ScriptId", "RunId");
|
||||
|
||||
b.HasIndex("State", "NotBeforeUtc");
|
||||
|
||||
b.ToTable("Jobs", (string)null);
|
||||
});
|
||||
|
||||
@@ -688,6 +788,9 @@ namespace Backend.Data.Migrations.Postgres
|
||||
.IsRequired()
|
||||
.HasColumnType("text");
|
||||
|
||||
b.Property<string>("SnapshotProtected")
|
||||
.HasColumnType("text");
|
||||
|
||||
b.Property<string>("Source")
|
||||
.IsRequired()
|
||||
.HasMaxLength(32)
|
||||
@@ -819,6 +922,14 @@ namespace Backend.Data.Migrations.Postgres
|
||||
b.Property<Guid>("ScriptId")
|
||||
.HasColumnType("uuid");
|
||||
|
||||
b.Property<int>("StaggerBatchSize")
|
||||
.ValueGeneratedOnAdd()
|
||||
.HasColumnType("integer")
|
||||
.HasDefaultValue(1);
|
||||
|
||||
b.Property<int>("StaggerSeconds")
|
||||
.HasColumnType("integer");
|
||||
|
||||
b.Property<string>("TimeZoneId")
|
||||
.IsRequired()
|
||||
.HasColumnType("text");
|
||||
|
||||
+1301
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,67 @@
|
||||
using System;
|
||||
using Microsoft.EntityFrameworkCore.Migrations;
|
||||
|
||||
#nullable disable
|
||||
|
||||
namespace Backend.Data.Migrations.SqlServer
|
||||
{
|
||||
/// <inheritdoc />
|
||||
public partial class DeviceRemovals : Migration
|
||||
{
|
||||
/// <inheritdoc />
|
||||
protected override void Up(MigrationBuilder migrationBuilder)
|
||||
{
|
||||
migrationBuilder.CreateTable(
|
||||
name: "DeviceRemovals",
|
||||
columns: table => new
|
||||
{
|
||||
Id = table.Column<Guid>(type: "uniqueidentifier", nullable: false),
|
||||
ClientId = table.Column<string>(type: "nvarchar(450)", nullable: false),
|
||||
NkeyPublic = table.Column<string>(type: "nvarchar(max)", nullable: false),
|
||||
EnrollmentId = table.Column<Guid>(type: "uniqueidentifier", nullable: false),
|
||||
Hostname = table.Column<string>(type: "nvarchar(max)", nullable: true),
|
||||
OsCaption = table.Column<string>(type: "nvarchar(max)", nullable: true),
|
||||
AgentVersion = table.Column<string>(type: "nvarchar(max)", nullable: true),
|
||||
State = table.Column<string>(type: "nvarchar(16)", maxLength: 16, nullable: false),
|
||||
Reason = table.Column<string>(type: "nvarchar(32)", maxLength: 32, nullable: true),
|
||||
Detail = table.Column<string>(type: "nvarchar(2000)", maxLength: 2000, nullable: true),
|
||||
Attempts = table.Column<int>(type: "int", nullable: false),
|
||||
RequestedBy = table.Column<string>(type: "nvarchar(256)", maxLength: 256, nullable: true),
|
||||
Note = table.Column<string>(type: "nvarchar(500)", maxLength: 500, nullable: true),
|
||||
RequestedUtc = table.Column<DateTimeOffset>(type: "datetimeoffset", nullable: false),
|
||||
UpdatedUtc = table.Column<DateTimeOffset>(type: "datetimeoffset", nullable: false),
|
||||
SentUtc = table.Column<DateTimeOffset>(type: "datetimeoffset", nullable: true),
|
||||
LastSeenUtc = table.Column<DateTimeOffset>(type: "datetimeoffset", nullable: true),
|
||||
FinishedUtc = table.Column<DateTimeOffset>(type: "datetimeoffset", nullable: true),
|
||||
GroupIdsJson = table.Column<string>(type: "nvarchar(max)", nullable: true),
|
||||
ConcurrencyStamp = table.Column<Guid>(type: "uniqueidentifier", nullable: false)
|
||||
},
|
||||
constraints: table =>
|
||||
{
|
||||
table.PrimaryKey("PK_DeviceRemovals", x => x.Id);
|
||||
});
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_DeviceRemovals_ClientId",
|
||||
table: "DeviceRemovals",
|
||||
column: "ClientId");
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_DeviceRemovals_RequestedUtc",
|
||||
table: "DeviceRemovals",
|
||||
column: "RequestedUtc");
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_DeviceRemovals_State",
|
||||
table: "DeviceRemovals",
|
||||
column: "State");
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void Down(MigrationBuilder migrationBuilder)
|
||||
{
|
||||
migrationBuilder.DropTable(
|
||||
name: "DeviceRemovals");
|
||||
}
|
||||
}
|
||||
}
|
||||
+1246
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,119 @@
|
||||
using System;
|
||||
using Microsoft.EntityFrameworkCore.Migrations;
|
||||
|
||||
#nullable disable
|
||||
|
||||
namespace Backend.Data.Migrations.SqlServer
|
||||
{
|
||||
/// <inheritdoc />
|
||||
public partial class JobStagger : Migration
|
||||
{
|
||||
/// <inheritdoc />
|
||||
protected override void Up(MigrationBuilder migrationBuilder)
|
||||
{
|
||||
migrationBuilder.AddColumn<int>(
|
||||
name: "StaggerBatchSize",
|
||||
table: "ScriptTriggers",
|
||||
type: "int",
|
||||
nullable: false,
|
||||
defaultValue: 1);
|
||||
|
||||
migrationBuilder.AddColumn<int>(
|
||||
name: "StaggerSeconds",
|
||||
table: "ScriptTriggers",
|
||||
type: "int",
|
||||
nullable: false,
|
||||
defaultValue: 0);
|
||||
|
||||
migrationBuilder.AddColumn<string>(
|
||||
name: "SnapshotProtected",
|
||||
table: "Runs",
|
||||
type: "nvarchar(max)",
|
||||
nullable: true);
|
||||
|
||||
migrationBuilder.AlterColumn<string>(
|
||||
name: "State",
|
||||
table: "Jobs",
|
||||
type: "nvarchar(450)",
|
||||
nullable: false,
|
||||
oldClrType: typeof(string),
|
||||
oldType: "nvarchar(max)");
|
||||
|
||||
migrationBuilder.AddColumn<string>(
|
||||
name: "ErrorId",
|
||||
table: "Jobs",
|
||||
type: "nvarchar(300)",
|
||||
maxLength: 300,
|
||||
nullable: true);
|
||||
|
||||
migrationBuilder.AddColumn<string>(
|
||||
name: "ErrorMessage",
|
||||
table: "Jobs",
|
||||
type: "nvarchar(2000)",
|
||||
maxLength: 2000,
|
||||
nullable: true);
|
||||
|
||||
migrationBuilder.AddColumn<string>(
|
||||
name: "ErrorType",
|
||||
table: "Jobs",
|
||||
type: "nvarchar(300)",
|
||||
maxLength: 300,
|
||||
nullable: true);
|
||||
|
||||
migrationBuilder.AddColumn<DateTimeOffset>(
|
||||
name: "NotBeforeUtc",
|
||||
table: "Jobs",
|
||||
type: "datetimeoffset",
|
||||
nullable: true);
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_Jobs_State_NotBeforeUtc",
|
||||
table: "Jobs",
|
||||
columns: new[] { "State", "NotBeforeUtc" });
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void Down(MigrationBuilder migrationBuilder)
|
||||
{
|
||||
migrationBuilder.DropIndex(
|
||||
name: "IX_Jobs_State_NotBeforeUtc",
|
||||
table: "Jobs");
|
||||
|
||||
migrationBuilder.DropColumn(
|
||||
name: "StaggerBatchSize",
|
||||
table: "ScriptTriggers");
|
||||
|
||||
migrationBuilder.DropColumn(
|
||||
name: "StaggerSeconds",
|
||||
table: "ScriptTriggers");
|
||||
|
||||
migrationBuilder.DropColumn(
|
||||
name: "SnapshotProtected",
|
||||
table: "Runs");
|
||||
|
||||
migrationBuilder.DropColumn(
|
||||
name: "ErrorId",
|
||||
table: "Jobs");
|
||||
|
||||
migrationBuilder.DropColumn(
|
||||
name: "ErrorMessage",
|
||||
table: "Jobs");
|
||||
|
||||
migrationBuilder.DropColumn(
|
||||
name: "ErrorType",
|
||||
table: "Jobs");
|
||||
|
||||
migrationBuilder.DropColumn(
|
||||
name: "NotBeforeUtc",
|
||||
table: "Jobs");
|
||||
|
||||
migrationBuilder.AlterColumn<string>(
|
||||
name: "State",
|
||||
table: "Jobs",
|
||||
type: "nvarchar(max)",
|
||||
nullable: false,
|
||||
oldClrType: typeof(string),
|
||||
oldType: "nvarchar(450)");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -417,6 +417,89 @@ namespace Backend.Data.Migrations.SqlServer
|
||||
b.ToTable("DeviceInventory", (string)null);
|
||||
});
|
||||
|
||||
modelBuilder.Entity("Backend.Data.DeviceRemovalEntity", b =>
|
||||
{
|
||||
b.Property<Guid>("Id")
|
||||
.ValueGeneratedOnAdd()
|
||||
.HasColumnType("uniqueidentifier");
|
||||
|
||||
b.Property<string>("AgentVersion")
|
||||
.HasColumnType("nvarchar(max)");
|
||||
|
||||
b.Property<int>("Attempts")
|
||||
.HasColumnType("int");
|
||||
|
||||
b.Property<string>("ClientId")
|
||||
.IsRequired()
|
||||
.HasColumnType("nvarchar(450)");
|
||||
|
||||
b.Property<Guid>("ConcurrencyStamp")
|
||||
.IsConcurrencyToken()
|
||||
.HasColumnType("uniqueidentifier");
|
||||
|
||||
b.Property<string>("Detail")
|
||||
.HasMaxLength(2000)
|
||||
.HasColumnType("nvarchar(2000)");
|
||||
|
||||
b.Property<Guid>("EnrollmentId")
|
||||
.HasColumnType("uniqueidentifier");
|
||||
|
||||
b.Property<DateTimeOffset?>("FinishedUtc")
|
||||
.HasColumnType("datetimeoffset");
|
||||
|
||||
b.Property<string>("GroupIdsJson")
|
||||
.HasColumnType("nvarchar(max)");
|
||||
|
||||
b.Property<string>("Hostname")
|
||||
.HasColumnType("nvarchar(max)");
|
||||
|
||||
b.Property<DateTimeOffset?>("LastSeenUtc")
|
||||
.HasColumnType("datetimeoffset");
|
||||
|
||||
b.Property<string>("NkeyPublic")
|
||||
.IsRequired()
|
||||
.HasColumnType("nvarchar(max)");
|
||||
|
||||
b.Property<string>("Note")
|
||||
.HasMaxLength(500)
|
||||
.HasColumnType("nvarchar(500)");
|
||||
|
||||
b.Property<string>("OsCaption")
|
||||
.HasColumnType("nvarchar(max)");
|
||||
|
||||
b.Property<string>("Reason")
|
||||
.HasMaxLength(32)
|
||||
.HasColumnType("nvarchar(32)");
|
||||
|
||||
b.Property<string>("RequestedBy")
|
||||
.HasMaxLength(256)
|
||||
.HasColumnType("nvarchar(256)");
|
||||
|
||||
b.Property<DateTimeOffset>("RequestedUtc")
|
||||
.HasColumnType("datetimeoffset");
|
||||
|
||||
b.Property<DateTimeOffset?>("SentUtc")
|
||||
.HasColumnType("datetimeoffset");
|
||||
|
||||
b.Property<string>("State")
|
||||
.IsRequired()
|
||||
.HasMaxLength(16)
|
||||
.HasColumnType("nvarchar(16)");
|
||||
|
||||
b.Property<DateTimeOffset>("UpdatedUtc")
|
||||
.HasColumnType("datetimeoffset");
|
||||
|
||||
b.HasKey("Id");
|
||||
|
||||
b.HasIndex("ClientId");
|
||||
|
||||
b.HasIndex("RequestedUtc");
|
||||
|
||||
b.HasIndex("State");
|
||||
|
||||
b.ToTable("DeviceRemovals", (string)null);
|
||||
});
|
||||
|
||||
modelBuilder.Entity("Backend.Data.FunctionModuleEntity", b =>
|
||||
{
|
||||
b.Property<Guid>("Id")
|
||||
@@ -466,9 +549,24 @@ namespace Backend.Data.Migrations.SqlServer
|
||||
b.Property<DateTimeOffset>("CreatedUtc")
|
||||
.HasColumnType("datetimeoffset");
|
||||
|
||||
b.Property<string>("ErrorId")
|
||||
.HasMaxLength(300)
|
||||
.HasColumnType("nvarchar(300)");
|
||||
|
||||
b.Property<string>("ErrorMessage")
|
||||
.HasMaxLength(2000)
|
||||
.HasColumnType("nvarchar(2000)");
|
||||
|
||||
b.Property<string>("ErrorType")
|
||||
.HasMaxLength(300)
|
||||
.HasColumnType("nvarchar(300)");
|
||||
|
||||
b.Property<int?>("ExitCode")
|
||||
.HasColumnType("int");
|
||||
|
||||
b.Property<DateTimeOffset?>("NotBeforeUtc")
|
||||
.HasColumnType("datetimeoffset");
|
||||
|
||||
b.Property<string>("OutputSummary")
|
||||
.HasColumnType("nvarchar(max)");
|
||||
|
||||
@@ -490,7 +588,7 @@ namespace Backend.Data.Migrations.SqlServer
|
||||
|
||||
b.Property<string>("State")
|
||||
.IsRequired()
|
||||
.HasColumnType("nvarchar(max)");
|
||||
.HasColumnType("nvarchar(450)");
|
||||
|
||||
b.HasKey("JobId");
|
||||
|
||||
@@ -502,6 +600,8 @@ namespace Backend.Data.Migrations.SqlServer
|
||||
|
||||
b.HasIndex("ScriptId", "RunId");
|
||||
|
||||
b.HasIndex("State", "NotBeforeUtc");
|
||||
|
||||
b.ToTable("Jobs", (string)null);
|
||||
});
|
||||
|
||||
@@ -688,6 +788,9 @@ namespace Backend.Data.Migrations.SqlServer
|
||||
.IsRequired()
|
||||
.HasColumnType("nvarchar(max)");
|
||||
|
||||
b.Property<string>("SnapshotProtected")
|
||||
.HasColumnType("nvarchar(max)");
|
||||
|
||||
b.Property<string>("Source")
|
||||
.IsRequired()
|
||||
.HasMaxLength(32)
|
||||
@@ -819,6 +922,14 @@ namespace Backend.Data.Migrations.SqlServer
|
||||
b.Property<Guid>("ScriptId")
|
||||
.HasColumnType("uniqueidentifier");
|
||||
|
||||
b.Property<int>("StaggerBatchSize")
|
||||
.ValueGeneratedOnAdd()
|
||||
.HasColumnType("int")
|
||||
.HasDefaultValue(1);
|
||||
|
||||
b.Property<int>("StaggerSeconds")
|
||||
.HasColumnType("int");
|
||||
|
||||
b.Property<string>("TimeZoneId")
|
||||
.IsRequired()
|
||||
.HasColumnType("nvarchar(max)");
|
||||
|
||||
+1288
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,67 @@
|
||||
using System;
|
||||
using Microsoft.EntityFrameworkCore.Migrations;
|
||||
|
||||
#nullable disable
|
||||
|
||||
namespace Backend.Data.Migrations.Sqlite
|
||||
{
|
||||
/// <inheritdoc />
|
||||
public partial class DeviceRemovals : Migration
|
||||
{
|
||||
/// <inheritdoc />
|
||||
protected override void Up(MigrationBuilder migrationBuilder)
|
||||
{
|
||||
migrationBuilder.CreateTable(
|
||||
name: "DeviceRemovals",
|
||||
columns: table => new
|
||||
{
|
||||
Id = table.Column<Guid>(type: "TEXT", nullable: false),
|
||||
ClientId = table.Column<string>(type: "TEXT", nullable: false),
|
||||
NkeyPublic = table.Column<string>(type: "TEXT", nullable: false),
|
||||
EnrollmentId = table.Column<Guid>(type: "TEXT", nullable: false),
|
||||
Hostname = table.Column<string>(type: "TEXT", nullable: true),
|
||||
OsCaption = table.Column<string>(type: "TEXT", nullable: true),
|
||||
AgentVersion = table.Column<string>(type: "TEXT", nullable: true),
|
||||
State = table.Column<string>(type: "TEXT", maxLength: 16, nullable: false),
|
||||
Reason = table.Column<string>(type: "TEXT", maxLength: 32, nullable: true),
|
||||
Detail = table.Column<string>(type: "TEXT", maxLength: 2000, nullable: true),
|
||||
Attempts = table.Column<int>(type: "INTEGER", nullable: false),
|
||||
RequestedBy = table.Column<string>(type: "TEXT", maxLength: 256, nullable: true),
|
||||
Note = table.Column<string>(type: "TEXT", maxLength: 500, nullable: true),
|
||||
RequestedUtc = table.Column<long>(type: "INTEGER", nullable: false),
|
||||
UpdatedUtc = table.Column<long>(type: "INTEGER", nullable: false),
|
||||
SentUtc = table.Column<long>(type: "INTEGER", nullable: true),
|
||||
LastSeenUtc = table.Column<long>(type: "INTEGER", nullable: true),
|
||||
FinishedUtc = table.Column<long>(type: "INTEGER", nullable: true),
|
||||
GroupIdsJson = table.Column<string>(type: "TEXT", nullable: true),
|
||||
ConcurrencyStamp = table.Column<Guid>(type: "TEXT", nullable: false)
|
||||
},
|
||||
constraints: table =>
|
||||
{
|
||||
table.PrimaryKey("PK_DeviceRemovals", x => x.Id);
|
||||
});
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_DeviceRemovals_ClientId",
|
||||
table: "DeviceRemovals",
|
||||
column: "ClientId");
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_DeviceRemovals_RequestedUtc",
|
||||
table: "DeviceRemovals",
|
||||
column: "RequestedUtc");
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_DeviceRemovals_State",
|
||||
table: "DeviceRemovals",
|
||||
column: "State");
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void Down(MigrationBuilder migrationBuilder)
|
||||
{
|
||||
migrationBuilder.DropTable(
|
||||
name: "DeviceRemovals");
|
||||
}
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,102 @@
|
||||
using Microsoft.EntityFrameworkCore.Migrations;
|
||||
|
||||
#nullable disable
|
||||
|
||||
namespace Backend.Data.Migrations.Sqlite
|
||||
{
|
||||
/// <inheritdoc />
|
||||
public partial class JobStagger : Migration
|
||||
{
|
||||
/// <inheritdoc />
|
||||
protected override void Up(MigrationBuilder migrationBuilder)
|
||||
{
|
||||
migrationBuilder.AddColumn<int>(
|
||||
name: "StaggerBatchSize",
|
||||
table: "ScriptTriggers",
|
||||
type: "INTEGER",
|
||||
nullable: false,
|
||||
defaultValue: 1);
|
||||
|
||||
migrationBuilder.AddColumn<int>(
|
||||
name: "StaggerSeconds",
|
||||
table: "ScriptTriggers",
|
||||
type: "INTEGER",
|
||||
nullable: false,
|
||||
defaultValue: 0);
|
||||
|
||||
migrationBuilder.AddColumn<string>(
|
||||
name: "SnapshotProtected",
|
||||
table: "Runs",
|
||||
type: "TEXT",
|
||||
nullable: true);
|
||||
|
||||
migrationBuilder.AddColumn<string>(
|
||||
name: "ErrorId",
|
||||
table: "Jobs",
|
||||
type: "TEXT",
|
||||
maxLength: 300,
|
||||
nullable: true);
|
||||
|
||||
migrationBuilder.AddColumn<string>(
|
||||
name: "ErrorMessage",
|
||||
table: "Jobs",
|
||||
type: "TEXT",
|
||||
maxLength: 2000,
|
||||
nullable: true);
|
||||
|
||||
migrationBuilder.AddColumn<string>(
|
||||
name: "ErrorType",
|
||||
table: "Jobs",
|
||||
type: "TEXT",
|
||||
maxLength: 300,
|
||||
nullable: true);
|
||||
|
||||
migrationBuilder.AddColumn<long>(
|
||||
name: "NotBeforeUtc",
|
||||
table: "Jobs",
|
||||
type: "INTEGER",
|
||||
nullable: true);
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_Jobs_State_NotBeforeUtc",
|
||||
table: "Jobs",
|
||||
columns: new[] { "State", "NotBeforeUtc" });
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void Down(MigrationBuilder migrationBuilder)
|
||||
{
|
||||
migrationBuilder.DropIndex(
|
||||
name: "IX_Jobs_State_NotBeforeUtc",
|
||||
table: "Jobs");
|
||||
|
||||
migrationBuilder.DropColumn(
|
||||
name: "StaggerBatchSize",
|
||||
table: "ScriptTriggers");
|
||||
|
||||
migrationBuilder.DropColumn(
|
||||
name: "StaggerSeconds",
|
||||
table: "ScriptTriggers");
|
||||
|
||||
migrationBuilder.DropColumn(
|
||||
name: "SnapshotProtected",
|
||||
table: "Runs");
|
||||
|
||||
migrationBuilder.DropColumn(
|
||||
name: "ErrorId",
|
||||
table: "Jobs");
|
||||
|
||||
migrationBuilder.DropColumn(
|
||||
name: "ErrorMessage",
|
||||
table: "Jobs");
|
||||
|
||||
migrationBuilder.DropColumn(
|
||||
name: "ErrorType",
|
||||
table: "Jobs");
|
||||
|
||||
migrationBuilder.DropColumn(
|
||||
name: "NotBeforeUtc",
|
||||
table: "Jobs");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -408,6 +408,89 @@ namespace Backend.Data.Migrations.Sqlite
|
||||
b.ToTable("DeviceInventory", (string)null);
|
||||
});
|
||||
|
||||
modelBuilder.Entity("Backend.Data.DeviceRemovalEntity", b =>
|
||||
{
|
||||
b.Property<Guid>("Id")
|
||||
.ValueGeneratedOnAdd()
|
||||
.HasColumnType("TEXT");
|
||||
|
||||
b.Property<string>("AgentVersion")
|
||||
.HasColumnType("TEXT");
|
||||
|
||||
b.Property<int>("Attempts")
|
||||
.HasColumnType("INTEGER");
|
||||
|
||||
b.Property<string>("ClientId")
|
||||
.IsRequired()
|
||||
.HasColumnType("TEXT");
|
||||
|
||||
b.Property<Guid>("ConcurrencyStamp")
|
||||
.IsConcurrencyToken()
|
||||
.HasColumnType("TEXT");
|
||||
|
||||
b.Property<string>("Detail")
|
||||
.HasMaxLength(2000)
|
||||
.HasColumnType("TEXT");
|
||||
|
||||
b.Property<Guid>("EnrollmentId")
|
||||
.HasColumnType("TEXT");
|
||||
|
||||
b.Property<long?>("FinishedUtc")
|
||||
.HasColumnType("INTEGER");
|
||||
|
||||
b.Property<string>("GroupIdsJson")
|
||||
.HasColumnType("TEXT");
|
||||
|
||||
b.Property<string>("Hostname")
|
||||
.HasColumnType("TEXT");
|
||||
|
||||
b.Property<long?>("LastSeenUtc")
|
||||
.HasColumnType("INTEGER");
|
||||
|
||||
b.Property<string>("NkeyPublic")
|
||||
.IsRequired()
|
||||
.HasColumnType("TEXT");
|
||||
|
||||
b.Property<string>("Note")
|
||||
.HasMaxLength(500)
|
||||
.HasColumnType("TEXT");
|
||||
|
||||
b.Property<string>("OsCaption")
|
||||
.HasColumnType("TEXT");
|
||||
|
||||
b.Property<string>("Reason")
|
||||
.HasMaxLength(32)
|
||||
.HasColumnType("TEXT");
|
||||
|
||||
b.Property<string>("RequestedBy")
|
||||
.HasMaxLength(256)
|
||||
.HasColumnType("TEXT");
|
||||
|
||||
b.Property<long>("RequestedUtc")
|
||||
.HasColumnType("INTEGER");
|
||||
|
||||
b.Property<long?>("SentUtc")
|
||||
.HasColumnType("INTEGER");
|
||||
|
||||
b.Property<string>("State")
|
||||
.IsRequired()
|
||||
.HasMaxLength(16)
|
||||
.HasColumnType("TEXT");
|
||||
|
||||
b.Property<long>("UpdatedUtc")
|
||||
.HasColumnType("INTEGER");
|
||||
|
||||
b.HasKey("Id");
|
||||
|
||||
b.HasIndex("ClientId");
|
||||
|
||||
b.HasIndex("RequestedUtc");
|
||||
|
||||
b.HasIndex("State");
|
||||
|
||||
b.ToTable("DeviceRemovals", (string)null);
|
||||
});
|
||||
|
||||
modelBuilder.Entity("Backend.Data.FunctionModuleEntity", b =>
|
||||
{
|
||||
b.Property<Guid>("Id")
|
||||
@@ -457,9 +540,24 @@ namespace Backend.Data.Migrations.Sqlite
|
||||
b.Property<long>("CreatedUtc")
|
||||
.HasColumnType("INTEGER");
|
||||
|
||||
b.Property<string>("ErrorId")
|
||||
.HasMaxLength(300)
|
||||
.HasColumnType("TEXT");
|
||||
|
||||
b.Property<string>("ErrorMessage")
|
||||
.HasMaxLength(2000)
|
||||
.HasColumnType("TEXT");
|
||||
|
||||
b.Property<string>("ErrorType")
|
||||
.HasMaxLength(300)
|
||||
.HasColumnType("TEXT");
|
||||
|
||||
b.Property<int?>("ExitCode")
|
||||
.HasColumnType("INTEGER");
|
||||
|
||||
b.Property<long?>("NotBeforeUtc")
|
||||
.HasColumnType("INTEGER");
|
||||
|
||||
b.Property<string>("OutputSummary")
|
||||
.HasColumnType("TEXT");
|
||||
|
||||
@@ -493,6 +591,8 @@ namespace Backend.Data.Migrations.Sqlite
|
||||
|
||||
b.HasIndex("ScriptId", "RunId");
|
||||
|
||||
b.HasIndex("State", "NotBeforeUtc");
|
||||
|
||||
b.ToTable("Jobs", (string)null);
|
||||
});
|
||||
|
||||
@@ -677,6 +777,9 @@ namespace Backend.Data.Migrations.Sqlite
|
||||
.IsRequired()
|
||||
.HasColumnType("TEXT");
|
||||
|
||||
b.Property<string>("SnapshotProtected")
|
||||
.HasColumnType("TEXT");
|
||||
|
||||
b.Property<string>("Source")
|
||||
.IsRequired()
|
||||
.HasMaxLength(32)
|
||||
@@ -808,6 +911,14 @@ namespace Backend.Data.Migrations.Sqlite
|
||||
b.Property<Guid>("ScriptId")
|
||||
.HasColumnType("TEXT");
|
||||
|
||||
b.Property<int>("StaggerBatchSize")
|
||||
.ValueGeneratedOnAdd()
|
||||
.HasColumnType("INTEGER")
|
||||
.HasDefaultValue(1);
|
||||
|
||||
b.Property<int>("StaggerSeconds")
|
||||
.HasColumnType("INTEGER");
|
||||
|
||||
b.Property<string>("TimeZoneId")
|
||||
.IsRequired()
|
||||
.HasColumnType("TEXT");
|
||||
|
||||
@@ -15,7 +15,9 @@ namespace Backend.Enrollment;
|
||||
/// Agent bootstrap and credential refresh. Bootstrap validates and consumes a
|
||||
/// one-time token, binds the agent's public NKey to the device and returns a
|
||||
/// scoped NATS user JWT. Refresh proves possession of the enrolled NKey by
|
||||
/// verifying a signature over <c>{clientId}|{timestampUtc:O}</c>.
|
||||
/// verifying a signature over <c>{clientId}|{timestampUtc:O}</c>. Both answer
|
||||
/// to a device's removal: tokens from before it no longer enroll the device, and
|
||||
/// a renewal is told the device has gone.
|
||||
/// </summary>
|
||||
public static class EnrollmentEndpoints
|
||||
{
|
||||
@@ -39,6 +41,7 @@ public static class EnrollmentEndpoints
|
||||
JetStreamProvisioner provisioner,
|
||||
Messaging.BrowserEventBroadcaster broadcaster,
|
||||
Groups.ScriptTriggerService triggers,
|
||||
Removals.IDeviceRemovalSender removalSender,
|
||||
AuditService audit,
|
||||
HttpContext context,
|
||||
ILoggerFactory loggerFactory,
|
||||
@@ -88,6 +91,45 @@ public static class EnrollmentEndpoints
|
||||
return Results.Unauthorized();
|
||||
}
|
||||
|
||||
// A removed device comes back only on a token made after its removal, so
|
||||
// the install it already has, or a GPO still pushing an old one, cannot.
|
||||
var removal = await db.DeviceRemovals
|
||||
.Where(r => r.ClientId == request.ClientId)
|
||||
.OrderByDescending(r => r.RequestedUtc)
|
||||
.FirstOrDefaultAsync(cancellationToken);
|
||||
if (removal is not null && token.CreatedUtc <= removal.RequestedUtc)
|
||||
{
|
||||
await audit.LogAsync(AuditCategories.Enrollment, "bootstrap-rejected",
|
||||
clientId: request.ClientId, detail: $"token predates the device's removal {removal.Id:D}", remoteIp: remoteIp,
|
||||
cancellationToken: cancellationToken);
|
||||
logger.LogWarning("Bootstrap rejected for removed client {ClientId} from {RemoteIp}", request.ClientId, remoteIp);
|
||||
return Results.Unauthorized();
|
||||
}
|
||||
|
||||
// Enrolling again: nothing queued for the old enrollment may replay, and
|
||||
// the agent retries when the messaging system could not make sure of that.
|
||||
if (removal is not null && device is null)
|
||||
{
|
||||
try
|
||||
{
|
||||
await removalSender.ClearCommandsAsync(request.ClientId, cancellationToken);
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
logger.LogWarning(ex, "Could not clear old commands of re-enrolling client {ClientId}", request.ClientId);
|
||||
return Results.Problem(statusCode: StatusCodes.Status503ServiceUnavailable, title: "Messaging unavailable.");
|
||||
}
|
||||
|
||||
if (Shared.Contracts.Models.DeviceRemovalStates.Open.Contains(removal.State))
|
||||
{
|
||||
removal.State = Shared.Contracts.Models.DeviceRemovalStates.Removed;
|
||||
removal.Detail = "The device enrolled again.";
|
||||
removal.FinishedUtc = now;
|
||||
removal.UpdatedUtc = now;
|
||||
Api.DeviceHistory.RecordRemoved(db, request.ClientId, "The device enrolled again on a new token before its agent uninstalled.");
|
||||
}
|
||||
}
|
||||
|
||||
// Idempotent enrollment by client id: re-enrollment rebinds the NKey.
|
||||
var enrollmentId = Guid.NewGuid();
|
||||
|
||||
@@ -146,7 +188,12 @@ public static class EnrollmentEndpoints
|
||||
|
||||
device.BootstrapTokenId = token.Id;
|
||||
|
||||
var userJwt = issuer.IssueAgentCredential(request.ClientId, request.NkeyPublic);
|
||||
// An agent too old to renew keeps a credential that never expires: one
|
||||
// that expired would strand it, and it cannot take a remove command either.
|
||||
DateTimeOffset? credentialExpires = RenewsItsCredential(request.AgentVersion)
|
||||
? now + Shared.Contracts.Models.AgentCredentialDefaults.Lifetime
|
||||
: null;
|
||||
var userJwt = issuer.IssueAgentCredential(request.ClientId, request.NkeyPublic, credentialExpires);
|
||||
|
||||
await db.SaveChangesAsync(cancellationToken);
|
||||
await audit.LogAsync(AuditCategories.Enrollment, "bootstrap-succeeded",
|
||||
@@ -179,9 +226,12 @@ public static class EnrollmentEndpoints
|
||||
await triggers.FireAsync(
|
||||
Shared.Contracts.Models.ScriptTriggerType.OnRegistration, request.ClientId, cancellationToken);
|
||||
|
||||
return Results.Ok(new BootstrapResponse(enrollmentId, request.ClientId, userJwt, CredentialExpiresUtc: null));
|
||||
return Results.Ok(new BootstrapResponse(enrollmentId, request.ClientId, userJwt, credentialExpires));
|
||||
}
|
||||
|
||||
private static bool RenewsItsCredential(string? agentVersion)
|
||||
=> Shared.Contracts.Models.AgentVersions.Compare(agentVersion, Shared.Contracts.Models.DeviceRemovalDefaults.MinimumAgentVersion) >= 0;
|
||||
|
||||
private static bool IsValidUserPublicKey(string candidate)
|
||||
{
|
||||
try
|
||||
@@ -199,6 +249,7 @@ public static class EnrollmentEndpoints
|
||||
CredentialRefreshRequest request,
|
||||
AppDbContext db,
|
||||
NatsCredentialIssuer issuer,
|
||||
Removals.DeviceRemovalService removals,
|
||||
AuditService audit,
|
||||
HttpContext context,
|
||||
CancellationToken cancellationToken)
|
||||
@@ -209,8 +260,12 @@ public static class EnrollmentEndpoints
|
||||
}
|
||||
|
||||
var device = await db.Devices.SingleOrDefaultAsync(d => d.ClientId == request.ClientId, cancellationToken);
|
||||
if (device is null
|
||||
|| !string.Equals(device.NkeyPublic, request.NkeyPublic, StringComparison.Ordinal)
|
||||
if (device is null)
|
||||
{
|
||||
return await RefuseRemovedAsync(request, db, removals, audit, context, cancellationToken);
|
||||
}
|
||||
|
||||
if (!string.Equals(device.NkeyPublic, request.NkeyPublic, StringComparison.Ordinal)
|
||||
|| !issuer.IsConfigured)
|
||||
{
|
||||
return Results.Unauthorized();
|
||||
@@ -226,32 +281,11 @@ public static class EnrollmentEndpoints
|
||||
return Results.Unauthorized();
|
||||
}
|
||||
|
||||
var skew = DateTimeOffset.UtcNow - request.TimestampUtc;
|
||||
if (skew.Duration() > RefreshTimestampTolerance)
|
||||
if (!await ProvesKeyAsync(request, device.NkeyPublic, audit, context, cancellationToken))
|
||||
{
|
||||
return Results.Unauthorized();
|
||||
}
|
||||
|
||||
byte[] signature;
|
||||
try
|
||||
{
|
||||
signature = Convert.FromBase64String(request.SignatureBase64);
|
||||
}
|
||||
catch (FormatException)
|
||||
{
|
||||
return Results.Unauthorized();
|
||||
}
|
||||
|
||||
var payload = Encoding.UTF8.GetBytes($"{request.ClientId}|{request.TimestampUtc:O}");
|
||||
using var verifier = KeyPair.FromPublicKey(device.NkeyPublic);
|
||||
if (!verifier.Verify(payload, signature))
|
||||
{
|
||||
await audit.LogAsync(AuditCategories.Enrollment, "credential-refresh-rejected",
|
||||
clientId: request.ClientId, detail: "signature verification failed",
|
||||
remoteIp: ClientIp.EffectiveText(context), cancellationToken: cancellationToken);
|
||||
return Results.Unauthorized();
|
||||
}
|
||||
|
||||
// A verified refresh is the freshest sighting of the agent's public IP.
|
||||
var publicIp = ClientIp.EffectiveText(context);
|
||||
if (!string.Equals(device.PublicIp, publicIp, StringComparison.Ordinal))
|
||||
@@ -260,7 +294,102 @@ public static class EnrollmentEndpoints
|
||||
await db.SaveChangesAsync(cancellationToken);
|
||||
}
|
||||
|
||||
var userJwt = issuer.IssueAgentCredential(request.ClientId, request.NkeyPublic);
|
||||
return Results.Ok(new CredentialRefreshResponse(userJwt, CredentialExpiresUtc: null));
|
||||
// Only agents that renew call this, so the new credential always expires.
|
||||
var expires = DateTimeOffset.UtcNow + Shared.Contracts.Models.AgentCredentialDefaults.Lifetime;
|
||||
var userJwt = issuer.IssueAgentCredential(request.ClientId, request.NkeyPublic, expires);
|
||||
return Results.Ok(new CredentialRefreshResponse(userJwt, expires));
|
||||
}
|
||||
|
||||
/// <summary>True when the request is fresh and signed with the enrolled key.</summary>
|
||||
private static async Task<bool> ProvesKeyAsync(
|
||||
CredentialRefreshRequest request, string nkeyPublic, AuditService audit, HttpContext context,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var skew = DateTimeOffset.UtcNow - request.TimestampUtc;
|
||||
if (skew.Duration() > RefreshTimestampTolerance)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
byte[] signature;
|
||||
try
|
||||
{
|
||||
signature = Convert.FromBase64String(request.SignatureBase64);
|
||||
}
|
||||
catch (FormatException)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
var payload = Encoding.UTF8.GetBytes($"{request.ClientId}|{request.TimestampUtc:O}");
|
||||
using var verifier = KeyPair.FromPublicKey(nkeyPublic);
|
||||
if (verifier.Verify(payload, signature))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
await audit.LogAsync(AuditCategories.Enrollment, "credential-refresh-rejected",
|
||||
clientId: request.ClientId, detail: "signature verification failed",
|
||||
remoteIp: ClientIp.EffectiveText(context), cancellationToken: cancellationToken);
|
||||
return false;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A renewal for a client id with no device row. Only the agent holding the
|
||||
/// enrolled key learns anything: everyone else gets the plain 401 an unknown
|
||||
/// id gets. An open removal answers 410 and the agent uninstalls itself, which
|
||||
/// also reaches a device that came back too briefly for a command. So does a
|
||||
/// closed one, as an agent that returns from the dead should still go. A
|
||||
/// stopped removal gets the plain 401: Stop Trying means its agent stays
|
||||
/// installed and stays blocked, and 410 would uninstall it after all.
|
||||
/// </summary>
|
||||
private static async Task<IResult> RefuseRemovedAsync(
|
||||
CredentialRefreshRequest request, AppDbContext db, Removals.DeviceRemovalService removals, AuditService audit,
|
||||
HttpContext context, CancellationToken cancellationToken)
|
||||
{
|
||||
var removal = await removals.LatestAsync(request.ClientId, cancellationToken);
|
||||
if (removal is null
|
||||
|| !string.Equals(removal.NkeyPublic, request.NkeyPublic, StringComparison.Ordinal)
|
||||
|| !await ProvesKeyAsync(request, removal.NkeyPublic, audit, context, cancellationToken))
|
||||
{
|
||||
return Results.Unauthorized();
|
||||
}
|
||||
|
||||
var remoteIp = ClientIp.EffectiveText(context);
|
||||
if (removal.State == Shared.Contracts.Models.DeviceRemovalStates.Stopped)
|
||||
{
|
||||
await audit.LogAsync(AuditCategories.Enrollment, "credential-refresh-refused",
|
||||
clientId: request.ClientId, detail: "removal stopped: the device stays blocked", remoteIp: remoteIp,
|
||||
cancellationToken: cancellationToken);
|
||||
return Results.Unauthorized();
|
||||
}
|
||||
|
||||
if (Shared.Contracts.Models.DeviceRemovalStates.Open.Contains(removal.State))
|
||||
{
|
||||
var now = DateTimeOffset.UtcNow;
|
||||
removal.State = Shared.Contracts.Models.DeviceRemovalStates.Uninstalling;
|
||||
removal.Reason = null;
|
||||
removal.UpdatedUtc = now;
|
||||
removal.LastSeenUtc = now;
|
||||
try
|
||||
{
|
||||
await db.SaveChangesAsync(cancellationToken);
|
||||
await removals.AnnounceAsync(removal, cancellationToken);
|
||||
}
|
||||
catch (DbUpdateConcurrencyException)
|
||||
{
|
||||
// Changed on another node meanwhile; the answer is the same, and the event or next renewal says so again.
|
||||
db.ChangeTracker.Clear();
|
||||
}
|
||||
}
|
||||
|
||||
await audit.LogAsync(AuditCategories.Enrollment, "credential-refresh-refused",
|
||||
clientId: request.ClientId, detail: $"device removed ({removal.Id:D})", remoteIp: remoteIp,
|
||||
cancellationToken: cancellationToken);
|
||||
return Results.Problem(
|
||||
statusCode: StatusCodes.Status410Gone,
|
||||
type: Shared.Contracts.Models.AgentCredentialDefaults.RemovedProblemType,
|
||||
title: "This device was removed.",
|
||||
extensions: new Dictionary<string, object?> { ["removalId"] = removal.Id });
|
||||
}
|
||||
}
|
||||
|
||||
@@ -103,10 +103,15 @@ public sealed class ScriptTriggerService(
|
||||
continue;
|
||||
}
|
||||
|
||||
await runner.DispatchAsync(
|
||||
prepared, job, TriggerExecutor.ParseArguments(trigger.ArgumentsJson),
|
||||
var arguments = TriggerExecutor.ParseArguments(trigger.ArgumentsJson);
|
||||
var sent = await runner.DispatchAsync(
|
||||
prepared, job, arguments,
|
||||
$"Trigger ({triggerType}) queued “{script.Name}” for {device.Hostname}",
|
||||
cancellationToken);
|
||||
if (!sent)
|
||||
{
|
||||
await runner.LeaveForDispatcherAsync(runId, prepared, arguments, [job], cancellationToken);
|
||||
}
|
||||
|
||||
logger.LogInformation(
|
||||
"Trigger {TriggerId} ({TriggerType}) fired for {ClientId} as job {JobId}",
|
||||
|
||||
@@ -48,7 +48,9 @@ public static class TriggerEndpoints
|
||||
Name: t.Name,
|
||||
LastRunId: t.LastRunId,
|
||||
Arguments: TriggerExecutor.ParseArguments(t.ArgumentsJson),
|
||||
CreatedBy: t.CreatedBy)).ToList());
|
||||
CreatedBy: t.CreatedBy,
|
||||
StaggerSeconds: t.StaggerSeconds,
|
||||
StaggerBatchSize: t.StaggerBatchSize)).ToList());
|
||||
}
|
||||
|
||||
private static async Task<IResult> CreateAsync(
|
||||
@@ -348,6 +350,11 @@ public static class TriggerEndpoints
|
||||
return "Name must be 200 characters or fewer.";
|
||||
}
|
||||
|
||||
if (JobStagger.Validate(request.StaggerSeconds, request.StaggerBatchSize) is { } badStagger)
|
||||
{
|
||||
return badStagger;
|
||||
}
|
||||
|
||||
var timeZone = string.IsNullOrWhiteSpace(request.TimeZoneId) ? "UTC" : request.TimeZoneId.Trim();
|
||||
string? cron = null;
|
||||
DateTimeOffset? runOnce = null;
|
||||
@@ -409,6 +416,11 @@ public static class TriggerEndpoints
|
||||
entity.RunOnceUtc = runOnce;
|
||||
entity.NextRunUtc = nextRun;
|
||||
entity.ArgumentsJson = request.Arguments is { Count: > 0 } ? JsonDefaults.Serialize(request.Arguments) : null;
|
||||
|
||||
// Event triggers fire on one device at a time: nothing to spread.
|
||||
var scheduled = request.TriggerType == ScriptTriggerType.Scheduled;
|
||||
entity.StaggerSeconds = scheduled ? request.StaggerSeconds : 0;
|
||||
entity.StaggerBatchSize = scheduled ? request.StaggerBatchSize : 1;
|
||||
return null;
|
||||
}
|
||||
|
||||
|
||||
@@ -11,11 +11,17 @@ public interface IAgentUpgradeSender
|
||||
Task<bool> DispatchAgentUpgradeAsync(string clientId, AgentUpgradeCommandPayload command, CancellationToken cancellationToken);
|
||||
}
|
||||
|
||||
/// <summary>Sends script commands (the job runner's seam; tests record instead).</summary>
|
||||
public interface IScriptCommandSender
|
||||
{
|
||||
Task<bool> DispatchScriptAsync(string clientId, ScriptCommandPayload command, CancellationToken cancellationToken);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Publishes durable agent commands through JetStream (awaiting stream acks)
|
||||
/// so script dispatch survives agent disconnects and backend restarts.
|
||||
/// </summary>
|
||||
public sealed class CommandDispatcher(NatsHub hub, ILogger<CommandDispatcher> logger) : IAgentUpgradeSender
|
||||
public sealed class CommandDispatcher(NatsHub hub, ILogger<CommandDispatcher> logger) : IAgentUpgradeSender, IScriptCommandSender
|
||||
{
|
||||
private static readonly EndpointRef BackendSource = new(EndpointKinds.Backend, "backend");
|
||||
|
||||
@@ -31,6 +37,18 @@ public sealed class CommandDispatcher(NatsHub hub, ILogger<CommandDispatcher> lo
|
||||
return await PublishAsync(NatsSubjects.AgentCommandUpgrade(clientId), envelope, cancellationToken);
|
||||
}
|
||||
|
||||
public async Task<bool> DispatchAgentRemoveAsync(
|
||||
string clientId, AgentRemoveCommandPayload command, CancellationToken cancellationToken)
|
||||
{
|
||||
var envelope = MessageEnvelope.Create(
|
||||
MessageTypes.CommandAgentRemove,
|
||||
BackendSource,
|
||||
command,
|
||||
new EndpointRef(EndpointKinds.Agent, clientId));
|
||||
|
||||
return await PublishAsync(NatsSubjects.AgentCommandRemove(clientId), envelope, cancellationToken);
|
||||
}
|
||||
|
||||
public async Task<bool> DispatchScriptAsync(
|
||||
string clientId, ScriptCommandPayload command, CancellationToken cancellationToken)
|
||||
{
|
||||
|
||||
@@ -0,0 +1,240 @@
|
||||
using Backend.Data;
|
||||
using Backend.Secrets;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Shared.Contracts.Models;
|
||||
using Shared.Contracts.Serialization;
|
||||
using Shared.Infrastructure.Constants;
|
||||
|
||||
namespace Backend.Jobs;
|
||||
|
||||
/// <summary>What one dispatcher pass did (for tests and logs).</summary>
|
||||
public sealed record DispatchPass(int Sent, int Skipped, int Failed, bool More, bool Refused = false);
|
||||
|
||||
/// <summary>
|
||||
/// Sends the jobs whose time has come: staggered jobs at their NotBeforeUtc,
|
||||
/// and any job whose publish failed. Everything it needs is in the database
|
||||
/// (the job's start time, the run's sealed snapshot), so a restart loses
|
||||
/// nothing. Secrets and presigned URLs are prepared afresh for each pass, since
|
||||
/// a run can outlast both.
|
||||
///
|
||||
/// Like the schedule worker it runs on the node with SCHEDULE_WORKER on. A job
|
||||
/// is published first and then marked, only if still queued: a second send
|
||||
/// after a crash is dropped by the agent, which dedupes on JobId.
|
||||
/// </summary>
|
||||
public sealed class JobDispatcher(IServiceScopeFactory scopeFactory, ILogger<JobDispatcher> logger) : BackgroundService
|
||||
{
|
||||
public static readonly TimeSpan Tick = TimeSpan.FromSeconds(2);
|
||||
public const int PassLimit = 500;
|
||||
|
||||
/// <summary>The pause after messaging refused a publish, so an outage is not retried and logged every tick.</summary>
|
||||
public static readonly TimeSpan RefusedDelay = TimeSpan.FromSeconds(15);
|
||||
|
||||
/// <summary>A vault or S3 failure is retried this long before the job fails.</summary>
|
||||
public static readonly TimeSpan PrepareRetryWindow = TimeSpan.FromMinutes(5);
|
||||
|
||||
/// <summary>A job that could not be sent for this long fails, rather than running long after anyone expects it.</summary>
|
||||
public static readonly TimeSpan SendExpiry = TimeSpan.FromHours(24);
|
||||
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
while (!stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
var more = false;
|
||||
var delay = Tick;
|
||||
try
|
||||
{
|
||||
var pass = await RunPassAsync(DateTimeOffset.UtcNow, stoppingToken);
|
||||
more = pass.More;
|
||||
delay = pass.Refused ? RefusedDelay : Tick;
|
||||
}
|
||||
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
return;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
logger.LogError(ex, "Job dispatch pass failed");
|
||||
}
|
||||
|
||||
if (more)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
await Task.Delay(delay, stoppingToken);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
internal async Task<DispatchPass> RunPassAsync(DateTimeOffset now, CancellationToken cancellationToken)
|
||||
{
|
||||
using var scope = scopeFactory.CreateScope();
|
||||
var services = scope.ServiceProvider;
|
||||
var db = services.GetRequiredService<AppDbContext>();
|
||||
var runner = services.GetRequiredService<JobRunner>();
|
||||
var snapshots = services.GetRequiredService<RunSnapshots>();
|
||||
|
||||
var due = await db.Jobs
|
||||
.Where(j => j.State == JobState.Queued && j.NotBeforeUtc != null && j.NotBeforeUtc <= now)
|
||||
.OrderBy(j => j.NotBeforeUtc)
|
||||
.Take(PassLimit)
|
||||
.ToListAsync(cancellationToken);
|
||||
|
||||
var tally = new Tally();
|
||||
var delivering = true;
|
||||
foreach (var group in due.GroupBy(j => j.RunId))
|
||||
{
|
||||
if (group.Key is not { } runId)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
delivering = await DispatchRunAsync(db, runner, snapshots, runId, group.ToList(), now, delivering, tally, cancellationToken);
|
||||
}
|
||||
|
||||
await runner.ClearFinishedSnapshotsAsync(cancellationToken);
|
||||
|
||||
if (tally.Sent + tally.Skipped + tally.Failed > 0)
|
||||
{
|
||||
logger.LogDebug("Dispatch pass: {Sent} sent, {Skipped} skipped, {Failed} failed", tally.Sent, tally.Skipped, tally.Failed);
|
||||
}
|
||||
|
||||
// Messaging being down is no reason to spin: the next pass tries again, after a pause.
|
||||
return new DispatchPass(tally.Sent, tally.Skipped, tally.Failed, More: delivering && due.Count == PassLimit, Refused: !delivering);
|
||||
}
|
||||
|
||||
/// <summary>One run's due jobs. Returns false once a publish was refused, so the rest of the pass sends nothing.</summary>
|
||||
private async Task<bool> DispatchRunAsync(
|
||||
AppDbContext db, JobRunner runner, RunSnapshots snapshots, Guid runId, List<JobEntity> jobs, DateTimeOffset now,
|
||||
bool delivering, Tally tally, CancellationToken cancellationToken)
|
||||
{
|
||||
var run = await db.Runs.SingleOrDefaultAsync(r => r.RunId == runId, cancellationToken);
|
||||
var snapshot = snapshots.Open(run?.SnapshotProtected);
|
||||
if (run is null || snapshot is null)
|
||||
{
|
||||
await FailAllAsync(runner, jobs, "The run's saved script could not be read.", tally, cancellationToken);
|
||||
return delivering;
|
||||
}
|
||||
|
||||
if (!await db.Scripts.AnyAsync(s => s.Id == snapshot.ScriptId, cancellationToken))
|
||||
{
|
||||
await FailAllAsync(runner, jobs, "The script was deleted before this device's start.", tally, cancellationToken);
|
||||
return delivering;
|
||||
}
|
||||
|
||||
var clientIds = jobs.Select(j => j.ClientId).ToList();
|
||||
var devices = await db.Devices.AsNoTracking()
|
||||
.Where(d => clientIds.Contains(d.ClientId))
|
||||
.ToDictionaryAsync(d => d.ClientId, StringComparer.Ordinal, cancellationToken);
|
||||
|
||||
var skips = new List<RunSkip>();
|
||||
var sendable = new List<(JobEntity Job, DeviceEntity Device)>();
|
||||
foreach (var job in jobs)
|
||||
{
|
||||
if (now - job.NotBeforeUtc > SendExpiry)
|
||||
{
|
||||
await FailAsync(runner, job, $"Could not be sent within {SendExpiry.TotalHours:0} hours.", tally, cancellationToken);
|
||||
}
|
||||
else if (!devices.TryGetValue(job.ClientId, out var device))
|
||||
{
|
||||
await FailAsync(runner, job, "The device was removed before its start.", tally, cancellationToken);
|
||||
}
|
||||
else if (device.IsOrphaned)
|
||||
{
|
||||
skips.Add(new RunSkip(device.ClientId, device.Hostname, SkipReasons.Orphaned));
|
||||
await RemoveAsync(db, job, cancellationToken);
|
||||
}
|
||||
else if (run.OnlineOnly && ProtocolDefaults.CalculateStatus(device.LastSeenUtc, now) != DeviceStatus.Online)
|
||||
{
|
||||
skips.Add(new RunSkip(device.ClientId, device.Hostname, SkipReasons.Offline));
|
||||
await RemoveAsync(db, job, cancellationToken);
|
||||
}
|
||||
else
|
||||
{
|
||||
sendable.Add((job, device));
|
||||
}
|
||||
}
|
||||
|
||||
if (skips.Count > 0)
|
||||
{
|
||||
var skipped = JsonDefaults.Deserialize<List<RunSkip>>(run.SkippedJson) ?? [];
|
||||
skipped.AddRange(skips);
|
||||
run.SkippedJson = JsonDefaults.Serialize(skipped);
|
||||
await db.SaveChangesAsync(cancellationToken);
|
||||
tally.Skipped += skips.Count;
|
||||
}
|
||||
|
||||
if (sendable.Count == 0 || !delivering)
|
||||
{
|
||||
return delivering;
|
||||
}
|
||||
|
||||
PreparedScript prepared;
|
||||
try
|
||||
{
|
||||
prepared = await runner.PrepareAsync(snapshot, cancellationToken);
|
||||
}
|
||||
catch (Exception ex) when (ex is SecretResolutionException or Backend.Files.FileResolutionException)
|
||||
{
|
||||
foreach (var (job, _) in sendable.Where(s => now - s.Job.NotBeforeUtc > PrepareRetryWindow))
|
||||
{
|
||||
await FailAsync(runner, job, ex.Message, tally, cancellationToken);
|
||||
}
|
||||
|
||||
logger.LogWarning("Run {RunId}: could not prepare “{Script}”: {Message}", runId, snapshot.Name, ex.Message);
|
||||
return delivering;
|
||||
}
|
||||
|
||||
foreach (var (job, device) in sendable)
|
||||
{
|
||||
var sent = await runner.DispatchAsync(
|
||||
prepared, job, snapshot.Arguments, $"Job “{snapshot.Name}” started for {device.Hostname}", cancellationToken);
|
||||
if (!sent)
|
||||
{
|
||||
logger.LogWarning("Run {RunId}: job {JobId} could not be sent; retrying next pass", runId, job.JobId);
|
||||
return false;
|
||||
}
|
||||
|
||||
tally.Sent++;
|
||||
await runner.PruneHistoryAsync(job.ClientId, cancellationToken);
|
||||
}
|
||||
|
||||
return delivering;
|
||||
}
|
||||
|
||||
private static async Task FailAllAsync(
|
||||
JobRunner runner, List<JobEntity> jobs, string message, Tally tally, CancellationToken cancellationToken)
|
||||
{
|
||||
foreach (var job in jobs)
|
||||
{
|
||||
await FailAsync(runner, job, message, tally, cancellationToken);
|
||||
}
|
||||
}
|
||||
|
||||
private static async Task FailAsync(
|
||||
JobRunner runner, JobEntity job, string message, Tally tally, CancellationToken cancellationToken)
|
||||
{
|
||||
await runner.MarkFailedAsync(job, message, cancellationToken);
|
||||
tally.Failed++;
|
||||
}
|
||||
|
||||
/// <summary>A skipped device has no job: a run counts its targets as jobs plus skips.</summary>
|
||||
private static Task RemoveAsync(AppDbContext db, JobEntity job, CancellationToken cancellationToken)
|
||||
{
|
||||
db.Entry(job).State = EntityState.Detached;
|
||||
return db.Jobs.Where(j => j.JobId == job.JobId && j.State == JobState.Queued).ExecuteDeleteAsync(cancellationToken);
|
||||
}
|
||||
|
||||
private sealed class Tally
|
||||
{
|
||||
public int Sent;
|
||||
public int Skipped;
|
||||
public int Failed;
|
||||
}
|
||||
}
|
||||
@@ -27,6 +27,70 @@ public static class JobEndpoints
|
||||
app.MapGet(ApiRoutes.JobOutputDownload, DownloadJobOutputAsync).RequirePermission(Permissions.JobsView);
|
||||
app.MapGet(ApiRoutes.ExecutionOutputDownload, DownloadExecutionOutputAsync).RequirePermission(Permissions.JobsView);
|
||||
app.MapPost(ApiRoutes.ExecutionOutputArchive, DownloadExecutionArchiveAsync).RequirePermission(Permissions.JobsView);
|
||||
app.MapPost(ApiRoutes.ExecutionCancelWaiting, CancelWaitingAsync).RequirePermission(Permissions.JobsRun);
|
||||
app.MapGet(ApiRoutes.ExecutionBreakdown, GetBreakdownAsync).RequirePermission(Permissions.JobsView);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A run's exit codes, grouped errors and completions over time, for the
|
||||
/// jobs in the caller's scope. Reads only the columns it needs; a job's
|
||||
/// output is read only when nothing else says why it failed.
|
||||
/// </summary>
|
||||
private static async Task<IResult> GetBreakdownAsync(
|
||||
Guid runId, AppDbContext db, DeviceScopeService scope, HttpContext context, CancellationToken cancellationToken)
|
||||
{
|
||||
var jobs = await (await ScopedJobsAsync(db, scope, context, cancellationToken))
|
||||
.Where(j => j.RunId == runId || j.JobId == runId)
|
||||
.Select(j => new RunBreakdownJob(
|
||||
j.JobId, j.ClientId, j.State, j.ExitCode, j.CreatedUtc, j.CompletedUtc, j.ErrorType, j.ErrorMessage, j.OutputSummary))
|
||||
.ToListAsync(cancellationToken);
|
||||
if (jobs.Count == 0)
|
||||
{
|
||||
var run = await db.Runs.AsNoTracking().SingleOrDefaultAsync(r => r.RunId == runId, cancellationToken);
|
||||
var allowed = await scope.GetAllowedClientIdsAsync(context.User, cancellationToken);
|
||||
return SkipsOf(run, allowed).Count == 0 ? Results.NotFound() : Results.Ok(RunBreakdown.Build([], new Dictionary<Guid, string>(), new Dictionary<string, string>()));
|
||||
}
|
||||
|
||||
// The first Error-stream line of each job that has nothing better (agents before error reporting).
|
||||
var silent = RunBreakdown.NeedingOutput(jobs);
|
||||
var firstLines = silent.Count == 0
|
||||
? []
|
||||
: await db.JobOutput.AsNoTracking()
|
||||
.Where(o => silent.Contains(o.JobId) && o.Stream == OutputStreamKind.Error
|
||||
&& !db.JobOutput.Any(p => p.JobId == o.JobId && p.Stream == OutputStreamKind.Error && p.Sequence < o.Sequence))
|
||||
.ToDictionaryAsync(o => o.JobId, o => o.Text, cancellationToken);
|
||||
|
||||
var hostnames = await HostnamesAsync(db, jobs.Select(j => j.ClientId).Distinct().ToList(), cancellationToken);
|
||||
return Results.Ok(RunBreakdown.Build(jobs, firstLines, hostnames));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Cancels the run's jobs not yet sent (those the dispatcher owns), within
|
||||
/// the caller's device scope; a job already on its way is left alone.
|
||||
/// </summary>
|
||||
private static async Task<IResult> CancelWaitingAsync(
|
||||
Guid runId, AppDbContext db, DeviceScopeService scope, JobRunner runner, AuditService audit, HttpContext context,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var jobs = (await ScopedJobsAsync(db, scope, context, cancellationToken)).Where(j => j.RunId == runId);
|
||||
if (!await jobs.AnyAsync(cancellationToken))
|
||||
{
|
||||
return Results.NotFound();
|
||||
}
|
||||
|
||||
var now = DateTimeOffset.UtcNow;
|
||||
var cancelled = await jobs
|
||||
.Where(j => j.State == JobState.Queued && j.NotBeforeUtc != null)
|
||||
.ExecuteUpdateAsync(
|
||||
s => s.SetProperty(j => j.State, JobState.Cancelled)
|
||||
.SetProperty(j => j.CompletedUtc, now)
|
||||
.SetProperty(j => j.OutputSummary, "Cancelled before its start."),
|
||||
cancellationToken);
|
||||
await runner.ClearFinishedSnapshotsAsync(cancellationToken);
|
||||
|
||||
await audit.LogAsync(AuditCategories.Jobs, "cancel-waiting", context.User.Identity?.Name,
|
||||
detail: $"run={runId} cancelled={cancelled}", cancellationToken: cancellationToken);
|
||||
return Results.Ok(new CancelWaitingResultDto(cancelled));
|
||||
}
|
||||
|
||||
/// <summary>One job's complete stored output as a text file.</summary>
|
||||
@@ -255,7 +319,7 @@ public static class JobEndpoints
|
||||
var allowed = await scope.GetAllowedClientIdsAsync(context.User, cancellationToken);
|
||||
var online = await OnlineAsync(db, groups.SelectMany(g => g).Select(j => j.ClientId), cancellationToken);
|
||||
|
||||
return groups.Select(g =>
|
||||
var summaries = groups.Select(g =>
|
||||
{
|
||||
var run = runs.GetValueOrDefault(g.Key);
|
||||
var counts = Count(g.ToList(), SkipsOf(run, allowed), online);
|
||||
@@ -268,12 +332,33 @@ public static class JobEndpoints
|
||||
g.Count(),
|
||||
counts.Succeeded,
|
||||
counts.Failed,
|
||||
counts.Pending + counts.Running,
|
||||
counts.Waiting + counts.Pending + counts.Running,
|
||||
run?.ScriptName ?? first.ScriptName,
|
||||
run?.Source,
|
||||
run?.TargetsLabel,
|
||||
counts);
|
||||
}).ToList();
|
||||
|
||||
// A staggered run whose devices were all offline at their start has only skips left.
|
||||
var emptied = await db.Runs.AsNoTracking()
|
||||
.Where(r => (scriptId == null || r.ScriptId == scriptId)
|
||||
&& r.SkippedJson != "[]"
|
||||
&& !db.Jobs.Any(j => j.RunId == r.RunId))
|
||||
.OrderByDescending(r => r.CreatedUtc)
|
||||
.Take(limit)
|
||||
.ToListAsync(cancellationToken);
|
||||
foreach (var run in emptied)
|
||||
{
|
||||
var skips = SkipsOf(run, allowed);
|
||||
if (skips.Count > 0)
|
||||
{
|
||||
summaries.Add(new ExecutionSummaryDto(
|
||||
run.RunId, run.ScriptId, run.CreatedUtc, run.RequestedBy, 0, 0, 0, 0, run.ScriptName, run.Source,
|
||||
run.TargetsLabel, Count([], skips, online)));
|
||||
}
|
||||
}
|
||||
|
||||
return summaries.OrderByDescending(s => s.StartedUtc).Take(limit).ToList();
|
||||
}
|
||||
|
||||
/// <summary>A single execution: every target with its state, the skipped devices and the counts.</summary>
|
||||
@@ -284,27 +369,32 @@ public static class JobEndpoints
|
||||
.Where(j => j.RunId == runId || j.JobId == runId)
|
||||
.OrderBy(j => j.CreatedUtc)
|
||||
.ToListAsync(cancellationToken);
|
||||
if (jobs.Count == 0)
|
||||
{
|
||||
return Results.NotFound();
|
||||
}
|
||||
|
||||
var run = await db.Runs.AsNoTracking().SingleOrDefaultAsync(r => r.RunId == runId, cancellationToken);
|
||||
var allowed = await scope.GetAllowedClientIdsAsync(context.User, cancellationToken);
|
||||
var skipped = SkipsOf(run, allowed);
|
||||
|
||||
// No jobs but visible skips: a staggered run whose devices were all offline at their start.
|
||||
if (jobs.Count == 0 && skipped.Count == 0)
|
||||
{
|
||||
return Results.NotFound();
|
||||
}
|
||||
|
||||
var hostnames = await HostnamesAsync(db, jobs, cancellationToken);
|
||||
var online = await OnlineAsync(db, jobs.Select(j => j.ClientId), cancellationToken);
|
||||
var targets = jobs.Select(j => new ExecutionTargetDto(
|
||||
j.JobId, j.ClientId, hostnames.GetValueOrDefault(j.ClientId), j.State, j.ExitCode,
|
||||
j.CreatedUtc, j.StartedUtc, j.CompletedUtc, online.Contains(j.ClientId), j.OutputSummary)).ToList();
|
||||
j.CreatedUtc, j.StartedUtc, j.CompletedUtc, online.Contains(j.ClientId), j.OutputSummary ?? j.ErrorMessage,
|
||||
j.NotBeforeUtc)).ToList();
|
||||
|
||||
var first = jobs[0];
|
||||
var first = jobs.FirstOrDefault();
|
||||
return Results.Ok(new ExecutionDetailDto(
|
||||
runId, first.ScriptId, run?.ScriptName ?? first.ScriptName, jobs.Min(j => j.CreatedUtc),
|
||||
runId, first?.ScriptId ?? run!.ScriptId, run?.ScriptName ?? first!.ScriptName, first?.CreatedUtc ?? run!.CreatedUtc,
|
||||
run?.RequestedBy ?? jobs.Select(j => j.RequestedBy).FirstOrDefault(r => r is not null), targets,
|
||||
run?.Source, run?.TargetsLabel, run?.OnlineOnly ?? false,
|
||||
skipped.Select(s => new ExecutionSkipDto(s.ClientId, s.Hostname, s.Reason)).ToList(),
|
||||
Count(jobs, skipped, online)));
|
||||
Count(jobs, skipped, online),
|
||||
DateTimeOffset.UtcNow));
|
||||
}
|
||||
|
||||
/// <summary>The run's skipped devices a scoped role may see.</summary>
|
||||
@@ -336,15 +426,21 @@ public static class JobEndpoints
|
||||
}
|
||||
|
||||
private static RunCountsDto Count(IReadOnlyCollection<JobEntity> jobs, IReadOnlyCollection<RunSkip> skipped, HashSet<string> online)
|
||||
=> new(
|
||||
{
|
||||
var now = DateTimeOffset.UtcNow;
|
||||
var waiting = jobs.Count(j => j.State == JobState.Queued && j.NotBeforeUtc > now);
|
||||
return new(
|
||||
Targets: jobs.Count + skipped.Count,
|
||||
Pending: jobs.Count(j => j.State is JobState.Queued or JobState.Dispatched),
|
||||
Pending: jobs.Count(j => j.State is JobState.Queued or JobState.Dispatched) - waiting,
|
||||
Running: jobs.Count(j => j.State == JobState.Running),
|
||||
Succeeded: jobs.Count(j => j.State == JobState.Succeeded),
|
||||
Failed: jobs.Count(j => j.State is JobState.Failed or JobState.TimedOut),
|
||||
Cancelled: jobs.Count(j => j.State == JobState.Cancelled),
|
||||
Skipped: skipped.Count,
|
||||
Online: jobs.Select(j => j.ClientId).Distinct().Count(online.Contains));
|
||||
Online: jobs.Select(j => j.ClientId).Distinct().Count(online.Contains),
|
||||
Waiting: waiting,
|
||||
TimedOut: jobs.Count(j => j.State == JobState.TimedOut));
|
||||
}
|
||||
|
||||
private static async Task<IResult> ListAsync(
|
||||
AppDbContext db, DeviceScopeService scope, HttpContext context, string? clientId, Guid? scriptId, int? limit,
|
||||
@@ -409,11 +505,23 @@ public static class JobEndpoints
|
||||
|
||||
private static Task<Dictionary<string, string>> HostnamesAsync(
|
||||
AppDbContext db, IReadOnlyCollection<JobEntity> jobs, CancellationToken cancellationToken)
|
||||
=> HostnamesAsync(db, jobs.Select(j => j.ClientId).Distinct().ToList(), cancellationToken);
|
||||
|
||||
private static async Task<Dictionary<string, string>> HostnamesAsync(
|
||||
AppDbContext db, List<string> clientIds, CancellationToken cancellationToken)
|
||||
{
|
||||
var clientIds = jobs.Select(j => j.ClientId).Distinct().ToList();
|
||||
return db.Devices.AsNoTracking()
|
||||
var hostnames = await db.Devices.AsNoTracking()
|
||||
.Where(d => clientIds.Contains(d.ClientId))
|
||||
.ToDictionaryAsync(d => d.ClientId, d => d.Hostname, cancellationToken);
|
||||
|
||||
// Old runs of a removed device keep its name.
|
||||
foreach (var (clientId, hostname) in await Removals.DeviceRemovalService.RemovedHostnamesAsync(
|
||||
db, clientIds.Where(id => !hostnames.ContainsKey(id)), cancellationToken))
|
||||
{
|
||||
hostnames[clientId] = hostname;
|
||||
}
|
||||
|
||||
return hostnames;
|
||||
}
|
||||
|
||||
private static async Task<IResult> CreateAsync(
|
||||
@@ -444,6 +552,11 @@ public static class JobEndpoints
|
||||
return Results.BadRequest(new { error = "Invalid client id." });
|
||||
}
|
||||
|
||||
if (JobStagger.Validate(request.StaggerSeconds, request.StaggerBatchSize) is { } badStagger)
|
||||
{
|
||||
return Results.BadRequest(new { error = badStagger });
|
||||
}
|
||||
|
||||
var script = await db.Scripts.AsNoTracking()
|
||||
.SingleOrDefaultAsync(s => s.Id == request.ScriptId, cancellationToken);
|
||||
if (script is null)
|
||||
@@ -502,7 +615,6 @@ public static class JobEndpoints
|
||||
.ToListAsync(cancellationToken);
|
||||
|
||||
var now = DateTimeOffset.UtcNow;
|
||||
var jobs = new List<JobDto>();
|
||||
var runId = Guid.NewGuid();
|
||||
var via = selected.Count == 1 && clientIds.Count == 0 ? $" (group “{selected[0].Name}”)" : string.Empty;
|
||||
|
||||
@@ -535,14 +647,12 @@ public static class JobEndpoints
|
||||
cancellationToken, request.OnlineOnly);
|
||||
}
|
||||
|
||||
foreach (var device in eligible)
|
||||
{
|
||||
var job = await runner.CreateJobAsync(script, device.ClientId, actor, cancellationToken, runId);
|
||||
await runner.DispatchAsync(
|
||||
prepared, job, request.Arguments,
|
||||
$"Job “{script.Name}” queued for {device.Hostname}{via}", cancellationToken);
|
||||
jobs.Add(ToDto(job, device.Hostname));
|
||||
}
|
||||
var hostnames = eligible.ToDictionary(d => d.ClientId, d => d.Hostname, StringComparer.Ordinal);
|
||||
var jobs = (await runner.StartRunAsync(
|
||||
runId, prepared, request.Arguments, eligible, actor, request.StaggerSeconds, request.StaggerBatchSize,
|
||||
device => $"Job “{script.Name}” queued for {device.Hostname}{via}", cancellationToken))
|
||||
.Select(job => ToDto(job, hostnames.GetValueOrDefault(job.ClientId)))
|
||||
.ToList();
|
||||
|
||||
if (singleDevice && jobs.Count == 0)
|
||||
{
|
||||
@@ -561,7 +671,8 @@ public static class JobEndpoints
|
||||
|
||||
await audit.LogAsync(AuditCategories.Jobs, "group-run", actor,
|
||||
detail: $"script={script.Name} run={runId} groups={selected.Count} devices={clientIds.Count} " +
|
||||
$"jobs={jobs.Count} skippedOrphaned={skippedOrphaned.Count} skippedOffline={skippedOffline.Count}",
|
||||
$"jobs={jobs.Count} skippedOrphaned={skippedOrphaned.Count} skippedOffline={skippedOffline.Count}" +
|
||||
(request.StaggerSeconds > 0 ? $" staggerSeconds={request.StaggerSeconds} staggerBatchSize={request.StaggerBatchSize}" : string.Empty),
|
||||
cancellationToken: cancellationToken);
|
||||
|
||||
return Results.Ok(new GroupRunDto(
|
||||
|
||||
+157
-10
@@ -30,10 +30,11 @@ public sealed record PreparedScript(
|
||||
/// </summary>
|
||||
public sealed class JobRunner(
|
||||
AppDbContext db,
|
||||
CommandDispatcher dispatcher,
|
||||
IScriptCommandSender dispatcher,
|
||||
BrowserEventBroadcaster broadcaster,
|
||||
SecretsProviderService secretsService,
|
||||
Backend.Files.S3PresignService s3,
|
||||
RunSnapshots snapshots,
|
||||
IConfiguration configuration)
|
||||
{
|
||||
/// <summary>
|
||||
@@ -43,9 +44,18 @@ public sealed class JobRunner(
|
||||
/// </summary>
|
||||
public async Task<PreparedScript> PrepareAsync(ScriptEntity script, CancellationToken cancellationToken)
|
||||
{
|
||||
var parameters = Backend.Scripts.ScriptEndpoints.DeserializeParameters(script.ParametersJson);
|
||||
var moduleIds = Backend.Scripts.ScriptEndpoints.DeserializeModuleIds(script.FunctionModuleIdsJson);
|
||||
var prelude = await ComposePreludeAsync(moduleIds, cancellationToken);
|
||||
return await PrepareAsync(script, prelude, cancellationToken);
|
||||
}
|
||||
|
||||
/// <summary>The same from a run's snapshot: secrets and presigned URLs are new, the script and prelude as they were.</summary>
|
||||
public Task<PreparedScript> PrepareAsync(RunSnapshot snapshot, CancellationToken cancellationToken)
|
||||
=> PrepareAsync(snapshot.ToScript(), snapshot.Prelude, cancellationToken);
|
||||
|
||||
private async Task<PreparedScript> PrepareAsync(ScriptEntity script, string? prelude, CancellationToken cancellationToken)
|
||||
{
|
||||
var parameters = Backend.Scripts.ScriptEndpoints.DeserializeParameters(script.ParametersJson);
|
||||
|
||||
IReadOnlyList<ScriptSecret>? secrets = null;
|
||||
var bindings = Backend.Scripts.ScriptEndpoints.DeserializeSecretBindings(script.SecretBindingsJson);
|
||||
@@ -173,12 +183,113 @@ public sealed class JobRunner(
|
||||
return job;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Starts a recorded run on its devices. Without a stagger, each device's
|
||||
/// job is created and sent here; a publish that fails is left for the
|
||||
/// dispatcher to retry. With one, every job is written at once with its
|
||||
/// start time and the dispatcher sends them, so this returns immediately
|
||||
/// however many devices there are.
|
||||
/// </summary>
|
||||
public async Task<IReadOnlyList<JobEntity>> StartRunAsync(
|
||||
Guid runId,
|
||||
PreparedScript prepared,
|
||||
IReadOnlyDictionary<string, JsonElement>? arguments,
|
||||
IReadOnlyList<DeviceEntity> devices,
|
||||
string? requestedBy,
|
||||
int staggerSeconds,
|
||||
int staggerBatchSize,
|
||||
Func<DeviceEntity, string> activityText,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
if (JobStagger.Applies(staggerSeconds, staggerBatchSize, devices.Count))
|
||||
{
|
||||
return await ScheduleStaggeredAsync(
|
||||
runId, prepared, arguments, devices, requestedBy, staggerSeconds, staggerBatchSize, cancellationToken);
|
||||
}
|
||||
|
||||
var jobs = new List<JobEntity>(devices.Count);
|
||||
var unsent = new List<JobEntity>();
|
||||
foreach (var device in devices)
|
||||
{
|
||||
var job = await CreateJobAsync(prepared.Script, device.ClientId, requestedBy, cancellationToken, runId);
|
||||
jobs.Add(job);
|
||||
if (!await DispatchAsync(prepared, job, arguments, activityText(device), cancellationToken))
|
||||
{
|
||||
unsent.Add(job);
|
||||
}
|
||||
}
|
||||
|
||||
if (unsent.Count > 0)
|
||||
{
|
||||
await LeaveForDispatcherAsync(runId, prepared, arguments, unsent, cancellationToken);
|
||||
}
|
||||
|
||||
return jobs;
|
||||
}
|
||||
|
||||
private async Task<IReadOnlyList<JobEntity>> ScheduleStaggeredAsync(
|
||||
Guid runId, PreparedScript prepared, IReadOnlyDictionary<string, JsonElement>? arguments,
|
||||
IReadOnlyList<DeviceEntity> devices, string? requestedBy, int staggerSeconds, int staggerBatchSize,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var script = prepared.Script;
|
||||
var start = DateTimeOffset.UtcNow;
|
||||
var jobs = JobStagger
|
||||
.Plan(runId, devices.Select(d => d.ClientId), staggerSeconds, staggerBatchSize)
|
||||
.Select(slot => new JobEntity
|
||||
{
|
||||
JobId = Guid.NewGuid(),
|
||||
RunId = runId,
|
||||
ClientId = slot.ClientId,
|
||||
ScriptId = script.Id,
|
||||
ScriptName = script.Name,
|
||||
State = JobState.Queued,
|
||||
CreatedUtc = start,
|
||||
RequestedBy = requestedBy,
|
||||
NotBeforeUtc = start + slot.Offset,
|
||||
})
|
||||
.ToList();
|
||||
db.Jobs.AddRange(jobs);
|
||||
|
||||
// One save, so the dispatcher never sees a job without its snapshot.
|
||||
var run = await db.Runs.SingleAsync(r => r.RunId == runId, cancellationToken);
|
||||
run.SnapshotProtected = snapshots.Seal(RunSnapshot.Of(prepared, arguments));
|
||||
await db.SaveChangesAsync(cancellationToken);
|
||||
|
||||
await broadcaster.BroadcastActivityAsync(new ActivityEventPayload(
|
||||
"job-created", $"Job “{script.Name}” staggered across {jobs.Count} devices",
|
||||
null, null, DateTimeOffset.UtcNow), cancellationToken);
|
||||
return jobs;
|
||||
}
|
||||
|
||||
/// <summary>Hands jobs whose publish failed to the dispatcher, with what it needs to send them.</summary>
|
||||
public async Task LeaveForDispatcherAsync(
|
||||
Guid runId, PreparedScript prepared, IReadOnlyDictionary<string, JsonElement>? arguments,
|
||||
IReadOnlyList<JobEntity> jobs, CancellationToken cancellationToken)
|
||||
{
|
||||
var now = DateTimeOffset.UtcNow;
|
||||
foreach (var job in jobs)
|
||||
{
|
||||
job.NotBeforeUtc = now;
|
||||
}
|
||||
|
||||
var run = await db.Runs.SingleAsync(r => r.RunId == runId, cancellationToken);
|
||||
run.SnapshotProtected = snapshots.Seal(RunSnapshot.Of(prepared, arguments));
|
||||
await db.SaveChangesAsync(cancellationToken);
|
||||
}
|
||||
|
||||
/// <summary>Drops the snapshots of runs with nothing left to send: arguments stay at rest no longer than needed.</summary>
|
||||
public Task<int> ClearFinishedSnapshotsAsync(CancellationToken cancellationToken)
|
||||
=> db.Runs
|
||||
.Where(r => r.SnapshotProtected != null && !db.Jobs.Any(j => j.RunId == r.RunId && j.State == JobState.Queued))
|
||||
.ExecuteUpdateAsync(s => s.SetProperty(r => r.SnapshotProtected, (string?)null), cancellationToken);
|
||||
|
||||
/// <summary>
|
||||
/// Execution history retention: keeps the newest JOB_HISTORY_LIMIT jobs
|
||||
/// per device (default 200, 0 disables pruning) and drops the stored
|
||||
/// output of anything older.
|
||||
/// </summary>
|
||||
private async Task PruneHistoryAsync(string clientId, CancellationToken cancellationToken)
|
||||
public async Task PruneHistoryAsync(string clientId, CancellationToken cancellationToken)
|
||||
{
|
||||
var limit = int.TryParse(configuration["JOB_HISTORY_LIMIT"], out var configured) && configured >= 0
|
||||
? configured
|
||||
@@ -228,8 +339,12 @@ public sealed class JobRunner(
|
||||
|
||||
if (dispatched)
|
||||
{
|
||||
job.State = JobState.Dispatched;
|
||||
await db.SaveChangesAsync(cancellationToken);
|
||||
// Only from Queued: a fast agent's Running may already be stored, and
|
||||
// an operator may have cancelled the job meanwhile.
|
||||
var moved = await db.Jobs
|
||||
.Where(j => j.JobId == job.JobId && j.State == JobState.Queued)
|
||||
.ExecuteUpdateAsync(s => s.SetProperty(j => j.State, JobState.Dispatched), cancellationToken);
|
||||
await SyncAsync(job, moved > 0, j => j.State = JobState.Dispatched, cancellationToken);
|
||||
}
|
||||
|
||||
await broadcaster.BroadcastActivityAsync(new ActivityEventPayload(
|
||||
@@ -238,13 +353,45 @@ public sealed class JobRunner(
|
||||
return dispatched;
|
||||
}
|
||||
|
||||
/// <summary>Terminal failure before dispatch (e.g. the vault was unreachable).</summary>
|
||||
/// <summary>Terminal failure before dispatch (e.g. the vault was unreachable). Only a job still queued changes.</summary>
|
||||
public async Task MarkFailedAsync(JobEntity job, string message, CancellationToken cancellationToken)
|
||||
{
|
||||
job.State = JobState.Failed;
|
||||
job.CompletedUtc = DateTimeOffset.UtcNow;
|
||||
job.OutputSummary = message;
|
||||
await db.SaveChangesAsync(cancellationToken);
|
||||
var now = DateTimeOffset.UtcNow;
|
||||
var moved = await db.Jobs
|
||||
.Where(j => j.JobId == job.JobId && j.State == JobState.Queued)
|
||||
.ExecuteUpdateAsync(
|
||||
s => s.SetProperty(j => j.State, JobState.Failed)
|
||||
.SetProperty(j => j.CompletedUtc, now)
|
||||
.SetProperty(j => j.OutputSummary, message),
|
||||
cancellationToken);
|
||||
await SyncAsync(job, moved > 0, j =>
|
||||
{
|
||||
j.State = JobState.Failed;
|
||||
j.CompletedUtc = now;
|
||||
j.OutputSummary = message;
|
||||
}, cancellationToken);
|
||||
}
|
||||
|
||||
/// <summary>Brings a tracked job in line with a direct update (or, when another writer won, with the database).</summary>
|
||||
private async Task SyncAsync(JobEntity job, bool moved, Action<JobEntity> apply, CancellationToken cancellationToken)
|
||||
{
|
||||
var entry = db.Entry(job);
|
||||
if (!moved)
|
||||
{
|
||||
if (entry.State != EntityState.Detached)
|
||||
{
|
||||
await entry.ReloadAsync(cancellationToken);
|
||||
}
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
apply(job);
|
||||
if (entry.State != EntityState.Detached)
|
||||
{
|
||||
entry.OriginalValues.SetValues(entry.CurrentValues);
|
||||
entry.State = EntityState.Unchanged;
|
||||
}
|
||||
}
|
||||
|
||||
private async Task<string?> ComposePreludeAsync(
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
|
||||
namespace Backend.Jobs;
|
||||
|
||||
/// <summary>
|
||||
/// When each device of a staggered run starts. The devices fall into batches
|
||||
/// (slots) of StaggerBatchSize, evenly spaced over StaggerSeconds: the first
|
||||
/// starts at once, the last when the spread ends. The order is shuffled by a
|
||||
/// hash of (run, device), so it is repeatable for a run but the same device is
|
||||
/// not always last.
|
||||
/// </summary>
|
||||
public static class JobStagger
|
||||
{
|
||||
public const int MaxSeconds = 86_400;
|
||||
public const int MaxBatchSize = 10_000;
|
||||
|
||||
/// <summary>The problem with these settings, or null.</summary>
|
||||
public static string? Validate(int seconds, int batchSize)
|
||||
{
|
||||
if (seconds is < 0 or > MaxSeconds)
|
||||
{
|
||||
return $"Spread Over must be between 0 and {MaxSeconds} seconds.";
|
||||
}
|
||||
|
||||
return batchSize is < 1 or > MaxBatchSize
|
||||
? $"Devices at a Time must be between 1 and {MaxBatchSize}."
|
||||
: null;
|
||||
}
|
||||
|
||||
public static int Slots(int targets, int batchSize) =>
|
||||
targets <= 0 ? 0 : (targets + Math.Max(batchSize, 1) - 1) / Math.Max(batchSize, 1);
|
||||
|
||||
/// <summary>False when everything would start at once anyway, and the run dispatches as usual.</summary>
|
||||
public static bool Applies(int seconds, int batchSize, int targets) => seconds > 0 && Slots(targets, batchSize) > 1;
|
||||
|
||||
/// <summary>How long after the run's start a slot begins: slot × spread ÷ (slots − 1).</summary>
|
||||
public static TimeSpan SlotOffset(int slot, int slots, int seconds) =>
|
||||
slots <= 1 ? TimeSpan.Zero : TimeSpan.FromTicks(slot * (seconds * TimeSpan.TicksPerSecond) / (slots - 1));
|
||||
|
||||
/// <summary>Each client id with how long after the run's start it begins, earliest first.</summary>
|
||||
public static IReadOnlyList<(string ClientId, TimeSpan Offset)> Plan(
|
||||
Guid runId, IEnumerable<string> clientIds, int seconds, int batchSize)
|
||||
{
|
||||
var ordered = clientIds
|
||||
.Select(id => (Id: id, Rank: Rank(runId, id)))
|
||||
.OrderBy(d => d.Rank)
|
||||
.ThenBy(d => d.Id, StringComparer.Ordinal)
|
||||
.Select(d => d.Id)
|
||||
.ToList();
|
||||
var size = Math.Max(batchSize, 1);
|
||||
var slots = Slots(ordered.Count, size);
|
||||
return ordered.Select((id, index) => (id, SlotOffset(index / size, slots, seconds))).ToList();
|
||||
}
|
||||
|
||||
private static ulong Rank(Guid runId, string clientId)
|
||||
{
|
||||
Span<byte> hash = stackalloc byte[SHA256.HashSizeInBytes];
|
||||
SHA256.HashData([.. runId.ToByteArray(), .. Encoding.UTF8.GetBytes(clientId)], hash);
|
||||
return BitConverter.ToUInt64(hash);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,143 @@
|
||||
using System.Text.RegularExpressions;
|
||||
using Shared.Contracts.Api;
|
||||
using Shared.Contracts.Models;
|
||||
|
||||
namespace Backend.Jobs;
|
||||
|
||||
/// <summary>What the breakdown needs of one job, so thousands of them stay cheap to read.</summary>
|
||||
public sealed record RunBreakdownJob(
|
||||
Guid JobId,
|
||||
string ClientId,
|
||||
JobState State,
|
||||
int? ExitCode,
|
||||
DateTimeOffset CreatedUtc,
|
||||
DateTimeOffset? CompletedUtc,
|
||||
string? ErrorType,
|
||||
string? ErrorMessage,
|
||||
string? OutputSummary);
|
||||
|
||||
/// <summary>Turns a run's jobs into exit-code counts, grouped errors and completions over time.</summary>
|
||||
public static partial class RunBreakdown
|
||||
{
|
||||
public const int MaxSamples = 5;
|
||||
public const int MaxMessageLength = 300;
|
||||
public const string NoMessage = "No Error Reported";
|
||||
|
||||
private const int MaxBuckets = 60;
|
||||
private static readonly int[] BucketSeconds = [5, 10, 30, 60, 300, 900, 1800, 3600, 7200, 14400, 43200, 86400];
|
||||
|
||||
/// <summary>The failed jobs nothing says why about: their first Error-stream line stands in (the caller reads those).</summary>
|
||||
public static List<Guid> NeedingOutput(IEnumerable<RunBreakdownJob> jobs) => jobs
|
||||
.Where(j => Failed(j) && string.IsNullOrWhiteSpace(j.ErrorMessage) && string.IsNullOrWhiteSpace(j.OutputSummary))
|
||||
.Select(j => j.JobId)
|
||||
.ToList();
|
||||
|
||||
public static ExecutionBreakdownDto Build(
|
||||
IReadOnlyCollection<RunBreakdownJob> jobs,
|
||||
IReadOnlyDictionary<Guid, string> firstErrorLines,
|
||||
IReadOnlyDictionary<string, string> hostnames)
|
||||
{
|
||||
var exitCodes = jobs
|
||||
.Where(j => j.CompletedUtc is not null && j.ExitCode is not null)
|
||||
.GroupBy(j => j.ExitCode!.Value)
|
||||
.Select(g => new ExitCodeCountDto(g.Key, g.Count()))
|
||||
.OrderByDescending(c => c.Count)
|
||||
.ThenBy(c => c.ExitCode)
|
||||
.ToList();
|
||||
|
||||
var errors = jobs
|
||||
.Where(Failed)
|
||||
.Select(j => (Job: j, Message: Normalize(j.ErrorMessage ?? j.OutputSummary ?? firstErrorLines.GetValueOrDefault(j.JobId))))
|
||||
.GroupBy(e => (e.Job.ErrorType, e.Message))
|
||||
.Select(g => new RunErrorGroupDto(
|
||||
g.Key.ErrorType,
|
||||
g.Key.Message,
|
||||
g.Count(),
|
||||
g.Take(MaxSamples)
|
||||
.Select(e => new RunErrorSampleDto(e.Job.JobId, e.Job.ClientId, hostnames.GetValueOrDefault(e.Job.ClientId)))
|
||||
.ToList(),
|
||||
g.Select(e => e.Job.JobId).ToList()))
|
||||
.OrderByDescending(g => g.Count)
|
||||
.ThenBy(g => g.Message, StringComparer.Ordinal)
|
||||
.ToList();
|
||||
|
||||
var (bucketSeconds, completions) = Completions(jobs);
|
||||
return new ExecutionBreakdownDto(exitCodes, errors, bucketSeconds, completions);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// An error's first line, without what differs from device to device or
|
||||
/// run to run: "At line:1 char:1" position text and file system paths.
|
||||
/// </summary>
|
||||
public static string Normalize(string? text)
|
||||
{
|
||||
var line = text?.Split('\n', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries).FirstOrDefault();
|
||||
if (string.IsNullOrEmpty(line))
|
||||
{
|
||||
return NoMessage;
|
||||
}
|
||||
|
||||
line = PositionText().Replace(line, string.Empty);
|
||||
line = QuotedPath().Replace(line, m => $"{m.Groups[1].Value}…{m.Groups[1].Value}");
|
||||
line = BarePath().Replace(line, "…");
|
||||
line = Whitespace().Replace(line, " ").Trim();
|
||||
if (line.Length == 0)
|
||||
{
|
||||
return NoMessage;
|
||||
}
|
||||
|
||||
return line.Length > MaxMessageLength ? line[..(MaxMessageLength - 1)] + "…" : line;
|
||||
}
|
||||
|
||||
private static bool Failed(RunBreakdownJob job) => job.State is JobState.Failed or JobState.TimedOut;
|
||||
|
||||
private static (int Seconds, List<RunCompletionBucketDto> Buckets) Completions(IReadOnlyCollection<RunBreakdownJob> jobs)
|
||||
{
|
||||
var finished = jobs.Where(j => j.CompletedUtc is not null).ToList();
|
||||
if (finished.Count == 0)
|
||||
{
|
||||
return (BucketSeconds[0], []);
|
||||
}
|
||||
|
||||
var start = jobs.Min(j => j.CreatedUtc);
|
||||
var span = Math.Max(0, (finished.Max(j => j.CompletedUtc!.Value) - start).TotalSeconds);
|
||||
var seconds = BucketSeconds.FirstOrDefault(b => span / b < MaxBuckets);
|
||||
if (seconds == 0)
|
||||
{
|
||||
seconds = (int)Math.Ceiling(span / MaxBuckets);
|
||||
}
|
||||
|
||||
var count = (int)(span / seconds) + 1;
|
||||
var tally = new (int Succeeded, int Failed, int TimedOut, int Cancelled)[count];
|
||||
foreach (var job in finished)
|
||||
{
|
||||
// An agent's clock can put a completion before the run's start.
|
||||
var index = Math.Clamp((int)((job.CompletedUtc!.Value - start).TotalSeconds / seconds), 0, count - 1);
|
||||
switch (job.State)
|
||||
{
|
||||
case JobState.Succeeded: tally[index].Succeeded++; break;
|
||||
case JobState.Failed: tally[index].Failed++; break;
|
||||
case JobState.TimedOut: tally[index].TimedOut++; break;
|
||||
case JobState.Cancelled: tally[index].Cancelled++; break;
|
||||
}
|
||||
}
|
||||
|
||||
return (seconds, tally
|
||||
.Select((t, i) => new RunCompletionBucketDto(start.AddSeconds(i * seconds), t.Succeeded, t.Failed, t.TimedOut, t.Cancelled))
|
||||
.ToList());
|
||||
}
|
||||
|
||||
[GeneratedRegex(@"\s*\bAt (?:line:\d+|\S*:\d+) char:\d+.*$", RegexOptions.IgnoreCase)]
|
||||
private static partial Regex PositionText();
|
||||
|
||||
/// <summary>A quoted path, spaces included: 'C:\Program Files\x', "\\server\share\y", '/var/log/z'.</summary>
|
||||
[GeneratedRegex("""(['"])(?:[A-Za-z]:[\\/]|\\\\|/)[^'"]*\1""")]
|
||||
private static partial Regex QuotedPath();
|
||||
|
||||
/// <summary>An unquoted path: C:\Temp\x.log, \\server\share, /var/log/z (two segments at least, so "and/or" stays).</summary>
|
||||
[GeneratedRegex(@"(?:(?<![\w])[A-Za-z]:[\\/][^\s'"",;]*|\\\\[^\s'"",;]+|(?<![\w/:.])/(?:[\w.\-]+/)+[\w.\-]*)")]
|
||||
private static partial Regex BarePath();
|
||||
|
||||
[GeneratedRegex(@"\s+")]
|
||||
private static partial Regex Whitespace();
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
using System.Security.Cryptography;
|
||||
using System.Text.Json;
|
||||
using Backend.Data;
|
||||
using Microsoft.AspNetCore.DataProtection;
|
||||
using Shared.Contracts.Serialization;
|
||||
|
||||
namespace Backend.Jobs;
|
||||
|
||||
/// <summary>
|
||||
/// A script as it was when its run started, with the run's arguments: every
|
||||
/// device of a run the dispatcher sends later gets this, whatever the script
|
||||
/// has become since. It holds secret and file <em>bindings</em>; their values
|
||||
/// and presigned URLs are resolved anew at each dispatch and never stored.
|
||||
/// </summary>
|
||||
public sealed record RunSnapshot(
|
||||
Guid ScriptId,
|
||||
string Name,
|
||||
string Language,
|
||||
string Content,
|
||||
string? ParametersJson,
|
||||
string? Prelude,
|
||||
string? SecretBindingsJson,
|
||||
string? SecretsVariableName,
|
||||
string? FileBindingsJson,
|
||||
IReadOnlyDictionary<string, JsonElement>? Arguments)
|
||||
{
|
||||
public static RunSnapshot Of(PreparedScript prepared, IReadOnlyDictionary<string, JsonElement>? arguments) => new(
|
||||
prepared.Script.Id,
|
||||
prepared.Script.Name,
|
||||
prepared.Script.Language,
|
||||
prepared.Script.Content,
|
||||
prepared.Script.ParametersJson,
|
||||
prepared.Prelude,
|
||||
prepared.Script.SecretBindingsJson,
|
||||
prepared.Script.SecretsVariableName,
|
||||
prepared.Script.FileBindingsJson,
|
||||
arguments);
|
||||
|
||||
/// <summary>A detached script entity to prepare and dispatch from.</summary>
|
||||
public ScriptEntity ToScript() => new()
|
||||
{
|
||||
Id = ScriptId,
|
||||
Name = Name,
|
||||
Language = Language,
|
||||
Content = Content,
|
||||
ParametersJson = ParametersJson,
|
||||
SecretBindingsJson = SecretBindingsJson,
|
||||
SecretsVariableName = SecretsVariableName,
|
||||
FileBindingsJson = FileBindingsJson,
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>Seals run snapshots with Data Protection: arguments can be sensitive, so the database never holds them in the clear.</summary>
|
||||
public sealed class RunSnapshots(IDataProtectionProvider provider)
|
||||
{
|
||||
private readonly IDataProtector protector = provider.CreateProtector("Backend.Jobs.RunSnapshot");
|
||||
|
||||
public string Seal(RunSnapshot snapshot) => protector.Protect(JsonDefaults.Serialize(snapshot));
|
||||
|
||||
/// <summary>The snapshot, or null when it can no longer be opened (the key ring was replaced).</summary>
|
||||
public RunSnapshot? Open(string? sealedValue)
|
||||
{
|
||||
if (sealedValue is null)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
return JsonDefaults.Deserialize<RunSnapshot>(protector.Unprotect(sealedValue));
|
||||
}
|
||||
catch (Exception ex) when (ex is CryptographicException or JsonException)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -62,14 +62,10 @@ public sealed class TriggerExecutor(
|
||||
.ToList();
|
||||
await runner.RecordRunAsync(
|
||||
runId, script, requestedBy, RunSources.Schedule, targetsLabel, skipped, cancellationToken, triggerId: trigger.Id);
|
||||
foreach (var device in devices)
|
||||
{
|
||||
var job = await runner.CreateJobAsync(script, device.ClientId, requestedBy, cancellationToken, runId);
|
||||
await runner.DispatchAsync(
|
||||
prepared, job, arguments,
|
||||
$"{label} queued “{script.Name}” for {device.Hostname}",
|
||||
cancellationToken);
|
||||
}
|
||||
await runner.StartRunAsync(
|
||||
runId, prepared, arguments, devices, requestedBy, trigger.StaggerSeconds, trigger.StaggerBatchSize,
|
||||
device => $"{label} queued “{script.Name}” for {device.Hostname}",
|
||||
cancellationToken);
|
||||
|
||||
logger.LogInformation(
|
||||
"Trigger {TriggerId} queued {Count} job(s) of “{Script}”", trigger.Id, devices.Count, script.Name);
|
||||
|
||||
@@ -92,7 +92,9 @@ public sealed class MessagingWorkers(
|
||||
var device = await db.Devices.SingleOrDefaultAsync(d => d.ClientId == payload.ClientId, cancellationToken);
|
||||
if (device is null)
|
||||
{
|
||||
continue; // Presence from a device we have no enrollment for.
|
||||
// No enrollment: a removal may be waiting for this device; otherwise dropped.
|
||||
await Removals.DeviceRemovalEvents.NoteSightingAsync(scope.ServiceProvider, msg.Subject, payload.ClientId, payload.AgentVersion, cancellationToken);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (device.IsOrphaned)
|
||||
@@ -161,6 +163,7 @@ public sealed class MessagingWorkers(
|
||||
var device = await db.Devices.SingleOrDefaultAsync(d => d.ClientId == clientId, cancellationToken);
|
||||
if (device is null)
|
||||
{
|
||||
await Removals.DeviceRemovalEvents.NoteSightingAsync(scope.ServiceProvider, msg.Subject, clientId, null, cancellationToken);
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -208,8 +211,8 @@ public sealed class MessagingWorkers(
|
||||
/// <summary>
|
||||
/// Agent lifecycle events. A "ready" notice means the agent is connected
|
||||
/// and consuming job commands, which fires OnAgentReady script triggers.
|
||||
/// Agent upgrade progress and outcomes share the subject, told apart by
|
||||
/// the envelope's message type.
|
||||
/// Agent upgrade and removal progress share the subject, told apart by the
|
||||
/// envelope's message type.
|
||||
/// </summary>
|
||||
private async Task ConsumeAgentEventsAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
@@ -230,6 +233,20 @@ public sealed class MessagingWorkers(
|
||||
continue;
|
||||
}
|
||||
|
||||
if (envelope?.MessageType == MessageTypes.AgentRemoval)
|
||||
{
|
||||
try
|
||||
{
|
||||
await Removals.DeviceRemovalEvents.HandleAsync(scopeFactory, msg.Subject, envelope, cancellationToken);
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
logger.LogError(ex, "Agent removal event from {Subject} failed", msg.Subject);
|
||||
}
|
||||
|
||||
continue;
|
||||
}
|
||||
|
||||
AgentEventPayload? payload = null;
|
||||
try
|
||||
{
|
||||
@@ -253,6 +270,7 @@ public sealed class MessagingWorkers(
|
||||
var device = await db.Devices.SingleOrDefaultAsync(d => d.ClientId == payload.ClientId, cancellationToken);
|
||||
if (device is null)
|
||||
{
|
||||
await Removals.DeviceRemovalEvents.NoteSightingAsync(scope.ServiceProvider, msg.Subject, payload.ClientId, null, cancellationToken);
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -350,7 +368,9 @@ public sealed class MessagingWorkers(
|
||||
ApplyJobState(job, payload);
|
||||
await db.SaveChangesAsync(cancellationToken);
|
||||
|
||||
await broadcaster.BroadcastJobEventAsync(payload.JobId, MessageTypes.JobState, payload, cancellationToken);
|
||||
// Browsers get the run too (agents do not know it), so a run page can pick out its own jobs.
|
||||
await broadcaster.BroadcastJobEventAsync(
|
||||
payload.JobId, MessageTypes.JobState, payload with { RunId = job.RunId ?? job.JobId }, cancellationToken);
|
||||
|
||||
if (payload.State is JobState.Running)
|
||||
{
|
||||
@@ -370,7 +390,7 @@ public sealed class MessagingWorkers(
|
||||
}
|
||||
}
|
||||
|
||||
private static void ApplyJobState(JobEntity job, JobStatePayload payload)
|
||||
internal static void ApplyJobState(JobEntity job, JobStatePayload payload)
|
||||
{
|
||||
// Never regress a terminal state (redelivery safety).
|
||||
if (job.State is JobState.Succeeded or JobState.Failed or JobState.Cancelled or JobState.TimedOut)
|
||||
@@ -388,8 +408,13 @@ public sealed class MessagingWorkers(
|
||||
job.CompletedUtc = payload.TimestampUtc;
|
||||
job.ExitCode = payload.ExitCode;
|
||||
job.OutputSummary = payload.Message;
|
||||
job.ErrorType = Cut(payload.ErrorType, 300);
|
||||
job.ErrorId = Cut(payload.ErrorId, 300);
|
||||
job.ErrorMessage = Cut(payload.ErrorMessage, 2000);
|
||||
break;
|
||||
}
|
||||
|
||||
static string? Cut(string? value, int length) => value is { } text && text.Length > length ? text[..length] : value;
|
||||
}
|
||||
|
||||
private async Task SweepStatusesAsync(CancellationToken cancellationToken)
|
||||
|
||||
@@ -74,4 +74,34 @@ public sealed class JetStreamProvisioner(NatsHub hub, ILogger<JetStreamProvision
|
||||
|
||||
logger.LogInformation("Durable command consumer ensured for {ClientId}", clientId);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Deletes one agent's durable command consumer and purges its command
|
||||
/// subjects. The stream keeps acked messages until they age out, and a new
|
||||
/// consumer delivers everything still there, so without this a device enrolled
|
||||
/// again would replay its old commands.
|
||||
/// </summary>
|
||||
public async Task RemoveAgentCommandsAsync(string clientId, CancellationToken cancellationToken)
|
||||
{
|
||||
if (!hub.IsConfigured)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var js = new NatsJSContext(hub.Require());
|
||||
try
|
||||
{
|
||||
await js.DeleteConsumerAsync(
|
||||
JetStreamDefaults.CommandStreamName, JetStreamDefaults.AgentCommandConsumerName(clientId), cancellationToken);
|
||||
}
|
||||
catch (NatsJSApiException ex) when (ex.Error.Code == 404)
|
||||
{
|
||||
// Already gone.
|
||||
}
|
||||
|
||||
var stream = await js.GetStreamAsync(JetStreamDefaults.CommandStreamName, cancellationToken: cancellationToken);
|
||||
await stream.PurgeAsync(new StreamPurgeRequest { Filter = NatsSubjects.AgentCommandWildcard(clientId) }, cancellationToken);
|
||||
|
||||
logger.LogInformation("Durable command consumer and commands cleared for {ClientId}", clientId);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7,9 +7,10 @@ namespace Backend.Nats;
|
||||
|
||||
/// <summary>
|
||||
/// Issues scoped NATS user JWTs signed with the account signing key
|
||||
/// (NATS_ACCOUNT_SIGNING_SEED). Agents get long-lived credentials restricted
|
||||
/// to their own client id; browsers get short-lived credentials restricted to
|
||||
/// their session subjects. The signing seed never appears in logs or responses.
|
||||
/// (NATS_ACCOUNT_SIGNING_SEED). Agents get credentials restricted to their own
|
||||
/// client id, expiring within a day for agents that renew them; browsers get
|
||||
/// short-lived credentials restricted to their session subjects. The signing
|
||||
/// seed never appears in logs or responses.
|
||||
/// </summary>
|
||||
public sealed class NatsCredentialIssuer : IDisposable
|
||||
{
|
||||
@@ -30,12 +31,18 @@ public sealed class NatsCredentialIssuer : IDisposable
|
||||
|
||||
public bool IsConfigured => signingKey is not null;
|
||||
|
||||
public string IssueAgentCredential(string clientId, string userPublicKey)
|
||||
/// <param name="expiresUtc">Null only for agents too old to renew a credential.</param>
|
||||
public string IssueAgentCredential(string clientId, string userPublicKey, DateTimeOffset? expiresUtc)
|
||||
{
|
||||
var (key, account) = RequireConfiguration();
|
||||
|
||||
var claims = NatsJwt.NewUserClaims(userPublicKey);
|
||||
claims.Name = $"agent-{clientId}";
|
||||
if (expiresUtc is { } expires)
|
||||
{
|
||||
claims.Expires = expires;
|
||||
}
|
||||
|
||||
claims.User.IssuerAccount = account;
|
||||
claims.User.Pub.Allow = [.. NatsSubjects.AgentPublishPermissions(clientId)];
|
||||
claims.User.Sub.Allow = [.. NatsSubjects.AgentSubscribePermissions(clientId)];
|
||||
|
||||
+11
-3
@@ -164,6 +164,11 @@ builder.Services.AddSingleton<Backend.Upgrades.AgentUpgradeSettingsStore>();
|
||||
builder.Services.AddScoped<Backend.Upgrades.AgentCatalogService>();
|
||||
builder.Services.AddScoped<Backend.Upgrades.AgentUpgradeService>();
|
||||
builder.Services.AddSingleton<IAgentUpgradeSender>(sp => sp.GetRequiredService<CommandDispatcher>());
|
||||
// Device removals: the tombstones and the sender their scheduler (with the schedules) uses.
|
||||
builder.Services.AddScoped<Backend.Removals.DeviceRemovalService>();
|
||||
builder.Services.AddSingleton<Backend.Removals.IDeviceRemovalSender, Backend.Removals.DeviceRemovalSender>();
|
||||
builder.Services.AddSingleton<IScriptCommandSender>(sp => sp.GetRequiredService<CommandDispatcher>());
|
||||
builder.Services.AddSingleton<RunSnapshots>();
|
||||
if (builder.Configuration["AGENT_CATALOG_SYNC"] != "0")
|
||||
{
|
||||
builder.Services.AddHostedService<Backend.Upgrades.AgentCatalogWorker>();
|
||||
@@ -182,13 +187,15 @@ builder.Services.AddSingleton<Backend.Secrets.ISecretsProviderDriver, Backend.Se
|
||||
builder.Services.AddScoped<Backend.Secrets.SecretsProviderService>();
|
||||
builder.Services.AddSingleton<ShellBridge>();
|
||||
builder.Services.AddHostedService<MessagingWorkers>();
|
||||
// SCHEDULE_WORKER=0 keeps the background sweeps (schedules, agent upgrades)
|
||||
// out of test hosts, where they race the tests' deterministic manual sweeps,
|
||||
// and off all but the one worker replica in a scaled deployment.
|
||||
// SCHEDULE_WORKER=0 keeps the background sweeps (schedules, staggered jobs,
|
||||
// agent upgrades) out of test hosts, where they race the tests' deterministic
|
||||
// manual sweeps, and off all but the one worker replica in a scaled deployment.
|
||||
if (builder.Configuration["SCHEDULE_WORKER"] != "0")
|
||||
{
|
||||
builder.Services.AddHostedService<ScheduleWorker>();
|
||||
builder.Services.AddHostedService<JobDispatcher>();
|
||||
builder.Services.AddHostedService<Backend.Upgrades.AgentUpgradeScheduler>();
|
||||
builder.Services.AddHostedService<Backend.Removals.DeviceRemovalScheduler>();
|
||||
}
|
||||
builder.Services.AddHostedService(sp => sp.GetRequiredService<ShellBridge>());
|
||||
|
||||
@@ -403,6 +410,7 @@ GroupEndpoints.Map(app);
|
||||
TriggerEndpoints.Map(app);
|
||||
SettingsEndpoints.Map(app);
|
||||
Backend.Upgrades.AgentUpgradeEndpoints.Map(app);
|
||||
Backend.Removals.DeviceRemovalEndpoints.Map(app);
|
||||
Backend.Secrets.SecretsEndpoints.Map(app);
|
||||
|
||||
if (staticUiPath is not null)
|
||||
|
||||
@@ -0,0 +1,189 @@
|
||||
using Backend.Api;
|
||||
using Backend.Auth;
|
||||
using Backend.Data;
|
||||
using Backend.Groups;
|
||||
using Backend.Messaging;
|
||||
using Backend.Security;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Shared.Contracts.Api;
|
||||
using Shared.Contracts.Messages;
|
||||
using Shared.Contracts.Models;
|
||||
using Shared.Infrastructure.Nats;
|
||||
|
||||
namespace Backend.Removals;
|
||||
|
||||
/// <summary>Removing devices: the request, the list behind the Devices page, and Stop Trying.</summary>
|
||||
public static class DeviceRemovalEndpoints
|
||||
{
|
||||
public const int DefaultListLimit = 200;
|
||||
|
||||
public static void Map(WebApplication app)
|
||||
{
|
||||
app.MapGet(ApiRoutes.DeviceRemovals, ListAsync).RequirePermission(Permissions.DevicesView);
|
||||
app.MapPost(ApiRoutes.DeviceRemovals, RemoveAsync).RequirePermission(Permissions.DevicesRemove);
|
||||
app.MapPost(ApiRoutes.DeviceRemovalStop, StopAsync).RequirePermission(Permissions.DevicesRemove);
|
||||
}
|
||||
|
||||
private static async Task<IResult> ListAsync(
|
||||
string? state, string? clientId, int? limit, AppDbContext db, DeviceScopeService scope, HttpContext context,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var query = db.DeviceRemovals.AsNoTracking();
|
||||
if (!string.IsNullOrWhiteSpace(state))
|
||||
{
|
||||
query = query.Where(r => r.State == state);
|
||||
}
|
||||
|
||||
if (!string.IsNullOrWhiteSpace(clientId))
|
||||
{
|
||||
query = query.Where(r => r.ClientId == clientId);
|
||||
}
|
||||
|
||||
var take = Math.Clamp(limit ?? DefaultListLimit, 1, 5000);
|
||||
query = query.OrderByDescending(r => r.RequestedUtc);
|
||||
|
||||
// A scoped role sees removals by the groups kept on each row: they are
|
||||
// JSON, so the page is cut after this filter for such a role.
|
||||
var scopeGroupIds = await scope.GetScopeGroupIdsAsync(context.User, cancellationToken);
|
||||
var rows = scopeGroupIds is null
|
||||
? await query.Take(take).ToListAsync(cancellationToken)
|
||||
: (await query.ToListAsync(cancellationToken))
|
||||
.Where(r => DeviceRemovalService.IsVisibleTo(r, scopeGroupIds))
|
||||
.Take(take)
|
||||
.ToList();
|
||||
|
||||
var now = DateTimeOffset.UtcNow;
|
||||
return Results.Ok(rows.Select(r => DeviceRemovalService.ToDto(r, now)).ToList());
|
||||
}
|
||||
|
||||
private static async Task<IResult> RemoveAsync(
|
||||
RemoveDevicesRequest request, AppDbContext db, DeviceRemovalService removals, DeviceGroupService groups,
|
||||
DeviceScopeService scope, AuditService audit, BrowserEventBroadcaster broadcaster, HttpContext context,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var named = (request.ClientIds ?? []).Distinct(StringComparer.Ordinal).ToList();
|
||||
var groupIds = (request.GroupIds ?? []).Distinct().ToList();
|
||||
if (named.Count == 0 && groupIds.Count == 0)
|
||||
{
|
||||
return Results.BadRequest(new { error = "Choose at least one device or group." });
|
||||
}
|
||||
|
||||
if (named.Any(id => !SubjectIdentifier.IsValid(id)))
|
||||
{
|
||||
return Results.BadRequest(new { error = "Invalid client id." });
|
||||
}
|
||||
|
||||
var selected = await db.DeviceGroups.AsNoTracking()
|
||||
.Where(g => groupIds.Contains(g.Id))
|
||||
.ToListAsync(cancellationToken);
|
||||
if (selected.Count < groupIds.Count)
|
||||
{
|
||||
return Results.NotFound(new { error = groupIds.Count == 1 ? "Group not found." : "Some groups no longer exist." });
|
||||
}
|
||||
|
||||
var fromGroups = new HashSet<string>(StringComparer.Ordinal);
|
||||
foreach (var group in selected)
|
||||
{
|
||||
fromGroups.UnionWith(await groups.GetMembersAsync(group, cancellationToken));
|
||||
}
|
||||
|
||||
// A scoped role only reaches the devices inside its own scope. A device it
|
||||
// named outside that reads the same whether it exists or not; one a group
|
||||
// brought in is not its to know about, as with agent upgrades.
|
||||
var allowed = await scope.GetAllowedClientIdsAsync(context.User, cancellationToken);
|
||||
var skipped = new List<DeviceRemovalSkipDto>();
|
||||
var targets = new HashSet<string>(StringComparer.Ordinal);
|
||||
foreach (var clientId in named)
|
||||
{
|
||||
if (allowed is null || allowed.Contains(clientId))
|
||||
{
|
||||
targets.Add(clientId);
|
||||
}
|
||||
else
|
||||
{
|
||||
skipped.Add(new DeviceRemovalSkipDto(clientId, null, DeviceRemovalService.SkipOutOfScope));
|
||||
}
|
||||
}
|
||||
|
||||
targets.UnionWith(allowed is null ? fromGroups : fromGroups.Where(allowed.Contains));
|
||||
|
||||
var note = string.IsNullOrWhiteSpace(request.Reason) ? null : request.Reason.Trim();
|
||||
if (note is { Length: > DeviceRemovalService.NoteMaxLength })
|
||||
{
|
||||
note = note[..DeviceRemovalService.NoteMaxLength];
|
||||
}
|
||||
|
||||
var actor = context.User.Identity?.Name;
|
||||
var (removed, unknown) = await removals.RemoveAsync(targets.Order(StringComparer.Ordinal).ToList(), note, actor, cancellationToken);
|
||||
skipped.AddRange(unknown);
|
||||
|
||||
var left = string.Join(' ', skipped.GroupBy(s => s.Reason).Select(g => $"{g.Key}={g.Count()}"));
|
||||
await audit.LogAsync(AuditCategories.Admin, "device-removal-requested", actor,
|
||||
clientId: removed.Count == 1 ? removed[0].ClientId : null,
|
||||
detail: $"removed={removed.Count}{(left.Length > 0 ? " skipped: " + left : string.Empty)}",
|
||||
cancellationToken: cancellationToken);
|
||||
|
||||
if (removed.Count > 0)
|
||||
{
|
||||
await broadcaster.BroadcastActivityAsync(new ActivityEventPayload(
|
||||
DeviceRemovalService.ActivityKind,
|
||||
removed.Count == 1 ? $"{removed[0].Hostname} removal requested" : $"Removal requested for {removed.Count} devices",
|
||||
removed.Count == 1 ? removed[0].ClientId : null, null, DateTimeOffset.UtcNow), cancellationToken);
|
||||
}
|
||||
|
||||
var now = DateTimeOffset.UtcNow;
|
||||
return Results.Ok(new DeviceRemovalBatchDto(
|
||||
removed.OrderBy(r => r.Hostname, StringComparer.OrdinalIgnoreCase).Select(r => DeviceRemovalService.ToDto(r, now)).ToList(),
|
||||
skipped));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Stops chasing a device: it stays blocked and its agent stays installed.
|
||||
/// A remove command still waiting for it is purged so it cannot arrive later;
|
||||
/// that is best effort, since a command left behind expires with the stream.
|
||||
/// </summary>
|
||||
private static async Task<IResult> StopAsync(
|
||||
Guid id, AppDbContext db, DeviceRemovalService removals, IDeviceRemovalSender sender, DeviceScopeService scope,
|
||||
AuditService audit, HttpContext context, ILogger<DeviceRemovalService> logger, CancellationToken cancellationToken)
|
||||
{
|
||||
var row = await db.DeviceRemovals.SingleOrDefaultAsync(r => r.Id == id, cancellationToken);
|
||||
if (row is null || !DeviceRemovalService.IsVisibleTo(row, await scope.GetScopeGroupIdsAsync(context.User, cancellationToken)))
|
||||
{
|
||||
return Results.NotFound();
|
||||
}
|
||||
|
||||
if (!DeviceRemovalStates.Open.Contains(row.State))
|
||||
{
|
||||
return Results.Conflict(new { error = $"This removal has already finished; it is {row.State}." });
|
||||
}
|
||||
|
||||
var now = DateTimeOffset.UtcNow;
|
||||
row.State = DeviceRemovalStates.Stopped;
|
||||
row.FinishedUtc = now;
|
||||
row.UpdatedUtc = now;
|
||||
var actor = context.User.Identity?.Name;
|
||||
DeviceHistory.RecordRemovalStopped(db, row.ClientId, actor);
|
||||
try
|
||||
{
|
||||
await db.SaveChangesAsync(cancellationToken);
|
||||
}
|
||||
catch (DbUpdateConcurrencyException)
|
||||
{
|
||||
return Results.Conflict(new { error = "The removal changed meanwhile. Try again." });
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
await sender.ClearCommandsAsync(row.ClientId, cancellationToken);
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
logger.LogWarning(ex, "Could not clear the commands of device {ClientId} after its removal stopped", row.ClientId);
|
||||
}
|
||||
|
||||
await audit.LogAsync(AuditCategories.Admin, "device-removal-stopped", actor, clientId: row.ClientId,
|
||||
detail: $"removal={row.Id}", cancellationToken: cancellationToken);
|
||||
await removals.AnnounceAsync(row, cancellationToken);
|
||||
return Results.Ok(DeviceRemovalService.ToDto(row, now));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,288 @@
|
||||
using Backend.Api;
|
||||
using Backend.Data;
|
||||
using Backend.Jobs;
|
||||
using Backend.Nats;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Shared.Contracts.Messages;
|
||||
using Shared.Contracts.Models;
|
||||
using Shared.Infrastructure.Constants;
|
||||
|
||||
namespace Backend.Removals;
|
||||
|
||||
/// <summary>Reaches an agent for its removal (the seam the tests record through).</summary>
|
||||
public interface IDeviceRemovalSender
|
||||
{
|
||||
/// <summary>Publishes the remove command; false when messaging is down.</summary>
|
||||
Task<bool> SendRemoveAsync(string clientId, AgentRemoveCommandPayload command, CancellationToken cancellationToken);
|
||||
|
||||
/// <summary>
|
||||
/// Deletes the agent's durable consumer and purges its command subjects, so
|
||||
/// a device enrolled again never replays old commands. Throws when messaging
|
||||
/// is down and something may be left behind.
|
||||
/// </summary>
|
||||
Task ClearCommandsAsync(string clientId, CancellationToken cancellationToken);
|
||||
}
|
||||
|
||||
public sealed class DeviceRemovalSender(CommandDispatcher dispatcher, JetStreamProvisioner provisioner, ILogger<DeviceRemovalSender> logger)
|
||||
: IDeviceRemovalSender
|
||||
{
|
||||
public async Task<bool> SendRemoveAsync(string clientId, AgentRemoveCommandPayload command, CancellationToken cancellationToken)
|
||||
{
|
||||
try
|
||||
{
|
||||
// A removal reopened after it closed has lost its consumer.
|
||||
await provisioner.EnsureAgentConsumerAsync(clientId, cancellationToken);
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
logger.LogWarning(ex, "Could not ensure the command consumer for {ClientId}", clientId);
|
||||
return false;
|
||||
}
|
||||
|
||||
return await dispatcher.DispatchAgentRemoveAsync(clientId, command, cancellationToken);
|
||||
}
|
||||
|
||||
public Task ClearCommandsAsync(string clientId, CancellationToken cancellationToken)
|
||||
=> provisioner.RemoveAgentCommandsAsync(clientId, cancellationToken);
|
||||
}
|
||||
|
||||
/// <summary>What one scheduler pass did (for tests and logs).</summary>
|
||||
public sealed record RemovalPass(int Sent, int Finished);
|
||||
|
||||
/// <summary>
|
||||
/// Drives open removals every 15 seconds:
|
||||
/// <list type="bullet">
|
||||
/// <item>Sends the remove command to a device seen online, not more often than
|
||||
/// <see cref="ResendAfter"/>. An agent too old to take it is marked and stays
|
||||
/// open: the server keeps blocking it.</item>
|
||||
/// <item>A command nobody answered goes back to waiting after
|
||||
/// <see cref="ReportDeadline"/>, for when the device next returns.</item>
|
||||
/// <item>An agent that said it was uninstalling and then went silent counts as
|
||||
/// gone, since nobody is left to report success: its consumer and commands are
|
||||
/// cleared and the removal closes. One that keeps talking was not uninstalled,
|
||||
/// and goes back to waiting.</item>
|
||||
/// </list>
|
||||
/// Like the schedule worker it runs on the node with SCHEDULE_WORKER on; each
|
||||
/// row's write is guarded by its concurrency stamp, so a second scheduler, an
|
||||
/// event on another node or an operator's stop can never double-send a row.
|
||||
/// </summary>
|
||||
public sealed class DeviceRemovalScheduler(
|
||||
IServiceScopeFactory scopeFactory,
|
||||
IDeviceRemovalSender sender,
|
||||
ILogger<DeviceRemovalScheduler> logger) : BackgroundService
|
||||
{
|
||||
public static readonly TimeSpan Tick = TimeSpan.FromSeconds(15);
|
||||
public static readonly TimeSpan ResendAfter = TimeSpan.FromMinutes(5);
|
||||
public static readonly TimeSpan ReportDeadline = TimeSpan.FromMinutes(30);
|
||||
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
while (!stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
try
|
||||
{
|
||||
await RunPassAsync(DateTimeOffset.UtcNow, stoppingToken);
|
||||
}
|
||||
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
return;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
logger.LogError(ex, "Device removal pass failed");
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
await Task.Delay(Tick, stoppingToken);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
internal async Task<RemovalPass> RunPassAsync(DateTimeOffset now, CancellationToken cancellationToken)
|
||||
{
|
||||
using var scope = scopeFactory.CreateScope();
|
||||
var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
|
||||
var removals = scope.ServiceProvider.GetRequiredService<DeviceRemovalService>();
|
||||
|
||||
var open = DeviceRemovalStates.Open.ToList();
|
||||
var rows = await db.DeviceRemovals.Where(r => open.Contains(r.State)).ToListAsync(cancellationToken);
|
||||
|
||||
var sent = 0;
|
||||
var finished = 0;
|
||||
var messagingDown = false;
|
||||
foreach (var row in rows)
|
||||
{
|
||||
switch (row.State)
|
||||
{
|
||||
case DeviceRemovalStates.Uninstalling:
|
||||
finished += await AdvanceUninstallingAsync(db, removals, row, now, cancellationToken) ? 1 : 0;
|
||||
break;
|
||||
|
||||
case DeviceRemovalStates.Sent:
|
||||
if (now - (row.SentUtc ?? row.UpdatedUtc) > ReportDeadline)
|
||||
{
|
||||
row.State = DeviceRemovalStates.Waiting;
|
||||
row.UpdatedUtc = now;
|
||||
if (await TrySaveAsync(db, row, cancellationToken))
|
||||
{
|
||||
await removals.AnnounceAsync(row, cancellationToken);
|
||||
}
|
||||
}
|
||||
|
||||
break;
|
||||
|
||||
case DeviceRemovalStates.Waiting or DeviceRemovalStates.Failed when !messagingDown:
|
||||
switch (await TrySendAsync(db, removals, row, now, cancellationToken))
|
||||
{
|
||||
case SendResult.Sent:
|
||||
sent++;
|
||||
break;
|
||||
case SendResult.MessagingDown:
|
||||
messagingDown = true;
|
||||
logger.LogWarning("Removal {RemovalId} could not be sent; retrying next pass", row.Id);
|
||||
break;
|
||||
}
|
||||
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
return new RemovalPass(sent, finished);
|
||||
}
|
||||
|
||||
private enum SendResult
|
||||
{
|
||||
NotDue,
|
||||
Sent,
|
||||
MessagingDown,
|
||||
}
|
||||
|
||||
private async Task<SendResult> TrySendAsync(
|
||||
AppDbContext db, DeviceRemovalService removals, DeviceRemovalEntity row, DateTimeOffset now, CancellationToken cancellationToken)
|
||||
{
|
||||
// Presence brings the running version, so an agent upgraded by hand gets past this.
|
||||
var tooOld = !string.IsNullOrWhiteSpace(row.AgentVersion)
|
||||
&& AgentVersions.Compare(row.AgentVersion, DeviceRemovalDefaults.MinimumAgentVersion) < 0;
|
||||
if (tooOld)
|
||||
{
|
||||
if (row.Reason != DeviceRemovalReasons.AgentTooOld)
|
||||
{
|
||||
row.Reason = DeviceRemovalReasons.AgentTooOld;
|
||||
row.Detail = $"The agent runs {row.AgentVersion}; removing it needs {DeviceRemovalDefaults.MinimumAgentVersion} or later.";
|
||||
row.UpdatedUtc = now;
|
||||
if (await TrySaveAsync(db, row, cancellationToken))
|
||||
{
|
||||
await removals.AnnounceAsync(row, cancellationToken);
|
||||
}
|
||||
}
|
||||
|
||||
return SendResult.NotDue;
|
||||
}
|
||||
|
||||
var online = ProtocolDefaults.CalculateStatus(row.LastSeenUtc, now) == DeviceStatus.Online;
|
||||
if (!online
|
||||
|| string.IsNullOrWhiteSpace(row.AgentVersion)
|
||||
|| (row.SentUtc is { } lastSent && now - lastSent < ResendAfter))
|
||||
{
|
||||
return SendResult.NotDue;
|
||||
}
|
||||
|
||||
// Mark it sent first: a row is never handed to JetStream twice.
|
||||
var (state, reason, detail, sentUtc, updatedUtc) = (row.State, row.Reason, row.Detail, row.SentUtc, row.UpdatedUtc);
|
||||
row.State = DeviceRemovalStates.Sent;
|
||||
row.Reason = null;
|
||||
row.Detail = null;
|
||||
row.SentUtc = now;
|
||||
row.UpdatedUtc = now;
|
||||
row.Attempts++;
|
||||
if (!await TrySaveAsync(db, row, cancellationToken))
|
||||
{
|
||||
return SendResult.NotDue; // Stopped or taken meanwhile.
|
||||
}
|
||||
|
||||
if (!await sender.SendRemoveAsync(row.ClientId, new AgentRemoveCommandPayload(row.Id, row.EnrollmentId), cancellationToken))
|
||||
{
|
||||
// Messaging is down: as it was, the next pass tries again.
|
||||
(row.State, row.Reason, row.Detail, row.SentUtc, row.UpdatedUtc) = (state, reason, detail, sentUtc, updatedUtc);
|
||||
row.Attempts--;
|
||||
await TrySaveAsync(db, row, cancellationToken);
|
||||
return SendResult.MessagingDown;
|
||||
}
|
||||
|
||||
await removals.AnnounceAsync(row, cancellationToken);
|
||||
return SendResult.Sent;
|
||||
}
|
||||
|
||||
/// <summary>True when the removal closed.</summary>
|
||||
private async Task<bool> AdvanceUninstallingAsync(
|
||||
AppDbContext db, DeviceRemovalService removals, DeviceRemovalEntity row, DateTimeOffset now, CancellationToken cancellationToken)
|
||||
{
|
||||
// UpdatedUtc is when it said so: presence moves only LastSeenUtc.
|
||||
if (row.LastSeenUtc is { } seen && seen - row.UpdatedUtc > DeviceRemovalDefaults.UninstallSilence)
|
||||
{
|
||||
row.State = DeviceRemovalStates.Waiting;
|
||||
row.Detail = "The agent kept reporting after it said it was uninstalling.";
|
||||
row.UpdatedUtc = now;
|
||||
if (await TrySaveAsync(db, row, cancellationToken))
|
||||
{
|
||||
await removals.AnnounceAsync(row, cancellationToken);
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
var lastSign = row.LastSeenUtc is { } last && last > row.UpdatedUtc ? last : row.UpdatedUtc;
|
||||
if (now - lastSign <= DeviceRemovalDefaults.UninstallSilence)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
await sender.ClearCommandsAsync(row.ClientId, cancellationToken);
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
// Close it only once nothing can replay; the next pass tries again.
|
||||
logger.LogWarning(ex, "Could not clear the commands of removed device {ClientId}", row.ClientId);
|
||||
return false;
|
||||
}
|
||||
|
||||
row.State = DeviceRemovalStates.Removed;
|
||||
row.FinishedUtc = now;
|
||||
row.UpdatedUtc = now;
|
||||
DeviceHistory.RecordRemoved(db, row.ClientId, "The agent said it was uninstalling, then went silent.");
|
||||
if (!await TrySaveAsync(db, row, cancellationToken))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
await removals.AnnounceAsync(row, cancellationToken);
|
||||
return true;
|
||||
}
|
||||
|
||||
/// <summary>Saves one row's change; false (the row re-read, its history dropped) when it changed elsewhere first.</summary>
|
||||
private static async Task<bool> TrySaveAsync(AppDbContext db, DeviceRemovalEntity row, CancellationToken cancellationToken)
|
||||
{
|
||||
try
|
||||
{
|
||||
await db.SaveChangesAsync(cancellationToken);
|
||||
return true;
|
||||
}
|
||||
catch (DbUpdateConcurrencyException)
|
||||
{
|
||||
foreach (var added in db.ChangeTracker.Entries<AuditEventEntity>().Where(e => e.State == EntityState.Added).ToList())
|
||||
{
|
||||
added.State = EntityState.Detached;
|
||||
}
|
||||
|
||||
await db.Entry(row).ReloadAsync(cancellationToken);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,400 @@
|
||||
using Backend.Api;
|
||||
using Backend.Data;
|
||||
using Backend.Groups;
|
||||
using Backend.Messaging;
|
||||
using Backend.Security;
|
||||
using Backend.Upgrades;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Shared.Contracts.Api;
|
||||
using Shared.Contracts.Messages;
|
||||
using Shared.Contracts.Models;
|
||||
using Shared.Contracts.Serialization;
|
||||
using Shared.Infrastructure.Nats;
|
||||
|
||||
namespace Backend.Removals;
|
||||
|
||||
/// <summary>
|
||||
/// Removing devices: the tombstone that stands in for a deleted device row, the
|
||||
/// agent's sightings and events, and the device-history trail. The scheduler
|
||||
/// sends what is open here; nothing is sent from this class.
|
||||
/// </summary>
|
||||
public sealed class DeviceRemovalService(
|
||||
AppDbContext db,
|
||||
DeviceGroupService groups,
|
||||
BrowserEventBroadcaster broadcaster,
|
||||
ILogger<DeviceRemovalService> logger)
|
||||
{
|
||||
/// <summary>Activity kind consoles refresh devices and removals on.</summary>
|
||||
public const string ActivityKind = "device-removal";
|
||||
|
||||
public const string SkipOutOfScope = "out-of-scope";
|
||||
public const string SkipUnknown = "unknown";
|
||||
|
||||
public const int NoteMaxLength = 500;
|
||||
private const int DetailMaxLength = 2000;
|
||||
private const int ReasonMaxLength = 32;
|
||||
|
||||
/// <summary>States an agent may report.</summary>
|
||||
private static readonly IReadOnlySet<string> AgentStates = new HashSet<string>(StringComparer.Ordinal)
|
||||
{
|
||||
DeviceRemovalStates.Uninstalling, DeviceRemovalStates.Failed,
|
||||
};
|
||||
|
||||
public static DeviceRemovalDto ToDto(DeviceRemovalEntity row, DateTimeOffset now) => new(
|
||||
row.Id, row.ClientId, row.Hostname, row.OsCaption, row.AgentVersion, row.State, row.Reason, row.Detail,
|
||||
row.Attempts, row.RequestedBy, row.Note, row.RequestedUtc, row.UpdatedUtc, row.SentUtc, row.LastSeenUtc,
|
||||
row.FinishedUtc, ShowInDeviceList(row, now));
|
||||
|
||||
public static bool ShowInDeviceList(DeviceRemovalEntity row, DateTimeOffset now)
|
||||
=> now - row.UpdatedUtc < DeviceRemovalDefaults.VisibleInDeviceList;
|
||||
|
||||
/// <summary>
|
||||
/// For each device: its tombstone in, its row and everything that would
|
||||
/// reattach to a returning client id out, in one save. A device that is
|
||||
/// already gone (a second request racing this one) is skipped as unknown.
|
||||
/// </summary>
|
||||
public async Task<(List<DeviceRemovalEntity> Removed, List<DeviceRemovalSkipDto> Skipped)> RemoveAsync(
|
||||
IReadOnlyList<string> clientIds, string? note, string? requestedBy, CancellationToken cancellationToken)
|
||||
{
|
||||
var removed = new List<DeviceRemovalEntity>();
|
||||
var skipped = new List<DeviceRemovalSkipDto>();
|
||||
if (clientIds.Count == 0)
|
||||
{
|
||||
return (removed, skipped);
|
||||
}
|
||||
|
||||
// Read while the rows exist: a dynamic group's rules need them. Group ids
|
||||
// are what lets a scoped role keep seeing the removals of its devices.
|
||||
var groupsOf = await EffectiveGroupsAsync(clientIds, cancellationToken);
|
||||
foreach (var clientId in clientIds)
|
||||
{
|
||||
if (await TryRemoveOneAsync(clientId, groupsOf.GetValueOrDefault(clientId), note, requestedBy, cancellationToken) is { } row)
|
||||
{
|
||||
removed.Add(row);
|
||||
}
|
||||
else
|
||||
{
|
||||
skipped.Add(new DeviceRemovalSkipDto(clientId, null, SkipUnknown));
|
||||
}
|
||||
}
|
||||
|
||||
return (removed, skipped);
|
||||
}
|
||||
|
||||
private async Task<Dictionary<string, List<Guid>>> EffectiveGroupsAsync(
|
||||
IReadOnlyCollection<string> clientIds, CancellationToken cancellationToken)
|
||||
{
|
||||
var result = new Dictionary<string, List<Guid>>(StringComparer.Ordinal);
|
||||
if (!await db.DeviceGroups.AnyAsync(cancellationToken))
|
||||
{
|
||||
return result;
|
||||
}
|
||||
|
||||
var wanted = clientIds.ToHashSet(StringComparer.Ordinal);
|
||||
var membership = await groups.GetMembershipAsync(cancellationToken);
|
||||
foreach (var group in membership.Groups)
|
||||
{
|
||||
foreach (var clientId in membership.Effective(group.Id).Where(wanted.Contains))
|
||||
{
|
||||
if (!result.TryGetValue(clientId, out var list))
|
||||
{
|
||||
result[clientId] = list = [];
|
||||
}
|
||||
|
||||
list.Add(group.Id);
|
||||
}
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
private async Task<DeviceRemovalEntity?> TryRemoveOneAsync(
|
||||
string clientId, List<Guid>? groupIds, string? note, string? requestedBy, CancellationToken cancellationToken)
|
||||
{
|
||||
for (var attempt = 0; attempt < 3; attempt++)
|
||||
{
|
||||
var device = await db.Devices.SingleOrDefaultAsync(d => d.ClientId == clientId, cancellationToken);
|
||||
if (device is null)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
var now = DateTimeOffset.UtcNow;
|
||||
var row = new DeviceRemovalEntity
|
||||
{
|
||||
Id = Guid.NewGuid(),
|
||||
ClientId = clientId,
|
||||
NkeyPublic = device.NkeyPublic,
|
||||
EnrollmentId = device.EnrollmentId,
|
||||
Hostname = device.Hostname,
|
||||
OsCaption = device.OsCaption,
|
||||
AgentVersion = device.AgentVersion,
|
||||
State = DeviceRemovalStates.Waiting,
|
||||
RequestedBy = requestedBy,
|
||||
Note = note,
|
||||
RequestedUtc = now,
|
||||
UpdatedUtc = now,
|
||||
LastSeenUtc = device.LastSeenUtc,
|
||||
GroupIdsJson = groupIds is { Count: > 0 } ? JsonDefaults.Serialize(groupIds) : null,
|
||||
};
|
||||
db.DeviceRemovals.Add(row);
|
||||
await StageDependentsAsync(clientId, now, cancellationToken);
|
||||
db.Devices.Remove(device);
|
||||
DeviceHistory.RecordRemovalRequested(db, clientId, requestedBy, note);
|
||||
|
||||
try
|
||||
{
|
||||
await db.SaveChangesAsync(cancellationToken);
|
||||
return row;
|
||||
}
|
||||
catch (DbUpdateConcurrencyException)
|
||||
{
|
||||
// A schedule fired or the device went meanwhile: read it all again.
|
||||
db.ChangeTracker.Clear();
|
||||
}
|
||||
}
|
||||
|
||||
logger.LogWarning("Device {ClientId} kept changing; its removal was left out", clientId);
|
||||
return null;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// What references the client id without a foreign key. Group memberships
|
||||
/// would attach to the device again if the id returns; device-specific
|
||||
/// schedules would keep firing (disabled, as when their group is deleted);
|
||||
/// trigger runs would stop on-registration triggers firing after a legitimate
|
||||
/// re-enrollment. Jobs that never started would sit queued forever. Finished
|
||||
/// jobs, their output and the audit trail stay as history.
|
||||
/// </summary>
|
||||
private async Task StageDependentsAsync(string clientId, DateTimeOffset now, CancellationToken cancellationToken)
|
||||
{
|
||||
db.DeviceGroupMembers.RemoveRange(
|
||||
await db.DeviceGroupMembers.Where(m => m.ClientId == clientId).ToListAsync(cancellationToken));
|
||||
db.ScriptTriggerRuns.RemoveRange(
|
||||
await db.ScriptTriggerRuns.Where(r => r.ClientId == clientId).ToListAsync(cancellationToken));
|
||||
|
||||
foreach (var trigger in await db.ScriptTriggers.Where(t => t.ClientId == clientId && t.Enabled).ToListAsync(cancellationToken))
|
||||
{
|
||||
trigger.Enabled = false;
|
||||
trigger.LastResult = "Disabled: its target device was removed.";
|
||||
}
|
||||
|
||||
var unstarted = await db.Jobs
|
||||
.Where(j => j.ClientId == clientId && (j.State == JobState.Queued || j.State == JobState.Dispatched))
|
||||
.ToListAsync(cancellationToken);
|
||||
foreach (var job in unstarted)
|
||||
{
|
||||
job.State = JobState.Cancelled;
|
||||
job.CompletedUtc = now;
|
||||
job.OutputSummary = "Cancelled: the device was removed.";
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>The newest removal of a client id, tracked; older ones are history only.</summary>
|
||||
public Task<DeviceRemovalEntity?> LatestAsync(string clientId, CancellationToken cancellationToken)
|
||||
=> db.DeviceRemovals
|
||||
.Where(r => r.ClientId == clientId)
|
||||
.OrderByDescending(r => r.RequestedUtc)
|
||||
.FirstOrDefaultAsync(cancellationToken);
|
||||
|
||||
/// <summary>
|
||||
/// Presence, inventory or a ready notice from a client id with no device
|
||||
/// row. An open removal notes the sighting (the scheduler sends on it), so
|
||||
/// the device row is never resurrected. An agent still talking after its
|
||||
/// removal closed as removed was not gone after all: back to waiting, with
|
||||
/// its command consumer restored when the next command is sent. A stopped
|
||||
/// removal stays stopped. True when a removal took the sighting.
|
||||
/// </summary>
|
||||
public async Task<bool> NoteSightingAsync(
|
||||
string? subjectClientId, string clientId, string? agentVersion, CancellationToken cancellationToken)
|
||||
{
|
||||
if (subjectClientId is not null && !string.Equals(subjectClientId, clientId, StringComparison.Ordinal))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
var row = await LatestAsync(clientId, cancellationToken);
|
||||
if (row is null || row.State == DeviceRemovalStates.Stopped)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
var now = DateTimeOffset.UtcNow;
|
||||
var reopened = row.State == DeviceRemovalStates.Removed;
|
||||
row.LastSeenUtc = now;
|
||||
row.AgentVersion = string.IsNullOrWhiteSpace(agentVersion) ? row.AgentVersion : agentVersion;
|
||||
if (reopened)
|
||||
{
|
||||
row.State = DeviceRemovalStates.Waiting;
|
||||
row.FinishedUtc = null;
|
||||
row.UpdatedUtc = now;
|
||||
row.Detail = "The device came back after it was marked as removed.";
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
await db.SaveChangesAsync(cancellationToken);
|
||||
}
|
||||
catch (DbUpdateConcurrencyException)
|
||||
{
|
||||
// The scheduler or an event got there first; the next sighting is seconds away.
|
||||
db.ChangeTracker.Clear();
|
||||
return true;
|
||||
}
|
||||
|
||||
if (reopened)
|
||||
{
|
||||
await AnnounceAsync(row, cancellationToken);
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Applies an agent's removal event. Ignored: unknown ids, another device's
|
||||
/// removal, states only the server sets, and anything after the removal
|
||||
/// closed. True when the row changed.
|
||||
/// </summary>
|
||||
public async Task<bool> ApplyEventAsync(
|
||||
string? subjectClientId, AgentRemovalEventPayload payload, CancellationToken cancellationToken)
|
||||
{
|
||||
if (!SubjectIdentifier.IsValid(payload.ClientId)
|
||||
|| (subjectClientId is not null && !string.Equals(subjectClientId, payload.ClientId, StringComparison.Ordinal))
|
||||
|| !AgentStates.Contains(payload.State))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
var openStates = DeviceRemovalStates.Open.ToList();
|
||||
for (var attempt = 0; attempt < 3; attempt++)
|
||||
{
|
||||
// A removal the agent started from a refused renewal may not know its id; its device names it.
|
||||
var row = payload.RemovalId != Guid.Empty
|
||||
? await db.DeviceRemovals.SingleOrDefaultAsync(r => r.Id == payload.RemovalId, cancellationToken)
|
||||
: await db.DeviceRemovals
|
||||
.Where(r => r.ClientId == payload.ClientId && openStates.Contains(r.State))
|
||||
.OrderByDescending(r => r.RequestedUtc)
|
||||
.FirstOrDefaultAsync(cancellationToken);
|
||||
if (row is null
|
||||
|| !string.Equals(row.ClientId, payload.ClientId, StringComparison.Ordinal)
|
||||
|| !DeviceRemovalStates.Open.Contains(row.State))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
var now = DateTimeOffset.UtcNow;
|
||||
row.State = payload.State;
|
||||
row.Reason = Truncate(payload.Reason, ReasonMaxLength);
|
||||
row.Detail = Truncate(payload.Detail, DetailMaxLength);
|
||||
row.UpdatedUtc = now;
|
||||
row.LastSeenUtc = now;
|
||||
|
||||
try
|
||||
{
|
||||
await db.SaveChangesAsync(cancellationToken);
|
||||
}
|
||||
catch (DbUpdateConcurrencyException)
|
||||
{
|
||||
db.ChangeTracker.Clear();
|
||||
continue;
|
||||
}
|
||||
|
||||
await AnnounceAsync(row, cancellationToken);
|
||||
return true;
|
||||
}
|
||||
|
||||
logger.LogWarning("Removal {RemovalId} kept changing; event {State} dropped", payload.RemovalId, payload.State);
|
||||
return false;
|
||||
}
|
||||
|
||||
/// <summary>Tells consoles a removal changed, with a line for the activity feed.</summary>
|
||||
public ValueTask AnnounceAsync(DeviceRemovalEntity row, CancellationToken cancellationToken)
|
||||
=> broadcaster.BroadcastActivityAsync(new ActivityEventPayload(
|
||||
ActivityKind, Describe(row), row.ClientId, null, DateTimeOffset.UtcNow), cancellationToken);
|
||||
|
||||
private static string Describe(DeviceRemovalEntity row)
|
||||
{
|
||||
var name = row.Hostname ?? row.ClientId;
|
||||
return row.State switch
|
||||
{
|
||||
DeviceRemovalStates.Sent => $"{name} remove command sent",
|
||||
DeviceRemovalStates.Uninstalling => $"{name} agent uninstalling",
|
||||
DeviceRemovalStates.Removed => $"{name} agent removed",
|
||||
DeviceRemovalStates.Failed => $"{name} agent could not uninstall ({row.Reason ?? "unknown"})",
|
||||
DeviceRemovalStates.Stopped => $"{name} removal stopped",
|
||||
_ => $"{name} removal waiting for the device",
|
||||
};
|
||||
}
|
||||
|
||||
private static string? Truncate(string? value, int length)
|
||||
=> value is not null && value.Length > length ? value[..length] : value;
|
||||
|
||||
/// <summary>Names of removed devices, for what outlives them (old runs, old upgrades); the newest removal wins.</summary>
|
||||
public static async Task<Dictionary<string, string>> RemovedHostnamesAsync(
|
||||
AppDbContext db, IEnumerable<string> clientIds, CancellationToken cancellationToken)
|
||||
{
|
||||
var ids = clientIds.Distinct(StringComparer.Ordinal).ToList();
|
||||
if (ids.Count == 0)
|
||||
{
|
||||
return new Dictionary<string, string>(StringComparer.Ordinal);
|
||||
}
|
||||
|
||||
var rows = await db.DeviceRemovals.AsNoTracking()
|
||||
.Where(r => ids.Contains(r.ClientId) && r.Hostname != null)
|
||||
.OrderBy(r => r.RequestedUtc)
|
||||
.Select(r => new { r.ClientId, r.Hostname })
|
||||
.ToListAsync(cancellationToken);
|
||||
|
||||
var names = new Dictionary<string, string>(StringComparer.Ordinal);
|
||||
foreach (var row in rows)
|
||||
{
|
||||
names[row.ClientId] = row.Hostname!;
|
||||
}
|
||||
|
||||
return names;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// True when the caller's roles may see this removal: an unscoped role sees
|
||||
/// all, a scoped role those of devices that were in one of its scope groups
|
||||
/// when they were removed.
|
||||
/// </summary>
|
||||
public static bool IsVisibleTo(DeviceRemovalEntity row, IReadOnlyList<Guid>? scopeGroupIds)
|
||||
{
|
||||
if (scopeGroupIds is null)
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
if (string.IsNullOrEmpty(row.GroupIdsJson))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
return (JsonDefaults.Deserialize<List<Guid>>(row.GroupIdsJson) ?? []).Intersect(scopeGroupIds).Any();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Removal events on the agent event subject (MessageTypes.AgentRemoval).</summary>
|
||||
public static class DeviceRemovalEvents
|
||||
{
|
||||
public static async Task HandleAsync(
|
||||
IServiceScopeFactory scopeFactory, string? subject, MessageEnvelope envelope, CancellationToken cancellationToken)
|
||||
{
|
||||
var payload = envelope.GetPayload<AgentRemovalEventPayload>();
|
||||
if (payload is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
using var scope = scopeFactory.CreateScope();
|
||||
await scope.ServiceProvider.GetRequiredService<DeviceRemovalService>()
|
||||
.ApplyEventAsync(AgentUpgradeEvents.SubjectClientId(subject), payload, cancellationToken);
|
||||
}
|
||||
|
||||
/// <summary>A sighting from a client id with no device row; <paramref name="agentVersion"/> comes with presence.</summary>
|
||||
public static Task NoteSightingAsync(
|
||||
IServiceProvider services, string? subject, string clientId, string? agentVersion, CancellationToken cancellationToken)
|
||||
=> services.GetRequiredService<DeviceRemovalService>()
|
||||
.NoteSightingAsync(AgentUpgradeEvents.SubjectClientId(subject), clientId, agentVersion, cancellationToken);
|
||||
}
|
||||
@@ -240,13 +240,22 @@ public static class AgentUpgradeEndpoints
|
||||
return Results.Ok(AgentUpgradeService.ToDto(row, hostname));
|
||||
}
|
||||
|
||||
private static Task<Dictionary<string, string>> HostnamesAsync(
|
||||
private static async Task<Dictionary<string, string>> HostnamesAsync(
|
||||
AppDbContext db, IEnumerable<string> clientIds, CancellationToken cancellationToken)
|
||||
{
|
||||
var ids = clientIds.Distinct(StringComparer.Ordinal).ToList();
|
||||
return db.Devices.AsNoTracking()
|
||||
var hostnames = await db.Devices.AsNoTracking()
|
||||
.Where(d => ids.Contains(d.ClientId))
|
||||
.ToDictionaryAsync(d => d.ClientId, d => d.Hostname, cancellationToken);
|
||||
|
||||
// Old upgrades of a removed device keep its name.
|
||||
foreach (var (clientId, hostname) in await Removals.DeviceRemovalService.RemovedHostnamesAsync(
|
||||
db, ids.Where(id => !hostnames.ContainsKey(id)), cancellationToken))
|
||||
{
|
||||
hostnames[clientId] = hostname;
|
||||
}
|
||||
|
||||
return hostnames;
|
||||
}
|
||||
|
||||
private static async Task<IResult> GetSettingsAsync(
|
||||
|
||||
@@ -237,6 +237,16 @@ public sealed class AgentUpgradeScheduler(
|
||||
continue;
|
||||
}
|
||||
|
||||
// Queued before the device moved past the minimum version, or by a
|
||||
// request that raced it: an older agent could not renew its credential.
|
||||
if (AgentUpgradeService.IsCredentialDowngrade(runningVersion(row.ClientId), row.TargetVersion))
|
||||
{
|
||||
upgrades.Finish(row, AgentUpgradeStates.Cancelled, AgentUpgradeService.SkipCredentialDowngrade,
|
||||
$"The device runs {runningVersion(row.ClientId)}, past the oldest version that renews its credential.", now);
|
||||
await TrySaveRowAsync(db, row, cancellationToken);
|
||||
continue;
|
||||
}
|
||||
|
||||
// Mark it sent first: a row is never handed to JetStream twice.
|
||||
row.State = AgentUpgradeStates.Sent;
|
||||
row.SentUtc = now;
|
||||
|
||||
@@ -26,6 +26,20 @@ public sealed class AgentUpgradeService(
|
||||
public const string SkipOrphaned = SkipReasons.Orphaned;
|
||||
public const string SkipNoPrevious = "no-previous";
|
||||
|
||||
/// <summary>
|
||||
/// The target is older than the first agent that renews its credential, and
|
||||
/// the device already holds a credential that expires: it would strand within a day.
|
||||
/// </summary>
|
||||
public const string SkipCredentialDowngrade = "credential-downgrade";
|
||||
|
||||
/// <summary>
|
||||
/// True when a device on <paramref name="running"/> must not go to
|
||||
/// <paramref name="target"/>: exactly the minimum version still renews.
|
||||
/// </summary>
|
||||
public static bool IsCredentialDowngrade(string? running, string? target)
|
||||
=> AgentVersions.Compare(running, DeviceRemovalDefaults.MinimumAgentVersion) >= 0
|
||||
&& AgentVersions.Compare(target, DeviceRemovalDefaults.MinimumAgentVersion) < 0;
|
||||
|
||||
/// <summary>Detail of a row the scheduler closed because presence already showed the target version.</summary>
|
||||
public const string PresenceConfirmed = "Confirmed by presence: the device reports the target version.";
|
||||
|
||||
@@ -181,6 +195,10 @@ public sealed class AgentUpgradeService(
|
||||
// Already there (or, without an explicit version, ahead of the newest held).
|
||||
skip = AgentUpgradeReasons.Already;
|
||||
}
|
||||
else if (IsCredentialDowngrade(device.AgentVersion, package.Version))
|
||||
{
|
||||
skip = SkipCredentialDowngrade;
|
||||
}
|
||||
}
|
||||
|
||||
if (skip is not null)
|
||||
@@ -231,6 +249,10 @@ public sealed class AgentUpgradeService(
|
||||
{
|
||||
skip = AgentUpgradeReasons.Already;
|
||||
}
|
||||
else if (IsCredentialDowngrade(device.AgentVersion, from))
|
||||
{
|
||||
skip = SkipCredentialDowngrade;
|
||||
}
|
||||
else if ((package = await catalog.FindAsync(from, types[device.ClientId]!, cancellationToken)) is null)
|
||||
{
|
||||
skip = SkipNoPrevious;
|
||||
|
||||
@@ -83,6 +83,11 @@ public static class ApiRoutes
|
||||
/// <summary>POST: cancel an upgrade that has not been sent yet.</summary>
|
||||
public const string AgentUpgradeCancel = "/api/agent-upgrades/{id}/cancel";
|
||||
public const string AgentUpgradeSettings = "/api/settings/agent-upgrades";
|
||||
|
||||
/// <summary>GET (?state=, ?clientId=, ?limit=): removals, newest first. POST: remove devices.</summary>
|
||||
public const string DeviceRemovals = "/api/device-removals";
|
||||
/// <summary>POST: stop trying to uninstall; the device stays blocked.</summary>
|
||||
public const string DeviceRemovalStop = "/api/device-removals/{id}/stop";
|
||||
/// <summary>GET (anonymous, ranges): one catalog package by SHA-256; what an upgrade command's DownloadUrl points at.</summary>
|
||||
public const string AgentPackageBySha = "/api/agent-packages/{sha256}";
|
||||
|
||||
@@ -99,6 +104,10 @@ public static class ApiRoutes
|
||||
public const string ExecutionOutputDownload = "/api/executions/{runId}/output/download";
|
||||
/// <summary>POST: a zip with one log per chosen job of the run (all when none are named).</summary>
|
||||
public const string ExecutionOutputArchive = "/api/executions/{runId}/output/archive";
|
||||
/// <summary>POST: cancels the run's jobs that have not been sent yet (in the caller's scope).</summary>
|
||||
public const string ExecutionCancelWaiting = "/api/executions/{runId}/cancel-waiting";
|
||||
/// <summary>Exit codes, grouped errors and completions over time of the run's jobs in the caller's scope.</summary>
|
||||
public const string ExecutionBreakdown = "/api/executions/{runId}/breakdown";
|
||||
|
||||
public const string Dashboard = "/api/dashboard";
|
||||
|
||||
@@ -128,6 +137,7 @@ public static class ApiRoutes
|
||||
public static string Device(string clientId) => $"/api/devices/{Uri.EscapeDataString(clientId)}";
|
||||
public static string DeviceInventory(string clientId) => $"/api/devices/{Uri.EscapeDataString(clientId)}/inventory";
|
||||
public static string DeviceHistory(string clientId) => $"/api/devices/{Uri.EscapeDataString(clientId)}/history";
|
||||
public static string DeviceRemovalStop(Guid id) => $"/api/device-removals/{id:D}/stop";
|
||||
public static string AgentVersion(string version) => $"/api/agent-versions/{Uri.EscapeDataString(version)}";
|
||||
public static string AgentUpgradeCancel(Guid id) => $"/api/agent-upgrades/{id:D}/cancel";
|
||||
public static string AgentPackageBySha(string sha256) => $"/api/agent-packages/{Uri.EscapeDataString(sha256)}";
|
||||
@@ -150,6 +160,8 @@ public static class ApiRoutes
|
||||
public static string Execution(Guid runId) => $"/api/executions/{runId:D}";
|
||||
public static string ExecutionOutputDownload(Guid runId) => $"/api/executions/{runId:D}/output/download";
|
||||
public static string ExecutionOutputArchive(Guid runId) => $"/api/executions/{runId:D}/output/archive";
|
||||
public static string ExecutionCancelWaiting(Guid runId) => $"/api/executions/{runId:D}/cancel-waiting";
|
||||
public static string ExecutionBreakdown(Guid runId) => $"/api/executions/{runId:D}/breakdown";
|
||||
public static string AdminBootstrapToken(Guid id) => $"/api/admin/bootstrap-tokens/{id:D}";
|
||||
public static string AdminRole(Guid id) => $"/api/admin/roles/{id:D}";
|
||||
public static string AdminUser(Guid id) => $"/api/admin/users/{id:D}";
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
namespace Shared.Contracts.Api;
|
||||
|
||||
/// <summary>
|
||||
/// POST /api/device-removals: remove these devices and every device in these
|
||||
/// groups (nested groups and folders included). Each leaves the system at once;
|
||||
/// its agent is uninstalled now if online, or when it next returns.
|
||||
/// </summary>
|
||||
public sealed record RemoveDevicesRequest(
|
||||
IReadOnlyList<string>? ClientIds = null,
|
||||
IReadOnlyList<Guid>? GroupIds = null,
|
||||
string? Reason = null);
|
||||
|
||||
/// <summary>
|
||||
/// One device's removal. <c>ShowInDeviceList</c> is true for the first
|
||||
/// <c>DeviceRemovalDefaults.VisibleInDeviceList</c> after <c>UpdatedUtc</c>; after
|
||||
/// that only the Removals filter shows it.
|
||||
/// </summary>
|
||||
public sealed record DeviceRemovalDto(
|
||||
Guid Id,
|
||||
string ClientId,
|
||||
string? Hostname,
|
||||
string? OsCaption,
|
||||
string? AgentVersion,
|
||||
string State,
|
||||
string? Reason,
|
||||
string? Detail,
|
||||
int Attempts,
|
||||
string? RequestedBy,
|
||||
string? Note,
|
||||
DateTimeOffset RequestedUtc,
|
||||
DateTimeOffset UpdatedUtc,
|
||||
DateTimeOffset? SentUtc,
|
||||
DateTimeOffset? LastSeenUtc,
|
||||
DateTimeOffset? FinishedUtc,
|
||||
bool ShowInDeviceList);
|
||||
|
||||
/// <summary>A device a removal request left out, and why.</summary>
|
||||
public sealed record DeviceRemovalSkipDto(string ClientId, string? Hostname, string Reason);
|
||||
|
||||
/// <summary>What a removal request did.</summary>
|
||||
public sealed record DeviceRemovalBatchDto(
|
||||
IReadOnlyList<DeviceRemovalDto> Removals,
|
||||
IReadOnlyList<DeviceRemovalSkipDto> Skipped);
|
||||
@@ -85,13 +85,17 @@ public sealed record ScriptTriggerDto(
|
||||
string? Name = null,
|
||||
Guid? LastRunId = null,
|
||||
IReadOnlyDictionary<string, System.Text.Json.JsonElement>? Arguments = null,
|
||||
string? CreatedBy = null);
|
||||
string? CreatedBy = null,
|
||||
int StaggerSeconds = 0,
|
||||
int StaggerBatchSize = 1);
|
||||
|
||||
/// <summary>
|
||||
/// POST /api/triggers and PUT /api/triggers/{id}. Target: exactly one of
|
||||
/// ClientId / GroupId, or neither for all devices. For TriggerType=Scheduled,
|
||||
/// supply CronExpression (recurring) or RunOnceUtc (one-shot) plus an optional
|
||||
/// TimeZoneId and Arguments.
|
||||
/// TimeZoneId and Arguments. <c>StaggerSeconds</c> and <c>StaggerBatchSize</c>
|
||||
/// spread each firing as on <see cref="CreateJobRequest"/>; event triggers fire
|
||||
/// on one device and ignore them.
|
||||
/// </summary>
|
||||
public sealed record SaveScriptTriggerRequest(
|
||||
Guid ScriptId,
|
||||
@@ -103,7 +107,9 @@ public sealed record SaveScriptTriggerRequest(
|
||||
DateTimeOffset? RunOnceUtc = null,
|
||||
string? TimeZoneId = null,
|
||||
IReadOnlyDictionary<string, System.Text.Json.JsonElement>? Arguments = null,
|
||||
string? Name = null);
|
||||
string? Name = null,
|
||||
int StaggerSeconds = 0,
|
||||
int StaggerBatchSize = 1);
|
||||
|
||||
/// <summary>POST /api/triggers/preview — upcoming fire times of a cadence, evaluated server-side.</summary>
|
||||
public sealed record SchedulePreviewRequest(string? CronExpression, string? TimeZoneId, int Count = 5);
|
||||
@@ -118,8 +124,10 @@ public sealed record TriggerRunResultDto(Guid? RunId, int Queued, string Message
|
||||
public sealed record SetTriggerEnabledRequest(bool Enabled);
|
||||
|
||||
/// <summary>
|
||||
/// Where every target of a run stands. Targets = jobs + skipped. Pending is
|
||||
/// queued or sent and not started; Online counts job targets online now.
|
||||
/// Where every target of a run stands. Targets = jobs + skipped. Waiting is
|
||||
/// held back by a stagger until its start time; Pending is queued or sent and
|
||||
/// not started; Failed includes the TimedOut ones; Online counts job targets
|
||||
/// online now.
|
||||
/// </summary>
|
||||
public sealed record RunCountsDto(
|
||||
int Targets,
|
||||
@@ -129,7 +137,9 @@ public sealed record RunCountsDto(
|
||||
int Failed,
|
||||
int Cancelled,
|
||||
int Skipped,
|
||||
int Online);
|
||||
int Online,
|
||||
int Waiting = 0,
|
||||
int TimedOut = 0);
|
||||
|
||||
/// <summary>One execution (run) of a script — the per-device jobs it created, summarized.</summary>
|
||||
public sealed record ExecutionSummaryDto(
|
||||
@@ -157,12 +167,17 @@ public sealed record ExecutionTargetDto(
|
||||
DateTimeOffset? StartedUtc,
|
||||
DateTimeOffset? CompletedUtc,
|
||||
bool Online = false,
|
||||
string? Summary = null);
|
||||
string? Summary = null,
|
||||
DateTimeOffset? NotBeforeUtc = null);
|
||||
|
||||
/// <summary>A targeted device that was given no job: orphaned, or offline with Only Devices Online Now.</summary>
|
||||
public sealed record ExecutionSkipDto(string ClientId, string Hostname, string Reason);
|
||||
|
||||
/// <summary>GET /api/executions/{runId} — a run and the targets it ran on.</summary>
|
||||
/// <summary>
|
||||
/// GET /api/executions/{runId} — a run and the targets it ran on.
|
||||
/// <c>ServerTimeUtc</c> lets a console count down to each target's
|
||||
/// <c>NotBeforeUtc</c> whatever its own clock says.
|
||||
/// </summary>
|
||||
public sealed record ExecutionDetailDto(
|
||||
Guid RunId,
|
||||
Guid ScriptId,
|
||||
@@ -174,7 +189,39 @@ public sealed record ExecutionDetailDto(
|
||||
string? TargetsLabel = null,
|
||||
bool OnlineOnly = false,
|
||||
IReadOnlyList<ExecutionSkipDto>? Skipped = null,
|
||||
RunCountsDto? Counts = null);
|
||||
RunCountsDto? Counts = null,
|
||||
DateTimeOffset? ServerTimeUtc = null);
|
||||
|
||||
/// <summary>POST /api/executions/{runId}/cancel-waiting — how many waiting jobs were cancelled.</summary>
|
||||
public sealed record CancelWaitingResultDto(int Cancelled);
|
||||
|
||||
/// <summary>How many of a run's finished jobs reported this exit code.</summary>
|
||||
public sealed record ExitCodeCountDto(int ExitCode, int Count);
|
||||
|
||||
/// <summary>A device one error group happened on.</summary>
|
||||
public sealed record RunErrorSampleDto(Guid JobId, string ClientId, string? Hostname);
|
||||
|
||||
/// <summary>
|
||||
/// Failed jobs that stopped on the same error: its exception type (when the
|
||||
/// agent reported one) and its first line, with position text and paths cut.
|
||||
/// <c>Samples</c> holds up to five devices, <c>JobIds</c> every job in the group.
|
||||
/// </summary>
|
||||
public sealed record RunErrorGroupDto(
|
||||
string? Type,
|
||||
string Message,
|
||||
int Count,
|
||||
IReadOnlyList<RunErrorSampleDto> Samples,
|
||||
IReadOnlyList<Guid> JobIds);
|
||||
|
||||
/// <summary>Jobs that finished during one bucket of a run.</summary>
|
||||
public sealed record RunCompletionBucketDto(DateTimeOffset StartUtc, int Succeeded, int Failed, int TimedOut, int Cancelled);
|
||||
|
||||
/// <summary>GET /api/executions/{runId}/breakdown — a run's exit codes, grouped errors and completions over time.</summary>
|
||||
public sealed record ExecutionBreakdownDto(
|
||||
IReadOnlyList<ExitCodeCountDto> ExitCodes,
|
||||
IReadOnlyList<RunErrorGroupDto> Errors,
|
||||
int BucketSeconds,
|
||||
IReadOnlyList<RunCompletionBucketDto> Completions);
|
||||
|
||||
/// <summary>Device detail for GET /api/devices/{clientId}.</summary>
|
||||
public sealed record DeviceDetailDto(
|
||||
@@ -296,7 +343,11 @@ public sealed record JobDto(
|
||||
/// (<c>GroupId</c>/<c>GroupIds</c>), deduplicated into one execution. A lone
|
||||
/// <c>ClientId</c> answers with the single <see cref="JobDto"/>; anything else
|
||||
/// with a <see cref="GroupRunDto"/>. <c>OnlineOnly</c> skips devices that are
|
||||
/// not online now instead of queueing for their return.
|
||||
/// not online now instead of queueing for their return. <c>StaggerSeconds</c>
|
||||
/// (0–86400; 0 = all at once) spreads the starts over that long, in batches of
|
||||
/// <c>StaggerBatchSize</c> devices (1–10000): the first batch starts now and
|
||||
/// the last when the spread ends. With OnlineOnly, a device is checked again
|
||||
/// when its batch comes up.
|
||||
/// </summary>
|
||||
public sealed record CreateJobRequest(
|
||||
string? ClientId,
|
||||
@@ -305,7 +356,9 @@ public sealed record CreateJobRequest(
|
||||
Guid? GroupId = null,
|
||||
IReadOnlyList<Guid>? GroupIds = null,
|
||||
IReadOnlyList<string>? ClientIds = null,
|
||||
bool OnlineOnly = false);
|
||||
bool OnlineOnly = false,
|
||||
int StaggerSeconds = 0,
|
||||
int StaggerBatchSize = 1);
|
||||
|
||||
/// <summary>
|
||||
/// POST /api/executions/{runId}/output/archive request body: the run's jobs to
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
namespace Shared.Contracts.Messages;
|
||||
|
||||
/// <summary>
|
||||
/// Removes the agent from the device; durable command on
|
||||
/// <c>agent.<clientId>.command.remove</c>. The agent acts only when
|
||||
/// <paramref name="EnrollmentId"/> is the enrollment it holds, so a stale
|
||||
/// command can never uninstall a device that was enrolled again since.
|
||||
/// </summary>
|
||||
public sealed record AgentRemoveCommandPayload(Guid RemovalId, Guid EnrollmentId);
|
||||
|
||||
/// <summary>
|
||||
/// Progress of a removal, published by the agent on
|
||||
/// <c>agent.<clientId>.event</c> with message type <see cref="MessageTypes.AgentRemoval"/>.
|
||||
/// <paramref name="State"/> is uninstalling or failed (<c>DeviceRemovalStates</c>):
|
||||
/// after a successful uninstall nobody is left to say so.
|
||||
/// </summary>
|
||||
public sealed record AgentRemovalEventPayload(
|
||||
Guid RemovalId,
|
||||
string ClientId,
|
||||
string State,
|
||||
string? Reason,
|
||||
string? Detail,
|
||||
DateTimeOffset TimestampUtc);
|
||||
@@ -11,11 +11,14 @@ public static class MessageTypes
|
||||
public const string AgentEvent = "agent.event";
|
||||
/// <summary>Upgrade progress or outcome on the agent event subject (<see cref="AgentUpgradeEventPayload"/>).</summary>
|
||||
public const string AgentUpgrade = "agent.upgrade";
|
||||
/// <summary>Removal progress on the agent event subject (<see cref="AgentRemovalEventPayload"/>).</summary>
|
||||
public const string AgentRemoval = "agent.removal";
|
||||
|
||||
public const string CommandInventory = "command.inventory";
|
||||
public const string CommandScript = "command.script";
|
||||
public const string CommandFileDownload = "command.file.download";
|
||||
public const string CommandAgentUpgrade = "command.agent.upgrade";
|
||||
public const string CommandAgentRemove = "command.agent.remove";
|
||||
public const string CommandShellOpen = "command.shell.open";
|
||||
public const string CommandShellInput = "command.shell.input";
|
||||
public const string CommandShellResize = "command.shell.resize";
|
||||
|
||||
@@ -127,14 +127,24 @@ public sealed record JobProgressPayload(
|
||||
int? SecondsRemaining,
|
||||
DateTimeOffset TimestampUtc);
|
||||
|
||||
/// <summary>Job state transition on <c>job.<jobId>.state</c>.</summary>
|
||||
/// <summary>
|
||||
/// Job state transition on <c>job.<jobId>.state</c>. A failed job says
|
||||
/// which error stopped it: the exception's full type name, the error record's
|
||||
/// fully qualified id and the message (older agents omit all three). The
|
||||
/// backend adds <paramref name="RunId"/> when relaying this to browsers, so a
|
||||
/// run page can pick out its own jobs; agents never send it.
|
||||
/// </summary>
|
||||
public sealed record JobStatePayload(
|
||||
Guid JobId,
|
||||
string ClientId,
|
||||
JobState State,
|
||||
int? ExitCode,
|
||||
DateTimeOffset TimestampUtc,
|
||||
string? Message);
|
||||
string? Message,
|
||||
string? ErrorType = null,
|
||||
string? ErrorId = null,
|
||||
string? ErrorMessage = null,
|
||||
Guid? RunId = null);
|
||||
|
||||
/// <summary>
|
||||
/// Opens a remote shell; sent on <c>agent.<clientId>.command.shell.open</c>.
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
namespace Shared.Contracts.Models;
|
||||
|
||||
/// <summary>
|
||||
/// States of one device's removal. The device row is gone from the first moment;
|
||||
/// this record stands in for it until the agent has gone, and stays afterwards
|
||||
/// as the history. A removal waits for as long as the device stays away.
|
||||
/// </summary>
|
||||
public static class DeviceRemovalStates
|
||||
{
|
||||
/// <summary>Waiting for the device to come back (or for the next send).</summary>
|
||||
public const string Waiting = "waiting";
|
||||
/// <summary>The remove command was sent; no answer yet.</summary>
|
||||
public const string Sent = "sent";
|
||||
/// <summary>The agent accepted and is uninstalling (it told us, or its credential renewal was refused).</summary>
|
||||
public const string Uninstalling = "uninstalling";
|
||||
/// <summary>The agent has gone: it uninstalled, or went silent for good after saying it was uninstalling.</summary>
|
||||
public const string Removed = "removed";
|
||||
/// <summary>The uninstall failed on the device; the agent says why and the server asks again when it next returns.</summary>
|
||||
public const string Failed = "failed";
|
||||
/// <summary>An operator stopped trying. The device stays blocked; its agent stays installed.</summary>
|
||||
public const string Stopped = "stopped";
|
||||
|
||||
/// <summary>Not finished: the scheduler still acts on these.</summary>
|
||||
public static readonly IReadOnlySet<string> Open = new HashSet<string>(StringComparer.Ordinal)
|
||||
{ Waiting, Sent, Uninstalling, Failed };
|
||||
}
|
||||
|
||||
/// <summary>Why a removal failed, or what the agent could not do.</summary>
|
||||
public static class DeviceRemovalReasons
|
||||
{
|
||||
/// <summary>The agent is too old to take a remove command: uninstall it by hand, or upgrade it first.</summary>
|
||||
public const string AgentTooOld = "agent-too-old";
|
||||
/// <summary>The agent cannot remove itself here (foreground or development run, container).</summary>
|
||||
public const string Unsupported = "unsupported";
|
||||
/// <summary>An agent upgrade is in progress; the server asks again.</summary>
|
||||
public const string Busy = "busy";
|
||||
/// <summary>Windows Installer failed to uninstall the agent.</summary>
|
||||
public const string Msi = "msi";
|
||||
/// <summary>Files or services could not be removed.</summary>
|
||||
public const string Uninstall = "uninstall";
|
||||
/// <summary>The command's enrollment is not the one this agent holds (a stale command); it was ignored.</summary>
|
||||
public const string StaleEnrollment = "stale-enrollment";
|
||||
}
|
||||
|
||||
/// <summary>Timings shared by the server's removal scheduler and the console.</summary>
|
||||
public static class DeviceRemovalDefaults
|
||||
{
|
||||
/// <summary>
|
||||
/// How long a removal shows in the default Devices list after its last
|
||||
/// change. After that only the Removals status filter shows it, so the main
|
||||
/// list stays uncluttered while nothing is lost.
|
||||
/// </summary>
|
||||
public static readonly TimeSpan VisibleInDeviceList = TimeSpan.FromDays(7);
|
||||
/// <summary>Silence after "uninstalling" that counts as gone: nobody is left to report success.</summary>
|
||||
public static readonly TimeSpan UninstallSilence = TimeSpan.FromMinutes(15);
|
||||
/// <summary>The first agent release that understands the remove command and renews its credential.</summary>
|
||||
public const string MinimumAgentVersion = "1.1.0";
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The agent's lifetime on its NATS credential. It renews over HTTPS at half
|
||||
/// life and at every start (older non-expiring credentials are replaced then),
|
||||
/// so a removed device is cut off within a day: renewal answers 410 Gone.
|
||||
/// </summary>
|
||||
public static class AgentCredentialDefaults
|
||||
{
|
||||
public static readonly TimeSpan Lifetime = TimeSpan.FromHours(24);
|
||||
/// <summary>Problem type of the 410 answer: the agent then uninstalls itself.</summary>
|
||||
public const string RemovedProblemType = "urn:problem:device-removed";
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A removal handed from the agent to the watchdog, in
|
||||
/// <c><data>/upgrade/removal.json</c>. The watchdog runs it from a detached
|
||||
/// copy of itself outside the install and data folders, since it wipes both.
|
||||
/// </summary>
|
||||
public sealed record WatchdogRemoval(
|
||||
Guid RemovalId,
|
||||
string InstallKind,
|
||||
string Phase,
|
||||
int Attempts,
|
||||
string? LastError,
|
||||
DateTimeOffset RequestedUtc,
|
||||
string? RunnerDirectory = null);
|
||||
|
||||
/// <summary>How a removal runs on the device, shared by the agent, the watchdog and the runner.</summary>
|
||||
public static class WatchdogRemovalFiles
|
||||
{
|
||||
/// <summary>Hidden watchdog verb of the detached runner: <c>remove-run <id> --install-dir … --data-dir … --runner-dir …</c>.</summary>
|
||||
public const string RunVerb = "remove-run";
|
||||
/// <summary>The same runner in the agent's binary, for a device where no watchdog can take the removal.</summary>
|
||||
public const string AgentRunVerb = "remove-self";
|
||||
/// <summary>In <see cref="WatchdogRemoval.RunnerDirectory"/>: held open (no sharing) by the runner for as long as it runs.</summary>
|
||||
public const string RunnerLock = "removal.lock";
|
||||
/// <summary>Attempts at uninstalling before the removal ends failed.</summary>
|
||||
public const int MaxAttempts = 3;
|
||||
public static readonly TimeSpan AttemptDelay = TimeSpan.FromMinutes(5);
|
||||
/// <summary>A removal the watchdog has not taken by then: the agent starts the runner itself.</summary>
|
||||
public static readonly TimeSpan PickupDeadline = TimeSpan.FromMinutes(2);
|
||||
}
|
||||
|
||||
/// <summary>Phases of <see cref="WatchdogRemoval"/>.</summary>
|
||||
public static class WatchdogRemovalPhases
|
||||
{
|
||||
public const string Requested = "requested";
|
||||
public const string Uninstalling = "uninstalling";
|
||||
public const string Failed = "failed";
|
||||
}
|
||||
@@ -29,6 +29,8 @@ public static class WatchdogFiles
|
||||
|
||||
/// <summary>MSI upgrades: the runner's journal (<see cref="WatchdogMsiJournal"/>), beside handover.json.</summary>
|
||||
public const string MsiJournal = "msi.json";
|
||||
/// <summary>A removal handed to the watchdog (<see cref="WatchdogRemoval"/>), beside handover.json.</summary>
|
||||
public const string Removal = "removal.json";
|
||||
/// <summary>MSI upgrades: <c>upgrade/msi/</c> holds the MSIs being installed and the runner's copy of the watchdog binary.</summary>
|
||||
public const string MsiFolder = "msi";
|
||||
/// <summary>
|
||||
@@ -38,6 +40,12 @@ public static class WatchdogFiles
|
||||
/// later upgrades brought; the newest 3 are kept.
|
||||
/// </summary>
|
||||
public const string KeptPackages = "packages";
|
||||
/// <summary>
|
||||
/// Beside <c>credentials.dat</c>, one thumbprint per line: the server CA
|
||||
/// certificates the agent itself added to the machine's trust store, the only
|
||||
/// ones a removal takes out (the same CA may be there for the server's own sake).
|
||||
/// </summary>
|
||||
public const string ManagedCa = "managed-ca.txt";
|
||||
public const int KeptPackagesMax = 3;
|
||||
/// <summary>The longest an MSI install (or the reinstall of the previous one) may run.</summary>
|
||||
public static readonly TimeSpan MsiInstallTimeout = TimeSpan.FromMinutes(30);
|
||||
|
||||
@@ -20,4 +20,5 @@ namespace Shared.Contracts.Serialization;
|
||||
[JsonSerializable(typeof(WatchdogJournal))]
|
||||
[JsonSerializable(typeof(WatchdogOutcome))]
|
||||
[JsonSerializable(typeof(WatchdogMsiJournal))]
|
||||
[JsonSerializable(typeof(WatchdogRemoval))]
|
||||
public sealed partial class WatchdogJsonContext : JsonSerializerContext;
|
||||
|
||||
@@ -15,6 +15,7 @@ public static class JetStreamDefaults
|
||||
"agent.*.command.inventory",
|
||||
"agent.*.command.file",
|
||||
"agent.*.command.upgrade",
|
||||
"agent.*.command.remove",
|
||||
];
|
||||
|
||||
public const string JobEventsStreamName = "JOB_EVENTS";
|
||||
|
||||
@@ -18,6 +18,7 @@ public static class NatsSubjects
|
||||
public static string AgentCommandScript(string clientId) => $"agent.{SubjectIdentifier.Require(clientId)}.command.script";
|
||||
public static string AgentCommandFile(string clientId) => $"agent.{SubjectIdentifier.Require(clientId)}.command.file";
|
||||
public static string AgentCommandUpgrade(string clientId) => $"agent.{SubjectIdentifier.Require(clientId)}.command.upgrade";
|
||||
public static string AgentCommandRemove(string clientId) => $"agent.{SubjectIdentifier.Require(clientId)}.command.remove";
|
||||
public static string AgentCommandShellOpen(string clientId) => $"agent.{SubjectIdentifier.Require(clientId)}.command.shell.open";
|
||||
public static string AgentCommandShellInput(string clientId) => $"agent.{SubjectIdentifier.Require(clientId)}.command.shell.input";
|
||||
public static string AgentCommandShellResize(string clientId) => $"agent.{SubjectIdentifier.Require(clientId)}.command.shell.resize";
|
||||
|
||||
@@ -10,6 +10,8 @@ import type {
|
||||
DashboardDto,
|
||||
DeviceDetailDto,
|
||||
DeviceDto,
|
||||
DeviceRemovalBatchDto,
|
||||
DeviceRemovalDto,
|
||||
DeviceGroupDto,
|
||||
ExecutionDetailDto,
|
||||
ExecutionSummaryDto,
|
||||
@@ -37,6 +39,7 @@ import type {
|
||||
SecretsProviderDto,
|
||||
SecretsProvidersResponse,
|
||||
SecretsTestResultDto,
|
||||
RemoveDevicesBody,
|
||||
ServerInfoDto,
|
||||
StartAgentUpgradeBody,
|
||||
TriggerRunResultDto
|
||||
@@ -206,6 +209,9 @@ const enc = encodeURIComponent;
|
||||
|
||||
const upgradeQuery = ({ clientId, open, limit }: AgentUpgradeQuery) => qs({ clientId, open: open ? 'true' : undefined, limit });
|
||||
|
||||
/** Every removal the page may show (the API's upper limit): the Devices list and its Removals filter share this read. */
|
||||
export const removalsKey = '/api/device-removals?limit=5000';
|
||||
|
||||
/** Every REST call the console makes. Reads also have SWR keys (see `keys`). */
|
||||
export const api = {
|
||||
// Devices
|
||||
@@ -227,6 +233,8 @@ export const api = {
|
||||
rollBackAgent: (body: RollBackAgentBody) => request<AgentUpgradeBatchDto>('/api/agent-upgrades/rollback', json('POST', body)),
|
||||
/** Only an upgrade still queued can be cancelled. */
|
||||
cancelAgentUpgrade: (id: string) => request<void>(`/api/agent-upgrades/${enc(id)}/cancel`, { method: 'POST' }),
|
||||
removeDevices: (body: RemoveDevicesBody) => request<DeviceRemovalBatchDto>('/api/device-removals', json('POST', body)),
|
||||
stopDeviceRemoval: (id: string) => request<DeviceRemovalDto>(`/api/device-removals/${enc(id)}/stop`, { method: 'POST' }),
|
||||
agentUpgradeSettings: () => request<AgentUpgradeSettingsDto>('/api/settings/agent-upgrades'),
|
||||
saveAgentUpgradeSettings: (body: AgentUpgradeSettingsDto) =>
|
||||
request<AgentUpgradeSettingsDto | undefined>('/api/settings/agent-upgrades', json('PUT', body)),
|
||||
@@ -257,6 +265,8 @@ export const api = {
|
||||
executions: (limit = 50) => request<ExecutionSummaryDto[]>(`/api/executions?limit=${limit}`),
|
||||
/** Runs a script on any mix of devices and groups as one execution. */
|
||||
runScript: (body: RunScriptBody) => request<GroupRunDto>('/api/jobs', json('POST', body)),
|
||||
/** Cancels the run's jobs that have not been sent yet. */
|
||||
cancelWaiting: (runId: string) => request<{ cancelled: number }>(`/api/executions/${runId}/cancel-waiting`, json('POST', {})),
|
||||
|
||||
// Groups
|
||||
groups: () => request<DeviceGroupDto[]>('/api/groups'),
|
||||
@@ -382,6 +392,7 @@ export const keys = {
|
||||
jobs: (params: { clientId?: string; scriptId?: string; limit?: number } = {}) => `/api/jobs${qs(params)}`,
|
||||
scriptExecutions: (scriptId: string) => `/api/scripts/${scriptId}/executions`,
|
||||
execution: (runId: string) => `/api/executions/${runId}`,
|
||||
executionBreakdown: (runId: string) => `/api/executions/${runId}/breakdown`,
|
||||
executions: (limit = 50) => `/api/executions?limit=${limit}`,
|
||||
roles: '/api/admin/roles',
|
||||
permissionCatalog: '/api/admin/permissions'
|
||||
|
||||
@@ -230,6 +230,48 @@ export interface AgentUpgradeBatchDto {
|
||||
skipped: AgentUpgradeSkipDto[];
|
||||
}
|
||||
|
||||
export type DeviceRemovalState = 'waiting' | 'sent' | 'uninstalling' | 'removed' | 'failed' | 'stopped';
|
||||
|
||||
/** One device's removal; it stands in for the device row, and stays as the history. */
|
||||
export interface DeviceRemovalDto {
|
||||
id: string;
|
||||
clientId: string;
|
||||
hostname?: string | null;
|
||||
osCaption?: string | null;
|
||||
agentVersion?: string | null;
|
||||
state: DeviceRemovalState;
|
||||
reason?: string | null;
|
||||
detail?: string | null;
|
||||
attempts: number;
|
||||
requestedBy?: string | null;
|
||||
note?: string | null;
|
||||
requestedUtc: string;
|
||||
updatedUtc: string;
|
||||
sentUtc?: string | null;
|
||||
lastSeenUtc?: string | null;
|
||||
finishedUtc?: string | null;
|
||||
showInDeviceList: boolean;
|
||||
}
|
||||
|
||||
export interface RemoveDevicesBody {
|
||||
clientIds?: string[];
|
||||
groupIds?: string[];
|
||||
/** The operator's note. */
|
||||
reason?: string | null;
|
||||
}
|
||||
|
||||
/** A device a request left out: unknown or out-of-scope. */
|
||||
export interface DeviceRemovalSkipDto {
|
||||
clientId: string;
|
||||
hostname?: string | null;
|
||||
reason: string;
|
||||
}
|
||||
|
||||
export interface DeviceRemovalBatchDto {
|
||||
removals: DeviceRemovalDto[];
|
||||
skipped: DeviceRemovalSkipDto[];
|
||||
}
|
||||
|
||||
export interface AgentUpgradeSettingsDto {
|
||||
automaticUpgrades: boolean;
|
||||
/** Queued online devices start downloading within this many minutes (1–30), in paced batches. */
|
||||
@@ -263,6 +305,9 @@ export interface ScriptTriggerDto {
|
||||
lastRunId?: string;
|
||||
arguments?: Record<string, unknown>;
|
||||
createdBy?: string;
|
||||
/** Spreads each firing's starts over this long; 0 starts them all at once. */
|
||||
staggerSeconds?: number;
|
||||
staggerBatchSize?: number;
|
||||
}
|
||||
|
||||
/** POST /api/triggers and PUT /api/triggers/{id}. Target: clientId OR groupId, or neither for all devices. */
|
||||
@@ -277,6 +322,8 @@ export interface SaveTriggerBody {
|
||||
runOnceUtc?: string;
|
||||
timeZoneId?: string;
|
||||
arguments?: Record<string, unknown>;
|
||||
staggerSeconds?: number;
|
||||
staggerBatchSize?: number;
|
||||
}
|
||||
|
||||
export interface SchedulePreviewDto {
|
||||
@@ -418,6 +465,10 @@ export interface RunScriptBody {
|
||||
arguments?: Record<string, unknown>;
|
||||
/** Skip devices that are not online now instead of queueing for their return. */
|
||||
onlineOnly?: boolean;
|
||||
/** Spreads the starts over this long (0–86400); 0 starts them all at once. */
|
||||
staggerSeconds?: number;
|
||||
/** Devices started together in each batch of a stagger. */
|
||||
staggerBatchSize?: number;
|
||||
}
|
||||
|
||||
export interface CountPoint {
|
||||
@@ -566,11 +617,15 @@ export interface RunCountsDto {
|
||||
pending: number;
|
||||
running: number;
|
||||
succeeded: number;
|
||||
/** Includes the timed out ones. */
|
||||
failed: number;
|
||||
cancelled: number;
|
||||
skipped: number;
|
||||
/** Job targets online now. */
|
||||
online: number;
|
||||
/** Held back by a stagger until their start. */
|
||||
waiting?: number;
|
||||
timedOut?: number;
|
||||
}
|
||||
|
||||
export type RunSource = 'manual' | 'schedule' | 'trigger';
|
||||
@@ -602,6 +657,8 @@ export interface ExecutionTargetDto {
|
||||
online?: boolean;
|
||||
/** Failure or completion summary from the agent. */
|
||||
summary?: string;
|
||||
/** When a stagger starts it, while it is queued. */
|
||||
notBeforeUtc?: string;
|
||||
}
|
||||
|
||||
/** A targeted device given no job. */
|
||||
@@ -623,6 +680,41 @@ export interface ExecutionDetailDto {
|
||||
onlineOnly?: boolean;
|
||||
skipped?: ExecutionSkipDto[];
|
||||
counts?: RunCountsDto;
|
||||
/** The server's clock at this answer, to count down whatever the browser's says. */
|
||||
serverTimeUtc?: string;
|
||||
}
|
||||
|
||||
export interface RunErrorSampleDto {
|
||||
jobId: string;
|
||||
clientId: string;
|
||||
hostname?: string;
|
||||
}
|
||||
|
||||
/** Failed jobs that stopped on the same error. */
|
||||
export interface RunErrorGroupDto {
|
||||
/** The exception's full type name, when the agent reported one. */
|
||||
type?: string;
|
||||
message: string;
|
||||
count: number;
|
||||
/** Up to five devices. */
|
||||
samples: RunErrorSampleDto[];
|
||||
jobIds: string[];
|
||||
}
|
||||
|
||||
export interface RunCompletionBucketDto {
|
||||
startUtc: string;
|
||||
succeeded: number;
|
||||
failed: number;
|
||||
timedOut: number;
|
||||
cancelled: number;
|
||||
}
|
||||
|
||||
/** GET /api/executions/{runId}/breakdown */
|
||||
export interface ExecutionBreakdownDto {
|
||||
exitCodes: { exitCode: number; count: number }[];
|
||||
errors: RunErrorGroupDto[];
|
||||
bucketSeconds: number;
|
||||
completions: RunCompletionBucketDto[];
|
||||
}
|
||||
|
||||
export interface RoleDto {
|
||||
@@ -728,6 +820,11 @@ export interface JobStatePayload {
|
||||
exitCode?: number;
|
||||
timestampUtc: string;
|
||||
message?: string;
|
||||
errorType?: string;
|
||||
errorId?: string;
|
||||
errorMessage?: string;
|
||||
/** The run this job belongs to, added by the server. */
|
||||
runId?: string;
|
||||
}
|
||||
|
||||
export interface ShellOutputPayload {
|
||||
|
||||
@@ -60,6 +60,8 @@ interface Props<T> {
|
||||
rowLabel?: (row: T) => string;
|
||||
/** Marks the row currently open in a side panel. */
|
||||
activeRowId?: string;
|
||||
/** Dims rows that only stand in for something (not selectable, fewer actions). */
|
||||
dimmedRow?: (row: T) => boolean;
|
||||
minWidth?: number;
|
||||
}
|
||||
|
||||
@@ -93,6 +95,7 @@ export default function DataTable<T>({
|
||||
dense = true,
|
||||
rowLabel,
|
||||
activeRowId,
|
||||
dimmedRow,
|
||||
minWidth = 720
|
||||
}: Props<T>) {
|
||||
const [sort, setSort] = useState(defaultSort);
|
||||
@@ -207,7 +210,7 @@ export default function DataTable<T>({
|
||||
hover
|
||||
selected={isChosen || activeRowId === id}
|
||||
onClick={onRowClick ? () => onRowClick(row) : undefined}
|
||||
sx={{ cursor: onRowClick ? 'pointer' : 'default' }}
|
||||
sx={{ cursor: onRowClick ? 'pointer' : 'default', ...(dimmedRow?.(row) && { '& > td': { opacity: 0.65 } }) }}
|
||||
>
|
||||
{selection && (
|
||||
<TableCell padding="checkbox" sx={{ pl: 2 }} onClick={(e) => e.stopPropagation()}>
|
||||
|
||||
@@ -15,6 +15,7 @@ const TONES: Record<string, { tone: Tone; label: string }> = {
|
||||
offline: { tone: 'secondary', label: 'Offline' },
|
||||
orphaned: { tone: 'error', label: 'Orphaned' },
|
||||
queued: { tone: 'secondary', label: 'Queued' },
|
||||
waiting: { tone: 'info', label: 'Waiting' },
|
||||
dispatched: { tone: 'info', label: 'Dispatched' },
|
||||
running: { tone: 'primary', label: 'Running' },
|
||||
succeeded: { tone: 'success', label: 'Succeeded' },
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import { KeyboardEvent } from 'react';
|
||||
|
||||
// material-ui
|
||||
import Box from '@mui/material/Box';
|
||||
import Stack from '@mui/material/Stack';
|
||||
@@ -19,12 +21,27 @@ interface Props {
|
||||
label: string;
|
||||
/** Unit for the tooltip ("jobs"). */
|
||||
unit?: string;
|
||||
/** Makes each row a toggle, e.g. to filter a table. */
|
||||
onSelect?: (item: CountPoint) => void;
|
||||
/** The label of the row that is on. */
|
||||
selected?: string | null;
|
||||
/** A hue per row, when rows are outcomes rather than one series. */
|
||||
colorOf?: (item: CountPoint) => string;
|
||||
}
|
||||
|
||||
// ==============================|| CHART - BAR LIST ||============================== //
|
||||
|
||||
/** Horizontal bars for ranked counts: label, proportional bar, value. */
|
||||
export default function BarList({ items, color = 'var(--chart-1)', emptyText = 'No data yet.', label, unit = 'jobs' }: Props) {
|
||||
export default function BarList({
|
||||
items,
|
||||
color = 'var(--chart-1)',
|
||||
emptyText = 'No data yet.',
|
||||
label,
|
||||
unit = 'jobs',
|
||||
onSelect,
|
||||
selected,
|
||||
colorOf
|
||||
}: Props) {
|
||||
if (items.length === 0) {
|
||||
return (
|
||||
<Typography variant="body2" color="text.secondary">
|
||||
@@ -40,16 +57,28 @@ export default function BarList({ items, color = 'var(--chart-1)', emptyText = '
|
||||
<Stack component="ul" aria-label={label} sx={{ listStyle: 'none', m: 0, p: 0, gap: 0.5 }}>
|
||||
{items.map((item) => {
|
||||
const share = total > 0 ? Math.round((item.count / total) * 100) : 0;
|
||||
const fill = colorOf?.(item) ?? color;
|
||||
return (
|
||||
<Tooltip
|
||||
key={item.label}
|
||||
placement="top"
|
||||
title={<ChartTooltip heading={item.label} rows={[{ label: `${unit} · ${share}%`, value: item.count, color }]} />}
|
||||
title={<ChartTooltip heading={item.label} rows={[{ label: `${unit} · ${share}%`, value: item.count, color: fill }]} />}
|
||||
>
|
||||
<Box
|
||||
component="li"
|
||||
tabIndex={0}
|
||||
aria-label={`${item.label}: ${item.count}`}
|
||||
{...(onSelect && {
|
||||
role: 'button',
|
||||
'aria-pressed': selected === item.label,
|
||||
onClick: () => onSelect(item),
|
||||
onKeyDown: (event: KeyboardEvent) => {
|
||||
if (event.key === 'Enter' || event.key === ' ') {
|
||||
event.preventDefault();
|
||||
onSelect(item);
|
||||
}
|
||||
}
|
||||
})}
|
||||
sx={(theme) => ({
|
||||
display: 'grid',
|
||||
gridTemplateColumns: 'minmax(0, 9rem) 1fr auto',
|
||||
@@ -59,6 +88,8 @@ export default function BarList({ items, color = 'var(--chart-1)', emptyText = '
|
||||
px: 0.5,
|
||||
borderRadius: 1,
|
||||
outline: 'none',
|
||||
cursor: onSelect ? 'pointer' : undefined,
|
||||
bgcolor: selected === item.label ? theme.vars.palette.action.selected : undefined,
|
||||
'& .bar': { transition: 'opacity 0.15s' },
|
||||
'&:hover, &:focus-visible': { bgcolor: theme.vars.palette.action.hover },
|
||||
'&:hover .bar, &:focus-visible .bar': { opacity: 0.8 }
|
||||
@@ -69,7 +100,7 @@ export default function BarList({ items, color = 'var(--chart-1)', emptyText = '
|
||||
</Typography>
|
||||
<Box component="svg" aria-hidden sx={{ width: 1, height: 8, display: 'block' }}>
|
||||
<Box component="rect" width="100%" height={8} rx={4} sx={(theme) => ({ fill: theme.vars.palette.divider })} />
|
||||
<rect className="bar" width={`${Math.max((item.count / max) * 100, 1)}%`} height={8} rx={4} style={{ fill: color }} />
|
||||
<rect className="bar" width={`${Math.max((item.count / max) * 100, 1)}%`} height={8} rx={4} style={{ fill }} />
|
||||
</Box>
|
||||
<Typography variant="caption" sx={{ fontWeight: 600, fontVariantNumeric: 'tabular-nums', textAlign: 'right' }}>
|
||||
{item.count.toLocaleString()}
|
||||
|
||||
@@ -0,0 +1,245 @@
|
||||
import { useEffect, useRef, useState } from 'react';
|
||||
|
||||
// material-ui
|
||||
import { useTheme } from '@mui/material/styles';
|
||||
import Box from '@mui/material/Box';
|
||||
import Stack from '@mui/material/Stack';
|
||||
import Tooltip from '@mui/material/Tooltip';
|
||||
import Typography from '@mui/material/Typography';
|
||||
|
||||
// project imports
|
||||
import ChartTooltip from './ChartTooltip';
|
||||
|
||||
export interface ColumnSeries {
|
||||
key: string;
|
||||
label: string;
|
||||
color: string;
|
||||
}
|
||||
|
||||
export interface ColumnPoint {
|
||||
/** Stable and unique among the points. */
|
||||
key: string;
|
||||
/** Under the column, where there is room. */
|
||||
label: string;
|
||||
/** Tooltip heading; the label when not given. */
|
||||
heading?: string;
|
||||
values: Record<string, number>;
|
||||
}
|
||||
|
||||
interface Props {
|
||||
points: ColumnPoint[];
|
||||
series: ColumnSeries[];
|
||||
height?: number;
|
||||
/** Accessible name ("Completions over time"). */
|
||||
label: string;
|
||||
emptyText?: string;
|
||||
/** Column heading of the screen-reader table ("Day"). */
|
||||
pointHeading?: string;
|
||||
/** Puts each series' total beside its name in the legend. */
|
||||
legendTotals?: boolean;
|
||||
}
|
||||
|
||||
const AXIS_HEIGHT = 20;
|
||||
const TOP_PAD = 14;
|
||||
/** Surface gap between stacked segments. */
|
||||
const GAP = 2;
|
||||
const RADIUS = 4;
|
||||
|
||||
/** Off-screen but still read by assistive technology. */
|
||||
const visuallyHidden = {
|
||||
position: 'absolute',
|
||||
width: '1px',
|
||||
height: '1px',
|
||||
overflow: 'hidden',
|
||||
clip: 'rect(0 0 0 0)',
|
||||
whiteSpace: 'nowrap',
|
||||
border: 0,
|
||||
p: 0,
|
||||
m: '-1px'
|
||||
} as const;
|
||||
|
||||
/** A rect with only its top corners rounded (the data end; the baseline end stays square). */
|
||||
function topRoundedRect(x: number, y: number, w: number, h: number, r: number) {
|
||||
const radius = Math.min(r, h, w / 2);
|
||||
return `M${x},${y + h} V${y + radius} Q${x},${y} ${x + radius},${y} H${x + w - radius} Q${x + w},${y} ${x + w},${y + radius} V${y + h} Z`;
|
||||
}
|
||||
|
||||
/** Tracks an element's content width. */
|
||||
function useWidth<T extends HTMLElement>() {
|
||||
const ref = useRef<T>(null);
|
||||
const [width, setWidth] = useState(0);
|
||||
useEffect(() => {
|
||||
const element = ref.current;
|
||||
if (!element) return;
|
||||
const observer = new ResizeObserver(([entry]) => setWidth(entry.contentRect.width));
|
||||
observer.observe(element);
|
||||
return () => observer.disconnect();
|
||||
}, []);
|
||||
return [ref, width] as const;
|
||||
}
|
||||
|
||||
// ==============================|| CHART - STACKED COLUMNS ||============================== //
|
||||
|
||||
/** Stacked columns over a sequence (days, minutes of a run), with a per-column hover readout. */
|
||||
export default function StackedColumns({
|
||||
points,
|
||||
series,
|
||||
height = 150,
|
||||
label,
|
||||
emptyText = 'No data yet.',
|
||||
pointHeading = 'Time',
|
||||
legendTotals
|
||||
}: Props) {
|
||||
const theme = useTheme();
|
||||
const [ref, width] = useWidth<HTMLDivElement>();
|
||||
const [hovered, setHovered] = useState<number | null>(null);
|
||||
|
||||
const sumOf = (point: ColumnPoint) => series.reduce((sum, s) => sum + (point.values[s.key] ?? 0), 0);
|
||||
const max = Math.max(1, ...points.map(sumOf));
|
||||
const total = points.reduce((sum, point) => sum + sumOf(point), 0);
|
||||
|
||||
const plotHeight = height - AXIS_HEIGHT - TOP_PAD;
|
||||
const baseline = TOP_PAD + plotHeight;
|
||||
const slot = points.length > 0 ? width / points.length : 0;
|
||||
const barWidth = Math.max(Math.min(slot * 0.62, 28), 2);
|
||||
const labelStep = Math.max(1, Math.ceil((points.length * 52) / Math.max(width, 1)));
|
||||
|
||||
return (
|
||||
<Stack sx={{ gap: 1 }}>
|
||||
<Box ref={ref} sx={{ width: 1, position: 'relative' }}>
|
||||
{total === 0 ? (
|
||||
<Typography variant="body2" color="text.secondary">
|
||||
{emptyText}
|
||||
</Typography>
|
||||
) : (
|
||||
width > 0 && (
|
||||
<svg width={width} height={height} role="img" aria-label={`${label}: ${total} in total`} style={{ display: 'block' }}>
|
||||
<title>{`${label}: ${total} in total`}</title>
|
||||
|
||||
{/* recessive grid: top line labelled with the scale maximum, and the baseline */}
|
||||
<line
|
||||
x1={0}
|
||||
x2={width}
|
||||
y1={TOP_PAD}
|
||||
y2={TOP_PAD}
|
||||
style={{ stroke: theme.vars.palette.divider, strokeDasharray: '3 3', strokeWidth: 1 }}
|
||||
/>
|
||||
<text x={width} y={TOP_PAD - 4} textAnchor="end" style={{ fill: theme.vars.palette.text.secondary, fontSize: 10 }}>
|
||||
{max}
|
||||
</text>
|
||||
<line x1={0} x2={width} y1={baseline} y2={baseline} style={{ stroke: theme.vars.palette.divider, strokeWidth: 1 }} />
|
||||
|
||||
{points.map((point, i) => {
|
||||
const x = i * slot + (slot - barWidth) / 2;
|
||||
let y = baseline;
|
||||
const stacked = series.filter((s) => (point.values[s.key] ?? 0) > 0);
|
||||
const segments = stacked.map((s, index) => {
|
||||
const h = (point.values[s.key] / max) * plotHeight;
|
||||
const top = y - h;
|
||||
// Every segment above the first gives up GAP px at its bottom.
|
||||
const drawn = index > 0 ? Math.max(h - GAP, 0.5) : Math.max(h, 1);
|
||||
y = top;
|
||||
return { ...s, top, drawn, last: index === stacked.length - 1 };
|
||||
});
|
||||
|
||||
const readout = (
|
||||
<ChartTooltip
|
||||
heading={point.heading ?? point.label}
|
||||
rows={[
|
||||
...series.map((s) => ({ label: s.label, value: point.values[s.key] ?? 0, color: s.color })),
|
||||
{ label: 'Total', value: sumOf(point) }
|
||||
]}
|
||||
/>
|
||||
);
|
||||
|
||||
return (
|
||||
<g key={point.key} style={{ opacity: hovered !== null && hovered !== i ? 0.6 : 1, transition: 'opacity 0.15s' }}>
|
||||
{segments.map((s) =>
|
||||
s.last ? (
|
||||
<path key={s.key} d={topRoundedRect(x, s.top, barWidth, s.drawn, RADIUS)} style={{ fill: s.color }} />
|
||||
) : (
|
||||
<rect key={s.key} x={x} y={s.top} width={barWidth} height={s.drawn} style={{ fill: s.color }} />
|
||||
)
|
||||
)}
|
||||
{i % labelStep === 0 && (
|
||||
<text
|
||||
// Labels at the edges hang inward instead of being cut off.
|
||||
x={i * slot + slot / 2 < 26 ? 0 : i * slot + slot / 2 > width - 26 ? width : i * slot + slot / 2}
|
||||
y={height - 4}
|
||||
textAnchor={i * slot + slot / 2 < 26 ? 'start' : i * slot + slot / 2 > width - 26 ? 'end' : 'middle'}
|
||||
style={{ fill: theme.vars.palette.text.secondary, fontSize: 10 }}
|
||||
>
|
||||
{point.label}
|
||||
</text>
|
||||
)}
|
||||
{/* the whole column is the hit target */}
|
||||
<Tooltip title={readout} placement="top" followCursor>
|
||||
<rect
|
||||
x={i * slot}
|
||||
y={0}
|
||||
width={slot}
|
||||
height={baseline}
|
||||
tabIndex={0}
|
||||
aria-label={`${point.heading ?? point.label}: ${series.map((s) => `${point.values[s.key] ?? 0} ${s.label.toLowerCase()}`).join(', ')}`}
|
||||
onMouseEnter={() => setHovered(i)}
|
||||
onMouseLeave={() => setHovered(null)}
|
||||
onFocus={() => setHovered(i)}
|
||||
onBlur={() => setHovered(null)}
|
||||
style={{ fill: 'transparent', outline: 'none' }}
|
||||
/>
|
||||
</Tooltip>
|
||||
</g>
|
||||
);
|
||||
})}
|
||||
</svg>
|
||||
)
|
||||
)}
|
||||
</Box>
|
||||
|
||||
{total > 0 && (
|
||||
<>
|
||||
<Stack direction="row" sx={{ gap: 2, flexWrap: 'wrap' }}>
|
||||
{series.map((s) => (
|
||||
<Stack key={s.key} direction="row" sx={{ alignItems: 'center', gap: 0.75 }}>
|
||||
<Box sx={{ width: 10, height: 10, borderRadius: '3px', bgcolor: s.color }} />
|
||||
<Typography variant="caption" color="text.secondary">
|
||||
{s.label}
|
||||
</Typography>
|
||||
{legendTotals && (
|
||||
<Typography variant="caption" sx={{ fontWeight: 600, fontVariantNumeric: 'tabular-nums' }}>
|
||||
{points.reduce((sum, point) => sum + (point.values[s.key] ?? 0), 0).toLocaleString()}
|
||||
</Typography>
|
||||
)}
|
||||
</Stack>
|
||||
))}
|
||||
</Stack>
|
||||
|
||||
{/* the same numbers as a table, for screen readers */}
|
||||
<Box component="table" sx={visuallyHidden}>
|
||||
<caption>{label}</caption>
|
||||
<thead>
|
||||
<tr>
|
||||
<th scope="col">{pointHeading}</th>
|
||||
{series.map((s) => (
|
||||
<th key={s.key} scope="col">
|
||||
{s.label}
|
||||
</th>
|
||||
))}
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{points.map((point) => (
|
||||
<tr key={point.key}>
|
||||
<th scope="row">{point.heading ?? point.label}</th>
|
||||
{series.map((s) => (
|
||||
<td key={s.key}>{point.values[s.key] ?? 0}</td>
|
||||
))}
|
||||
</tr>
|
||||
))}
|
||||
</tbody>
|
||||
</Box>
|
||||
</>
|
||||
)}
|
||||
</Stack>
|
||||
);
|
||||
}
|
||||
@@ -1,14 +1,8 @@
|
||||
import { useEffect, useRef, useState } from 'react';
|
||||
|
||||
// material-ui
|
||||
import { useTheme } from '@mui/material/styles';
|
||||
import Box from '@mui/material/Box';
|
||||
import Stack from '@mui/material/Stack';
|
||||
import Tooltip from '@mui/material/Tooltip';
|
||||
import Typography from '@mui/material/Typography';
|
||||
|
||||
// project imports
|
||||
import ChartTooltip from './ChartTooltip';
|
||||
import StackedColumns from './StackedColumns';
|
||||
|
||||
// types
|
||||
import type { DayPoint } from 'api/types';
|
||||
@@ -21,47 +15,6 @@ interface Props {
|
||||
emptyText?: string;
|
||||
}
|
||||
|
||||
type SeriesKey = 'succeeded' | 'failed' | 'other';
|
||||
|
||||
const AXIS_HEIGHT = 20;
|
||||
const TOP_PAD = 14;
|
||||
/** Surface gap between stacked segments. */
|
||||
const GAP = 2;
|
||||
const RADIUS = 4;
|
||||
|
||||
/** Off-screen but still read by assistive technology. */
|
||||
const visuallyHidden = {
|
||||
position: 'absolute',
|
||||
width: '1px',
|
||||
height: '1px',
|
||||
overflow: 'hidden',
|
||||
clip: 'rect(0 0 0 0)',
|
||||
whiteSpace: 'nowrap',
|
||||
border: 0,
|
||||
p: 0,
|
||||
m: '-1px'
|
||||
} as const;
|
||||
|
||||
/** A rect with only its top corners rounded (the data end; the baseline end stays square). */
|
||||
function topRoundedRect(x: number, y: number, w: number, h: number, r: number) {
|
||||
const radius = Math.min(r, h, w / 2);
|
||||
return `M${x},${y + h} V${y + radius} Q${x},${y} ${x + radius},${y} H${x + w - radius} Q${x + w},${y} ${x + w},${y + radius} V${y + h} Z`;
|
||||
}
|
||||
|
||||
/** Tracks an element's content width. */
|
||||
function useWidth<T extends HTMLElement>() {
|
||||
const ref = useRef<T>(null);
|
||||
const [width, setWidth] = useState(0);
|
||||
useEffect(() => {
|
||||
const element = ref.current;
|
||||
if (!element) return;
|
||||
const observer = new ResizeObserver(([entry]) => setWidth(entry.contentRect.width));
|
||||
observer.observe(element);
|
||||
return () => observer.disconnect();
|
||||
}, []);
|
||||
return [ref, width] as const;
|
||||
}
|
||||
|
||||
// ==============================|| CHART - EXECUTIONS TIMELINE ||============================== //
|
||||
|
||||
/** Stacked columns of executions per day (succeeded / failed / other) with a per-day hover readout. */
|
||||
@@ -72,156 +25,24 @@ export default function TimelineChart({
|
||||
emptyText = 'No executions in the last 14 days.'
|
||||
}: Props) {
|
||||
const theme = useTheme();
|
||||
const [ref, width] = useWidth<HTMLDivElement>();
|
||||
const [hovered, setHovered] = useState<number | null>(null);
|
||||
|
||||
// Status colors: these segments are outcomes, so they wear the status palette.
|
||||
const series: { key: SeriesKey; label: string; color: string }[] = [
|
||||
{ key: 'succeeded', label: 'Succeeded', color: theme.vars.palette.success.main },
|
||||
{ key: 'failed', label: 'Failed', color: theme.vars.palette.error.main },
|
||||
{ key: 'other', label: 'Other', color: 'var(--chart-6)' }
|
||||
];
|
||||
|
||||
const sumOf = (d: DayPoint) => d.succeeded + d.failed + d.other;
|
||||
const max = Math.max(1, ...days.map(sumOf));
|
||||
const total = days.reduce((sum, d) => sum + sumOf(d), 0);
|
||||
|
||||
const plotHeight = height - AXIS_HEIGHT - TOP_PAD;
|
||||
const baseline = TOP_PAD + plotHeight;
|
||||
const slot = days.length > 0 ? width / days.length : 0;
|
||||
const barWidth = Math.max(Math.min(slot * 0.62, 28), 2);
|
||||
const labelStep = Math.max(1, Math.ceil((days.length * 52) / Math.max(width, 1)));
|
||||
|
||||
return (
|
||||
<Stack sx={{ gap: 1 }}>
|
||||
<Box ref={ref} sx={{ width: 1, position: 'relative' }}>
|
||||
{total === 0 ? (
|
||||
<Typography variant="body2" color="text.secondary">
|
||||
{emptyText}
|
||||
</Typography>
|
||||
) : (
|
||||
width > 0 && (
|
||||
<svg width={width} height={height} role="img" aria-label={`${label}: ${total} in total`} style={{ display: 'block' }}>
|
||||
<title>{`${label}: ${total} in total`}</title>
|
||||
|
||||
{/* recessive grid: top line labelled with the scale maximum, and the baseline */}
|
||||
<line
|
||||
x1={0}
|
||||
x2={width}
|
||||
y1={TOP_PAD}
|
||||
y2={TOP_PAD}
|
||||
style={{ stroke: theme.vars.palette.divider, strokeDasharray: '3 3', strokeWidth: 1 }}
|
||||
/>
|
||||
<text x={width} y={TOP_PAD - 4} textAnchor="end" style={{ fill: theme.vars.palette.text.secondary, fontSize: 10 }}>
|
||||
{max}
|
||||
</text>
|
||||
<line x1={0} x2={width} y1={baseline} y2={baseline} style={{ stroke: theme.vars.palette.divider, strokeWidth: 1 }} />
|
||||
|
||||
{days.map((day, i) => {
|
||||
const x = i * slot + (slot - barWidth) / 2;
|
||||
let y = baseline;
|
||||
const stacked = series.filter((s) => day[s.key] > 0);
|
||||
const segments = stacked.map((s, index) => {
|
||||
const h = (day[s.key] / max) * plotHeight;
|
||||
const top = y - h;
|
||||
// Every segment above the first gives up GAP px at its bottom.
|
||||
const drawn = index > 0 ? Math.max(h - GAP, 0.5) : Math.max(h, 1);
|
||||
y = top;
|
||||
return { ...s, top, drawn, last: index === stacked.length - 1 };
|
||||
});
|
||||
|
||||
const readout = (
|
||||
<ChartTooltip
|
||||
heading={day.date}
|
||||
rows={[
|
||||
...series.map((s) => ({ label: s.label, value: day[s.key], color: s.color })),
|
||||
{ label: 'Total', value: sumOf(day) }
|
||||
]}
|
||||
/>
|
||||
);
|
||||
|
||||
return (
|
||||
<g key={day.date} style={{ opacity: hovered !== null && hovered !== i ? 0.6 : 1, transition: 'opacity 0.15s' }}>
|
||||
{segments.map((s) =>
|
||||
s.last ? (
|
||||
<path key={s.key} d={topRoundedRect(x, s.top, barWidth, s.drawn, RADIUS)} style={{ fill: s.color }} />
|
||||
) : (
|
||||
<rect key={s.key} x={x} y={s.top} width={barWidth} height={s.drawn} style={{ fill: s.color }} />
|
||||
)
|
||||
)}
|
||||
{i % labelStep === 0 && (
|
||||
<text
|
||||
x={i * slot + slot / 2}
|
||||
y={height - 4}
|
||||
textAnchor="middle"
|
||||
style={{ fill: theme.vars.palette.text.secondary, fontSize: 10 }}
|
||||
>
|
||||
{day.date}
|
||||
</text>
|
||||
)}
|
||||
{/* the whole column is the hit target */}
|
||||
<Tooltip title={readout} placement="top" followCursor>
|
||||
<rect
|
||||
x={i * slot}
|
||||
y={0}
|
||||
width={slot}
|
||||
height={baseline}
|
||||
tabIndex={0}
|
||||
aria-label={`${day.date}: ${day.succeeded} succeeded, ${day.failed} failed, ${day.other} other`}
|
||||
onMouseEnter={() => setHovered(i)}
|
||||
onMouseLeave={() => setHovered(null)}
|
||||
onFocus={() => setHovered(i)}
|
||||
onBlur={() => setHovered(null)}
|
||||
style={{ fill: 'transparent', outline: 'none' }}
|
||||
/>
|
||||
</Tooltip>
|
||||
</g>
|
||||
);
|
||||
})}
|
||||
</svg>
|
||||
)
|
||||
)}
|
||||
</Box>
|
||||
|
||||
{total > 0 && (
|
||||
<>
|
||||
<Stack direction="row" sx={{ gap: 2, flexWrap: 'wrap' }}>
|
||||
{series.map((s) => (
|
||||
<Stack key={s.key} direction="row" sx={{ alignItems: 'center', gap: 0.75 }}>
|
||||
<Box sx={{ width: 10, height: 10, borderRadius: '3px', bgcolor: s.color }} />
|
||||
<Typography variant="caption" color="text.secondary">
|
||||
{s.label}
|
||||
</Typography>
|
||||
</Stack>
|
||||
))}
|
||||
</Stack>
|
||||
|
||||
{/* the same numbers as a table, for screen readers */}
|
||||
<Box component="table" sx={visuallyHidden}>
|
||||
<caption>{label}</caption>
|
||||
<thead>
|
||||
<tr>
|
||||
<th scope="col">Day</th>
|
||||
{series.map((s) => (
|
||||
<th key={s.key} scope="col">
|
||||
{s.label}
|
||||
</th>
|
||||
))}
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{days.map((day) => (
|
||||
<tr key={day.date}>
|
||||
<th scope="row">{day.date}</th>
|
||||
{series.map((s) => (
|
||||
<td key={s.key}>{day[s.key]}</td>
|
||||
))}
|
||||
</tr>
|
||||
))}
|
||||
</tbody>
|
||||
</Box>
|
||||
</>
|
||||
)}
|
||||
</Stack>
|
||||
<StackedColumns
|
||||
// Status colors: these segments are outcomes, so they wear the status palette.
|
||||
series={[
|
||||
{ key: 'succeeded', label: 'Succeeded', color: theme.vars.palette.success.main },
|
||||
{ key: 'failed', label: 'Failed', color: theme.vars.palette.error.main },
|
||||
{ key: 'other', label: 'Other', color: 'var(--chart-6)' }
|
||||
]}
|
||||
points={days.map((day) => ({
|
||||
key: day.date,
|
||||
label: day.date,
|
||||
values: { succeeded: day.succeeded, failed: day.failed, other: day.other }
|
||||
}))}
|
||||
height={height}
|
||||
label={label}
|
||||
emptyText={emptyText}
|
||||
pointHeading="Day"
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -19,7 +19,6 @@ import useApi from 'hooks/useApi';
|
||||
import useLiveRefresh from 'hooks/useLiveRefresh';
|
||||
import { useGate } from 'hooks/useSession';
|
||||
import ActivityFeed from 'sections/dashboard/ActivityFeed';
|
||||
import LiveDeployment from 'sections/dashboard/LiveDeployment';
|
||||
import RecentRuns from 'sections/dashboard/RecentRuns';
|
||||
import { errorMessage, keys } from 'api/client';
|
||||
import type { CountPoint, DashboardDto } from 'api/types';
|
||||
@@ -111,8 +110,6 @@ export default function Dashboard() {
|
||||
|
||||
{error && !data && <Alert severity="error">{errorMessage(error, 'The dashboard could not be loaded.')}</Alert>}
|
||||
|
||||
<LiveDeployment onFinished={() => void mutate()} />
|
||||
|
||||
<Grid container spacing={2}>
|
||||
<Grid size={{ xs: 12, sm: 6, lg: 'grow' }}>
|
||||
<KpiCard
|
||||
|
||||
@@ -71,7 +71,7 @@ export default function DeviceDetailPage() {
|
||||
() => mutate(),
|
||||
(event) => event.clientId === clientId
|
||||
);
|
||||
const actions = useDeviceActions({ onDone: () => void mutate() });
|
||||
const actions = useDeviceActions({ onDone: () => void mutate(), onRemoved: () => navigate('/devices') });
|
||||
|
||||
const tab = (TABS.find((t) => t.id === params.get('tab'))?.id ?? 'overview') as TabId;
|
||||
const device = detail?.device;
|
||||
|
||||
@@ -5,6 +5,10 @@ import { Link as RouterLink, useNavigate, useSearchParams } from 'react-router-d
|
||||
import Box from '@mui/material/Box';
|
||||
import Button from '@mui/material/Button';
|
||||
import Chip from '@mui/material/Chip';
|
||||
import Dialog from '@mui/material/Dialog';
|
||||
import DialogActions from '@mui/material/DialogActions';
|
||||
import DialogContent from '@mui/material/DialogContent';
|
||||
import DialogTitle from '@mui/material/DialogTitle';
|
||||
import FormControlLabel from '@mui/material/FormControlLabel';
|
||||
import Grid from '@mui/material/Grid';
|
||||
import InputAdornment from '@mui/material/InputAdornment';
|
||||
@@ -29,7 +33,11 @@ import OsIcon, { isWinPe, platformOf, WinPeChip } from 'components/OsIcon';
|
||||
import PageHeader from 'components/PageHeader';
|
||||
import { DeviceStatusChip } from 'components/StatusChip';
|
||||
import AddDeviceDialog from 'sections/devices/AddDeviceDialog';
|
||||
import DeviceHistory from 'sections/devices/DeviceHistory';
|
||||
import { isOpenRemoval, isRemovalActivity } from 'sections/devices/removalLabels';
|
||||
import RemovalStateChip from 'sections/devices/RemovalStateChip';
|
||||
import useDeviceActions from 'sections/devices/useDeviceActions';
|
||||
import useDeviceRemovals from 'sections/devices/useDeviceRemovals';
|
||||
import GroupTreePanel, { ALL_NODE, UNGROUPED_NODE } from 'sections/groups/GroupTreePanel';
|
||||
import useGroupActions from 'sections/groups/useGroupActions';
|
||||
import { isUpgradeActivity, newestFor, versionSortKey } from 'sections/upgrades/agentVersions';
|
||||
@@ -38,10 +46,10 @@ import useApi from 'hooks/useApi';
|
||||
import useAutomation from 'hooks/useAutomation';
|
||||
import useLiveRefresh from 'hooks/useLiveRefresh';
|
||||
import useSession, { useGate } from 'hooks/useSession';
|
||||
import { keys } from 'api/client';
|
||||
import { keys, removalsKey } from 'api/client';
|
||||
import { plural } from 'utils/format';
|
||||
import { ancestors, devicesInGroup, ungroupedDevices } from 'utils/groupTree';
|
||||
import type { AgentUpgradeDto, AgentVersionDto, DeviceDto, DeviceGroupDto } from 'api/types';
|
||||
import type { AgentUpgradeDto, AgentVersionDto, DeviceDto, DeviceGroupDto, DeviceRemovalDto } from 'api/types';
|
||||
|
||||
// assets
|
||||
import CalendarOutlined from '@ant-design/icons/CalendarOutlined';
|
||||
@@ -51,13 +59,33 @@ import PlayCircleOutlined from '@ant-design/icons/PlayCircleOutlined';
|
||||
import PlusOutlined from '@ant-design/icons/PlusOutlined';
|
||||
import SearchOutlined from '@ant-design/icons/SearchOutlined';
|
||||
|
||||
type StatusFilter = 'all' | 'online' | 'stale' | 'offline' | 'orphaned';
|
||||
type StatusFilter = 'all' | 'online' | 'stale' | 'offline' | 'orphaned' | 'removals';
|
||||
|
||||
/** A row of the table: a device, or a removal standing in for one. */
|
||||
type ListRow = DeviceDto & { removal?: DeviceRemovalDto };
|
||||
|
||||
const removalRow = (removal: DeviceRemovalDto): ListRow => ({
|
||||
clientId: removal.clientId,
|
||||
hostname: removal.hostname ?? removal.clientId,
|
||||
osCaption: removal.osCaption ?? undefined,
|
||||
agentVersion: removal.agentVersion ?? undefined,
|
||||
ipAddresses: [],
|
||||
status: 'offline',
|
||||
isOrphaned: false,
|
||||
lastSeenUtc: removal.lastSeenUtc ?? undefined,
|
||||
enrolledUtc: removal.requestedUtc,
|
||||
removal
|
||||
});
|
||||
|
||||
const listRowId = (row: ListRow) => (row.removal ? `removal:${row.removal.id}` : row.clientId);
|
||||
|
||||
const KIND_LABEL = { static: 'Static Group', dynamic: 'Dynamic Group', folder: 'Folder' } as const;
|
||||
|
||||
function matches(device: DeviceDto, query: string): boolean {
|
||||
function matches(device: ListRow, query: string): boolean {
|
||||
return [
|
||||
device.hostname,
|
||||
device.removal?.requestedBy,
|
||||
device.removal?.note,
|
||||
device.clientId,
|
||||
device.manufacturer,
|
||||
device.model,
|
||||
@@ -92,12 +120,14 @@ export default function DevicesPage() {
|
||||
refreshInterval: 60_000
|
||||
});
|
||||
const openByDevice = useMemo(() => new Map((openUpgrades ?? []).map((u) => [u.clientId, u])), [openUpgrades]);
|
||||
const { data: removals, mutate: refreshRemovals } = useApi<DeviceRemovalDto[]>(removalsKey, { refreshInterval: 60_000 });
|
||||
// Presence and inventory changes arrive as activity carrying a client id.
|
||||
useLiveRefresh(
|
||||
() => Promise.all([refreshDevices(), refreshGroups()]),
|
||||
(event) => Boolean(event.clientId)
|
||||
);
|
||||
useLiveRefresh(() => refreshUpgrades(), isUpgradeActivity);
|
||||
useLiveRefresh(() => Promise.all([refreshRemovals(), refreshDevices(), refreshGroups()]), isRemovalActivity);
|
||||
|
||||
const selectedId = params.get('group') ?? ALL_NODE;
|
||||
const search = params.get('q') ?? '';
|
||||
@@ -120,6 +150,7 @@ export default function DevicesPage() {
|
||||
const [menu, setMenu] = useState<{ anchor: HTMLElement; devices: DeviceDto[] } | null>(null);
|
||||
const [groupMenu, setGroupMenu] = useState<HTMLElement | null>(null);
|
||||
const [addOpen, setAddOpen] = useState(false);
|
||||
const [historyOf, setHistoryOf] = useState<ListRow | null>(null);
|
||||
|
||||
const selectedGroup = groups?.find((g) => g.id === selectedId) ?? null;
|
||||
const groupActions = useGroupActions({ onDeleted: () => setParam('group', null) });
|
||||
@@ -127,6 +158,9 @@ export default function DevicesPage() {
|
||||
staticGroup: selectedGroup?.kind === 'static' ? selectedGroup : null,
|
||||
onDone: () => setChosen(new Set())
|
||||
});
|
||||
const removalActions = useDeviceRemovals();
|
||||
const canRemove = can('devices.remove');
|
||||
const showingRemovals = status === 'removals';
|
||||
|
||||
const inScope = useMemo(() => {
|
||||
const list = devices ?? [];
|
||||
@@ -135,33 +169,54 @@ export default function DevicesPage() {
|
||||
return devicesInGroup(list, groups ?? [], selectedId, nested);
|
||||
}, [devices, groups, selectedId, nested]);
|
||||
|
||||
const rows = useMemo(() => {
|
||||
const rows = useMemo<ListRow[]>(() => {
|
||||
const query = search.trim().toLowerCase();
|
||||
return inScope.filter((d) => {
|
||||
if (query && !matches(d, query)) return false;
|
||||
if (status === 'orphaned' ? !d.isOrphaned : status !== 'all' && (d.status !== status || d.isOrphaned)) return false;
|
||||
if (platform !== 'all' && platformOf(d.osCaption) !== platform) return false;
|
||||
return true;
|
||||
});
|
||||
}, [inScope, search, status, platform]);
|
||||
const passes = (d: ListRow) => (!query || matches(d, query)) && (platform === 'all' || platformOf(d.osCaption) === platform);
|
||||
if (showingRemovals) return (removals ?? []).map(removalRow).filter(passes);
|
||||
|
||||
const chosenRows = rows.filter((d) => chosen.has(d.clientId));
|
||||
const listed = inScope.filter((d) => {
|
||||
if (status === 'orphaned' ? !d.isOrphaned : status !== 'all' && (d.status !== status || d.isOrphaned)) return false;
|
||||
return passes(d);
|
||||
});
|
||||
// Recent removals sit in the plain list; a removed device has no groups, and one enrolled again has its own row.
|
||||
if (status !== 'all' || selectedId !== ALL_NODE) return listed;
|
||||
const enrolled = new Set((devices ?? []).map((d) => d.clientId));
|
||||
return [
|
||||
...listed,
|
||||
...(removals ?? [])
|
||||
.filter((r) => r.showInDeviceList && !enrolled.has(r.clientId))
|
||||
.map(removalRow)
|
||||
.filter(passes)
|
||||
];
|
||||
}, [devices, inScope, removals, search, showingRemovals, status, platform, selectedId]);
|
||||
|
||||
const chosenRows = rows.filter((d) => !d.removal && chosen.has(d.clientId));
|
||||
const groupName = (id: string) => groups?.find((g) => g.id === id)?.name ?? '…';
|
||||
|
||||
const title =
|
||||
selectedId === ALL_NODE ? 'All Devices' : selectedId === UNGROUPED_NODE ? 'Ungrouped Devices' : (selectedGroup?.name ?? 'Group');
|
||||
const title = showingRemovals
|
||||
? 'Removals'
|
||||
: selectedId === ALL_NODE
|
||||
? 'All Devices'
|
||||
: selectedId === UNGROUPED_NODE
|
||||
? 'Ungrouped Devices'
|
||||
: (selectedGroup?.name ?? 'Group');
|
||||
const trail = selectedGroup ? [...ancestors(groups ?? [], selectedGroup.id).reverse(), selectedGroup] : [];
|
||||
const online = inScope.filter((d) => d.status === 'online').length;
|
||||
|
||||
const columns: Column<DeviceDto>[] = [
|
||||
{
|
||||
id: 'hostname',
|
||||
header: 'Name',
|
||||
sortValue: (d) => d.hostname,
|
||||
cell: (d) => (
|
||||
<Stack direction="row" sx={{ alignItems: 'center', gap: 1.25, minWidth: 0 }}>
|
||||
<OsIcon osCaption={d.osCaption} />
|
||||
<Stack sx={{ minWidth: 0 }}>
|
||||
// A removal has no device page; its row opens the history it left behind.
|
||||
const nameColumn: Column<ListRow> = {
|
||||
id: 'hostname',
|
||||
header: 'Name',
|
||||
sortValue: (d) => d.hostname,
|
||||
cell: (d) => (
|
||||
<Stack direction="row" sx={{ alignItems: 'center', gap: 1.25, minWidth: 0 }}>
|
||||
<OsIcon osCaption={d.osCaption} />
|
||||
<Stack sx={{ minWidth: 0 }}>
|
||||
{d.removal ? (
|
||||
<Typography variant="body2" sx={{ fontWeight: 600 }}>
|
||||
{d.hostname}
|
||||
</Typography>
|
||||
) : (
|
||||
<Link
|
||||
component={RouterLink}
|
||||
to={`/devices/${encodeURIComponent(d.clientId)}`}
|
||||
@@ -172,18 +227,77 @@ export default function DevicesPage() {
|
||||
>
|
||||
{d.hostname}
|
||||
</Link>
|
||||
<Typography variant="caption" color="text.secondary" noWrap sx={{ fontFamily: 'var(--font-mono)', maxWidth: 240 }}>
|
||||
{d.clientId}
|
||||
</Typography>
|
||||
</Stack>
|
||||
)}
|
||||
<Typography variant="caption" color="text.secondary" noWrap sx={{ fontFamily: 'var(--font-mono)', maxWidth: 240 }}>
|
||||
{d.clientId}
|
||||
</Typography>
|
||||
</Stack>
|
||||
</Stack>
|
||||
)
|
||||
};
|
||||
|
||||
const stopTrying = (removal: DeviceRemovalDto) => (
|
||||
<GatedButton
|
||||
size="small"
|
||||
color="secondary"
|
||||
gate={canRemove ? undefined : 'Needs the Remove Devices permission.'}
|
||||
onClick={(e: MouseEvent<HTMLButtonElement>) => {
|
||||
e.stopPropagation();
|
||||
removalActions.stop(removal);
|
||||
}}
|
||||
>
|
||||
Stop Trying
|
||||
</GatedButton>
|
||||
);
|
||||
|
||||
const removalColumns: Column<ListRow>[] = [
|
||||
nameColumn,
|
||||
{
|
||||
id: 'state',
|
||||
header: 'State',
|
||||
sortValue: (d) => d.removal?.state,
|
||||
cell: (d) => <RemovalStateChip removal={d.removal!} />
|
||||
},
|
||||
{ id: 'requestedBy', header: 'Requested By', sortValue: (d) => d.removal?.requestedBy, cell: (d) => d.removal?.requestedBy ?? '—' },
|
||||
{
|
||||
id: 'requested',
|
||||
header: 'Requested',
|
||||
nowrap: true,
|
||||
sortValue: (d) => d.removal?.requestedUtc,
|
||||
cell: (d) => <RelativeTime value={d.removal?.requestedUtc} />
|
||||
},
|
||||
{
|
||||
id: 'finished',
|
||||
header: 'Finished',
|
||||
nowrap: true,
|
||||
sortValue: (d) => d.removal?.finishedUtc,
|
||||
cell: (d) => <RelativeTime value={d.removal?.finishedUtc} empty="—" />
|
||||
},
|
||||
{
|
||||
id: 'detail',
|
||||
header: 'Detail',
|
||||
cell: (d) => (
|
||||
<Typography variant="body2" sx={{ wordBreak: 'break-word', maxWidth: 320 }}>
|
||||
{[d.removal?.note, d.removal?.detail].filter(Boolean).join(' · ') || '—'}
|
||||
</Typography>
|
||||
)
|
||||
},
|
||||
{
|
||||
id: 'actions',
|
||||
header: '',
|
||||
align: 'right',
|
||||
nowrap: true,
|
||||
cell: (d) => (d.removal && isOpenRemoval(d.removal.state) ? stopTrying(d.removal) : null)
|
||||
}
|
||||
];
|
||||
|
||||
const columns: Column<ListRow>[] = [
|
||||
nameColumn,
|
||||
{
|
||||
id: 'status',
|
||||
header: 'Status',
|
||||
sortValue: (d) => (d.isOrphaned ? 'z' : d.status),
|
||||
cell: (d) => <DeviceStatusChip status={d.status} orphaned={d.isOrphaned} />
|
||||
sortValue: (d) => d.removal?.state ?? (d.isOrphaned ? 'z' : d.status),
|
||||
cell: (d) => (d.removal ? <RemovalStateChip removal={d.removal} /> : <DeviceStatusChip status={d.status} orphaned={d.isOrphaned} />)
|
||||
},
|
||||
{
|
||||
id: 'os',
|
||||
@@ -292,7 +406,7 @@ export default function DevicesPage() {
|
||||
cell: (d) => (
|
||||
<AgentVersion
|
||||
version={d.agentVersion}
|
||||
newest={newestFor(versions, d.osCaption, d.agentInstall)}
|
||||
newest={d.removal ? undefined : newestFor(versions, d.osCaption, d.agentInstall)}
|
||||
open={openByDevice.get(d.clientId)}
|
||||
install={d.agentInstall}
|
||||
/>
|
||||
@@ -310,19 +424,24 @@ export default function DevicesPage() {
|
||||
header: '',
|
||||
width: 56,
|
||||
align: 'right',
|
||||
cell: (d) => (
|
||||
<IconButton
|
||||
size="small"
|
||||
color="secondary"
|
||||
aria-label={`Actions for ${d.hostname}`}
|
||||
onClick={(e: MouseEvent<HTMLButtonElement>) => {
|
||||
e.stopPropagation();
|
||||
setMenu({ anchor: e.currentTarget, devices: [d] });
|
||||
}}
|
||||
>
|
||||
<MoreOutlined />
|
||||
</IconButton>
|
||||
)
|
||||
cell: (d) =>
|
||||
d.removal ? (
|
||||
isOpenRemoval(d.removal.state) ? (
|
||||
stopTrying(d.removal)
|
||||
) : null
|
||||
) : (
|
||||
<IconButton
|
||||
size="small"
|
||||
color="secondary"
|
||||
aria-label={`Actions for ${d.hostname}`}
|
||||
onClick={(e: MouseEvent<HTMLButtonElement>) => {
|
||||
e.stopPropagation();
|
||||
setMenu({ anchor: e.currentTarget, devices: [d] });
|
||||
}}
|
||||
>
|
||||
<MoreOutlined />
|
||||
</IconButton>
|
||||
)
|
||||
}
|
||||
];
|
||||
|
||||
@@ -386,6 +505,11 @@ export default function DevicesPage() {
|
||||
<MenuItem value="stale">Stale</MenuItem>
|
||||
<MenuItem value="offline">Offline</MenuItem>
|
||||
<MenuItem value="orphaned">Orphaned</MenuItem>
|
||||
<MenuItem value="removals">
|
||||
<Tooltip title="Every removed device, however long ago." placement="right">
|
||||
<span>Removals</span>
|
||||
</Tooltip>
|
||||
</MenuItem>
|
||||
</TextField>
|
||||
<TextField
|
||||
select
|
||||
@@ -400,7 +524,7 @@ export default function DevicesPage() {
|
||||
<MenuItem value="linux">Linux</MenuItem>
|
||||
<MenuItem value="macos">macOS</MenuItem>
|
||||
</TextField>
|
||||
{selectedGroup && (
|
||||
{selectedGroup && !showingRemovals && (
|
||||
<FormControlLabel
|
||||
control={<Switch size="small" checked={nested} onChange={(e) => setParam('nested', e.target.checked ? null : '0')} />}
|
||||
label={
|
||||
@@ -473,16 +597,23 @@ export default function DevicesPage() {
|
||||
<Typography variant="h5" noWrap>
|
||||
{title}
|
||||
</Typography>
|
||||
{selectedGroup && <Chip size="small" variant="outlined" label={KIND_LABEL[selectedGroup.kind]} />}
|
||||
{selectedGroup && !showingRemovals && <Chip size="small" variant="outlined" label={KIND_LABEL[selectedGroup.kind]} />}
|
||||
</Stack>
|
||||
<Typography variant="caption" color="text.secondary">
|
||||
{plural(inScope.length, 'device')}, {online} online
|
||||
{selectedGroup?.description ? ` · ${selectedGroup.description}` : ''}
|
||||
{showingRemovals ? (
|
||||
plural(removals?.length ?? 0, 'removal')
|
||||
) : (
|
||||
<>
|
||||
{plural(inScope.length, 'device')}, {online} online
|
||||
{selectedGroup?.description ? ` · ${selectedGroup.description}` : ''}
|
||||
</>
|
||||
)}
|
||||
</Typography>
|
||||
</Stack>
|
||||
}
|
||||
secondary={
|
||||
selectedGroup && (
|
||||
selectedGroup &&
|
||||
!showingRemovals && (
|
||||
<Stack direction="row" sx={{ gap: 1 }}>
|
||||
<GatedButton
|
||||
size="small"
|
||||
@@ -521,31 +652,40 @@ export default function DevicesPage() {
|
||||
}
|
||||
>
|
||||
<DataTable
|
||||
columns={columns}
|
||||
key={showingRemovals ? 'removals' : 'devices'}
|
||||
columns={showingRemovals ? removalColumns : columns}
|
||||
rows={devices ? rows : undefined}
|
||||
rowId={(d) => d.clientId}
|
||||
rowId={listRowId}
|
||||
loading={loading}
|
||||
refreshing={validating && !loading}
|
||||
error={error?.message}
|
||||
toolbar={toolbar}
|
||||
defaultSort={{ id: 'hostname', direction: 'asc' }}
|
||||
defaultSort={showingRemovals ? { id: 'requested', direction: 'desc' } : { id: 'hostname', direction: 'asc' }}
|
||||
rowLabel={(d) => d.hostname}
|
||||
onRowClick={(d) => navigate(`/devices/${encodeURIComponent(d.clientId)}`)}
|
||||
onRowClick={(d) => (d.removal ? setHistoryOf(d) : navigate(`/devices/${encodeURIComponent(d.clientId)}`))}
|
||||
minWidth={1100}
|
||||
selection={{
|
||||
selected: chosen,
|
||||
onChange: (ids, on) =>
|
||||
setChosen((current) => {
|
||||
const next = new Set(current);
|
||||
for (const id of ids) {
|
||||
if (on) next.add(id);
|
||||
else next.delete(id);
|
||||
dimmedRow={(d) => !showingRemovals && Boolean(d.removal)}
|
||||
selection={
|
||||
showingRemovals
|
||||
? undefined
|
||||
: {
|
||||
selected: chosen,
|
||||
selectable: (d) => !d.removal,
|
||||
onChange: (ids, on) =>
|
||||
setChosen((current) => {
|
||||
const next = new Set(current);
|
||||
for (const id of ids) {
|
||||
if (on) next.add(id);
|
||||
else next.delete(id);
|
||||
}
|
||||
return next;
|
||||
})
|
||||
}
|
||||
return next;
|
||||
})
|
||||
}}
|
||||
}
|
||||
empty={
|
||||
(devices?.length ?? 0) === 0 ? (
|
||||
showingRemovals ? (
|
||||
<EmptyState dense title={(removals?.length ?? 0) === 0 ? 'No Removals Yet' : 'No Matches'} />
|
||||
) : (devices?.length ?? 0) === 0 ? (
|
||||
<EmptyState
|
||||
title="No Devices Yet"
|
||||
detail="Enrolled devices appear here."
|
||||
@@ -582,6 +722,16 @@ export default function DevicesPage() {
|
||||
{groupMenu && <ActionMenu anchorEl={groupMenu} onClose={() => setGroupMenu(null)} entries={groupMenuEntries} />}
|
||||
{deviceActions.elements}
|
||||
{groupActions.elements}
|
||||
{removalActions.elements}
|
||||
<Dialog open={historyOf !== null} onClose={() => setHistoryOf(null)} maxWidth="md" fullWidth>
|
||||
<DialogTitle>{historyOf?.hostname}</DialogTitle>
|
||||
<DialogContent>{historyOf && <DeviceHistory clientId={historyOf.clientId} />}</DialogContent>
|
||||
<DialogActions sx={{ px: 3, pb: 2 }}>
|
||||
<Button variant="contained" onClick={() => setHistoryOf(null)}>
|
||||
Close
|
||||
</Button>
|
||||
</DialogActions>
|
||||
</Dialog>
|
||||
<Box>
|
||||
<AddDeviceDialog open={addOpen} onClose={() => setAddOpen(false)} />
|
||||
</Box>
|
||||
|
||||
@@ -10,7 +10,6 @@ import Typography from '@mui/material/Typography';
|
||||
// project imports
|
||||
import { RelativeTime } from 'components/DateTime';
|
||||
import GatedButton from 'components/GatedButton';
|
||||
import MainCard from 'components/MainCard';
|
||||
import PageHeader from 'components/PageHeader';
|
||||
import StatusChip from 'components/StatusChip';
|
||||
import JobOutputDrawer from 'sections/jobs/JobOutputDrawer';
|
||||
@@ -29,7 +28,10 @@ import type { ExecutionDetailDto } from 'api/types';
|
||||
/** Running / Complete / Complete With Failures, for the header. */
|
||||
function RunStatus({ run }: { run: ExecutionDetailDto }) {
|
||||
const counts = runCounts(run);
|
||||
if (!runFinished(counts)) return <StatusChip status="running" label={counts.running ? 'Running' : 'In Progress'} />;
|
||||
if (!runFinished(counts)) {
|
||||
if (!counts.running && counts.pending === 0) return <StatusChip status="waiting" />;
|
||||
return <StatusChip status="running" label={counts.running ? 'Running' : 'In Progress'} />;
|
||||
}
|
||||
return counts.failed > 0 ? (
|
||||
<StatusChip status="failed" label="Complete With Failures" />
|
||||
) : (
|
||||
@@ -105,15 +107,14 @@ export default function RunPage() {
|
||||
{execution.error && !run ? (
|
||||
<Alert severity="error">{errorMessage(execution.error, 'Could not load the run.')}</Alert>
|
||||
) : (
|
||||
<MainCard content={false}>
|
||||
<RunProgress
|
||||
run={run}
|
||||
loading={execution.loading}
|
||||
refreshing={execution.validating && !execution.loading}
|
||||
openJobId={openJobId}
|
||||
onOpenJob={setJob}
|
||||
/>
|
||||
</MainCard>
|
||||
<RunProgress
|
||||
dashboard
|
||||
run={run}
|
||||
loading={execution.loading}
|
||||
refreshing={execution.validating && !execution.loading}
|
||||
openJobId={openJobId}
|
||||
onOpenJob={setJob}
|
||||
/>
|
||||
)}
|
||||
|
||||
<JobOutputDrawer jobId={openJobId} onClose={() => setJob(null)} />
|
||||
|
||||
@@ -1,81 +0,0 @@
|
||||
import { useEffect, useEffectEvent, useState } from 'react';
|
||||
|
||||
// material-ui
|
||||
import { keyframes } from '@mui/material/styles';
|
||||
import Box from '@mui/material/Box';
|
||||
import Button from '@mui/material/Button';
|
||||
import LinearProgress from '@mui/material/LinearProgress';
|
||||
import Stack from '@mui/material/Stack';
|
||||
import Typography from '@mui/material/Typography';
|
||||
|
||||
// project imports
|
||||
import MainCard from 'components/MainCard';
|
||||
import { TERMINAL_JOB_STATES } from 'components/StatusChip';
|
||||
import { nats } from 'messaging/nats';
|
||||
|
||||
const pulse = keyframes`
|
||||
0%, 100% { opacity: 1; }
|
||||
50% { opacity: 0.3; }
|
||||
`;
|
||||
|
||||
// ==============================|| DASHBOARD - LIVE DEPLOYMENT TRACKER ||============================== //
|
||||
|
||||
/**
|
||||
* Rolls live job.state events into a running count, so a group run of N
|
||||
* endpoints shows progress as results land. Calls `onFinished` whenever a
|
||||
* job reaches a terminal state.
|
||||
*/
|
||||
export default function LiveDeployment({ onFinished }: { onFinished: () => void }) {
|
||||
const [total, setTotal] = useState(0);
|
||||
const [finished, setFinished] = useState(0);
|
||||
const finishedOne = useEffectEvent(onFinished);
|
||||
|
||||
useEffect(
|
||||
() =>
|
||||
nats.onJobEvent((_jobId, event) => {
|
||||
if (event.type !== 'job.state') return;
|
||||
const state = event.payload.state;
|
||||
if (state === 'queued' || state === 'dispatched') {
|
||||
setTotal((n) => n + 1);
|
||||
} else if (TERMINAL_JOB_STATES.includes(state)) {
|
||||
setFinished((n) => n + 1);
|
||||
finishedOne();
|
||||
}
|
||||
}),
|
||||
[]
|
||||
);
|
||||
|
||||
if (total === 0 || finished >= total) return null;
|
||||
|
||||
const dismiss = () => {
|
||||
setTotal(0);
|
||||
setFinished(0);
|
||||
};
|
||||
|
||||
return (
|
||||
<MainCard sx={(theme) => ({ borderColor: theme.vars.palette.info.main })} contentSX={{ py: 1.5, '&:last-child': { pb: 1.5 } }}>
|
||||
<Stack sx={{ gap: 1 }}>
|
||||
<Stack direction="row" sx={{ alignItems: 'center', gap: 1 }}>
|
||||
<Box
|
||||
aria-hidden
|
||||
sx={{ width: 9, height: 9, borderRadius: '50%', bgcolor: 'info.main', animation: `${pulse} 1.2s ease-in-out infinite` }}
|
||||
/>
|
||||
<Typography variant="subtitle1">Deploying</Typography>
|
||||
<Typography variant="body2" color="text.secondary" sx={{ flex: 1 }} aria-live="polite">
|
||||
{finished} of {total} endpoints finished
|
||||
</Typography>
|
||||
<Button size="small" color="secondary" onClick={dismiss}>
|
||||
Dismiss
|
||||
</Button>
|
||||
</Stack>
|
||||
<LinearProgress
|
||||
variant="determinate"
|
||||
color="info"
|
||||
value={(finished / total) * 100}
|
||||
aria-label="Deployment progress"
|
||||
sx={{ height: 8, borderRadius: 4 }}
|
||||
/>
|
||||
</Stack>
|
||||
</MainCard>
|
||||
);
|
||||
}
|
||||
@@ -20,7 +20,10 @@ const EVENTS: Record<string, { label: string; color: ChipProps['color'] }> = {
|
||||
'device-os-changed': { label: 'OS Changed', color: 'primary' },
|
||||
'agent-upgraded': { label: 'Agent Upgraded', color: 'success' },
|
||||
'agent-rolled-back': { label: 'Agent Rolled Back', color: 'warning' },
|
||||
'agent-upgrade-failed': { label: 'Agent Upgrade Failed', color: 'error' }
|
||||
'agent-upgrade-failed': { label: 'Agent Upgrade Failed', color: 'error' },
|
||||
'device-removal-requested': { label: 'Removal Requested', color: 'warning' },
|
||||
'device-removed': { label: 'Removed', color: 'secondary' },
|
||||
'device-removal-stopped': { label: 'Removal Stopped', color: 'warning' }
|
||||
};
|
||||
|
||||
// ==============================|| DEVICE - HISTORY ||============================== //
|
||||
@@ -65,7 +68,7 @@ export default function DeviceHistory({ clientId }: { clientId: string }) {
|
||||
title={
|
||||
<>
|
||||
Device History
|
||||
<HoverHelp title="Enrollments, renames, OS changes and agent upgrades, newest first — e.g. Windows PE before imaging, then the installed OS." />
|
||||
<HoverHelp title="Enrollments, renames, OS changes, agent upgrades and removals, newest first — e.g. Windows PE before imaging, then the installed OS." />
|
||||
</>
|
||||
}
|
||||
>
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
// material-ui
|
||||
import Chip from '@mui/material/Chip';
|
||||
import Stack from '@mui/material/Stack';
|
||||
import Tooltip from '@mui/material/Tooltip';
|
||||
import Typography from '@mui/material/Typography';
|
||||
|
||||
// project imports
|
||||
import Dot from 'components/@extended/Dot';
|
||||
import { REMOVAL_REASONS, REMOVAL_STATES } from './removalLabels';
|
||||
import type { DeviceRemovalDto } from 'api/types';
|
||||
|
||||
// ==============================|| REMOVAL STATE CHIP ||============================== //
|
||||
|
||||
/** A removal's state, explaining itself on hover, and under it what stands in its way when something does. */
|
||||
export default function RemovalStateChip({ removal }: { removal: DeviceRemovalDto }) {
|
||||
const known = REMOVAL_STATES[removal.state] ?? { label: removal.state, help: '', tone: 'secondary' as const };
|
||||
const why = removal.reason ? REMOVAL_REASONS[removal.reason] : undefined;
|
||||
const chip = (
|
||||
<Chip
|
||||
size="small"
|
||||
variant="light"
|
||||
color={known.tone}
|
||||
label={known.label}
|
||||
icon={<Dot size={6} color={known.tone} sx={{ ml: 0.75 }} />}
|
||||
sx={{ fontWeight: 500 }}
|
||||
/>
|
||||
);
|
||||
return (
|
||||
<Stack sx={{ alignItems: 'flex-start', gap: 0.25 }}>
|
||||
{known.help ? <Tooltip title={known.help}>{chip}</Tooltip> : chip}
|
||||
{removal.reason && (
|
||||
<Tooltip title={why?.help ?? ''}>
|
||||
<Typography variant="caption" color="text.secondary">
|
||||
{why?.label ?? removal.reason}
|
||||
</Typography>
|
||||
</Tooltip>
|
||||
)}
|
||||
</Stack>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
// types
|
||||
import type { ActivityEventPayload, DeviceRemovalState } from 'api/types';
|
||||
|
||||
type Tone = 'success' | 'warning' | 'error' | 'info' | 'primary' | 'secondary';
|
||||
|
||||
/** Label, hover text and tone per removal state (one map, so a state always reads the same). */
|
||||
export const REMOVAL_STATES: Record<DeviceRemovalState, { label: string; help: string; tone: Tone }> = {
|
||||
waiting: { label: 'Waiting for Device', help: 'The agent uninstalls when the device next comes online.', tone: 'secondary' },
|
||||
sent: { label: 'Sent', help: 'The remove command was sent, waiting for the agent to answer.', tone: 'info' },
|
||||
uninstalling: { label: 'Uninstalling', help: 'The agent is uninstalling itself.', tone: 'primary' },
|
||||
removed: { label: 'Removed', help: 'The agent has gone.', tone: 'success' },
|
||||
failed: {
|
||||
label: "Couldn't Uninstall",
|
||||
help: 'The uninstall failed on the device; the server asks again while it is online.',
|
||||
tone: 'error'
|
||||
},
|
||||
stopped: { label: 'Stopped', help: 'Nobody is trying any more. The device stays blocked and its agent stays installed.', tone: 'warning' }
|
||||
};
|
||||
|
||||
/** States still being worked on: only these can be stopped. */
|
||||
export const OPEN_REMOVAL_STATES: DeviceRemovalState[] = ['waiting', 'sent', 'uninstalling', 'failed'];
|
||||
|
||||
export const isOpenRemoval = (state: DeviceRemovalState) => OPEN_REMOVAL_STATES.includes(state);
|
||||
|
||||
/** Why a removal could not go ahead. */
|
||||
export const REMOVAL_REASONS: Record<string, { label: string; help: string }> = {
|
||||
'agent-too-old': {
|
||||
label: 'Uninstall It by Hand or Upgrade It First',
|
||||
help: 'This agent is too old to take a remove command; the server keeps blocking it.'
|
||||
},
|
||||
unsupported: { label: "Can't Remove Itself Here", help: 'A foreground or development run, or a container. Uninstall it by hand.' },
|
||||
busy: { label: 'Upgrade in Progress', help: 'An agent upgrade is under way; the server asks again.' },
|
||||
msi: { label: 'Windows Installer Failed', help: 'Windows Installer did not uninstall the agent; the server asks again.' },
|
||||
uninstall: { label: "Files Couldn't Be Removed", help: 'Files or services could not be removed; the server asks again.' },
|
||||
'stale-enrollment': { label: 'Another Enrollment', help: 'The agent holds a different enrollment, so it ignored the command.' }
|
||||
};
|
||||
|
||||
/** Why a request left a device out. */
|
||||
export const REMOVAL_SKIPS: Record<string, { label: string; help: string }> = {
|
||||
unknown: { label: 'Not Found', help: 'No such device; it may have been removed already.' },
|
||||
'out-of-scope': { label: 'Outside Your Scope', help: 'Your roles do not reach this device.' }
|
||||
};
|
||||
|
||||
/** The console refreshes devices and removals on this activity. */
|
||||
export const isRemovalActivity = (event: ActivityEventPayload) => event.kind === 'device-removal';
|
||||
@@ -15,6 +15,7 @@ import ConfirmDialog from 'components/ConfirmDialog';
|
||||
import { MenuEntry } from 'components/ActionMenu';
|
||||
import HoverHelp from 'components/HoverHelp';
|
||||
import GroupEditor from 'sections/groups/GroupEditor';
|
||||
import useDeviceRemovals from 'sections/devices/useDeviceRemovals';
|
||||
import useAgentUpgrades, { UpgradeTarget } from 'sections/upgrades/useAgentUpgrades';
|
||||
import useApi from 'hooks/useApi';
|
||||
import useAutomation from 'hooks/useAutomation';
|
||||
@@ -38,8 +39,11 @@ const upgradeTarget = (devices: DeviceDto[]): UpgradeTarget => ({
|
||||
/**
|
||||
* Everything the device "…" and bulk Actions menus can do. Render `elements`
|
||||
* once; `menuFor(devices)` builds the entries for one device or a selection.
|
||||
* `onRemoved` runs after a removal (`onDone` when not given).
|
||||
*/
|
||||
export default function useDeviceActions(options: { staticGroup?: DeviceGroupDto | null; onDone?: () => void } = {}) {
|
||||
export default function useDeviceActions(
|
||||
options: { staticGroup?: DeviceGroupDto | null; onDone?: () => void; onRemoved?: () => void } = {}
|
||||
) {
|
||||
const navigate = useNavigate();
|
||||
const { can } = useSession();
|
||||
const { openRun, openSchedule } = useAutomation();
|
||||
@@ -53,6 +57,7 @@ export default function useDeviceActions(options: { staticGroup?: DeviceGroupDto
|
||||
const [removing, setRemoving] = useState<DeviceDto[] | null>(null);
|
||||
|
||||
const upgrades = useAgentUpgrades();
|
||||
const removals = useDeviceRemovals({ onRemoved: options.onRemoved ?? options.onDone });
|
||||
|
||||
const refresh = () => Promise.all([mutate(keys.devices), mutate(keys.groups)]);
|
||||
const staticGroups = flattenTree(buildGroupTree(groups ?? [])).filter((n) => n.group.kind === 'static');
|
||||
@@ -89,6 +94,7 @@ export default function useDeviceActions(options: { staticGroup?: DeviceGroupDto
|
||||
refreshInventory: () => void refreshInventory(devices),
|
||||
upgradeAgent: () => upgrades.upgrade(upgradeTarget(devices)),
|
||||
rollBackAgent: () => upgrades.rollBack(upgradeTarget(devices)),
|
||||
removeAgent: () => removals.remove(upgradeTarget(devices)),
|
||||
details: () => navigate(`/devices/${encodeURIComponent(devices[0].clientId)}`),
|
||||
history: () => navigate(`/jobs?clientId=${encodeURIComponent(devices[0].clientId)}`),
|
||||
copyId: () =>
|
||||
@@ -104,6 +110,7 @@ export default function useDeviceActions(options: { staticGroup?: DeviceGroupDto
|
||||
const elements = (
|
||||
<>
|
||||
{upgrades.elements}
|
||||
{removals.elements}
|
||||
<ConfirmDialog
|
||||
open={grouping !== null}
|
||||
title="Add to Group"
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user