Semantic versioning (1.0.0) and MSI-based agent upgrades #19

Merged
gsadmin merged 27 commits from dev into main 2026-10-24 02:57:05 +00:00
97 changed files with 14619 additions and 365 deletions
+37 -20
View File
@@ -2147,35 +2147,50 @@ Never print token values.
# 48. Unified Versioning
All build outputs use:
> Updated: semantic versions replace the calendar versions (`yyyy.MM.dd.HHmm`)
> this section first specified. The README's Versioning section is the
> operator-facing description.
All build outputs of one build share one version, computed once
(`scripts/Get-Version.ps1`) and passed to every project as the
`UnifiedVersion` MSBuild property:
```text
yyyy.MM.dd.HHmm
X.Y.Z a release, from its git tag vX.Y.Z (1.4.0)
X.Y.Z.N any other build: N commits after release X.Y.Z (1.4.0.7)
0.0.0.N before the first release tag
0.0.0.0 no version handed in and no git (development)
```
Example:
- Numbers only (no `-dev`/`-rc` labels): every version is a valid assembly,
file and MSI version. Builds fail when major or minor > 255, another field >
65535, or major >= 2000 (the calendar era).
- Releases are numbered from conventional commits by semantic-release in
dry-run mode (`build/release`, `scripts/Get-NextRelease.ps1`): fix/perf the
patch, feat the minor, `BREAKING CHANGE:` / `type!:` the major; no release
when none of those landed. Our own script tags `vX.Y.Z` and creates the Gitea
release. Calendar-era tags never count (release tags have a 1-3 digit
major); history is seeded with `v0.1.0`, and the switch-over's breaking
change makes the first release 1.0.0.
- One version order everywhere (`AgentVersions`): field by field, missing
fourth field = 0, `+metadata` ignored, calendar era before every semantic
version, `0.0.0.0` and unparseable lowest. Mirrored in the console and the
MSIs' custom action.
```text
2026.08.06.2151
```
The same version is embedded into:
Use UTC for CI-generated versions.
The same version must be embedded into:
- Agent executable
- Agent and watchdog executables (informational version `X.Y.Z[.N]+<commit>`)
- Backend assembly
- MSI ProductVersion-compatible mapped version
- Docker image labels
- Docker image tags where allowed
- MSI ProductVersion: the four-field version (`X.Y.Z.0` for a release); each
MSI removes every installed version and a custom action refuses a newer one
(Windows Installer compares only three fields and knows nothing of the
calendar era, whose MSIs carried `yy.MM.minutes`)
- Docker image labels and tags
- frontend build metadata
- release metadata
Because MSI ProductVersion has constraints, create a deterministic compatible numeric transform while retaining the full unified version in MSI display metadata.
`Directory.Build.props` should centralize assembly version properties.
Generate a build metadata source file or MSBuild property once per build, not independently per project.
`Directory.Build.props` centralizes the assembly version properties; a build
not handed a version reads it from git (`build/Version.targets`).
---
@@ -2264,7 +2279,9 @@ Behavior:
- push:
- `<version>`
- `latest`
- create Gitea release/tag for unified version if token is available
- when the commits since the last release call for one (§48), release it
instead: build as `X.Y.Z`, create the tag `vX.Y.Z` and the Gitea release if
the token is available, push the images as `X.Y.Z` and `latest`
- attach:
- Agent executable ZIP
- Agent MSI
+3 -1
View File
@@ -178,7 +178,9 @@ LOG_MAX_BYTES=10485760
AGENT_PACKAGE_PATH=/data/agent
# ── Build / version ────────────────────────────────────────────────────────
# Set by CI; local builds derive it via scripts/Build.ps1.
# X.Y.Z (a release) or X.Y.Z.N (N commits after it), never yyyy.MM.dd.HHmm.
# Picks the image tag to pull (empty = latest) and stamps a local build;
# CI sets it, scripts/Get-Version.ps1 and deploy.sh derive it from git.
UNIFIED_VERSION=
# ── Device lifecycle ───────────────────────────────────────────────────────
+50 -9
View File
@@ -13,6 +13,7 @@ on:
- 'tests/**'
- 'scripts/**'
- 'assets/**'
- 'build/**'
- '.gitea/workflows/**'
- 'ExecuteR.slnx'
- 'Directory.Build.props'
@@ -26,6 +27,7 @@ on:
- 'tests/**'
- 'scripts/**'
- 'assets/**'
- 'build/**'
- '.gitea/workflows/**'
- 'ExecuteR.slnx'
- 'Directory.Build.props'
@@ -42,11 +44,15 @@ jobs:
# instead of recomputing, so binaries and container images never drift.
outputs:
version: ${{ steps.ver.outputs.version }}
short: ${{ steps.ver.outputs.short }}
release: ${{ steps.ver.outputs.release }}
steps:
# This repository uses the SHA-256 object format. actions/checkout
# cannot clone it (it inits a SHA-1 repo then fetches: "mismatched
# algorithms"), so check out with plain git clone, which negotiates
# the object format correctly.
# the object format correctly. The version needs the branch's whole
# history and its tags (the last release tag and the commits since it),
# so no --depth.
- name: Checkout
shell: pwsh
env:
@@ -57,7 +63,7 @@ jobs:
$ref = '${{ github.head_ref }}'
if (-not $ref) { $ref = '${{ github.ref_name }}' }
$url = '${{ github.server_url }}/${{ github.repository }}.git' -replace '^https://', "https://x-access-token:$env:CHECKOUT_TOKEN@"
git clone --depth 1 --branch $ref $url .
git clone --single-branch --branch $ref $url .
git log -1 --format='%H %s'
- name: Setup .NET 10
@@ -71,19 +77,43 @@ jobs:
# @nats-io/* packages require Node >= 22.
node-version: '22'
- name: Compute unified version
# README, Versioning. A build is X.Y.Z.N (N commits after the last
# release tag vX.Y.Z; 0.0.0.N before the first). A push to main that
# release.yml releases (a fix, feat or breaking change since the last
# release) builds as that release, X.Y.Z, the version release.yml
# computes the same way (semantic-release, dry run): the two workflows
# agree on a release commit's version, and its images come from
# release.yml (release=true skips them here).
- name: Version
id: ver
shell: pwsh
run: |
$v = & ./scripts/Get-UnifiedVersion.ps1
$ErrorActionPreference = 'Stop'
$v = $null
$release = 'false'
if ('${{ github.event_name }}' -eq 'push' -and '${{ github.ref_name }}' -eq 'main') {
try {
$next = ./scripts/Get-NextRelease.ps1
if ($next) { $v = $next.Version; $release = 'true' }
}
catch {
Write-Host "::warning::The next release could not be computed ($_); building as a build between releases."
}
}
if (-not $v) { $v = ./scripts/Get-Version.ps1 }
$short = ./scripts/Get-Version.ps1 -Version $v -Format Short
"version=$v" | Out-File -FilePath $env:GITHUB_OUTPUT -Append
Write-Host "Unified version: $v"
"short=$short" | Out-File -FilePath $env:GITHUB_OUTPUT -Append
"release=$release" | Out-File -FilePath $env:GITHUB_OUTPUT -Append
Write-Host "Version: $v ($short; release: $release)"
# A refused dry run above leaves its exit code behind; this step succeeded.
$global:LASTEXITCODE = 0
- name: Restore
run: dotnet restore ExecuteR.slnx
- name: Build
run: dotnet build ExecuteR.slnx -c Release --no-restore "-p:UnifiedVersion=${{ steps.ver.outputs.version }}"
run: dotnet build ExecuteR.slnx -c Release --no-restore "-p:UnifiedVersion=${{ steps.ver.outputs.version }}" "-p:GitCommitShort=${{ steps.ver.outputs.short }}"
- name: Unit tests
run: |
@@ -92,6 +122,11 @@ jobs:
dotnet test tests/Agent.Watchdog.Tests -c Release --no-build --nologo
dotnet test tests/Backend.Tests -c Release --no-build --nologo
# The version scripts and the release numbering, in a scratch repository.
- name: Versioning checks
shell: pwsh
run: ./scripts/tests/Test-Versioning.ps1
- name: Integration tests
run: dotnet test tests/Integration.Tests -c Release --no-build --nologo
@@ -164,8 +199,9 @@ jobs:
# registry. Image names match what docker-compose.yml pulls:
# <gitea-host>/<owner>/executer-backend:<tag>
# <gitea-host>/<owner>/executer-webui:<tag>
# A release commit's images (X.Y.Z and latest) come from release.yml.
needs: build-test
if: ${{ github.event_name == 'push' }}
if: ${{ github.event_name == 'push' && needs.build-test.outputs.release != 'true' }}
# Needs Docker; label must match a registered Linux runner.
runs-on: ubuntu-host
env:
@@ -249,12 +285,13 @@ jobs:
- name: Build images
env:
VERSION: ${{ needs.build-test.outputs.version }}
GIT_COMMIT_SHORT: ${{ needs.build-test.outputs.short }}
PREFIX: ${{ steps.ver.outputs.prefix }}
SOURCE_URL: ${{ github.server_url }}/${{ github.repository }}
run: |
set -e
GIT_COMMIT="$(git rev-parse HEAD)"
GIT_COMMIT_SHORT="$(git rev-parse --short HEAD)"
[ -n "$GIT_COMMIT_SHORT" ] || GIT_COMMIT_SHORT="$(git rev-parse --short HEAD)"
BUILD_DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
for svc in backend webui; do
docker build -f "src/Infra/${svc}/Dockerfile" \
@@ -271,11 +308,15 @@ jobs:
VERSION: ${{ needs.build-test.outputs.version }}
PREFIX: ${{ steps.ver.outputs.prefix }}
REF_NAME: ${{ github.ref_name }}
# main: <version> (X.Y.Z.N) and latest. Other branches: dev and
# <version>-dev only. A branch counts its own commits since the same
# release tag, so dev's 1.4.0.3 is not main's 1.4.0.3 and must not
# take main's version tag.
run: |
set -e
for svc in backend webui; do
docker push "${PREFIX}-${svc}:$VERSION"
if [ "$REF_NAME" = "main" ]; then
docker push "${PREFIX}-${svc}:$VERSION"
docker tag "${PREFIX}-${svc}:$VERSION" "${PREFIX}-${svc}:latest"
docker push "${PREFIX}-${svc}:latest"
else
+174 -18
View File
@@ -1,7 +1,20 @@
name: Release
# Docs-only merges to main skip the release build the same way CI skips
# docs-only commits — a release only makes sense when shippable bits changed.
# README, Versioning. A push to main releases when the conventional commits
# since the last release tag vX.Y.Z call for one: semantic-release (dry run,
# scripts/Get-NextRelease.ps1) bumps the patch for fix/perf, the minor for
# feat and the major for a BREAKING CHANGE footer or "type!:". Only docs,
# tests, chores, refactors... release nothing: the run stops after numbering
# (no tag, no release, no images; CI's X.Y.Z.N build and images stand).
#
# A release builds every binary as X.Y.Z, then scripts/Publish-Release.ps1
# creates the tag vX.Y.Z on the commit and the Gitea release with the notes,
# and the images job publishes the backend and webui images as X.Y.Z and
# latest (CI skips its images for a release commit and builds it as X.Y.Z
# too, so both workflows agree on its version). A re-run of a released commit
# publishes the same version again (missing assets only).
#
# Docs-only merges skip the run the same way CI skips docs-only commits.
on:
push:
branches: [main]
@@ -10,23 +23,35 @@ on:
- 'tests/**'
- 'scripts/**'
- 'assets/**'
- 'build/**'
- '.gitea/workflows/**'
- 'ExecuteR.slnx'
- 'Directory.Build.props'
- 'Directory.Packages.props'
- 'global.json'
- 'dotnet-tools.json'
workflow_dispatch:
# One release at a time: a second push waits for the first to tag its commit,
# so it numbers after it.
concurrency:
group: release
cancel-in-progress: false
jobs:
agent-artifacts:
# Publishes the self-contained agent for every platform, the MSI, the
# combined standalone server (win/linux) and checksums, then attaches them
# to a Gitea release for the unified version.
release:
# Publishes the self-contained agent for every platform, the MSIs, the
# combined standalone server (win/linux) and checksums, then tags the
# commit and attaches them to the Gitea release.
# Label must match a registered runner: windows-host.
runs-on: windows-host
outputs:
version: ${{ steps.next.outputs.version }}
short: ${{ steps.next.outputs.short }}
steps:
# SHA-256 repo: plain git clone (actions/checkout inits SHA-1 and
# cannot fetch from a SHA-256 server).
# cannot fetch from a SHA-256 server). The whole history and its tags:
# semantic-release reads the commits since the last release tag.
- name: Checkout
shell: pwsh
env:
@@ -35,7 +60,7 @@ jobs:
$ErrorActionPreference = 'Stop'
Get-ChildItem -Force | Remove-Item -Recurse -Force -ErrorAction SilentlyContinue
$url = '${{ github.server_url }}/${{ github.repository }}.git' -replace '^https://', "https://x-access-token:$env:CHECKOUT_TOKEN@"
git clone --depth 1 --branch '${{ github.ref_name }}' $url .
git clone --single-branch --branch '${{ github.ref_name }}' $url .
git log -1 --format='%H %s'
- name: Setup .NET 10
@@ -46,31 +71,62 @@ jobs:
- name: Setup Node
uses: actions/setup-node@v4
with:
# @nats-io/* packages require Node >= 22 (static WebUI build).
# semantic-release needs Node >= 20.8.1; @nats-io/* (WebUI) >= 22.
node-version: '22'
- name: Compute unified version
id: ver
# Nothing to release leaves version empty and every later step skips.
- name: Next release
id: next
shell: pwsh
run: |
$v = & ./scripts/Get-UnifiedVersion.ps1
"version=$v" | Out-File -FilePath $env:GITHUB_OUTPUT -Append
$ErrorActionPreference = 'Stop'
$next = ./scripts/Get-NextRelease.ps1 -NotesPath artifacts/release-notes.md
if ($next) {
$short = ./scripts/Get-Version.ps1 -Version $next.Version -Format Short
"version=$($next.Version)" | Out-File -FilePath $env:GITHUB_OUTPUT -Append
"short=$short" | Out-File -FilePath $env:GITHUB_OUTPUT -Append
}
$global:LASTEXITCODE = 0
- name: Generate icon
if: ${{ steps.next.outputs.version != '' }}
shell: pwsh
run: ./scripts/Generate-Icons.ps1
- name: Package agent (all platforms + MSI + checksums)
if: ${{ steps.next.outputs.version != '' }}
shell: pwsh
run: |
./scripts/Package-Agent.ps1 -Version ${{ steps.ver.outputs.version }} `
./scripts/Package-Agent.ps1 -Version ${{ steps.next.outputs.version }} `
-Runtimes win-x64,win-arm64,linux-x64,linux-arm64,osx-x64,osx-arm64 -SkipIcon
# The installers the backend image serves (see the images job).
$stage = 'artifacts/agent-package'
New-Item -ItemType Directory -Force $stage | Out-Null
$msi = "artifacts/ExecuteR-Agent-${{ steps.next.outputs.version }}.msi"
$zip = "artifacts/ExecuteR-Agent-${{ steps.next.outputs.version }}-win-x64.zip"
if (Test-Path $msi) { Copy-Item $msi "$stage/Agent.msi" }
if (Test-Path $zip) { Copy-Item $zip "$stage/Agent.zip" }
foreach ($rid in 'linux-x64','linux-arm64','osx-x64','osx-arm64') {
$unix = "artifacts/ExecuteR-Agent-${{ steps.next.outputs.version }}-$rid.zip"
if (Test-Path $unix) { Copy-Item $unix "$stage/Agent-$rid.zip" }
}
'${{ steps.next.outputs.version }}' | Set-Content "$stage/VERSION" -NoNewline
- name: Package server (win-x64 + linux-x64)
if: ${{ steps.next.outputs.version != '' }}
shell: pwsh
run: ./scripts/Package-Server.ps1 -Version ${{ steps.ver.outputs.version }} -SkipIcon
run: ./scripts/Package-Server.ps1 -Version ${{ steps.next.outputs.version }} -SkipIcon
- name: Create Gitea release and upload assets
- name: Upload agent package
if: ${{ steps.next.outputs.version != '' }}
uses: actions/upload-artifact@v3
with:
name: agent-package
path: artifacts/agent-package/
retention-days: 7
- name: Tag, create the Gitea release and upload assets
if: ${{ steps.next.outputs.version != '' }}
shell: pwsh
env:
# GITEA_TOKEN is injected automatically by Gitea Actions; SERVER_URL
@@ -80,5 +136,105 @@ jobs:
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
GITEA_OWNER: ${{ github.repository_owner }}
GITEA_REPOSITORY: ${{ github.event.repository.name }}
VERSION: ${{ steps.ver.outputs.version }}
run: ./scripts/Publish-Release.ps1 -Version $env:VERSION
run: |
$notes = if (Test-Path artifacts/release-notes.md) { 'artifacts/release-notes.md' } else { '' }
./scripts/Publish-Release.ps1 -Version ${{ steps.next.outputs.version }} -NotesFile $notes -Commit "$(git rev-parse HEAD)"
images:
# The release's backend + webui images, tagged X.Y.Z and latest, in the
# Gitea container registry (the same names CI publishes and
# docker-compose.yml pulls). Only once the release is published.
needs: release
if: ${{ needs.release.outputs.version != '' }}
runs-on: ubuntu-host
env:
# Gitea's built-in registry by default; REGISTRY_* override it with an
# external one (see ci.yml).
REGISTRY_HOST: ${{ secrets.REGISTRY_HOST }}
REGISTRY_USER: ${{ secrets.REGISTRY_USERNAME }}
REGISTRY_PASS: ${{ secrets.REGISTRY_PASSWORD }}
SERVER_URL: ${{ github.server_url }}
ACTOR: ${{ github.actor }}
BUILTIN_TOKEN: ${{ secrets.GITEA_TOKEN }}
OWNER: ${{ github.repository_owner }}
steps:
# SHA-256 repo: plain git clone (see the release job). The image build
# takes its version from the release job, not from git.
- name: Checkout
shell: bash
env:
CHECKOUT_TOKEN: ${{ github.token }}
run: |
set -e
find . -mindepth 1 -maxdepth 1 -exec rm -rf {} + 2>/dev/null || true
URL="$(echo '${{ github.server_url }}/${{ github.repository }}.git' | sed "s#^https://#https://x-access-token:${CHECKOUT_TOKEN}@#")"
git clone --depth 1 --branch '${{ github.ref_name }}' "$URL" .
git log -1 --format='%H %s'
- name: Resolve registry target
id: reg
shell: bash
run: |
set -e
if [ -n "$REGISTRY_HOST" ]; then
HOST="$REGISTRY_HOST"; USER="$REGISTRY_USER"; PASS="$REGISTRY_PASS"
echo "Using external registry $HOST"
else
HOST="$(echo "$SERVER_URL" | sed -E 's#^https?://##; s#/$##')"
USER="$ACTOR"; PASS="$BUILTIN_TOKEN"
echo "Using the built-in Gitea registry at $HOST"
fi
OWNER_LC="$(echo "$OWNER" | tr '[:upper:]' '[:lower:]')"
{
echo "host=$HOST"
echo "user=$USER"
echo "prefix=${HOST}/${OWNER_LC}/executer"
} >> "$GITHUB_OUTPUT"
echo "::add-mask::$PASS"
echo "REGISTRY_LOGIN_PASSWORD=$PASS" >> "$GITHUB_ENV"
- name: Registry login
run: echo "$REGISTRY_LOGIN_PASSWORD" | docker login "${{ steps.reg.outputs.host }}" -u "${{ steps.reg.outputs.user }}" --password-stdin
- name: Download agent package
uses: actions/download-artifact@v3
continue-on-error: true
with:
name: agent-package
path: src/Infra/backend/agent-package
- name: Build images
env:
VERSION: ${{ needs.release.outputs.version }}
GIT_COMMIT_SHORT: ${{ needs.release.outputs.short }}
PREFIX: ${{ steps.reg.outputs.prefix }}
SOURCE_URL: ${{ github.server_url }}/${{ github.repository }}
run: |
set -e
if ! ls src/Infra/backend/agent-package/Agent.* >/dev/null 2>&1; then
echo "::warning::No agent package staged - the backend image will ship without an installer."
fi
GIT_COMMIT="$(git rev-parse HEAD)"
BUILD_DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
for svc in backend webui; do
docker build -f "src/Infra/${svc}/Dockerfile" \
--build-arg UNIFIED_VERSION="$VERSION" \
--build-arg GIT_COMMIT="$GIT_COMMIT" \
--build-arg GIT_COMMIT_SHORT="$GIT_COMMIT_SHORT" \
--build-arg BUILD_DATE="$BUILD_DATE" \
--build-arg SOURCE_URL="$SOURCE_URL" \
-t "${PREFIX}-${svc}:$VERSION" .
done
- name: Tag and push
env:
VERSION: ${{ needs.release.outputs.version }}
PREFIX: ${{ steps.reg.outputs.prefix }}
run: |
set -e
for svc in backend webui; do
docker push "${PREFIX}-${svc}:$VERSION"
docker tag "${PREFIX}-${svc}:$VERSION" "${PREFIX}-${svc}:latest"
docker push "${PREFIX}-${svc}:latest"
done
echo "Published ${PREFIX}-backend and ${PREFIX}-webui as $VERSION and latest"
+19 -10
View File
@@ -1,19 +1,26 @@
<Project>
<!--
Unified versioning: yyyy.MM.dd.HHmm (UTC).
CI and scripts/Build.ps1 compute the version ONCE and pass it as
-p:UnifiedVersion=... so every artifact in a build shares one version.
The fallback below only applies to ad-hoc IDE/local builds.
Versioning (README, Versioning): a release is X.Y.Z (from its tag vX.Y.Z),
a build between releases X.Y.Z.N (N commits after the last release).
CI and the scripts compute the version ONCE (scripts/Get-Version.ps1) and
pass -p:UnifiedVersion=... (and -p:GitCommitShort=...) so every artifact
of a build shares one version. A build that is not handed one (IDE, plain
dotnet build) reads the same rule from git (build/Version.targets); without
git it stays 0.0.0.0, the development version nothing upgrades to.
-->
<PropertyGroup>
<UnifiedVersion Condition="'$(UnifiedVersion)' == ''">$([System.DateTime]::get_UtcNow().ToString('yyyy.MM.dd.HHmm'))</UnifiedVersion>
<VersionFromGit Condition="'$(VersionFromGit)' == '' And '$(UnifiedVersion)' == ''">true</VersionFromGit>
<UnifiedVersion Condition="'$(UnifiedVersion)' == ''">0.0.0.0</UnifiedVersion>
<!-- Assembly, file and MSI versions take four fields: a release X.Y.Z is X.Y.Z.0. -->
<UnifiedFileVersion>$(UnifiedVersion)</UnifiedFileVersion>
<UnifiedFileVersion Condition="$([System.Text.RegularExpressions.Regex]::IsMatch('$(UnifiedVersion)', '^\d+\.\d+\.\d+$'))">$(UnifiedVersion).0</UnifiedFileVersion>
<Version>$(UnifiedVersion)</Version>
<AssemblyVersion>$(UnifiedVersion)</AssemblyVersion>
<FileVersion>$(UnifiedVersion)</FileVersion>
<!-- ProductVersion shown in file properties: date version-shortcommit.
Packaging passes -p:GitCommitShort; plain local builds omit it. -->
<InformationalVersion Condition="'$(GitCommitShort)' != ''">$(UnifiedVersion)-$(GitCommitShort)</InformationalVersion>
<AssemblyVersion>$(UnifiedFileVersion)</AssemblyVersion>
<FileVersion>$(UnifiedFileVersion)</FileVersion>
<!-- ProductVersion shown in file properties: version+shortcommit (semantic
version build metadata, which every comparison ignores). -->
<InformationalVersion Condition="'$(GitCommitShort)' != ''">$(UnifiedVersion)+$(GitCommitShort)</InformationalVersion>
<InformationalVersion Condition="'$(GitCommitShort)' == ''">$(UnifiedVersion)</InformationalVersion>
<IncludeSourceRevisionInInformationalVersion>false</IncludeSourceRevisionInInformationalVersion>
<Company>ExecuteR</Company>
@@ -30,4 +37,6 @@
<AccelerateBuildsInVisualStudio>true</AccelerateBuildsInVisualStudio>
</PropertyGroup>
<Import Project="$(MSBuildThisFileDirectory)build/Version.targets" Condition="Exists('$(MSBuildThisFileDirectory)build/Version.targets')" />
</Project>
+3
View File
@@ -57,6 +57,9 @@
<!-- Installer -->
<PackageVersion Include="WixToolset.Util.wixext" Version="5.0.2" />
<PackageVersion Include="WixToolset.UI.wixext" Version="5.0.2" />
<!-- Managed custom actions (src/Installer.CustomActions), the same WiX 5 line. -->
<PackageVersion Include="WixToolset.Dtf.CustomAction" Version="5.0.2" />
<PackageVersion Include="WixToolset.Dtf.WindowsInstaller" Version="5.0.2" />
<!-- Tests -->
<PackageVersion Include="Microsoft.NET.Test.Sdk" Version="18.8.1" />
+1
View File
@@ -3,6 +3,7 @@
<Project Path="src/Agent/Agent.csproj" />
<Project Path="src/Agent.Watchdog/Agent.Watchdog.csproj" />
<Project Path="src/Backend/Backend.csproj" />
<Project Path="src/Installer.CustomActions/Installer.CustomActions.csproj" />
<Project Path="src/Shared.Contracts/Shared.Contracts.csproj" />
<Project Path="src/Shared.Infrastructure/Shared.Infrastructure.csproj" />
</Folder>
+2
View File
@@ -72,6 +72,8 @@ services:
# Published image by default; `build` is the fallback used when the image
# is not present locally and cannot be pulled (pull_policy: missing).
# scripts/deploy.sh pulls first and builds only if the pull fails.
# Tags: X.Y.Z (a release), latest (newest main), X.Y.Z.N (a main build
# between releases), dev and X.Y.Z.N-dev (other branches).
image: '${REGISTRY:-git.example.com/owner}/${BACKEND_IMAGE:-executer-backend}:${BACKEND_VERSION:-${UNIFIED_VERSION:-latest}}'
pull_policy: '${IMAGE_PULL_POLICY:-missing}'
container_name: EXECUTER-API-00001
+2 -1
View File
@@ -1,6 +1,7 @@
#=============================================================================
# ExecuteR - Kubernetes: backend (API) replicas, scheduler worker, WebUI
# Replace REGISTRY / VERSION before applying (or kustomize the image).
# Replace REGISTRY / VERSION before applying (or kustomize the image);
# VERSION is a release such as 1.4.0 (or latest).
#=============================================================================
apiVersion: apps/v1
kind: Deployment
+1 -1
View File
@@ -11,7 +11,7 @@ docker run --rm -v "$PWD/nats:/out" <registry>/executer-backend:latest \
nats-auth init --output /out
# -> NATS_ACCOUNT_SIGNING_SEED = account-signing.nk, NATS_ACCOUNT_PUBLIC_KEY = account.pub
# 3. Deploy
# 3. Deploy (VERSION: a release such as 1.4.0, or latest)
export REGISTRY=... VERSION=... POSTGRES_PASSWORD=... \
NATS_ACCOUNT_SIGNING_SEED=... NATS_ACCOUNT_PUBLIC_KEY=... \
PUBLIC_SERVER_URL=https://control.example.com
+148 -17
View File
@@ -223,7 +223,7 @@ Full script-execution, enrollment, and remote-shell sequence diagrams are in
# One-time setup: restores packages, installs WebUI deps, generates secrets
./scripts/Initialize-Repository.ps1
# Build everything with one unified version
# Build everything with one version (from git: 1.4.0.7, see Versioning)
./scripts/Build.ps1 -IncludeWebUI
# Run tests
@@ -611,8 +611,9 @@ the package over.
1. The agent takes the `command.agent.upgrade` command. It answers at once,
without downloading, when it cannot or need not upgrade: **unsupported**
(not running as the installed service, or the package is for another
platform), **busy** (another hand-over, swap or unreported outcome is
(not running as the installed service, the package is for another
platform, or a zip for an agent the MSI installed and the other way round),
**busy** (another hand-over, swap, MSI run or unreported outcome is
pending), or **succeeded / already** (it runs that version). A redelivered
command (same upgrade ID) is ignored.
2. It waits a random start jitter (0 to the seconds the server sends), then
@@ -653,6 +654,59 @@ never moved. The watchdog replaces **its own binary last**, after the new agent
is kept, and only when the package's watchdog is newer and runs (`version`
exits 0 within 30 s); it then restarts onto it through the service manager.
### Agents the MSI installed
A swap would leave Windows Installer, Programs and Features and every
detection tool that reads them showing the version the MSI installed, so an
agent the MSI installed upgrades (and rolls back) **through its MSI** instead.
The agent finds out at start: Windows Installer holds a product of the agent's
UpgradeCode (`MsiEnumRelatedProductsW`) installed in its own folder; no
product, another folder, or no Windows Installer at all (WinPE) is a portable
install. It reports `msi` or `portable` on presence, the device page shows it
(**Installed by MSI** / **Portable**), and the server sends an MSI install the
version's `Agent.msi` (a version the server holds no MSI of is skipped as
**no-package**; it never gets the zip). Nothing runs `msiexec.exe` or
PowerShell: the MSI is installed in process through the Windows Installer API.
1. The agent downloads and checks the MSI (same jitter, size and SHA-256 as a
zip) into `upgrade/msi/ExecuteR-Agent-<version>.msi` and confirms it is the
agent's MSI of that version (its UpgradeCode and `UNIFIEDVERSION`). For a
rollback it sets the MSI of the version it runs now beside it
(`upgrade/msi/rollback/`): its own kept copy, else the copy the server
sends with the command (downloaded, checked and kept). Without either the
upgrade still goes ahead, and the hand-over and the outcome say a new
version that is not healthy cannot be rolled back automatically.
2. The watchdog checks both MSIs again, writes `upgrade/msi.json`, copies its
own binary to `upgrade/msi/runner/` and starts the copy **detached** (its
own process group, out of the service's job, no console), because the MSI
stops both services. No file is swapped.
3. The runner calls Windows Installer with no UI and a verbose log
(`/l*v`) in `<install>\Logs\agent-msi-<version>-<id>.log`, with
`REBOOT=ReallySuppress`. 0, 3010 and 1641 mean installed; **1618** (another
installation under way) is tried again every 30 s for up to 10 minutes,
then the upgrade fails as **installer-busy**; any other code, or 30 minutes
without an answer, fails as **msi** with the code, the log path and its last
lines (Windows Installer's own transaction leaves the previous version in
place).
4. The MSI starts the new services; the runner keeps the version when its
beacon reports it, connected, within the health deadline. Otherwise it
installs the previous version's MSI again with `ALLOWDOWNGRADE=1` (its own
log, `agent-msi-<previous>-<id>.log`) and ends **rolled back** with the new
agent's last log lines; with no MSI to go back to, the new version stays
and the upgrade fails as **health-deadline**. A **Roll Back Agent** from the
console installs the older MSI with `ALLOWDOWNGRADE=1` from the start.
5. The runner leaves both services running, writes the outcome and exits. A
watchdog that starts meanwhile (the new version's, started by the MSI)
leaves a live runner alone; one that died is started again from
`msi.json`, at most three times, and the agent is never started by the
watchdog while the run is pending.
MSIs are kept for later rollbacks in `%ProgramData%\ExecuteR\packages\<sha256>\`:
the one the device was installed from (the installer keeps it) and every one an
upgrade brought, the newest three. The first upgrade of an MSI install that has
none kept yet rolls back with the server's copy of its version, when the server
still holds it.
### What you see on disk
| Path | Meaning |
@@ -663,6 +717,10 @@ exits 0 within 30 s); it then restarts onto it through the service manager.
| `upgrade/outcome.json` | how the last swap ended, until the agent has reported it |
| `upgrade/downloads/`, `upgrade/staging/` | the newest package, zipped and unpacked |
| `<install>/.next`, `.previous`, `.failed` | exist only during a swap |
| `upgrade/msi.json`, `upgrade/msi.lock` | an MSI upgrade under way: its phase (`installing`, `verifying`, `rolling-back`), Windows Installer's last code and busy retries; the runner holds the lock while it lives |
| `upgrade/msi/` | the MSI being installed, the one to go back to (`rollback/`) and the runner's copy of the watchdog (`runner/`) |
| `<install>\Logs\agent-msi-<version>-<id>.log`, `ExecuteR-Watchdog-msi.log` | Windows Installer's verbose log of each MSI run, and the runner's own log |
| `%ProgramData%\ExecuteR\packages\<sha256>\` | agent MSIs kept for rollback (the newest three) |
If the watchdog itself stops part way (power loss, a kill), its next start
finishes or undoes the run from the journal: `stopping` starts over (the agent
@@ -1448,9 +1506,12 @@ How upgrades are started, paced and rolled back. What happens on the device is i
**Versions arrive with each server release.** The agent packages that ship in
the backend image (or are dropped into `AGENT_PACKAGE_PATH`, with their
`VERSION` marker) are imported into the server's **version catalog** at
startup and every five minutes: the portable Windows zip and the Linux/macOS
builds (the MSI is for installs only — Windows upgrades swap files from the
zip). Each package is copied to `<DATA_PATH>/agent-packages/<version>/`, so a
startup and every five minutes: the MSI (listed as **Windows (MSI)**), the
portable Windows zip and the Linux/macOS builds. Each device gets the package
of how its agent was installed: an agent the MSI installed upgrades through
the MSI, with the MSI of the version it runs now in the command for a
rollback when the catalog holds it; a portable one swaps files from the zip.
Each package is copied to `<DATA_PATH>/agent-packages/<version>/`, so a
device can still be rolled back after the image moves on. A development build
(`0.0.0.0`) or a package without a version is not catalogued.
@@ -1504,7 +1565,7 @@ version that already failed or rolled back on that device.
| ExcludedGroupIds | none | existing groups; nested members are excluded too |
| KeepVersions | 5 | 2–50 |
**Device History** shows `agent-upgraded` (`2026.10.20.1200 → 2026.10.23.1400`),
**Device History** shows `agent-upgraded` (`1.4.0 → 1.5.0`),
`agent-rolled-back` (an operator's roll back, or the watchdog undoing an
upgrade, with its reason) and `agent-upgrade-failed` (with the reason).
@@ -1644,11 +1705,14 @@ Gitea Actions workflows live in `.gitea/workflows`:
- **Push to `dev`** — build + test, agent binaries for every platform
(win-x64, win-arm64, linux-x64, linux-arm64, osx-x64, osx-arm64) + MSI, the
standalone server (win-x64 + MSI, linux-x64), backend/webui images tagged
`dev` and `<version>-dev`. All binaries upload as the `binaries` run
artifact.
- **Push to `main`** — the same set, images tagged `<version>` and `latest`,
and a Gitea release with every agent ZIP, both MSIs, the Server ZIPs and
checksums.
`dev` and `<version>-dev` (`X.Y.Z.N-dev`). All binaries upload as the
`binaries` run artifact.
- **Push to `main`** — the same set, images tagged `<version>` (`X.Y.Z.N`)
and `latest`. When the commits since the last release call for one (a fix,
feat or breaking change), `release.yml` releases it instead: every binary as
`X.Y.Z`, the tag `vX.Y.Z`, a Gitea release with the notes, every agent ZIP,
both MSIs, the Server ZIPs and checksums, and the images tagged `X.Y.Z` and
`latest` (see [Versioning](#versioning)).
Scripts discover context from optional environment variables and fall back
gracefully when run locally:
@@ -1665,11 +1729,78 @@ Token values are never printed. `dev` is not merged into `main` automatically.
## Versioning
All artifacts share one unified version, `yyyy.MM.dd.HHmm` (UTC), computed once
per build (`scripts/Get-UnifiedVersion.ps1`) and embedded into the agent, the
backend assembly, Docker image labels/tags, the MSI (via a deterministic
MSI-compatible numeric transform, with the full version retained in display
metadata), and the frontend build metadata.
Semantic versions from git tags. Every artifact of a build shares one version —
the agent, the watchdog, the server, both MSIs, the images and the console
footer — computed once (`scripts/Get-Version.ps1`) and handed to every step.
| Build | Version | Example |
|---|---|---|
| A release | `X.Y.Z`, from its tag `vX.Y.Z` | `1.4.0` |
| Any other build | `X.Y.Z.N`: N commits after release `X.Y.Z` | `1.4.0.7` |
| Before the first release tag | `0.0.0.N` (commits in the history) | `0.0.0.253` |
| No version handed in and no git | `0.0.0.0`, the development version | |
Numbers only, no `-dev` or `-rc` labels, so the same version is a valid
assembly, file and MSI version. A build stops when the major or minor is over
255, another field over 65535, or the major is 2000 or more (a calendar
version). Binaries carry `1.4.0.7+1a2b3c4` (the short commit, `.dirty` for a
local build with changes) as their informational version; the version they
report drops the `+…`.
**One order everywhere.** `AgentVersions` (Shared.Contracts) compares field by
field, a missing fourth field is 0 and `+metadata` is ignored. The calendar-era
versions before 1.0.0 (`2026.10.10.0140`) are older than every semantic
version; `0.0.0.0` and anything unparseable are oldest. The server's catalog
and upgrades, the agent, the watchdog, the console, `agentVersion` group rules
(`>`, `<`; `1.4` means `1.4.0`) and the MSIs all use it.
**Releases come from conventional commits.** On a push to `main`,
`release.yml` runs semantic-release in dry-run mode
(`scripts/Get-NextRelease.ps1`, pinned in `build/release`) over the commits
since the last release tag: `fix`/`perf` bump the patch, `feat` the minor, a
`BREAKING CHANGE:` footer or `type!:` the major. Only `docs`, `test`, `chore`,
`refactor`, `ci`, `build`… release nothing; CI's `X.Y.Z.N` build stands. When a
release is due, the run builds every binary as `X.Y.Z`,
`scripts/Publish-Release.ps1` tags the commit `vX.Y.Z` and creates the Gitea
release "ExecuteR X.Y.Z" with the generated notes and the artifacts, and the
images are pushed as `X.Y.Z` and `latest`. CI builds that commit as `X.Y.Z`
too (the same dry run) and leaves its images to the release. A re-run of a
released commit publishes the same version again; a version never names a
second commit.
```powershell
./scripts/Get-Version.ps1 # this tree: 1.4.0.7
./scripts/Get-Version.ps1 -Format FileVersion # four fields: 1.4.0.7, or 1.4.0.0 for 1.4.0
./scripts/Get-NextRelease.ps1 # what main would release now (main, node)
./scripts/tests/Test-Versioning.ps1 # the rules and the numbering, in a scratch repository
```
**Seeding (once).** The calendar tags (`v2026.10.10.0140`) never count: release
tags have a 1–3 digit major. Before the first semantic release:
1. Tag the last calendar-era release commit on `main` `v0.1.0` and push the
tag (`git tag v0.1.0 <commit>`, `git push origin v0.1.0`).
2. The switch-over commit (`build(version)!: semantic versions from release
tags`) carries `BREAKING CHANGE:`, so the first release after the seed is
**1.0.0** (semantic-release takes 0.x plus a breaking change to 1.0.0). If
the change is squash-merged, keep the footer in the squashed message.
Until the seed tag exists, builds are `0.0.0.N`, release runs refuse ("No
release tag … is reachable") and CI warns. Seed before images built this way
reach a server: `0.0.0.N` already sorts after every calendar version.
**MSIs.** The ProductVersion is the four-field version (`1.4.0.0` for 1.4.0,
`1.4.0.7` between releases); Programs and Features shows it, the comments show
`ExecuteR Agent 1.4.0`. Windows Installer compares only three fields and the
calendar-era MSIs carry `26.x`, so each MSI removes every installed version and
its `FindNewerRelease` custom action first refuses a newer one, all four fields
and the era considered: *A newer version of ExecuteR Agent, 1.5.0.0, is already
installed.* `ALLOWDOWNGRADE=1` installs an older version deliberately (an MSI
upgrade's rollback passes it); credentials and data stay either way. Upgrades
and repairs lay every file down (`REINSTALLMODE=amus`), because calendar-era
binaries carry file versions `2026.x`. The agent MSI keeps a copy of itself
under `%ProgramData%\ExecuteR\packages\<sha256>\` (the 3 most recently
installed, writable only by SYSTEM and Administrators) for that rollback.
---
+75
View File
@@ -0,0 +1,75 @@
<Project>
<!-- The version rules of scripts/Get-Version.ps1 (README, Versioning) for MSBuild: a handed-in UnifiedVersion is checked, and a build
that was not handed one reads it from git. Both run before the assembly attributes are generated. -->
<!-- A handed-in version: 3 or 4 numbers (no -label), fitting an MSI ProductVersion (major and minor up to 255, the rest up to
65535) and not of the calendar era (major 2000 or more). -->
<Target Name="CheckUnifiedVersion" BeforeTargets="GetAssemblyVersion;GenerateAssemblyInfo" Condition="'$(VersionFromGit)' != 'true'">
<PropertyGroup>
<_VersionPattern>^(\d{1,9})\.(\d{1,9})\.(\d{1,9})(\.(\d{1,9}))?$</_VersionPattern>
<_VersionWellFormed>$([System.Text.RegularExpressions.Regex]::IsMatch('$(UnifiedVersion)', '$(_VersionPattern)'))</_VersionWellFormed>
</PropertyGroup>
<Error Condition="'$(_VersionWellFormed)' != 'True'"
Text="UnifiedVersion '$(UnifiedVersion)' is not major.minor.patch or major.minor.patch.build (numbers only, no -label)." />
<PropertyGroup>
<_VersionMajor>$([System.Text.RegularExpressions.Regex]::Replace('$(UnifiedVersion)', '$(_VersionPattern)', '$1'))</_VersionMajor>
<_VersionMinor>$([System.Text.RegularExpressions.Regex]::Replace('$(UnifiedVersion)', '$(_VersionPattern)', '$2'))</_VersionMinor>
<_VersionPatch>$([System.Text.RegularExpressions.Regex]::Replace('$(UnifiedVersion)', '$(_VersionPattern)', '$3'))</_VersionPatch>
<_VersionBuild>$([System.Text.RegularExpressions.Regex]::Replace('$(UnifiedVersion)', '$(_VersionPattern)', '$5'))</_VersionBuild>
<_VersionBuild Condition="'$(_VersionBuild)' == ''">0</_VersionBuild>
</PropertyGroup>
<Error Condition="$(_VersionMajor) &gt;= 2000"
Text="UnifiedVersion '$(UnifiedVersion)' is a calendar-era version (major $(_VersionMajor)), not a semantic one." />
<Error Condition="$(_VersionMajor) &gt; 255 Or $(_VersionMinor) &gt; 255 Or $(_VersionPatch) &gt; 65535 Or $(_VersionBuild) &gt; 65535"
Text="UnifiedVersion '$(UnifiedVersion)' does not fit an MSI ProductVersion (major and minor up to 255, the rest up to 65535)." />
</Target>
<!-- No version handed in: the newest release tag vX.Y.Z behind HEAD and the commits since it (1.4.0.7; the release itself on the
tagged commit), and the short hash, ".dirty" when tracked files differ. Without a release tag the base is 0.0.0 and the
build the number of commits; without git, Directory.Build.props' 0.0.0.0 stands. -->
<Target Name="StampVersionFromGit" BeforeTargets="GetAssemblyVersion;GenerateAssemblyInfo" Condition="'$(VersionFromGit)' == 'true'">
<!-- One to three digits of major: the calendar-era tags (v2026.10.10.0140) never match. -->
<Exec Command="git describe --tags --long --match &quot;v[0-9].[0-9]*.[0-9]*&quot; --match &quot;v[0-9][0-9].[0-9]*.[0-9]*&quot; --match &quot;v[0-9][0-9][0-9].[0-9]*.[0-9]*&quot;"
WorkingDirectory="$(MSBuildThisFileDirectory).." ConsoleToMSBuild="true" IgnoreExitCode="true" EchoOff="true"
StandardOutputImportance="low" StandardErrorImportance="low" ContinueOnError="true">
<Output TaskParameter="ConsoleOutput" PropertyName="_Describe" />
</Exec>
<PropertyGroup>
<_DescribePattern>^v(\d{1,3}\.\d{1,9}\.\d{1,9})-(\d+)-g([0-9a-f]+)$</_DescribePattern>
<_Described>$([System.Text.RegularExpressions.Regex]::IsMatch('$(_Describe)', '$(_DescribePattern)'))</_Described>
</PropertyGroup>
<Exec Command="git rev-parse --short HEAD" Condition="'$(_Described)' != 'True'" WorkingDirectory="$(MSBuildThisFileDirectory).."
ConsoleToMSBuild="true" IgnoreExitCode="true" EchoOff="true" StandardOutputImportance="low" StandardErrorImportance="low" ContinueOnError="true">
<Output TaskParameter="ConsoleOutput" PropertyName="_HeadShort" />
</Exec>
<Exec Command="git rev-list --count HEAD" Condition="'$(_Described)' != 'True'" WorkingDirectory="$(MSBuildThisFileDirectory).."
ConsoleToMSBuild="true" IgnoreExitCode="true" EchoOff="true" StandardOutputImportance="low" StandardErrorImportance="low" ContinueOnError="true">
<Output TaskParameter="ConsoleOutput" PropertyName="_Count" />
</Exec>
<PropertyGroup Condition="'$(_Described)' == 'True'">
<_Base>$([System.Text.RegularExpressions.Regex]::Replace('$(_Describe)', '$(_DescribePattern)', '$1'))</_Base>
<_Since>$([System.Text.RegularExpressions.Regex]::Replace('$(_Describe)', '$(_DescribePattern)', '$2'))</_Since>
<_Short>$([System.Text.RegularExpressions.Regex]::Replace('$(_Describe)', '$(_DescribePattern)', '$3'))</_Short>
</PropertyGroup>
<PropertyGroup Condition="'$(_Described)' != 'True' And $([System.Text.RegularExpressions.Regex]::IsMatch('$(_Count)', '^\d+$')) And $([System.Text.RegularExpressions.Regex]::IsMatch('$(_HeadShort)', '^[0-9a-f]+$'))">
<_Base>0.0.0</_Base>
<_Since>$(_Count)</_Since>
<_Short>$(_HeadShort)</_Short>
</PropertyGroup>
<Exec Command="git status --porcelain --untracked-files=no" Condition="'$(_Base)' != ''" WorkingDirectory="$(MSBuildThisFileDirectory).."
ConsoleToMSBuild="true" IgnoreExitCode="true" EchoOff="true" StandardOutputImportance="low" StandardErrorImportance="low" ContinueOnError="true">
<Output TaskParameter="ConsoleOutput" PropertyName="_Status" />
</Exec>
<PropertyGroup Condition="'$(_Base)' != ''">
<UnifiedVersion>$(_Base)</UnifiedVersion>
<UnifiedVersion Condition="'$(_Since)' != '0'">$(_Base).$(_Since)</UnifiedVersion>
<UnifiedFileVersion>$(_Base).$(_Since)</UnifiedFileVersion>
<GitCommitShort>$(_Short)</GitCommitShort>
<GitCommitShort Condition="'$(_Status)' != ''">$(GitCommitShort).dirty</GitCommitShort>
<Version>$(UnifiedVersion)</Version>
<AssemblyVersion>$(UnifiedFileVersion)</AssemblyVersion>
<FileVersion>$(UnifiedFileVersion)</FileVersion>
<InformationalVersion>$(UnifiedVersion)+$(GitCommitShort)</InformationalVersion>
</PropertyGroup>
</Target>
</Project>
+9
View File
@@ -0,0 +1,9 @@
// A semantic-release plugin that only looks: it records the last release semantic-release found, because a run with nothing to
// release returns no result at all. It hands it to next-release.mjs in the same process through a global (semantic-release copies
// plugin options). It never changes the bump (its commit type is none).
export const seenKey = Symbol.for('executer.release.lastRelease');
export function analyzeCommits(_pluginConfig, context) {
globalThis[seenKey] = context.lastRelease;
return null;
}
+56
View File
@@ -0,0 +1,56 @@
// The next release: semantic-release in dry-run mode on the repository, written as JSON to the file named by the first argument:
// { "version": "1.1.0", "gitTag": "v1.1.0", "lastVersion": "1.0.0", "type": "minor", "notes": "..." }
// or { "version": null, "lastVersion": "1.0.0" } when no commit since the last release calls for one (only docs, tests, chores...).
// Usage: node build/release/next-release.mjs <out.json> [repository directory, default the repository this file is in]
import { writeFile } from 'node:fs/promises';
import { dirname, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
import semanticRelease from 'semantic-release';
import { seenKey } from './last-release-plugin.mjs';
import { calendarEraMajor, cleanNotes, releaseConfig } from './release-config.mjs';
export async function nextRelease(cwd, { env = process.env, log = process.stderr } = {}) {
const plugins = [...releaseConfig.plugins, fileURLToPath(new URL('./last-release-plugin.mjs', import.meta.url))];
globalThis[seenKey] = undefined;
// semantic-release reports on stdout; this tool's output is the file, so its report goes to stderr.
const result = await semanticRelease({ ...releaseConfig, plugins, dryRun: true }, { cwd, env, stdout: log, stderr: log });
const lastRelease = globalThis[seenKey];
if (!lastRelease) {
throw new Error('semantic-release did not analyze this branch (not main, or behind its remote); see its report above.');
}
if (!lastRelease.version) {
// Never a "first release": history starts before semantic versions, at the seed tag (README, Versioning).
throw new Error('No release tag v<major>.<minor>.<patch> is reachable from HEAD. Fetch the tags (full clone), or place the seed tag v0.1.0.');
}
const next = result ? result.nextRelease : null;
for (const version of [lastRelease.version, next?.version].filter(Boolean)) {
if (Number.parseInt(version, 10) >= calendarEraMajor) {
throw new Error(`${version} is a calendar-era version, not a semantic release.`);
}
}
return next?.version
? { version: next.version, gitTag: next.gitTag, lastVersion: lastRelease.version, type: next.type, notes: cleanNotes(next.notes) }
: { version: null, lastVersion: lastRelease.version };
}
if (process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
const out = resolve(process.argv[2] ?? 'next-release.json');
const cwd = resolve(process.argv[3] ?? resolve(dirname(fileURLToPath(import.meta.url)), '../..'));
try {
const release = await nextRelease(cwd);
await writeFile(out, JSON.stringify(release, null, 2) + '\n', 'utf8');
process.stderr.write(
release.version
? `Next release: ${release.version} (${release.type}, after ${release.lastVersion})\n`
: `Nothing to release since ${release.lastVersion}\n`,
);
} catch (error) {
// The reason goes into the file too, so the calling script can say it.
process.stderr.write(`${error.message}\n`);
await writeFile(out, JSON.stringify({ error: error.message }) + '\n', 'utf8').catch(() => {});
process.exitCode = 1;
}
}
+5553
View File
File diff suppressed because it is too large Load Diff
+18
View File
@@ -0,0 +1,18 @@
{
"name": "executer-release",
"private": true,
"description": "Computes the next ExecuteR release version and its notes from the conventional commits since the last release tag (semantic-release, dry run only).",
"type": "module",
"engines": {
"node": ">=20.8.1"
},
"scripts": {
"next": "node next-release.mjs"
},
"dependencies": {
"@semantic-release/commit-analyzer": "13.0.1",
"@semantic-release/release-notes-generator": "14.0.3",
"conventional-changelog-conventionalcommits": "8.0.0",
"semantic-release": "24.2.9"
}
}
+25
View File
@@ -0,0 +1,25 @@
// How the next ExecuteR release is numbered (README, Versioning). semantic-release reads the conventional commits since the newest
// release tag v<major>.<minor>.<patch> and bumps once for all of them: fix/perf the patch, feat the minor, a "BREAKING CHANGE:" footer
// (or "!" after the type) the major. It only computes (dry run); scripts/Publish-Release.ps1 creates the tag and the Gitea release.
// The releases before semantic versions were numbered by date (v2026.10.10.0140). Those tags are not semantic versions, so
// semantic-release passes over them; a version whose major is this or more is one of them and is refused, never read as 2026.0.0.
export const calendarEraMajor = 2000;
export const releaseConfig = {
branches: ['main'],
tagFormat: 'v${version}',
plugins: [
['@semantic-release/commit-analyzer', { preset: 'conventionalcommits' }],
['@semantic-release/release-notes-generator', { preset: 'conventionalcommits' }],
],
};
/** Release notes as Gitea shows them: without the trailers (Co-Authored-By, Signed-off-by) a breaking-change note drags along. */
export function cleanNotes(notes) {
return (notes ?? '')
.replace(/[ \t]*\[skip ci\]/gi, '')
.replace(/^[ \t]*(Co-Authored-By|Signed-off-by):.*$/gim, '')
.replace(/\n{3,}/g, '\n\n')
.trim();
}
+6 -4
View File
@@ -1,6 +1,7 @@
<#
.SYNOPSIS
Builds the full solution (and optionally the WebUI) with one unified version.
Builds the full solution (and optionally the WebUI) with one version
(scripts/Get-Version.ps1: X.Y.Z for a release, X.Y.Z.N between releases).
.EXAMPLE
./scripts/Build.ps1
@@ -16,11 +17,12 @@ param(
$ErrorActionPreference = 'Stop'
$repoRoot = Split-Path -Parent $PSScriptRoot
$Version = & (Join-Path $PSScriptRoot 'Get-UnifiedVersion.ps1') -Version $Version
$Version = & (Join-Path $PSScriptRoot 'Get-Version.ps1') -Version $Version
$short = & (Join-Path $PSScriptRoot 'Get-Version.ps1') -Version $Version -Format Short
Write-Host "Unified version: $Version"
Write-Host "Version: $Version"
dotnet build (Join-Path $repoRoot 'ExecuteR.slnx') -c $Configuration --nologo "-p:UnifiedVersion=$Version"
dotnet build (Join-Path $repoRoot 'ExecuteR.slnx') -c $Configuration --nologo "-p:UnifiedVersion=$Version" "-p:GitCommitShort=$short"
if ($LASTEXITCODE -ne 0) { throw "dotnet build failed ($LASTEXITCODE)." }
if ($IncludeWebUI) {
+93
View File
@@ -0,0 +1,93 @@
<#
.SYNOPSIS
The version a release run publishes: semantic-release's next version from the
conventional commits since the last release tag (a dry run, build/release),
or nothing when no release is due.
.DESCRIPTION
fix/perf since the last release bump the patch, feat the minor, a
"BREAKING CHANGE:" footer (or "type!:") the major; only docs, tests, chores,
refactors... release nothing, and this returns nothing. A commit that is
already a release (a re-run after a failed upload) returns that release's
version again, with no notes: the release exists.
Refuses when no release tag v<major>.<minor>.<patch> is reachable (the seed tag
must be present: a full clone with tags), when the branch is not main, and for
calendar-era versions (major 2000 or more).
Installs build/release's pinned packages with npm ci when they are missing.
.PARAMETER NotesPath
Where the release notes (Markdown) are written; removed when there are none.
.OUTPUTS
Nothing, or an object with Version (1.1.0), Tag (v1.1.0), LastVersion, Type
(major, minor, patch, or rerun) and Notes (the notes file, or $null).
.EXAMPLE
$next = ./scripts/Get-NextRelease.ps1 -NotesPath artifacts/release-notes.md
if ($next) { ./scripts/Package-Agent.ps1 -Version $next.Version }
#>
[CmdletBinding()]
param(
[string]$NotesPath,
[string]$Root = (Split-Path -Parent $PSScriptRoot)
)
$ErrorActionPreference = 'Stop'
Set-StrictMode -Version 3.0
$tool = Join-Path (Split-Path -Parent $PSScriptRoot) 'build/release'
if (-not (Test-Path -LiteralPath (Join-Path $tool 'node_modules/semantic-release'))) {
# From inside the folder: npm ci needs the lockfile beside it.
Push-Location -LiteralPath $tool
try {
& npm ci --no-audit --no-fund | Out-Host
if ($LASTEXITCODE -ne 0) { throw "npm ci in build/release failed ($LASTEXITCODE)." }
}
finally {
Pop-Location
}
}
$out = Join-Path ([System.IO.Path]::GetTempPath()) "executer-next-release-$([guid]::NewGuid().ToString('N')).json"
try {
# semantic-release's report goes to stderr (the console); the result is the file.
& node (Join-Path $tool 'next-release.mjs') $out $Root | Out-Host
$exit = $LASTEXITCODE
$next = if (Test-Path -LiteralPath $out) { Get-Content -Raw -LiteralPath $out | ConvertFrom-Json } else { $null }
if ($exit -ne 0 -or -not $next -or $next.PSObject.Properties['error']) {
$reason = if ($next -and $next.PSObject.Properties['error']) { $next.error } else { "exit code $exit; see above" }
throw "Computing the next release failed: $reason"
}
}
finally {
Remove-Item -LiteralPath $out -ErrorAction SilentlyContinue
}
if ($NotesPath) { Remove-Item -LiteralPath $NotesPath -ErrorAction SilentlyContinue }
if ($next.version) {
$notes = $null
if ($NotesPath) {
New-Item -ItemType Directory -Force -Path (Split-Path -Parent ([System.IO.Path]::GetFullPath($NotesPath))) | Out-Null
[System.IO.File]::WriteAllText($NotesPath, "$($next.notes)`n", [System.Text.UTF8Encoding]::new($false))
$notes = $NotesPath
}
$result = [pscustomobject]@{ Version = $next.version; Tag = $next.gitTag; LastVersion = $next.lastVersion; Type = $next.type; Notes = $notes }
}
else {
$own = @(& git -C $Root tag --points-at HEAD | Where-Object { $_ -match '^v\d{1,3}\.\d+\.\d+$' })
if ($own.Count -eq 0) {
Write-Host "Nothing to release: no fix, feat or breaking change since $($next.lastVersion)."
return
}
$version = $own[0].Substring(1)
$result = [pscustomobject]@{ Version = $version; Tag = $own[0]; LastVersion = $version; Type = 'rerun'; Notes = $null }
}
# The same limits every build version meets (major/minor up to 255, the rest up to 65535, no calendar era).
$null = & (Join-Path $PSScriptRoot 'Get-Version.ps1') -Version $result.Version
Write-Host "Release $($result.Version) ($($result.Type), after $($result.LastVersion))"
$result
+4 -23
View File
@@ -1,11 +1,8 @@
<#
.SYNOPSIS
Computes the unified build version (yyyy.MM.dd.HHmm, UTC).
Honors an explicit -Version argument or the UNIFIED_VERSION environment variable
first. Otherwise it derives the version from the HEAD commit's committer time, so
every job and workflow building the same commit lands on the same version — no
drift between the binaries and the container images. Falls back to the current
time only when git is unavailable.
Kept for callers of the old name: the build version from scripts/Get-Version.ps1
(X.Y.Z for a release, X.Y.Z.N for a build N commits after it). Honors -Version
and UNIFIED_VERSION the same way.
#>
[CmdletBinding()]
param(
@@ -13,20 +10,4 @@ param(
)
$ErrorActionPreference = 'Stop'
if (-not $Version) { $Version = $env:UNIFIED_VERSION }
if (-not $Version) {
$epoch = "$(git show -s --format=%ct HEAD 2>$null)".Trim()
if ($epoch -match '^\d+$') {
$Version = [DateTimeOffset]::FromUnixTimeSeconds([long]$epoch).UtcDateTime.ToString('yyyy.MM.dd.HHmm')
}
else {
$Version = [DateTime]::UtcNow.ToString('yyyy.MM.dd.HHmm')
}
}
if ($Version -notmatch '^\d{4}\.\d{2}\.\d{2}\.\d{4}$') {
throw "Unified version '$Version' does not match yyyy.MM.dd.HHmm."
}
$Version
& (Join-Path $PSScriptRoot 'Get-Version.ps1') -Version $Version -Format Version
+125
View File
@@ -0,0 +1,125 @@
<#
.SYNOPSIS
Computes the build version: a release's X.Y.Z, or X.Y.Z.N for a build N commits
after the last release (README, Versioning).
.DESCRIPTION
Order of precedence:
1. -Version (explicit), then the UNIFIED_VERSION environment variable: a
release run hands in the version semantic-release computed
(scripts/Get-NextRelease.ps1); CI hands one value to every step.
2. git: the newest release tag vX.Y.Z reachable from HEAD and the commits
since it (git describe). 1.4.0.7 is seven commits after 1.4.0; a build of
the tagged commit itself is 1.4.0.
3. No release tag reachable: 0.0.0.<commits in HEAD>. No git: 0.0.0.0.
Release tags have a 1-3 digit major, so the calendar-era tags
(v2026.10.10.0140) never match. Every version is 3 or 4 numeric fields (no
-prerelease labels) so assembly, file and MSI versions accept it, and is
refused when it does not fit an MSI ProductVersion (major and minor up to 255,
the rest up to 65535) or has a calendar-era major (2000 or more).
.PARAMETER Format
Version (default): 1.4.0 or 1.4.0.7.
FileVersion: always four fields (1.4.0.0), for assembly/file/MSI versions.
Informational: 1.4.0.7+<short commit>.
Short: the short commit hash (empty without git).
MSBuild: -p:UnifiedVersion=... and -p:GitCommitShort=... lines.
Env: UNIFIED_VERSION=... and GIT_COMMIT_SHORT=... lines (for $GITHUB_ENV).
Object: all of the above.
.PARAMETER Describe
What `git describe --tags --long` would print (v1.4.0-7-g1a2b3c4), in place of
git; an empty string means no release tag (0.0.0.<-Commits>). For tests.
.EXAMPLE
./scripts/Get-Version.ps1 # 0.0.0.253, 1.4.0, 1.4.0.7
./scripts/Get-Version.ps1 -Format FileVersion # 1.4.0.0
./scripts/Get-Version.ps1 -Version 1.5.0 # validated and echoed
#>
[CmdletBinding()]
param(
[string]$Version,
[ValidateSet('Version', 'FileVersion', 'Informational', 'Short', 'MSBuild', 'Env', 'Object')]
[string]$Format = 'Version',
[string]$Root = (Split-Path -Parent $PSScriptRoot),
[string]$Describe,
[int]$Commits = 0
)
$ErrorActionPreference = 'Stop'
Set-StrictMode -Version 3.0
# Majors from here up are the calendar era (yyyy.MM.dd.HHmm), never a semantic version.
$calendarEraMajor = 2000
function Assert-Fields([long[]]$Fields, [string]$Text) {
if ($Fields[0] -ge $calendarEraMajor) {
throw "Version '$Text' is a calendar-era version (major $($Fields[0])), not a semantic one."
}
if ($Fields[0] -gt 255 -or $Fields[1] -gt 255 -or @($Fields | Select-Object -Skip 2 | Where-Object { $_ -gt 65535 }).Count -gt 0) {
throw "Version '$Text' does not fit an MSI ProductVersion (major and minor up to 255, the rest up to 65535)."
}
}
$git = [bool](Get-Command git -ErrorAction SilentlyContinue)
$short = ''
if ($git) {
$short = "$(& git -C $Root rev-parse --short HEAD 2>$null)".Trim()
if ($LASTEXITCODE -ne 0) { $short = '' }
}
if (-not $Version) { $Version = $env:UNIFIED_VERSION }
if ($Version) {
if ($Version -notmatch '^(\d{1,9})\.(\d{1,9})\.(\d{1,9})(?:\.(\d{1,9}))?$') {
throw "Version '$Version' is not major.minor.patch or major.minor.patch.build (numbers only, no -label)."
}
$base = [long[]]@($Matches[1], $Matches[2], $Matches[3])
$since = if ($Matches[4]) { [long]$Matches[4] } else { 0 }
if ($env:GIT_COMMIT_SHORT) { $short = $env:GIT_COMMIT_SHORT }
}
else {
if (-not $PSBoundParameters.ContainsKey('Describe')) {
$Describe = ''
if ($git -and $short) {
# One to three digits of major: v2026.10.10.0140 (calendar era) never matches.
$Describe = "$(& git -C $Root describe --tags --long --match 'v[0-9].[0-9]*.[0-9]*' --match 'v[0-9][0-9].[0-9]*.[0-9]*' --match 'v[0-9][0-9][0-9].[0-9]*.[0-9]*' 2>$null)".Trim()
if ($LASTEXITCODE -ne 0) { $Describe = '' }
$Commits = [int]"$(& git -C $Root rev-list --count HEAD 2>$null)".Trim()
}
}
if ($Describe -match '^v(\d{1,3})\.(\d{1,9})\.(\d{1,9})-(\d+)-g[0-9a-f]+$') {
$base = [long[]]@($Matches[1], $Matches[2], $Matches[3])
$since = [long]$Matches[4]
}
elseif ($Describe) {
throw "git describe returned '$Describe', which is not v<major>.<minor>.<patch>-<commits>-g<hash>."
}
else {
$base = [long[]]@(0, 0, 0)
$since = [long]$Commits
}
}
$semantic = $base -join '.'
$text = if ($since -gt 0) { "$semantic.$since" } else { $semantic }
Assert-Fields ($base + $since) $(if ($Version) { $Version } else { $text })
$result = [pscustomobject]@{
Version = $text
FileVersion = "$semantic.$since"
Short = $short
Informational = if ($short) { "$text+$short" } else { $text }
}
switch ($Format) {
'Version' { $result.Version }
'FileVersion' { $result.FileVersion }
'Informational' { $result.Informational }
'Short' { $result.Short }
'MSBuild' { "-p:UnifiedVersion=$($result.Version)"; if ($result.Short) { "-p:GitCommitShort=$($result.Short)" } }
'Env' { "UNIFIED_VERSION=$($result.Version)"; "GIT_COMMIT_SHORT=$($result.Short)" }
default { $result }
}
+15 -13
View File
@@ -10,7 +10,7 @@ systemd/launchd, no remote shell).
.EXAMPLE
./scripts/Package-Agent.ps1
./scripts/Package-Agent.ps1 -Version 2026.08.06.2151 -SkipMsi
./scripts/Package-Agent.ps1 -Version 1.4.0 -SkipMsi
./scripts/Package-Agent.ps1 -Runtimes win-x64,linux-x64,linux-arm64,osx-x64,osx-arm64
#>
[CmdletBinding()]
@@ -23,7 +23,10 @@ param(
$ErrorActionPreference = 'Stop'
$repoRoot = Split-Path -Parent $PSScriptRoot
$Version = & (Join-Path $PSScriptRoot 'Get-UnifiedVersion.ps1') -Version $Version
$Version = & (Join-Path $PSScriptRoot 'Get-Version.ps1') -Version $Version
# The MSI ProductVersion takes four fields: a release X.Y.Z is X.Y.Z.0.
$fileVersion = & (Join-Path $PSScriptRoot 'Get-Version.ps1') -Version $Version -Format FileVersion
$short = & (Join-Path $PSScriptRoot 'Get-Version.ps1') -Version $Version -Format Short
$artifacts = Join-Path $repoRoot 'artifacts'
Write-Host "Packaging Agent $Version for: $($Runtimes -join ', ')"
@@ -50,7 +53,7 @@ foreach ($rid in $Runtimes) {
-p:EnableCompressionInSingleFile=true `
-p:DebugType=None `
"-p:UnifiedVersion=$Version" `
"-p:GitCommitShort=$(git -C $repoRoot rev-parse --short HEAD 2>$null)" `
"-p:GitCommitShort=$short" `
-o $ridPublishDir
if ($LASTEXITCODE -ne 0) { throw "Agent publish for $rid failed ($LASTEXITCODE)." }
@@ -72,7 +75,7 @@ foreach ($rid in $Runtimes) {
-r $rid `
-p:DebugType=None `
"-p:UnifiedVersion=$Version" `
"-p:GitCommitShort=$(git -C $repoRoot rev-parse --short HEAD 2>$null)" `
"-p:GitCommitShort=$short" `
-o $watchdogPublishDir
if ($LASTEXITCODE -ne 0) { throw "Watchdog publish for $rid failed ($LASTEXITCODE)." }
if ($rid -like 'win-*') {
@@ -95,17 +98,14 @@ $publishDir = Join-Path $artifacts "agent/$Version/win-x64"
$wixProject = Join-Path $repoRoot 'src/Installer/Agent.Installer.wixproj'
if (-not $SkipMsi -and (Test-Path $wixProject) -and (Test-Path $publishDir)) {
# Deterministic MSI ProductVersion transform (see BuildVersion.ToMsiProductVersion):
# yyyy.MM.dd.HHmm -> (yyyy-2000).(MM).((dd-1)*1440 + HH*60 + mm)
$parts = $Version.Split('.')
$hhmm = [int]$parts[3]
$msiVersion = '{0}.{1}.{2}' -f ([int]$parts[0] - 2000), [int]$parts[1],
((([int]$parts[2] - 1) * 1440) + ([math]::Floor($hhmm / 100) * 60) + ($hhmm % 100))
Write-Host "MSI ProductVersion: $msiVersion"
# ProductVersion is the four-field version (1.4.0.0, 1.4.0.7). Windows
# Installer compares only three fields, so the MSI removes every installed
# version and its FindNewerRelease custom action refuses a newer one.
Write-Host "MSI ProductVersion: $fileVersion"
dotnet build $wixProject -c Release --nologo `
"-p:UnifiedVersion=$Version" `
"-p:MsiProductVersion=$msiVersion" `
"-p:MsiProductVersion=$fileVersion" `
"-p:AgentPublishDir=$publishDir"
if ($LASTEXITCODE -ne 0) { throw "MSI build failed ($LASTEXITCODE)." }
@@ -120,7 +120,9 @@ if (-not $SkipMsi -and (Test-Path $wixProject) -and (Test-Path $publishDir)) {
# Checksums for release attachment.
$checksums = Join-Path $artifacts "ExecuteR-Agent-$Version-checksums.txt"
Get-ChildItem $artifacts -File | Where-Object { $_.Name -like "ExecuteR-Agent-$Version*" -and $_.Extension -ne '.txt' } | ForEach-Object {
# This version's files only: 1.4.0's pattern must not take 1.4.0.7's.
$ownFile = '^ExecuteR-Agent-' + [regex]::Escape($Version) + '(-.+\.zip|\.msi)$'
Get-ChildItem $artifacts -File | Where-Object { $_.Name -match $ownFile } | ForEach-Object {
"{0} {1}" -f (Get-FileHash $_.FullName -Algorithm SHA256).Hash.ToLowerInvariant(), $_.Name
} | Set-Content $checksums -Encoding utf8NoBOM
Write-Host "Created $checksums"
+11 -11
View File
@@ -23,7 +23,10 @@ param(
$ErrorActionPreference = 'Stop'
$repoRoot = Split-Path -Parent $PSScriptRoot
$Version = & (Join-Path $PSScriptRoot 'Get-UnifiedVersion.ps1') -Version $Version
$Version = & (Join-Path $PSScriptRoot 'Get-Version.ps1') -Version $Version
# The MSI ProductVersion takes four fields: a release X.Y.Z is X.Y.Z.0.
$fileVersion = & (Join-Path $PSScriptRoot 'Get-Version.ps1') -Version $Version -Format FileVersion
$short = & (Join-Path $PSScriptRoot 'Get-Version.ps1') -Version $Version -Format Short
$artifacts = Join-Path $repoRoot 'artifacts'
$natsVersion = '2.14.4'
@@ -60,7 +63,7 @@ foreach ($rid in $Runtimes) {
-p:PublishSingleFile=true `
-p:DebugType=None `
-p:UnifiedVersion=$Version `
"-p:GitCommitShort=$(git -C $repoRoot rev-parse --short HEAD 2>$null)" `
"-p:GitCommitShort=$short" `
-o $publishDir
if ($LASTEXITCODE -ne 0) { throw "dotnet publish for $rid failed ($LASTEXITCODE)." }
@@ -185,17 +188,12 @@ TRUSTED_HOSTS, TRUST_PROXY=0 to disable.
$serverWix = Join-Path $repoRoot 'src/Installer.Server/Server.Installer.wixproj'
$winPublish = Join-Path $artifacts "server/$Version/win-x64"
if (-not $SkipMsi -and (Test-Path $serverWix) -and (Test-Path $winPublish)) {
# Deterministic MSI ProductVersion transform (same as the agent):
# yyyy.MM.dd.HHmm -> (yyyy-2000).(MM).((dd-1)*1440 + HH*60 + mm)
$parts = $Version.Split('.')
$hhmm = [int]$parts[3]
$msiVersion = '{0}.{1}.{2}' -f ([int]$parts[0] - 2000), [int]$parts[1],
((([int]$parts[2] - 1) * 1440) + ([math]::Floor($hhmm / 100) * 60) + ($hhmm % 100))
Write-Host "Server MSI ProductVersion: $msiVersion"
# ProductVersion is the four-field version, as for the agent MSI.
Write-Host "Server MSI ProductVersion: $fileVersion"
dotnet build $serverWix -c Release --nologo `
"-p:UnifiedVersion=$Version" `
"-p:MsiProductVersion=$msiVersion" `
"-p:MsiProductVersion=$fileVersion" `
"-p:ServerPublishDir=$winPublish"
if ($LASTEXITCODE -ne 0) { throw "Server MSI build failed ($LASTEXITCODE)." }
@@ -210,7 +208,9 @@ if (-not $SkipMsi -and (Test-Path $serverWix) -and (Test-Path $winPublish)) {
# Checksums for release attachment.
$checksums = Join-Path $artifacts "ExecuteR-Server-$Version-checksums.txt"
Get-ChildItem $artifacts -File | Where-Object { $_.Name -like "ExecuteR-Server-$Version*" -and $_.Extension -ne '.txt' } | ForEach-Object {
# This version's files only: 1.4.0's pattern must not take 1.4.0.7's.
$ownFile = '^ExecuteR-Server-' + [regex]::Escape($Version) + '(-.+\.zip|\.msi)$'
Get-ChildItem $artifacts -File | Where-Object { $_.Name -match $ownFile } | ForEach-Object {
"{0} {1}" -f (Get-FileHash $_.FullName -Algorithm SHA256).Hash.ToLowerInvariant(), $_.Name
} | Set-Content $checksums -Encoding utf8NoBOM
Write-Host "Created $checksums"
+81 -27
View File
@@ -1,22 +1,44 @@
<#
.SYNOPSIS
Creates (or reuses) a Gitea release for the unified version and uploads the
agent artifacts from ./artifacts. No-ops with a clear message when GITEA_TOKEN
or GITEA_URL are absent (local runs). Never prints the token.
Publishes a release on Gitea: the tag vX.Y.Z on the commit, the release
"ExecuteR X.Y.Z" with the notes semantic-release wrote, and every agent and
server artifact of that version from ./artifacts attached.
.DESCRIPTION
The version is a release version X.Y.Z (scripts/Get-NextRelease.ps1). Safe to
run again: an existing tag or release of the same commit is reused and files
already uploaded whole are skipped. A tag or release of that version on another
commit is refused: a version names one commit.
No-ops with a clear message when GITEA_URL/TOKEN/OWNER/REPOSITORY are not all
set (local runs). Never prints the token.
.NOTES
Uses the Gitea REST API:
POST /api/v1/repos/{owner}/{repo}/releases
POST /api/v1/repos/{owner}/{repo}/releases/{id}/assets
Gitea REST API:
GET/POST /api/v1/repos/{owner}/{repo}/tags
GET /api/v1/repos/{owner}/{repo}/releases/tags/{tag}
POST /api/v1/repos/{owner}/{repo}/releases
POST /api/v1/repos/{owner}/{repo}/releases/{id}/assets
.EXAMPLE
./scripts/Publish-Release.ps1 -Version 1.1.0 -NotesFile artifacts/release-notes.md -Commit $env:GITHUB_SHA
#>
[CmdletBinding()]
param(
[string]$Version
[Parameter(Mandatory)]
[ValidatePattern('^\d{1,3}\.\d+\.\d+$')]
[string]$Version,
# The release notes (Markdown) from semantic-release; none on a re-run.
[string]$NotesFile,
[string]$Commit = "$(git rev-parse HEAD 2>$null)".Trim(),
# Marks the Gitea release as a pre-release.
[switch]$Prerelease
)
$ErrorActionPreference = 'Stop'
$repoRoot = Split-Path -Parent $PSScriptRoot
$Version = & (Join-Path $PSScriptRoot 'Get-UnifiedVersion.ps1') -Version $Version
# The same limits as every build version.
$null = & (Join-Path $PSScriptRoot 'Get-Version.ps1') -Version $Version
$giteaUrl = $env:GITEA_URL
$token = $env:GITEA_TOKEN
@@ -27,41 +49,73 @@ if (-not $giteaUrl -or -not $token -or -not $owner -or -not $repo) {
Write-Host 'GITEA_URL/TOKEN/OWNER/REPOSITORY not all set; skipping release publish.'
return
}
if (-not $Commit) { throw 'No commit to tag (-Commit, or run inside the repository).' }
$headers = @{ Authorization = "token $token" }
$apiBase = "$($giteaUrl.TrimEnd('/'))/api/v1/repos/$owner/$repo"
$tag = "v$Version"
Write-Host "Creating release $tag on $owner/$repo"
$body = @{
tag_name = $tag
target_commitish = 'main'
name = "ExecuteR $Version"
body = "Automated release for unified version $Version."
draft = $false
prerelease = $false
} | ConvertTo-Json
function Get-OrNull([string]$Uri) {
try { Invoke-RestMethod -Method Get -Uri $Uri -Headers $headers }
catch {
if ($_.Exception.Response -and [int]$_.Exception.Response.StatusCode -eq 404) { return $null }
throw
}
}
try {
# ── The tag on the commit ────────────────────────────────────────────────────
$existingTag = Get-OrNull "$apiBase/tags/$tag"
if ($existingTag) {
if ($existingTag.commit.sha -ne $Commit) {
throw "Tag $tag already exists on commit $($existingTag.commit.sha), not $Commit; a version names one commit."
}
Write-Host "Tag $tag already on $Commit"
}
else {
Write-Host "Creating tag $tag on $Commit"
$tagBody = @{ tag_name = $tag; target = $Commit; message = "ExecuteR $Version" } | ConvertTo-Json
Invoke-RestMethod -Method Post -Uri "$apiBase/tags" -Headers $headers -ContentType 'application/json' -Body $tagBody | Out-Null
}
# ── The release ──────────────────────────────────────────────────────────────
$release = Get-OrNull "$apiBase/releases/tags/$tag"
if (-not $release) {
$notes = if ($NotesFile -and (Test-Path -LiteralPath $NotesFile)) { (Get-Content -Raw -LiteralPath $NotesFile).Trim() } else { '' }
$short = $Commit.Substring(0, [Math]::Min(7, $Commit.Length))
$body = (@($notes, "ExecuteR $Version (commit ``$short``)") | Where-Object { $_ }) -join "`n`n"
Write-Host "Creating release $tag on $owner/$repo"
$releaseBody = @{
tag_name = $tag
target_commitish = $Commit
name = "ExecuteR $Version"
body = $body
draft = $false
prerelease = [bool]$Prerelease
} | ConvertTo-Json
$release = Invoke-RestMethod -Method Post -Uri "$apiBase/releases" -Headers $headers `
-ContentType 'application/json' -Body $body
-ContentType 'application/json' -Body $releaseBody
}
catch {
# Release for this tag may already exist; fetch it.
$release = Invoke-RestMethod -Method Get -Uri "$apiBase/releases/tags/$tag" -Headers $headers
else {
Write-Host "Release $tag exists; adding missing assets"
}
# ── Assets: this version's agent and server files only ───────────────────────
$artifacts = Join-Path $repoRoot 'artifacts'
$assets = Get-ChildItem $artifacts -File |
Where-Object { $_.Name -like "ExecuteR-Agent-$Version*" -or $_.Name -like "ExecuteR-Server-$Version*" }
if (-not $assets) {
throw "No artifacts matching ExecuteR-Agent-$Version* or ExecuteR-Server-$Version* under $artifacts."
$ownFile = '^ExecuteR-(Agent|Server)-' + [regex]::Escape($Version) + '(-.+\.(zip|txt)|\.msi)$'
$assets = @(Get-ChildItem $artifacts -File | Where-Object { $_.Name -match $ownFile })
if ($assets.Count -eq 0) {
throw "No artifacts of $Version (ExecuteR-Agent-$Version-*, ExecuteR-Server-$Version-*, *.msi) under $artifacts."
}
$uploaded = @($release.assets | Where-Object { $_ })
foreach ($asset in $assets) {
if ($uploaded | Where-Object { $_.name -eq $asset.Name -and $_.size -eq $asset.Length }) {
Write-Host "Already uploaded $($asset.Name)"
continue
}
Write-Host "Uploading $($asset.Name)"
$uri = "$apiBase/releases/$($release.id)/assets?name=$([uri]::EscapeDataString($asset.Name))"
Invoke-RestMethod -Method Post -Uri $uri -Headers $headers -InFile $asset.FullName -ContentType 'application/octet-stream' | Out-Null
}
Write-Host "Release $tag published with $($assets.Count) asset(s)."
Write-Host "Release $tag published with $($assets.Count) asset(s): $($giteaUrl.TrimEnd('/'))/$owner/$repo/releases/tag/$tag"
+2 -1
View File
@@ -17,11 +17,12 @@ param(
$ErrorActionPreference = 'Stop'
$repoRoot = Split-Path -Parent $PSScriptRoot
$Version = & (Join-Path $PSScriptRoot 'Get-UnifiedVersion.ps1') -Version $Version
$Version = & (Join-Path $PSScriptRoot 'Get-Version.ps1') -Version $Version
$unitProjects = @(
'tests/Shared.Tests',
'tests/Agent.Tests',
'tests/Agent.Watchdog.Tests',
'tests/Backend.Tests'
)
+21 -2
View File
@@ -168,8 +168,27 @@ fi
if [[ $BUILD_NEEDED -eq 1 ]]; then
[[ -f src/Infra/backend/Dockerfile ]] \
|| die "cannot build: source not present. Clone the repo, or make the registry reachable."
log "Building images from source"
UNIFIED_VERSION="${UNIFIED_VERSION:-$(date -u +%Y.%m.%d.%H%M)}" compose build
# The build version (README, Versioning; scripts/Get-Version.ps1): the
# release tag vX.Y.Z behind HEAD and the commits since it (1.4.0.7, or
# 1.4.0 on the tagged commit); 0.0.0.<commits> without a release tag;
# 0.0.0.0 without git. One to three digits of major: calendar-era tags
# (v2026.10.10.0140) never match.
if [[ -z "${UNIFIED_VERSION:-}" ]]; then
describe="$(git describe --tags --long --match 'v[0-9].[0-9]*.[0-9]*' \
--match 'v[0-9][0-9].[0-9]*.[0-9]*' --match 'v[0-9][0-9][0-9].[0-9]*.[0-9]*' 2>/dev/null || true)"
if [[ $describe =~ ^v([0-9]{1,3}\.[0-9]+\.[0-9]+)-([0-9]+)-g[0-9a-f]+$ ]]; then
UNIFIED_VERSION="${BASH_REMATCH[1]}"
[[ ${BASH_REMATCH[2]} == 0 ]] || UNIFIED_VERSION="$UNIFIED_VERSION.${BASH_REMATCH[2]}"
elif commits="$(git rev-list --count HEAD 2>/dev/null)"; then
UNIFIED_VERSION="0.0.0.$commits"
else
UNIFIED_VERSION="0.0.0.0"
fi
fi
# Exported: the steps below run the images just built, by this tag.
export UNIFIED_VERSION
log "Building images from source ($UNIFIED_VERSION)"
compose build
fi
# ── NATS operator/account keys (generated by the backend image itself) ──────
+149
View File
@@ -0,0 +1,149 @@
<#
.SYNOPSIS
Checks scripts/Get-Version.ps1 and scripts/Get-NextRelease.ps1 (README,
Versioning). Plain asserts, no Pester needed; exits 1 on the first failure.
.DESCRIPTION
The version rules run against canned `git describe` output. Unless -SkipGit,
a throwaway repository (SHA-256, like this one) with a bare "origin" is built
in the temp folder and walked through the seed and the first releases:
calendar-era tags are ignored, no release tag is refused, v0.1.0 plus a
breaking change releases 1.0.0, a re-run of the tagged commit publishes 1.0.0
again, builds after it are 1.0.0.N, a fix releases 1.0.1, docs alone release
nothing, a feat releases 1.1.0. Needs git and node (build/release is installed
with npm ci when missing). Never touches this repository's tags.
.EXAMPLE
./scripts/tests/Test-Versioning.ps1
./scripts/tests/Test-Versioning.ps1 -SkipGit
#>
[CmdletBinding()]
param(
[switch]$SkipGit
)
$ErrorActionPreference = 'Stop'
Set-StrictMode -Version 3.0
$scripts = Split-Path -Parent $PSScriptRoot
$getVersion = Join-Path $scripts 'Get-Version.ps1'
$getNext = Join-Path $scripts 'Get-NextRelease.ps1'
$failures = 0
# The version overrides, and the CI variables semantic-release reads the branch from (in CI they name the
# pipeline's branch, not the scratch repository's), are cleared while the checks run.
$cleared = 'UNIFIED_VERSION', 'GIT_COMMIT_SHORT', 'CI', 'GITHUB_ACTIONS', 'GITEA_ACTIONS', 'GITHUB_REF', 'GITHUB_HEAD_REF',
'GITHUB_BASE_REF', 'GITHUB_EVENT_NAME', 'GITHUB_EVENT_PATH', 'GITHUB_SHA', 'GITHUB_REPOSITORY'
$saved = @{}
foreach ($name in $cleared) {
$saved[$name] = [Environment]::GetEnvironmentVariable($name)
[Environment]::SetEnvironmentVariable($name, $null)
}
function Assert-Equal($Expected, $Actual, [string]$What) {
if ("$Expected" -ceq "$Actual") { Write-Host " ok $What = $Actual" }
else { Write-Host " FAIL $What = '$Actual', expected '$Expected'" -ForegroundColor Red; $script:failures++ }
}
function Assert-Throws([scriptblock]$Action, [string]$Pattern, [string]$What) {
try { & $Action | Out-Null; Write-Host " FAIL $What did not throw" -ForegroundColor Red; $script:failures++ }
catch {
if ("$_" -match $Pattern) { Write-Host " ok $What refused: $_" }
else { Write-Host " FAIL $What threw '$_', expected /$Pattern/" -ForegroundColor Red; $script:failures++ }
}
}
try {
Write-Host 'Version rules'
Assert-Equal '1.4.0.7' (& $getVersion -Describe 'v1.4.0-7-g1a2b3c4') 'seven commits after v1.4.0'
Assert-Equal '1.4.0' (& $getVersion -Describe 'v1.4.0-0-g1a2b3c4') 'the tagged commit'
Assert-Equal '1.4.0.0' (& $getVersion -Describe 'v1.4.0-0-g1a2b3c4' -Format FileVersion) 'file version of a release'
Assert-Equal '1.4.0.7' (& $getVersion -Describe 'v1.4.0-7-g1a2b3c4' -Format FileVersion) 'file version of a build'
Assert-Equal '0.0.0.253' (& $getVersion -Describe '' -Commits 253) 'no release tag'
Assert-Equal '0.0.0' (& $getVersion -Describe '' -Commits 0) 'no git'
Assert-Equal '255.255.65535.65535' (& $getVersion -Version '255.255.65535.65535') 'the largest version'
Assert-Equal '1.5.0' (& $getVersion -Version '1.5.0') 'explicit release'
$env:UNIFIED_VERSION = '2.0.0.3'
Assert-Equal '2.0.0.3' (& $getVersion) 'UNIFIED_VERSION'
Assert-Equal '2.0.0.3' (& (Join-Path $scripts 'Get-UnifiedVersion.ps1')) 'the old entry point'
$env:GIT_COMMIT_SHORT = 'abc1234'
Assert-Equal '2.0.0.3+abc1234' (& $getVersion -Format Informational) 'informational version'
$env:UNIFIED_VERSION = $null
$env:GIT_COMMIT_SHORT = $null
Assert-Throws { & $getVersion -Version '2026.10.10.0140' } 'calendar-era' 'a calendar version'
Assert-Throws { & $getVersion -Version '256.0.0' } 'does not fit' 'major 256'
Assert-Throws { & $getVersion -Version '1.256.0' } 'does not fit' 'minor 256'
Assert-Throws { & $getVersion -Version '1.0.65536' } 'does not fit' 'patch 65536'
Assert-Throws { & $getVersion -Version '1.0.0.65536' } 'does not fit' 'build 65536'
Assert-Throws { & $getVersion -Version '1.0.0-rc.1' } 'no -label' 'a prerelease label'
Assert-Throws { & $getVersion -Version '1.0' } 'no -label' 'two fields'
Assert-Throws { & $getVersion -Describe 'v1.0.0-rc1-3-gabc' } 'not v<major>' 'an odd describe'
if (-not $SkipGit) {
Write-Host 'Releases in a scratch repository'
$temp = Join-Path ([System.IO.Path]::GetTempPath()) "executer-versioning-$([guid]::NewGuid().ToString('N'))"
$origin = Join-Path $temp 'origin.git'
$work = Join-Path $temp 'work'
New-Item -ItemType Directory -Force $temp | Out-Null
try {
function Invoke-Git { & git -C $work @args 2>&1 | Out-Null; if ($LASTEXITCODE -ne 0) { throw "git $args failed ($LASTEXITCODE)" } }
function Add-Commit([string]$Message) {
Add-Content -LiteralPath (Join-Path $work 'history.txt') -Value $Message
Invoke-Git add history.txt
Invoke-Git -c user.name=test -c user.email=test@example.invalid commit -q -m $Message
Invoke-Git push -q origin main
}
function Next { $r = & $getNext -Root $work 6>$null; if ($r) { "$($r.Version) $($r.Type)" } else { 'none' } }
& git init -q --bare --object-format=sha256 --initial-branch=main $origin
& git init -q --object-format=sha256 --initial-branch=main $work
Invoke-Git remote add origin $origin
Invoke-Git config commit.gpgsign false
Invoke-Git config tag.gpgsign false
Add-Commit 'feat: the calendar era'
Invoke-Git tag v2026.10.10.0140
Add-Commit 'fix: still calendar'
Invoke-Git push -q origin --tags
Assert-Equal '0.0.0.2' (& $getVersion -Root $work) 'calendar tags only'
Assert-Throws { & $getNext -Root $work 6>$null } 'No release tag' 'a release without the seed tag'
Invoke-Git tag v0.1.0
Invoke-Git push -q origin v0.1.0
Assert-Equal '0.1.0' (& $getVersion -Root $work) 'the seed commit'
Add-Commit "build(version)!: semantic versions`n`nBREAKING CHANGE: versions are X.Y.Z from release tags."
Assert-Equal '0.1.0.1' (& $getVersion -Root $work) 'a build after the seed'
Assert-Equal '1.0.0 major' (Next) 'the first release after the seed'
Invoke-Git tag v1.0.0
Invoke-Git push -q origin v1.0.0
Assert-Equal '1.0.0' (& $getVersion -Root $work) 'the released commit'
Assert-Equal '1.0.0 rerun' (Next) 'a re-run of the released commit'
Add-Commit 'docs: a note'
Assert-Equal '1.0.0.1' (& $getVersion -Root $work) 'docs after 1.0.0'
Assert-Equal 'none' (Next) 'docs alone'
Add-Commit 'fix: a bug'
Assert-Equal '1.0.0.2' (& $getVersion -Root $work) 'a fix after 1.0.0'
Assert-Equal '1.0.1 patch' (Next) 'a fix'
Add-Commit 'feat: a feature'
Assert-Equal '1.1.0 minor' (Next) 'a feat and a fix'
Invoke-Git checkout -q -b topic
Assert-Throws { & $getNext -Root $work 6>$null } 'did not analyze' 'a branch other than main'
}
finally {
Remove-Item -LiteralPath $temp -Recurse -Force -ErrorAction SilentlyContinue
}
}
}
finally {
foreach ($name in $cleared) { [Environment]::SetEnvironmentVariable($name, $saved[$name]) }
}
if ($failures -gt 0) {
Write-Host "$failures check(s) failed." -ForegroundColor Red
exit 1
}
Write-Host 'All versioning checks passed.'
# The refusals above leave a non-zero $LASTEXITCODE behind; the result is this.
exit 0
+437
View File
@@ -0,0 +1,437 @@
using System.Text;
using Shared.Contracts.Models;
using Shared.Contracts.Serialization;
namespace Agent.Watchdog.Msi;
/// <summary>Timings of an MSI run; shortened in tests.</summary>
public sealed record MsiRunnerSettings
{
/// <summary>Between tries while Windows Installer is busy with another installation (1618).</summary>
public TimeSpan BusyRetryDelay { get; init; } = TimeSpan.FromSeconds(30);
/// <summary>How long a busy Windows Installer is waited out before the run gives up (installer-busy).</summary>
public TimeSpan BusyRetryFor { get; init; } = TimeSpan.FromMinutes(10);
/// <summary>The longest one install may take (real time: Windows Installer's call cannot be cut short).</summary>
public TimeSpan InstallTimeout { get; init; } = WatchdogFiles.MsiInstallTimeout;
}
/// <summary>
/// The detached MSI runner (<c>ExecuteR-Watchdog.exe msi-run &lt;upgradeId&gt;</c>,
/// started from a copy of the watchdog binary in <c>upgrade/msi/runner/</c>).
/// It owns the run in <c>msi.json</c> for as long as it holds
/// <c>msi.lock</c>, and depends on neither service, since the MSI stops both:
/// <list type="number">
/// <item><c>installing</c>: Windows Installer in process (no UI, verbose log
/// <c>&lt;logs&gt;/agent-msi-&lt;version&gt;-&lt;id&gt;.log</c>, <c>REBOOT=ReallySuppress</c>,
/// <c>ALLOWDOWNGRADE=1</c> for a rollback from the console); 0, 3010 and 1641
/// are success; 1618 (another installation) is tried again every 30 s for up
/// to 10 minutes, then installer-busy; anything else, or 30 minutes without an
/// answer, is failed / msi (Windows Installer's transaction leaves the previous
/// version in place);</item>
/// <item><c>verifying</c>: the services the MSI started; a beacon of the new
/// version, connected, written since the install began, before the deadline;</item>
/// <item><c>rolling-back</c>: the MSI of the previous version with
/// <c>ALLOWDOWNGRADE=1</c> (its own log), then the previous version's health;
/// without that MSI the new version stays and the run ends failed.</item>
/// </list>
/// The outcome goes to <c>outcome.json</c>, both services are left running,
/// <c>msi.json</c> goes and the runner exits.
/// </summary>
public sealed class MsiRunner(
WatchdogOptions options,
IAgentService agent,
IAgentService? watchdogService,
IWindowsInstaller installer,
IWatchdogClock clock,
WatchdogLog log,
MsiRunnerSettings? settings = null)
{
public const int LogLines = UpgradeApplier.LogLines;
public const int MaxExits = UpgradeApplier.MaxExits;
public const string InstallProperties = "REBOOT=ReallySuppress";
public const string DowngradeProperties = "REBOOT=ReallySuppress ALLOWDOWNGRADE=1";
private readonly MsiRunnerSettings timings = settings ?? new MsiRunnerSettings();
/// <summary>The new version's last log lines, taken before it is rolled back.</summary>
private List<string> newVersionLog = [];
/// <summary>Why the new version is being rolled back (a resumed rollback says health-deadline).</summary>
private string rollbackReason = AgentUpgradeReasons.HealthDeadline;
/// <summary>Runs (or resumes) the run of <paramref name="upgradeId"/>; 0 unless the run could not be read at all.</summary>
public async Task<int> RunAsync(Guid upgradeId, CancellationToken cancellationToken)
{
using var held = MsiRunLock.TryHold(options.MsiRunnerLockPath);
if (held is null)
{
log.Info($"Another MSI runner owns upgrade {upgradeId}; this one exits");
return 0;
}
var journal = MsiUpgrade.ReadJournal(options);
if (journal is null || journal.UpgradeId != upgradeId)
{
log.Warn($"No MSI run of upgrade {upgradeId} is journalled; nothing to do");
return journal is null ? 1 : 0;
}
log.BeginRun();
journal = journal with { RunnerProcessId = Environment.ProcessId };
Write(journal);
log.Info($"MSI runner {Environment.ProcessId} owns upgrade {journal.UpgradeId}: {journal.FromVersion} -> {journal.Version} ({journal.Phase})");
try
{
while (true)
{
var next = journal.Phase switch
{
WatchdogMsiPhases.Installing => await InstallAsync(journal, cancellationToken),
WatchdogMsiPhases.Verifying => await VerifyAsync(journal, cancellationToken),
WatchdogMsiPhases.RollingBack => await RollBackAsync(journal, cancellationToken),
_ => await EndAsync(journal, AgentUpgradeStates.Failed, AgentUpgradeReasons.Watchdog,
$"The MSI run was in an unknown phase ({journal.Phase}).", log.RunTail(LogLines), cancellationToken),
};
if (next is null)
{
return 0;
}
journal = next;
}
}
catch (Exception exception) when (exception is not OperationCanceledException)
{
// Never leave the device without its services: end the run and say why.
log.Error("The MSI run failed", exception);
await EndAsync(journal, AgentUpgradeStates.Failed, AgentUpgradeReasons.Msi,
$"The MSI run failed ({exception.GetBaseException().Message}).", log.RunTail(LogLines), CancellationToken.None);
return 0;
}
}
/// <summary>Installs the new version's MSI; the journal of the next phase, or null when the run ended.</summary>
private async Task<WatchdogMsiJournal?> InstallAsync(WatchdogMsiJournal journal, CancellationToken cancellationToken)
{
var (held, problem) = MsiUpgrade.Hold(journal.Msi, options);
if (held is null)
{
log.Error($"The MSI of {journal.Version} does not check out ({problem}); nothing was installed");
return await EndAsync(journal, AgentUpgradeStates.Failed, AgentUpgradeReasons.Hash,
$"The MSI of {journal.Version} no longer matched what the agent checked ({problem}); nothing was installed, {journal.FromVersion} stays.",
log.RunTail(LogLines), cancellationToken);
}
journal = journal with { StartedUtc = clock.UtcNow, DeadlineUtc = null };
Write(journal);
var logPath = options.MsiLogPath(journal.Version, journal.UpgradeId);
var properties = journal.Revert ? DowngradeProperties : InstallProperties;
log.Info($"Installing {journal.Version} through Windows Installer ({properties}); its log is {logPath}");
InstallResult result;
int code;
using (held)
{
(result, code, journal) = await InstallWithRetriesAsync(journal, journal.Msi.Path, properties, logPath, cancellationToken);
}
switch (result)
{
case InstallResult.TimedOut:
log.Error($"Windows Installer did not finish installing {journal.Version} within {timings.InstallTimeout.TotalMinutes:0} minutes");
return await EndAsync(journal, AgentUpgradeStates.Failed, AgentUpgradeReasons.Msi,
$"Windows Installer did not finish installing {journal.Version} within {timings.InstallTimeout.TotalMinutes:0} minutes; its log is {logPath}.",
TailMsiLog(logPath), cancellationToken);
case InstallResult.Busy:
log.Error($"Windows Installer stayed busy with another installation for {timings.BusyRetryFor.TotalMinutes:0} minutes; {journal.FromVersion} stays");
return await EndAsync(journal, AgentUpgradeStates.Failed, AgentUpgradeReasons.InstallerBusy,
$"Another installation kept Windows Installer busy (1618) for {timings.BusyRetryFor.TotalMinutes:0} minutes; {journal.FromVersion} stays.",
log.RunTail(LogLines), cancellationToken);
case InstallResult.Failed:
log.Error($"Windows Installer ended with {code} ({MsiExitCodes.Describe(code)}); its transaction left {journal.FromVersion} in place");
return await EndAsync(journal, AgentUpgradeStates.Failed, AgentUpgradeReasons.Msi,
$"Windows Installer ended with {code} ({MsiExitCodes.Describe(code)}) and left {journal.FromVersion} in place; its log is {logPath}.",
TailMsiLog(logPath), cancellationToken);
}
var deadline = clock.UtcNow + (journal.HealthDeadlineSeconds > 0
? TimeSpan.FromSeconds(journal.HealthDeadlineSeconds)
: options.DefaultHealthDeadline);
log.Info(code == MsiExitCodes.Success
? $"Windows Installer installed {journal.Version}; waiting until {deadline:O} for it to report healthy"
: $"Windows Installer installed {journal.Version} ({code}: {MsiExitCodes.Describe(code)}); waiting until {deadline:O} for it to report healthy");
journal = journal with { Phase = WatchdogMsiPhases.Verifying, DeadlineUtc = deadline };
Write(journal);
return journal;
}
private async Task<WatchdogMsiJournal?> VerifyAsync(WatchdogMsiJournal journal, CancellationToken cancellationToken)
{
var deadline = journal.DeadlineUtc ?? clock.UtcNow + options.DefaultHealthDeadline;
var (healthy, exits) = await WaitHealthyAsync(journal.Version, journal.StartedUtc, deadline, cancellationToken);
if (healthy)
{
log.Info($"{journal.Version} is healthy; the upgrade is kept");
return await EndAsync(journal, AgentUpgradeStates.Succeeded, null, null, [], cancellationToken);
}
var reason = exits >= MaxExits ? AgentUpgradeReasons.Crashed : AgentUpgradeReasons.HealthDeadline;
rollbackReason = reason;
newVersionLog = UpgradeApplier.TailAgentLog(options, journal.StartedUtc);
var why = reason == AgentUpgradeReasons.Crashed ? "kept exiting" : "did not report healthy in time";
if (journal.RollbackMsi is null || journal.RollbackVersion is null)
{
log.Error($"{journal.Version} {why}, and no MSI of {journal.FromVersion} was kept to go back to; it stays");
return await EndAsync(journal, AgentUpgradeStates.Failed, reason,
$"{journal.Version} {why}; no MSI of {journal.FromVersion} was kept to go back to, so it stays.",
newVersionLog.Count > 0 ? newVersionLog : log.RunTail(LogLines), cancellationToken);
}
log.Warn($"{journal.Version} {why}; installing the MSI of {journal.RollbackVersion} again");
journal = journal with { Phase = WatchdogMsiPhases.RollingBack, StartedUtc = clock.UtcNow, DeadlineUtc = null, Attempts = 0, LastExitCode = null };
Write(journal);
return journal;
}
private async Task<WatchdogMsiJournal?> RollBackAsync(WatchdogMsiJournal journal, CancellationToken cancellationToken)
{
var lastLog = newVersionLog.Count > 0 ? newVersionLog : log.RunTail(LogLines);
var previous = journal.RollbackVersion ?? journal.FromVersion;
var why = rollbackReason == AgentUpgradeReasons.Crashed ? "kept exiting" : "did not report healthy in time";
var rollback = journal.RollbackMsi;
var (held, problem) = rollback is null ? (null, "it is not there") : MsiUpgrade.Hold(rollback, options);
if (rollback is null || held is null)
{
log.Error($"The MSI of {previous} does not check out ({problem}); {journal.Version} stays");
return await EndAsync(journal, AgentUpgradeStates.Failed, rollbackReason,
$"{journal.Version} {why}, and the MSI of {previous} to go back to no longer checked out; {journal.Version} stays.",
lastLog, cancellationToken);
}
var started = clock.UtcNow;
journal = journal with { StartedUtc = started };
Write(journal);
var logPath = options.MsiLogPath(previous, journal.UpgradeId);
log.Info($"Installing {previous} again through Windows Installer ({DowngradeProperties}); its log is {logPath}");
InstallResult result;
int code;
using (held)
{
(result, code, journal) = await InstallWithRetriesAsync(journal, rollback.Path, DowngradeProperties, logPath, cancellationToken);
}
if (result != InstallResult.Installed)
{
var what = result switch
{
InstallResult.TimedOut => $"did not finish within {timings.InstallTimeout.TotalMinutes:0} minutes",
InstallResult.Busy => "could not run: Windows Installer stayed busy (1618)",
_ => $"ended with {code} ({MsiExitCodes.Describe(code)})",
};
log.Error($"Installing {previous} again {what}; {journal.Version} stays");
return await EndAsync(journal, AgentUpgradeStates.Failed, rollbackReason,
$"{journal.Version} {why}, and installing the MSI of {previous} again {what}; its log is {logPath}.",
lastLog, cancellationToken);
}
// The new version's beacon must not speak for the previous one.
WatchdogFileStore.TryDelete(options.HealthPath);
var (back, _) = await WaitHealthyAsync(previous, started, clock.UtcNow + options.RollbackHealthDeadline, cancellationToken);
log.Info(back
? $"{previous} is back and healthy"
: $"{previous} is back; it has not reported healthy within {options.RollbackHealthDeadline.TotalMinutes:0} minutes");
return await EndAsync(journal, AgentUpgradeStates.RolledBack, rollbackReason,
$"{journal.Version} {why}; the MSI of {previous} was installed again{(back ? "" : ", and it has not reported healthy yet")}.",
lastLog, cancellationToken);
}
private enum InstallResult
{
Installed,
Failed,
Busy,
TimedOut,
}
/// <summary>One install, tried again while Windows Installer answers 1618; every attempt and answer is journalled.</summary>
private async Task<(InstallResult Result, int Code, WatchdogMsiJournal Journal)> InstallWithRetriesAsync(
WatchdogMsiJournal journal, string package, string properties, string logPath, CancellationToken cancellationToken)
{
var giveUp = clock.UtcNow + timings.BusyRetryFor;
while (true)
{
journal = journal with { Attempts = journal.Attempts + 1 };
Write(journal);
var code = await InstallOnceAsync(package, properties, logPath);
if (code is not { } answer)
{
return (InstallResult.TimedOut, -1, journal);
}
journal = journal with { LastExitCode = answer };
Write(journal);
if (answer != MsiExitCodes.InstallerBusy)
{
return (MsiExitCodes.Succeeded(answer) ? InstallResult.Installed : InstallResult.Failed, answer, journal);
}
if (clock.UtcNow + timings.BusyRetryDelay > giveUp)
{
return (InstallResult.Busy, answer, journal);
}
log.Warn($"Windows Installer is busy with another installation (1618); trying again in {timings.BusyRetryDelay.TotalSeconds:0} s (attempt {journal.Attempts})");
await clock.DelayAsync(timings.BusyRetryDelay, cancellationToken);
}
}
/// <summary>
/// Windows Installer's answer, or null when it gave none within the install
/// timeout. The call runs on a background thread of its own (the three calls
/// on the same thread); one cut off by the timeout ends with the process.
/// </summary>
private async Task<int?> InstallOnceAsync(string package, string properties, string logPath)
{
var install = Task.Factory.StartNew(() =>
{
installer.SetSilentUI();
installer.EnableLog(logPath);
return installer.InstallProduct(package, properties);
}, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default);
try
{
return await install.WaitAsync(timings.InstallTimeout);
}
catch (TimeoutException)
{
return null;
}
}
/// <summary>
/// Healthy: a beacon of <paramref name="version"/>, connected, written since
/// <paramref name="since"/>, before <paramref name="deadline"/>. The agent
/// found not running is started again; the third time ends the wait.
/// </summary>
private async Task<(bool Healthy, int Exits)> WaitHealthyAsync(
string version, DateTimeOffset since, DateTimeOffset deadline, CancellationToken cancellationToken)
{
var exits = 0;
var reported = clock.UtcNow;
while (clock.UtcNow < deadline && exits < MaxExits)
{
if (MsiUpgrade.IsHealthy(options, version, since))
{
return (true, exits);
}
if (clock.UtcNow - reported >= TimeSpan.FromMinutes(1))
{
reported = clock.UtcNow;
log.Info($"Waiting for {version} to report healthy");
}
var status = await agent.QueryAsync(cancellationToken);
if (status.State == AgentRunState.Stopped)
{
exits++;
if (exits >= MaxExits)
{
log.Warn($"{version} exited {exits} times; the wait ends");
break;
}
log.Warn($"{version} is not running (exit {exits}); starting it");
try
{
await agent.StartAsync(cancellationToken);
}
catch (Exception exception) when (exception is not OperationCanceledException)
{
log.Warn("The agent did not start", exception);
}
}
await clock.DelayAsync(options.PollInterval, cancellationToken);
}
return (MsiUpgrade.IsHealthy(options, version, since), exits);
}
/// <summary>Both services running, the outcome written, the run's files gone; null (the run is over).</summary>
private async Task<WatchdogMsiJournal?> EndAsync(
WatchdogMsiJournal journal, string result, string? reason, string? detail, IReadOnlyList<string> lastLog, CancellationToken cancellationToken)
{
await EnsureRunningAsync(agent, "agent", cancellationToken);
if (watchdogService is not null)
{
await EnsureRunningAsync(watchdogService, "watchdog", cancellationToken);
}
MsiUpgrade.WriteOutcome(options, journal, result, reason, detail, lastLog, clock.UtcNow);
log.Info($"Upgrade {journal.UpgradeId} to {journal.Version} ended: {result}{(reason is null ? "" : $" ({reason})")}");
WatchdogFileStore.TryDelete(options.MsiJournalPath);
MsiUpgrade.DeletePackages(options);
return null;
}
private async Task EnsureRunningAsync(IAgentService service, string name, CancellationToken cancellationToken)
{
try
{
var status = await service.QueryAsync(cancellationToken);
if (status.State != AgentRunState.NotInstalled && !status.IsRunning)
{
await service.StartAsync(cancellationToken);
}
}
catch (Exception exception) when (exception is not OperationCanceledException)
{
log.Warn($"The {name} service could not be started", exception);
}
}
private void Write(WatchdogMsiJournal journal)
=> WatchdogFileStore.Write(options.MsiJournalPath, journal, WatchdogJsonContext.Default.WatchdogMsiJournal);
/// <summary>The last lines of Windows Installer's log (UTF-16 with a BOM, or whatever its BOM says).</summary>
internal static List<string> TailMsiLog(string path)
{
try
{
if (!File.Exists(path))
{
return [];
}
using var stream = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite | FileShare.Delete);
using var reader = new StreamReader(stream, Encoding.Unicode, detectEncodingFromByteOrderMarks: true);
var lines = new Queue<string>();
while (reader.ReadLine() is { } line)
{
if (line.Length == 0)
{
continue;
}
lines.Enqueue(line.Length <= 2000 ? line : line[..2000]);
if (lines.Count > LogLines)
{
lines.Dequeue();
}
}
return [.. lines];
}
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
{
return [];
}
}
}
+200
View File
@@ -0,0 +1,200 @@
using System.ComponentModel;
using System.Runtime.InteropServices;
using System.Runtime.Versioning;
using System.Text;
using Shared.Contracts.Models;
namespace Agent.Watchdog.Msi;
/// <summary>Starts the MSI runner apart from the watchdog (the MSI stops the watchdog service); replaced in tests.</summary>
public interface IMsiRunnerLauncher
{
/// <summary>Starts <c>msi-run &lt;upgradeId&gt;</c>; null when it started, otherwise why not.</summary>
string? Launch(Guid upgradeId);
}
/// <summary>
/// Copies the watchdog's own binary to <c>upgrade/msi/runner/</c> (the MSI
/// replaces the installed one while it runs) and starts the copy detached: its
/// own process group, no console, out of any job the service runs in, so
/// stopping both services (which the MSI does) never takes it down.
/// </summary>
[SupportedOSPlatform("windows")]
public sealed class WindowsMsiRunnerLauncher(WatchdogOptions options, WatchdogLog log) : IMsiRunnerLauncher
{
public string? Launch(Guid upgradeId)
{
if (options.SelfPath is not { } self || !File.Exists(self)
|| !string.Equals(Path.GetFileName(self), Shared.Infrastructure.Constants.Branding.WatchdogExecutableName, StringComparison.OrdinalIgnoreCase))
{
return "the watchdog does not run from its own binary, so it has none to start the MSI runner from";
}
string runner;
try
{
Directory.CreateDirectory(options.MsiRunnerDirectory);
runner = Path.Combine(options.MsiRunnerDirectory, Path.GetFileName(self));
File.Copy(self, runner, overwrite: true);
}
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
{
return $"the watchdog binary could not be copied for the MSI runner ({exception.Message})";
}
string[] arguments =
[
WatchdogMsiFiles.RunVerb, upgradeId.ToString("D"),
"--install-dir", options.InstallDirectory,
"--data-dir", options.DataDirectory,
];
try
{
var processId = DetachedProcess.Start(runner, arguments, options.MsiRunnerDirectory);
log.Info($"MSI runner started (process {processId}) from {runner}");
return null;
}
catch (Win32Exception exception)
{
return $"the MSI runner did not start ({exception.Message})";
}
}
}
/// <summary>CreateProcessW for a process that outlives its parent and the parent's service.</summary>
[SupportedOSPlatform("windows")]
public static partial class DetachedProcess
{
private const uint DetachedProcessFlag = 0x00000008;
private const uint CreateNewProcessGroup = 0x00000200;
private const uint CreateBreakawayFromJob = 0x01000000;
private const uint CreateNoWindow = 0x08000000;
private const int ErrorAccessDenied = 5;
/// <summary>
/// Starts <paramref name="executable"/> with DETACHED_PROCESS |
/// CREATE_NEW_PROCESS_GROUP | CREATE_BREAKAWAY_FROM_JOB | CREATE_NO_WINDOW
/// (without the breakaway when the job refuses it) and returns its process id.
/// </summary>
public static unsafe int Start(string executable, IReadOnlyList<string> arguments, string workingDirectory)
{
var commandLine = BuildCommandLine(executable, arguments);
var flags = DetachedProcessFlag | CreateNewProcessGroup | CreateNoWindow;
foreach (var breakaway in new[] { true, false })
{
var buffer = (commandLine + '\0').ToCharArray();
var startup = new StartupInfo { cb = sizeof(StartupInfo) };
fixed (char* line = buffer)
{
if (CreateProcessW(executable, line, 0, 0, false, flags | (breakaway ? CreateBreakawayFromJob : 0), 0, workingDirectory,
ref startup, out var info))
{
CloseHandle(info.hThread);
CloseHandle(info.hProcess);
return info.dwProcessId;
}
}
var error = Marshal.GetLastPInvokeError();
if (!breakaway || error != ErrorAccessDenied)
{
throw new Win32Exception(error);
}
// A job that does not allow breakaway: the process stays in it, still detached.
}
throw new Win32Exception(ErrorAccessDenied);
}
/// <summary>Quotes each argument the way CommandLineToArgvW reads it back.</summary>
public static string BuildCommandLine(string executable, IReadOnlyList<string> arguments)
{
var builder = new StringBuilder();
Append(builder, executable);
foreach (var argument in arguments)
{
builder.Append(' ');
Append(builder, argument);
}
return builder.ToString();
}
private static void Append(StringBuilder builder, string argument)
{
if (argument.Length > 0 && !argument.Any(c => c is ' ' or '\t' or '"'))
{
builder.Append(argument);
return;
}
builder.Append('"');
var backslashes = 0;
foreach (var c in argument)
{
if (c == '\\')
{
backslashes++;
continue;
}
builder.Append('\\', c == '"' ? backslashes * 2 + 1 : backslashes);
backslashes = 0;
builder.Append(c);
}
builder.Append('\\', backslashes * 2);
builder.Append('"');
}
[StructLayout(LayoutKind.Sequential)]
private struct StartupInfo
{
public int cb;
public nint lpReserved;
public nint lpDesktop;
public nint lpTitle;
public int dwX;
public int dwY;
public int dwXSize;
public int dwYSize;
public int dwXCountChars;
public int dwYCountChars;
public int dwFillAttribute;
public int dwFlags;
public short wShowWindow;
public short cbReserved2;
public nint lpReserved2;
public nint hStdInput;
public nint hStdOutput;
public nint hStdError;
}
[StructLayout(LayoutKind.Sequential)]
private struct ProcessInformation
{
public nint hProcess;
public nint hThread;
public int dwProcessId;
public int dwThreadId;
}
[LibraryImport("kernel32.dll", EntryPoint = "CreateProcessW", StringMarshalling = StringMarshalling.Utf16, SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
private static unsafe partial bool CreateProcessW(
string applicationName,
char* commandLine,
nint processAttributes,
nint threadAttributes,
[MarshalAs(UnmanagedType.Bool)] bool inheritHandles,
uint creationFlags,
nint environment,
string currentDirectory,
ref StartupInfo startupInfo,
out ProcessInformation processInformation);
[LibraryImport("kernel32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
private static partial bool CloseHandle(nint handle);
}
+346
View File
@@ -0,0 +1,346 @@
using Shared.Contracts.Models;
using Shared.Contracts.Serialization;
namespace Agent.Watchdog.Msi;
/// <summary>
/// The watchdog's side of an MSI upgrade (hand-over kind <c>msi</c>): it checks
/// the MSIs the agent handed over (in <c>upgrade/msi/</c>, size and SHA-256),
/// journals the run in <c>msi.json</c> and starts the detached runner
/// (<see cref="MsiRunner"/>), which does the rest while the MSI stops and starts
/// both services. Whichever watchdog runs next (the new version's, started by the
/// MSI) leaves a live runner alone; a runner that died is started again from the
/// journal, at most <see cref="MaxResumes"/> times, or, when it died installing
/// and the new version already reports healthy, it resumes in <c>verifying</c>.
/// No file is swapped and the agent is never started while a run is pending:
/// that is the runner's to decide.
/// </summary>
public sealed class MsiUpgrade(WatchdogOptions options, IMsiRunnerLauncher? launcher, IAgentService agent, IWatchdogClock clock, WatchdogLog log)
{
public const int MaxResumes = 3;
private const string Mismatch = "its size or SHA-256 does not match";
/// <summary>A runner this watchdog just started is given this long to take its lock.</summary>
public static readonly TimeSpan LaunchGrace = TimeSpan.FromMinutes(1);
private DateTimeOffset launchedUtc = DateTimeOffset.MinValue;
/// <summary>An MSI run is journalled: the watchdog neither supervises the agent nor takes another hand-over.</summary>
public bool Pending => File.Exists(options.MsiJournalPath);
/// <summary>Checks the hand-over's MSIs, journals the run and starts the runner; the run ends here (outcome written) when it cannot start.</summary>
public async Task StartAsync(WatchdogHandover handover, CancellationToken cancellationToken)
{
log.BeginRun();
log.Info($"Upgrade {handover.UpgradeId} handed over: {handover.FromVersion} -> {handover.Version}{(handover.Revert ? " (rollback)" : "")} through its MSI");
WatchdogFileStore.TryDelete(options.HandoverPath);
var journal = new WatchdogMsiJournal(handover.UpgradeId, handover.Version, handover.FromVersion, WatchdogMsiPhases.Installing,
handover.Msi ?? new WatchdogFile("", 0, ""), null, null, handover.HealthDeadlineSeconds, RunnerProcessId: 0, Attempts: 0,
LastExitCode: null, clock.UtcNow, DeadlineUtc: null, handover.Revert);
var detail = handover.Msi is null ? "the hand-over names no MSI" : Check(handover.Msi, options);
if (detail is not null)
{
log.Error($"The MSI handed over does not check out ({detail}); nothing was changed");
WriteOutcome(options, journal, AgentUpgradeStates.Failed,
detail == Mismatch ? AgentUpgradeReasons.Hash : AgentUpgradeReasons.Package,
$"The MSI of {handover.Version} did not check out ({detail}); nothing was changed.", log.RunTail(UpgradeApplier.LogLines), clock.UtcNow);
return;
}
if (handover.RollbackMsi is { } rollback && handover.RollbackVersion is { } rollbackVersion)
{
if (Check(rollback, options) is { } rollbackProblem)
{
log.Warn($"The MSI of {rollbackVersion} to go back to does not check out ({rollbackProblem}); this upgrade cannot be rolled back automatically");
}
else
{
journal = journal with { RollbackMsi = rollback, RollbackVersion = rollbackVersion };
}
}
if (journal.RollbackMsi is null)
{
log.Warn($"No MSI of {handover.FromVersion} is kept: a {handover.Version} that does not report healthy stays");
}
if (launcher is null)
{
log.Error("This watchdog cannot run Windows Installer (not Windows); nothing was changed");
WriteOutcome(options, journal, AgentUpgradeStates.Failed, AgentUpgradeReasons.Unsupported,
"MSI upgrades run only on Windows; nothing was changed.", log.RunTail(UpgradeApplier.LogLines), clock.UtcNow);
return;
}
Write(journal);
await LaunchAsync(journal, cancellationToken);
}
/// <summary>Each pass (and at start) while a run is journalled: a live runner is left alone, a dead one resumed or the run ended.</summary>
public async Task CheckAsync(CancellationToken cancellationToken)
{
var journal = ReadJournal(options);
if (journal is null)
{
if (Pending && !RunnerAlive)
{
var id = WatchdogFileStore.ReadUpgradeId(options.MsiJournalPath);
log.Error($"The MSI run's journal does not read; the run ends (upgrade {id?.ToString() ?? "unknown"})");
WatchdogFileStore.TryDelete(options.MsiJournalPath);
if (id is { } upgradeId)
{
WatchdogFileStore.Write(options.OutcomePath,
new WatchdogOutcome(upgradeId, AgentUpgradeStates.Failed, AgentUpgradeReasons.Watchdog, "", "", log.RunTail(UpgradeApplier.LogLines),
clock.UtcNow, "The MSI run's journal could not be read."),
WatchdogJsonContext.Default.WatchdogOutcome);
}
await EnsureAgentRunningAsync(cancellationToken);
}
return;
}
if (RunnerAlive || clock.UtcNow - launchedUtc < LaunchGrace)
{
return;
}
log.BeginRun();
if (journal.Resumes >= MaxResumes || launcher is null)
{
log.Error($"The MSI runner of upgrade {journal.UpgradeId} stopped {journal.Resumes + 1} times before the run ended; it is given up");
await EnsureAgentRunningAsync(cancellationToken);
WriteOutcome(options, journal, AgentUpgradeStates.Failed, AgentUpgradeReasons.Msi,
$"The MSI runner stopped before the upgrade to {journal.Version} ended, {journal.Resumes + 1} times.", log.RunTail(UpgradeApplier.LogLines), clock.UtcNow);
WatchdogFileStore.TryDelete(options.MsiJournalPath);
DeletePackages(options);
return;
}
if (journal.Phase == WatchdogMsiPhases.Installing && IsHealthy(options, journal.Version, journal.StartedUtc))
{
// Installed (the MSI started the new version, which reports healthy)
// before the runner could say so: it goes on from verifying.
var deadline = journal.DeadlineUtc is { } at && at > clock.UtcNow ? at : clock.UtcNow + UpgradeApplier.RecheckWindow;
log.Warn($"The MSI runner stopped, but {journal.Version} is installed and healthy; the run goes on from verifying");
journal = journal with { Phase = WatchdogMsiPhases.Verifying, DeadlineUtc = deadline };
}
else
{
log.Warn($"The MSI runner of upgrade {journal.UpgradeId} stopped in {journal.Phase}; it is started again (resume {journal.Resumes + 1} of {MaxResumes})");
}
journal = journal with { Resumes = journal.Resumes + 1 };
Write(journal);
await LaunchAsync(journal, cancellationToken);
}
/// <summary>Removes the runner's binary copy, unless a run is pending.</summary>
public void Tidy()
{
if (!Pending)
{
SwapPaths.TryDeleteDirectory(options.MsiRunnerDirectory);
}
}
private async Task LaunchAsync(WatchdogMsiJournal journal, CancellationToken cancellationToken)
{
if (launcher!.Launch(journal.UpgradeId) is { } problem)
{
log.Error($"The MSI runner could not be started: {problem}; nothing was installed");
await EnsureAgentRunningAsync(cancellationToken);
WriteOutcome(options, journal, AgentUpgradeStates.Failed, AgentUpgradeReasons.Watchdog,
$"The watchdog could not start Windows Installer: {problem}.", log.RunTail(UpgradeApplier.LogLines), clock.UtcNow);
WatchdogFileStore.TryDelete(options.MsiJournalPath);
DeletePackages(options);
return;
}
launchedUtc = clock.UtcNow;
log.Info($"Upgrade {journal.UpgradeId}: the MSI runner installs {journal.Version}; Windows Installer stops and starts both services");
}
private bool RunnerAlive => MsiRunLock.IsHeld(options.MsiRunnerLockPath);
private async Task EnsureAgentRunningAsync(CancellationToken cancellationToken)
{
try
{
var status = await agent.QueryAsync(cancellationToken);
if (status.State != AgentRunState.NotInstalled && !status.IsRunning)
{
await agent.StartAsync(cancellationToken);
}
}
catch (Exception exception) when (exception is not OperationCanceledException || !cancellationToken.IsCancellationRequested)
{
log.Warn("The agent could not be started", exception);
}
}
private void Write(WatchdogMsiJournal journal)
=> WatchdogFileStore.Write(options.MsiJournalPath, journal, WatchdogJsonContext.Default.WatchdogMsiJournal);
public static WatchdogMsiJournal? ReadJournal(WatchdogOptions options)
=> WatchdogFileStore.Read(options.MsiJournalPath, WatchdogJsonContext.Default.WatchdogMsiJournal);
/// <summary>Null when <paramref name="file"/> is an MSI in <c>upgrade/msi/</c> of the size and SHA-256 named; otherwise what is wrong.</summary>
public static string? Check(WatchdogFile file, WatchdogOptions options)
{
if (string.IsNullOrWhiteSpace(file.Path) || !Path.IsPathFullyQualified(file.Path)
|| !SwapPaths.IsInside(file.Path, options.MsiDirectory)
|| !string.Equals(Path.GetExtension(file.Path), ".msi", StringComparison.OrdinalIgnoreCase))
{
return "it is not an MSI in the data folder's upgrade/msi folder";
}
if (!File.Exists(file.Path))
{
return "it is not there";
}
try
{
return new FileInfo(file.Path).Length == file.Size
&& string.Equals(SwapPaths.Sha256(file.Path), file.Sha256, StringComparison.OrdinalIgnoreCase)
? null
: Mismatch;
}
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
{
return $"it cannot be read ({exception.Message})";
}
}
/// <summary>
/// <see cref="Check"/>, then the file opened sharing read only and its
/// SHA-256 checked again from that handle: held while Windows Installer
/// installs it, nothing can write, replace or rename it meanwhile, so what
/// installs is what was checked. Null with the problem when it does not check out.
/// </summary>
public static (FileStream? Held, string? Problem) Hold(WatchdogFile file, WatchdogOptions options)
{
if (Check(file, options) is { } problem)
{
return (null, problem);
}
FileStream? held = null;
try
{
held = new FileStream(file.Path, FileMode.Open, FileAccess.Read, FileShare.Read);
var sha = Convert.ToHexStringLower(System.Security.Cryptography.SHA256.HashData(held));
held.Position = 0;
if (held.Length == file.Size && string.Equals(sha, file.Sha256, StringComparison.OrdinalIgnoreCase))
{
return (held, null);
}
held.Dispose();
return (null, Mismatch);
}
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
{
held?.Dispose();
return (null, $"it cannot be held open ({exception.Message})");
}
}
/// <summary>A beacon of <paramref name="version"/>, connected, written at or after <paramref name="since"/>.</summary>
public static bool IsHealthy(WatchdogOptions options, string version, DateTimeOffset since)
=> WatchdogFileStore.Read(options.HealthPath, WatchdogJsonContext.Default.WatchdogHealthBeacon) is { Connected: true } beacon
&& beacon.AtUtc >= since
&& AgentVersions.AreSame(beacon.Version, version);
public static void WriteOutcome(WatchdogOptions options, WatchdogMsiJournal journal, string result, string? reason, string? detail,
IReadOnlyList<string> lastLog, DateTimeOffset now)
=> WatchdogFileStore.Write(options.OutcomePath,
new WatchdogOutcome(journal.UpgradeId, result, reason, journal.Version, journal.FromVersion,
[.. lastLog.Skip(Math.Max(0, lastLog.Count - UpgradeApplier.LogLines))], now, detail),
WatchdogJsonContext.Default.WatchdogOutcome);
/// <summary>The MSIs of a finished run (kept copies live in <c>&lt;data&gt;/packages</c>); the runner's binary stays until the next tidy.</summary>
public static void DeletePackages(WatchdogOptions options)
{
if (!Directory.Exists(options.MsiDirectory))
{
return;
}
foreach (var entry in new DirectoryInfo(options.MsiDirectory).GetFileSystemInfos())
{
if (string.Equals(entry.Name, WatchdogMsiFiles.RunnerFolder, StringComparison.OrdinalIgnoreCase))
{
continue;
}
try
{
if (entry is DirectoryInfo folder)
{
folder.Delete(recursive: true);
}
else
{
entry.Delete();
}
}
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
{
}
}
}
}
/// <summary>
/// <c>msi.lock</c>, held open with no sharing by the runner for as long as it
/// lives: Windows releases it when the process ends however it ends, so a lock
/// nobody holds is a runner that is gone (a process id can be reused, a lock
/// cannot outlive its holder).
/// </summary>
public static class MsiRunLock
{
public static FileStream? TryHold(string path)
{
try
{
Directory.CreateDirectory(Path.GetDirectoryName(path)!);
return new FileStream(path, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None);
}
catch (IOException)
{
return null;
}
}
public static bool IsHeld(string path)
{
if (!File.Exists(path))
{
return false;
}
try
{
using var probe = new FileStream(path, FileMode.Open, FileAccess.ReadWrite, FileShare.None);
return false;
}
catch (FileNotFoundException)
{
return false;
}
catch (IOException)
{
return true;
}
catch (UnauthorizedAccessException)
{
return false;
}
}
}
+100
View File
@@ -0,0 +1,100 @@
using System.Runtime.InteropServices;
namespace Agent.Watchdog.Msi;
/// <summary>
/// The three Windows Installer calls an MSI upgrade makes, in process (msi.dll,
/// which is what msiexec.exe itself calls; never msiexec, never PowerShell).
/// Replaced in tests, which never run an MSI.
/// </summary>
public interface IWindowsInstaller
{
/// <summary><c>MsiSetInternalUI(INSTALLUILEVEL_NONE)</c>: no UI at all (msiexec's <c>/qn</c>).</summary>
void SetSilentUI();
/// <summary><c>MsiEnableLogW</c>: msiexec's <c>/l*v</c> into <paramref name="logPath"/>, appended to.</summary>
void EnableLog(string logPath);
/// <summary><c>MsiInstallProductW</c>: installs the package and returns Windows Installer's own code (what msiexec would exit with).</summary>
int InstallProduct(string package, string commandLine);
}
/// <summary>Windows Installer's answers an MSI run acts on.</summary>
public static class MsiExitCodes
{
public const int Success = 0;
/// <summary>ERROR_SUCCESS_REBOOT_REQUIRED: installed; a restart finishes it.</summary>
public const int SuccessRebootRequired = 3010;
/// <summary>ERROR_SUCCESS_REBOOT_INITIATED: installed; Windows Installer started a restart.</summary>
public const int SuccessRebootInitiated = 1641;
/// <summary>ERROR_INSTALL_ALREADY_RUNNING: another installation is under way; try again later.</summary>
public const int InstallerBusy = 1618;
/// <summary>ERROR_INSTALL_USEREXIT.</summary>
public const int UserExit = 1602;
/// <summary>ERROR_INSTALL_FAILURE: a fatal error; Windows Installer rolled its transaction back.</summary>
public const int InstallFailure = 1603;
/// <summary>ERROR_INSTALL_PACKAGE_INVALID.</summary>
public const int PackageInvalid = 1620;
/// <summary>ERROR_PRODUCT_VERSION: a newer version is installed (a downgrade without ALLOWDOWNGRADE=1).</summary>
public const int ProductVersion = 1638;
/// <summary>Done, done with a restart pending, or done with a restart started.</summary>
public static bool Succeeded(int code) => code is Success or SuccessRebootRequired or SuccessRebootInitiated;
public static string Describe(int code) => code switch
{
Success => "success",
SuccessRebootRequired => "success, a restart is required to finish",
SuccessRebootInitiated => "success, a restart was started",
InstallerBusy => "another installation is in progress",
UserExit => "the installation was cancelled",
InstallFailure => "a fatal error during installation",
PackageInvalid => "the package could not be opened",
ProductVersion => "another version of the product is already installed",
1619 => "the package could not be opened",
1625 => "the installation is forbidden by system policy",
1633 => "the package is not supported on this platform",
_ => "see the Windows Installer log",
};
}
/// <summary>msi.dll through P/Invoke.</summary>
public sealed partial class WindowsInstallerApi : IWindowsInstaller
{
/// <summary>INSTALLUILEVEL_NONE.</summary>
private const int InstallUiLevelNone = 2;
/// <summary>
/// msiexec's <c>/l*v</c> (msi.h INSTALLLOGMODE_*): FATALEXIT 0x1, ERROR 0x2,
/// WARNING 0x4, USER 0x8, INFO 0x10, RESOLVESOURCE 0x40, OUTOFDISKSPACE 0x80,
/// ACTIONSTART 0x100, ACTIONDATA 0x200, PROPERTYDUMP 0x400, COMMONDATA 0x800
/// (the <c>*</c> kinds, source resolution included) and VERBOSE 0x1000.
/// </summary>
public const uint LogEverythingVerbose = 0x1FDF;
/// <summary>INSTALLLOGATTRIBUTES_APPEND: a retry adds to the run's log rather than replacing it.</summary>
private const uint LogAttributesAppend = 0x1;
public void SetSilentUI() => MsiSetInternalUI(InstallUiLevelNone, 0);
public void EnableLog(string logPath)
{
Directory.CreateDirectory(Path.GetDirectoryName(logPath)!);
var result = MsiEnableLogW(LogEverythingVerbose, logPath, LogAttributesAppend);
if (result != 0)
{
throw new IOException($"Windows Installer did not accept the log file {logPath} (error {result}).");
}
}
public int InstallProduct(string package, string commandLine) => (int)MsiInstallProductW(package, commandLine);
[LibraryImport("msi.dll")]
private static partial int MsiSetInternalUI(int level, nint window);
[LibraryImport("msi.dll", StringMarshalling = StringMarshalling.Utf16)]
private static partial uint MsiEnableLogW(uint mode, string file, uint attributes);
[LibraryImport("msi.dll", StringMarshalling = StringMarshalling.Utf16)]
private static partial uint MsiInstallProductW(string package, string commandLine);
}
+86 -12
View File
@@ -34,7 +34,9 @@ public enum PassResult
/// brings a newer one; or <c>rolling-back</c>: the new files go to <c>.failed/&lt;id&gt;/</c>, the previous ones come back,
/// the previous agent starts, and the outcome says rolled-back with the new version's last log lines.</item>
/// </list>
/// The watchdog's own binary is never part of the swap.
/// The watchdog's own binary is never part of the swap. A hand-over of kind
/// <c>msi</c> is not swapped at all: it goes to <see cref="Msi.MsiUpgrade"/>,
/// whose detached runner installs it through Windows Installer.
/// </summary>
public sealed class UpgradeApplier(
WatchdogOptions options,
@@ -42,7 +44,8 @@ public sealed class UpgradeApplier(
IWatchdogClock clock,
WatchdogLog log,
ISmokeTest? smoke = null,
IWatchdogRestart? restart = null)
IWatchdogRestart? restart = null,
Msi.MsiUpgrade? msi = null)
{
public const int MaxAttempts = 3;
public const int MoveAttempts = 10;
@@ -63,7 +66,7 @@ public sealed class UpgradeApplier(
private string Install => options.InstallDirectory;
/// <summary>An upgrade waits or is under way: the agent is not supervised meanwhile.</summary>
public bool Pending => File.Exists(options.HandoverPath) || File.Exists(options.JournalPath);
public bool Pending => File.Exists(options.HandoverPath) || File.Exists(options.JournalPath) || File.Exists(options.MsiJournalPath);
/// <summary>The watchdog's own binary relative to the install folder, when it runs from there.</summary>
private string? SelfRelative => options.SelfPath is { } self && SwapPaths.IsInside(self, Install)
@@ -93,7 +96,15 @@ public sealed class UpgradeApplier(
/// <summary>One pass: takes a hand-over, or carries a waiting run on. Idle when there is nothing to do.</summary>
public async Task<PassResult> PassAsync(CancellationToken cancellationToken)
{
TakeHandover();
if (await MsiPassAsync(cancellationToken))
{
return PassResult.Worked;
}
if (await TakeHandoverAsync(cancellationToken))
{
return PassResult.Worked;
}
var journal = ReadJournal();
if (journal is null)
@@ -132,6 +143,11 @@ public sealed class UpgradeApplier(
/// </summary>
public async Task<PassResult> RecoverAsync(CancellationToken cancellationToken)
{
if (await MsiPassAsync(cancellationToken))
{
return PassResult.Worked;
}
var journal = ReadJournal();
if (journal is null)
{
@@ -217,6 +233,7 @@ public sealed class UpgradeApplier(
/// </summary>
public void Tidy()
{
msi?.Tidy();
if (Pending || File.Exists(options.TakenHandoverPath))
{
return;
@@ -233,12 +250,62 @@ public sealed class UpgradeApplier(
}
}
/// <summary>handover.json → handover.taken.json and a new journal; a duplicate or unreadable one is set aside.</summary>
private void TakeHandover()
/// <summary>
/// While an MSI run is journalled it is the only work (the runner's, or
/// resuming a runner that died); true then.
/// </summary>
private async Task<bool> MsiPassAsync(CancellationToken cancellationToken)
{
if (!File.Exists(options.MsiJournalPath))
{
return false;
}
if (msi is null)
{
// A watchdog without MSI support leaves the run alone, and the agent too.
return true;
}
await msi.CheckAsync(cancellationToken);
return true;
}
/// <summary>
/// handover.json → handover.taken.json and a new journal; a duplicate or
/// unreadable one is set aside. An MSI hand-over goes to the MSI upgrade
/// (true: this pass worked on it).
/// </summary>
private async Task<bool> TakeHandoverAsync(CancellationToken cancellationToken)
{
if (TakeHandover() is not { Kind: WatchdogHandoverKinds.Msi } handover)
{
return false;
}
if (msi is null)
{
log.Error($"Upgrade {handover.UpgradeId} is an MSI upgrade, which this watchdog cannot run; nothing was changed");
WatchdogFileStore.TryDelete(options.HandoverPath);
WatchdogFileStore.Write(options.OutcomePath,
new WatchdogOutcome(handover.UpgradeId, AgentUpgradeStates.Failed, AgentUpgradeReasons.Unsupported, handover.Version,
handover.FromVersion, log.RunTail(LogLines), clock.UtcNow, "MSI upgrades run only on Windows; nothing was changed."),
WatchdogJsonContext.Default.WatchdogOutcome);
finished.Add(handover.UpgradeId);
return true;
}
await msi.StartAsync(handover, cancellationToken);
finished.Add(handover.UpgradeId);
return true;
}
/// <summary>The MSI hand-over to start, after the checks every hand-over gets; null when there is none (a files one is journalled here).</summary>
private WatchdogHandover? TakeHandover()
{
if (!File.Exists(options.HandoverPath))
{
return;
return null;
}
var handover = WatchdogFileStore.Read(options.HandoverPath, WatchdogJsonContext.Default.WatchdogHandover);
@@ -254,7 +321,7 @@ public sealed class UpgradeApplier(
WatchdogJsonContext.Default.WatchdogOutcome);
}
return;
return null;
}
if (ReadJournal() is { } open)
@@ -264,14 +331,20 @@ public sealed class UpgradeApplier(
? $"Upgrade {handover.UpgradeId} is already under way ({open.Phase}); the repeated hand-over is not applied again"
: $"Upgrade {open.UpgradeId} is under way; hand-over {handover.UpgradeId} is set aside");
WatchdogFileStore.TryDelete(options.HandoverPath);
return;
return null;
}
if (finished.Contains(handover.UpgradeId) || ReadOutcome()?.UpgradeId == handover.UpgradeId)
{
log.Warn($"Upgrade {handover.UpgradeId} has already ended; the repeated hand-over is not applied again");
WatchdogFileStore.TryDelete(options.HandoverPath);
return;
return null;
}
if (handover.Kind == WatchdogHandoverKinds.Msi)
{
// Installed by Windows Installer, never swapped.
return handover;
}
log.BeginRun();
@@ -290,6 +363,7 @@ public sealed class UpgradeApplier(
DeadlineUtc: null,
Attempts: 0));
notBefore = DateTimeOffset.MinValue;
return null;
}
private async Task<PassResult> AttemptAsync(WatchdogHandover handover, WatchdogJournal journal, CancellationToken cancellationToken)
@@ -651,7 +725,7 @@ public sealed class UpgradeApplier(
// A version that did not come up healthy is told by its own last log
// lines (taken before it is stopped); a swap that broke, by the watchdog's.
var lastLog = reason is AgentUpgradeReasons.HealthDeadline or AgentUpgradeReasons.Crashed ? TailAgentLog(since) : [];
var lastLog = reason is AgentUpgradeReasons.HealthDeadline or AgentUpgradeReasons.Crashed ? TailAgentLog(options, since) : [];
try
{
await agent.StopAsync(cancellationToken);
@@ -794,7 +868,7 @@ public sealed class UpgradeApplier(
}
/// <summary>The agent's last log lines, from its newest log file written since <paramref name="since"/>.</summary>
private List<string> TailAgentLog(DateTimeOffset since)
internal static List<string> TailAgentLog(WatchdogOptions options, DateTimeOffset since)
{
try
{
+28 -1
View File
@@ -1,7 +1,10 @@
using System.Globalization;
using System.Runtime.InteropServices;
using System.Runtime.Versioning;
using System.ServiceProcess;
using Agent.Watchdog.Msi;
using Agent.Watchdog.Platform;
using Shared.Contracts.Models;
using Shared.Infrastructure.Constants;
namespace Agent.Watchdog;
@@ -53,6 +56,10 @@ public static class WatchdogCli
&& int.TryParse(args[1], NumberStyles.None, CultureInfo.InvariantCulture, out var processId):
return WindowsWatchdogRegistration.RestartAfterExit(processId);
case WatchdogMsiFiles.RunVerb when OperatingSystem.IsWindows() && args.Length > 1
&& Guid.TryParse(args[1], out var upgradeId):
return await RunMsiAsync(upgradeId, args[2..]);
case null or "help" or "-h" or "--help" or "/?":
PrintUsage();
return command is null ? 1 : 0;
@@ -129,7 +136,9 @@ public static class WatchdogCli
: new UnixWatchdogRestart(log);
var clock = SystemClock.Instance;
var applier = new UpgradeApplier(options, agent, clock, log, new ProcessSmokeTest(), restart);
var msi = new MsiUpgrade(options, OperatingSystem.IsWindows() && !processMode ? new WindowsMsiRunnerLauncher(options, log) : null,
agent, clock, log);
var applier = new UpgradeApplier(options, agent, clock, log, new ProcessSmokeTest(), restart, msi);
var supervisor = new AgentSupervisor(options, agent, clock, log);
var service = new WatchdogService(options, applier, supervisor, clock, log);
@@ -168,6 +177,24 @@ public static class WatchdogCli
}
}
/// <summary>
/// Hidden: the detached MSI runner, started by the watchdog from its copy in
/// <c>upgrade/msi/runner/</c> (see <see cref="MsiRunner"/>). It logs to its own
/// file (the watchdog service writes the main one meanwhile).
/// </summary>
[SupportedOSPlatform("windows")]
private static async Task<int> RunMsiAsync(Guid upgradeId, string[] args)
{
var (options, _) = ParseOptions(args);
options = options with { SelfPath = null };
var log = new WatchdogLog(options.LogDirectory, console: false, name: "ExecuteR-Watchdog-msi");
var agent = new WindowsAgentService(Branding.AgentServiceName, Path.Combine(options.InstallDirectory, options.AgentExecutable), log);
var watchdog = new WindowsAgentService(Branding.WatchdogServiceName,
Path.Combine(options.InstallDirectory, Branding.WatchdogExecutableName), log);
var runner = new MsiRunner(options, agent, watchdog, new WindowsInstallerApi(), SystemClock.Instance, log);
return await runner.RunAsync(upgradeId, CancellationToken.None);
}
private static void PrintUsage()
{
Console.WriteLine($"{Branding.WatchdogServiceDisplayName} {WatchdogVersion.Current}");
+14
View File
@@ -52,4 +52,18 @@ public sealed record WatchdogOptions
public string JournalPath => Path.Combine(UpgradeDirectory, WatchdogFiles.Journal);
public string OutcomePath => Path.Combine(UpgradeDirectory, WatchdogFiles.Outcome);
/// <summary>MSI upgrades: <c>upgrade/msi/</c>, where the MSIs the agent hands over lie.</summary>
public string MsiDirectory => Path.Combine(UpgradeDirectory, WatchdogFiles.MsiFolder);
/// <summary>The runner's copy of the watchdog binary (the MSI replaces the installed one while it runs).</summary>
public string MsiRunnerDirectory => Path.Combine(MsiDirectory, WatchdogMsiFiles.RunnerFolder);
public string MsiJournalPath => Path.Combine(UpgradeDirectory, WatchdogFiles.MsiJournal);
public string MsiRunnerLockPath => Path.Combine(UpgradeDirectory, WatchdogMsiFiles.RunnerLock);
/// <summary>Windows Installer's verbose log of one MSI run, beside the agent's own logs.</summary>
public string MsiLogPath(string version, Guid upgradeId)
=> Path.Combine(LogDirectory, $"agent-msi-{version.Split('+')[0]}-{upgradeId:N}.log");
}
+8 -7
View File
@@ -3,25 +3,26 @@ using System.Reflection;
namespace Agent.Watchdog;
/// <summary>
/// The watchdog's unified version (<c>yyyy.MM.dd.HHmm</c>), read the way the
/// agent's BuildVersion reads its own: the informational version without the
/// short-commit suffix or build metadata.
/// The watchdog's version (<c>X.Y.Z</c> or <c>X.Y.Z.N</c>), read the way the
/// agent's BuildVersion reads its own: the informational version without its
/// <c>+metadata</c> (the commit).
/// </summary>
public static class WatchdogVersion
{
public static string Current { get; } = Resolve(typeof(WatchdogVersion).Assembly);
public static string Resolve(Assembly assembly)
=> FromInformational(assembly.GetCustomAttribute<AssemblyInformationalVersionAttribute>()?.InformationalVersion);
public static string FromInformational(string? informational)
{
var informational = assembly.GetCustomAttribute<AssemblyInformationalVersionAttribute>()?.InformationalVersion;
if (string.IsNullOrWhiteSpace(informational))
{
return "0.0.0.0";
}
var plus = informational.IndexOf('+');
var trimmed = plus >= 0 ? informational[..plus] : informational;
var dash = trimmed.IndexOf('-');
return dash >= 0 ? trimmed[..dash] : trimmed;
var version = (plus >= 0 ? informational[..plus] : informational).Trim();
return version.Length == 0 ? "0.0.0.0" : version;
}
}
+3 -1
View File
@@ -33,7 +33,9 @@ public sealed class PresencePublisher(
UptimeSeconds: Environment.TickCount64 / 1000,
ActiveJobs: activeJobCounter(),
// The build's RID ("win-x64"): the server picks the upgrade package by it.
Runtime: RuntimeInformation.RuntimeIdentifier);
Runtime: RuntimeInformation.RuntimeIdentifier,
// msi or portable: an MSI install upgrades through its MSI.
Install: Upgrade.AgentInstall.Current);
await publisher.PublishAsync(subject, MessageTypes.AgentPresence, payload, cancellationToken: cancellationToken);
}
+1
View File
@@ -87,6 +87,7 @@ public sealed class AgentSession : IAgentSession
Folders = folders,
ClientId = credentials.ClientId,
CanSelfUpgrade = runtime.CanSelfUpgrade,
InstallKind = AgentInstall.Current,
ServerUrl = configuration.ServerUrl,
TlsSkipVerify = configuration.TlsSkipVerify.Value,
},
+204 -25
View File
@@ -46,6 +46,13 @@ public sealed class AgentUpgradeContext
public string CurrentVersion { get; init; } = BuildVersion.Resolve(typeof(AgentUpgradeContext).Assembly);
public string? RuntimeIdentifier { get; init; } = RuntimeInformation.RuntimeIdentifier;
public bool CanSelfUpgrade { get; init; }
/// <summary>How this agent was installed (<see cref="AgentInstallKinds"/>): an MSI install upgrades through its MSI.</summary>
public string InstallKind { get; init; } = AgentInstallKinds.Portable;
/// <summary>Reads an MSI's agent version and UpgradeCode (Windows Installer, read only).</summary>
public IMsiInspector MsiInspector { get; init; } = WindowsMsiInspector.Instance;
public ServerUrl? ServerUrl { get; init; }
public bool TlsSkipVerify { get; init; }
@@ -73,7 +80,10 @@ public sealed class AgentUpgradeContext
/// and otherwise waits its start jitter, downloads the package (size and SHA-256
/// checked), unpacks it into the staging folder, makes sure the watchdog runs,
/// and hands the package over to it. The swap's outcome is reported by whichever
/// version runs afterwards (<see cref="UpgradeOutcomeReporter"/>).
/// version runs afterwards (<see cref="UpgradeOutcomeReporter"/>). An agent the
/// MSI installed takes the MSI instead (package type <c>msi</c>), which the
/// watchdog's detached runner installs through the Windows Installer API, so
/// Windows Installer and Programs and Features carry the real version.
/// </summary>
public sealed class AgentUpgradeHandler(AgentUpgradeContext context, IUpgradeEvents events, IWatchdogInstaller watchdog, ILogger logger)
{
@@ -129,13 +139,17 @@ public sealed class AgentUpgradeHandler(AgentUpgradeContext context, IUpgradeEve
return;
}
var expected = AgentPackageTypes.ForRuntime(context.RuntimeIdentifier);
var expected = AgentPackageTypes.ForDevice(context.RuntimeIdentifier, context.InstallKind);
if (expected is null || !string.Equals(expected, command.PackageType, StringComparison.OrdinalIgnoreCase))
{
await FailAsync(command, AgentUpgradeReasons.Unsupported,
expected is null
? $"No upgrade package is built for this platform ({context.RuntimeIdentifier})."
: $"This agent ({context.RuntimeIdentifier}) upgrades from the {expected} package, not {command.PackageType}.",
: string.Equals(command.PackageType, AgentPackageTypes.Msi, StringComparison.OrdinalIgnoreCase)
? $"This agent was not installed by the MSI; it upgrades from the {expected} package."
: expected == AgentPackageTypes.Msi
? $"This agent was installed by the MSI; it upgrades through the MSI, not the {command.PackageType} package."
: $"This agent ({context.RuntimeIdentifier}) upgrades from the {expected} package, not {command.PackageType}.",
cancellationToken);
return;
}
@@ -181,6 +195,12 @@ public sealed class AgentUpgradeHandler(AgentUpgradeContext context, IUpgradeEve
return;
}
if (string.Equals(command.PackageType, AgentPackageTypes.Msi, StringComparison.OrdinalIgnoreCase))
{
await UpgradeByMsiAsync(command, url, cancellationToken);
return;
}
var package = await DownloadAsync(command, url, cancellationToken);
if (package is null)
{
@@ -226,24 +246,55 @@ public sealed class AgentUpgradeHandler(AgentUpgradeContext context, IUpgradeEve
/// <summary>The verified package in the downloads folder (reused when already there), or null after a failure was reported.</summary>
private async Task<string?> DownloadAsync(AgentUpgradeCommandPayload command, Uri url, CancellationToken cancellationToken)
{
var sha = command.Sha256.Trim().ToLowerInvariant();
if (sha.Length != 64 || !sha.All(Uri.IsHexDigit))
if (NormalizeSha(command.Sha256) is not { } sha)
{
await FailAsync(command, AgentUpgradeReasons.Hash, "The command carries no valid SHA-256.", cancellationToken);
return null;
}
Directory.CreateDirectory(Folders.Downloads);
var path = Path.Combine(Folders.Downloads, sha + ".zip");
if (File.Exists(path) && new FileInfo(path).Length == command.SizeBytes && await HashAsync(path, cancellationToken) == sha)
if (await FetchAsync(command, url, sha, command.SizeBytes, path, report: true, cancellationToken) is { } problem)
{
logger.LogInformation("Upgrade {UpgradeId}: package already downloaded", command.UpgradeId);
File.SetLastWriteTimeUtc(path, DateTime.UtcNow);
await PublishAsync(command, AgentUpgradeStates.Downloading, null, "Package already downloaded.", cancellationToken);
return path;
await FailAsync(command, problem.Reason, problem.Detail, cancellationToken);
return null;
}
return path;
}
private static string? NormalizeSha(string? value)
{
var sha = value?.Trim().ToLowerInvariant();
return sha is { Length: 64 } && sha.All(Uri.IsHexDigit) ? sha : null;
}
/// <summary>
/// Downloads <paramref name="url"/> to <paramref name="path"/> (reused when
/// already there) and checks size and SHA-256; null when it is there and
/// matches, otherwise the reason and detail. <paramref name="report"/>: the
/// upgrade's own package, whose progress is published.
/// </summary>
private async Task<(string Reason, string Detail)?> FetchAsync(
AgentUpgradeCommandPayload command, Uri url, string sha, long sizeBytes, string path, bool report, CancellationToken cancellationToken)
{
Directory.CreateDirectory(Path.GetDirectoryName(path)!);
if (File.Exists(path) && new FileInfo(path).Length == sizeBytes && await HashAsync(path, cancellationToken) == sha)
{
logger.LogInformation("Upgrade {UpgradeId}: {File} already downloaded", command.UpgradeId, Path.GetFileName(path));
File.SetLastWriteTimeUtc(path, DateTime.UtcNow);
if (report)
{
await PublishAsync(command, AgentUpgradeStates.Downloading, null, "Package already downloaded.", cancellationToken);
}
return null;
}
if (report)
{
await PublishAsync(command, AgentUpgradeStates.Downloading, null, $"Downloading 0% of {Megabytes(sizeBytes)}.", cancellationToken);
}
await PublishAsync(command, AgentUpgradeStates.Downloading, null, $"Downloading 0% of {Megabytes(command.SizeBytes)}.", cancellationToken);
var partial = path + ".partial";
try
{
@@ -261,17 +312,17 @@ public sealed class AgentUpgradeHandler(AgentUpgradeContext context, IUpgradeEve
{
await target.WriteAsync(buffer.AsMemory(0, read), cancellationToken);
done += read;
if (done > command.SizeBytes)
if (done > sizeBytes)
{
break;
}
var percent = command.SizeBytes > 0 ? (int)(done * 100 / command.SizeBytes) : 0;
if (percent >= reported + 10 && percent < 100)
var percent = sizeBytes > 0 ? (int)(done * 100 / sizeBytes) : 0;
if (report && percent >= reported + 10 && percent < 100)
{
reported = percent / 10 * 10;
await PublishAsync(command, AgentUpgradeStates.Downloading, null,
$"Downloading {reported}% of {Megabytes(command.SizeBytes)}.", cancellationToken);
$"Downloading {reported}% of {Megabytes(sizeBytes)}.", cancellationToken);
}
}
}
@@ -280,23 +331,150 @@ public sealed class AgentUpgradeHandler(AgentUpgradeContext context, IUpgradeEve
|| (exception is TaskCanceledException && !cancellationToken.IsCancellationRequested))
{
TryDelete(partial);
await FailAsync(command, AgentUpgradeReasons.Download, $"The package could not be downloaded ({exception.Message}).", cancellationToken);
return null;
return (AgentUpgradeReasons.Download, $"The package could not be downloaded ({exception.Message}).");
}
var size = new FileInfo(partial).Length;
var actual = await HashAsync(partial, cancellationToken);
if (size != command.SizeBytes || actual != sha)
if (size != sizeBytes || actual != sha)
{
TryDelete(partial);
await FailAsync(command, AgentUpgradeReasons.Hash,
$"The package does not match: {size} bytes with SHA-256 {actual}, expected {command.SizeBytes} bytes with {sha}.", cancellationToken);
return null;
return (AgentUpgradeReasons.Hash,
$"The package does not match: {size} bytes with SHA-256 {actual}, expected {sizeBytes} bytes with {sha}.");
}
File.Move(partial, path, overwrite: true);
logger.LogInformation("Upgrade {UpgradeId}: package downloaded and verified", command.UpgradeId);
return path;
logger.LogInformation("Upgrade {UpgradeId}: {File} downloaded and verified", command.UpgradeId, Path.GetFileName(path));
return null;
}
/// <summary>
/// An agent the MSI installed upgrades through its MSI: downloaded and
/// checked into <c>upgrade/msi/</c>, and it is the agent's MSI of the version
/// asked for. The MSI of the version running now goes beside it
/// (<c>upgrade/msi/rollback/</c>) for a rollback: a kept one, else the one the
/// command names (downloaded, checked and kept); without either the upgrade
/// still goes on, but a new version that is not healthy stays. Both are kept
/// under <c>&lt;data&gt;/packages/</c>. The watchdog's detached runner installs
/// it through the Windows Installer API.
/// </summary>
private async Task UpgradeByMsiAsync(AgentUpgradeCommandPayload command, Uri url, CancellationToken cancellationToken)
{
if (NormalizeSha(command.Sha256) is not { } sha)
{
await FailAsync(command, AgentUpgradeReasons.Hash, "The command carries no valid SHA-256.", cancellationToken);
return;
}
var msi = Path.Combine(Folders.Msi, Branding.AgentMsiFileName(command.Version));
if (await FetchAsync(command, url, sha, command.SizeBytes, msi, report: true, cancellationToken) is { } problem)
{
await FailAsync(command, problem.Reason, problem.Detail, cancellationToken);
return;
}
var carried = context.MsiInspector.ReadAgentVersion(msi);
if (carried is null || !AgentVersions.AreSame(carried, command.Version))
{
TryDelete(msi);
await FailAsync(command, AgentUpgradeReasons.Package, carried is null
? "The package is not the agent's MSI."
: $"The MSI is of version {carried}, not {command.Version}.", cancellationToken);
return;
}
var kept = new KeptMsiPackages(Folders.KeptPackages, context.MsiInspector, logger);
var rollback = await RollbackMsiAsync(command, kept, cancellationToken);
TryKeep(kept, msi, sha, command.Version, rollback?.Version);
if (await watchdog.EnsureRunningAsync(Folders.Msi, cancellationToken) is { } watchdogProblem)
{
await FailAsync(command, AgentUpgradeReasons.Watchdog, watchdogProblem, cancellationToken);
return;
}
var handover = new WatchdogHandover(command.UpgradeId, command.Version, context.CurrentVersion, Folders.Msi, [],
command.HealthDeadlineSeconds, command.Revert, context.UtcNow(),
Kind: WatchdogHandoverKinds.Msi,
Msi: new WatchdogFile(msi, new FileInfo(msi).Length, sha),
RollbackMsi: rollback?.File,
RollbackVersion: rollback?.Version);
WatchdogFileStore.Write(Folders.Handover, handover, WatchdogJsonContext.Default.WatchdogHandover);
logger.LogInformation("Upgrade {UpgradeId} handed to the watchdog to install {Msi} through Windows Installer (rollback MSI: {Rollback})",
command.UpgradeId, msi, rollback?.File.Path ?? "none");
await PublishAsync(command, AgentUpgradeStates.Installing, null, rollback is null
? $"Handed to the watchdog to install through Windows Installer. No MSI of {context.CurrentVersion} is kept, so an unhealthy {command.Version} cannot be rolled back automatically."
: "Handed to the watchdog to install through Windows Installer.", cancellationToken);
await WaitForPickupAsync(command, cancellationToken);
}
/// <summary>The MSI of the running version in <c>upgrade/msi/rollback/</c>, or null when there is none to be had.</summary>
private async Task<(WatchdogFile File, string Version)?> RollbackMsiAsync(
AgentUpgradeCommandPayload command, KeptMsiPackages kept, CancellationToken cancellationToken)
{
var current = context.CurrentVersion;
var target = Path.Combine(Folders.MsiRollback, Branding.AgentMsiFileName(current));
try
{
if (kept.Find(current) is { } own)
{
Directory.CreateDirectory(Folders.MsiRollback);
File.Copy(own.Path, target, overwrite: true);
File.SetLastWriteTimeUtc(target, DateTime.UtcNow);
if (await HashAsync(target, cancellationToken) == own.Sha256)
{
logger.LogInformation("Upgrade {UpgradeId}: the kept MSI of {Version} is the rollback", command.UpgradeId, current);
return (new WatchdogFile(target, own.Size, own.Sha256), current);
}
TryDelete(target);
}
}
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
{
logger.LogWarning(exception, "Upgrade {UpgradeId}: the kept MSI of {Version} could not be set aside", command.UpgradeId, current);
}
if (!AgentVersions.AreSame(command.RollbackVersion, current)
|| NormalizeSha(command.RollbackSha256) is not { } sha
|| command.RollbackSizeBytes is not { } size
|| ResolveDownloadUrl(command.RollbackDownloadUrl ?? "", context.ServerUrl) is not { } url)
{
logger.LogWarning("Upgrade {UpgradeId}: no MSI of {Version} is kept and the server offers none; no automatic rollback",
command.UpgradeId, current);
return null;
}
if (await FetchAsync(command, url, sha, size, target, report: false, cancellationToken) is { } problem)
{
logger.LogWarning("Upgrade {UpgradeId}: the MSI of {Version} to roll back to could not be had ({Detail}); no automatic rollback",
command.UpgradeId, current, problem.Detail);
return null;
}
if (!AgentVersions.AreSame(context.MsiInspector.ReadAgentVersion(target), current))
{
TryDelete(target);
logger.LogWarning("Upgrade {UpgradeId}: the server's MSI of {Version} is not that version's agent MSI; no automatic rollback",
command.UpgradeId, current);
return null;
}
TryKeep(kept, target, sha, current);
return (new WatchdogFile(target, size, sha), current);
}
private void TryKeep(KeptMsiPackages kept, string msi, string sha, string version, params string?[] protectedVersions)
{
try
{
kept.Keep(msi, sha, version, [context.CurrentVersion, .. protectedVersions.OfType<string>()]);
}
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
{
logger.LogWarning(exception, "The MSI of {Version} could not be kept; the upgrade goes on", version);
}
}
private HttpMessageHandler CreateHandler()
@@ -350,7 +528,8 @@ public sealed class AgentUpgradeHandler(AgentUpgradeContext context, IUpgradeEve
{
if (context.UtcNow() >= deadline)
{
if (WatchdogFileStore.Read(Folders.Journal, WatchdogJsonContext.Default.WatchdogJournal)?.UpgradeId == command.UpgradeId)
if (WatchdogFileStore.Read(Folders.Journal, WatchdogJsonContext.Default.WatchdogJournal)?.UpgradeId == command.UpgradeId
|| WatchdogFileStore.Read(Folders.MsiJournal, WatchdogJsonContext.Default.WatchdogMsiJournal)?.UpgradeId == command.UpgradeId)
{
return;
}
+160
View File
@@ -0,0 +1,160 @@
using System.Security.Cryptography;
using System.Text.Json;
using Microsoft.Extensions.Logging;
using Shared.Contracts.Models;
using Shared.Contracts.Serialization;
namespace Agent.Upgrade;
/// <summary>One agent MSI kept for rollback.</summary>
public sealed record KeptMsi(string Path, string Sha256, long Size, string Version);
/// <summary>
/// The agent MSIs kept under <c>&lt;data&gt;/packages/&lt;sha256&gt;/&lt;file&gt;.msi</c>
/// (<see cref="WatchdogFiles.KeptPackages"/>): the one the device was installed
/// from (the installer's KeepInstallerPackage custom action copies it there) and
/// those later upgrades brought (kept here). The newest
/// <see cref="WatchdogFiles.KeptPackagesMax"/> stay. A package's version is read
/// from the MSI itself (its UNIFIEDVERSION property) unless a <c>package.json</c>
/// this class wrote beside it says; a file whose SHA-256 is not its folder's name
/// is never used.
/// </summary>
public sealed class KeptMsiPackages(string root, IMsiInspector inspector, ILogger logger)
{
public const string DescriptionFile = "package.json";
public string Root { get; } = root;
/// <summary>The kept MSI of <paramref name="version"/> whose bytes still match its hash, or null.</summary>
public KeptMsi? Find(string version)
{
foreach (var folder in Folders())
{
foreach (var file in folder.EnumerateFiles("*.msi"))
{
var kept = Version(folder, file);
if (!AgentVersions.AreSame(kept, version))
{
continue;
}
try
{
var sha = Sha256(file.FullName);
if (string.Equals(sha, folder.Name, StringComparison.OrdinalIgnoreCase))
{
return new KeptMsi(file.FullName, sha, file.Length, kept!);
}
logger.LogWarning("Kept MSI {Path} does not match its SHA-256; it is not used", file.FullName);
}
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
{
logger.LogWarning(exception, "Kept MSI {Path} could not be read", file.FullName);
}
}
}
return null;
}
/// <summary>
/// Keeps a verified MSI (a copy, unless it is already kept), then removes
/// the oldest beyond the newest <see cref="WatchdogFiles.KeptPackagesMax"/>,
/// never one of <paramref name="protectedVersions"/>.
/// </summary>
public void Keep(string source, string sha256, string version, params string[] protectedVersions)
{
var sha = sha256.ToLowerInvariant();
var folder = Path.Combine(Root, sha);
var target = Path.Combine(folder, Shared.Infrastructure.Constants.Branding.AgentMsiFileName(version));
Directory.CreateDirectory(folder);
if (!File.Exists(target) || new FileInfo(target).Length != new FileInfo(source).Length)
{
var partial = target + ".partial";
File.Copy(source, partial, overwrite: true);
File.Move(partial, target, overwrite: true);
}
File.WriteAllText(Path.Combine(folder, DescriptionFile),
JsonDefaults.Serialize(new Description(version, sha, new FileInfo(target).Length, Path.GetFileName(target))));
Directory.SetLastWriteTimeUtc(folder, DateTime.UtcNow);
logger.LogInformation("Agent MSI {Version} kept at {Path}", version, target);
Prune([version, .. protectedVersions]);
}
private void Prune(IReadOnlyCollection<string> protectedVersions)
{
foreach (var folder in Folders().Skip(WatchdogFiles.KeptPackagesMax))
{
var versions = folder.EnumerateFiles("*.msi").Select(f => Version(folder, f)).ToList();
if (versions.Any(v => protectedVersions.Any(p => AgentVersions.AreSame(v, p))))
{
continue;
}
try
{
folder.Delete(recursive: true);
logger.LogInformation("Kept MSI folder {Folder} removed (more than {Max} kept)", folder.FullName, WatchdogFiles.KeptPackagesMax);
}
catch (Exception exception) when (exception is IOException or UnauthorizedAccessException)
{
// In use; the next upgrade tries again.
}
}
}
/// <summary>Package folders (named by a SHA-256), newest first.</summary>
private List<DirectoryInfo> Folders()
{
if (!Directory.Exists(Root))
{
return [];
}
return new DirectoryInfo(Root).GetDirectories()
.Where(d => d.Name.Length == 64 && d.Name.All(Uri.IsHexDigit))
.OrderByDescending(Newest)
.ToList();
}
private static DateTime Newest(DirectoryInfo folder)
{
var newest = folder.LastWriteTimeUtc;
foreach (var file in folder.GetFiles("*.msi"))
{
newest = file.LastWriteTimeUtc > newest ? file.LastWriteTimeUtc : newest;
}
return newest;
}
private string? Version(DirectoryInfo folder, FileInfo file)
{
var description = Path.Combine(folder.FullName, DescriptionFile);
try
{
if (File.Exists(description)
&& JsonDefaults.Deserialize<Description>(File.ReadAllText(description)) is { } d
&& string.Equals(d.FileName, file.Name, StringComparison.OrdinalIgnoreCase)
&& AgentVersions.IsValid(d.Version))
{
return d.Version;
}
}
catch (Exception exception) when (exception is IOException or JsonException or UnauthorizedAccessException)
{
}
return inspector.ReadAgentVersion(file.FullName);
}
private static string Sha256(string path)
{
using var stream = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Read | FileShare.Delete);
return Convert.ToHexStringLower(SHA256.HashData(stream));
}
private sealed record Description(string Version, string Sha256, long Size, string FileName);
}
+233
View File
@@ -0,0 +1,233 @@
using System.Runtime.InteropServices;
using Shared.Contracts.Models;
using Shared.Infrastructure.Constants;
namespace Agent.Upgrade;
/// <summary>What the agent asks Windows Installer (read only); replaced in tests.</summary>
public interface IMsiInspector
{
/// <summary>
/// <c>msi</c> when Windows Installer holds the agent product (its UpgradeCode)
/// installed in <paramref name="installDirectory"/>; otherwise <c>portable</c>
/// (no product, another folder, linux/macOS, or no Windows Installer as in WinPE).
/// </summary>
string DetectInstall(string installDirectory);
/// <summary>
/// The agent version an MSI file carries (its UNIFIEDVERSION property, else
/// ProductVersion); null when the file is not the agent's MSI (another
/// UpgradeCode) or does not open as one.
/// </summary>
string? ReadAgentVersion(string msiPath);
}
/// <summary>
/// Windows Installer through msi.dll (what msiexec itself calls), read only:
/// related products of the agent's UpgradeCode, and an MSI file's Property
/// table. Never msiexec, never PowerShell.
/// </summary>
public sealed partial class WindowsMsiInspector : IMsiInspector
{
public static WindowsMsiInspector Instance { get; } = new();
private const uint ErrorSuccess = 0;
private const uint ErrorMoreData = 234;
private const uint ErrorNoMoreItems = 259;
/// <summary>MSIDBOPEN_READONLY: the persist argument is the integer 0, not a string.</summary>
private const nint MsiDbOpenReadOnly = 0;
public string DetectInstall(string installDirectory)
{
if (!OperatingSystem.IsWindows())
{
return AgentInstallKinds.Portable;
}
try
{
var product = RelatedProduct(Branding.AgentMsiUpgradeCode);
if (product is null)
{
return AgentInstallKinds.Portable;
}
// A portable copy run from elsewhere on a machine that also has the
// MSI install is not the MSI's agent.
var location = ProductInfo(product, "InstallLocation");
return string.IsNullOrWhiteSpace(location) || SameFolder(location, installDirectory)
? AgentInstallKinds.Msi
: AgentInstallKinds.Portable;
}
catch (Exception exception) when (exception is DllNotFoundException or EntryPointNotFoundException)
{
// No Windows Installer (WinPE): only the portable package runs there.
return AgentInstallKinds.Portable;
}
}
public string? ReadAgentVersion(string msiPath)
{
if (!OperatingSystem.IsWindows() || !File.Exists(msiPath))
{
return null;
}
try
{
var properties = ReadProperties(msiPath, "UpgradeCode", "UNIFIEDVERSION", "ProductVersion");
if (properties is null
|| !properties.TryGetValue("UpgradeCode", out var upgradeCode)
|| !string.Equals(upgradeCode, Branding.AgentMsiUpgradeCode, StringComparison.OrdinalIgnoreCase))
{
return null;
}
return properties.TryGetValue("UNIFIEDVERSION", out var unified) && !string.IsNullOrWhiteSpace(unified)
? unified.Trim()
: properties.GetValueOrDefault("ProductVersion")?.Trim();
}
catch (Exception exception) when (exception is DllNotFoundException or EntryPointNotFoundException)
{
return null;
}
}
internal static bool SameFolder(string left, string right)
=> string.Equals(
Path.TrimEndingDirectorySeparator(Path.GetFullPath(left)),
Path.TrimEndingDirectorySeparator(Path.GetFullPath(right)),
StringComparison.OrdinalIgnoreCase);
private static unsafe string? RelatedProduct(string upgradeCode)
{
var buffer = stackalloc char[39];
return MsiEnumRelatedProductsW(upgradeCode, 0, 0, buffer) == ErrorSuccess ? new string(buffer, 0, 38) : null;
}
private static unsafe string? ProductInfo(string productCode, string property)
{
uint length = 512;
var buffer = new char[length];
fixed (char* pointer = buffer)
{
var result = MsiGetProductInfoW(productCode, property, pointer, ref length);
if (result == ErrorMoreData)
{
return null;
}
return result == ErrorSuccess ? new string(pointer, 0, (int)length) : null;
}
}
/// <summary>The named rows of the package's Property table; null when it does not open.</summary>
private static Dictionary<string, string>? ReadProperties(string msiPath, params string[] names)
{
if (MsiOpenDatabaseW(msiPath, MsiDbOpenReadOnly, out var database) != ErrorSuccess)
{
return null;
}
try
{
if (MsiDatabaseOpenViewW(database, "SELECT `Property`, `Value` FROM `Property`", out var view) != ErrorSuccess)
{
return null;
}
try
{
if (MsiViewExecute(view, 0) != ErrorSuccess)
{
return null;
}
var wanted = new HashSet<string>(names, StringComparer.Ordinal);
var found = new Dictionary<string, string>(StringComparer.Ordinal);
while (MsiViewFetch(view, out var record) is var fetched && fetched != ErrorNoMoreItems)
{
if (fetched != ErrorSuccess)
{
break;
}
try
{
if (RecordString(record, 1) is { } name && wanted.Contains(name) && RecordString(record, 2) is { } value)
{
found[name] = value;
}
}
finally
{
MsiCloseHandle(record);
}
}
return found;
}
finally
{
MsiCloseHandle(view);
}
}
finally
{
MsiCloseHandle(database);
}
}
private static unsafe string? RecordString(nint record, uint field)
{
uint length = 0;
char empty = '\0';
var result = MsiRecordGetStringW(record, field, &empty, ref length);
if (result != ErrorMoreData && result != ErrorSuccess)
{
return null;
}
length++;
var buffer = new char[length];
fixed (char* pointer = buffer)
{
return MsiRecordGetStringW(record, field, pointer, ref length) == ErrorSuccess ? new string(pointer, 0, (int)length) : null;
}
}
[LibraryImport("msi.dll", StringMarshalling = StringMarshalling.Utf16)]
private static unsafe partial uint MsiEnumRelatedProductsW(string upgradeCode, uint reserved, uint index, char* productCode);
[LibraryImport("msi.dll", StringMarshalling = StringMarshalling.Utf16)]
private static unsafe partial uint MsiGetProductInfoW(string product, string property, char* value, ref uint length);
[LibraryImport("msi.dll", StringMarshalling = StringMarshalling.Utf16)]
private static partial uint MsiOpenDatabaseW(string databasePath, nint persist, out nint database);
[LibraryImport("msi.dll", StringMarshalling = StringMarshalling.Utf16)]
private static partial uint MsiDatabaseOpenViewW(nint database, string query, out nint view);
[LibraryImport("msi.dll")]
private static partial uint MsiViewExecute(nint view, nint record);
[LibraryImport("msi.dll")]
private static partial uint MsiViewFetch(nint view, out nint record);
[LibraryImport("msi.dll")]
private static unsafe partial uint MsiRecordGetStringW(nint record, uint field, char* value, ref uint length);
[LibraryImport("msi.dll")]
private static partial uint MsiCloseHandle(nint handle);
}
/// <summary>How the running agent was installed, asked once (presence reports it; upgrades choose the package by it).</summary>
public static class AgentInstall
{
private static readonly Lazy<string> Detected = new(() =>
WindowsMsiInspector.Instance.DetectInstall(AppContext.BaseDirectory));
/// <summary><c>msi</c> or <c>portable</c> (<see cref="AgentInstallKinds"/>).</summary>
public static string Current => Detected.Value;
}
+35 -4
View File
@@ -13,6 +13,16 @@ public sealed class UpgradeFolders(string dataDirectory, string installDirectory
public string Root { get; } = Path.Combine(dataDirectory, WatchdogFiles.Folder);
/// <summary><c>&lt;data&gt;/packages</c>: the agent MSIs kept for rollback (<see cref="KeptMsiPackages"/>).</summary>
public string KeptPackages { get; } = Path.Combine(dataDirectory, WatchdogFiles.KeptPackages);
/// <summary><c>upgrade/msi/</c>: the MSI an upgrade installs, the one it rolls back to (<c>rollback/</c>) and the runner's binary.</summary>
public string Msi => Path.Combine(Root, WatchdogFiles.MsiFolder);
public string MsiRollback => Path.Combine(Msi, WatchdogMsiFiles.RollbackFolder);
public string MsiJournal => Path.Combine(Root, WatchdogFiles.MsiJournal);
public string Downloads => Path.Combine(Root, WatchdogFiles.Downloads);
public string Staging => Path.Combine(Root, WatchdogFiles.Staging);
@@ -25,20 +35,41 @@ public sealed class UpgradeFolders(string dataDirectory, string installDirectory
public string Outcome => Path.Combine(Root, WatchdogFiles.Outcome);
/// <summary>A hand-over, a swap or an outcome not yet reported: another upgrade must wait.</summary>
public bool UpgradePending => File.Exists(Handover) || File.Exists(Journal) || File.Exists(Outcome);
/// <summary>A hand-over, a swap, an MSI run or an outcome not yet reported: another upgrade must wait.</summary>
public bool UpgradePending => File.Exists(Handover) || File.Exists(Journal) || File.Exists(MsiJournal) || File.Exists(Outcome);
/// <summary>
/// Keeps the newest download and the newest staged package, removes the rest.
/// Never while a hand-over or a swap is under way (the watchdog reads the staging then).
/// Never while a hand-over, a swap or an MSI run is under way (the watchdog reads the staging then).
/// The MSI folder goes whole: what a later MSI upgrade needs is in <see cref="KeptPackages"/>.
/// </summary>
public void Tidy()
{
if (File.Exists(Handover) || File.Exists(Journal))
if (File.Exists(Handover) || File.Exists(Journal) || File.Exists(MsiJournal))
{
return;
}
if (Directory.Exists(Msi))
{
// Not what an upgrade is downloading right now (written in the last hour).
var before = DateTime.UtcNow - TimeSpan.FromHours(1);
foreach (var entry in new DirectoryInfo(Msi).GetFileSystemInfos().Where(e => e.LastWriteTimeUtc < before))
{
TryDelete(() =>
{
if (entry is DirectoryInfo folder)
{
folder.Delete(recursive: true);
}
else
{
entry.Delete();
}
});
}
}
if (Directory.Exists(Downloads))
{
foreach (var file in new DirectoryInfo(Downloads).GetFiles().OrderByDescending(f => f.LastWriteTimeUtc).Skip(1))
+4 -2
View File
@@ -63,7 +63,7 @@ public sealed class UpgradeOutcomeReporter(
reported = true;
tidied = false;
}
else if (File.Exists(folders.Outcome) && !File.Exists(folders.Journal))
else if (File.Exists(folders.Outcome) && !File.Exists(folders.Journal) && !File.Exists(folders.MsiJournal))
{
// Unreadable, and no run that could still rewrite it: drop it.
logger.LogWarning("Upgrade outcome at {Path} does not read; it is removed", folders.Outcome);
@@ -81,7 +81,7 @@ public sealed class UpgradeOutcomeReporter(
public static AgentUpgradeEventPayload ToEvent(WatchdogOutcome outcome, string clientId)
{
var detail = outcome.Result switch
var detail = !string.IsNullOrWhiteSpace(outcome.Detail) ? outcome.Detail : outcome.Result switch
{
AgentUpgradeStates.Succeeded => $"Upgraded from {outcome.FromVersion} to {outcome.Version}.",
AgentUpgradeStates.RolledBack => outcome.Reason switch
@@ -96,6 +96,8 @@ public sealed class UpgradeOutcomeReporter(
AgentUpgradeReasons.Hash => "The staged package did not match its SHA-256; nothing was changed.",
AgentUpgradeReasons.Package => "The staged package was not usable; nothing was changed.",
AgentUpgradeReasons.Swap => "The previous version could not be put back whole.",
AgentUpgradeReasons.Msi => $"Windows Installer did not install {outcome.Version}; {outcome.FromVersion} stays.",
AgentUpgradeReasons.InstallerBusy => $"Another installation kept Windows Installer busy; {outcome.FromVersion} stays.",
_ => "The watchdog could not apply the package; nothing was changed.",
},
};
+2 -1
View File
@@ -178,6 +178,7 @@ public static class DeviceEndpoints
Tags: Groups.DeviceGroupService.ParseTags(device.TagsJson),
IsOrphaned: device.IsOrphaned,
PublicIp: device.PublicIp,
GroupIds: groupIds);
GroupIds: groupIds,
AgentInstall: device.AgentInstall);
}
}
+1
View File
@@ -185,6 +185,7 @@ public class AppDbContext(DbContextOptions<AppDbContext> options) : DbContext(op
entity.HasKey(e => e.ClientId);
entity.HasIndex(e => e.Hostname);
entity.Property(e => e.AgentRuntime).HasMaxLength(32);
entity.Property(e => e.AgentInstall).HasMaxLength(16);
entity.HasOne(e => e.Inventory)
.WithOne()
.HasForeignKey<DeviceInventoryEntity>(e => e.ClientId)
+2
View File
@@ -187,6 +187,8 @@ public sealed class DeviceEntity
public string? AgentVersion { get; set; }
/// <summary>The agent build's runtime identifier from presence (e.g. win-x64); null from older agents.</summary>
public string? AgentRuntime { get; set; }
/// <summary>How the agent was installed, from presence: msi or portable (AgentInstallKinds); null from older agents.</summary>
public string? AgentInstall { get; set; }
/// <summary>Enrolled NATS user public key bound to this device.</summary>
public required string NkeyPublic { get; set; }
public Guid EnrollmentId { get; set; }
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,29 @@
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace Backend.Data.Migrations.Postgres
{
/// <inheritdoc />
public partial class AgentInstallKind : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.AddColumn<string>(
name: "AgentInstall",
table: "Devices",
type: "character varying(16)",
maxLength: 16,
nullable: true);
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropColumn(
name: "AgentInstall",
table: "Devices");
}
}
}
@@ -266,6 +266,10 @@ namespace Backend.Data.Migrations.Postgres
b.Property<string>("ClientId")
.HasColumnType("text");
b.Property<string>("AgentInstall")
.HasMaxLength(16)
.HasColumnType("character varying(16)");
b.Property<string>("AgentRuntime")
.HasMaxLength(32)
.HasColumnType("character varying(32)");
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,29 @@
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace Backend.Data.Migrations.SqlServer
{
/// <inheritdoc />
public partial class AgentInstallKind : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.AddColumn<string>(
name: "AgentInstall",
table: "Devices",
type: "nvarchar(16)",
maxLength: 16,
nullable: true);
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropColumn(
name: "AgentInstall",
table: "Devices");
}
}
}
@@ -266,6 +266,10 @@ namespace Backend.Data.Migrations.SqlServer
b.Property<string>("ClientId")
.HasColumnType("nvarchar(450)");
b.Property<string>("AgentInstall")
.HasMaxLength(16)
.HasColumnType("nvarchar(16)");
b.Property<string>("AgentRuntime")
.HasMaxLength(32)
.HasColumnType("nvarchar(32)");
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,29 @@
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace Backend.Data.Migrations.Sqlite
{
/// <inheritdoc />
public partial class AgentInstallKind : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.AddColumn<string>(
name: "AgentInstall",
table: "Devices",
type: "TEXT",
maxLength: 16,
nullable: true);
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropColumn(
name: "AgentInstall",
table: "Devices");
}
}
}
@@ -259,6 +259,10 @@ namespace Backend.Data.Migrations.Sqlite
b.Property<string>("ClientId")
.HasColumnType("TEXT");
b.Property<string>("AgentInstall")
.HasMaxLength(16)
.HasColumnType("TEXT");
b.Property<string>("AgentRuntime")
.HasMaxLength(32)
.HasColumnType("TEXT");
+2 -2
View File
@@ -107,8 +107,8 @@ public sealed class AgentPackageService(
: "volume";
/// <summary>
/// The upgradable packages visible right now (portable zip and the Unix
/// builds, not the MSI), each with the version from its VERSION marker:
/// The upgradable packages visible right now (the MSI, the portable zip and
/// the Unix builds), each with the version from its VERSION marker:
/// what the agent catalog imports.
/// </summary>
public IEnumerable<(string PackageType, string Path, string? Version)> LocateUpgradePackages()
@@ -111,6 +111,8 @@ public sealed class MessagingWorkers(
device.AgentVersion = payload.AgentVersion;
// Older agents send no runtime; what a newer build reported stands.
device.AgentRuntime = payload.Runtime ?? device.AgentRuntime;
// msi or portable: an MSI install upgrades through its MSI.
device.AgentInstall = payload.Install is AgentInstallKinds.Msi or AgentInstallKinds.Portable ? payload.Install : device.AgentInstall;
await db.SaveChangesAsync(cancellationToken);
lock (lastKnownStatus)
+17 -6
View File
@@ -42,13 +42,9 @@ public sealed class AgentCatalogService(
S3SettingsService s3Settings,
ILogger<AgentCatalogService> logger)
{
/// <summary>A real release version (four numeric parts); a development 0.0.0.0 or a label is never catalogued.</summary>
/// <summary>A real version (3 or 4 numeric parts, 0.x included); a development 0.0.0.0, +metadata or a label is never catalogued.</summary>
public static bool IsCatalogVersion(string? version)
=> !string.IsNullOrWhiteSpace(version)
&& version.Length <= 64
&& Version.TryParse(version, out var parsed)
&& parsed.Major > 0
&& version.All(c => char.IsAsciiDigit(c) || c == '.');
=> AgentVersions.IsValid(version) && version!.All(c => char.IsAsciiDigit(c) || c == '.');
/// <summary>One pass: import what this node can see, offload to S3 when on, prune.</summary>
public async Task SyncAsync(CancellationToken cancellationToken)
@@ -321,6 +317,21 @@ public sealed class AgentCatalogService(
=> db.AgentPackages.AsNoTracking()
.SingleOrDefaultAsync(p => p.Version == version && p.PackageType == packageType, cancellationToken);
/// <summary>
/// The package of the same version (<see cref="AgentVersions.AreSame"/>:
/// 1.4.0 and 1.4.0.0 match, +metadata ignored) for the platform, or null.
/// </summary>
public async Task<AgentPackageEntity?> FindSameAsync(string? version, string packageType, CancellationToken cancellationToken)
{
if (!AgentVersions.IsValid(version))
{
return null;
}
return (await db.AgentPackages.AsNoTracking().Where(p => p.PackageType == packageType).ToListAsync(cancellationToken))
.FirstOrDefault(p => AgentVersions.AreSame(p.Version, version));
}
/// <summary>The newest package per platform (what upgrades default to).</summary>
public async Task<Dictionary<string, AgentPackageEntity>> NewestByTypeAsync(CancellationToken cancellationToken)
=> (await db.AgentPackages.AsNoTracking().ToListAsync(cancellationToken))
+15 -3
View File
@@ -167,7 +167,8 @@ public sealed class AgentUpgradeScheduler(
await upgrades.BroadcastAsync(row, devices.GetValueOrDefault(row.ClientId)?.Hostname ?? row.ClientId, cancellationToken);
}
var sent = await SendBatchAsync(db, upgrades, catalog, settings, rows, Online, now, cancellationToken);
var sent = await SendBatchAsync(db, upgrades, catalog, settings, rows, Online,
clientId => devices.GetValueOrDefault(clientId)?.AgentVersion, now, cancellationToken);
if (sent > 0 || queued.Count > 0)
{
await services.GetRequiredService<BrowserEventBroadcaster>().BroadcastActivityAsync(new ActivityEventPayload(
@@ -188,7 +189,8 @@ public sealed class AgentUpgradeScheduler(
/// </summary>
private async Task<int> SendBatchAsync(
AppDbContext db, AgentUpgradeService upgrades, AgentCatalogService catalog, Shared.Contracts.Api.AgentUpgradeSettingsDto settings,
List<AgentUpgradeEntity> rows, Func<string, bool> online, DateTimeOffset now, CancellationToken cancellationToken)
List<AgentUpgradeEntity> rows, Func<string, bool> online, Func<string, string?> runningVersion, DateTimeOffset now,
CancellationToken cancellationToken)
{
var ready = rows.Where(r => r.State == AgentUpgradeStates.Queued && online(r.ClientId)).ToList();
var readyIds = ready.Select(r => r.Id).ToHashSet();
@@ -245,13 +247,23 @@ public sealed class AgentUpgradeScheduler(
continue; // Cancelled or taken meanwhile.
}
// An MSI upgrade carries the MSI of the version the device runs now,
// when the catalog holds it: the device's own kept copy comes first,
// this one is for a device that kept none (its first upgrade).
var rollback = row.PackageType == AgentPackageTypes.Msi
? await catalog.FindSameAsync(runningVersion(row.ClientId) ?? row.FromVersion, AgentPackageTypes.Msi, cancellationToken)
: null;
var command = new AgentUpgradeCommandPayload(
row.Id, row.TargetVersion, row.PackageType,
await catalog.DownloadUrlAsync(package, cancellationToken),
package.Sha256, package.SizeBytes,
Revert: row.Source == AgentUpgradeSources.Rollback,
HealthDeadlineSeconds: settings.HealthDeadlineMinutes * 60,
StartJitterSeconds: StartJitterSeconds);
StartJitterSeconds: StartJitterSeconds,
RollbackVersion: rollback?.Version,
RollbackDownloadUrl: rollback is null ? null : await catalog.DownloadUrlAsync(rollback, cancellationToken),
RollbackSha256: rollback?.Sha256,
RollbackSizeBytes: rollback?.SizeBytes);
if (!await sender.DispatchAgentUpgradeAsync(row.ClientId, command, cancellationToken))
{
+9 -6
View File
@@ -42,13 +42,15 @@ public sealed class AgentUpgradeService(
};
/// <summary>
/// The package a device upgrades from: its reported runtime, or for older
/// agents that do not report one, the OS platform and architecture from
/// inventory (Windows is only built for x64). Null when there is no build.
/// The package a device upgrades from: the MSI for a Windows agent the MSI
/// installed (so Windows Installer and detection tools see the real
/// version), otherwise its reported runtime's package, or for older agents
/// that report no runtime, the OS platform and architecture from inventory
/// (Windows is only built for x64). Null when there is no build.
/// </summary>
public static string? ResolvePackageType(string? runtime, string? osCaption, string? architecture)
public static string? ResolvePackageType(string? runtime, string? osCaption, string? architecture, string? install = null)
{
if (AgentPackageTypes.ForRuntime(runtime) is { } fromRuntime)
if (AgentPackageTypes.ForDevice(runtime, install) is { } fromRuntime)
{
return fromRuntime;
}
@@ -123,7 +125,8 @@ public sealed class AgentUpgradeService(
result[device.ClientId] = ResolvePackageType(
device.AgentRuntime,
report?.Os.Caption ?? device.OsCaption,
report?.Os.Architecture ?? report?.Identity.SystemType);
report?.Os.Architecture ?? report?.Identity.SystemType,
device.AgentInstall);
}
return result;
+7 -1
View File
@@ -2,11 +2,15 @@
# Build context is the repository root.
FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
# The version (README, Versioning): X.Y.Z for a release, X.Y.Z.N between
# releases; CI passes it. 0.0.0.0, the development version, otherwise (the
# image build has no git to read one from).
ARG UNIFIED_VERSION=0.0.0.0
WORKDIR /source
# Restore layer: copy only project/props files first for build caching.
COPY global.json Directory.Build.props Directory.Packages.props ./
COPY build/Version.targets build/
COPY src/Shared.Contracts/Shared.Contracts.csproj src/Shared.Contracts/
COPY src/Shared.Infrastructure/Shared.Infrastructure.csproj src/Shared.Infrastructure/
COPY src/Backend/Backend.csproj src/Backend/
@@ -15,8 +19,10 @@ RUN dotnet restore src/Backend/Backend.csproj
COPY src/Shared.Contracts/ src/Shared.Contracts/
COPY src/Shared.Infrastructure/ src/Shared.Infrastructure/
COPY src/Backend/ src/Backend/
# The short commit goes into the informational version (X.Y.Z+<commit>).
ARG GIT_COMMIT_SHORT=""
RUN dotnet publish src/Backend/Backend.csproj -c Release -o /app --no-restore \
-p:UnifiedVersion=${UNIFIED_VERSION}
-p:UnifiedVersion=${UNIFIED_VERSION} -p:GitCommitShort=${GIT_COMMIT_SHORT}
FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS runtime
ARG UNIFIED_VERSION=0.0.0.0
@@ -0,0 +1,6 @@
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<startup useLegacyV2RuntimeActivationPolicy="true">
<supportedRuntime version="v4.0" />
</startup>
</configuration>
@@ -0,0 +1,112 @@
namespace ExecuteR.Installer.CustomActions;
/// <summary>
/// Which installed release an MSI may replace (README, Versioning). Windows
/// Installer compares only the first three fields of a ProductVersion and knows
/// nothing of the calendar era, so each package removes every installed version
/// (MajorUpgrade AllowDowngrades) and this decides first whether one of them is
/// newer, by the product's one version order. <see cref="Compare"/> is a copy of
/// Shared.Contracts' AgentVersions.Compare (this assembly runs on .NET
/// Framework); tests/Shared.Tests compiles this file and checks the two agree.
/// No usings: the file is also compiled into the tests.
/// </summary>
public static class InstalledRelease
{
/// <summary>Majors from this value on are calendar-era versions (2026.10.23.1400).</summary>
public const int CalendarEraMajor = 2000;
/// <summary>
/// The product version an installed MSI's version string stands for. From
/// semantic versions on it is the version itself (1.4.0.7). The calendar
/// era's MSIs carried three fields, <c>yy.MM.minute-of-the-month</c>
/// (26.10.13060 for 2026.10.10.0140), read back to that version. Null when
/// it does not read as a version.
/// </summary>
public static string? FromMsiVersion(string? text)
{
var trimmed = text?.Trim();
if (!System.Version.TryParse(trimmed, out var version) || version.Build < 0)
{
return null;
}
if (version.Revision < 0 && version.Major >= 26 && version.Major <= 99
&& version.Minor >= 1 && version.Minor <= 12 && version.Build < 31 * 1440)
{
var minute = version.Build;
return string.Format(
System.Globalization.CultureInfo.InvariantCulture,
"{0}.{1:00}.{2:00}.{3:00}{4:00}",
2000 + version.Major, version.Minor, (minute / 1440) + 1, minute % 1440 / 60, minute % 60);
}
return trimmed;
}
/// <summary>
/// The first installed version (as <see cref="FromMsiVersion"/> reads it)
/// newer than <paramref name="incoming"/>, the package's own full version,
/// or null when none is. The same version again is a reinstall, not newer.
/// </summary>
public static string? Newer(string? incoming, System.Collections.Generic.IEnumerable<string?> installedMsiVersions)
{
foreach (var text in installedMsiVersions)
{
var installed = FromMsiVersion(text);
if (installed is not null && Compare(installed, incoming) > 0)
{
return installed;
}
}
return null;
}
/// <summary>
/// AgentVersions.Compare: semantic <c>major.minor.patch[.build]</c> (missing
/// fourth field = 0, <c>+metadata</c> ignored) after the calendar era (major
/// 2000 or more); anything that does not parse, and 0.0.0.0, before
/// everything, then ordinally.
/// </summary>
public static int Compare(string? left, string? right)
{
var a = Parse(left);
var b = Parse(right);
if (a is not null && b is not null)
{
var era = Era(a).CompareTo(Era(b));
return era != 0 ? era : a.CompareTo(b);
}
if (a is not null)
{
return 1;
}
if (b is not null)
{
return -1;
}
return string.CompareOrdinal(left, right);
}
private static int Era(System.Version version) => version.Major >= CalendarEraMajor ? 0 : 1;
private static System.Version? Parse(string? value)
{
if (string.IsNullOrWhiteSpace(value) || value!.Length > 64)
{
return null;
}
var core = value.Split('+')[0].Trim();
if (!System.Version.TryParse(core, out var version) || core.Split('.').Length < 3)
{
return null;
}
var normalized = new System.Version(version.Major, version.Minor, version.Build, System.Math.Max(version.Revision, 0));
return normalized == new System.Version(0, 0, 0, 0) ? null : normalized;
}
}
@@ -0,0 +1,22 @@
<Project Sdk="Microsoft.NET.Sdk">
<!-- Managed custom actions for both MSIs. Windows Installer hosts them on
.NET Framework; DTF packs the assembly into Installer.CustomActions.CA.dll,
which the installers embed as a Binary. -->
<PropertyGroup>
<TargetFramework>net472</TargetFramework>
<ImplicitUsings>disable</ImplicitUsings>
<IsPackable>false</IsPackable>
<RootNamespace>ExecuteR.Installer.CustomActions</RootNamespace>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="WixToolset.Dtf.CustomAction" />
<PackageReference Include="WixToolset.Dtf.WindowsInstaller" />
</ItemGroup>
<ItemGroup>
<Content Include="CustomAction.config" CopyToOutputDirectory="PreserveNewest" />
</ItemGroup>
</Project>
+166
View File
@@ -0,0 +1,166 @@
namespace ExecuteR.Installer.CustomActions;
/// <summary>
/// The agent MSIs kept for rollback (Shared.Contracts' WatchdogFiles.KeptPackages):
/// <c>%ProgramData%\ExecuteR\packages\&lt;sha256&gt;\&lt;file name&gt;.msi</c>, the
/// newest <see cref="Max"/> by when each was last installed. An MSI upgrade's
/// rollback reinstalls the previous version from here. No usings: the file is
/// also compiled into tests/Shared.Tests.
/// </summary>
public static class KeptPackages
{
/// <summary>WatchdogFiles.KeptPackagesMax.</summary>
public const int Max = 3;
/// <summary>The description beside each kept MSI, as the agent writes it: <c>{ "version", "sha256" }</c>.</summary>
public const string DescriptionFile = "package.json";
/// <summary>
/// Copies <paramref name="source"/> to <c>&lt;root&gt;\&lt;sha256&gt;\&lt;its file name&gt;</c>
/// (unless that copy is already there whole), writes <see cref="DescriptionFile"/>
/// beside it with <paramref name="version"/> (the package's full version) and
/// the hash, marks its folder as the newest, and removes the package folders
/// beyond the newest <paramref name="max"/>. Returns the kept file's path.
/// Throws on failure; the caller decides what a failure means (the installer
/// logs it and goes on).
/// </summary>
public static string Keep(string source, string? version, string root, int max, System.Action<string> log)
{
var sha256 = Sha256(source);
var folder = System.IO.Path.Combine(root, sha256);
var target = System.IO.Path.Combine(folder, System.IO.Path.GetFileName(source));
System.IO.Directory.CreateDirectory(folder);
if (System.IO.File.Exists(target) && Sha256(target) == sha256)
{
log($"{target} is already kept.");
}
else
{
// Copied under a temporary name and renamed, so a reader never sees half an MSI.
var partial = target + ".partial";
System.IO.File.Copy(source, partial, overwrite: true);
if (System.IO.File.Exists(target))
{
System.IO.File.Delete(target);
}
System.IO.File.Move(partial, target);
log($"Kept {source} as {target}.");
}
if (!string.IsNullOrWhiteSpace(version))
{
// The same temporary-name-and-rename, so the agent never reads half a description.
var description = System.IO.Path.Combine(folder, DescriptionFile);
var partial = description + ".partial";
System.IO.File.WriteAllText(partial, Description(version!.Trim(), sha256), new System.Text.UTF8Encoding(false));
if (System.IO.File.Exists(description))
{
System.IO.File.Delete(description);
}
System.IO.File.Move(partial, description);
}
System.IO.Directory.SetLastWriteTimeUtc(folder, System.DateTime.UtcNow);
Prune(root, sha256, max, log);
return target;
}
/// <summary>Removes the package folders (named by a SHA-256) beyond the newest <paramref name="max"/>; <paramref name="keep"/> always stays.</summary>
public static void Prune(string root, string keep, int max, System.Action<string> log)
{
var folders = new System.Collections.Generic.List<System.IO.DirectoryInfo>();
foreach (var directory in new System.IO.DirectoryInfo(root).GetDirectories())
{
if (IsSha256(directory.Name))
{
folders.Add(directory);
}
}
// The kept one first, then the most recently installed.
folders.Sort((a, b) =>
{
var first = string.Equals(b.Name, keep, System.StringComparison.OrdinalIgnoreCase)
.CompareTo(string.Equals(a.Name, keep, System.StringComparison.OrdinalIgnoreCase));
return first != 0 ? first : b.LastWriteTimeUtc.CompareTo(a.LastWriteTimeUtc);
});
for (var i = System.Math.Max(max, 1); i < folders.Count; i++)
{
try
{
folders[i].Delete(recursive: true);
log($"Removed the kept package {folders[i].FullName}.");
}
catch (System.Exception exception) when (exception is System.IO.IOException or System.UnauthorizedAccessException)
{
log($"Could not remove the kept package {folders[i].FullName}: {exception.Message}");
}
}
}
/// <summary><c>{"version":"1.4.0","sha256":"…"}</c> (camelCase, as the agent reads it).</summary>
public static string Description(string version, string sha256)
=> "{\"version\":" + JsonString(version) + ",\"sha256\":" + JsonString(sha256) + "}";
private static string JsonString(string value)
{
var text = new System.Text.StringBuilder(value.Length + 2).Append('"');
foreach (var c in value)
{
switch (c)
{
case '"': text.Append("\\\""); break;
case '\\': text.Append("\\\\"); break;
default:
if (c < ' ')
{
text.Append("\\u").Append(((int)c).ToString("x4", System.Globalization.CultureInfo.InvariantCulture));
}
else
{
text.Append(c);
}
break;
}
}
return text.Append('"').ToString();
}
public static string Sha256(string path)
{
using var stream = System.IO.File.OpenRead(path);
using var sha = System.Security.Cryptography.SHA256.Create();
var hash = sha.ComputeHash(stream);
var text = new System.Text.StringBuilder(hash.Length * 2);
foreach (var b in hash)
{
text.Append(b.ToString("x2", System.Globalization.CultureInfo.InvariantCulture));
}
return text.ToString();
}
private static bool IsSha256(string name)
{
if (name.Length != 64)
{
return false;
}
foreach (var c in name)
{
if (!(c is >= '0' and <= '9' or >= 'a' and <= 'f' or >= 'A' and <= 'F'))
{
return false;
}
}
return true;
}
}
@@ -0,0 +1,115 @@
using System;
using System.IO;
using System.Linq;
using System.Security.AccessControl;
using System.Security.Principal;
using WixToolset.Dtf.WindowsInstaller;
namespace ExecuteR.Installer.CustomActions;
/// <summary>The installers' side of versioning and MSI upgrades (README, Versioning).</summary>
public static class UpgradeActions
{
/// <summary>
/// Immediate, after FindRelatedProducts (both sequences, <c>NOT Installed</c>).
/// MajorUpgrade lists every installed version of the product in
/// <c>WIX_UPGRADE_DETECTED</c>; when one is newer than this package's own
/// version (<c>UNIFIEDVERSION</c>, all four fields and the calendar era
/// considered by <see cref="InstalledRelease"/>), <c>NEWER_RELEASE_INSTALLED</c>
/// is set to it and the launch condition refuses the install, unless
/// <c>ALLOWDOWNGRADE=1</c> (an MSI upgrade's rollback).
/// </summary>
[CustomAction]
public static ActionResult FindNewerRelease(Session session)
{
var incoming = session["UNIFIEDVERSION"];
if (string.IsNullOrEmpty(incoming))
{
incoming = session["ProductVersion"];
}
var codes = (session["WIX_UPGRADE_DETECTED"] ?? string.Empty).Split(new[] { ';' }, StringSplitOptions.RemoveEmptyEntries);
var installed = codes.Select(code => InstalledVersion(session, code)).ToList();
var newer = InstalledRelease.Newer(incoming, installed);
session.Log($"FindNewerRelease: this package is {incoming}; installed: {(installed.Count == 0 ? "none" : string.Join(", ", installed))}; newer: {newer ?? "none"}.");
if (newer is not null)
{
session["NEWER_RELEASE_INSTALLED"] = newer;
}
return ActionResult.Success;
}
/// <summary>
/// Deferred, as SYSTEM, after InstallFiles on an install or upgrade (agent MSI
/// only). Copies the MSI being installed (<c>Source</c> = [OriginalDatabase])
/// to <c>Destination\&lt;sha256&gt;\&lt;file name&gt;</c> (the original name, e.g.
/// ExecuteR-Agent-1.4.0.msi), writes <c>package.json</c> (<c>{ version, sha256 }</c>,
/// <c>Version</c> = [UNIFIEDVERSION]) beside it and keeps the newest
/// <see cref="KeptPackages.Max"/>, so an MSI upgrade can roll back to it.
/// Never fails the install: a copy that cannot be made is logged.
/// </summary>
[CustomAction]
public static ActionResult KeepInstallerPackage(Session session)
{
try
{
var data = session.CustomActionData;
var source = data["Source"];
var destination = data["Destination"];
if (string.IsNullOrEmpty(source) || !File.Exists(source))
{
session.Log($"KeepInstallerPackage: the package '{source}' cannot be read; nothing kept.");
return ActionResult.Success;
}
// The full version the agent reports (UNIFIEDVERSION), for package.json.
data.TryGetValue("Version", out var version);
ProtectFolder(destination);
KeptPackages.Keep(source, version, destination, KeptPackages.Max, message => session.Log($"KeepInstallerPackage: {message}"));
}
catch (Exception exception)
{
// Whatever went wrong, the agent installs; only the rollback copy is missing.
session.Log($"KeepInstallerPackage: keeping the package failed, the install goes on: {exception}");
}
return ActionResult.Success;
}
private static string? InstalledVersion(Session session, string productCode)
{
try
{
return new ProductInstallation(productCode)["VersionString"];
}
catch (Exception exception) when (exception is InstallerException or ArgumentException)
{
session.Log($"FindNewerRelease: the version of {productCode} could not be read: {exception.Message}");
return null;
}
}
/// <summary>
/// The kept packages are what SYSTEM reinstalls on a rollback, so only SYSTEM
/// and Administrators may write there (as the agent protects its data folder),
/// nothing inherited from %ProgramData%, where every user may create files.
/// </summary>
private static void ProtectFolder(string path)
{
var directory = Directory.CreateDirectory(path);
var security = new DirectorySecurity();
security.SetAccessRuleProtection(isProtected: true, preserveInheritance: false);
foreach (var sid in new[] { WellKnownSidType.LocalSystemSid, WellKnownSidType.BuiltinAdministratorsSid })
{
security.AddAccessRule(new FileSystemAccessRule(
new SecurityIdentifier(sid, null),
FileSystemRights.FullControl,
InheritanceFlags.ContainerInherit | InheritanceFlags.ObjectInherit,
PropagationFlags.None,
AccessControlType.Allow));
}
directory.SetAccessControl(security);
}
}
+31 -3
View File
@@ -32,15 +32,38 @@
<SummaryInformation Description="ExecuteR Server $(UnifiedVersion)" />
<MajorUpgrade AllowSameVersionUpgrades="yes"
Schedule="afterInstallInitialize"
DowngradeErrorMessage="A newer version of ExecuteR Server is already installed." />
<!--
Versioning (README, Versioning), as for the agent MSI: ProductVersion is
the four-field version, every installed version is removed
(AllowDowngrades, so 1.x replaces the calendar era's 26.x) and
FindNewerRelease refuses a newer one first, all four fields and the era
considered. ALLOWDOWNGRADE=1 installs an older one deliberately. The old
product's uninstall during the upgrade sets UPGRADINGPRODUCTCODE, so the
service and data are kept.
-->
<MajorUpgrade AllowDowngrades="yes"
Schedule="afterInstallInitialize" />
<MediaTemplate EmbedCab="yes" />
<!-- The full version (X.Y.Z or X.Y.Z.N), for FindNewerRelease and ARP. -->
<Property Id="UNIFIEDVERSION" Value="$(UnifiedVersion)" />
<Property Id="ARPCOMMENTS" Value="ExecuteR Server $(UnifiedVersion)" />
<Binary Id="ExecuteRCustomActions" SourceFile="$(var.Installer.CustomActions.TargetDir)Installer.CustomActions.CA.dll" />
<CustomAction Id="FindNewerRelease" BinaryRef="ExecuteRCustomActions" DllEntry="FindNewerRelease"
Execute="immediate" Return="check" />
<Property Id="ALLOWDOWNGRADE" Secure="yes" />
<Launch Condition="Installed OR NOT NEWER_RELEASE_INSTALLED OR ALLOWDOWNGRADE = &quot;1&quot;"
Message="A newer version of ExecuteR Server, [NEWER_RELEASE_INSTALLED], is already installed." />
<!-- An upgrade (and a repair) lays every file down: a calendar-era release
stamped its binaries 2026.x, which Windows Installer counts as newer
than 1.x, so it would keep them, then remove them with the old product
and install none. -->
<SetProperty Id="REINSTALLMODE" Value="amus" Before="CostInitialize" Sequence="both"
Condition="REINSTALL OR WIX_UPGRADE_DETECTED" />
<Icon Id="AppIcon" SourceFile="$(AppIconPath)" />
<Property Id="ARPPRODUCTICON" Value="AppIcon" />
@@ -326,7 +349,12 @@
Condition="WIXUI_EXITDIALOGOPTIONALCHECKBOX = 1 AND NOT Installed" />
</UI>
<InstallUISequence>
<Custom Action="FindNewerRelease" After="FindRelatedProducts" Condition="NOT Installed" />
</InstallUISequence>
<InstallExecuteSequence>
<Custom Action="FindNewerRelease" After="FindRelatedProducts" Condition="NOT Installed" />
<Custom Action="RunServerSetup"
After="InstallFiles"
Condition="NOT (REMOVE=&quot;ALL&quot;)" />
@@ -5,10 +5,12 @@
<PropertyGroup>
<OutputName>Server</OutputName>
<InstallerPlatform>x64</InstallerPlatform>
<!-- ICE61: AllowSameVersionUpgrades="yes". ICE60: the harvested bundle
<!-- ICE61: every installed version is removed (AllowDowngrades) and a
newer one refused by FindNewerRelease. ICE60: the harvested bundle
carries versioned-but-language-less binaries (nats-server is Go). -->
<SuppressIces>ICE61;ICE60</SuppressIces>
<MsiProductVersion Condition="'$(MsiProductVersion)' == ''">0.0.0</MsiProductVersion>
<!-- The four-field version (X.Y.Z.0 / X.Y.Z.N), passed by scripts/Package-Server.ps1. -->
<MsiProductVersion Condition="'$(MsiProductVersion)' == ''">0.0.0.0</MsiProductVersion>
<!-- Directory containing the packaged standalone server (exe + wwwroot +
nats + agent-package), produced by scripts/Package-Server.ps1. -->
<ServerPublishDir Condition="'$(ServerPublishDir)' == ''">$(MSBuildThisFileDirectory)..\..\artifacts\server\current\win-x64</ServerPublishDir>
@@ -21,4 +23,9 @@
<PackageReference Include="WixToolset.UI.wixext" />
</ItemGroup>
<ItemGroup>
<!-- FindNewerRelease (Installer.CustomActions.CA.dll). -->
<ProjectReference Include="..\Installer.CustomActions\Installer.CustomActions.csproj" />
</ItemGroup>
</Project>
+10 -4
View File
@@ -5,11 +5,12 @@
<PropertyGroup>
<OutputName>Agent</OutputName>
<InstallerPlatform>x64</InstallerPlatform>
<!-- ICE61 intentionally suppressed: AllowSameVersionUpgrades="yes". -->
<!-- ICE61 intentionally suppressed: every installed version is removed
(AllowDowngrades) and a newer one refused by FindNewerRelease. -->
<SuppressIces>ICE61</SuppressIces>
<!-- MsiProductVersion is the deterministic MSI-compatible transform of
UnifiedVersion, computed by scripts/Package-Agent.ps1 / CI. -->
<MsiProductVersion Condition="'$(MsiProductVersion)' == ''">0.0.0</MsiProductVersion>
<!-- MsiProductVersion is the four-field version (X.Y.Z.0 for a release,
X.Y.Z.N between releases), passed by scripts/Package-Agent.ps1 / CI. -->
<MsiProductVersion Condition="'$(MsiProductVersion)' == ''">0.0.0.0</MsiProductVersion>
<!-- Directory containing the published self-contained Agent.exe. -->
<AgentPublishDir Condition="'$(AgentPublishDir)' == ''">$(MSBuildThisFileDirectory)..\Agent\bin\Release\net10.0-windows\win-x64\publish</AgentPublishDir>
<AppIconPath Condition="'$(AppIconPath)' == ''">$(MSBuildThisFileDirectory)..\..\assets\icons\app.ico</AppIconPath>
@@ -22,4 +23,9 @@
<PackageReference Include="WixToolset.UI.wixext" />
</ItemGroup>
<ItemGroup>
<!-- FindNewerRelease and KeepInstallerPackage (Installer.CustomActions.CA.dll). -->
<ProjectReference Include="..\Installer.CustomActions\Installer.CustomActions.csproj" />
</ItemGroup>
</Project>
+49 -4
View File
@@ -30,16 +30,53 @@
<SummaryInformation Description="ExecuteR Agent $(UnifiedVersion)" />
<MajorUpgrade AllowSameVersionUpgrades="yes"
Schedule="afterInstallInitialize"
DowngradeErrorMessage="A newer version of ExecuteR Agent is already installed." />
<!--
Versioning (README, Versioning). ProductVersion is the four-field version
(1.4.0.0 for release 1.4.0, 1.4.0.7 for a build seven commits after it).
Windows Installer compares only three fields and knows nothing of the
calendar era before 1.0.0, whose MSIs carry 26.x, so every installed
version is removed (AllowDowngrades) and FindNewerRelease refuses a newer
one first, all four fields and the era considered. The same version again
reinstalls; ALLOWDOWNGRADE=1 (an MSI upgrade's rollback) installs an
older one. Uninstalling the old product as part of the upgrade sets
UPGRADINGPRODUCTCODE, so its credentials are kept either way.
-->
<MajorUpgrade AllowDowngrades="yes"
Schedule="afterInstallInitialize" />
<MediaTemplate EmbedCab="yes" />
<!-- Full unified version retained in MSI metadata / ARP. -->
<!-- The full version the agent reports (X.Y.Z or X.Y.Z.N), for
FindNewerRelease, KeepInstallerPackage's package.json and ARP. The
agent's MSI upgrades read an MSI's version from it (MsiOpenDatabase,
falling back to ProductVersion): keep it. -->
<Property Id="UNIFIEDVERSION" Value="$(UnifiedVersion)" />
<Property Id="ARPCOMMENTS" Value="ExecuteR Agent $(UnifiedVersion)" />
<Binary Id="ExecuteRCustomActions" SourceFile="$(var.Installer.CustomActions.TargetDir)Installer.CustomActions.CA.dll" />
<CustomAction Id="FindNewerRelease" BinaryRef="ExecuteRCustomActions" DllEntry="FindNewerRelease"
Execute="immediate" Return="check" />
<Property Id="ALLOWDOWNGRADE" Secure="yes" />
<Launch Condition="Installed OR NOT NEWER_RELEASE_INSTALLED OR ALLOWDOWNGRADE = &quot;1&quot;"
Message="A newer version of ExecuteR Agent, [NEWER_RELEASE_INSTALLED], is already installed." />
<!-- An upgrade (and a repair) lays every file down: a calendar-era release
stamped its binaries 2026.x, which Windows Installer counts as newer
than 1.x, so it would keep them, then remove them with the old product
and install none. -->
<SetProperty Id="REINSTALLMODE" Value="amus" Before="CostInitialize" Sequence="both"
Condition="REINSTALL OR WIX_UPGRADE_DETECTED" />
<!-- The MSI being installed, kept for an MSI upgrade's rollback under
%ProgramData%\ExecuteR\packages\<sha256>\<original file name> with a
package.json { version, sha256 } beside it (the newest 3). -->
<SetProperty Id="KeepInstallerPackage"
Value="Source=[OriginalDatabase];Version=[UNIFIEDVERSION];Destination=[CommonAppDataFolder]ExecuteR\packages"
Sequence="execute"
Before="KeepInstallerPackage" />
<CustomAction Id="KeepInstallerPackage" BinaryRef="ExecuteRCustomActions" DllEntry="KeepInstallerPackage"
Execute="deferred" Impersonate="no" Return="ignore" />
<?ifdef AppIconPath ?>
<Icon Id="AppIcon" SourceFile="$(AppIconPath)" />
<Property Id="ARPPRODUCTICON" Value="AppIcon" />
@@ -279,7 +316,15 @@
Order="99" Condition="NOT Installed" />
</UI>
<InstallUISequence>
<Custom Action="FindNewerRelease" After="FindRelatedProducts" Condition="NOT Installed" />
</InstallUISequence>
<InstallExecuteSequence>
<Custom Action="FindNewerRelease" After="FindRelatedProducts" Condition="NOT Installed" />
<Custom Action="KeepInstallerPackage"
After="InstallFiles"
Condition="NOT Installed AND NOT (REMOVE=&quot;ALL&quot;)" />
<Custom Action="ConfigureServiceExtras"
After="InstallServices"
Condition="NOT (REMOVE=&quot;ALL&quot;)" />
+2 -1
View File
@@ -18,7 +18,8 @@ public sealed record DeviceDto(
IReadOnlyList<string>? Tags = null,
bool IsOrphaned = false,
string? PublicIp = null,
IReadOnlyList<Guid>? GroupIds = null);
IReadOnlyList<Guid>? GroupIds = null,
string? AgentInstall = null);
/// <summary>
/// Device group resource. <c>MemberCount</c>/<c>OnlineCount</c> include the
@@ -11,6 +11,13 @@ namespace Shared.Contracts.Messages;
/// <paramref name="HealthDeadlineSeconds"/>. The agent waits a random 0 to
/// <paramref name="StartJitterSeconds"/> seconds before downloading, so one
/// batch of devices does not start in the same instant.
/// <para>
/// For an MSI upgrade (<paramref name="PackageType"/> "msi") the Rollback fields
/// name the MSI of the version the device runs now, when the server holds it, so
/// the runner can reinstall it if the new version is not healthy. The agent
/// prefers its own kept copy (<c>WatchdogFiles.KeptPackages</c>) and downloads
/// this one only when it has none.
/// </para>
/// </summary>
public sealed record AgentUpgradeCommandPayload(
Guid UpgradeId,
@@ -21,7 +28,11 @@ public sealed record AgentUpgradeCommandPayload(
long SizeBytes,
bool Revert,
int HealthDeadlineSeconds,
int StartJitterSeconds = 0);
int StartJitterSeconds = 0,
string? RollbackVersion = null,
string? RollbackDownloadUrl = null,
string? RollbackSha256 = null,
long? RollbackSizeBytes = null);
/// <summary>
/// Progress or outcome of one upgrade, published by the agent on
+4 -2
View File
@@ -5,7 +5,8 @@ namespace Shared.Contracts.Messages;
/// <summary>
/// Periodic agent heartbeat published on <c>agent.&lt;clientId&gt;.presence</c>.
/// <c>Runtime</c> is the agent build's runtime identifier (e.g. win-x64), which
/// picks the package it upgrades from; older agents leave it out.
/// picks the package it upgrades from; <c>Install</c> is an <c>AgentInstallKinds</c>
/// value (msi or portable). Older agents leave both out.
/// </summary>
public sealed record PresencePayload(
string ClientId,
@@ -15,7 +16,8 @@ public sealed record PresencePayload(
DateTimeOffset TimestampUtc,
long UptimeSeconds,
int ActiveJobs,
string? Runtime = null);
string? Runtime = null,
string? Install = null);
/// <summary>Inventory snapshot published on <c>agent.&lt;clientId&gt;.inventory</c>.</summary>
public sealed record InventoryPayload(InventoryReport Report);
+69 -11
View File
@@ -50,6 +50,10 @@ public static class AgentUpgradeReasons
public const string Watchdog = "watchdog";
/// <summary>Files could not be swapped (the previous version was put back).</summary>
public const string Swap = "swap";
/// <summary>Windows Installer failed or did not finish in time (its own transaction put the previous version back).</summary>
public const string Msi = "msi";
/// <summary>Another installation kept Windows Installer busy (1618) through every retry.</summary>
public const string InstallerBusy = "installer-busy";
/// <summary>The new version did not report healthy before the deadline (rolled back).</summary>
public const string HealthDeadline = "health-deadline";
/// <summary>The new version kept exiting (rolled back).</summary>
@@ -75,10 +79,20 @@ public static class AgentUpgradeSources
public const string Rollback = "rollback";
}
/// <summary>How an agent was installed, reported on presence (<c>PresencePayload.Install</c>).</summary>
public static class AgentInstallKinds
{
/// <summary>Installed by the MSI (Windows Installer knows the product): upgrades go through the MSI.</summary>
public const string Msi = "msi";
/// <summary>Unpacked from the portable package: upgrades swap files.</summary>
public const string Portable = "portable";
}
/// <summary>
/// Agent package types: the file the server holds for each platform. Windows
/// upgrades always use the portable zip (also for MSI installs: files are
/// swapped in place, the MSI keeps its registration).
/// Agent package types: the file the server holds for each platform. An agent
/// the MSI installed upgrades through the MSI (installed in process through the
/// Windows Installer API by the watchdog's detached runner); a portable install
/// upgrades by swapping files (see <see cref="ForDevice"/>).
/// </summary>
public static class AgentPackageTypes
{
@@ -101,25 +115,44 @@ public static class AgentPackageTypes
_ => null,
};
/// <summary>Types an upgrade can use (everything but the MSI).</summary>
public static readonly IReadOnlyList<string> Upgradable = [WindowsZip, LinuxX64, LinuxArm64, MacX64, MacArm64];
/// <summary>
/// The package a device upgrades from: the MSI for a Windows agent the MSI
/// installed (so Windows Installer, Programs and Features and detection tools
/// see the real version), otherwise the portable package of its runtime.
/// </summary>
public static string? ForDevice(string? runtimeIdentifier, string? install)
=> install == AgentInstallKinds.Msi && string.Equals(runtimeIdentifier, "win-x64", StringComparison.OrdinalIgnoreCase)
? Msi
: ForRuntime(runtimeIdentifier);
/// <summary>Types the version catalog holds and upgrades use (the MSI included).</summary>
public static readonly IReadOnlyList<string> Upgradable = [Msi, WindowsZip, LinuxX64, LinuxArm64, MacX64, MacArm64];
}
/// <summary>
/// The one version comparison used everywhere. Versions are
/// <c>yyyy.MM.dd.HHmm</c> (four numeric parts); anything that does not parse
/// (e.g. a development "0.0.0.0" or a custom label) sorts before every real
/// version, then ordinally.
/// The one version comparison used everywhere (server, agent, watchdog,
/// installer custom action, console). Versions are semantic
/// <c>major.minor.patch</c>, with a fourth field on builds between releases
/// (<c>1.4.0.7</c> = seven commits after 1.4.0); a missing fourth field is 0,
/// and <c>+metadata</c> (the commit) is ignored. The calendar era that came
/// before (<c>yyyy.MM.dd.HHmm</c>, major 2000 or more) sorts before every
/// semantic version. Anything that does not parse, and the development
/// fallback <c>0.0.0.0</c>, sorts before everything, then ordinally.
/// </summary>
public static class AgentVersions
{
/// <summary>Majors from this value on are calendar-era versions (2026.10.23.1400).</summary>
public const int CalendarEraMajor = 2000;
public static int Compare(string? left, string? right)
{
var a = Parse(left);
var b = Parse(right);
if (a is not null && b is not null)
{
return a.CompareTo(b);
// Semantic (era 1) after calendar (era 0), then field by field.
var era = Era(a).CompareTo(Era(b));
return era != 0 ? era : a.CompareTo(b);
}
if (a is not null)
@@ -137,8 +170,33 @@ public static class AgentVersions
public static bool IsNewer(string? candidate, string? than) => Compare(candidate, than) > 0;
/// <summary>The same version, ignoring a missing fourth field and <c>+metadata</c> (1.4.0 == 1.4.0.0).</summary>
public static bool AreSame(string? left, string? right) => Parse(left) is not null && Compare(left, right) == 0;
/// <summary>True for a version this comparison understands (not the 0.0.0.0 fallback).</summary>
public static bool IsValid(string? value) => Parse(value) is not null;
public static bool IsCalendarEra(string? value) => Parse(value) is { } version && Era(version) == 0;
public static IComparer<string?> Comparer { get; } = Comparer<string?>.Create(Compare);
private static int Era(Version version) => version.Major >= CalendarEraMajor ? 0 : 1;
/// <summary>3 or 4 numeric fields, <c>+metadata</c> stripped; missing fourth field normalized to 0.</summary>
private static Version? Parse(string? value)
=> Version.TryParse(value, out var version) && version.Major > 0 ? version : null;
{
if (string.IsNullOrWhiteSpace(value) || value.Length > 64)
{
return null;
}
var core = value.Split('+')[0].Trim();
if (!Version.TryParse(core, out var version) || core.Split('.').Length < 3)
{
return null;
}
var normalized = new Version(version.Major, version.Minor, version.Build, Math.Max(version.Revision, 0));
return normalized == new Version(0, 0, 0, 0) ? null : normalized;
}
}
+90 -3
View File
@@ -27,6 +27,21 @@ public static class WatchdogFiles
public const string PreviousFolder = ".previous";
public const string FailedFolder = ".failed";
/// <summary>MSI upgrades: the runner's journal (<see cref="WatchdogMsiJournal"/>), beside handover.json.</summary>
public const string MsiJournal = "msi.json";
/// <summary>MSI upgrades: <c>upgrade/msi/</c> holds the MSIs being installed and the runner's copy of the watchdog binary.</summary>
public const string MsiFolder = "msi";
/// <summary>
/// MSIs kept for rollback under <c>&lt;data&gt;/packages/&lt;sha256&gt;/&lt;file&gt;.msi</c>
/// (a data-root folder, not under upgrade/): the MSI an install ran from
/// (copied by the installer's KeepInstallerPackage custom action) and those
/// later upgrades brought; the newest 3 are kept.
/// </summary>
public const string KeptPackages = "packages";
public const int KeptPackagesMax = 3;
/// <summary>The longest an MSI install (or the reinstall of the previous one) may run.</summary>
public static readonly TimeSpan MsiInstallTimeout = TimeSpan.FromMinutes(30);
public static readonly TimeSpan BeaconInterval = TimeSpan.FromSeconds(5);
/// <summary>A beacon older than this while the agent should run: the watchdog restarts it (at most once per period).</summary>
public static readonly TimeSpan BeaconStaleAfter = TimeSpan.FromMinutes(3);
@@ -48,7 +63,24 @@ public sealed record WatchdogHandover(
IReadOnlyList<WatchdogFile> Files,
int HealthDeadlineSeconds,
bool Revert,
DateTimeOffset CreatedUtc);
DateTimeOffset CreatedUtc,
string Kind = WatchdogHandoverKinds.Files,
WatchdogFile? Msi = null,
WatchdogFile? RollbackMsi = null,
string? RollbackVersion = null);
/// <summary>
/// How a hand-over is installed: <c>files</c> = the journalled in-place swap;
/// <c>msi</c> = the watchdog starts a detached copy of itself that installs
/// <c>Msi</c> through the Windows Installer API (no msiexec), checks the new
/// version's health and reinstalls <c>RollbackMsi</c> if it is not healthy.
/// For <c>msi</c>, <c>Msi.Path</c> and <c>RollbackMsi.Path</c> are absolute.
/// </summary>
public static class WatchdogHandoverKinds
{
public const string Files = "files";
public const string Msi = "msi";
}
/// <summary>
/// The agent's heartbeat to the watchdog. <c>Connected</c> means its command
@@ -88,7 +120,61 @@ public sealed record WatchdogJournal(
DateTimeOffset? DeadlineUtc,
int Attempts);
/// <summary>How a swap ended: <c>Result</c> is succeeded, failed or rolled-back (<c>AgentUpgradeStates</c>).</summary>
/// <summary>Phases of an MSI upgrade, run by the detached runner.</summary>
public static class WatchdogMsiPhases
{
public const string Installing = "installing";
public const string Verifying = "verifying";
public const string RollingBack = "rolling-back";
}
/// <summary>
/// The MSI runner's journal. <c>RunnerProcessId</c> is the runner that owns the
/// run (it also holds <see cref="WatchdogMsiFiles.RunnerLock"/> while it lives, which
/// is how a restarted watchdog tells a live runner from one that died and
/// resumes it); <c>Attempts</c> counts Windows Installer busy (1618) retries.
/// <c>StartedUtc</c> is when the current phase's install began (a beacon must be
/// newer), <c>DeadlineUtc</c> the health deadline while verifying.
/// <c>Revert</c>: a rollback from the console, installed with
/// <c>ALLOWDOWNGRADE=1</c> from the start. <c>Resumes</c>: how many times a dead
/// runner was started again for this run (bounded).
/// </summary>
public sealed record WatchdogMsiJournal(
Guid UpgradeId,
string Version,
string FromVersion,
string Phase,
WatchdogFile Msi,
WatchdogFile? RollbackMsi,
string? RollbackVersion,
int HealthDeadlineSeconds,
int RunnerProcessId,
int Attempts,
int? LastExitCode,
DateTimeOffset StartedUtc,
DateTimeOffset? DeadlineUtc,
bool Revert = false,
int Resumes = 0);
/// <summary>Files of an MSI upgrade beside <see cref="WatchdogFiles.MsiJournal"/>, private to the watchdog and its runner.</summary>
public static class WatchdogMsiFiles
{
/// <summary>Held open (no sharing) by the runner for as long as it runs.</summary>
public const string RunnerLock = "msi.lock";
/// <summary><c>upgrade/msi/runner/</c>: the copy of the watchdog binary the runner runs from (the MSI replaces the installed one).</summary>
public const string RunnerFolder = "runner";
/// <summary><c>upgrade/msi/rollback/</c>: the MSI of the version the device ran, to go back to.</summary>
public const string RollbackFolder = "rollback";
/// <summary>Hidden watchdog verb the detached runner is started with: <c>msi-run &lt;upgradeId&gt; --install-dir … --data-dir …</c>.</summary>
public const string RunVerb = "msi-run";
}
/// <summary>
/// How a swap or an MSI run ended: <c>Result</c> is succeeded, failed or
/// rolled-back (<c>AgentUpgradeStates</c>); <c>Detail</c>, when set, is the
/// sentence the agent reports (an MSI run's exit code, log path, or that no MSI
/// was kept to go back to).
/// </summary>
public sealed record WatchdogOutcome(
Guid UpgradeId,
string Result,
@@ -96,4 +182,5 @@ public sealed record WatchdogOutcome(
string Version,
string FromVersion,
IReadOnlyList<string> LastLog,
DateTimeOffset FinishedUtc);
DateTimeOffset FinishedUtc,
string? Detail = null);
@@ -19,4 +19,5 @@ namespace Shared.Contracts.Serialization;
[JsonSerializable(typeof(WatchdogHealthBeacon))]
[JsonSerializable(typeof(WatchdogJournal))]
[JsonSerializable(typeof(WatchdogOutcome))]
[JsonSerializable(typeof(WatchdogMsiJournal))]
public sealed partial class WatchdogJsonContext : JsonSerializerContext;
@@ -41,6 +41,16 @@ public static class Branding
public const string WatchdogSystemdUnit = "executer-watchdog";
public const string WatchdogLaunchdLabel = "info.executer.watchdog";
/// <summary>
/// The agent MSI's UpgradeCode (src/Installer/Product.wxs): Windows Installer
/// knows an MSI install of the agent by it (MsiEnumRelatedProducts), and the
/// agent checks a downloaded or kept MSI is the agent's before using it.
/// </summary>
public const string AgentMsiUpgradeCode = "{7F9C1A52-3F89-4BD4-9E11-2A67C33A5F10}";
/// <summary>The file name an agent MSI of a version is kept and installed under.</summary>
public static string AgentMsiFileName(string version) => $"ExecuteR-Agent-{version.Split('+')[0]}.msi";
/// <summary>The agent binary's file name on the running platform.</summary>
public static string AgentBinaryName => OperatingSystem.IsWindows() ? AgentExecutableName : AgentUnixExecutableName;
@@ -14,6 +14,9 @@ public static class RuleEvaluator
{
private static readonly TimeSpan RegexTimeout = TimeSpan.FromMilliseconds(250);
/// <summary>Ordered by version, not by number or text (<see cref="CompareVersion"/>).</summary>
private const string AgentVersionField = "agentVersion";
/// <summary>Operators that are inherently negative and so must hold for ALL values.</summary>
private static bool IsUniversal(RuleOperator op)
=> op is RuleOperator.NotEquals or RuleOperator.NotContains or RuleOperator.NotIn
@@ -129,7 +132,9 @@ public static class RuleEvaluator
case RuleOperator.GreaterOrEqual:
case RuleOperator.LessThan:
case RuleOperator.LessOrEqual:
return CompareNumeric(actual, expected, condition.Operator);
return string.Equals(condition.Field, AgentVersionField, StringComparison.OrdinalIgnoreCase)
? CompareVersion(actual, expected, condition.Operator)
: CompareNumeric(actual, expected, condition.Operator);
case RuleOperator.InCidr:
return MatchesCidr(actual, expected);
@@ -203,6 +208,32 @@ public static class RuleEvaluator
return true;
}
/// <summary>
/// The agent version orders the way upgrades do (<see cref="AgentVersions"/>):
/// 1.10.0 after 1.9.0, 1.4.0.7 after 1.4.0, every calendar-era version
/// before 1.0.0. A rule value of one or two numbers means its release line
/// (<c>1.4</c> is <c>1.4.0</c>).
/// </summary>
private static bool CompareVersion(string actual, string expected, RuleOperator op)
{
var bound = expected.Trim();
var fields = bound.Split('.');
if (fields.Length < 3 && fields.All(f => f.Length > 0 && f.All(char.IsAsciiDigit)))
{
bound += string.Concat(Enumerable.Repeat(".0", 3 - fields.Length));
}
var comparison = AgentVersions.Compare(actual, bound);
return op switch
{
RuleOperator.GreaterThan => comparison > 0,
RuleOperator.GreaterOrEqual => comparison >= 0,
RuleOperator.LessThan => comparison < 0,
RuleOperator.LessOrEqual => comparison <= 0,
_ => false,
};
}
private static bool CompareNumeric(string actual, string expected, RuleOperator op)
{
if (!double.TryParse(actual, NumberStyles.Float, CultureInfo.InvariantCulture, out var left)
@@ -1,74 +1,38 @@
using System.Globalization;
using System.Reflection;
namespace Shared.Infrastructure.Versioning;
/// <summary>
/// Unified build version (<c>yyyy.MM.dd.HHmm</c>, UTC) shared by every artifact.
/// The value is stamped at build time via the UnifiedVersion MSBuild property.
/// The build version shared by every artifact: a release's <c>X.Y.Z</c>, or
/// <c>X.Y.Z.N</c> for a build N commits after it (README, Versioning). It is
/// stamped at build time through the UnifiedVersion MSBuild property; versions
/// are compared with <see cref="Shared.Contracts.Models.AgentVersions"/>.
/// </summary>
public static class BuildVersion
{
/// <summary>The unified version of the running application's entry assembly.</summary>
/// <summary>The development version of a binary built without one.</summary>
public const string Development = "0.0.0.0";
/// <summary>The version of the running application's entry assembly.</summary>
public static string Current { get; } = Resolve(Assembly.GetEntryAssembly());
public static string Resolve(Assembly? assembly)
{
var informational = assembly?
.GetCustomAttribute<AssemblyInformationalVersionAttribute>()?
.InformationalVersion;
if (string.IsNullOrWhiteSpace(informational))
{
return "0.0.0.0";
}
// The binary keeps its full informational version (date-hash) for
// support; the REPORTED version (enrollment, presence, /health, CLI)
// is the clean unified yyyy.MM.dd.HHmm. Strip build metadata (+sha)
// and the short-commit suffix (-sha) — the date part has no '-'.
var trimmed = informational;
var plus = trimmed.IndexOf('+');
if (plus >= 0)
{
trimmed = trimmed[..plus];
}
var dash = trimmed.IndexOf('-');
if (dash >= 0)
{
trimmed = trimmed[..dash];
}
return trimmed;
}
=> FromInformational(assembly?.GetCustomAttribute<AssemblyInformationalVersionAttribute>()?.InformationalVersion);
/// <summary>
/// Deterministic MSI ProductVersion transform. MSI limits: major ≤ 255,
/// minor ≤ 255, build ≤ 65535. Mapping keeps ordering within a year:
/// <c>yyyy.MM.dd.HHmm</c> → <c>(yyyy-2000).(MM).((dd-1)*1440 + HH*60 + mm)</c>.
/// The full unified version is retained in MSI display metadata.
/// The reported version (enrollment, presence, /health, CLI) from an
/// informational version: the binary keeps <c>1.4.0.7+1a2b3c4</c> for
/// support; the version is the part before the build metadata.
/// </summary>
public static string ToMsiProductVersion(string unifiedVersion)
public static string FromInformational(string? informational)
{
var parts = unifiedVersion.Split('.');
if (parts.Length != 4
|| !int.TryParse(parts[0], NumberStyles.None, CultureInfo.InvariantCulture, out var year)
|| !int.TryParse(parts[1], NumberStyles.None, CultureInfo.InvariantCulture, out var month)
|| !int.TryParse(parts[2], NumberStyles.None, CultureInfo.InvariantCulture, out var day)
|| !int.TryParse(parts[3], NumberStyles.None, CultureInfo.InvariantCulture, out var hhmm))
if (string.IsNullOrWhiteSpace(informational))
{
throw new FormatException($"'{unifiedVersion}' is not a unified yyyy.MM.dd.HHmm version.");
return Development;
}
var major = year - 2000;
var build = (day - 1) * 1440 + (hhmm / 100) * 60 + hhmm % 100;
if (major is < 0 or > 255 || month is < 1 or > 12 || build is < 0 or > 65535)
{
throw new FormatException($"'{unifiedVersion}' cannot be mapped to an MSI ProductVersion.");
}
return $"{major}.{month}.{build}";
var plus = informational.IndexOf('+');
var version = (plus >= 0 ? informational[..plus] : informational).Trim();
return version.Length == 0 ? Development : version;
}
}
+4
View File
@@ -27,8 +27,12 @@ export interface DeviceDto {
isOrphaned?: boolean;
/** Groups the device is directly in (absent when no groups exist). */
groupIds?: string[];
/** How the agent was installed (from presence): msi upgrades through Windows Installer, portable by swapping files. */
agentInstall?: AgentInstallKind;
}
export type AgentInstallKind = 'msi' | 'portable';
// ── Device grouping ───────────────────────────────────────────────────
/** Folders hold no devices of their own; every kind includes its nested groups' devices. */
+8 -1
View File
@@ -289,7 +289,14 @@ export default function DevicesPage() {
header: 'Agent',
nowrap: true,
sortValue: (d) => versionSortKey(d.agentVersion),
cell: (d) => <AgentVersion version={d.agentVersion} newest={newestFor(versions, d.osCaption)} open={openByDevice.get(d.clientId)} />
cell: (d) => (
<AgentVersion
version={d.agentVersion}
newest={newestFor(versions, d.osCaption, d.agentInstall)}
open={openByDevice.get(d.clientId)}
install={d.agentInstall}
/>
)
},
{
id: 'lastSeen',
@@ -22,7 +22,7 @@ import Properties from 'components/Properties';
import { deviceUpgradesKey } from 'sections/devices/DeviceUpgrades';
import { newestFor } from 'sections/upgrades/agentVersions';
import { AgentVersion, UpgradeStateChip } from 'sections/upgrades/UpgradeChips';
import { isOpenUpgrade } from 'sections/upgrades/upgradeLabels';
import { INSTALL_KINDS, isOpenUpgrade } from 'sections/upgrades/upgradeLabels';
import useApi from 'hooks/useApi';
import useLiveRefresh from 'hooks/useLiveRefresh';
import { api, errorMessage, keys } from 'api/client';
@@ -74,6 +74,7 @@ export default function DeviceOverview({ detail, canManage, onChanged }: Props)
const byNewest = [...(upgrades ?? [])].sort((a, b) => b.createdUtc.localeCompare(a.createdUtc));
const open = byNewest.find((u) => isOpenUpgrade(u.state));
const last = byNewest.find((u) => !isOpenUpgrade(u.state) && u.finishedUtc);
const installKind = device.agentInstall ? INSTALL_KINDS[device.agentInstall] : undefined;
return (
<Grid container spacing={3}>
@@ -147,7 +148,15 @@ export default function DeviceOverview({ detail, canManage, onChanged }: Props)
items={[
{
label: 'Version',
value: <AgentVersion version={device.agentVersion} newest={newestFor(versions, device.osCaption)} />
value: <AgentVersion version={device.agentVersion} newest={newestFor(versions, device.osCaption, device.agentInstall)} />
},
{
label: 'Install',
value: installKind ? (
<Tooltip title={installKind.help}>
<Chip size="small" variant="outlined" label={installKind.label} />
</Tooltip>
) : undefined
},
...(open
? [
@@ -8,7 +8,7 @@ import Typography from '@mui/material/Typography';
// project imports
import Dot from 'components/@extended/Dot';
import { upgradeAvailable } from './agentVersions';
import { UPGRADE_REASONS, UPGRADE_STATES } from './upgradeLabels';
import { INSTALL_KINDS, UPGRADE_REASONS, UPGRADE_STATES } from './upgradeLabels';
import type { AgentUpgradeDto, AgentUpgradeState } from 'api/types';
// assets
@@ -47,13 +47,27 @@ export function UpgradeStateChip({
// ==============================|| AGENT VERSION (WITH UPGRADE MARKERS) ||============================== //
/** A device's agent version, marked when a newer one is available and while an upgrade is open. */
export function AgentVersion({ version, newest, open }: { version?: string; newest?: string; open?: AgentUpgradeDto }) {
/** A device's agent version, marked when a newer one is available and while an upgrade is open; how it was installed on hover. */
export function AgentVersion({
version,
newest,
open,
install
}: {
version?: string;
newest?: string;
open?: AgentUpgradeDto;
install?: string;
}) {
const kind = install ? INSTALL_KINDS[install] : undefined;
const text = (
<Typography variant="body2" component="span" sx={{ fontFamily: 'var(--font-mono)' }} color={version ? undefined : 'text.secondary'}>
{version ?? '—'}
</Typography>
);
return (
<Stack direction="row" sx={{ alignItems: 'center', gap: 0.75, whiteSpace: 'nowrap' }}>
<Typography variant="body2" component="span" sx={{ fontFamily: 'var(--font-mono)' }} color={version ? undefined : 'text.secondary'}>
{version ?? '—'}
</Typography>
{kind && version ? <Tooltip title={`${kind.label}. ${kind.help}`}>{text}</Tooltip> : text}
{!open && upgradeAvailable(version, newest) && (
<Tooltip title={`${newest} is available.`}>
<Box
@@ -6,36 +6,53 @@ import { platformOf } from 'components/OsIcon';
import { keys } from 'api/client';
import type { ActivityEventPayload, AgentVersionDto } from 'api/types';
/** The four numbers of a yyyy.MM.dd.HHmm version, or null for anything else (a development 0.0.0.0, a custom label). */
// The server's one version order (AgentVersions in Shared.Contracts), mirrored exactly: semantic major.minor.patch with a fourth
// field on builds between releases (1.4.0.7; missing = 0), +metadata ignored. The calendar era before it (yyyy.MM.dd.HHmm, major
// 2000 or more) sorts before every semantic version; anything that does not parse, and the development 0.0.0.0, before everything.
const calendarEraMajor = 2000;
// One field the way .NET's Version.TryParse reads it (an int, ASCII whitespace around it, a sign).
const field = /^[\t\n\v\f\r ]*[+-]?\d+[\t\n\v\f\r ]*$/;
/** Era (0 calendar, 1 semantic) then the four numbers, or null for anything else (0.0.0.0, a label). */
function parse(version?: string | null): number[] | null {
const parts = (version ?? '').split('.');
if (parts.length < 2 || parts.length > 4 || !parts.every((p) => /^\d+$/.test(p))) return null;
const numbers = [0, 1, 2, 3].map((i) => Number(parts[i] ?? 0));
return numbers[0] > 0 ? numbers : null;
if (!version || !version.trim() || version.length > 64) return null;
const parts = version.split('+')[0].trim().split('.');
if (parts.length < 3 || parts.length > 4 || !parts.every((p) => field.test(p))) return null;
// -0 reads as 0 (as in .NET); any other negative, or a number past Int32, does not parse.
const numbers = [0, 1, 2, 3].map((i) => Math.abs(Number(parts[i] ?? 0)));
if (parts.some((p) => Number(p) < 0) || numbers.some((n) => n > 2147483647)) return null;
if (numbers.every((n) => n === 0)) return null;
return [numbers[0] >= calendarEraMajor ? 0 : 1, ...numbers];
}
/** The server's one version order: real versions by their numbers; anything else before them, then by text. */
/** The server's one version order: by era, then field by field; anything that does not parse before them, then ordinally. */
export function compareVersions(left?: string | null, right?: string | null): number {
const a = parse(left);
const b = parse(right);
if (a && b) return a.reduce((result, n, i) => result || n - b[i], 0);
if (a && b) return Math.sign(a.reduce((result, n, i) => result || n - b[i], 0));
if (a) return 1;
if (b) return -1;
return (left ?? '').localeCompare(right ?? '');
// string.CompareOrdinal: no value before any text, then UTF-16 code unit by code unit.
const l = left ?? null;
const r = right ?? null;
if (l === null || r === null) return l === r ? 0 : l === null ? -1 : 1;
return l < r ? -1 : l > r ? 1 : 0;
}
/** A text key that sorts like `compareVersions` (for table columns). */
export function versionSortKey(version?: string | null): string | undefined {
if (!version) return undefined;
const parts = parse(version);
return parts ? `1${parts.map((n) => String(n).padStart(10, '0')).join('')}` : `0${version}`;
if (!parts) return `0${version}`;
const [era, ...numbers] = parts;
return `1${era}${numbers.map((n) => String(n).padStart(10, '0')).join('')}`;
}
/** Package types an agent on this OS upgrades from. */
function packagesFor(osCaption?: string | null): string[] {
/** Package types an agent on this OS upgrades from (the MSI when the MSI installed it). */
function packagesFor(osCaption?: string | null, install?: string | null): string[] {
switch (platformOf(osCaption)) {
case 'windows':
return ['zip'];
return install === 'msi' ? ['msi'] : ['zip'];
case 'linux':
return ['linux-x64', 'linux-arm64'];
case 'macos':
@@ -46,8 +63,8 @@ function packagesFor(osCaption?: string | null): string[] {
}
/** The newest catalog version holding a package for the device's platform (by its OS; the server decides by architecture). */
export function newestFor(versions: AgentVersionDto[] | undefined, osCaption?: string | null): string | undefined {
const types = packagesFor(osCaption);
export function newestFor(versions: AgentVersionDto[] | undefined, osCaption?: string | null, install?: string | null): string | undefined {
const types = packagesFor(osCaption, install);
return (versions ?? [])
.filter((v) => v.packages.some((p) => types.includes(p.packageType)))
.map((v) => v.version)
@@ -10,14 +10,14 @@ export const UPGRADE_STATES: Record<AgentUpgradeState, { label: string; help: st
downloading: { label: 'Downloading', help: 'The agent is downloading the package.', tone: 'primary' },
installing: {
label: 'Installing',
help: 'The watchdog is swapping the files and waiting for the new version to report healthy.',
help: 'The watchdog is installing the new version and waiting for it to report healthy.',
tone: 'primary'
},
succeeded: { label: 'Upgraded', help: 'The new version reported healthy.', tone: 'success' },
failed: { label: 'Failed', help: 'The upgrade stopped; the previous version still runs.', tone: 'error' },
'rolled-back': {
label: 'Rolled Back',
help: 'The new version was not healthy, so the watchdog put the previous one back.',
help: 'The new version was not healthy, so the previous one was put back.',
tone: 'warning'
},
cancelled: { label: 'Cancelled', help: 'Cancelled or replaced before it was sent.', tone: 'secondary' }
@@ -42,6 +42,11 @@ export const UPGRADE_REASONS: Record<string, { label: string; help: string }> =
package: { label: 'Bad Package', help: 'The package could not be unpacked or has no agent in it.' },
watchdog: { label: 'Watchdog Unavailable', help: 'The watchdog is missing or did not pick the upgrade up in time.' },
swap: { label: "Files Couldn't Be Swapped", help: 'The files could not be swapped; the previous version was put back.' },
msi: { label: 'Windows Installer Failed', help: 'Windows Installer did not install the new version; the previous one stays.' },
'installer-busy': {
label: 'Windows Installer Busy',
help: 'Another installation kept Windows Installer busy through every retry; the previous version stays.'
},
'health-deadline': {
label: 'Not Healthy in Time',
help: 'The new version did not report healthy before the deadline, so it was rolled back.'
@@ -66,13 +71,22 @@ export const UPGRADE_SOURCES: Record<string, { label: string; help: string }> =
/** Platform names for the package types a version holds. */
export const PACKAGE_TYPES: Record<string, { label: string; help?: string }> = {
zip: { label: 'Windows' },
msi: { label: 'Windows MSI', help: 'For new installs; upgrades use the Windows zip.' },
zip: { label: 'Windows', help: 'Upgrades portable Windows installs by swapping files.' },
msi: { label: 'Windows (MSI)', help: 'New installs, and upgrades of agents the MSI installed, through Windows Installer.' },
'linux-x64': { label: 'Linux x64' },
'linux-arm64': { label: 'Linux Arm64' },
'osx-x64': { label: 'macOS x64' },
'osx-arm64': { label: 'macOS Arm64' }
};
/** How a device's agent was installed, which decides how it upgrades. */
export const INSTALL_KINDS: Record<string, { label: string; help: string }> = {
msi: {
label: 'Installed by MSI',
help: 'MSI installs upgrade through Windows Installer, so Programs and Features shows the real version.'
},
portable: { label: 'Portable', help: 'Portable installs upgrade by swapping files in place.' }
};
/** Display order of package types. */
export const PACKAGE_ORDER = ['zip', 'msi', 'linux-x64', 'linux-arm64', 'osx-x64', 'osx-arm64'];
+269
View File
@@ -0,0 +1,269 @@
using System.Net;
using System.Security.Cryptography;
using System.Text;
using Agent.Upgrade;
using Microsoft.Extensions.Logging.Abstractions;
using Shared.Contracts.Messages;
using Shared.Contracts.Models;
using Shared.Contracts.Serialization;
using Shared.Infrastructure.Routing;
using Xunit;
namespace Agent.Tests;
/// <summary>An MSI is a file whose text is "msi &lt;version&gt;"; anything else is not the agent's MSI.</summary>
internal sealed class FakeMsiInspector : IMsiInspector
{
public string Install { get; set; } = AgentInstallKinds.Msi;
public string DetectInstall(string installDirectory) => Install;
public string? ReadAgentVersion(string msiPath)
{
var text = File.Exists(msiPath) ? File.ReadAllText(msiPath) : "";
return text.StartsWith("msi ", StringComparison.Ordinal) ? text["msi ".Length..] : null;
}
}
public sealed class AgentMsiUpgradeTests : IDisposable
{
private const string Current = "1.3.0";
private const string Target = "1.4.0";
private const string MsiUrl = "/api/agent/packages/by-sha/target";
private const string RollbackUrl = "/api/agent/packages/by-sha/rollback";
private readonly string root = Path.Combine(Path.GetTempPath(), "executer-msi-upgrade-tests", Guid.NewGuid().ToString("N"));
private readonly UpgradeFolders folders;
private readonly RecordingEvents events = new();
private readonly FakeWatchdog watchdog = new();
private readonly FakeMsiInspector inspector = new();
private readonly List<string> requests = [];
private byte[] target = Encoding.UTF8.GetBytes($"msi {Target}");
private readonly byte[] previous = Encoding.UTF8.GetBytes($"msi {Current}");
private WatchdogHandover? handedOver;
public AgentMsiUpgradeTests()
{
folders = new UpgradeFolders(Path.Combine(root, "data"), Path.Combine(root, "install"));
Directory.CreateDirectory(folders.InstallDirectory);
}
public void Dispose()
{
try
{
Directory.Delete(root, recursive: true);
}
catch (IOException)
{
}
}
private static string Sha(byte[] bytes) => Convert.ToHexStringLower(SHA256.HashData(bytes));
private AgentUpgradeHandler CreateHandler(string install = AgentInstallKinds.Msi)
{
inspector.Install = install;
return new AgentUpgradeHandler(new AgentUpgradeContext
{
Folders = folders,
ClientId = "client-1",
CurrentVersion = Current,
RuntimeIdentifier = "win-x64",
CanSelfUpgrade = true,
InstallKind = install,
MsiInspector = inspector,
ServerUrl = ServerUrl.Parse("https://control.example.com"),
CreateHttpHandler = () => new Stub(request =>
{
var path = request.RequestUri!.AbsolutePath;
requests.Add(path);
return new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new ByteArrayContent(path.EndsWith("rollback", StringComparison.Ordinal) ? previous : target),
};
}),
Delay = (_, _) =>
{
// The watchdog takes the hand-over at its first look.
handedOver ??= WatchdogFileStore.Read(folders.Handover, WatchdogJsonContext.Default.WatchdogHandover);
File.Delete(folders.Handover);
return Task.CompletedTask;
},
RandomMilliseconds = max => max,
Gate = new UpgradeGate(),
}, events, watchdog, NullLogger.Instance);
}
private AgentUpgradeCommandPayload Command(bool rollbackFields = false, string packageType = AgentPackageTypes.Msi)
=> new(Guid.NewGuid(), Target, packageType, MsiUrl, Sha(target), target.Length, Revert: false, HealthDeadlineSeconds: 600,
RollbackVersion: rollbackFields ? Current : null,
RollbackDownloadUrl: rollbackFields ? RollbackUrl : null,
RollbackSha256: rollbackFields ? Sha(previous) : null,
RollbackSizeBytes: rollbackFields ? previous.Length : null);
/// <summary>The MSI the device was installed from, as the installer's KeepInstallerPackage leaves it (no description file).</summary>
private string KeepInstalled(byte[] bytes, string? folderName = null)
{
var folder = Path.Combine(folders.KeptPackages, folderName ?? Sha(bytes));
Directory.CreateDirectory(folder);
var path = Path.Combine(folder, "ExecuteR-Agent.msi");
File.WriteAllBytes(path, bytes);
return path;
}
private AgentUpgradeEventPayload Last => events.Published[^1];
[Fact]
public async Task AnMsiInstallHandsTheMsiOverWithTheKeptMsiOfItsVersionToRollBackTo()
{
KeepInstalled(previous);
var command = Command();
await CreateHandler().RunAsync(command, CancellationToken.None);
Assert.Equal([MsiUrl], requests);
Assert.Equal(AgentUpgradeStates.Installing, Last.State);
Assert.NotNull(handedOver);
Assert.Equal(WatchdogHandoverKinds.Msi, handedOver.Kind);
Assert.Empty(handedOver.Files);
Assert.Equal(Current, handedOver.FromVersion);
var msi = handedOver.Msi!;
Assert.Equal(Path.Combine(folders.Msi, "ExecuteR-Agent-1.4.0.msi"), msi.Path);
Assert.True(Path.IsPathFullyQualified(msi.Path));
Assert.Equal(Sha(target), msi.Sha256);
Assert.Equal(target.Length, msi.Size);
var rollback = handedOver.RollbackMsi!;
Assert.Equal(Path.Combine(folders.MsiRollback, "ExecuteR-Agent-1.3.0.msi"), rollback.Path);
Assert.Equal(Sha(previous), rollback.Sha256);
Assert.Equal(previous, File.ReadAllBytes(rollback.Path));
Assert.Equal(Current, handedOver.RollbackVersion);
// The MSI the upgrade brought is kept for the next one's rollback.
Assert.Equal(target, File.ReadAllBytes(Path.Combine(folders.KeptPackages, Sha(target), "ExecuteR-Agent-1.4.0.msi")));
Assert.Equal(folders.Msi, watchdog.EnsuredFor);
}
[Fact]
public async Task WithoutAKeptMsiTheServersMsiOfTheRunningVersionIsDownloadedCheckedAndKept()
{
await CreateHandler().RunAsync(Command(rollbackFields: true), CancellationToken.None);
Assert.Equal([MsiUrl, RollbackUrl], requests);
Assert.Equal(Sha(previous), handedOver!.RollbackMsi!.Sha256);
Assert.Equal(Current, handedOver.RollbackVersion);
Assert.True(File.Exists(Path.Combine(folders.KeptPackages, Sha(previous), "ExecuteR-Agent-1.3.0.msi")));
}
[Fact]
public async Task WithNoMsiToGoBackToTheUpgradeGoesOnAndSaysItCannotRollBack()
{
await CreateHandler().RunAsync(Command(), CancellationToken.None);
Assert.NotNull(handedOver);
Assert.Null(handedOver.RollbackMsi);
Assert.Null(handedOver.RollbackVersion);
Assert.Equal(AgentUpgradeStates.Installing, Last.State);
Assert.Contains("cannot be rolled back automatically", Last.Detail, StringComparison.Ordinal);
}
[Fact]
public async Task AKeptMsiWhoseBytesDoNotMatchItsFolderIsNotUsed()
{
KeepInstalled(previous, folderName: new string('a', 64));
await CreateHandler().RunAsync(Command(rollbackFields: true), CancellationToken.None);
Assert.Equal([MsiUrl, RollbackUrl], requests);
Assert.NotNull(handedOver!.RollbackMsi);
}
[Fact]
public async Task APortableInstallRefusesTheMsi()
{
await CreateHandler(AgentInstallKinds.Portable).RunAsync(Command(), CancellationToken.None);
Assert.Equal(AgentUpgradeStates.Failed, Last.State);
Assert.Equal(AgentUpgradeReasons.Unsupported, Last.Reason);
Assert.Empty(requests);
}
[Fact]
public async Task AnMsiInstallRefusesThePortableZip()
{
await CreateHandler().RunAsync(Command(packageType: AgentPackageTypes.WindowsZip), CancellationToken.None);
Assert.Equal(AgentUpgradeReasons.Unsupported, Last.Reason);
Assert.Contains("through the MSI", Last.Detail, StringComparison.Ordinal);
Assert.Empty(requests);
}
[Fact]
public async Task AnMsiOfAnotherVersionFailsAsPackage()
{
target = Encoding.UTF8.GetBytes("msi 1.5.0");
await CreateHandler().RunAsync(Command(), CancellationToken.None);
Assert.Equal(AgentUpgradeReasons.Package, Last.Reason);
Assert.Null(handedOver);
}
[Fact]
public async Task APackageThatIsNotTheAgentsMsiFailsAsPackage()
{
target = Encoding.UTF8.GetBytes("PK zip bytes");
await CreateHandler().RunAsync(Command(), CancellationToken.None);
Assert.Equal(AgentUpgradeReasons.Package, Last.Reason);
Assert.False(File.Exists(folders.Handover));
}
[Fact]
public async Task AnMsiRunUnderWayMakesTheAgentBusy()
{
Directory.CreateDirectory(folders.Root);
File.WriteAllText(folders.MsiJournal, "{}");
await CreateHandler().RunAsync(Command(), CancellationToken.None);
Assert.Equal(AgentUpgradeReasons.Busy, Last.Reason);
Assert.Empty(requests);
}
[Fact]
public void KeptPackagesStayAtTheNewestThreePlusTheOnesStillNeeded()
{
var kept = new KeptMsiPackages(folders.KeptPackages, inspector, NullLogger.Instance);
var source = Path.Combine(root, "source.msi");
var start = DateTime.UtcNow.AddHours(-10);
string[] versions = ["1.0.0", "1.1.0", "1.2.0", Current, Target];
for (var i = 0; i < versions.Length; i++)
{
var bytes = Encoding.UTF8.GetBytes($"msi {versions[i]}");
File.WriteAllBytes(source, bytes);
kept.Keep(source, Sha(bytes), versions[i], "1.0.0");
var folder = Path.Combine(folders.KeptPackages, Sha(bytes));
foreach (var file in Directory.GetFiles(folder))
{
File.SetLastWriteTimeUtc(file, start.AddHours(i));
}
Directory.SetLastWriteTimeUtc(folder, start.AddHours(i));
}
var left = Directory.GetDirectories(folders.KeptPackages).Select(d => File.ReadAllText(Directory.GetFiles(d, "*.msi").Single())).Order().ToList();
Assert.Equal(["msi 1.0.0", "msi 1.2.0", $"msi {Current}", $"msi {Target}"], left);
Assert.Equal(Target, kept.Find(Target)!.Version);
Assert.Null(kept.Find("1.1.0"));
}
private sealed class Stub(Func<HttpRequestMessage, HttpResponseMessage> respond) : HttpMessageHandler
{
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
=> Task.FromResult(respond(request));
}
}
@@ -54,6 +54,27 @@ public sealed class UpgradeOutcomeReporterTests : IDisposable
Assert.False(File.Exists(folders.Outcome));
}
[Fact]
public async Task AnMsiRunsOwnDetailIsWhatIsReported()
{
var outcome = new WatchdogOutcome(Guid.NewGuid(), AgentUpgradeStates.Failed, AgentUpgradeReasons.HealthDeadline, "1.4.0", "1.3.0", [],
DateTimeOffset.UtcNow, "1.4.0 did not report healthy in time; no MSI of 1.3.0 was kept to go back to, so it stays.");
WatchdogFileStore.Write(folders.Outcome, outcome, WatchdogJsonContext.Default.WatchdogOutcome);
Assert.True(await CreateReporter().ReportPendingAsync(CancellationToken.None));
Assert.Equal(outcome.Detail, Assert.Single(events.Published).Detail);
}
[Fact]
public void AnMsiFailureWithoutDetailSaysThePreviousVersionStays()
{
var reported = UpgradeOutcomeReporter.ToEvent(new WatchdogOutcome(Guid.NewGuid(), AgentUpgradeStates.Failed, AgentUpgradeReasons.InstallerBusy,
"1.4.0", "1.3.0", [], DateTimeOffset.UtcNow), "client-1");
Assert.Contains("1.3.0 stays", reported.Detail, StringComparison.Ordinal);
}
[Fact]
public async Task ReportsASuccess()
{
@@ -0,0 +1,584 @@
using System.Security.Cryptography;
using System.Text;
using Agent.Watchdog.Msi;
using Shared.Contracts.Models;
using Shared.Contracts.Serialization;
using Xunit;
namespace Agent.Watchdog.Tests;
/// <summary>
/// Windows Installer as the runner sees it. A successful install "installs" the
/// package's version (the agent binary says it) and, as the MSI's
/// ServiceControl does, stops and starts the agent; each answer comes from
/// <see cref="Answer"/>, by package and attempt.
/// </summary>
public sealed class FakeWindowsInstaller(WatchdogFixture fixture) : IWindowsInstaller
{
public List<(string Package, string CommandLine, string Log)> Installs { get; } = [];
public int SilentCalls { get; private set; }
public string? CurrentLog { get; private set; }
/// <summary>Windows Installer's answer for a package (by its version) and attempt; 0 by default.</summary>
public Func<string, int, int> Answer { get; set; } = (_, _) => MsiExitCodes.Success;
/// <summary>Set to block every install until released (a hung Windows Installer).</summary>
public ManualResetEventSlim? Block { get; set; }
public void SetSilentUI() => SilentCalls++;
public void EnableLog(string logPath)
{
CurrentLog = logPath;
Directory.CreateDirectory(Path.GetDirectoryName(logPath)!);
File.AppendAllText(logPath, "=== Verbose logging started ===\r\nMSI (s) (00:00) [12:00:00:000]: Product: ExecuteR Agent -- done.\r\n", Encoding.Unicode);
}
public int InstallProduct(string package, string commandLine)
{
Block?.Wait(TimeSpan.FromSeconds(30));
Installs.Add((package, commandLine, CurrentLog!));
var version = File.ReadAllText(package)["msi ".Length..];
var code = Answer(version, Installs.Count(i => i.Package == package));
if (MsiExitCodes.Succeeded(code))
{
fixture.Agent.StopAsync(CancellationToken.None).GetAwaiter().GetResult();
fixture.WriteInstalled(WatchdogFixture.AgentExe, $"agent {version}");
fixture.Agent.StartAsync(CancellationToken.None).GetAwaiter().GetResult();
}
return code;
}
}
public sealed class FakeLauncher : IMsiRunnerLauncher
{
public List<Guid> Launches { get; } = [];
public string? Problem { get; set; }
public string? Launch(Guid upgradeId)
{
Launches.Add(upgradeId);
return Problem;
}
}
/// <summary>The watchdog's own service, for the runner to leave running.</summary>
public sealed class FakeService : IAgentService
{
public bool Running { get; set; } = true;
public int Starts { get; private set; }
public Task<AgentServiceStatus> QueryAsync(CancellationToken cancellationToken)
=> Task.FromResult(new AgentServiceStatus(Running ? AgentRunState.Running : AgentRunState.Stopped, true, false));
public Task StartAsync(CancellationToken cancellationToken)
{
Starts++;
Running = true;
return Task.CompletedTask;
}
public Task StopAsync(CancellationToken cancellationToken)
{
Running = false;
return Task.CompletedTask;
}
}
public sealed class MsiUpgradeTests : IDisposable
{
private readonly WatchdogFixture fixture = new();
private readonly FakeLauncher launcher = new();
private readonly FakeService watchdogService = new();
private readonly FakeWindowsInstaller installer;
public MsiUpgradeTests()
{
installer = new FakeWindowsInstaller(fixture);
}
public void Dispose() => fixture.Dispose();
private WatchdogOptions Options => fixture.Options;
private MsiUpgrade CreateMsiUpgrade() => new(Options, launcher, fixture.Agent, fixture.Clock, fixture.Log);
private MsiRunner CreateRunner(MsiRunnerSettings? settings = null)
=> new(Options, fixture.Agent, watchdogService, installer, fixture.Clock, fixture.Log, settings);
/// <summary>An MSI "of a version" in upgrade/msi/ (its content names the version), as the agent leaves it.</summary>
private WatchdogFile Msi(string version, string? folder = null)
{
var directory = folder is null ? Options.MsiDirectory : Path.Combine(Options.MsiDirectory, folder);
Directory.CreateDirectory(directory);
var path = Path.Combine(directory, $"ExecuteR-Agent-{version}.msi");
File.WriteAllText(path, $"msi {version}");
return new WatchdogFile(path, new FileInfo(path).Length, Convert.ToHexStringLower(SHA256.HashData(File.ReadAllBytes(path))));
}
private WatchdogHandover HandOver(bool rollback = true, bool revert = false, WatchdogFile? msi = null)
{
var handover = new WatchdogHandover(Guid.NewGuid(), WatchdogFixture.ToVersion, WatchdogFixture.FromVersion, Options.MsiDirectory, [], 600,
revert, fixture.Clock.UtcNow, WatchdogHandoverKinds.Msi,
msi ?? Msi(WatchdogFixture.ToVersion),
rollback ? Msi(WatchdogFixture.FromVersion, WatchdogMsiFiles.RollbackFolder) : null,
rollback ? WatchdogFixture.FromVersion : null);
WatchdogFileStore.Write(Options.HandoverPath, handover, WatchdogJsonContext.Default.WatchdogHandover);
return handover;
}
private WatchdogMsiJournal Journal(string phase = WatchdogMsiPhases.Installing, bool rollback = true, bool revert = false, int resumes = 0)
{
var journal = new WatchdogMsiJournal(Guid.NewGuid(), WatchdogFixture.ToVersion, WatchdogFixture.FromVersion, phase,
Msi(WatchdogFixture.ToVersion),
rollback ? Msi(WatchdogFixture.FromVersion, WatchdogMsiFiles.RollbackFolder) : null,
rollback ? WatchdogFixture.FromVersion : null,
HealthDeadlineSeconds: 600, RunnerProcessId: 0, Attempts: 0, LastExitCode: null, fixture.Clock.UtcNow,
DeadlineUtc: phase == WatchdogMsiPhases.Verifying ? fixture.Clock.UtcNow.AddMinutes(10) : null, revert, resumes);
WatchdogFileStore.Write(Options.MsiJournalPath, journal, WatchdogJsonContext.Default.WatchdogMsiJournal);
return journal;
}
private WatchdogMsiJournal? ReadJournal() => MsiUpgrade.ReadJournal(Options);
private void AssertRunEnded()
{
Assert.False(File.Exists(Options.MsiJournalPath));
Assert.Empty(Directory.EnumerateFiles(Options.MsiDirectory, "*.msi", SearchOption.AllDirectories));
Assert.True(fixture.Agent.Running);
Assert.True(watchdogService.Running);
}
// ---- the watchdog takes the hand-over ----------------------------------
[Fact]
public async Task AnMsiHandOverIsJournalledAndGivenToTheDetachedRunnerWithoutSwappingAFile()
{
var handover = HandOver();
var before = fixture.InstalledFiles();
var applier = new UpgradeApplier(Options, fixture.Agent, fixture.Clock, fixture.Log, msi: CreateMsiUpgrade());
Assert.Equal(PassResult.Worked, await applier.PassAsync(CancellationToken.None));
Assert.Equal([handover.UpgradeId], launcher.Launches);
Assert.False(File.Exists(Options.HandoverPath));
Assert.False(File.Exists(Options.JournalPath));
var journal = ReadJournal();
Assert.NotNull(journal);
Assert.Equal(WatchdogMsiPhases.Installing, journal.Phase);
Assert.Equal(handover.Msi, journal.Msi);
Assert.Equal(handover.RollbackMsi, journal.RollbackMsi);
Assert.Equal(WatchdogFixture.FromVersion, journal.RollbackVersion);
// Nothing was swapped and the agent was not touched: the MSI does that.
Assert.Equal(before, fixture.InstalledFiles());
Assert.Equal(0, fixture.Agent.Stops);
Assert.True(applier.Pending);
}
[Fact]
public async Task AnMsiThatDoesNotMatchItsHashEndsTheRunBeforeAnythingStarts()
{
var msi = Msi(WatchdogFixture.ToVersion);
HandOver(msi: msi with { Sha256 = new string('0', 64) });
await new UpgradeApplier(Options, fixture.Agent, fixture.Clock, fixture.Log, msi: CreateMsiUpgrade()).PassAsync(CancellationToken.None);
Assert.Empty(launcher.Launches);
Assert.False(File.Exists(Options.MsiJournalPath));
Assert.Equal(AgentUpgradeStates.Failed, fixture.Outcome!.Result);
Assert.Equal(AgentUpgradeReasons.Hash, fixture.Outcome!.Reason);
}
[Fact]
public async Task AnMsiOutsideTheUpgradeFolderIsRefused()
{
var outside = Path.Combine(fixture.Data, "elsewhere.msi");
File.WriteAllText(outside, $"msi {WatchdogFixture.ToVersion}");
HandOver(msi: new WatchdogFile(outside, new FileInfo(outside).Length, Convert.ToHexStringLower(SHA256.HashData(File.ReadAllBytes(outside)))));
await CreateMsiUpgrade().StartAsync(WatchdogFileStore.Read(Options.HandoverPath, WatchdogJsonContext.Default.WatchdogHandover)!, CancellationToken.None);
Assert.Empty(launcher.Launches);
Assert.Equal(AgentUpgradeReasons.Package, fixture.Outcome!.Reason);
}
[Fact]
public async Task ARollbackMsiThatDoesNotCheckOutIsDroppedAndTheUpgradeGoesOn()
{
var handover = HandOver();
File.AppendAllText(handover.RollbackMsi!.Path, "tampered");
await CreateMsiUpgrade().StartAsync(handover, CancellationToken.None);
Assert.Single(launcher.Launches);
Assert.Null(ReadJournal()!.RollbackMsi);
}
[Fact]
public async Task ARunnerThatCannotBeStartedEndsTheRunWithTheAgentRunning()
{
launcher.Problem = "the runner did not start";
fixture.Agent.Running = false;
await CreateMsiUpgrade().StartAsync(HandOver(), CancellationToken.None);
Assert.Equal(AgentUpgradeReasons.Watchdog, fixture.Outcome!.Reason);
Assert.False(File.Exists(Options.MsiJournalPath));
Assert.True(fixture.Agent.Running);
}
[Fact]
public async Task WhileAnMsiRunIsPendingTheAgentIsNeitherSupervisedNorStarted()
{
Journal();
using var runner = MsiRunLock.TryHold(Options.MsiRunnerLockPath);
fixture.Agent.Running = false;
fixture.Agent.StoppedWithFailure = true;
var applier = fixture.CreateApplier();
var withMsi = new UpgradeApplier(Options, fixture.Agent, fixture.Clock, fixture.Log, msi: CreateMsiUpgrade());
Assert.Equal(PassResult.Worked, await withMsi.PassAsync(CancellationToken.None));
Assert.Equal(PassResult.Worked, await applier.PassAsync(CancellationToken.None));
Assert.Equal(PassResult.Worked, await withMsi.RecoverAsync(CancellationToken.None));
Assert.Equal(0, fixture.Agent.Starts);
Assert.Empty(launcher.Launches);
}
// ---- the runner --------------------------------------------------------
[Fact]
public async Task InstallsInProcessSilentlyWithAVerboseLogAndKeepsAHealthyNewVersion()
{
var journal = Journal();
await CreateRunner().RunAsync(journal.UpgradeId, CancellationToken.None);
var install = Assert.Single(installer.Installs);
Assert.Equal(journal.Msi.Path, install.Package);
Assert.Equal("REBOOT=ReallySuppress", install.CommandLine);
Assert.Equal(Path.Combine(fixture.Install, "Logs", $"agent-msi-{WatchdogFixture.ToVersion}-{journal.UpgradeId:N}.log"), install.Log);
Assert.Equal(1, installer.SilentCalls);
var outcome = fixture.Outcome!;
Assert.Equal(AgentUpgradeStates.Succeeded, outcome.Result);
Assert.Null(outcome.Reason);
Assert.Equal(WatchdogFixture.ToVersion, outcome.Version);
Assert.Equal(WatchdogFixture.FromVersion, outcome.FromVersion);
Assert.Equal($"agent {WatchdogFixture.ToVersion}", fixture.ReadInstalled(WatchdogFixture.AgentExe));
AssertRunEnded();
}
[Theory]
[InlineData(MsiExitCodes.SuccessRebootRequired)]
[InlineData(MsiExitCodes.SuccessRebootInitiated)]
public async Task RestartCodesCountAsInstalled(int code)
{
installer.Answer = (_, _) => code;
var journal = Journal();
await CreateRunner().RunAsync(journal.UpgradeId, CancellationToken.None);
Assert.Equal(AgentUpgradeStates.Succeeded, fixture.Outcome!.Result);
}
[Fact]
public async Task ABusyWindowsInstallerIsTriedAgainEvery30Seconds()
{
installer.Answer = (_, attempt) => attempt < 3 ? MsiExitCodes.InstallerBusy : MsiExitCodes.Success;
var journal = Journal();
var started = fixture.Clock.UtcNow;
var attempts = new List<int>();
fixture.Clock.Advanced += () => attempts.Add(ReadJournal()?.Attempts ?? -1);
await CreateRunner().RunAsync(journal.UpgradeId, CancellationToken.None);
Assert.Equal(3, installer.Installs.Count);
Assert.Equal([1, 2], attempts.Take(2));
Assert.True(fixture.Clock.UtcNow - started >= TimeSpan.FromSeconds(60));
Assert.Equal(AgentUpgradeStates.Succeeded, fixture.Outcome!.Result);
}
[Fact]
public async Task AWindowsInstallerBusyThroughEveryRetryFailsAsInstallerBusy()
{
installer.Answer = (_, _) => MsiExitCodes.InstallerBusy;
var journal = Journal();
var started = fixture.Clock.UtcNow;
await CreateRunner().RunAsync(journal.UpgradeId, CancellationToken.None);
var outcome = fixture.Outcome!;
Assert.Equal(AgentUpgradeStates.Failed, outcome.Result);
Assert.Equal(AgentUpgradeReasons.InstallerBusy, outcome.Reason);
Assert.InRange(installer.Installs.Count, 20, 21);
Assert.InRange(fixture.Clock.UtcNow - started, TimeSpan.FromMinutes(9.5), TimeSpan.FromMinutes(10));
Assert.Equal($"agent {WatchdogFixture.FromVersion}", fixture.ReadInstalled(WatchdogFixture.AgentExe));
AssertRunEnded();
}
[Theory]
[InlineData(MsiExitCodes.InstallFailure)]
[InlineData(MsiExitCodes.ProductVersion)]
[InlineData(MsiExitCodes.UserExit)]
public async Task AnyOtherAnswerFailsAsMsiWithItsCodeAndLog(int code)
{
installer.Answer = (_, _) => code;
fixture.Agent.Running = false; // the MSI stopped it before it failed
var journal = Journal();
await CreateRunner().RunAsync(journal.UpgradeId, CancellationToken.None);
var outcome = fixture.Outcome!;
Assert.Equal(AgentUpgradeStates.Failed, outcome.Result);
Assert.Equal(AgentUpgradeReasons.Msi, outcome.Reason);
Assert.Contains(code.ToString(System.Globalization.CultureInfo.InvariantCulture), outcome.Detail, StringComparison.Ordinal);
Assert.Contains(Options.MsiLogPath(WatchdogFixture.ToVersion, journal.UpgradeId), outcome.Detail, StringComparison.Ordinal);
Assert.Contains(outcome.LastLog, l => l.Contains("Verbose logging started", StringComparison.Ordinal));
Assert.Single(installer.Installs);
AssertRunEnded();
}
[Fact]
public async Task AnInstallWithoutAnAnswerWithinTheTimeoutFailsAsMsi()
{
using var block = new ManualResetEventSlim(false);
installer.Block = block;
var journal = Journal();
try
{
await CreateRunner(new MsiRunnerSettings { InstallTimeout = TimeSpan.FromMilliseconds(200) }).RunAsync(journal.UpgradeId, CancellationToken.None);
}
finally
{
block.Set();
}
Assert.Equal(AgentUpgradeStates.Failed, fixture.Outcome!.Result);
Assert.Equal(AgentUpgradeReasons.Msi, fixture.Outcome!.Reason);
Assert.Contains("did not finish", fixture.Outcome!.Detail, StringComparison.Ordinal);
Assert.False(File.Exists(Options.MsiJournalPath));
}
[Fact]
public async Task ANewVersionThatIsNotHealthyIsRolledBackByTheMsiOfThePreviousOne()
{
fixture.Agent.Behave = version => version == WatchdogFixture.ToVersion ? AgentBehavior.Unconnected : AgentBehavior.Healthy;
File.Delete(fixture.InstalledPath("Logs/ExecuteR-Agent-20261023.log"));
fixture.WriteInstalled("Logs/ExecuteR-Agent-20261024.log", "new agent: could not connect\n");
var journal = Journal();
await CreateRunner().RunAsync(journal.UpgradeId, CancellationToken.None);
Assert.Equal(2, installer.Installs.Count);
var rollback = installer.Installs[1];
Assert.Equal(journal.RollbackMsi!.Path, rollback.Package);
Assert.Equal("REBOOT=ReallySuppress ALLOWDOWNGRADE=1", rollback.CommandLine);
Assert.Equal(Options.MsiLogPath(WatchdogFixture.FromVersion, journal.UpgradeId), rollback.Log);
Assert.NotEqual(installer.Installs[0].Log, rollback.Log);
var outcome = fixture.Outcome!;
Assert.Equal(AgentUpgradeStates.RolledBack, outcome.Result);
Assert.Equal(AgentUpgradeReasons.HealthDeadline, outcome.Reason);
Assert.Contains("new agent: could not connect", outcome.LastLog);
Assert.Equal($"agent {WatchdogFixture.FromVersion}", fixture.ReadInstalled(WatchdogFixture.AgentExe));
AssertRunEnded();
}
[Fact]
public async Task WithoutAnMsiToGoBackToAnUnhealthyNewVersionStaysAndTheRunFails()
{
fixture.Agent.Behave = version => version == WatchdogFixture.ToVersion ? AgentBehavior.Silent : AgentBehavior.Healthy;
var journal = Journal(rollback: false);
await CreateRunner().RunAsync(journal.UpgradeId, CancellationToken.None);
Assert.Single(installer.Installs);
var outcome = fixture.Outcome!;
Assert.Equal(AgentUpgradeStates.Failed, outcome.Result);
Assert.Equal(AgentUpgradeReasons.HealthDeadline, outcome.Reason);
Assert.Contains($"no MSI of {WatchdogFixture.FromVersion} was kept to go back to", outcome.Detail, StringComparison.Ordinal);
Assert.Equal($"agent {WatchdogFixture.ToVersion}", fixture.ReadInstalled(WatchdogFixture.AgentExe));
AssertRunEnded();
}
[Fact]
public async Task ARollbackFromTheConsoleInstallsWithAllowDowngradeFromTheStart()
{
var journal = Journal(revert: true);
await CreateRunner().RunAsync(journal.UpgradeId, CancellationToken.None);
Assert.Equal("REBOOT=ReallySuppress ALLOWDOWNGRADE=1", Assert.Single(installer.Installs).CommandLine);
Assert.Equal(AgentUpgradeStates.Succeeded, fixture.Outcome!.Result);
}
[Fact]
public async Task AnMsiChangedSinceTheHandOverIsNotInstalled()
{
var journal = Journal();
File.AppendAllText(journal.Msi.Path, "changed");
await CreateRunner().RunAsync(journal.UpgradeId, CancellationToken.None);
Assert.Empty(installer.Installs);
Assert.Equal(AgentUpgradeReasons.Hash, fixture.Outcome!.Reason);
}
[Fact]
public async Task ASecondRunnerLeavesTheRunToTheOneHoldingTheLock()
{
var journal = Journal();
using var first = MsiRunLock.TryHold(Options.MsiRunnerLockPath);
await CreateRunner().RunAsync(journal.UpgradeId, CancellationToken.None);
Assert.Empty(installer.Installs);
Assert.Equal(WatchdogMsiPhases.Installing, ReadJournal()!.Phase);
}
[Fact]
public async Task TheRunnerRecordsItsProcessAndHoldsTheLockWhileItRuns()
{
var journal = Journal();
var held = false;
int? owner = null;
installer.Answer = (_, _) =>
{
held = MsiRunLock.IsHeld(Options.MsiRunnerLockPath);
owner = ReadJournal()?.RunnerProcessId;
return MsiExitCodes.Success;
};
await CreateRunner().RunAsync(journal.UpgradeId, CancellationToken.None);
Assert.True(held);
Assert.Equal(Environment.ProcessId, owner);
Assert.False(MsiRunLock.IsHeld(Options.MsiRunnerLockPath));
}
[Fact]
public async Task TheMsiIsHeldReadOnlyWhileWindowsInstallerInstallsIt()
{
var journal = Journal();
var writable = true;
installer.Answer = (_, _) =>
{
try
{
using var write = new FileStream(journal.Msi.Path, FileMode.Open, FileAccess.Write, FileShare.ReadWrite);
}
catch (IOException)
{
writable = false;
}
return MsiExitCodes.Success;
};
await CreateRunner().RunAsync(journal.UpgradeId, CancellationToken.None);
// Windows enforces the sharing mode (the only place an MSI runs).
Assert.True(!OperatingSystem.IsWindows() || !writable);
Assert.Equal(AgentUpgradeStates.Succeeded, fixture.Outcome!.Result);
}
// ---- recovery ----------------------------------------------------------
[Fact]
public async Task ALiveRunnerIsLeftAlone()
{
Journal();
using var runner = MsiRunLock.TryHold(Options.MsiRunnerLockPath);
await CreateMsiUpgrade().CheckAsync(CancellationToken.None);
Assert.Empty(launcher.Launches);
Assert.Equal(0, ReadJournal()!.Resumes);
}
[Fact]
public async Task ADeadRunnerIsStartedAgainFromTheJournal()
{
var journal = Journal();
await CreateMsiUpgrade().CheckAsync(CancellationToken.None);
Assert.Equal([journal.UpgradeId], launcher.Launches);
Assert.Equal(WatchdogMsiPhases.Installing, ReadJournal()!.Phase);
Assert.Equal(1, ReadJournal()!.Resumes);
}
[Fact]
public async Task ARunnerThatDiedInstallingAVersionNowHealthyGoesOnFromVerifying()
{
var journal = Journal();
fixture.WriteBeacon(WatchdogFixture.ToVersion, connected: true);
await CreateMsiUpgrade().CheckAsync(CancellationToken.None);
Assert.Single(launcher.Launches);
Assert.Equal(WatchdogMsiPhases.Verifying, ReadJournal()!.Phase);
// The resumed runner sees it healthy and installs nothing again.
await CreateRunner().RunAsync(journal.UpgradeId, CancellationToken.None);
Assert.Empty(installer.Installs);
Assert.Equal(AgentUpgradeStates.Succeeded, fixture.Outcome!.Result);
}
[Fact]
public async Task ARunnerThatKeepsDyingIsGivenUpWithTheAgentRunning()
{
Journal(resumes: MsiUpgrade.MaxResumes);
fixture.Agent.Running = false;
await CreateMsiUpgrade().CheckAsync(CancellationToken.None);
Assert.Empty(launcher.Launches);
Assert.Equal(AgentUpgradeStates.Failed, fixture.Outcome!.Result);
Assert.Equal(AgentUpgradeReasons.Msi, fixture.Outcome!.Reason);
Assert.False(File.Exists(Options.MsiJournalPath));
Assert.True(fixture.Agent.Running);
}
[Fact]
public async Task AResumedRollbackInstallsThePreviousMsiAgain()
{
var journal = Journal(WatchdogMsiPhases.RollingBack);
await CreateRunner().RunAsync(journal.UpgradeId, CancellationToken.None);
Assert.Equal(journal.RollbackMsi!.Path, Assert.Single(installer.Installs).Package);
Assert.Equal(AgentUpgradeStates.RolledBack, fixture.Outcome!.Result);
}
[Fact]
public void ExitCodesThatMeanInstalled()
{
Assert.True(MsiExitCodes.Succeeded(0));
Assert.True(MsiExitCodes.Succeeded(3010));
Assert.True(MsiExitCodes.Succeeded(1641));
Assert.False(MsiExitCodes.Succeeded(1618));
Assert.False(MsiExitCodes.Succeeded(1603));
}
[Fact]
public void TheLogModeIsMsiexecsVerboseEverything()
=> Assert.Equal(0x1FDFu, WindowsInstallerApi.LogEverythingVerbose);
[Theory]
[InlineData(new[] { "msi-run", "id" }, "exe msi-run id")]
[InlineData(new[] { "--data-dir", @"C:\Program Data\ExecuteR" }, @"exe --data-dir ""C:\Program Data\ExecuteR""")]
[InlineData(new[] { @"C:\trailing slash\" }, @"exe ""C:\trailing slash\\""")]
[InlineData(new[] { "" }, @"exe """"")]
public void TheRunnerCommandLineQuotesWhatNeedsQuoting(string[] arguments, string expected)
{
if (OperatingSystem.IsWindows())
{
Assert.Equal(expected, DetachedProcess.BuildCommandLine("exe", arguments));
}
}
}
@@ -0,0 +1,19 @@
using Xunit;
namespace Agent.Watchdog.Tests;
public sealed class WatchdogVersionTests
{
[Theory]
[InlineData("1.4.0+1a2b3c4", "1.4.0")]
[InlineData("1.4.0.7+1a2b3c4.dirty", "1.4.0.7")]
[InlineData("1.4.0", "1.4.0")]
[InlineData("2026.10.10.0140", "2026.10.10.0140")]
[InlineData(null, "0.0.0.0")]
[InlineData("", "0.0.0.0")]
[InlineData("+1a2b3c4", "0.0.0.0")]
public void TheVersionIsTheInformationalVersionBeforeItsMetadata(string? informational, string expected)
{
Assert.Equal(expected, WatchdogVersion.FromInformational(informational));
}
}
@@ -0,0 +1,24 @@
using Backend.Upgrades;
using Xunit;
namespace Backend.Tests;
public sealed class AgentCatalogVersionTests
{
[Theory]
[InlineData("0.1.0", true)] // semantic versions start at 0.x
[InlineData("0.0.0.253", true)] // a build before the first release tag
[InlineData("1.4.0", true)]
[InlineData("1.4.0.7", true)]
[InlineData("2026.10.10.0140", true)] // the calendar era still catalogues
[InlineData("0.0.0.0", false)] // the development version
[InlineData("1.4.0+1a2b3c4", false)] // metadata never names a catalog version
[InlineData("1.4", false)]
[InlineData("1.4.0-rc.1", false)]
[InlineData(" 1.4.0", false)]
[InlineData(null, false)]
public void CatalogVersions(string? version, bool catalogued)
{
Assert.Equal(catalogued, AgentCatalogService.IsCatalogVersion(version));
}
}
+146
View File
@@ -0,0 +1,146 @@
using System.Net.Http.Json;
using Backend.Api;
using Backend.Upgrades;
using Microsoft.Extensions.DependencyInjection;
using Shared.Contracts.Api;
using Shared.Contracts.Models;
using Shared.Contracts.Serialization;
using Xunit;
using static Backend.Tests.AgentUpgradeTestKit;
namespace Backend.Tests;
/// <summary>Devices the MSI installed upgrade (and roll back) through the MSI; portable ones through their zip.</summary>
public class AgentMsiUpgradeTests(BackendTestFixture fixture) : IClassFixture<BackendTestFixture>
{
[Fact]
public void TheInstallKindPicksThePackage()
{
Assert.Equal(AgentPackageTypes.Msi, AgentUpgradeService.ResolvePackageType("win-x64", null, null, AgentInstallKinds.Msi));
Assert.Equal(AgentPackageTypes.WindowsZip, AgentUpgradeService.ResolvePackageType("win-x64", null, null, AgentInstallKinds.Portable));
Assert.Equal(AgentPackageTypes.WindowsZip, AgentUpgradeService.ResolvePackageType("win-x64", null, null));
Assert.Equal(AgentPackageTypes.LinuxX64, AgentUpgradeService.ResolvePackageType("linux-x64", null, null, AgentInstallKinds.Msi));
}
[Fact]
public async Task AnMsiDeviceGetsTheMsiAPortableOneTheZipAndAnMsiDeviceWithoutAnMsiOfTheTargetIsLeftOut()
{
await fixture.ResetAsync();
await fixture.SeedPackageAsync(V2);
await fixture.SeedPackageAsync(V2, AgentPackageTypes.Msi);
await fixture.SeedPackageAsync(V3);
var msi = await fixture.SeedAgentAsync("MSI-1", V1, install: AgentInstallKinds.Msi);
var portable = await fixture.SeedAgentAsync("PORTABLE-1", V1, install: AgentInstallKinds.Portable);
using var admin = await fixture.SignedInAsync("msi-admin");
var batch = await (await admin.PostAsJsonAsync(ApiRoutes.AgentUpgrades,
new StartAgentUpgradeRequest([msi, portable], Version: V2), JsonDefaults.Options)).ReadAsync<AgentUpgradeBatchDto>();
Assert.Equal(2, batch.Upgrades.Count);
Assert.Equal(AgentPackageTypes.Msi, Assert.Single(await fixture.UpgradesOfAsync(msi)).PackageType);
Assert.Equal(AgentPackageTypes.WindowsZip, Assert.Single(await fixture.UpgradesOfAsync(portable)).PackageType);
// V3 has no MSI: the MSI device is left out, never handed the zip.
var other = await fixture.SeedAgentAsync("MSI-2", V1, install: AgentInstallKinds.Msi);
var newest = await (await admin.PostAsJsonAsync(ApiRoutes.AgentUpgrades,
new StartAgentUpgradeRequest([other], Version: V3), JsonDefaults.Options)).ReadAsync<AgentUpgradeBatchDto>();
Assert.Empty(newest.Upgrades);
Assert.Equal(AgentUpgradeReasons.NoPackage, Assert.Single(newest.Skipped).Reason);
// Without a version: the newest MSI (V2), not the newest zip (V3).
var latest = await (await admin.PostAsJsonAsync(ApiRoutes.AgentUpgrades,
new StartAgentUpgradeRequest([other]), JsonDefaults.Options)).ReadAsync<AgentUpgradeBatchDto>();
Assert.Equal(V2, Assert.Single(latest.Upgrades).TargetVersion);
}
[Fact]
public async Task TheMsiCommandCarriesTheMsiOfTheRunningVersionToRollBackTo()
{
await fixture.ResetAsync();
var current = await fixture.SeedPackageAsync(V1, AgentPackageTypes.Msi, content: "msi v1");
var target = await fixture.SeedPackageAsync(V2, AgentPackageTypes.Msi);
await fixture.SeedPackageAsync(V2);
var msi = await fixture.SeedAgentAsync("MSI-RB", V1, install: AgentInstallKinds.Msi);
var portable = await fixture.SeedAgentAsync("ZIP-RB", V1, install: AgentInstallKinds.Portable);
using (var admin = await fixture.SignedInAsync("msi-admin-1"))
{
await admin.PostAsJsonAsync(ApiRoutes.AgentUpgrades, new StartAgentUpgradeRequest([msi, portable], Version: V2), JsonDefaults.Options);
}
var sender = new RecordingUpgradeSender();
await fixture.Scheduler(sender).RunPassAsync(DateTimeOffset.UtcNow, CancellationToken.None);
var command = sender.Sent.Single(s => s.ClientId == msi).Command;
Assert.Equal(AgentPackageTypes.Msi, command.PackageType);
Assert.Equal(target.Sha256, command.Sha256);
Assert.Equal(V1, command.RollbackVersion);
Assert.Equal(ApiRoutes.For.AgentPackageBySha(current.Sha256), command.RollbackDownloadUrl);
Assert.Equal(current.Sha256, command.RollbackSha256);
Assert.Equal(current.SizeBytes, command.RollbackSizeBytes);
var zip = sender.Sent.Single(s => s.ClientId == portable).Command;
Assert.Equal(AgentPackageTypes.WindowsZip, zip.PackageType);
Assert.Null(zip.RollbackVersion);
Assert.Null(zip.RollbackDownloadUrl);
}
[Fact]
public async Task AnMsiCommandForAVersionTheServerHasNoMsiOfCarriesNoRollback()
{
await fixture.ResetAsync();
await fixture.SeedPackageAsync(V2, AgentPackageTypes.Msi);
var msi = await fixture.SeedAgentAsync("MSI-NORB", V1, install: AgentInstallKinds.Msi);
using (var admin = await fixture.SignedInAsync("msi-admin-2"))
{
await admin.PostAsJsonAsync(ApiRoutes.AgentUpgrades, new StartAgentUpgradeRequest([msi], Version: V2), JsonDefaults.Options);
}
var sender = new RecordingUpgradeSender();
await fixture.Scheduler(sender).RunPassAsync(DateTimeOffset.UtcNow, CancellationToken.None);
var command = Assert.Single(sender.Sent).Command;
Assert.Equal(AgentPackageTypes.Msi, command.PackageType);
Assert.Null(command.RollbackVersion);
Assert.Null(command.RollbackSha256);
}
[Fact]
public async Task RollingBackAnMsiDeviceGoesThroughTheMsiOfThePreviousVersion()
{
await fixture.ResetAsync();
var previous = await fixture.SeedPackageAsync(V1, AgentPackageTypes.Msi);
await fixture.SeedPackageAsync(V1);
await fixture.SeedPackageAsync(V2, AgentPackageTypes.Msi);
var device = await fixture.SeedAgentAsync("MSI-BACK", V2, install: AgentInstallKinds.Msi);
await fixture.SeedUpgradeAsync(device, V1, V2, AgentUpgradeStates.Succeeded);
using var admin = await fixture.SignedInAsync("msi-rollback");
var batch = await (await admin.PostAsJsonAsync(ApiRoutes.AgentUpgradesRollback,
new RollBackAgentRequest([device]), JsonDefaults.Options)).ReadAsync<AgentUpgradeBatchDto>();
Assert.Equal(V1, Assert.Single(batch.Upgrades).TargetVersion);
var row = (await fixture.UpgradesOfAsync(device)).Single(u => u.Source == AgentUpgradeSources.Rollback);
Assert.Equal(AgentPackageTypes.Msi, row.PackageType);
Assert.Equal(previous.Id, row.PackageId);
var sender = new RecordingUpgradeSender();
await fixture.Scheduler(sender).RunPassAsync(DateTimeOffset.UtcNow, CancellationToken.None);
var command = Assert.Single(sender.Sent).Command;
Assert.True(command.Revert);
Assert.Equal(AgentPackageTypes.Msi, command.PackageType);
}
[Fact]
public async Task TheDeviceSaysHowItsAgentWasInstalled()
{
await fixture.ResetAsync();
var device = await fixture.SeedAgentAsync("MSI-DTO", V1, install: AgentInstallKinds.Msi);
using var admin = await fixture.SignedInAsync("msi-dto");
var detail = await (await admin.GetAsync($"{ApiRoutes.Devices}/{device}")).ReadAsync<DeviceDetailDto>();
Assert.Equal(AgentInstallKinds.Msi, detail.Device.AgentInstall);
}
}
+2 -1
View File
@@ -69,7 +69,7 @@ internal static class AgentUpgradeTestKit
public static async Task<string> SeedAgentAsync(
this BackendTestFixture fixture, string hostname, string? version, string? runtime = "win-x64",
bool online = true, string? osCaption = null)
bool online = true, string? osCaption = null, string? install = null)
{
var clientId = await fixture.SeedDeviceAsync(hostname, online);
await fixture.WithDbAsync(async db =>
@@ -78,6 +78,7 @@ internal static class AgentUpgradeTestKit
device.AgentVersion = version;
device.AgentRuntime = runtime;
device.OsCaption = osCaption;
device.AgentInstall = install;
await db.SaveChangesAsync();
});
return clientId;
+8 -1
View File
@@ -41,8 +41,15 @@ public class AgentCatalogTests(BackendTestFixture fixture) : IClassFixture<Backe
var catalog = scope.ServiceProvider.GetRequiredService<AgentCatalogService>();
Assert.Null(await catalog.ImportAsync(AgentPackageTypes.WindowsZip, source, "0.0.0.0", CancellationToken.None));
Assert.Null(await catalog.ImportAsync(AgentPackageTypes.WindowsZip, source, null, CancellationToken.None));
Assert.Null(await catalog.ImportAsync(AgentPackageTypes.Msi, source, V2, CancellationToken.None));
Assert.Equal(1, await fixture.WithDbAsync(db => db.AgentPackages.CountAsync()));
// The MSI is catalogued too: MSI installs upgrade through it.
var msi = await catalog.ImportAsync(AgentPackageTypes.Msi, source, V1, CancellationToken.None);
Assert.NotNull(msi);
Assert.Equal("Agent.msi", msi.FileName);
Assert.Equal($"{V1}/Agent.msi", msi.StoragePath);
Assert.True(File.Exists(CatalogFile(V1, "Agent.msi")));
Assert.Equal(2, await fixture.WithDbAsync(db => db.AgentPackages.CountAsync()));
}
[Fact]
+166
View File
@@ -0,0 +1,166 @@
using ExecuteR.Installer.CustomActions;
using Shared.Contracts.Models;
using Xunit;
namespace Shared.Tests;
public class AgentVersionsTests
{
/// <summary>Oldest first: every comparison anywhere must keep this order.</summary>
private static readonly string[] Ordered =
[
"2026.08.06.2151",
"2026.08.11.1042",
"2026.10.10.0140",
"0.0.0.1",
"0.1.0",
"0.1.0.3",
"1.0.0",
"1.0.0.1",
"1.0.1",
"1.9.0",
"1.10.0",
"2.0.0",
"255.255.65535.65535",
];
/// <summary>Versions that do not parse, and 0.0.0.0, before every version above.</summary>
private static readonly string?[] Unparsed = [null, "", " ", "0.0.0.0", "0.0.0", "1.4", "1", "1.4.0.7.1", "1.4.0-rc.1", "v1.4.0", "abc", "1.-1.0"];
public static TheoryData<string, string> AdjacentPairs()
{
var data = new TheoryData<string, string>();
for (var i = 1; i < Ordered.Length; i++)
{
data.Add(Ordered[i - 1], Ordered[i]);
}
return data;
}
[Theory]
[MemberData(nameof(AdjacentPairs))]
public void VersionsOrderSemanticAfterTheCalendarEra(string older, string newer)
{
Assert.True(AgentVersions.Compare(older, newer) < 0);
Assert.True(AgentVersions.Compare(newer, older) > 0);
Assert.True(AgentVersions.IsNewer(newer, older));
Assert.False(AgentVersions.IsNewer(older, newer));
}
[Fact]
public void SortingIsTheOneOrder()
{
var shuffled = Ordered.Reverse().Concat(Unparsed.Where(u => u is not null).Select(u => u!)).ToList();
var sorted = shuffled.OrderBy(v => v, AgentVersions.Comparer).ToList();
Assert.Equal(Ordered, sorted.Skip(sorted.Count - Ordered.Length));
}
[Theory]
[InlineData("1.4.0", "1.4.0.0")]
[InlineData("1.4.0+1a2b3c4", "1.4.0")]
[InlineData("1.4.0.7+1a2b3c4.dirty", "1.4.0.7")]
[InlineData(" 1.4.0 ", "1.4.0")]
public void AMissingFourthFieldAndMetadataDoNotCount(string left, string right)
{
Assert.Equal(0, AgentVersions.Compare(left, right));
Assert.True(AgentVersions.AreSame(left, right));
}
[Theory]
[InlineData("0.1.0", true)]
[InlineData("0.0.0.1", true)]
[InlineData("1.4.0", true)]
[InlineData("1.4.0.7", true)]
[InlineData("1.4.0+1a2b3c4", true)]
[InlineData("2026.10.10.0140", true)]
[InlineData("0.0.0.0", false)]
[InlineData("0.0.0", false)]
[InlineData("1.4", false)]
[InlineData("1.4.0-rc.1", false)]
[InlineData("garbage", false)]
[InlineData(null, false)]
public void ValidVersions(string? version, bool valid) => Assert.Equal(valid, AgentVersions.IsValid(version));
[Theory]
[InlineData("2026.10.10.0140", true)]
[InlineData("2000.1.1.0", true)]
[InlineData("1999.1.1", false)]
[InlineData("1.4.0", false)]
public void CalendarEra(string version, bool calendar) => Assert.Equal(calendar, AgentVersions.IsCalendarEra(version));
[Fact]
public void UnparsedVersionsSortBeforeEveryVersionThenOrdinally()
{
foreach (var unparsed in Unparsed)
{
Assert.True(AgentVersions.Compare(unparsed, Ordered[0]) < 0, $"{unparsed} before {Ordered[0]}");
Assert.True(AgentVersions.Compare(Ordered[0], unparsed) > 0);
}
Assert.True(AgentVersions.Compare("abc", "abd") < 0);
Assert.Equal(0, AgentVersions.Compare("0.0.0.0", "0.0.0.0"));
Assert.False(AgentVersions.AreSame("0.0.0.0", "0.0.0.0"));
}
/// <summary>The installer's copy (it runs on .NET Framework) must give exactly the same answers.</summary>
[Fact]
public void TheInstallerComparesExactlyLikeAgentVersions()
{
var all = Ordered.Concat(Unparsed).Concat(["1.4.0.0", "1.4.0+x", "1.10.0.3", "2026.10.10.140", "-1.0.0", "1.2.3 "]).ToList();
foreach (var left in all)
{
foreach (var right in all)
{
Assert.True(
Math.Sign(AgentVersions.Compare(left, right)) == Math.Sign(InstalledRelease.Compare(left, right)),
$"'{left}' vs '{right}': AgentVersions {AgentVersions.Compare(left, right)}, installer {InstalledRelease.Compare(left, right)}");
}
}
}
[Theory]
// The calendar era's MSIs: yy.MM.((dd-1)*1440 + HH*60 + mm).
[InlineData("26.10.13060", "2026.10.10.0140")]
[InlineData("26.8.8511", "2026.08.06.2151")]
[InlineData("26.1.0", "2026.01.01.0000")]
[InlineData("26.12.44639", "2026.12.31.2359")]
// Semantic MSIs carry the version itself, in four fields.
[InlineData("1.0.0.0", "1.0.0.0")]
[InlineData("1.4.0.7", "1.4.0.7")]
// Three fields that cannot be a calendar version stay as they are.
[InlineData("26.13.5", "26.13.5")]
[InlineData("1.4.0", "1.4.0")]
[InlineData("26.10.44640", "26.10.44640")]
public void InstalledMsiVersionsReadBack(string msiVersion, string version)
{
Assert.Equal(version, InstalledRelease.FromMsiVersion(msiVersion));
}
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData("garbage")]
[InlineData("26")]
public void UnreadableMsiVersions(string? msiVersion) => Assert.Null(InstalledRelease.FromMsiVersion(msiVersion));
[Theory]
// 1.0.0 replaces any calendar-era install.
[InlineData("1.0.0", new[] { "26.10.13060" }, null)]
[InlineData("0.1.0.4", new[] { "26.10.13060", "26.8.8511" }, null)]
// The same version again reinstalls; an older build is refused, by all four fields.
[InlineData("1.4.0", new[] { "1.4.0.0" }, null)]
[InlineData("1.4.0.7", new[] { "1.4.0.3" }, null)]
[InlineData("1.4.0.3", new[] { "1.4.0.7" }, "1.4.0.7")]
[InlineData("1.4.0", new[] { "1.5.0.0" }, "1.5.0.0")]
[InlineData("1.10.0", new[] { "1.9.0.0" }, null)]
// A calendar-era package cannot go over a semantic install.
[InlineData("2026.10.10.0140", new[] { "1.0.0.0" }, "1.0.0.0")]
// Nothing installed, or nothing readable.
[InlineData("1.0.0", new string[0], null)]
[InlineData("1.0.0", new[] { "garbage" }, null)]
public void ANewerInstalledReleaseIsFound(string incoming, string[] installed, string? newer)
{
Assert.Equal(newer, InstalledRelease.Newer(incoming, installed));
}
}
+26 -21
View File
@@ -1,3 +1,4 @@
using System.Reflection;
using Shared.Infrastructure.Versioning;
using Xunit;
@@ -6,31 +7,35 @@ namespace Shared.Tests;
public class BuildVersionTests
{
[Theory]
[InlineData("2026.08.06.2151", "26.8.8511")]
[InlineData("2026.01.01.0000", "26.1.0")]
[InlineData("2026.12.31.2359", "26.12.44639")]
public void MsiProductVersionTransformIsDeterministic(string unified, string expected)
[InlineData("1.4.0+1a2b3c4", "1.4.0")]
[InlineData("1.4.0.7+1a2b3c4", "1.4.0.7")]
[InlineData("1.4.0.7+1a2b3c4.dirty", "1.4.0.7")]
[InlineData("1.4.0", "1.4.0")]
[InlineData("0.0.0.253+0ce444f", "0.0.0.253")]
// A calendar-era binary still reads (its version had no '+').
[InlineData("2026.10.10.0140", "2026.10.10.0140")]
public void TheReportedVersionDropsOnlyTheBuildMetadata(string informational, string expected)
{
Assert.Equal(expected, BuildVersion.ToMsiProductVersion(unified));
}
[Fact]
public void MsiProductVersionIsMonotonicWithinAMonth()
{
var earlier = BuildVersion.ToMsiProductVersion("2026.08.06.2151");
var later = BuildVersion.ToMsiProductVersion("2026.08.07.0001");
var earlierBuild = int.Parse(earlier.Split('.')[2]);
var laterBuild = int.Parse(later.Split('.')[2]);
Assert.True(laterBuild > earlierBuild);
Assert.Equal(expected, BuildVersion.FromInformational(informational));
}
[Theory]
[InlineData("not-a-version")]
[InlineData("2026.08.06")]
[InlineData("1.2.3.4.5")]
public void InvalidUnifiedVersionsAreRejected(string value)
[InlineData(null)]
[InlineData("")]
[InlineData(" ")]
[InlineData("+1a2b3c4")]
public void NoVersionIsTheDevelopmentVersion(string? informational)
{
Assert.Throws<FormatException>(() => BuildVersion.ToMsiProductVersion(value));
Assert.Equal(BuildVersion.Development, BuildVersion.FromInformational(informational));
Assert.Equal("0.0.0.0", BuildVersion.Development);
}
[Fact]
public void AnAssemblyWithoutAnInformationalVersionIsTheDevelopmentVersion()
{
Assert.Equal("0.0.0.0", BuildVersion.Resolve(null));
var stamped = BuildVersion.Resolve(typeof(BuildVersion).Assembly);
var informational = typeof(BuildVersion).Assembly.GetCustomAttribute<AssemblyInformationalVersionAttribute>()!.InformationalVersion;
Assert.Equal(informational.Split('+')[0], stamped);
}
}
+138
View File
@@ -0,0 +1,138 @@
using ExecuteR.Installer.CustomActions;
using Shared.Contracts.Models;
using Xunit;
namespace Shared.Tests;
/// <summary>The installer's KeepInstallerPackage: the MSIs an MSI upgrade rolls back to.</summary>
public sealed class KeptPackagesTests : IDisposable
{
private readonly string temp = Path.Combine(Path.GetTempPath(), $"kept-packages-{Guid.NewGuid():N}");
private readonly List<string> log = [];
public KeptPackagesTests() => Directory.CreateDirectory(temp);
public void Dispose() => Directory.Delete(temp, recursive: true);
private string Root => Path.Combine(temp, "packages");
private string Msi(string name, string content)
{
var folder = Path.Combine(temp, "downloads", Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(folder);
var path = Path.Combine(folder, name);
File.WriteAllText(path, content);
return path;
}
[Fact]
public void TheLimitIsTheWatchdogs()
{
Assert.Equal(WatchdogFiles.KeptPackagesMax, KeptPackages.Max);
}
[Fact]
public void AnMsiIsKeptUnderItsHashWithItsName()
{
var source = Msi("ExecuteR-Agent-1.4.0.msi", "1.4.0");
var kept = KeptPackages.Keep(source, "1.4.0", Root, KeptPackages.Max, log.Add);
var sha256 = KeptPackages.Sha256(source);
Assert.Equal(Path.Combine(Root, sha256, "ExecuteR-Agent-1.4.0.msi"), kept);
Assert.Equal("1.4.0", File.ReadAllText(kept));
Assert.Matches("^[0-9a-f]{64}$", sha256);
Assert.Empty(Directory.GetFiles(Path.GetDirectoryName(kept)!, "*.partial"));
}
[Fact]
public void ADescriptionWithTheVersionAndHashLiesBesideTheKeptMsi()
{
var source = Msi("ExecuteR-Agent-1.4.0.7.msi", "1.4.0.7");
var kept = KeptPackages.Keep(source, "1.4.0.7", Root, KeptPackages.Max, log.Add);
var description = Path.Combine(Path.GetDirectoryName(kept)!, "package.json");
using var json = System.Text.Json.JsonDocument.Parse(File.ReadAllText(description));
Assert.Equal("1.4.0.7", json.RootElement.GetProperty("version").GetString());
Assert.Equal(KeptPackages.Sha256(source), json.RootElement.GetProperty("sha256").GetString());
Assert.Equal(2, json.RootElement.EnumerateObject().Count());
Assert.Empty(Directory.GetFiles(Path.GetDirectoryName(kept)!, "*.partial"));
}
[Fact]
public void NoVersionWritesNoDescription()
{
var kept = KeptPackages.Keep(Msi("Agent.msi", "x"), "", Root, KeptPackages.Max, log.Add);
Assert.False(File.Exists(Path.Combine(Path.GetDirectoryName(kept)!, "package.json")));
}
[Fact]
public void TheDescriptionEscapesWhatJsonNeedsEscaped()
{
using var json = System.Text.Json.JsonDocument.Parse(KeptPackages.Description("a\"b\\c\n", "ab"));
Assert.Equal("a\"b\\c\n", json.RootElement.GetProperty("version").GetString());
}
[Fact]
public void KeepingTheSameMsiAgainCopiesNothing()
{
var source = Msi("Agent.msi", "1.4.0");
var kept = KeptPackages.Keep(source, "1.4.0", Root, KeptPackages.Max, log.Add);
var written = File.GetLastWriteTimeUtc(kept);
KeptPackages.Keep(source, "1.4.0", Root, KeptPackages.Max, log.Add);
Assert.Equal(written, File.GetLastWriteTimeUtc(kept));
Assert.Contains(log, line => line.Contains("already kept"));
Assert.Single(Directory.GetDirectories(Root));
}
[Fact]
public void OnlyTheNewestThreeAreKept()
{
var folders = new List<string>();
for (var i = 1; i <= 4; i++)
{
var kept = KeptPackages.Keep(Msi($"ExecuteR-Agent-1.{i}.0.msi", $"1.{i}.0"), "1.4.0", Root, KeptPackages.Max, log.Add);
var folder = Path.GetDirectoryName(kept)!;
folders.Add(folder);
// Installed an hour apart, oldest first.
Directory.SetLastWriteTimeUtc(folder, DateTime.UtcNow.AddHours(i - 10));
}
Assert.False(Directory.Exists(folders[0]));
Assert.All(folders.Skip(1), folder => Assert.True(Directory.Exists(folder)));
}
[Fact]
public void TheMsiJustInstalledStaysEvenWhenOthersLookNewer()
{
var others = new List<string>();
for (var i = 1; i <= 3; i++)
{
var folder = Path.GetDirectoryName(KeptPackages.Keep(Msi($"Agent-{i}.msi", $"{i}"), "1.4.0", Root, KeptPackages.Max, log.Add))!;
Directory.SetLastWriteTimeUtc(folder, DateTime.UtcNow.AddDays(1)); // a clock that was ahead
others.Add(folder);
}
var current = Path.GetDirectoryName(KeptPackages.Keep(Msi("Agent-0.msi", "0"), "1.4.0", Root, KeptPackages.Max, log.Add))!;
Assert.True(Directory.Exists(current));
Assert.Equal(3, Directory.GetDirectories(Root).Length);
}
[Fact]
public void FoldersThatAreNotPackagesAreLeftAlone()
{
var other = Directory.CreateDirectory(Path.Combine(Root, "notes")).FullName;
Directory.SetLastWriteTimeUtc(other, DateTime.UtcNow.AddYears(-1));
for (var i = 1; i <= 4; i++)
{
KeptPackages.Keep(Msi($"Agent-{i}.msi", $"{i}"), "1.4.0", Root, KeptPackages.Max, log.Add);
}
Assert.True(Directory.Exists(other));
Assert.Equal(4, Directory.GetDirectories(Root).Length); // three packages and the other folder
}
}
+20
View File
@@ -68,6 +68,26 @@ public class RuleEvaluatorTests
Assert.Equal(expected, RuleEvaluator.Matches(condition, Bag()));
}
[Theory]
[InlineData("1.10.0", RuleOperator.GreaterThan, "1.9.0", true)] // not 1.10 < 1.9 as text
[InlineData("1.9.0", RuleOperator.GreaterThan, "1.10.0", false)]
[InlineData("1.4.0.7", RuleOperator.GreaterThan, "1.4.0", true)] // a build after the release
[InlineData("1.4.0", RuleOperator.GreaterOrEqual, "1.4.0.0", true)] // the missing fourth field is 0
[InlineData("1.4.0", RuleOperator.LessOrEqual, "1.4.0.0", true)]
[InlineData("1.4.0+1a2b3c4", RuleOperator.GreaterOrEqual, "1.4.0", true)] // +metadata ignored
[InlineData("2026.10.10.0140", RuleOperator.LessThan, "1.0.0", true)] // the calendar era before semantic versions
[InlineData("2026.10.10.0140", RuleOperator.LessThan, "0.1.0", true)]
[InlineData("2026.10.10.0140", RuleOperator.GreaterThan, "2026.08.06.2151", true)]
[InlineData("0.0.0.0", RuleOperator.LessThan, "0.1.0", true)] // a development build is lowest
[InlineData("1.4.2", RuleOperator.GreaterOrEqual, "1.4", true)] // 1.4 is the 1.4.0 line
[InlineData("1.3.9", RuleOperator.GreaterOrEqual, "1.4", false)]
[InlineData("2.0.0", RuleOperator.GreaterThan, "1", true)]
public void AgentVersionOrdersByVersion(string agentVersion, RuleOperator op, string value, bool expected)
{
var bag = DeviceFieldBag.Build("c1", "PC-001", DeviceStatus.Online, agentVersion, null, null);
Assert.Equal(expected, RuleEvaluator.Matches(new RuleCondition("agentVersion", op, value), bag));
}
[Fact]
public void NegationInvertsAnyOperator()
{
+7
View File
@@ -20,4 +20,11 @@
<ProjectReference Include="..\..\src\Shared.Infrastructure\Shared.Infrastructure.csproj" />
</ItemGroup>
<ItemGroup>
<!-- The installer custom actions' logic (the project itself is .NET
Framework): its version order must equal AgentVersions'. -->
<Compile Include="..\..\src\Installer.CustomActions\InstalledRelease.cs" Link="Installer\InstalledRelease.cs" />
<Compile Include="..\..\src\Installer.CustomActions\KeptPackages.cs" Link="Installer\KeptPackages.cs" />
</ItemGroup>
</Project>