fix: retry the qemu-server flock instead of predicting it

WaitForStatusTransition refused to return while snapshot.Locked, and its
comment quoted the exact error it was meant to prevent. Locked reads the
guest config's lock: property; the failure is the flock on
/var/lock/qemu-server/lock-<vmid>.conf, which PVE exposes nowhere.

The flock cannot be observed, so it is retried. GuestLockRetry reissues an
operation for a bounded 45s while PVE reports failing to enter lock_config
for a guest, which it raises before doing any work.

Two seams, because the failure has two surfaces. PveHttpClient.SendAsync
retries the request for operations PVE serialises in the API handler; it
takes a request factory because an HttpRequestMessage cannot be resent.
PveCmdletBase.InvokeGuestTask reissues the call and re-waits its task for
operations serialised in the forked worker, where the POST returns 200 and
only the task fails. WaitForStatusTransition routes through the latter,
hence Func<PveTask>.

The predicate is path-specific and anchored at the start of what PVE said:
lock_file uses identical wording for storage, LVM and HA locks, and a
qmclone that fails after allocating disks must not be reissued into
"VM already exists". That requires the raw text, so it reads
PveTaskFailedException.ExitStatus and PveApiException.ApiMessage.

The Locked check stays — it is correct for the config lock — with a comment
that says so.

Closes #113
This commit is contained in:
goodolclint-claude[bot]
2026-09-01 21:19:05 -05:00
parent 23ad9840c6
commit db416a3f03
20 changed files with 653 additions and 109 deletions
+2 -1
View File
@@ -10,7 +10,8 @@ and this project adheres to [Conventional Commits](https://www.conventionalcommi
### Fixed ### Fixed
- `Restart-PveVm` now uses PVE's native reboot endpoint (`POST {vmid}/status/reboot`) instead of composing a shutdown followed by a start. The two-call form raced Proxmox's own post-stop cleanup: the start won the guest's config lock, `qm cleanup` then held that lock for 30 seconds waiting on the newly started process, and the caller's next operation failed with `can't lock file '/var/lock/qemu-server/lock-<vmid>.conf' - got timeout`. Reproduced in integration runs 183, 185 and 186 as a cascade of 4 failures. `Restart-PveContainer` is unchanged — LXC has no reboot endpoint. See `DECISIONS.md` D016. - `Restart-PveVm` now uses PVE's native reboot endpoint (`POST {vmid}/status/reboot`) instead of composing a shutdown followed by a start. The two-call form raced Proxmox's own post-stop cleanup: the start won the guest's config lock, `qm cleanup` then held that lock for 30 seconds waiting on the newly started process, and the caller's next operation failed with `can't lock file '/var/lock/qemu-server/lock-<vmid>.conf' - got timeout`. Reproduced in integration runs 183, 185 and 186 as a cascade of 4 failures. `Restart-PveContainer` is unchanged — LXC has no reboot endpoint. See `DECISIONS.md` D016.
- Lifecycle cmdlets with `-Wait` (`Start`/`Stop`/`Restart`/`Reset`/`Resume` for VMs and containers) also wait for the guest's config lock (the `lock:` property, e.g. `backup` or `migrate`) to clear before returning, and the post-timeout fallback tests the most recent poll rather than whether a match was ever seen. See `DECISIONS.md` D015 — note that entry misdiagnosed the failure above and is superseded by D016. - Guest operations that Proxmox rejects with `can't lock file '/var/lock/qemu-server/lock-<vmid>.conf' - got timeout` are now reissued for up to 45 seconds instead of surfacing as an error. That flock is taken by `qm cleanup` for up to 30 seconds after a guest stops and is not exposed through the API in any form, so it can only be retried past, never waited on. Covers both the synchronous form (`Set-PveVmConfig`, `Resize-PveVmDisk`, and every other call through the HTTP client) and the asynchronous form, where the request succeeds and the PVE task then fails (`Reset-PveVm`, `Copy-PveVm`). Reproduced on a client ~40% slower than CI, which failed three VM tests on a commit CI passed. (#113) See `DECISIONS.md` D020.
- Lifecycle cmdlets with `-Wait` (`Start`/`Stop`/`Restart`/`Reset`/`Resume` for VMs and containers) also wait for the guest's config lock (the `lock:` property, e.g. `backup` or `migrate`) to clear before returning, and the post-timeout fallback tests the most recent poll rather than whether a match was ever seen. See `DECISIONS.md` D015 — that guard covers the config lock only; the separate flock race is D020.
- `New-PveCluster -Wait` now blocks until the cluster reports quorum, not merely until the creation task finishes. PVE's create task returns before corosync converges (~6s earlier in testing), so the natural `New-PveCluster -Wait``Add-PveClusterMember` sequence failed with `cluster not ready - no quorum?`. Adds `-Timeout` (seconds, default 60, range 1-3600) following the `-Wait` timeout convention used by `Stop-PveContainer` and `Reset-PveVm`. See `DECISIONS.md` D014. - `New-PveCluster -Wait` now blocks until the cluster reports quorum, not merely until the creation task finishes. PVE's create task returns before corosync converges (~6s earlier in testing), so the natural `New-PveCluster -Wait``Add-PveClusterMember` sequence failed with `cluster not ready - no quorum?`. Adds `-Timeout` (seconds, default 60, range 1-3600) following the `-Wait` timeout convention used by `Stop-PveContainer` and `Reset-PveVm`. See `DECISIONS.md` D014.
## [0.2.0] - 2026-05-22 ## [0.2.0] - 2026-05-22
+111 -12
View File
@@ -466,19 +466,20 @@ Add-PveClusterMember ...
## D015 — Lifecycle -Wait blocks until the guest config lock clears ## D015 — Lifecycle -Wait blocks until the guest config lock clears
**Status**: Superseded by D016 (2026-09-01)the mechanism below is wrong **Status**: Active, rescoped 2026-09-01 — it guards the config lock only, never the flock
**Finding refs**: (none — found via integration runs 183/184, 2026-09-01) **Finding refs**: (none — found via integration runs 183/184, 2026-09-01; rescoped for #113)
**Resolved in scan**: n/a **Resolved in scan**: n/a
> **This entry misdiagnosed the failure it was written for.** Two different things in PVE are > **This entry was written for a failure it does not prevent.** Two different things in PVE are
> called "lock": the **config lock** (the `lock:` property — `migrate`, `backup`, `clone`, > called "lock": the **config lock** (the `lock:` property — `migrate`, `backup`, `clone`,
> `snapshot` — a persisted config field, exposed as `lock` in `status/current`) and the > `snapshot` — a persisted config field, exposed as `lock` in `status/current`) and the
> **flock** on `/var/lock/qemu-server/lock-<vmid>.conf` taken by `PVE::QemuConfig->lock_config`, > **flock** on `/var/lock/qemu-server/lock-<vmid>.conf` taken by `PVE::QemuConfig->lock_config`,
> which is not exposed through the API at all. The integration failures were the flock; this > which is not exposed through the API at all. The integration failures were the flock; this
> entry guards the config lock, which an ordinary start/stop never sets. Run 186 confirmed the > entry guards the config lock, which an ordinary start/stop never sets. Run 186 confirmed the
> check never fired — `Restart-PveVm` took 4.11 s, unchanged. The real cause and fix are in > check never fired — `Restart-PveVm` took 4.11 s, unchanged. The flock is handled by **D016**
> **D016**. The waiting behaviour described below is harmless and still applies when a genuine > (serialise server-side where an endpoint exists) and **D020** (retry where none does). The
> config lock is present, so the code stays. > guard below is correct for the config lock and stays, but must never be described as covering
> the flock.
### Decision ### Decision
`WaitForStatusTransition` returns only when the guest reports the expected status **and** `WaitForStatusTransition` returns only when the guest reports the expected status **and**
@@ -579,16 +580,19 @@ not exist, so `Restart-PveContainer` necessarily keeps shutdown + start.
### Anti-pattern (do not reintroduce) ### Anti-pattern (do not reintroduce)
```csharp ```csharp
// NEVER compose a VM restart from two client calls — it races qmeventd's cleanup // NEVER compose a VM restart from two client calls — it races qmeventd's cleanup
var shutdownTask = vmService.ShutdownVm(session, node, vmid, timeout); WaitForStatusTransition(session, node, () => vmService.ShutdownVm(session, node, vmid, timeout),
WaitForStatusTransition(session, node, shutdownTask, vmid, "stopped", timeout); vmid, "stopped", timeout);
var startTask = vmService.StartVm(session, node, vmid); WaitForStatusTransition(session, node, () => vmService.StartVm(session, node, vmid),
vmid, "running", timeout);
``` ```
### Correct pattern ### Correct pattern
```csharp ```csharp
var task = vmService.RebootVm(session, node, vmid, timeout); PveTask Issue() => vmService.RebootVm(session, node, vmid, timeout);
if (Wait.IsPresent)
task = WaitForStatusTransition(session, node, task, vmid, "running", timeout); var task = Wait.IsPresent
? WaitForStatusTransition(session, node, Issue, vmid, "running", timeout)
: Issue();
``` ```
--- ---
@@ -778,3 +782,98 @@ removes real friction. This one removed none.
A `dev.ps1`, `Makefile` target, or shell function that re-implements provisioning steps, A `dev.ps1`, `Makefile` target, or shell function that re-implements provisioning steps,
module installation, or test invocation. If a local flow is awkward, fix it in module installation, or test invocation. If a local flow is awkward, fix it in
`run-integration.sh` so CI gets the fix too. `run-integration.sh` so CI gets the fix too.
---
## D020 — The qemu-server flock is retried, never predicted
**Status**: Active
**Finding refs**: (none — issue #113, reproduced 2026-09-01 on a Rosetta-emulated client)
**Resolved in scan**: n/a
### Decision
An operation PVE rejects with `can't lock file '<guest lock path>' - got timeout` is reissued
for a bounded window (`GuestLockRetry.DefaultWindow`, 45 s). Nothing in the module may attempt
to *detect* that the flock is held before acting.
Two seams implement it, and both are required:
- **`PveHttpClient.SendAsync`** — retries the request itself. This covers every operation PVE
serialises inside the API handler, where the failure arrives as a 500: `Set-PveVmConfig`,
`Resize-PveVmDisk`'s config writes, and every future call that goes through the client.
The private send takes a `Func<HttpRequestMessage>` rather than a request because an
`HttpRequestMessage` cannot be sent twice.
- **`PveCmdletBase.InvokeGuestTask`** — reissues the API call *and* re-waits its task. PVE takes
the flock inside the forked worker for most guest operations (`qmreset`, `qmclone`), so the
POST returns 200 with a UPID and the failure only appears in the task's exit status. The HTTP
layer cannot see it and cannot retry it. `WaitForStatusTransition` routes through this helper,
which is why it takes a `Func<PveTask>` instead of an already-issued `PveTask`.
Reissuing is safe **only** for a failure to *enter* `lock_config`, which PVE raises before the
operation does any work. `GuestLockRetry.IsLockTimeout` must keep both properties that establish
this, and no failure may be added to it without them:
- **Path-specific.** `PVE::Tools::lock_file` emits the identical wording for storage, LVM, HA,
backup and firewall locks. Those are taken mid-worker and carry no such guarantee, so the match
names the two guest config paths (`/var/lock/qemu-server/lock-<vmid>.conf`,
`/run/lock/lxc/pve-config-<vmid>.lock`) rather than the generic phrasing.
- **Anchored at the start of what PVE said.** `qmclone` is the operation that makes this matter:
its worker creates and locks the target config, allocates disks, then re-locks. A timeout at one
of those later points reads the same as one at entry, and reissuing it would hit
`check_vmid_unused` — "VM <newid> already exists" — leaving an orphaned guest behind. PVE
prefixes the late form with its own context (`clone failed: ...`), so anchoring rejects it.
`Resize-PveVmDisk -Size '+1G'` is the case where getting this wrong is irreversible rather than
merely messy.
The anchor only works against the raw text, so the predicate reads
`PveTaskFailedException.ExitStatus` and `PveApiException.ApiMessage` — never `Exception.Message`,
which both types prefix with their own context. `ApiMessage` exists for this.
### Rationale
D015 tried to predict the lock and guarded the wrong one (see its note). D016 removed the race
for `Restart-PveVm` by handing the ordering to PVE, but that only works where a server-side
serialised endpoint exists. `Set-PveVmConfig`, `Resize-PveVmDisk` and clone have none, so for
them the choice is retry or nothing.
The window is 45 s because `qm cleanup` holds the flock while polling `vm_running_locally` for
up to 30 s, and each rejected attempt first burns PVE's own 10 s `lock_config` timeout. It bounds
when a *new* attempt may start, not total wall clock: an attempt beginning just inside the window
still runs to its own conclusion, so the real ceiling is roughly one attempt longer.
Two consequences are deliberate, and both are load-bearing enough to state rather than discover:
- **`-Timeout` does not bound the retry.** It is documented as the budget for the status
transition, and `WaitForStatusTransition` starts counting it only after the operation's task
completes. Binding the retry to it would defeat the fix at exactly the values that need it —
`Reset-PveVm -Wait -Timeout 30` needed ~31 s of retrying in the run that verified this change.
- **The two seams nest.** A cmdlet operation rejected synchronously burns the HTTP layer's window
inside `InvokeGuestTask`'s. The overlap costs a longer wait before the same failure, never a
different outcome, so it is not worth threading a shared budget through both layers.
CI never showed this. Runs 189200 were green because the CI client is fast enough to win the
race. It reproduces on a client roughly 40% slower — the CI container image run under Docker
Desktop's Rosetta emulation on Apple Silicon — which failed `Should hard-reset a running VM`,
`Should clone a VM` and `Should resize a VM disk (Resize-PveVmDisk)` on the same commit CI
passed. **A green CI run is not evidence about this class of bug.**
### Not yet adopted
`InvokeGuestTask` is the correct seam for every cmdlet that issues a guest operation and waits
on its task. `Remove-PveVm`, `Move-PveVm`, the snapshot and template cmdlets, and the container
equivalents still call `TaskService.WaitForTask` directly and remain exposed to the same race.
They adopt the helper as they are next touched.
### Anti-pattern (do not reintroduce)
```csharp
// NEVER try to observe the flock — PVE does not expose it in status/current or anywhere else
if (!snapshot.Locked)
return task; // reads the config `lock:` property; says nothing about the flock
```
### Correct pattern
```csharp
PveTask Issue() => vmService.CloneVm(session, sourceNode, vmid, newid, name, targetNode, full);
var task = Wait.IsPresent
? InvokeGuestTask(session, sourceNode, Issue)
: Issue();
```
+36 -18
View File
@@ -10,6 +10,7 @@ using System.Threading.Tasks;
using Newtonsoft.Json.Linq; using Newtonsoft.Json.Linq;
using PSProxmoxVE.Core.Authentication; using PSProxmoxVE.Core.Authentication;
using PSProxmoxVE.Core.Exceptions; using PSProxmoxVE.Core.Exceptions;
using PSProxmoxVE.Core.Utilities;
using System.Net.Security; using System.Net.Security;
using System.Security.Cryptography.X509Certificates; using System.Security.Cryptography.X509Certificates;
@@ -62,7 +63,7 @@ namespace PSProxmoxVE.Core.Client
/// <summary> /// <summary>
/// Creates a bare HTTP client for pre-session use (e.g. initial authentication). /// Creates a bare HTTP client for pre-session use (e.g. initial authentication).
/// No auth headers are added to requests made with this constructor. /// Requests it builds carry no authentication headers.
/// </summary> /// </summary>
internal PveHttpClient(string hostname, int port, bool skipCertificateCheck, TimeSpan? timeout = null) internal PveHttpClient(string hostname, int port, bool skipCertificateCheck, TimeSpan? timeout = null)
{ {
@@ -95,8 +96,8 @@ namespace PSProxmoxVE.Core.Client
/// <returns>Raw JSON response body</returns> /// <returns>Raw JSON response body</returns>
public async Task<string> GetAsync(string resource) public async Task<string> GetAsync(string resource)
{ {
var request = BuildRequest(HttpMethod.Get, resource); return await SendAsync(() => BuildRequest(HttpMethod.Get, resource), resource, "GET")
return await SendAsync(request, resource, "GET").ConfigureAwait(false); .ConfigureAwait(false);
} }
/// <summary>Performs a POST request against the specified API resource path.</summary> /// <summary>Performs a POST request against the specified API resource path.</summary>
@@ -105,10 +106,13 @@ namespace PSProxmoxVE.Core.Client
/// <returns>Raw JSON response body</returns> /// <returns>Raw JSON response body</returns>
public async Task<string> PostAsync(string resource, Dictionary<string, string>? data = null) public async Task<string> PostAsync(string resource, Dictionary<string, string>? data = null)
{ {
var request = BuildRequest(HttpMethod.Post, resource, mutating: true); return await SendAsync(() =>
if (data != null) {
request.Content = BuildFormContent(data); var request = BuildRequest(HttpMethod.Post, resource, mutating: true);
return await SendAsync(request, resource, "POST").ConfigureAwait(false); if (data != null)
request.Content = BuildFormContent(data);
return request;
}, resource, "POST").ConfigureAwait(false);
} }
/// <summary> /// <summary>
@@ -121,9 +125,12 @@ namespace PSProxmoxVE.Core.Client
public async Task<string> PostAsync(string resource, IEnumerable<KeyValuePair<string, string>> data) public async Task<string> PostAsync(string resource, IEnumerable<KeyValuePair<string, string>> data)
{ {
if (data == null) throw new ArgumentNullException(nameof(data)); if (data == null) throw new ArgumentNullException(nameof(data));
var request = BuildRequest(HttpMethod.Post, resource, mutating: true); return await SendAsync(() =>
request.Content = BuildFormContent(data); {
return await SendAsync(request, resource, "POST").ConfigureAwait(false); var request = BuildRequest(HttpMethod.Post, resource, mutating: true);
request.Content = BuildFormContent(data);
return request;
}, resource, "POST").ConfigureAwait(false);
} }
/// <summary>Performs a PUT request against the specified API resource path.</summary> /// <summary>Performs a PUT request against the specified API resource path.</summary>
@@ -132,10 +139,13 @@ namespace PSProxmoxVE.Core.Client
/// <returns>Raw JSON response body</returns> /// <returns>Raw JSON response body</returns>
public async Task<string> PutAsync(string resource, Dictionary<string, string>? data = null) public async Task<string> PutAsync(string resource, Dictionary<string, string>? data = null)
{ {
var request = BuildRequest(HttpMethod.Put, resource, mutating: true); return await SendAsync(() =>
if (data != null) {
request.Content = BuildFormContent(data); var request = BuildRequest(HttpMethod.Put, resource, mutating: true);
return await SendAsync(request, resource, "PUT").ConfigureAwait(false); if (data != null)
request.Content = BuildFormContent(data);
return request;
}, resource, "PUT").ConfigureAwait(false);
} }
/// <summary>Performs a DELETE request against the specified API resource path.</summary> /// <summary>Performs a DELETE request against the specified API resource path.</summary>
@@ -143,8 +153,8 @@ namespace PSProxmoxVE.Core.Client
/// <returns>Raw JSON response body</returns> /// <returns>Raw JSON response body</returns>
public async Task<string> DeleteAsync(string resource) public async Task<string> DeleteAsync(string resource)
{ {
var request = BuildRequest(HttpMethod.Delete, resource, mutating: true); return await SendAsync(() => BuildRequest(HttpMethod.Delete, resource, mutating: true), resource, "DELETE")
return await SendAsync(request, resource, "DELETE").ConfigureAwait(false); .ConfigureAwait(false);
} }
// ------------------------------------------------------------------------- // -------------------------------------------------------------------------
@@ -331,7 +341,7 @@ namespace PSProxmoxVE.Core.Client
var request = BuildRequest(HttpMethod.Post, resource, mutating: true); var request = BuildRequest(HttpMethod.Post, resource, mutating: true);
request.Content = multipart; request.Content = multipart;
return await SendAsync(request, resource, "POST").ConfigureAwait(false); return await SendOnceAsync(request, resource, "POST").ConfigureAwait(false);
} }
finally finally
{ {
@@ -366,7 +376,15 @@ namespace PSProxmoxVE.Core.Client
return request; return request;
} }
private async Task<string> SendAsync(HttpRequestMessage request, string resource, string httpMethod) /// <summary>
/// Sends a request, rebuilding it from <paramref name="buildRequest"/> for each attempt
/// while PVE rejects it for a guest's config flock. An <see cref="HttpRequestMessage"/>
/// cannot be resent, which is why this takes a factory rather than a request.
/// </summary>
private Task<string> SendAsync(Func<HttpRequestMessage> buildRequest, string resource, string httpMethod) =>
GuestLockRetry.ExecuteAsync(() => SendOnceAsync(buildRequest(), resource, httpMethod));
private async Task<string> SendOnceAsync(HttpRequestMessage request, string resource, string httpMethod)
{ {
HttpResponseMessage response; HttpResponseMessage response;
try try
@@ -15,6 +15,13 @@ namespace PSProxmoxVE.Core.Exceptions
/// <summary>The HTTP method used for the request (GET, POST, PUT, DELETE).</summary> /// <summary>The HTTP method used for the request (GET, POST, PUT, DELETE).</summary>
public string HttpMethod { get; } public string HttpMethod { get; }
/// <summary>
/// The error text PVE returned, without the status/resource prefix that
/// <see cref="Exception.Message"/> carries. Callers that match on what PVE said —
/// rather than on how this exception renders it — must read this.
/// </summary>
public string ApiMessage { get; }
/// <summary>Initializes a new instance for a failed PVE API request.</summary> /// <summary>Initializes a new instance for a failed PVE API request.</summary>
/// <param name="statusCode">The HTTP status code returned.</param> /// <param name="statusCode">The HTTP status code returned.</param>
/// <param name="message">The error message from the API.</param> /// <param name="message">The error message from the API.</param>
@@ -26,6 +33,7 @@ namespace PSProxmoxVE.Core.Exceptions
StatusCode = statusCode; StatusCode = statusCode;
Resource = resource; Resource = resource;
HttpMethod = httpMethod; HttpMethod = httpMethod;
ApiMessage = message;
} }
/// <summary>Initializes a new instance for a failed PVE API request, with an inner exception.</summary> /// <summary>Initializes a new instance for a failed PVE API request, with an inner exception.</summary>
@@ -40,6 +48,7 @@ namespace PSProxmoxVE.Core.Exceptions
StatusCode = statusCode; StatusCode = statusCode;
Resource = resource; Resource = resource;
HttpMethod = httpMethod; HttpMethod = httpMethod;
ApiMessage = message;
} }
} }
} }
@@ -0,0 +1,103 @@
using System;
using System.Diagnostics;
using System.Text.RegularExpressions;
using System.Threading;
using System.Threading.Tasks;
using PSProxmoxVE.Core.Exceptions;
namespace PSProxmoxVE.Core.Utilities
{
/// <summary>
/// Retries an operation PVE rejected because it could not acquire a guest's config
/// flock (<c>/var/lock/qemu-server/lock-&lt;vmid&gt;.conf</c> for VMs,
/// <c>/run/lock/lxc/pve-config-&lt;vmid&gt;.lock</c> for containers).
///
/// That flock is held by <c>qm cleanup</c> after a guest stops and is not exposed
/// through the API in any form, so a caller cannot wait for it — only retry past it.
/// </summary>
public static class GuestLockRetry
{
/// <summary>
/// How long <see cref="Execute{T}"/> and <see cref="ExecuteAsync{T}"/> keep retrying.
/// <c>qm cleanup</c> polls <c>vm_running_locally</c> for up to 30s while holding the
/// flock, and each rejected attempt first burns PVE's own 10s <c>lock_config</c> timeout.
/// </summary>
public static readonly TimeSpan DefaultWindow = TimeSpan.FromSeconds(45);
private static readonly TimeSpan RetryInterval = TimeSpan.FromSeconds(2);
// Anchored, and specific to the two guest lock paths. `PVE::Tools::lock_file` emits this
// same wording for storage, LVM, HA and firewall locks, none of which carry the
// reissue-safety guarantee below. The anchor is what separates a failure to *enter*
// lock_config from one PVE prefixed with its own context ("clone failed: ..."), which
// means the worker had already done work.
private static readonly Regex GuestLockTimeout = new Regex(
@"^can't lock file '(?:/var/lock/qemu-server/lock-\d+\.conf|/run/lock/lxc/pve-config-\d+\.lock)' - got timeout",
RegexOptions.IgnoreCase | RegexOptions.CultureInvariant);
/// <summary>
/// True when <paramref name="ex"/> reports PVE failing to enter <c>lock_config</c> for a
/// guest. That is raised before the operation performs any work, so a call failing this
/// way is known not to have run and is safe to reissue.
///
/// Matched against what PVE actually said — <see cref="PveTaskFailedException.ExitStatus"/>
/// and <see cref="PveApiException.ApiMessage"/> — never against the composed
/// <see cref="Exception.Message"/>, whose prefix would defeat the anchor.
/// </summary>
/// <param name="ex">The exception to classify.</param>
public static bool IsLockTimeout(Exception ex) => ex switch
{
PveTaskFailedException task => GuestLockTimeout.IsMatch((task.ExitStatus ?? string.Empty).Trim()),
PveApiException api => GuestLockTimeout.IsMatch((api.ApiMessage ?? string.Empty).Trim()),
_ => false,
};
/// <summary>
/// Runs <paramref name="operation"/>, reissuing it while it fails on the guest config
/// flock and <paramref name="window"/> has not elapsed. Any other exception propagates
/// on the first attempt.
/// </summary>
/// <param name="operation">The operation to run.</param>
/// <param name="window">Retry budget. Defaults to <see cref="DefaultWindow"/>.</param>
public static T Execute<T>(Func<T> operation, TimeSpan? window = null)
{
if (operation == null) throw new ArgumentNullException(nameof(operation));
var budget = window ?? DefaultWindow;
var elapsed = Stopwatch.StartNew();
while (true)
{
try
{
return operation();
}
catch (Exception ex) when (IsLockTimeout(ex) && elapsed.Elapsed < budget)
{
Thread.Sleep(RetryInterval);
}
}
}
/// <summary>Asynchronous counterpart of <see cref="Execute{T}"/>.</summary>
/// <param name="operation">The operation to run.</param>
/// <param name="window">Retry budget. Defaults to <see cref="DefaultWindow"/>.</param>
public static async Task<T> ExecuteAsync<T>(Func<Task<T>> operation, TimeSpan? window = null)
{
if (operation == null) throw new ArgumentNullException(nameof(operation));
var budget = window ?? DefaultWindow;
var elapsed = Stopwatch.StartNew();
while (true)
{
try
{
return await operation().ConfigureAwait(false);
}
catch (Exception ex) when (IsLockTimeout(ex) && elapsed.Elapsed < budget)
{
await Task.Delay(RetryInterval).ConfigureAwait(false);
}
}
}
}
}
@@ -18,8 +18,9 @@ namespace PSProxmoxVE.Core.Utilities
/// StatusMatched: the guest reports <paramref name="expectedStatus"/>. qmpstatus is /// StatusMatched: the guest reports <paramref name="expectedStatus"/>. qmpstatus is
/// preferred over status when present, because PVE reports status=running with /// preferred over status when present, because PVE reports status=running with
/// qmpstatus=paused for a suspended VM. /// qmpstatus=paused for a suspended VM.
/// Locked: the guest config still carries a lock, so the next API call against it /// Locked: the guest config carries a `lock:` property (backup, clone, migrate,
/// would fail to acquire the lock file. /// snapshot). This is not the /var/lock/qemu-server flock, which PVE does not expose
/// through status/current or any other endpoint — see DECISIONS.md D015 and D020.
/// </returns> /// </returns>
public static (bool StatusMatched, bool Locked) Evaluate(string json, string expectedStatus) public static (bool StatusMatched, bool Locked) Evaluate(string json, string expectedStatus)
{ {
@@ -56,17 +56,17 @@ namespace PSProxmoxVE.Cmdlets.Containers
WriteVerbose($"Restarting container {VmId} on node '{Node}'..."); WriteVerbose($"Restarting container {VmId} on node '{Node}'...");
// Graceful shutdown PveTask Shutdown() => containerService.ShutdownContainer(session, Node, VmId, Timeout);
var shutdownTask = containerService.ShutdownContainer(session, Node, VmId, Timeout); PveTask Start() => containerService.StartContainer(session, Node, VmId);
if (Wait.IsPresent) if (Wait.IsPresent)
WaitForStatusTransition(session, Node, shutdownTask, VmId, "stopped", Timeout, isContainer: true); WaitForStatusTransition(session, Node, Shutdown, VmId, "stopped", Timeout, isContainer: true);
else
Shutdown();
// Start var startTask = Wait.IsPresent
var startTask = containerService.StartContainer(session, Node, VmId); ? WaitForStatusTransition(session, Node, Start, VmId, "running", Timeout, isContainer: true)
: Start();
if (Wait.IsPresent)
startTask = WaitForStatusTransition(session, Node, startTask, VmId, "running", Timeout, isContainer: true);
WriteObject(startTask); WriteObject(startTask);
} }
@@ -50,12 +50,11 @@ namespace PSProxmoxVE.Cmdlets.Containers
var containerService = new ContainerService(); var containerService = new ContainerService();
WriteVerbose($"Starting container {VmId} on node '{Node}'..."); WriteVerbose($"Starting container {VmId} on node '{Node}'...");
var task = containerService.StartContainer(session, Node, VmId); PveTask Issue() => containerService.StartContainer(session, Node, VmId);
if (Wait.IsPresent) var task = Wait.IsPresent
{ ? WaitForStatusTransition(session, Node, Issue, VmId, "running", Timeout, isContainer: true)
task = WaitForStatusTransition(session, Node, task, VmId, "running", Timeout, isContainer: true); : Issue();
}
WriteObject(task); WriteObject(task);
} }
@@ -52,12 +52,11 @@ namespace PSProxmoxVE.Cmdlets.Containers
var containerService = new ContainerService(); var containerService = new ContainerService();
WriteVerbose($"Stopping container {VmId} on node '{Node}'..."); WriteVerbose($"Stopping container {VmId} on node '{Node}'...");
var task = containerService.StopContainer(session, Node, VmId); PveTask Issue() => containerService.StopContainer(session, Node, VmId);
if (Wait.IsPresent) var task = Wait.IsPresent
{ ? WaitForStatusTransition(session, Node, Issue, VmId, "stopped", Timeout, isContainer: true)
task = WaitForStatusTransition(session, Node, task, VmId, "stopped", Timeout, isContainer: true); : Issue();
}
WriteObject(task); WriteObject(task);
} }
+38 -18
View File
@@ -99,7 +99,10 @@ namespace PSProxmoxVE.Cmdlets
/// </summary> /// </summary>
/// <param name="session">The authenticated PVE session.</param> /// <param name="session">The authenticated PVE session.</param>
/// <param name="node">The cluster node name.</param> /// <param name="node">The cluster node name.</param>
/// <param name="task">The task returned by the lifecycle API call.</param> /// <param name="issueOperation">
/// Issues the lifecycle API call. Invoked again on each retry, so it must be safe to
/// repeat — see <see cref="InvokeGuestTask"/>.
/// </param>
/// <param name="vmid">The VM or container ID to poll.</param> /// <param name="vmid">The VM or container ID to poll.</param>
/// <param name="expectedStatus">The expected status string (e.g. "running", "stopped", "paused").</param> /// <param name="expectedStatus">The expected status string (e.g. "running", "stopped", "paused").</param>
/// <param name="timeoutSeconds">Maximum seconds to wait for the status transition. Default 60.</param> /// <param name="timeoutSeconds">Maximum seconds to wait for the status transition. Default 60.</param>
@@ -108,19 +111,13 @@ namespace PSProxmoxVE.Cmdlets
protected PveTask WaitForStatusTransition( protected PveTask WaitForStatusTransition(
PveSession session, PveSession session,
string node, string node,
PveTask task, Func<PveTask> issueOperation,
int vmid, int vmid,
string expectedStatus, string expectedStatus,
int timeoutSeconds = 60, int timeoutSeconds = 60,
bool isContainer = false) bool isContainer = false)
{ {
var taskService = new TaskService(); var task = InvokeGuestTask(session, node, issueOperation);
// First wait for the PVE task to complete
if (!string.IsNullOrEmpty(task.Upid))
{
task = taskService.WaitForTask(session, node, task.Upid, null, null, null);
}
// Then poll status/current until VM/container reaches the expected status. // Then poll status/current until VM/container reaches the expected status.
// We query the status/current endpoint directly instead of the list endpoint // We query the status/current endpoint directly instead of the list endpoint
@@ -141,15 +138,11 @@ namespace PSProxmoxVE.Cmdlets
var snapshot = GuestStatusSnapshot.Evaluate(json, expectedStatus); var snapshot = GuestStatusSnapshot.Evaluate(json, expectedStatus);
lastMatched = snapshot.StatusMatched; lastMatched = snapshot.StatusMatched;
if (snapshot.StatusMatched) // snapshot.Locked is the config `lock:` property (backup, clone, migrate,
{ // snapshot) — not the /var/lock/qemu-server flock, which PVE does not
// PVE reports the target status before the operation releases the // expose. The flock race is handled by retrying, not by waiting.
// config lock. A caller that issues its next request inside that if (snapshot.StatusMatched && !snapshot.Locked)
// window gets "can't lock file '/var/lock/qemu-server/lock-<vmid>.conf' return task;
// - got timeout" from its own API call.
if (!snapshot.Locked)
return task;
}
} }
catch (Exception ex) when (ex is not OutOfMemoryException and not StackOverflowException) catch (Exception ex) when (ex is not OutOfMemoryException and not StackOverflowException)
{ {
@@ -169,6 +162,33 @@ namespace PSProxmoxVE.Cmdlets
TimeSpan.FromSeconds(timeoutSeconds)); TimeSpan.FromSeconds(timeoutSeconds));
} }
/// <summary>
/// Issues a guest operation and waits for the task it returns, reissuing the pair while
/// PVE rejects it for the guest's config flock.
///
/// PVE takes that flock inside the worker for most guest operations, so the failure
/// surfaces as a failed task rather than a failed request and cannot be retried at the
/// HTTP layer. <c>lock_config</c> raises it before doing any work, so a reissue repeats
/// nothing.
/// </summary>
/// <param name="session">The authenticated PVE session.</param>
/// <param name="node">The node the task runs on.</param>
/// <param name="issueOperation">Issues the API call; invoked again on each retry.</param>
/// <returns>The completed task, or the issued task when the call returned no UPID.</returns>
protected PveTask InvokeGuestTask(PveSession session, string node, Func<PveTask> issueOperation)
{
if (issueOperation == null) throw new ArgumentNullException(nameof(issueOperation));
var taskService = new TaskService();
return GuestLockRetry.Execute(() =>
{
var task = issueOperation();
return string.IsNullOrEmpty(task.Upid)
? task
: taskService.WaitForTask(session, node, task.Upid, null, null, null);
});
}
/// <summary> /// <summary>
/// Extracts the node name from a UPID string (format: UPID:node:...). /// Extracts the node name from a UPID string (format: UPID:node:...).
/// Falls back to <paramref name="fallback"/> if the UPID is empty or cannot be parsed. /// Falls back to <paramref name="fallback"/> if the UPID is empty or cannot be parsed.
@@ -81,13 +81,11 @@ namespace PSProxmoxVE.Cmdlets.Vms
WriteVerbose($"Cloning VM {VmId}..."); WriteVerbose($"Cloning VM {VmId}...");
var newid = NewVmId ?? 0; var newid = NewVmId ?? 0;
var task = vmService.CloneVm(session, SourceNode, VmId, newid, NewName, TargetNode, Full.IsPresent); PveTask Issue() => vmService.CloneVm(session, SourceNode, VmId, newid, NewName, TargetNode, Full.IsPresent);
if (Wait.IsPresent) var task = Wait.IsPresent
{ ? InvokeGuestTask(session, SourceNode, Issue)
var taskService = new TaskService(); : Issue();
task = taskService.WaitForTask(session, SourceNode, task.Upid, null, null, null);
}
WriteObject(task); WriteObject(task);
} }
@@ -52,12 +52,11 @@ namespace PSProxmoxVE.Cmdlets.Vms
var vmService = new VmService(); var vmService = new VmService();
WriteVerbose($"Resetting VM {VmId} on node '{Node}'..."); WriteVerbose($"Resetting VM {VmId} on node '{Node}'...");
var task = vmService.ResetVm(session, Node, VmId); PveTask Issue() => vmService.ResetVm(session, Node, VmId);
if (Wait.IsPresent) var task = Wait.IsPresent
{ ? WaitForStatusTransition(session, Node, Issue, VmId, "running", Timeout)
task = WaitForStatusTransition(session, Node, task, VmId, "running", Timeout); : Issue();
}
WriteObject(task); WriteObject(task);
} }
@@ -65,13 +65,11 @@ namespace PSProxmoxVE.Cmdlets.Vms
var vmService = new VmService(); var vmService = new VmService();
WriteVerbose($"Resizing disk '{Disk}' on VM {VmId}..."); WriteVerbose($"Resizing disk '{Disk}' on VM {VmId}...");
var task = vmService.ResizeDisk(session, Node, VmId, Disk, Size); PveTask Issue() => vmService.ResizeDisk(session, Node, VmId, Disk, Size);
if (Wait.IsPresent) var task = Wait.IsPresent
{ ? InvokeGuestTask(session, Node, Issue)
var taskService = new TaskService(); : Issue();
task = taskService.WaitForTask(session, Node, task.Upid, null, null, null);
}
WriteObject(task); WriteObject(task);
} }
@@ -57,10 +57,11 @@ namespace PSProxmoxVE.Cmdlets.Vms
WriteVerbose($"Restarting VM {VmId} on node '{Node}'..."); WriteVerbose($"Restarting VM {VmId} on node '{Node}'...");
var task = vmService.RebootVm(session, Node, VmId, Timeout); PveTask Issue() => vmService.RebootVm(session, Node, VmId, Timeout);
if (Wait.IsPresent) var task = Wait.IsPresent
task = WaitForStatusTransition(session, Node, task, VmId, "running", Timeout); ? WaitForStatusTransition(session, Node, Issue, VmId, "running", Timeout)
: Issue();
WriteObject(task); WriteObject(task);
} }
@@ -50,12 +50,11 @@ namespace PSProxmoxVE.Cmdlets.Vms
var vmService = new VmService(); var vmService = new VmService();
WriteVerbose($"Resuming VM {VmId} on node '{Node}'..."); WriteVerbose($"Resuming VM {VmId} on node '{Node}'...");
var task = vmService.ResumeVm(session, Node, VmId); PveTask Issue() => vmService.ResumeVm(session, Node, VmId);
if (Wait.IsPresent) var task = Wait.IsPresent
{ ? WaitForStatusTransition(session, Node, Issue, VmId, "running", Timeout)
task = WaitForStatusTransition(session, Node, task, VmId, "running", Timeout); : Issue();
}
WriteObject(task); WriteObject(task);
} }
@@ -50,12 +50,11 @@ namespace PSProxmoxVE.Cmdlets.Vms
var vmService = new VmService(); var vmService = new VmService();
WriteVerbose($"Starting VM {VmId} on node '{Node}'..."); WriteVerbose($"Starting VM {VmId} on node '{Node}'...");
var task = vmService.StartVm(session, Node, VmId); PveTask Issue() => vmService.StartVm(session, Node, VmId);
if (Wait.IsPresent) var task = Wait.IsPresent
{ ? WaitForStatusTransition(session, Node, Issue, VmId, "running", Timeout)
task = WaitForStatusTransition(session, Node, task, VmId, "running", Timeout); : Issue();
}
WriteObject(task); WriteObject(task);
} }
@@ -52,12 +52,11 @@ namespace PSProxmoxVE.Cmdlets.Vms
var vmService = new VmService(); var vmService = new VmService();
WriteVerbose($"Stopping VM {VmId} on node '{Node}'..."); WriteVerbose($"Stopping VM {VmId} on node '{Node}'...");
var task = vmService.StopVm(session, Node, VmId); PveTask Issue() => vmService.StopVm(session, Node, VmId);
if (Wait.IsPresent) var task = Wait.IsPresent
{ ? WaitForStatusTransition(session, Node, Issue, VmId, "stopped", Timeout)
task = WaitForStatusTransition(session, Node, task, VmId, "stopped", Timeout); : Issue();
}
WriteObject(task); WriteObject(task);
} }
@@ -51,12 +51,11 @@ namespace PSProxmoxVE.Cmdlets.Vms
var vmService = new VmService(); var vmService = new VmService();
WriteVerbose($"Suspending VM {VmId} on node '{Node}'..."); WriteVerbose($"Suspending VM {VmId} on node '{Node}'...");
var task = vmService.SuspendVm(session, Node, VmId); PveTask Issue() => vmService.SuspendVm(session, Node, VmId);
if (Wait.IsPresent) var task = Wait.IsPresent
{ ? WaitForStatusTransition(session, Node, Issue, VmId, "paused", Timeout)
task = WaitForStatusTransition(session, Node, task, VmId, "paused", Timeout); : Issue();
}
WriteObject(task); WriteObject(task);
} }
@@ -0,0 +1,143 @@
using System;
using System.Collections.Generic;
using System.Net;
using System.Net.Http;
using System.Reflection;
using System.Threading;
using System.Threading.Tasks;
using PSProxmoxVE.Core.Authentication;
using PSProxmoxVE.Core.Client;
using PSProxmoxVE.Core.Exceptions;
using Xunit;
namespace PSProxmoxVE.Core.Tests.Client
{
public class PveHttpClientLockRetryTests
{
private const string LockTimeoutBody =
"{\"message\":\"can't lock file '/var/lock/qemu-server/lock-100.conf' - got timeout\"}";
private static void SetInnerHttpClient(PveHttpClient client, HttpClient newInner)
{
var field = typeof(PveHttpClient).GetField("_httpClient",
BindingFlags.Instance | BindingFlags.NonPublic)!;
((HttpClient)field.GetValue(client)!).Dispose();
field.SetValue(client, newInner);
}
private static (PveHttpClient client, ScriptedHandler handler) NewClient(
params (HttpStatusCode status, string body)[] responses)
{
var session = new PveSession("pve.example.com", 8006, false,
"root@pam!token=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee");
var client = new PveHttpClient(session);
var handler = new ScriptedHandler(responses);
SetInnerHttpClient(client, new HttpClient(handler));
return (client, handler);
}
private static Dictionary<string, string> ConfigBody() =>
new Dictionary<string, string> { ["scsi0"] = "local-lvm:1" };
[Fact]
public async Task PutAsync_ReissuesTheRequestWhilePveReportsTheGuestFlock()
{
var (client, handler) = NewClient(
(HttpStatusCode.InternalServerError, LockTimeoutBody),
(HttpStatusCode.InternalServerError, LockTimeoutBody),
(HttpStatusCode.OK, "{\"data\":null}"));
using (client)
{
var result = await client.PutAsync("nodes/pve9a/qemu/100/config", ConfigBody());
Assert.Equal("{\"data\":null}", result);
}
Assert.Equal(3, handler.Bodies.Count);
}
[Fact]
public async Task PutAsync_RebuildsTheRequestSoEveryAttemptCarriesTheSameBody()
{
var (client, handler) = NewClient(
(HttpStatusCode.InternalServerError, LockTimeoutBody),
(HttpStatusCode.OK, "{\"data\":null}"));
using (client)
{
await client.PutAsync("nodes/pve9a/qemu/100/config", ConfigBody());
}
Assert.Equal(2, handler.Bodies.Count);
Assert.Equal("scsi0=local-lvm:1", handler.Bodies[0]);
Assert.Equal(handler.Bodies[0], handler.Bodies[1]);
Assert.All(handler.Methods, m => Assert.Equal(HttpMethod.Put, m));
Assert.Equal(handler.Uris[0], handler.Uris[1]);
Assert.EndsWith("nodes/pve9a/qemu/100/config", handler.Uris[0]);
}
[Fact]
public async Task PostAsync_DoesNotReissueApiErrorsThatAreNotTheFlock()
{
var (client, handler) = NewClient(
(HttpStatusCode.InternalServerError, "{\"message\":\"VM 100 not running\"}"),
(HttpStatusCode.OK, "{\"data\":null}"));
using (client)
{
var ex = await Assert.ThrowsAsync<PveApiException>(
() => client.PostAsync("nodes/pve9a/qemu/100/status/reset"));
Assert.Contains("VM 100 not running", ex.Message);
}
Assert.Single(handler.Bodies);
}
[Fact]
public async Task GetAsync_ReissuesWithoutCarryingContent()
{
var (client, handler) = NewClient(
(HttpStatusCode.InternalServerError, LockTimeoutBody),
(HttpStatusCode.OK, "{\"data\":{}}"));
using (client)
{
await client.GetAsync("nodes/pve9a/qemu/100/status/current");
}
Assert.Equal(2, handler.Bodies.Count);
Assert.All(handler.Bodies, b => Assert.Equal(string.Empty, b));
}
private sealed class ScriptedHandler : HttpMessageHandler
{
private readonly (HttpStatusCode status, string body)[] _responses;
private int _index;
public List<string> Bodies { get; } = new List<string>();
public List<HttpMethod> Methods { get; } = new List<HttpMethod>();
public List<string> Uris { get; } = new List<string>();
public ScriptedHandler((HttpStatusCode status, string body)[] responses)
{
_responses = responses;
}
protected override async Task<HttpResponseMessage> SendAsync(
HttpRequestMessage request, CancellationToken cancellationToken)
{
Bodies.Add(request.Content == null
? string.Empty
: await request.Content.ReadAsStringAsync().ConfigureAwait(false));
Methods.Add(request.Method);
Uris.Add(request.RequestUri!.ToString());
if (_index >= _responses.Length)
throw new InvalidOperationException("ScriptedHandler ran out of responses.");
var (status, body) = _responses[_index++];
return new HttpResponseMessage(status) { Content = new StringContent(body) };
}
}
}
}
@@ -0,0 +1,160 @@
using System;
using System.Net;
using System.Threading.Tasks;
using PSProxmoxVE.Core.Exceptions;
using PSProxmoxVE.Core.Utilities;
using Xunit;
namespace PSProxmoxVE.Core.Tests.Utilities
{
public class GuestLockRetryTests
{
private const string VmLockError =
"can't lock file '/var/lock/qemu-server/lock-100.conf' - got timeout";
private const string LxcLockError =
"can't lock file '/run/lock/lxc/pve-config-100.lock' - got timeout";
private static PveApiException ApiError(string message) =>
new PveApiException(HttpStatusCode.InternalServerError, message, "nodes/pve9a/qemu/100/config", "PUT");
private static PveTaskFailedException TaskError(string exitStatus) =>
new PveTaskFailedException("UPID:pve9a:00000001:qmreset:100:root@pam:", exitStatus);
[Fact]
public void IsLockTimeout_MatchesTheVmFlockErrorFromBothSurfaces()
{
Assert.True(GuestLockRetry.IsLockTimeout(ApiError(VmLockError)));
Assert.True(GuestLockRetry.IsLockTimeout(TaskError(VmLockError)));
}
[Fact]
public void IsLockTimeout_MatchesTheContainerFlockError()
{
Assert.True(GuestLockRetry.IsLockTimeout(TaskError(LxcLockError)));
}
[Theory]
[InlineData("VM 100 not running")]
[InlineData("can't lock file '/var/lock/qemu-server/lock-100.conf'")]
[InlineData("got timeout")]
// PVE::Tools::lock_file uses this same wording for locks that carry no
// reissue-safety guarantee. Only the two guest config paths may retry.
[InlineData("can't lock file '/run/lock/pve-manager/pve-storage-local' - got timeout")]
[InlineData("can't lock file '/var/lock/pve-manager/pve-backup' - got timeout")]
[InlineData("can't lock file '/run/lock/lvm/V_pve' - got timeout")]
// A message PVE prefixed with its own context means the worker had already
// started; reissuing it could repeat work that landed.
[InlineData("clone failed: can't lock file '/var/lock/qemu-server/lock-100.conf' - got timeout")]
[InlineData("unable to resize: can't lock file '/var/lock/qemu-server/lock-100.conf' - got timeout")]
public void IsLockTimeout_RejectsEveryOtherFailure(string message)
{
Assert.False(GuestLockRetry.IsLockTimeout(ApiError(message)));
Assert.False(GuestLockRetry.IsLockTimeout(TaskError(message)));
}
[Fact]
public void IsLockTimeout_ReadsWhatPveSaidRatherThanTheComposedMessage()
{
// Both exception types prefix Message with their own context, which would
// defeat the anchor if the predicate matched on Message.
var api = ApiError(VmLockError);
var task = TaskError(VmLockError);
Assert.StartsWith("PVE API error", api.Message);
Assert.StartsWith("Task UPID:", task.Message);
Assert.True(GuestLockRetry.IsLockTimeout(api));
Assert.True(GuestLockRetry.IsLockTimeout(task));
}
[Fact]
public void IsLockTimeout_RejectsExceptionTypesThatAreNotApiFailures()
{
Assert.False(GuestLockRetry.IsLockTimeout(new InvalidOperationException(VmLockError)));
}
[Fact]
public void Execute_ReturnsWithoutRetryingWhenTheOperationSucceeds()
{
var attempts = 0;
var result = GuestLockRetry.Execute(() => { attempts++; return 42; });
Assert.Equal(42, result);
Assert.Equal(1, attempts);
}
[Fact]
public void Execute_ReissuesUntilTheLockClears()
{
var attempts = 0;
var result = GuestLockRetry.Execute(() =>
{
attempts++;
if (attempts < 2) throw TaskError(VmLockError);
return "cloned";
});
Assert.Equal("cloned", result);
Assert.Equal(2, attempts);
}
[Fact]
public void Execute_DoesNotRetryOtherFailures()
{
var attempts = 0;
Assert.Throws<PveApiException>(() => GuestLockRetry.Execute<int>(() =>
{
attempts++;
throw ApiError("VM 100 not running");
}));
Assert.Equal(1, attempts);
}
[Fact]
public async Task ExecuteAsync_ReissuesUntilTheLockClears()
{
var attempts = 0;
var result = await GuestLockRetry.ExecuteAsync(() =>
{
attempts++;
if (attempts < 2) throw ApiError(VmLockError);
return Task.FromResult("written");
});
Assert.Equal("written", result);
Assert.Equal(2, attempts);
}
[Fact]
public async Task ExecuteAsync_DoesNotRetryOtherFailures()
{
var attempts = 0;
await Assert.ThrowsAsync<PveApiException>(() => GuestLockRetry.ExecuteAsync<int>(() =>
{
attempts++;
throw ApiError("can't lock file '/run/lock/lvm/V_pve' - got timeout");
}));
Assert.Equal(1, attempts);
}
[Fact]
public void Execute_GivesUpAndRethrowsOnceTheWindowElapses()
{
var attempts = 0;
var ex = Assert.Throws<PveTaskFailedException>(() => GuestLockRetry.Execute<int>(
() => { attempts++; throw TaskError(VmLockError); },
TimeSpan.Zero));
Assert.Contains("got timeout", ex.Message);
Assert.Equal(1, attempts);
}
}
}