mirror of
https://github.com/GoodOlClint/PSProxmoxVE.git
synced 2026-09-04 11:15:34 +00:00
fix: retry the qemu-server flock instead of predicting it
WaitForStatusTransition refused to return while snapshot.Locked, and its comment quoted the exact error it was meant to prevent. Locked reads the guest config's lock: property; the failure is the flock on /var/lock/qemu-server/lock-<vmid>.conf, which PVE exposes nowhere. The flock cannot be observed, so it is retried. GuestLockRetry reissues an operation for a bounded 45s while PVE reports failing to enter lock_config for a guest, which it raises before doing any work. Two seams, because the failure has two surfaces. PveHttpClient.SendAsync retries the request for operations PVE serialises in the API handler; it takes a request factory because an HttpRequestMessage cannot be resent. PveCmdletBase.InvokeGuestTask reissues the call and re-waits its task for operations serialised in the forked worker, where the POST returns 200 and only the task fails. WaitForStatusTransition routes through the latter, hence Func<PveTask>. The predicate is path-specific and anchored at the start of what PVE said: lock_file uses identical wording for storage, LVM and HA locks, and a qmclone that fails after allocating disks must not be reissued into "VM already exists". That requires the raw text, so it reads PveTaskFailedException.ExitStatus and PveApiException.ApiMessage. The Locked check stays — it is correct for the config lock — with a comment that says so. Closes #113
This commit is contained in:
+2
-1
@@ -10,7 +10,8 @@ and this project adheres to [Conventional Commits](https://www.conventionalcommi
|
|||||||
### Fixed
|
### Fixed
|
||||||
|
|
||||||
- `Restart-PveVm` now uses PVE's native reboot endpoint (`POST {vmid}/status/reboot`) instead of composing a shutdown followed by a start. The two-call form raced Proxmox's own post-stop cleanup: the start won the guest's config lock, `qm cleanup` then held that lock for 30 seconds waiting on the newly started process, and the caller's next operation failed with `can't lock file '/var/lock/qemu-server/lock-<vmid>.conf' - got timeout`. Reproduced in integration runs 183, 185 and 186 as a cascade of 4 failures. `Restart-PveContainer` is unchanged — LXC has no reboot endpoint. See `DECISIONS.md` D016.
|
- `Restart-PveVm` now uses PVE's native reboot endpoint (`POST {vmid}/status/reboot`) instead of composing a shutdown followed by a start. The two-call form raced Proxmox's own post-stop cleanup: the start won the guest's config lock, `qm cleanup` then held that lock for 30 seconds waiting on the newly started process, and the caller's next operation failed with `can't lock file '/var/lock/qemu-server/lock-<vmid>.conf' - got timeout`. Reproduced in integration runs 183, 185 and 186 as a cascade of 4 failures. `Restart-PveContainer` is unchanged — LXC has no reboot endpoint. See `DECISIONS.md` D016.
|
||||||
- Lifecycle cmdlets with `-Wait` (`Start`/`Stop`/`Restart`/`Reset`/`Resume` for VMs and containers) also wait for the guest's config lock (the `lock:` property, e.g. `backup` or `migrate`) to clear before returning, and the post-timeout fallback tests the most recent poll rather than whether a match was ever seen. See `DECISIONS.md` D015 — note that entry misdiagnosed the failure above and is superseded by D016.
|
- Guest operations that Proxmox rejects with `can't lock file '/var/lock/qemu-server/lock-<vmid>.conf' - got timeout` are now reissued for up to 45 seconds instead of surfacing as an error. That flock is taken by `qm cleanup` for up to 30 seconds after a guest stops and is not exposed through the API in any form, so it can only be retried past, never waited on. Covers both the synchronous form (`Set-PveVmConfig`, `Resize-PveVmDisk`, and every other call through the HTTP client) and the asynchronous form, where the request succeeds and the PVE task then fails (`Reset-PveVm`, `Copy-PveVm`). Reproduced on a client ~40% slower than CI, which failed three VM tests on a commit CI passed. (#113) See `DECISIONS.md` D020.
|
||||||
|
- Lifecycle cmdlets with `-Wait` (`Start`/`Stop`/`Restart`/`Reset`/`Resume` for VMs and containers) also wait for the guest's config lock (the `lock:` property, e.g. `backup` or `migrate`) to clear before returning, and the post-timeout fallback tests the most recent poll rather than whether a match was ever seen. See `DECISIONS.md` D015 — that guard covers the config lock only; the separate flock race is D020.
|
||||||
- `New-PveCluster -Wait` now blocks until the cluster reports quorum, not merely until the creation task finishes. PVE's create task returns before corosync converges (~6s earlier in testing), so the natural `New-PveCluster -Wait` → `Add-PveClusterMember` sequence failed with `cluster not ready - no quorum?`. Adds `-Timeout` (seconds, default 60, range 1-3600) following the `-Wait` timeout convention used by `Stop-PveContainer` and `Reset-PveVm`. See `DECISIONS.md` D014.
|
- `New-PveCluster -Wait` now blocks until the cluster reports quorum, not merely until the creation task finishes. PVE's create task returns before corosync converges (~6s earlier in testing), so the natural `New-PveCluster -Wait` → `Add-PveClusterMember` sequence failed with `cluster not ready - no quorum?`. Adds `-Timeout` (seconds, default 60, range 1-3600) following the `-Wait` timeout convention used by `Stop-PveContainer` and `Reset-PveVm`. See `DECISIONS.md` D014.
|
||||||
|
|
||||||
## [0.2.0] - 2026-05-22
|
## [0.2.0] - 2026-05-22
|
||||||
|
|||||||
+111
-12
@@ -466,19 +466,20 @@ Add-PveClusterMember ...
|
|||||||
|
|
||||||
## D015 — Lifecycle -Wait blocks until the guest config lock clears
|
## D015 — Lifecycle -Wait blocks until the guest config lock clears
|
||||||
|
|
||||||
**Status**: Superseded by D016 (2026-09-01) — the mechanism below is wrong
|
**Status**: Active, rescoped 2026-09-01 — it guards the config lock only, never the flock
|
||||||
**Finding refs**: (none — found via integration runs 183/184, 2026-09-01)
|
**Finding refs**: (none — found via integration runs 183/184, 2026-09-01; rescoped for #113)
|
||||||
**Resolved in scan**: n/a
|
**Resolved in scan**: n/a
|
||||||
|
|
||||||
> **This entry misdiagnosed the failure it was written for.** Two different things in PVE are
|
> **This entry was written for a failure it does not prevent.** Two different things in PVE are
|
||||||
> called "lock": the **config lock** (the `lock:` property — `migrate`, `backup`, `clone`,
|
> called "lock": the **config lock** (the `lock:` property — `migrate`, `backup`, `clone`,
|
||||||
> `snapshot` — a persisted config field, exposed as `lock` in `status/current`) and the
|
> `snapshot` — a persisted config field, exposed as `lock` in `status/current`) and the
|
||||||
> **flock** on `/var/lock/qemu-server/lock-<vmid>.conf` taken by `PVE::QemuConfig->lock_config`,
|
> **flock** on `/var/lock/qemu-server/lock-<vmid>.conf` taken by `PVE::QemuConfig->lock_config`,
|
||||||
> which is not exposed through the API at all. The integration failures were the flock; this
|
> which is not exposed through the API at all. The integration failures were the flock; this
|
||||||
> entry guards the config lock, which an ordinary start/stop never sets. Run 186 confirmed the
|
> entry guards the config lock, which an ordinary start/stop never sets. Run 186 confirmed the
|
||||||
> check never fired — `Restart-PveVm` took 4.11 s, unchanged. The real cause and fix are in
|
> check never fired — `Restart-PveVm` took 4.11 s, unchanged. The flock is handled by **D016**
|
||||||
> **D016**. The waiting behaviour described below is harmless and still applies when a genuine
|
> (serialise server-side where an endpoint exists) and **D020** (retry where none does). The
|
||||||
> config lock is present, so the code stays.
|
> guard below is correct for the config lock and stays, but must never be described as covering
|
||||||
|
> the flock.
|
||||||
|
|
||||||
### Decision
|
### Decision
|
||||||
`WaitForStatusTransition` returns only when the guest reports the expected status **and**
|
`WaitForStatusTransition` returns only when the guest reports the expected status **and**
|
||||||
@@ -579,16 +580,19 @@ not exist, so `Restart-PveContainer` necessarily keeps shutdown + start.
|
|||||||
### Anti-pattern (do not reintroduce)
|
### Anti-pattern (do not reintroduce)
|
||||||
```csharp
|
```csharp
|
||||||
// NEVER compose a VM restart from two client calls — it races qmeventd's cleanup
|
// NEVER compose a VM restart from two client calls — it races qmeventd's cleanup
|
||||||
var shutdownTask = vmService.ShutdownVm(session, node, vmid, timeout);
|
WaitForStatusTransition(session, node, () => vmService.ShutdownVm(session, node, vmid, timeout),
|
||||||
WaitForStatusTransition(session, node, shutdownTask, vmid, "stopped", timeout);
|
vmid, "stopped", timeout);
|
||||||
var startTask = vmService.StartVm(session, node, vmid);
|
WaitForStatusTransition(session, node, () => vmService.StartVm(session, node, vmid),
|
||||||
|
vmid, "running", timeout);
|
||||||
```
|
```
|
||||||
|
|
||||||
### Correct pattern
|
### Correct pattern
|
||||||
```csharp
|
```csharp
|
||||||
var task = vmService.RebootVm(session, node, vmid, timeout);
|
PveTask Issue() => vmService.RebootVm(session, node, vmid, timeout);
|
||||||
if (Wait.IsPresent)
|
|
||||||
task = WaitForStatusTransition(session, node, task, vmid, "running", timeout);
|
var task = Wait.IsPresent
|
||||||
|
? WaitForStatusTransition(session, node, Issue, vmid, "running", timeout)
|
||||||
|
: Issue();
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -778,3 +782,98 @@ removes real friction. This one removed none.
|
|||||||
A `dev.ps1`, `Makefile` target, or shell function that re-implements provisioning steps,
|
A `dev.ps1`, `Makefile` target, or shell function that re-implements provisioning steps,
|
||||||
module installation, or test invocation. If a local flow is awkward, fix it in
|
module installation, or test invocation. If a local flow is awkward, fix it in
|
||||||
`run-integration.sh` so CI gets the fix too.
|
`run-integration.sh` so CI gets the fix too.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## D020 — The qemu-server flock is retried, never predicted
|
||||||
|
|
||||||
|
**Status**: Active
|
||||||
|
**Finding refs**: (none — issue #113, reproduced 2026-09-01 on a Rosetta-emulated client)
|
||||||
|
**Resolved in scan**: n/a
|
||||||
|
|
||||||
|
### Decision
|
||||||
|
An operation PVE rejects with `can't lock file '<guest lock path>' - got timeout` is reissued
|
||||||
|
for a bounded window (`GuestLockRetry.DefaultWindow`, 45 s). Nothing in the module may attempt
|
||||||
|
to *detect* that the flock is held before acting.
|
||||||
|
|
||||||
|
Two seams implement it, and both are required:
|
||||||
|
|
||||||
|
- **`PveHttpClient.SendAsync`** — retries the request itself. This covers every operation PVE
|
||||||
|
serialises inside the API handler, where the failure arrives as a 500: `Set-PveVmConfig`,
|
||||||
|
`Resize-PveVmDisk`'s config writes, and every future call that goes through the client.
|
||||||
|
The private send takes a `Func<HttpRequestMessage>` rather than a request because an
|
||||||
|
`HttpRequestMessage` cannot be sent twice.
|
||||||
|
- **`PveCmdletBase.InvokeGuestTask`** — reissues the API call *and* re-waits its task. PVE takes
|
||||||
|
the flock inside the forked worker for most guest operations (`qmreset`, `qmclone`), so the
|
||||||
|
POST returns 200 with a UPID and the failure only appears in the task's exit status. The HTTP
|
||||||
|
layer cannot see it and cannot retry it. `WaitForStatusTransition` routes through this helper,
|
||||||
|
which is why it takes a `Func<PveTask>` instead of an already-issued `PveTask`.
|
||||||
|
|
||||||
|
Reissuing is safe **only** for a failure to *enter* `lock_config`, which PVE raises before the
|
||||||
|
operation does any work. `GuestLockRetry.IsLockTimeout` must keep both properties that establish
|
||||||
|
this, and no failure may be added to it without them:
|
||||||
|
|
||||||
|
- **Path-specific.** `PVE::Tools::lock_file` emits the identical wording for storage, LVM, HA,
|
||||||
|
backup and firewall locks. Those are taken mid-worker and carry no such guarantee, so the match
|
||||||
|
names the two guest config paths (`/var/lock/qemu-server/lock-<vmid>.conf`,
|
||||||
|
`/run/lock/lxc/pve-config-<vmid>.lock`) rather than the generic phrasing.
|
||||||
|
- **Anchored at the start of what PVE said.** `qmclone` is the operation that makes this matter:
|
||||||
|
its worker creates and locks the target config, allocates disks, then re-locks. A timeout at one
|
||||||
|
of those later points reads the same as one at entry, and reissuing it would hit
|
||||||
|
`check_vmid_unused` — "VM <newid> already exists" — leaving an orphaned guest behind. PVE
|
||||||
|
prefixes the late form with its own context (`clone failed: ...`), so anchoring rejects it.
|
||||||
|
`Resize-PveVmDisk -Size '+1G'` is the case where getting this wrong is irreversible rather than
|
||||||
|
merely messy.
|
||||||
|
|
||||||
|
The anchor only works against the raw text, so the predicate reads
|
||||||
|
`PveTaskFailedException.ExitStatus` and `PveApiException.ApiMessage` — never `Exception.Message`,
|
||||||
|
which both types prefix with their own context. `ApiMessage` exists for this.
|
||||||
|
|
||||||
|
### Rationale
|
||||||
|
D015 tried to predict the lock and guarded the wrong one (see its note). D016 removed the race
|
||||||
|
for `Restart-PveVm` by handing the ordering to PVE, but that only works where a server-side
|
||||||
|
serialised endpoint exists. `Set-PveVmConfig`, `Resize-PveVmDisk` and clone have none, so for
|
||||||
|
them the choice is retry or nothing.
|
||||||
|
|
||||||
|
The window is 45 s because `qm cleanup` holds the flock while polling `vm_running_locally` for
|
||||||
|
up to 30 s, and each rejected attempt first burns PVE's own 10 s `lock_config` timeout. It bounds
|
||||||
|
when a *new* attempt may start, not total wall clock: an attempt beginning just inside the window
|
||||||
|
still runs to its own conclusion, so the real ceiling is roughly one attempt longer.
|
||||||
|
|
||||||
|
Two consequences are deliberate, and both are load-bearing enough to state rather than discover:
|
||||||
|
|
||||||
|
- **`-Timeout` does not bound the retry.** It is documented as the budget for the status
|
||||||
|
transition, and `WaitForStatusTransition` starts counting it only after the operation's task
|
||||||
|
completes. Binding the retry to it would defeat the fix at exactly the values that need it —
|
||||||
|
`Reset-PveVm -Wait -Timeout 30` needed ~31 s of retrying in the run that verified this change.
|
||||||
|
- **The two seams nest.** A cmdlet operation rejected synchronously burns the HTTP layer's window
|
||||||
|
inside `InvokeGuestTask`'s. The overlap costs a longer wait before the same failure, never a
|
||||||
|
different outcome, so it is not worth threading a shared budget through both layers.
|
||||||
|
|
||||||
|
CI never showed this. Runs 189–200 were green because the CI client is fast enough to win the
|
||||||
|
race. It reproduces on a client roughly 40% slower — the CI container image run under Docker
|
||||||
|
Desktop's Rosetta emulation on Apple Silicon — which failed `Should hard-reset a running VM`,
|
||||||
|
`Should clone a VM` and `Should resize a VM disk (Resize-PveVmDisk)` on the same commit CI
|
||||||
|
passed. **A green CI run is not evidence about this class of bug.**
|
||||||
|
|
||||||
|
### Not yet adopted
|
||||||
|
`InvokeGuestTask` is the correct seam for every cmdlet that issues a guest operation and waits
|
||||||
|
on its task. `Remove-PveVm`, `Move-PveVm`, the snapshot and template cmdlets, and the container
|
||||||
|
equivalents still call `TaskService.WaitForTask` directly and remain exposed to the same race.
|
||||||
|
They adopt the helper as they are next touched.
|
||||||
|
|
||||||
|
### Anti-pattern (do not reintroduce)
|
||||||
|
```csharp
|
||||||
|
// NEVER try to observe the flock — PVE does not expose it in status/current or anywhere else
|
||||||
|
if (!snapshot.Locked)
|
||||||
|
return task; // reads the config `lock:` property; says nothing about the flock
|
||||||
|
```
|
||||||
|
|
||||||
|
### Correct pattern
|
||||||
|
```csharp
|
||||||
|
PveTask Issue() => vmService.CloneVm(session, sourceNode, vmid, newid, name, targetNode, full);
|
||||||
|
|
||||||
|
var task = Wait.IsPresent
|
||||||
|
? InvokeGuestTask(session, sourceNode, Issue)
|
||||||
|
: Issue();
|
||||||
|
```
|
||||||
|
|||||||
@@ -10,6 +10,7 @@ using System.Threading.Tasks;
|
|||||||
using Newtonsoft.Json.Linq;
|
using Newtonsoft.Json.Linq;
|
||||||
using PSProxmoxVE.Core.Authentication;
|
using PSProxmoxVE.Core.Authentication;
|
||||||
using PSProxmoxVE.Core.Exceptions;
|
using PSProxmoxVE.Core.Exceptions;
|
||||||
|
using PSProxmoxVE.Core.Utilities;
|
||||||
|
|
||||||
using System.Net.Security;
|
using System.Net.Security;
|
||||||
using System.Security.Cryptography.X509Certificates;
|
using System.Security.Cryptography.X509Certificates;
|
||||||
@@ -62,7 +63,7 @@ namespace PSProxmoxVE.Core.Client
|
|||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Creates a bare HTTP client for pre-session use (e.g. initial authentication).
|
/// Creates a bare HTTP client for pre-session use (e.g. initial authentication).
|
||||||
/// No auth headers are added to requests made with this constructor.
|
/// Requests it builds carry no authentication headers.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
internal PveHttpClient(string hostname, int port, bool skipCertificateCheck, TimeSpan? timeout = null)
|
internal PveHttpClient(string hostname, int port, bool skipCertificateCheck, TimeSpan? timeout = null)
|
||||||
{
|
{
|
||||||
@@ -95,8 +96,8 @@ namespace PSProxmoxVE.Core.Client
|
|||||||
/// <returns>Raw JSON response body</returns>
|
/// <returns>Raw JSON response body</returns>
|
||||||
public async Task<string> GetAsync(string resource)
|
public async Task<string> GetAsync(string resource)
|
||||||
{
|
{
|
||||||
var request = BuildRequest(HttpMethod.Get, resource);
|
return await SendAsync(() => BuildRequest(HttpMethod.Get, resource), resource, "GET")
|
||||||
return await SendAsync(request, resource, "GET").ConfigureAwait(false);
|
.ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>Performs a POST request against the specified API resource path.</summary>
|
/// <summary>Performs a POST request against the specified API resource path.</summary>
|
||||||
@@ -105,10 +106,13 @@ namespace PSProxmoxVE.Core.Client
|
|||||||
/// <returns>Raw JSON response body</returns>
|
/// <returns>Raw JSON response body</returns>
|
||||||
public async Task<string> PostAsync(string resource, Dictionary<string, string>? data = null)
|
public async Task<string> PostAsync(string resource, Dictionary<string, string>? data = null)
|
||||||
{
|
{
|
||||||
var request = BuildRequest(HttpMethod.Post, resource, mutating: true);
|
return await SendAsync(() =>
|
||||||
if (data != null)
|
{
|
||||||
request.Content = BuildFormContent(data);
|
var request = BuildRequest(HttpMethod.Post, resource, mutating: true);
|
||||||
return await SendAsync(request, resource, "POST").ConfigureAwait(false);
|
if (data != null)
|
||||||
|
request.Content = BuildFormContent(data);
|
||||||
|
return request;
|
||||||
|
}, resource, "POST").ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
@@ -121,9 +125,12 @@ namespace PSProxmoxVE.Core.Client
|
|||||||
public async Task<string> PostAsync(string resource, IEnumerable<KeyValuePair<string, string>> data)
|
public async Task<string> PostAsync(string resource, IEnumerable<KeyValuePair<string, string>> data)
|
||||||
{
|
{
|
||||||
if (data == null) throw new ArgumentNullException(nameof(data));
|
if (data == null) throw new ArgumentNullException(nameof(data));
|
||||||
var request = BuildRequest(HttpMethod.Post, resource, mutating: true);
|
return await SendAsync(() =>
|
||||||
request.Content = BuildFormContent(data);
|
{
|
||||||
return await SendAsync(request, resource, "POST").ConfigureAwait(false);
|
var request = BuildRequest(HttpMethod.Post, resource, mutating: true);
|
||||||
|
request.Content = BuildFormContent(data);
|
||||||
|
return request;
|
||||||
|
}, resource, "POST").ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>Performs a PUT request against the specified API resource path.</summary>
|
/// <summary>Performs a PUT request against the specified API resource path.</summary>
|
||||||
@@ -132,10 +139,13 @@ namespace PSProxmoxVE.Core.Client
|
|||||||
/// <returns>Raw JSON response body</returns>
|
/// <returns>Raw JSON response body</returns>
|
||||||
public async Task<string> PutAsync(string resource, Dictionary<string, string>? data = null)
|
public async Task<string> PutAsync(string resource, Dictionary<string, string>? data = null)
|
||||||
{
|
{
|
||||||
var request = BuildRequest(HttpMethod.Put, resource, mutating: true);
|
return await SendAsync(() =>
|
||||||
if (data != null)
|
{
|
||||||
request.Content = BuildFormContent(data);
|
var request = BuildRequest(HttpMethod.Put, resource, mutating: true);
|
||||||
return await SendAsync(request, resource, "PUT").ConfigureAwait(false);
|
if (data != null)
|
||||||
|
request.Content = BuildFormContent(data);
|
||||||
|
return request;
|
||||||
|
}, resource, "PUT").ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>Performs a DELETE request against the specified API resource path.</summary>
|
/// <summary>Performs a DELETE request against the specified API resource path.</summary>
|
||||||
@@ -143,8 +153,8 @@ namespace PSProxmoxVE.Core.Client
|
|||||||
/// <returns>Raw JSON response body</returns>
|
/// <returns>Raw JSON response body</returns>
|
||||||
public async Task<string> DeleteAsync(string resource)
|
public async Task<string> DeleteAsync(string resource)
|
||||||
{
|
{
|
||||||
var request = BuildRequest(HttpMethod.Delete, resource, mutating: true);
|
return await SendAsync(() => BuildRequest(HttpMethod.Delete, resource, mutating: true), resource, "DELETE")
|
||||||
return await SendAsync(request, resource, "DELETE").ConfigureAwait(false);
|
.ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
// -------------------------------------------------------------------------
|
// -------------------------------------------------------------------------
|
||||||
@@ -331,7 +341,7 @@ namespace PSProxmoxVE.Core.Client
|
|||||||
var request = BuildRequest(HttpMethod.Post, resource, mutating: true);
|
var request = BuildRequest(HttpMethod.Post, resource, mutating: true);
|
||||||
request.Content = multipart;
|
request.Content = multipart;
|
||||||
|
|
||||||
return await SendAsync(request, resource, "POST").ConfigureAwait(false);
|
return await SendOnceAsync(request, resource, "POST").ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
finally
|
finally
|
||||||
{
|
{
|
||||||
@@ -366,7 +376,15 @@ namespace PSProxmoxVE.Core.Client
|
|||||||
return request;
|
return request;
|
||||||
}
|
}
|
||||||
|
|
||||||
private async Task<string> SendAsync(HttpRequestMessage request, string resource, string httpMethod)
|
/// <summary>
|
||||||
|
/// Sends a request, rebuilding it from <paramref name="buildRequest"/> for each attempt
|
||||||
|
/// while PVE rejects it for a guest's config flock. An <see cref="HttpRequestMessage"/>
|
||||||
|
/// cannot be resent, which is why this takes a factory rather than a request.
|
||||||
|
/// </summary>
|
||||||
|
private Task<string> SendAsync(Func<HttpRequestMessage> buildRequest, string resource, string httpMethod) =>
|
||||||
|
GuestLockRetry.ExecuteAsync(() => SendOnceAsync(buildRequest(), resource, httpMethod));
|
||||||
|
|
||||||
|
private async Task<string> SendOnceAsync(HttpRequestMessage request, string resource, string httpMethod)
|
||||||
{
|
{
|
||||||
HttpResponseMessage response;
|
HttpResponseMessage response;
|
||||||
try
|
try
|
||||||
|
|||||||
@@ -15,6 +15,13 @@ namespace PSProxmoxVE.Core.Exceptions
|
|||||||
/// <summary>The HTTP method used for the request (GET, POST, PUT, DELETE).</summary>
|
/// <summary>The HTTP method used for the request (GET, POST, PUT, DELETE).</summary>
|
||||||
public string HttpMethod { get; }
|
public string HttpMethod { get; }
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The error text PVE returned, without the status/resource prefix that
|
||||||
|
/// <see cref="Exception.Message"/> carries. Callers that match on what PVE said —
|
||||||
|
/// rather than on how this exception renders it — must read this.
|
||||||
|
/// </summary>
|
||||||
|
public string ApiMessage { get; }
|
||||||
|
|
||||||
/// <summary>Initializes a new instance for a failed PVE API request.</summary>
|
/// <summary>Initializes a new instance for a failed PVE API request.</summary>
|
||||||
/// <param name="statusCode">The HTTP status code returned.</param>
|
/// <param name="statusCode">The HTTP status code returned.</param>
|
||||||
/// <param name="message">The error message from the API.</param>
|
/// <param name="message">The error message from the API.</param>
|
||||||
@@ -26,6 +33,7 @@ namespace PSProxmoxVE.Core.Exceptions
|
|||||||
StatusCode = statusCode;
|
StatusCode = statusCode;
|
||||||
Resource = resource;
|
Resource = resource;
|
||||||
HttpMethod = httpMethod;
|
HttpMethod = httpMethod;
|
||||||
|
ApiMessage = message;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>Initializes a new instance for a failed PVE API request, with an inner exception.</summary>
|
/// <summary>Initializes a new instance for a failed PVE API request, with an inner exception.</summary>
|
||||||
@@ -40,6 +48,7 @@ namespace PSProxmoxVE.Core.Exceptions
|
|||||||
StatusCode = statusCode;
|
StatusCode = statusCode;
|
||||||
Resource = resource;
|
Resource = resource;
|
||||||
HttpMethod = httpMethod;
|
HttpMethod = httpMethod;
|
||||||
|
ApiMessage = message;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,103 @@
|
|||||||
|
using System;
|
||||||
|
using System.Diagnostics;
|
||||||
|
using System.Text.RegularExpressions;
|
||||||
|
using System.Threading;
|
||||||
|
using System.Threading.Tasks;
|
||||||
|
using PSProxmoxVE.Core.Exceptions;
|
||||||
|
|
||||||
|
namespace PSProxmoxVE.Core.Utilities
|
||||||
|
{
|
||||||
|
/// <summary>
|
||||||
|
/// Retries an operation PVE rejected because it could not acquire a guest's config
|
||||||
|
/// flock (<c>/var/lock/qemu-server/lock-<vmid>.conf</c> for VMs,
|
||||||
|
/// <c>/run/lock/lxc/pve-config-<vmid>.lock</c> for containers).
|
||||||
|
///
|
||||||
|
/// That flock is held by <c>qm cleanup</c> after a guest stops and is not exposed
|
||||||
|
/// through the API in any form, so a caller cannot wait for it — only retry past it.
|
||||||
|
/// </summary>
|
||||||
|
public static class GuestLockRetry
|
||||||
|
{
|
||||||
|
/// <summary>
|
||||||
|
/// How long <see cref="Execute{T}"/> and <see cref="ExecuteAsync{T}"/> keep retrying.
|
||||||
|
/// <c>qm cleanup</c> polls <c>vm_running_locally</c> for up to 30s while holding the
|
||||||
|
/// flock, and each rejected attempt first burns PVE's own 10s <c>lock_config</c> timeout.
|
||||||
|
/// </summary>
|
||||||
|
public static readonly TimeSpan DefaultWindow = TimeSpan.FromSeconds(45);
|
||||||
|
|
||||||
|
private static readonly TimeSpan RetryInterval = TimeSpan.FromSeconds(2);
|
||||||
|
|
||||||
|
// Anchored, and specific to the two guest lock paths. `PVE::Tools::lock_file` emits this
|
||||||
|
// same wording for storage, LVM, HA and firewall locks, none of which carry the
|
||||||
|
// reissue-safety guarantee below. The anchor is what separates a failure to *enter*
|
||||||
|
// lock_config from one PVE prefixed with its own context ("clone failed: ..."), which
|
||||||
|
// means the worker had already done work.
|
||||||
|
private static readonly Regex GuestLockTimeout = new Regex(
|
||||||
|
@"^can't lock file '(?:/var/lock/qemu-server/lock-\d+\.conf|/run/lock/lxc/pve-config-\d+\.lock)' - got timeout",
|
||||||
|
RegexOptions.IgnoreCase | RegexOptions.CultureInvariant);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// True when <paramref name="ex"/> reports PVE failing to enter <c>lock_config</c> for a
|
||||||
|
/// guest. That is raised before the operation performs any work, so a call failing this
|
||||||
|
/// way is known not to have run and is safe to reissue.
|
||||||
|
///
|
||||||
|
/// Matched against what PVE actually said — <see cref="PveTaskFailedException.ExitStatus"/>
|
||||||
|
/// and <see cref="PveApiException.ApiMessage"/> — never against the composed
|
||||||
|
/// <see cref="Exception.Message"/>, whose prefix would defeat the anchor.
|
||||||
|
/// </summary>
|
||||||
|
/// <param name="ex">The exception to classify.</param>
|
||||||
|
public static bool IsLockTimeout(Exception ex) => ex switch
|
||||||
|
{
|
||||||
|
PveTaskFailedException task => GuestLockTimeout.IsMatch((task.ExitStatus ?? string.Empty).Trim()),
|
||||||
|
PveApiException api => GuestLockTimeout.IsMatch((api.ApiMessage ?? string.Empty).Trim()),
|
||||||
|
_ => false,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Runs <paramref name="operation"/>, reissuing it while it fails on the guest config
|
||||||
|
/// flock and <paramref name="window"/> has not elapsed. Any other exception propagates
|
||||||
|
/// on the first attempt.
|
||||||
|
/// </summary>
|
||||||
|
/// <param name="operation">The operation to run.</param>
|
||||||
|
/// <param name="window">Retry budget. Defaults to <see cref="DefaultWindow"/>.</param>
|
||||||
|
public static T Execute<T>(Func<T> operation, TimeSpan? window = null)
|
||||||
|
{
|
||||||
|
if (operation == null) throw new ArgumentNullException(nameof(operation));
|
||||||
|
|
||||||
|
var budget = window ?? DefaultWindow;
|
||||||
|
var elapsed = Stopwatch.StartNew();
|
||||||
|
while (true)
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
return operation();
|
||||||
|
}
|
||||||
|
catch (Exception ex) when (IsLockTimeout(ex) && elapsed.Elapsed < budget)
|
||||||
|
{
|
||||||
|
Thread.Sleep(RetryInterval);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Asynchronous counterpart of <see cref="Execute{T}"/>.</summary>
|
||||||
|
/// <param name="operation">The operation to run.</param>
|
||||||
|
/// <param name="window">Retry budget. Defaults to <see cref="DefaultWindow"/>.</param>
|
||||||
|
public static async Task<T> ExecuteAsync<T>(Func<Task<T>> operation, TimeSpan? window = null)
|
||||||
|
{
|
||||||
|
if (operation == null) throw new ArgumentNullException(nameof(operation));
|
||||||
|
|
||||||
|
var budget = window ?? DefaultWindow;
|
||||||
|
var elapsed = Stopwatch.StartNew();
|
||||||
|
while (true)
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
return await operation().ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
catch (Exception ex) when (IsLockTimeout(ex) && elapsed.Elapsed < budget)
|
||||||
|
{
|
||||||
|
await Task.Delay(RetryInterval).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -18,8 +18,9 @@ namespace PSProxmoxVE.Core.Utilities
|
|||||||
/// StatusMatched: the guest reports <paramref name="expectedStatus"/>. qmpstatus is
|
/// StatusMatched: the guest reports <paramref name="expectedStatus"/>. qmpstatus is
|
||||||
/// preferred over status when present, because PVE reports status=running with
|
/// preferred over status when present, because PVE reports status=running with
|
||||||
/// qmpstatus=paused for a suspended VM.
|
/// qmpstatus=paused for a suspended VM.
|
||||||
/// Locked: the guest config still carries a lock, so the next API call against it
|
/// Locked: the guest config carries a `lock:` property (backup, clone, migrate,
|
||||||
/// would fail to acquire the lock file.
|
/// snapshot). This is not the /var/lock/qemu-server flock, which PVE does not expose
|
||||||
|
/// through status/current or any other endpoint — see DECISIONS.md D015 and D020.
|
||||||
/// </returns>
|
/// </returns>
|
||||||
public static (bool StatusMatched, bool Locked) Evaluate(string json, string expectedStatus)
|
public static (bool StatusMatched, bool Locked) Evaluate(string json, string expectedStatus)
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -56,17 +56,17 @@ namespace PSProxmoxVE.Cmdlets.Containers
|
|||||||
|
|
||||||
WriteVerbose($"Restarting container {VmId} on node '{Node}'...");
|
WriteVerbose($"Restarting container {VmId} on node '{Node}'...");
|
||||||
|
|
||||||
// Graceful shutdown
|
PveTask Shutdown() => containerService.ShutdownContainer(session, Node, VmId, Timeout);
|
||||||
var shutdownTask = containerService.ShutdownContainer(session, Node, VmId, Timeout);
|
PveTask Start() => containerService.StartContainer(session, Node, VmId);
|
||||||
|
|
||||||
if (Wait.IsPresent)
|
if (Wait.IsPresent)
|
||||||
WaitForStatusTransition(session, Node, shutdownTask, VmId, "stopped", Timeout, isContainer: true);
|
WaitForStatusTransition(session, Node, Shutdown, VmId, "stopped", Timeout, isContainer: true);
|
||||||
|
else
|
||||||
|
Shutdown();
|
||||||
|
|
||||||
// Start
|
var startTask = Wait.IsPresent
|
||||||
var startTask = containerService.StartContainer(session, Node, VmId);
|
? WaitForStatusTransition(session, Node, Start, VmId, "running", Timeout, isContainer: true)
|
||||||
|
: Start();
|
||||||
if (Wait.IsPresent)
|
|
||||||
startTask = WaitForStatusTransition(session, Node, startTask, VmId, "running", Timeout, isContainer: true);
|
|
||||||
|
|
||||||
WriteObject(startTask);
|
WriteObject(startTask);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -50,12 +50,11 @@ namespace PSProxmoxVE.Cmdlets.Containers
|
|||||||
var containerService = new ContainerService();
|
var containerService = new ContainerService();
|
||||||
|
|
||||||
WriteVerbose($"Starting container {VmId} on node '{Node}'...");
|
WriteVerbose($"Starting container {VmId} on node '{Node}'...");
|
||||||
var task = containerService.StartContainer(session, Node, VmId);
|
PveTask Issue() => containerService.StartContainer(session, Node, VmId);
|
||||||
|
|
||||||
if (Wait.IsPresent)
|
var task = Wait.IsPresent
|
||||||
{
|
? WaitForStatusTransition(session, Node, Issue, VmId, "running", Timeout, isContainer: true)
|
||||||
task = WaitForStatusTransition(session, Node, task, VmId, "running", Timeout, isContainer: true);
|
: Issue();
|
||||||
}
|
|
||||||
|
|
||||||
WriteObject(task);
|
WriteObject(task);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -52,12 +52,11 @@ namespace PSProxmoxVE.Cmdlets.Containers
|
|||||||
var containerService = new ContainerService();
|
var containerService = new ContainerService();
|
||||||
|
|
||||||
WriteVerbose($"Stopping container {VmId} on node '{Node}'...");
|
WriteVerbose($"Stopping container {VmId} on node '{Node}'...");
|
||||||
var task = containerService.StopContainer(session, Node, VmId);
|
PveTask Issue() => containerService.StopContainer(session, Node, VmId);
|
||||||
|
|
||||||
if (Wait.IsPresent)
|
var task = Wait.IsPresent
|
||||||
{
|
? WaitForStatusTransition(session, Node, Issue, VmId, "stopped", Timeout, isContainer: true)
|
||||||
task = WaitForStatusTransition(session, Node, task, VmId, "stopped", Timeout, isContainer: true);
|
: Issue();
|
||||||
}
|
|
||||||
|
|
||||||
WriteObject(task);
|
WriteObject(task);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -99,7 +99,10 @@ namespace PSProxmoxVE.Cmdlets
|
|||||||
/// </summary>
|
/// </summary>
|
||||||
/// <param name="session">The authenticated PVE session.</param>
|
/// <param name="session">The authenticated PVE session.</param>
|
||||||
/// <param name="node">The cluster node name.</param>
|
/// <param name="node">The cluster node name.</param>
|
||||||
/// <param name="task">The task returned by the lifecycle API call.</param>
|
/// <param name="issueOperation">
|
||||||
|
/// Issues the lifecycle API call. Invoked again on each retry, so it must be safe to
|
||||||
|
/// repeat — see <see cref="InvokeGuestTask"/>.
|
||||||
|
/// </param>
|
||||||
/// <param name="vmid">The VM or container ID to poll.</param>
|
/// <param name="vmid">The VM or container ID to poll.</param>
|
||||||
/// <param name="expectedStatus">The expected status string (e.g. "running", "stopped", "paused").</param>
|
/// <param name="expectedStatus">The expected status string (e.g. "running", "stopped", "paused").</param>
|
||||||
/// <param name="timeoutSeconds">Maximum seconds to wait for the status transition. Default 60.</param>
|
/// <param name="timeoutSeconds">Maximum seconds to wait for the status transition. Default 60.</param>
|
||||||
@@ -108,19 +111,13 @@ namespace PSProxmoxVE.Cmdlets
|
|||||||
protected PveTask WaitForStatusTransition(
|
protected PveTask WaitForStatusTransition(
|
||||||
PveSession session,
|
PveSession session,
|
||||||
string node,
|
string node,
|
||||||
PveTask task,
|
Func<PveTask> issueOperation,
|
||||||
int vmid,
|
int vmid,
|
||||||
string expectedStatus,
|
string expectedStatus,
|
||||||
int timeoutSeconds = 60,
|
int timeoutSeconds = 60,
|
||||||
bool isContainer = false)
|
bool isContainer = false)
|
||||||
{
|
{
|
||||||
var taskService = new TaskService();
|
var task = InvokeGuestTask(session, node, issueOperation);
|
||||||
|
|
||||||
// First wait for the PVE task to complete
|
|
||||||
if (!string.IsNullOrEmpty(task.Upid))
|
|
||||||
{
|
|
||||||
task = taskService.WaitForTask(session, node, task.Upid, null, null, null);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Then poll status/current until VM/container reaches the expected status.
|
// Then poll status/current until VM/container reaches the expected status.
|
||||||
// We query the status/current endpoint directly instead of the list endpoint
|
// We query the status/current endpoint directly instead of the list endpoint
|
||||||
@@ -141,15 +138,11 @@ namespace PSProxmoxVE.Cmdlets
|
|||||||
var snapshot = GuestStatusSnapshot.Evaluate(json, expectedStatus);
|
var snapshot = GuestStatusSnapshot.Evaluate(json, expectedStatus);
|
||||||
lastMatched = snapshot.StatusMatched;
|
lastMatched = snapshot.StatusMatched;
|
||||||
|
|
||||||
if (snapshot.StatusMatched)
|
// snapshot.Locked is the config `lock:` property (backup, clone, migrate,
|
||||||
{
|
// snapshot) — not the /var/lock/qemu-server flock, which PVE does not
|
||||||
// PVE reports the target status before the operation releases the
|
// expose. The flock race is handled by retrying, not by waiting.
|
||||||
// config lock. A caller that issues its next request inside that
|
if (snapshot.StatusMatched && !snapshot.Locked)
|
||||||
// window gets "can't lock file '/var/lock/qemu-server/lock-<vmid>.conf'
|
return task;
|
||||||
// - got timeout" from its own API call.
|
|
||||||
if (!snapshot.Locked)
|
|
||||||
return task;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
catch (Exception ex) when (ex is not OutOfMemoryException and not StackOverflowException)
|
catch (Exception ex) when (ex is not OutOfMemoryException and not StackOverflowException)
|
||||||
{
|
{
|
||||||
@@ -169,6 +162,33 @@ namespace PSProxmoxVE.Cmdlets
|
|||||||
TimeSpan.FromSeconds(timeoutSeconds));
|
TimeSpan.FromSeconds(timeoutSeconds));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Issues a guest operation and waits for the task it returns, reissuing the pair while
|
||||||
|
/// PVE rejects it for the guest's config flock.
|
||||||
|
///
|
||||||
|
/// PVE takes that flock inside the worker for most guest operations, so the failure
|
||||||
|
/// surfaces as a failed task rather than a failed request and cannot be retried at the
|
||||||
|
/// HTTP layer. <c>lock_config</c> raises it before doing any work, so a reissue repeats
|
||||||
|
/// nothing.
|
||||||
|
/// </summary>
|
||||||
|
/// <param name="session">The authenticated PVE session.</param>
|
||||||
|
/// <param name="node">The node the task runs on.</param>
|
||||||
|
/// <param name="issueOperation">Issues the API call; invoked again on each retry.</param>
|
||||||
|
/// <returns>The completed task, or the issued task when the call returned no UPID.</returns>
|
||||||
|
protected PveTask InvokeGuestTask(PveSession session, string node, Func<PveTask> issueOperation)
|
||||||
|
{
|
||||||
|
if (issueOperation == null) throw new ArgumentNullException(nameof(issueOperation));
|
||||||
|
|
||||||
|
var taskService = new TaskService();
|
||||||
|
return GuestLockRetry.Execute(() =>
|
||||||
|
{
|
||||||
|
var task = issueOperation();
|
||||||
|
return string.IsNullOrEmpty(task.Upid)
|
||||||
|
? task
|
||||||
|
: taskService.WaitForTask(session, node, task.Upid, null, null, null);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Extracts the node name from a UPID string (format: UPID:node:...).
|
/// Extracts the node name from a UPID string (format: UPID:node:...).
|
||||||
/// Falls back to <paramref name="fallback"/> if the UPID is empty or cannot be parsed.
|
/// Falls back to <paramref name="fallback"/> if the UPID is empty or cannot be parsed.
|
||||||
|
|||||||
@@ -81,13 +81,11 @@ namespace PSProxmoxVE.Cmdlets.Vms
|
|||||||
|
|
||||||
WriteVerbose($"Cloning VM {VmId}...");
|
WriteVerbose($"Cloning VM {VmId}...");
|
||||||
var newid = NewVmId ?? 0;
|
var newid = NewVmId ?? 0;
|
||||||
var task = vmService.CloneVm(session, SourceNode, VmId, newid, NewName, TargetNode, Full.IsPresent);
|
PveTask Issue() => vmService.CloneVm(session, SourceNode, VmId, newid, NewName, TargetNode, Full.IsPresent);
|
||||||
|
|
||||||
if (Wait.IsPresent)
|
var task = Wait.IsPresent
|
||||||
{
|
? InvokeGuestTask(session, SourceNode, Issue)
|
||||||
var taskService = new TaskService();
|
: Issue();
|
||||||
task = taskService.WaitForTask(session, SourceNode, task.Upid, null, null, null);
|
|
||||||
}
|
|
||||||
|
|
||||||
WriteObject(task);
|
WriteObject(task);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -52,12 +52,11 @@ namespace PSProxmoxVE.Cmdlets.Vms
|
|||||||
var vmService = new VmService();
|
var vmService = new VmService();
|
||||||
|
|
||||||
WriteVerbose($"Resetting VM {VmId} on node '{Node}'...");
|
WriteVerbose($"Resetting VM {VmId} on node '{Node}'...");
|
||||||
var task = vmService.ResetVm(session, Node, VmId);
|
PveTask Issue() => vmService.ResetVm(session, Node, VmId);
|
||||||
|
|
||||||
if (Wait.IsPresent)
|
var task = Wait.IsPresent
|
||||||
{
|
? WaitForStatusTransition(session, Node, Issue, VmId, "running", Timeout)
|
||||||
task = WaitForStatusTransition(session, Node, task, VmId, "running", Timeout);
|
: Issue();
|
||||||
}
|
|
||||||
|
|
||||||
WriteObject(task);
|
WriteObject(task);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -65,13 +65,11 @@ namespace PSProxmoxVE.Cmdlets.Vms
|
|||||||
var vmService = new VmService();
|
var vmService = new VmService();
|
||||||
|
|
||||||
WriteVerbose($"Resizing disk '{Disk}' on VM {VmId}...");
|
WriteVerbose($"Resizing disk '{Disk}' on VM {VmId}...");
|
||||||
var task = vmService.ResizeDisk(session, Node, VmId, Disk, Size);
|
PveTask Issue() => vmService.ResizeDisk(session, Node, VmId, Disk, Size);
|
||||||
|
|
||||||
if (Wait.IsPresent)
|
var task = Wait.IsPresent
|
||||||
{
|
? InvokeGuestTask(session, Node, Issue)
|
||||||
var taskService = new TaskService();
|
: Issue();
|
||||||
task = taskService.WaitForTask(session, Node, task.Upid, null, null, null);
|
|
||||||
}
|
|
||||||
|
|
||||||
WriteObject(task);
|
WriteObject(task);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -57,10 +57,11 @@ namespace PSProxmoxVE.Cmdlets.Vms
|
|||||||
|
|
||||||
WriteVerbose($"Restarting VM {VmId} on node '{Node}'...");
|
WriteVerbose($"Restarting VM {VmId} on node '{Node}'...");
|
||||||
|
|
||||||
var task = vmService.RebootVm(session, Node, VmId, Timeout);
|
PveTask Issue() => vmService.RebootVm(session, Node, VmId, Timeout);
|
||||||
|
|
||||||
if (Wait.IsPresent)
|
var task = Wait.IsPresent
|
||||||
task = WaitForStatusTransition(session, Node, task, VmId, "running", Timeout);
|
? WaitForStatusTransition(session, Node, Issue, VmId, "running", Timeout)
|
||||||
|
: Issue();
|
||||||
|
|
||||||
WriteObject(task);
|
WriteObject(task);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -50,12 +50,11 @@ namespace PSProxmoxVE.Cmdlets.Vms
|
|||||||
var vmService = new VmService();
|
var vmService = new VmService();
|
||||||
|
|
||||||
WriteVerbose($"Resuming VM {VmId} on node '{Node}'...");
|
WriteVerbose($"Resuming VM {VmId} on node '{Node}'...");
|
||||||
var task = vmService.ResumeVm(session, Node, VmId);
|
PveTask Issue() => vmService.ResumeVm(session, Node, VmId);
|
||||||
|
|
||||||
if (Wait.IsPresent)
|
var task = Wait.IsPresent
|
||||||
{
|
? WaitForStatusTransition(session, Node, Issue, VmId, "running", Timeout)
|
||||||
task = WaitForStatusTransition(session, Node, task, VmId, "running", Timeout);
|
: Issue();
|
||||||
}
|
|
||||||
|
|
||||||
WriteObject(task);
|
WriteObject(task);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -50,12 +50,11 @@ namespace PSProxmoxVE.Cmdlets.Vms
|
|||||||
var vmService = new VmService();
|
var vmService = new VmService();
|
||||||
|
|
||||||
WriteVerbose($"Starting VM {VmId} on node '{Node}'...");
|
WriteVerbose($"Starting VM {VmId} on node '{Node}'...");
|
||||||
var task = vmService.StartVm(session, Node, VmId);
|
PveTask Issue() => vmService.StartVm(session, Node, VmId);
|
||||||
|
|
||||||
if (Wait.IsPresent)
|
var task = Wait.IsPresent
|
||||||
{
|
? WaitForStatusTransition(session, Node, Issue, VmId, "running", Timeout)
|
||||||
task = WaitForStatusTransition(session, Node, task, VmId, "running", Timeout);
|
: Issue();
|
||||||
}
|
|
||||||
|
|
||||||
WriteObject(task);
|
WriteObject(task);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -52,12 +52,11 @@ namespace PSProxmoxVE.Cmdlets.Vms
|
|||||||
var vmService = new VmService();
|
var vmService = new VmService();
|
||||||
|
|
||||||
WriteVerbose($"Stopping VM {VmId} on node '{Node}'...");
|
WriteVerbose($"Stopping VM {VmId} on node '{Node}'...");
|
||||||
var task = vmService.StopVm(session, Node, VmId);
|
PveTask Issue() => vmService.StopVm(session, Node, VmId);
|
||||||
|
|
||||||
if (Wait.IsPresent)
|
var task = Wait.IsPresent
|
||||||
{
|
? WaitForStatusTransition(session, Node, Issue, VmId, "stopped", Timeout)
|
||||||
task = WaitForStatusTransition(session, Node, task, VmId, "stopped", Timeout);
|
: Issue();
|
||||||
}
|
|
||||||
|
|
||||||
WriteObject(task);
|
WriteObject(task);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -51,12 +51,11 @@ namespace PSProxmoxVE.Cmdlets.Vms
|
|||||||
var vmService = new VmService();
|
var vmService = new VmService();
|
||||||
|
|
||||||
WriteVerbose($"Suspending VM {VmId} on node '{Node}'...");
|
WriteVerbose($"Suspending VM {VmId} on node '{Node}'...");
|
||||||
var task = vmService.SuspendVm(session, Node, VmId);
|
PveTask Issue() => vmService.SuspendVm(session, Node, VmId);
|
||||||
|
|
||||||
if (Wait.IsPresent)
|
var task = Wait.IsPresent
|
||||||
{
|
? WaitForStatusTransition(session, Node, Issue, VmId, "paused", Timeout)
|
||||||
task = WaitForStatusTransition(session, Node, task, VmId, "paused", Timeout);
|
: Issue();
|
||||||
}
|
|
||||||
|
|
||||||
WriteObject(task);
|
WriteObject(task);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,143 @@
|
|||||||
|
using System;
|
||||||
|
using System.Collections.Generic;
|
||||||
|
using System.Net;
|
||||||
|
using System.Net.Http;
|
||||||
|
using System.Reflection;
|
||||||
|
using System.Threading;
|
||||||
|
using System.Threading.Tasks;
|
||||||
|
using PSProxmoxVE.Core.Authentication;
|
||||||
|
using PSProxmoxVE.Core.Client;
|
||||||
|
using PSProxmoxVE.Core.Exceptions;
|
||||||
|
using Xunit;
|
||||||
|
|
||||||
|
namespace PSProxmoxVE.Core.Tests.Client
|
||||||
|
{
|
||||||
|
public class PveHttpClientLockRetryTests
|
||||||
|
{
|
||||||
|
private const string LockTimeoutBody =
|
||||||
|
"{\"message\":\"can't lock file '/var/lock/qemu-server/lock-100.conf' - got timeout\"}";
|
||||||
|
|
||||||
|
private static void SetInnerHttpClient(PveHttpClient client, HttpClient newInner)
|
||||||
|
{
|
||||||
|
var field = typeof(PveHttpClient).GetField("_httpClient",
|
||||||
|
BindingFlags.Instance | BindingFlags.NonPublic)!;
|
||||||
|
((HttpClient)field.GetValue(client)!).Dispose();
|
||||||
|
field.SetValue(client, newInner);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static (PveHttpClient client, ScriptedHandler handler) NewClient(
|
||||||
|
params (HttpStatusCode status, string body)[] responses)
|
||||||
|
{
|
||||||
|
var session = new PveSession("pve.example.com", 8006, false,
|
||||||
|
"root@pam!token=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee");
|
||||||
|
var client = new PveHttpClient(session);
|
||||||
|
var handler = new ScriptedHandler(responses);
|
||||||
|
SetInnerHttpClient(client, new HttpClient(handler));
|
||||||
|
return (client, handler);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static Dictionary<string, string> ConfigBody() =>
|
||||||
|
new Dictionary<string, string> { ["scsi0"] = "local-lvm:1" };
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task PutAsync_ReissuesTheRequestWhilePveReportsTheGuestFlock()
|
||||||
|
{
|
||||||
|
var (client, handler) = NewClient(
|
||||||
|
(HttpStatusCode.InternalServerError, LockTimeoutBody),
|
||||||
|
(HttpStatusCode.InternalServerError, LockTimeoutBody),
|
||||||
|
(HttpStatusCode.OK, "{\"data\":null}"));
|
||||||
|
|
||||||
|
using (client)
|
||||||
|
{
|
||||||
|
var result = await client.PutAsync("nodes/pve9a/qemu/100/config", ConfigBody());
|
||||||
|
Assert.Equal("{\"data\":null}", result);
|
||||||
|
}
|
||||||
|
|
||||||
|
Assert.Equal(3, handler.Bodies.Count);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task PutAsync_RebuildsTheRequestSoEveryAttemptCarriesTheSameBody()
|
||||||
|
{
|
||||||
|
var (client, handler) = NewClient(
|
||||||
|
(HttpStatusCode.InternalServerError, LockTimeoutBody),
|
||||||
|
(HttpStatusCode.OK, "{\"data\":null}"));
|
||||||
|
|
||||||
|
using (client)
|
||||||
|
{
|
||||||
|
await client.PutAsync("nodes/pve9a/qemu/100/config", ConfigBody());
|
||||||
|
}
|
||||||
|
|
||||||
|
Assert.Equal(2, handler.Bodies.Count);
|
||||||
|
Assert.Equal("scsi0=local-lvm:1", handler.Bodies[0]);
|
||||||
|
Assert.Equal(handler.Bodies[0], handler.Bodies[1]);
|
||||||
|
Assert.All(handler.Methods, m => Assert.Equal(HttpMethod.Put, m));
|
||||||
|
Assert.Equal(handler.Uris[0], handler.Uris[1]);
|
||||||
|
Assert.EndsWith("nodes/pve9a/qemu/100/config", handler.Uris[0]);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task PostAsync_DoesNotReissueApiErrorsThatAreNotTheFlock()
|
||||||
|
{
|
||||||
|
var (client, handler) = NewClient(
|
||||||
|
(HttpStatusCode.InternalServerError, "{\"message\":\"VM 100 not running\"}"),
|
||||||
|
(HttpStatusCode.OK, "{\"data\":null}"));
|
||||||
|
|
||||||
|
using (client)
|
||||||
|
{
|
||||||
|
var ex = await Assert.ThrowsAsync<PveApiException>(
|
||||||
|
() => client.PostAsync("nodes/pve9a/qemu/100/status/reset"));
|
||||||
|
Assert.Contains("VM 100 not running", ex.Message);
|
||||||
|
}
|
||||||
|
|
||||||
|
Assert.Single(handler.Bodies);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task GetAsync_ReissuesWithoutCarryingContent()
|
||||||
|
{
|
||||||
|
var (client, handler) = NewClient(
|
||||||
|
(HttpStatusCode.InternalServerError, LockTimeoutBody),
|
||||||
|
(HttpStatusCode.OK, "{\"data\":{}}"));
|
||||||
|
|
||||||
|
using (client)
|
||||||
|
{
|
||||||
|
await client.GetAsync("nodes/pve9a/qemu/100/status/current");
|
||||||
|
}
|
||||||
|
|
||||||
|
Assert.Equal(2, handler.Bodies.Count);
|
||||||
|
Assert.All(handler.Bodies, b => Assert.Equal(string.Empty, b));
|
||||||
|
}
|
||||||
|
|
||||||
|
private sealed class ScriptedHandler : HttpMessageHandler
|
||||||
|
{
|
||||||
|
private readonly (HttpStatusCode status, string body)[] _responses;
|
||||||
|
private int _index;
|
||||||
|
|
||||||
|
public List<string> Bodies { get; } = new List<string>();
|
||||||
|
public List<HttpMethod> Methods { get; } = new List<HttpMethod>();
|
||||||
|
public List<string> Uris { get; } = new List<string>();
|
||||||
|
|
||||||
|
public ScriptedHandler((HttpStatusCode status, string body)[] responses)
|
||||||
|
{
|
||||||
|
_responses = responses;
|
||||||
|
}
|
||||||
|
|
||||||
|
protected override async Task<HttpResponseMessage> SendAsync(
|
||||||
|
HttpRequestMessage request, CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
Bodies.Add(request.Content == null
|
||||||
|
? string.Empty
|
||||||
|
: await request.Content.ReadAsStringAsync().ConfigureAwait(false));
|
||||||
|
Methods.Add(request.Method);
|
||||||
|
Uris.Add(request.RequestUri!.ToString());
|
||||||
|
|
||||||
|
if (_index >= _responses.Length)
|
||||||
|
throw new InvalidOperationException("ScriptedHandler ran out of responses.");
|
||||||
|
|
||||||
|
var (status, body) = _responses[_index++];
|
||||||
|
return new HttpResponseMessage(status) { Content = new StringContent(body) };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,160 @@
|
|||||||
|
using System;
|
||||||
|
using System.Net;
|
||||||
|
using System.Threading.Tasks;
|
||||||
|
using PSProxmoxVE.Core.Exceptions;
|
||||||
|
using PSProxmoxVE.Core.Utilities;
|
||||||
|
using Xunit;
|
||||||
|
|
||||||
|
namespace PSProxmoxVE.Core.Tests.Utilities
|
||||||
|
{
|
||||||
|
public class GuestLockRetryTests
|
||||||
|
{
|
||||||
|
private const string VmLockError =
|
||||||
|
"can't lock file '/var/lock/qemu-server/lock-100.conf' - got timeout";
|
||||||
|
|
||||||
|
private const string LxcLockError =
|
||||||
|
"can't lock file '/run/lock/lxc/pve-config-100.lock' - got timeout";
|
||||||
|
|
||||||
|
private static PveApiException ApiError(string message) =>
|
||||||
|
new PveApiException(HttpStatusCode.InternalServerError, message, "nodes/pve9a/qemu/100/config", "PUT");
|
||||||
|
|
||||||
|
private static PveTaskFailedException TaskError(string exitStatus) =>
|
||||||
|
new PveTaskFailedException("UPID:pve9a:00000001:qmreset:100:root@pam:", exitStatus);
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void IsLockTimeout_MatchesTheVmFlockErrorFromBothSurfaces()
|
||||||
|
{
|
||||||
|
Assert.True(GuestLockRetry.IsLockTimeout(ApiError(VmLockError)));
|
||||||
|
Assert.True(GuestLockRetry.IsLockTimeout(TaskError(VmLockError)));
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void IsLockTimeout_MatchesTheContainerFlockError()
|
||||||
|
{
|
||||||
|
Assert.True(GuestLockRetry.IsLockTimeout(TaskError(LxcLockError)));
|
||||||
|
}
|
||||||
|
|
||||||
|
[Theory]
|
||||||
|
[InlineData("VM 100 not running")]
|
||||||
|
[InlineData("can't lock file '/var/lock/qemu-server/lock-100.conf'")]
|
||||||
|
[InlineData("got timeout")]
|
||||||
|
// PVE::Tools::lock_file uses this same wording for locks that carry no
|
||||||
|
// reissue-safety guarantee. Only the two guest config paths may retry.
|
||||||
|
[InlineData("can't lock file '/run/lock/pve-manager/pve-storage-local' - got timeout")]
|
||||||
|
[InlineData("can't lock file '/var/lock/pve-manager/pve-backup' - got timeout")]
|
||||||
|
[InlineData("can't lock file '/run/lock/lvm/V_pve' - got timeout")]
|
||||||
|
// A message PVE prefixed with its own context means the worker had already
|
||||||
|
// started; reissuing it could repeat work that landed.
|
||||||
|
[InlineData("clone failed: can't lock file '/var/lock/qemu-server/lock-100.conf' - got timeout")]
|
||||||
|
[InlineData("unable to resize: can't lock file '/var/lock/qemu-server/lock-100.conf' - got timeout")]
|
||||||
|
public void IsLockTimeout_RejectsEveryOtherFailure(string message)
|
||||||
|
{
|
||||||
|
Assert.False(GuestLockRetry.IsLockTimeout(ApiError(message)));
|
||||||
|
Assert.False(GuestLockRetry.IsLockTimeout(TaskError(message)));
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void IsLockTimeout_ReadsWhatPveSaidRatherThanTheComposedMessage()
|
||||||
|
{
|
||||||
|
// Both exception types prefix Message with their own context, which would
|
||||||
|
// defeat the anchor if the predicate matched on Message.
|
||||||
|
var api = ApiError(VmLockError);
|
||||||
|
var task = TaskError(VmLockError);
|
||||||
|
|
||||||
|
Assert.StartsWith("PVE API error", api.Message);
|
||||||
|
Assert.StartsWith("Task UPID:", task.Message);
|
||||||
|
Assert.True(GuestLockRetry.IsLockTimeout(api));
|
||||||
|
Assert.True(GuestLockRetry.IsLockTimeout(task));
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void IsLockTimeout_RejectsExceptionTypesThatAreNotApiFailures()
|
||||||
|
{
|
||||||
|
Assert.False(GuestLockRetry.IsLockTimeout(new InvalidOperationException(VmLockError)));
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Execute_ReturnsWithoutRetryingWhenTheOperationSucceeds()
|
||||||
|
{
|
||||||
|
var attempts = 0;
|
||||||
|
|
||||||
|
var result = GuestLockRetry.Execute(() => { attempts++; return 42; });
|
||||||
|
|
||||||
|
Assert.Equal(42, result);
|
||||||
|
Assert.Equal(1, attempts);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Execute_ReissuesUntilTheLockClears()
|
||||||
|
{
|
||||||
|
var attempts = 0;
|
||||||
|
|
||||||
|
var result = GuestLockRetry.Execute(() =>
|
||||||
|
{
|
||||||
|
attempts++;
|
||||||
|
if (attempts < 2) throw TaskError(VmLockError);
|
||||||
|
return "cloned";
|
||||||
|
});
|
||||||
|
|
||||||
|
Assert.Equal("cloned", result);
|
||||||
|
Assert.Equal(2, attempts);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Execute_DoesNotRetryOtherFailures()
|
||||||
|
{
|
||||||
|
var attempts = 0;
|
||||||
|
|
||||||
|
Assert.Throws<PveApiException>(() => GuestLockRetry.Execute<int>(() =>
|
||||||
|
{
|
||||||
|
attempts++;
|
||||||
|
throw ApiError("VM 100 not running");
|
||||||
|
}));
|
||||||
|
|
||||||
|
Assert.Equal(1, attempts);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task ExecuteAsync_ReissuesUntilTheLockClears()
|
||||||
|
{
|
||||||
|
var attempts = 0;
|
||||||
|
|
||||||
|
var result = await GuestLockRetry.ExecuteAsync(() =>
|
||||||
|
{
|
||||||
|
attempts++;
|
||||||
|
if (attempts < 2) throw ApiError(VmLockError);
|
||||||
|
return Task.FromResult("written");
|
||||||
|
});
|
||||||
|
|
||||||
|
Assert.Equal("written", result);
|
||||||
|
Assert.Equal(2, attempts);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task ExecuteAsync_DoesNotRetryOtherFailures()
|
||||||
|
{
|
||||||
|
var attempts = 0;
|
||||||
|
|
||||||
|
await Assert.ThrowsAsync<PveApiException>(() => GuestLockRetry.ExecuteAsync<int>(() =>
|
||||||
|
{
|
||||||
|
attempts++;
|
||||||
|
throw ApiError("can't lock file '/run/lock/lvm/V_pve' - got timeout");
|
||||||
|
}));
|
||||||
|
|
||||||
|
Assert.Equal(1, attempts);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Execute_GivesUpAndRethrowsOnceTheWindowElapses()
|
||||||
|
{
|
||||||
|
var attempts = 0;
|
||||||
|
|
||||||
|
var ex = Assert.Throws<PveTaskFailedException>(() => GuestLockRetry.Execute<int>(
|
||||||
|
() => { attempts++; throw TaskError(VmLockError); },
|
||||||
|
TimeSpan.Zero));
|
||||||
|
|
||||||
|
Assert.Contains("got timeout", ex.Message);
|
||||||
|
Assert.Equal(1, attempts);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user