fix: inject the guest-lock retry delay so tests no longer race wall clock (#167)

PveHttpClientLockRetryTests set GuestLockRetry's retry budget via reflection
on a private field with no production writer, and scripted two lock
failures to land inside a 400ms window. A cold or loaded CI runner's
first-attempt JIT and scheduling could burn past 400ms before the second
attempt started, failing the test though the retry itself was correct
(#134).

GuestLockRetry.ExecuteAsync gains an internal overload that takes the
inter-attempt delay as a Func<TimeSpan, Task>; the public overload keeps
defaulting to Task.Delay, so production behaviour (45s window, budget/4
capped at 2s) is unchanged. PveHttpClient gains a matching internal
constructor seam (window, handler, delay), replacing the reflection the
tests used for both the private HttpClient and the retry window. Tests
now pass a no-op delay, so the retry loop's real elapsed time drops to
microseconds and the production 45s window can never be exhausted by
runner speed.

Two tests pin that production still waits for real: one records the
delay invocations through the internal seam and asserts the computed
interval, the other drives the public overload with a small window and
asserts wall-clock time actually advances. Both were mutation-tested
against a no-op-default regression and fail without the fix.

The give-up test was renamed (PutAsync_DoesNotReissueWhenTheRetryWindowIsAlreadySpent)
to describe what TimeSpan.Zero actually proves: the client never attempts
a reissue once the budget reads spent, not a multi-attempt exhaustion
sequence — a review finding on the original name.

Out of scope, noted for follow-up: GuestLockRetryTests.cs's synchronous
Execute() tests still use a 400ms ShortWindow with real Thread.Sleep,
which is the same flake shape on the sync path; Execute() has no delay
seam. PveHttpClientTimeoutTests.cs and PveHttpClientFormEncodingTests.cs
still reflect on the private _httpClient field, which the new handler
seam could also retire.

Co-authored-by: goodolclint-claude[bot] <323206664+goodolclint-claude[bot]@users.noreply.github.com>
This commit is contained in:
goodolclint-claude[bot]
2026-09-02 17:10:52 +00:00
committed by GitHub
parent 1bf7483a4e
commit 1bc46567f5
4 changed files with 133 additions and 39 deletions
@@ -1,5 +1,6 @@
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Net;
using System.Threading.Tasks;
using PSProxmoxVE.Core.Exceptions;
@@ -166,6 +167,50 @@ namespace PSProxmoxVE.Core.Tests.Utilities
Assert.Equal(1, attempts);
}
[Fact]
public async Task ExecuteAsync_InvokesTheInjectedDelayOncePerRetryWithTheComputedInterval()
{
var attempts = 0;
var delays = new List<TimeSpan>();
var result = await GuestLockRetry.ExecuteAsync(() =>
{
attempts++;
if (attempts < 3) throw ApiError(VmLockError);
return Task.FromResult("written");
}, GuestLockRetry.DefaultWindow, delay =>
{
delays.Add(delay);
return Task.CompletedTask;
});
Assert.Equal("written", result);
// budget/4 capped at 2s; DefaultWindow (45s) puts the quarter above the cap.
Assert.Equal(new[] { TimeSpan.FromSeconds(2), TimeSpan.FromSeconds(2) }, delays);
}
[Fact]
public async Task ExecuteAsync_PublicOverloadActuallyWaitsBetweenAttempts()
{
var attempts = 0;
var window = TimeSpan.FromMilliseconds(200);
var elapsed = Stopwatch.StartNew();
await GuestLockRetry.ExecuteAsync(() =>
{
attempts++;
if (attempts < 2) throw ApiError(VmLockError);
return Task.FromResult("written");
}, window);
elapsed.Stop();
// quarter of 200ms is 50ms; a no-op delay would leave this near zero. The
// threshold is well under 50ms so Task.Delay's own timer slop (it can return
// a fraction of a millisecond early) never makes this assertion itself flaky.
Assert.True(elapsed.Elapsed >= TimeSpan.FromMilliseconds(20),
$"Expected the public overload to wait close to one retry interval; took {elapsed.Elapsed}.");
}
[Fact]
public void Execute_GivesUpAndRethrowsOnceTheWindowElapses()
{