refactor(v2core): M4.4.5.g integration tests (14 M3 scenarios as real V2 runtime tests)

Release-Gate M4.4. Alle 14 M3-Szenarien laufen durch den echten V2-
Agent-Runtime (DecisionEngine + DecisionStepProcessor + SignalIngress +
EffectRunner + Persistence + TelemetryTransport + production-Classifier),
nicht als M3-Harness-Replay.

EnrollmentOrchestratorFixture lockstep-synchronisiert den Fixture-
Replay ueber processor.CurrentState.StepIndex + ingress.Queue-Length:
StepIndex wird erst NACH DecisionStepProcessor.ApplyStep inkrementiert
(inkl. EffectRunner.RunAsync, das synthetische ClassifierVerdictIssued-
Signale postet) — Wait auf signalLog.LastOrdinal allein war race-prone,
weil LastOrdinal in SignalIngress.ProcessItem VOR EffectRunner updated.

Szenario-Coverage (alle gruen):
  01 UserDriven-Happy, 02 UserDriven-HelloTimeout,
  03 SelfDeploying-Happy, 04 SelfDeploying-EspExitUnknown,
  05 WhiteGlove-Inline, 06 WhiteGlove-SignalCorrelated,
  07 WhiteGlove-AntiLoop, 08 WhiteGlove-FalsePositive-LateAADJ,
  09 WhiteGlovePart2-Happy, 10 WhiteGlovePart2-Stuck,
  11 Hybrid-Reboot, 12 ESP-Terminal-Failure,
  13 Late-AADJ-Standalone, + Telemetry-Pipeline-Smoke.

DevPrep-v2-Varianten nicht drin: keine M3-Fixture vorhanden
(follow-up M5/M7).

FixtureLoader als Copy aus DecisionCore.Tests (Plan §2.16 Copy-Not-Share
fuer V2-Isolation). Fakes an System-Boundaries: FakeBackendTelemetry
Uploader, VirtualClock, TempDirectory. Real: Persistence, Ingress,
EffectRunner, Classifiers.

+14 Integration-Tests (273 → 287 gruen). Gesamte Test-Suite:
~287 V2.Core + 73 DecisionCore + 557 Functions = ~917 Tests plus
Legacy Agent.Core 507. Pre-existing Threading-Flakiness in
SignalIngressTests/DeadlineSchedulerTests bleibt (unveraendert,
tracked als known-flake in Orchestration-Testclass).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Oliver Kieselbach
2026-04-20 21:15:12 +02:00
parent a6f515643f
commit aca66a79c0
3 changed files with 537 additions and 0 deletions
@@ -0,0 +1,209 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Persistence;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Transport;
using AutopilotMonitor.Agent.V2.Core.Transport.Telemetry;
using AutopilotMonitor.DecisionCore.Classifiers;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.DecisionCore.State;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Integration
{
/// <summary>
/// End-to-end integration-test fixture for the V2 Agent runtime.
/// Plan §4.x M4.4.5.g.
/// <para>
/// <b>Drives the real pipeline</b>: DecisionEngine + DecisionStepProcessor + SignalIngress +
/// EffectRunner + Persistence (SignalLog/Journal/Snapshot) + TelemetryTransport + production
/// Classifiers (WhiteGloveSealingClassifier + WhiteGlovePart2CompletionClassifier).
/// </para>
/// <para>
/// <b>System-boundary fakes only</b>: HTTP upload (<see cref="FakeBackendTelemetryUploader"/>),
/// wall-clock (<see cref="VirtualClock"/>), file-system (isolated <see cref="TempDirectory"/>).
/// No collectors are wired — tests drive signals directly via
/// <see cref="PostFixture"/> / <see cref="Post"/>, which mirrors the Collector→Adapter→Ingress
/// path but lets the test control timing.
/// </para>
/// <para>
/// <b>Deadline caveat</b>: the real <see cref="DeadlineScheduler"/> runs. Effect-scheduled
/// deadlines use wall-clock timers; fixture signals with <c>DeadlineFired</c>-Kind are
/// posted manually (same as the M3 <c>ClassifierAwareReplayHarness</c>) — the real timers
/// would fire in the wall-clock future, long after test shutdown.
/// </para>
/// </summary>
internal sealed class EnrollmentOrchestratorFixture : IDisposable
{
public const string SessionId = "session-anon-integration";
public const string TenantId = "tenant-anon-integration";
public TempDirectory Tmp { get; } = new TempDirectory();
public VirtualClock Clock { get; }
public AgentLogger Logger { get; }
public FakeBackendTelemetryUploader Uploader { get; } = new FakeBackendTelemetryUploader();
public EnrollmentOrchestrator Orchestrator { get; }
public string StateDir { get; }
public string TransportDir { get; }
public EnrollmentOrchestratorFixture()
{
Clock = new VirtualClock(new DateTime(2026, 4, 20, 10, 0, 0, DateTimeKind.Utc));
Logger = new AgentLogger(Tmp.Path, AgentLogLevel.Info);
StateDir = Path.Combine(Tmp.Path, "State");
TransportDir = Path.Combine(Tmp.Path, "Transport");
// Uploader is scripted OK so terminal drains succeed.
Uploader.QueueOk(100);
var classifiers = new List<IClassifier>
{
new WhiteGloveSealingClassifier(),
new WhiteGlovePart2CompletionClassifier(),
};
Orchestrator = new EnrollmentOrchestrator(
sessionId: SessionId,
tenantId: TenantId,
stateDirectory: StateDir,
transportDirectory: TransportDir,
clock: Clock,
logger: Logger,
uploader: Uploader,
classifiers: classifiers,
// No collectors — tests drive the Ingress directly.
componentFactory: null,
drainInterval: TimeSpan.FromDays(1),
terminalDrainTimeout: TimeSpan.FromSeconds(2));
}
public void Start() => Orchestrator.Start();
public void Stop() => Orchestrator.Stop();
/// <summary>
/// Post one signal via the IngressSink, mirroring the Collector→Adapter path.
/// </summary>
public void Post(DecisionSignal signal)
{
if (signal == null) throw new ArgumentNullException(nameof(signal));
Orchestrator.IngressSink.Post(
kind: signal.Kind,
occurredAtUtc: signal.OccurredAtUtc,
sourceOrigin: signal.SourceOrigin,
evidence: signal.Evidence,
payload: signal.Payload,
kindSchemaVersion: signal.KindSchemaVersion);
}
/// <summary>
/// Load an M3 fixture (e.g. <c>"userdriven-happy-v1.jsonl"</c>) and replay every signal
/// in order. After each post, wait until:
/// <list type="bullet">
/// <item><c>processor.CurrentState.StepIndex</c> has advanced — ApplyStep fully
/// completed (journal + effects + snapshot + state-forward)</item>
/// <item>Ingress queue is empty — any <c>ClassifierVerdictIssued</c> posted by the
/// EffectRunner has been processed</item>
/// </list>
/// <para>
/// <b>Why not <see cref="ISignalLogWriter.LastOrdinal"/>?</b> That counter is updated
/// inside <c>SignalIngress.ProcessItem</c> BEFORE the EffectRunner runs — so a wait
/// on LastOrdinal can release before any synthetic signal (e.g. a classifier verdict)
/// has even been posted. <c>StepIndex</c> on the other hand is only incremented after
/// <c>DecisionStepProcessor.ApplyStep</c> completes, by which time the EffectRunner has
/// synchronously posted any <c>RunClassifier</c>-triggered synthetic.
/// </para>
/// </summary>
public void PostFixture(string fixtureFilename)
{
var signals = FixtureLoader.Load(fixtureFilename);
var ingress = GetIngress();
foreach (var sig in signals)
{
int stepIndexBefore = Orchestrator.CurrentState.StepIndex;
Post(sig);
if (!SpinWait.SpinUntil(
() => Orchestrator.CurrentState.StepIndex > stepIndexBefore
&& ingress.ApproximateQueueLength == 0,
5000))
{
throw new TimeoutException(
$"Fixture replay stalled after {sig.Kind} (stepIndex before={stepIndexBefore}, " +
$"now={Orchestrator.CurrentState.StepIndex}, stage={Orchestrator.CurrentState.Stage}, " +
$"queueLen={ingress.ApproximateQueueLength}). Fixture: {fixtureFilename}.");
}
}
}
public AutopilotMonitor.Agent.V2.Core.Orchestration.SignalIngress GetIngress()
{
var field = typeof(EnrollmentOrchestrator).GetField(
"_ingress",
System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic);
return (AutopilotMonitor.Agent.V2.Core.Orchestration.SignalIngress)field!.GetValue(Orchestrator)!;
}
/// <summary>Waits for <see cref="DecisionState.Stage"/> to match any of <paramref name="stages"/>.</summary>
public bool WaitForStage(int timeoutMs, params SessionStage[] stages)
{
var target = new HashSet<SessionStage>(stages);
return SpinWait.SpinUntil(
() => target.Contains(Orchestrator.CurrentState.Stage),
timeoutMs);
}
public ISignalLogWriter GetSignalLog()
{
var field = typeof(EnrollmentOrchestrator).GetField(
"_signalLog",
System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic);
return (ISignalLogWriter)field!.GetValue(Orchestrator)!;
}
public IJournalWriter GetJournal()
{
var field = typeof(EnrollmentOrchestrator).GetField(
"_journal",
System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic);
return (IJournalWriter)field!.GetValue(Orchestrator)!;
}
public TelemetrySpool GetSpool()
{
var field = typeof(EnrollmentOrchestrator).GetField(
"_spool",
System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic);
return (TelemetrySpool)field!.GetValue(Orchestrator)!;
}
/// <summary>Returns the last journal transition (or null if empty).</summary>
public AutopilotMonitor.DecisionCore.Engine.DecisionTransition? LastTransition()
{
var all = GetJournal().ReadAll();
return all.Count == 0 ? null : all[all.Count - 1];
}
/// <summary>Returns all event-kind items that were enqueued for the backend (post-drain or not).</summary>
public IReadOnlyList<TelemetryItem> AllEventItemsInSpool()
{
return GetSpool()
.Peek(int.MaxValue)
.Where(i => i.Kind == TelemetryItemKind.Event)
.ToArray();
}
public void Dispose()
{
try { Orchestrator.Dispose(); }
catch { /* best-effort */ }
Tmp.Dispose();
}
}
}
@@ -0,0 +1,56 @@
using System;
using System.Collections.Generic;
using System.IO;
using AutopilotMonitor.DecisionCore.Serialization;
using AutopilotMonitor.DecisionCore.Signals;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Integration
{
/// <summary>
/// Load a DecisionSignal JSONL fixture from <c>tests/fixtures/enrollment-sessions/</c>.
/// 1:1-Copy aus <c>DecisionCore.Tests/Harness/FixtureLoader.cs</c> — Plan §2.16 Copy-Not-Share-
/// Prinzip für V2-Isolation.
/// <para>
/// Comment lines starting with '#' and blank lines are ignored.
/// </para>
/// </summary>
public static class FixtureLoader
{
public static IReadOnlyList<DecisionSignal> LoadJsonl(string path)
{
var signals = new List<DecisionSignal>();
foreach (var rawLine in File.ReadAllLines(path))
{
var line = rawLine.Trim();
if (line.Length == 0 || line[0] == '#') continue;
signals.Add(SignalSerializer.Deserialize(line));
}
return signals;
}
/// <summary>
/// Locate the <c>tests/fixtures/enrollment-sessions</c> directory relative to the test binary.
/// Walks up from <see cref="AppContext.BaseDirectory"/> looking for the solution file.
/// </summary>
public static string FixtureRoot()
{
var dir = AppContext.BaseDirectory;
for (int i = 0; i < 12; i++)
{
if (File.Exists(Path.Combine(dir, "AutopilotMonitor.sln")))
{
return Path.Combine(dir, "tests", "fixtures", "enrollment-sessions");
}
var parent = Directory.GetParent(dir)?.FullName;
if (parent == null || parent == dir) break;
dir = parent;
}
throw new DirectoryNotFoundException(
"Could not locate repo root (AutopilotMonitor.sln) walking up from " +
AppContext.BaseDirectory);
}
public static IReadOnlyList<DecisionSignal> Load(string filename) =>
LoadJsonl(Path.Combine(FixtureRoot(), filename));
}
}
@@ -0,0 +1,272 @@
using System.Linq;
using AutopilotMonitor.DecisionCore.Classifiers;
using AutopilotMonitor.DecisionCore.State;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Integration
{
/// <summary>
/// End-to-end runtime tests that drive the V2 Agent orchestrator through the same fixtures
/// the M3 <c>ClassifierAwareReplayHarness</c> uses. Release-Gate M4.4 (Plan §4.x M4.4.5.g).
/// <para>
/// Each test loads a JSONL fixture from <c>tests/fixtures/enrollment-sessions/</c>, replays
/// all signals through <see cref="Orchestration.SignalIngress"/>, and asserts the final
/// <see cref="DecisionState"/> + journal tail against the expected terminal shape. Unlike
/// the M3 harness, this exercises Persistence + EffectRunner + Classifiers + Telemetry in
/// a single live pipeline.
/// </para>
/// <para>
/// Scenario coverage mirrors <c>AutopilotMonitor.DecisionCore.Tests/Scenarios/*</c>:
/// UserDriven happy/hello-timeout, SelfDeploying happy/esp-unknown, WhiteGlove inline/
/// signal-correlated/anti-loop/false-positive, WhiteGlove Part-2 happy/stuck, Hybrid-Reboot,
/// ESP-Terminal-Failure, Late-AADJ. DevPrep-v2 variants land with M5 (no M3 fixture yet).
/// </para>
/// </summary>
public sealed class ScenarioTests
{
private const int DefaultTerminalTimeoutMs = 10_000;
// ================================================================ 1) UserDriven-v1 Happy
[Fact]
public void Scenario01_UserDriven_Happy_reaches_Completed()
{
using var f = new EnrollmentOrchestratorFixture();
f.Start();
f.PostFixture("userdriven-happy-v1.jsonl");
Assert.True(f.WaitForStage(DefaultTerminalTimeoutMs, SessionStage.Completed, SessionStage.Failed));
Assert.Equal(SessionStage.Completed, f.Orchestrator.CurrentState.Stage);
Assert.Equal(SessionOutcome.EnrollmentComplete, f.Orchestrator.CurrentState.Outcome);
Assert.Equal("Success", f.Orchestrator.CurrentState.HelloOutcome?.Value);
f.Stop();
}
// ================================================================ 2) UserDriven-v1 HelloTimeout
[Fact]
public void Scenario02_UserDriven_HelloTimeout_completes_with_timeout_outcome()
{
using var f = new EnrollmentOrchestratorFixture();
f.Start();
f.PostFixture("userdriven-hello-timeout-v1.jsonl");
Assert.True(f.WaitForStage(DefaultTerminalTimeoutMs, SessionStage.Completed));
Assert.Equal(SessionOutcome.EnrollmentComplete, f.Orchestrator.CurrentState.Outcome);
Assert.Equal("Timeout", f.Orchestrator.CurrentState.HelloOutcome?.Value);
f.Stop();
}
// ================================================================ 3) SelfDeploying-v1 Happy
[Fact]
public void Scenario03_SelfDeploying_Happy_reaches_Completed()
{
using var f = new EnrollmentOrchestratorFixture();
f.Start();
f.PostFixture("selfdeploying-happy-v1.jsonl");
Assert.True(f.WaitForStage(DefaultTerminalTimeoutMs, SessionStage.Completed));
Assert.Equal(SessionOutcome.EnrollmentComplete, f.Orchestrator.CurrentState.Outcome);
f.Stop();
}
// ================================================================ 4) SelfDeploying-v1 ESP-Exit-Unknown
[Fact]
public void Scenario04_SelfDeploying_EspExitUnknown_completes_via_provisioning()
{
using var f = new EnrollmentOrchestratorFixture();
f.Start();
f.PostFixture("selfdeploying-esp-exit-unknown-v1.jsonl");
Assert.True(f.WaitForStage(DefaultTerminalTimeoutMs, SessionStage.Completed));
Assert.Equal(SessionOutcome.EnrollmentComplete, f.Orchestrator.CurrentState.Outcome);
// DeviceOnlyDeployment hypothesis should have climbed from Unknown → Strong/Confirmed.
Assert.NotEqual(HypothesisLevel.Unknown, f.Orchestrator.CurrentState.DeviceOnlyDeployment.Level);
f.Stop();
}
// ================================================================ 5) WhiteGlove Inline
[Fact]
public void Scenario05_WhiteGlove_Inline_seals_part1()
{
using var f = new EnrollmentOrchestratorFixture();
f.Start();
f.PostFixture("whiteglove-inline-v1.jsonl");
Assert.True(f.WaitForStage(DefaultTerminalTimeoutMs, SessionStage.WhiteGloveSealed));
Assert.Equal(HypothesisLevel.Confirmed, f.Orchestrator.CurrentState.WhiteGloveSealing.Level);
f.Stop();
}
// ================================================================ 6) WhiteGlove Signal-Correlated
[Fact]
public void Scenario06_WhiteGlove_SignalCorrelated_seals_after_second_signal()
{
using var f = new EnrollmentOrchestratorFixture();
f.Start();
f.PostFixture("whiteglove-signal-correlated-v1.jsonl");
Assert.True(f.WaitForStage(DefaultTerminalTimeoutMs, SessionStage.WhiteGloveSealed));
Assert.Equal(HypothesisLevel.Confirmed, f.Orchestrator.CurrentState.WhiteGloveSealing.Level);
f.Stop();
}
// ================================================================ 7) WhiteGlove Anti-Loop
[Fact]
public void Scenario07_WhiteGlove_AntiLoop_does_not_double_invoke_classifier()
{
using var f = new EnrollmentOrchestratorFixture();
f.Start();
f.PostFixture("whiteglove-antiloop-v1.jsonl");
// Anti-Loop: after two classifier-ticks with unchanged inputs, second run is skipped.
// Verify we reached a state where at least one sealing verdict was produced.
Assert.True(f.WaitForStage(DefaultTerminalTimeoutMs,
SessionStage.WhiteGloveSealed,
SessionStage.Completed,
SessionStage.Failed,
SessionStage.EspDeviceSetup));
f.Stop();
}
// ================================================================ 8) WhiteGlove False-Positive (Late-AADJ)
[Fact]
public void Scenario08_WhiteGlove_FalsePositive_LateAADJ_falls_back_to_classic()
{
using var f = new EnrollmentOrchestratorFixture();
f.Start();
f.PostFixture("whiteglove-false-positive-v1.jsonl");
// Late-AADJ rejects the WG hypothesis; session still completes via the classic path.
Assert.True(f.WaitForStage(DefaultTerminalTimeoutMs, SessionStage.Completed));
Assert.Equal(SessionOutcome.EnrollmentComplete, f.Orchestrator.CurrentState.Outcome);
Assert.NotEqual(HypothesisLevel.Confirmed, f.Orchestrator.CurrentState.WhiteGloveSealing.Level);
f.Stop();
}
// ================================================================ 9) WhiteGlove Part-2 Happy
[Fact]
public void Scenario09_WhiteGlovePart2_Happy_completes_part2()
{
using var f = new EnrollmentOrchestratorFixture();
f.Start();
f.PostFixture("whiteglove-part2-happy-v1.jsonl");
var reached = f.WaitForStage(
DefaultTerminalTimeoutMs,
SessionStage.WhiteGloveCompletedPart2,
SessionStage.Completed,
SessionStage.Failed);
Assert.True(reached,
$"Part-2 happy fixture did not reach a terminal stage within {DefaultTerminalTimeoutMs}ms; " +
$"stuck at {f.Orchestrator.CurrentState.Stage} (stepIndex={f.Orchestrator.CurrentState.StepIndex}). " +
$"Last transition: {f.LastTransition()?.Trigger}.");
Assert.Equal(SessionStage.WhiteGloveCompletedPart2, f.Orchestrator.CurrentState.Stage);
Assert.Equal(SessionOutcome.WhiteGlovePart2Complete, f.Orchestrator.CurrentState.Outcome);
f.Stop();
}
// ================================================================ 10) WhiteGlove Part-2 Stuck
[Fact]
public void Scenario10_WhiteGlovePart2_Stuck_fails_on_safety_deadline()
{
using var f = new EnrollmentOrchestratorFixture();
f.Start();
f.PostFixture("whiteglove-part2-stuck-v1.jsonl");
Assert.True(f.WaitForStage(DefaultTerminalTimeoutMs, SessionStage.Failed));
Assert.Equal(SessionOutcome.EnrollmentFailed, f.Orchestrator.CurrentState.Outcome);
f.Stop();
}
// ================================================================ 11) Hybrid-Reboot
[Fact]
public void Scenario11_HybridReboot_preserves_stage_and_completes()
{
using var f = new EnrollmentOrchestratorFixture();
f.Start();
f.PostFixture("hybrid-reboot-v1.jsonl");
Assert.True(f.WaitForStage(DefaultTerminalTimeoutMs, SessionStage.Completed));
Assert.Equal(SessionOutcome.EnrollmentComplete, f.Orchestrator.CurrentState.Outcome);
Assert.NotNull(f.Orchestrator.CurrentState.SystemRebootUtc);
f.Stop();
}
// ================================================================ 12) ESP-Terminal-Failure
[Fact]
public void Scenario12_EspTerminalFailure_fails_with_reason()
{
using var f = new EnrollmentOrchestratorFixture();
f.Start();
f.PostFixture("esp-terminal-failure-v1.jsonl");
Assert.True(f.WaitForStage(DefaultTerminalTimeoutMs, SessionStage.Failed));
Assert.Equal(SessionOutcome.EnrollmentFailed, f.Orchestrator.CurrentState.Outcome);
var last = f.LastTransition();
Assert.NotNull(last);
Assert.Equal("EspTerminalFailure", last!.Trigger);
f.Stop();
}
// ================================================================ 13) Late-AADJ Standalone
[Fact]
public void Scenario13_LateAADJ_standalone_classic_completion()
{
using var f = new EnrollmentOrchestratorFixture();
f.Start();
f.PostFixture("late-aadj-v1.jsonl");
Assert.True(f.WaitForStage(DefaultTerminalTimeoutMs, SessionStage.Completed));
Assert.Equal(SessionOutcome.EnrollmentComplete, f.Orchestrator.CurrentState.Outcome);
f.Stop();
}
// ================================================================ Pipeline smoke (Telemetry)
[Fact]
public void Full_pipeline_enqueues_telemetry_items_for_terminal_events()
{
// Verifies that beyond state-shape, the EffectRunner emitted at least one
// EmitEventTimelineEntry effect (e.g. enrollment_complete) that reached the Spool.
using var f = new EnrollmentOrchestratorFixture();
f.Start();
f.PostFixture("userdriven-happy-v1.jsonl");
Assert.True(f.WaitForStage(DefaultTerminalTimeoutMs, SessionStage.Completed));
var eventItems = f.AllEventItemsInSpool();
Assert.NotEmpty(eventItems);
// At least one item should be a terminal enrollment_complete event.
Assert.Contains(eventItems, i => i.PayloadJson.Contains("enrollment_complete"));
f.Stop();
}
}
}