rcourtman
06b7f5a0b9
Keep fixture alerts from outliving a mock-mode switch
...
SetMockMode(false) switched the mock package off and then cleared active
alerts, but every path that evaluates mode-dependent data checked the
mode once, up front. A mock alert pass (started from the ticker with go,
so passes overlap), a frontend or ingest refresh evaluating a GetState
snapshot, a backup-age check over fixture recovery rollups, or a
connection check over the mock ledger could already hold fixture data
when the switch happened. Its evaluations then landed after the clear,
in live mode, where nothing evaluates or resolves fixture resources, so
the alerts stayed until the 24-hour stale sweep.
A pass-level lock around checkMockAlerts was tried and reverted: it left
the other paths open and made the mock-mode API wait behind a whole pass
of synchronous event-log writes.
The monitor now owns a mock-mode epoch fence. The mock pass, backup-age
and snapshot checks, connection checks, Docker pruning and the
resource-store refresh take a scope before reading anything
mode-dependent and send each alert-manager call through it. SetMockMode,
now serialized, flips the mode, ends the epoch, waits only for calls
already admitted, and then clears. Unified metrics are evaluated one
resource per call, so the wait covers single calls, not passes.
Registry rebuilds are tracked rather than waited on. A refresh evaluates
the registry it read back only if no rebuild from an ended epoch
overlapped it, and the readers that open or remove alerts from the
inventory use current state until a clean rebuild in the new epoch
replaces the previous mode's registry. The mock supplemental fixture
sources now serve nothing outside mock mode: the router swaps them for
the real pollers only after the switch, refreshing between sources, so
a single switch to live mode used to raise fixture vSphere incidents.
The new tests park a real evaluation at a fixed point (a held lock or a
blocking supplier), switch modes and release it. Against the unfenced
code the mock pass left about 40 fixture alerts, the frontend refresh 5
vSphere incidents and the backup check 137 backup-age alerts in live
mode, and a mock-mode connection observation counted toward a live
connection's confirmation. The drain, republish and pruning tests fail
against a fence without its wait, overlap check or registry tracking.
Switching mock mode with several tenant monitors running remains
uncovered and is noted in the monitoring contract.
2026-10-07 12:03:38 +01:00
..
2026-03-18 16:06:30 +00:00
2026-07-20 22:41:19 +01:00
2026-08-31 15:53:50 +01:00
2026-08-31 15:53:50 +01:00
2026-10-05 06:02:27 +01:00
2026-09-01 17:34:30 +01:00
2026-09-01 17:34:30 +01:00
2026-07-17 16:09:06 +01:00
2026-08-02 20:30:38 +01:00
2026-07-27 11:13:04 +01:00
2026-08-24 16:36:22 +01:00
2026-10-07 12:03:38 +01:00
2026-08-30 21:44:32 +01:00
2026-10-07 12:03:38 +01:00
2026-07-29 20:09:23 +01:00
2026-07-29 21:28:26 +01:00
2026-10-04 15:56:57 +01:00
2026-02-07 22:05:55 +00:00
2026-09-05 15:05:53 +01:00
2026-10-05 14:18:18 +01:00
2026-10-07 12:03:38 +01:00
2026-10-01 11:32:30 +01:00
2026-10-07 12:03:38 +01:00
2026-06-02 14:52:47 +01:00
2026-07-24 12:40:05 +01:00
2026-08-27 20:41:08 +01:00
2026-03-18 16:06:30 +00:00
2026-03-18 16:06:30 +00:00
2026-07-22 06:42:33 +01:00
2026-10-05 04:14:27 +01:00
2026-07-11 18:25:17 +01:00
2025-12-01 17:33:36 +00:00
2026-03-18 16:06:30 +00:00
2026-08-11 20:46:11 +01:00
2026-08-11 20:46:11 +01:00
2026-08-27 16:27:46 +01:00
2026-08-27 16:27:46 +01:00
2026-09-08 03:59:00 +01:00
2026-08-28 10:14:02 +01:00
2026-04-01 19:14:12 +01:00
2026-07-27 18:36:50 +01:00
2026-07-10 00:05:22 +01:00
2026-07-10 00:05:22 +01:00
2026-08-20 11:40:16 +01:00
2026-08-20 23:33:36 +01:00
2026-09-07 09:45:31 +01:00
2026-09-01 15:26:48 +01:00
2026-09-01 15:26:48 +01:00
2026-07-19 09:13:14 +01:00
2026-08-03 01:01:20 +01:00
2026-08-03 01:01:20 +01:00
2026-07-23 22:33:07 +01:00
2026-09-06 12:45:40 +01:00
2026-08-30 00:07:53 +01:00
2026-09-11 10:28:15 +01:00
2026-09-01 15:26:48 +01:00
2026-08-11 21:03:27 +01:00
2026-03-18 16:06:30 +00:00
2026-03-18 16:06:30 +00:00
2026-10-04 17:00:24 +01:00
2026-07-02 23:52:02 +01:00
2026-10-03 13:58:05 +01:00
2026-10-04 11:54:46 +01:00
2026-08-10 14:39:01 +01:00
2026-10-04 21:37:11 +01:00
2026-10-05 14:13:16 +01:00
2026-01-28 16:52:35 +00:00
2026-07-13 09:41:11 +01:00
2026-10-03 13:58:05 +01:00
2026-10-04 22:43:53 +01:00
2026-04-22 09:22:46 +01:00
2026-09-07 23:52:46 +01:00
2026-10-04 13:15:54 +01:00
2026-10-07 12:03:38 +01:00
2026-10-04 17:00:24 +01:00
2026-06-27 17:41:00 +01:00
2026-10-03 13:58:05 +01:00
2026-07-27 18:36:50 +01:00
2026-07-27 18:36:50 +01:00
2026-10-04 23:34:19 +01:00
2026-10-05 17:17:29 +01:00
2026-10-04 23:34:19 +01:00
2026-10-04 23:34:19 +01:00
2026-07-27 18:36:50 +01:00
2026-07-24 15:46:56 +01:00
2026-10-06 20:50:21 +01:00
2026-10-06 20:50:21 +01:00
2026-08-03 01:01:20 +01:00
2026-08-03 01:01:20 +01:00
2026-09-06 11:13:59 +01:00
2026-08-03 01:01:20 +01:00
2026-09-01 15:26:48 +01:00
2026-07-23 22:17:25 +01:00
2026-10-06 23:09:52 +01:00
2026-07-21 20:30:54 +01:00
2026-08-25 12:24:32 +01:00
2026-09-06 12:45:40 +01:00
2026-07-26 19:31:15 +01:00
2026-07-28 13:28:55 +01:00
2026-08-02 18:56:05 +01:00
2026-09-10 16:00:36 +01:00
2026-03-18 16:06:30 +00:00
2026-07-28 12:08:10 +01:00
2026-08-31 21:20:29 +01:00
2026-09-24 10:24:29 +01:00
2026-07-23 22:33:07 +01:00
2026-04-11 22:46:34 +01:00
2026-02-04 19:49:52 +00:00
2026-02-04 19:49:52 +00:00
2026-04-12 09:41:29 +01:00
2026-06-11 19:42:40 +01:00
2026-06-11 19:42:40 +01:00
2026-07-24 10:26:50 +01:00
2026-07-24 00:19:40 +01:00
2026-10-06 19:37:14 +01:00
2026-07-23 22:05:09 +01:00
2026-07-23 22:33:07 +01:00
2026-10-04 06:33:37 +01:00
2026-08-28 23:23:43 +01:00
2025-12-01 21:18:33 +00:00
2026-10-04 06:33:37 +01:00
2026-10-04 06:33:37 +01:00
2026-10-04 08:18:36 +01:00
2026-10-04 06:33:37 +01:00
2026-10-04 08:18:36 +01:00
2026-03-18 16:06:30 +00:00
2026-03-18 16:06:30 +00:00
2026-08-26 12:11:10 +01:00
2026-10-05 17:53:17 +01:00
2026-10-05 17:53:17 +01:00
2026-10-05 17:53:17 +01:00
2026-10-05 17:53:17 +01:00
2026-06-09 21:42:21 +01:00
2026-10-07 12:03:38 +01:00
2026-10-07 12:03:38 +01:00
2026-06-09 21:42:21 +01:00
2026-08-26 20:22:20 +01:00
2026-01-22 13:39:06 +00:00
2026-10-06 20:50:21 +01:00
2026-07-30 22:42:04 +01:00
2026-10-06 23:09:52 +01:00
2026-10-07 11:18:45 +01:00
2026-10-06 18:03:55 +01:00
2026-09-24 17:43:50 +01:00
2026-10-07 12:03:38 +01:00
2026-10-07 12:03:38 +01:00
2026-10-07 12:03:38 +01:00
2026-10-07 12:03:38 +01:00
2026-07-26 20:03:46 +01:00
2026-05-05 14:19:50 +01:00
2026-08-10 14:39:01 +01:00
2026-09-05 15:05:53 +01:00
2026-10-07 12:03:38 +01:00
2026-05-05 16:25:00 +01:00
2026-08-24 22:45:59 +01:00
2026-08-24 22:45:59 +01:00
2026-07-28 11:24:01 +01:00
2026-08-13 22:06:29 +01:00
2026-10-05 14:13:16 +01:00
2026-07-24 22:34:54 +01:00
2026-08-31 15:53:50 +01:00
2026-09-10 08:42:59 +01:00
2026-10-06 19:37:14 +01:00
2026-10-07 12:03:38 +01:00
2026-06-14 11:07:47 +01:00
2026-07-30 19:08:28 +01:00
2026-10-07 12:03:38 +01:00
2026-10-06 20:50:21 +01:00
2026-09-19 17:27:25 +01:00
2026-07-30 20:22:47 +01:00
2026-07-30 20:22:47 +01:00
2026-10-06 19:37:14 +01:00
2026-10-05 09:21:40 +01:00
2026-04-11 14:51:10 +01:00
2026-08-26 12:11:10 +01:00
2026-10-06 23:09:52 +01:00
2026-10-06 23:09:52 +01:00
2026-10-07 12:03:38 +01:00
2026-07-27 10:00:33 +01:00
2026-10-03 01:16:42 +01:00
2026-07-12 02:53:01 +01:00
2026-10-07 12:03:38 +01:00
2026-07-24 10:26:50 +01:00
2026-09-06 14:12:46 +01:00
2026-10-01 03:59:08 +01:00
2026-09-29 22:53:25 +01:00
2026-09-05 19:41:25 +01:00
2026-08-11 14:05:06 +01:00
2026-05-08 19:53:03 +01:00
2026-09-21 23:24:33 +01:00
2026-10-06 20:50:21 +01:00
2026-10-06 19:37:14 +01:00
2026-08-24 16:36:22 +01:00
2026-03-18 16:06:30 +00:00
2026-08-27 20:41:08 +01:00
2026-10-07 12:03:38 +01:00
2026-08-02 19:07:38 +01:00
2026-10-07 12:03:38 +01:00
2026-05-13 19:00:02 +01:00
2026-09-21 23:24:33 +01:00
2026-07-23 22:43:34 +01:00
2026-08-26 15:17:12 +01:00
2026-10-06 19:37:14 +01:00
2026-08-04 10:35:09 +01:00
2026-09-06 01:54:28 +01:00
2026-10-04 17:00:24 +01:00
2026-07-24 22:37:29 +01:00
2026-10-04 22:43:53 +01:00
2026-09-21 23:24:33 +01:00
2026-10-04 17:00:24 +01:00
2026-09-21 23:24:33 +01:00
2026-07-27 12:07:44 +01:00
2026-10-03 13:58:05 +01:00
2026-08-10 14:39:01 +01:00
2026-08-26 00:27:58 +01:00
2026-07-24 11:21:27 +01:00
2026-10-06 23:43:53 +01:00
2026-01-04 10:29:40 +00:00
2026-07-26 12:42:22 +01:00
2026-07-28 13:28:55 +01:00
2026-09-22 20:37:00 +01:00
2026-08-10 14:39:01 +01:00
2026-07-26 20:03:46 +01:00
2026-07-26 20:03:46 +01:00
2026-08-16 13:24:08 +01:00
2026-09-10 21:04:56 +01:00
2026-03-18 16:06:30 +00:00
2026-09-06 14:34:03 +01:00
2026-07-26 20:03:46 +01:00
2026-09-05 20:06:03 +01:00
2026-05-17 18:45:24 +01:00
2026-07-29 23:11:45 +01:00
2026-03-18 16:06:30 +00:00
2026-09-06 12:45:40 +01:00
2026-07-24 12:42:16 +01:00
2026-03-18 16:06:30 +00:00
2026-07-30 21:15:05 +01:00
2026-10-07 12:03:38 +01:00
2026-09-24 10:57:17 +01:00
2026-07-16 16:25:10 +01:00
2026-07-21 21:24:15 +01:00
2026-07-16 15:17:45 +01:00
2026-07-20 07:50:22 +01:00
2026-10-05 17:17:29 +01:00
2026-04-01 10:04:30 +01:00
2026-10-06 19:37:14 +01:00
2026-07-24 00:19:40 +01:00
2026-08-29 20:51:19 +01:00
2026-04-11 22:46:34 +01:00
2026-08-31 07:03:43 +01:00
2026-09-05 12:53:13 +01:00
2026-07-19 15:10:38 +01:00
2026-07-19 15:10:38 +01:00
2026-07-18 11:54:09 +01:00
2026-07-18 11:54:09 +01:00
2026-10-07 11:18:10 +01:00
2026-08-24 22:45:59 +01:00
2026-06-10 09:19:51 +01:00
2026-07-26 12:42:22 +01:00
2026-03-18 16:06:30 +00:00
2026-07-13 11:04:09 +01:00
2026-07-13 11:04:09 +01:00
2026-07-24 10:11:56 +01:00
2026-08-10 14:39:01 +01:00
2026-07-22 06:42:33 +01:00
2026-08-05 19:05:05 +01:00
2026-08-04 15:07:18 +01:00
2026-03-18 16:06:30 +00:00
2026-03-18 16:06:30 +00:00
2026-07-24 10:26:50 +01:00
2026-07-24 10:26:50 +01:00
2026-07-23 22:24:52 +01:00
2026-07-23 22:24:52 +01:00
2026-03-26 22:58:27 +00:00
2026-10-07 12:03:38 +01:00
2026-08-29 15:23:28 +01:00
2026-09-15 09:37:01 +01:00
2026-09-15 09:37:01 +01:00
2026-08-30 22:10:49 +01:00
2026-08-30 22:10:49 +01:00
2026-08-12 11:22:29 +01:00
2026-08-12 11:22:29 +01:00
2026-03-18 16:06:30 +00:00
2026-03-18 16:06:30 +00:00
2026-03-18 16:06:30 +00:00
2026-08-27 20:41:08 +01:00
2026-08-27 20:41:08 +01:00
2026-01-06 17:44:52 +00:00
2026-01-06 17:44:52 +00:00
2026-09-06 20:58:14 +01:00
2026-09-06 20:02:06 +01:00
2026-02-04 19:33:29 +00:00
2025-12-02 12:02:13 +00:00
2026-08-24 16:36:22 +01:00
2026-08-26 12:11:10 +01:00
2026-06-11 16:00:25 +01:00
2026-07-28 12:08:10 +01:00
2026-10-04 20:55:13 +01:00
2026-10-04 09:45:21 +01:00
2026-07-25 00:21:35 +01:00
2026-09-23 23:55:18 +01:00
2026-10-04 09:45:21 +01:00
2026-10-04 09:45:21 +01:00
2026-07-30 16:45:13 +01:00