Files
Alphaeus Mote bfbd9b1d4f fix(messaging): a fleet reconnecting at once gets every settings answer [skip ci]
A 10,000-device connect storm answered 1,526 settings pulls on SQLite and
4,365 on PostgreSQL; the rest came after the agent's 20 s. Measured (a
probe in the lanes): every pull queued in the 8 agent-event lanes behind
every device's ready event, and each event's replay id was one database
insert (19 to 36 ms under the storm's write contention) before anything
else; the lanes' wait reached 61 s on SQLite (41 s on PostgreSQL), 8,312
of 9,845 SQLite answers left after the agent had stopped waiting, and 155
more failed to publish. Nothing was dropped by a subscription.

- Requests on the event subject (anything with an inbox: settings pulls,
  CLI runs, uploads, restarts) leave the lanes for a request path of their
  own: up to 256 at a time opened side by side, their ids remembered for
  every replica in one insert (ServerReceipt.EventBatch, RememberEventsAsync;
  a replay is refused as before), settings pulls answered together.
- AgentSettingsAnswers: the enabled profiles (and every device's groups when
  a profile names groups) kept as a snapshot until AgentSettingsSignal fires
  (profile, group, member, tag changes) or 30 s; a batch's devices and
  stored sets in one query each. A stored set that is what the snapshot
  decides, not pending and not behind the agent, is the answer with no
  write; everything else goes through ReconcileDeviceAsync, read afresh.
- The lanes take what has queued (up to 64) and write it together:
  inventory reports in one transaction (one at a time when that fails),
  ready and logon events' device rows in one, event ids in one insert;
  order per device is kept. These writes, settings rows and replay ids
  take SqliteWriteGate on SQLite.
- Presence, inventory, events and requests read the keys of devices not
  cached a query per 500 (DeviceKeyCache.LoadAsync) and open side by side.
- The server's ECDH key is held in up to 8 instances: one instance under
  one lock made every first message of every device (after a server start)
  wait its turn, which starved the thread pool once opening ran in
  parallel (833 answers in that run).
- Subscriptions hold 65,536 messages (NATS.Net's default is 16,384); what
  a full one drops is counted by kind and logged every 10 s at most. An
  answer sent after the agent's 20 s is counted and logged too.
- An upgrade's key check reads the keys the command was just sealed with.

After: 10,000/10,000 first-try answers on SQLite (p50 0.5 s, p95 1.8 s,
max 2.1 s) and on PostgreSQL (p50 0.1 s, p95 0.3 s, max 0.9 s); every
inventory, presence and ready event stored by the end of the 20 s ramp.

Replies are sealed and signed for the asking device and bound to its inbox
as before; pulls and other requests are still replay-checked in the
database for every replica.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 10:56:23 -05:00
..