docs(v1.11.0): document what actually ships, with the measurements behind it

The release notes inherited from the feature branch described the version it was
written against, not the one going out.

- `SCHEMA_VERSION` is 11 -> 12, not 10 -> 11, and the upgrade notes now say why:
  11 was taken by v1.10.4 while this was in review, and the version gate would
  have skipped the migration entirely on every existing install. Includes the
  no-op recovery path for anyone running a pre-release build that recorded 11.
- Successful agent polls are not logged by default, with the measured table
  behind it: 2 424 bytes/row on PostgreSQL 15 against the real schema and all
  nine indexes, ~9 792 logged calls/day/agent, and what that means at 20, 200
  and 500 nodes both ways. The point is not the disk, it is that the row cap
  holds by DELETING, so without this the configured 7-day/30-day retention
  quietly becomes a few hours for everything in the table.
- Runtime cost stated as measured numbers rather than adjectives: 27.7 us per
  request, 1.4 us on an excluded path, 18.8 us per row on the writer, 0.096 %
  of one core at 500 nodes.
- REQUEST_LOG_QUEUE_MAX_BYTES documented in .env.template and CONFIG.md, with
  the reason it exists: the row count alone does not bound memory when
  max_body_bytes is operator-editable to 256 KB.
- The old "raise REQUEST_LOG_QUEUE_MAX if you see drops" advice is corrected -
  following it could OOM the worker. Lower max_body_bytes or sample_rate first;
  if you do raise the queue, raise its byte ceiling with it.
- Two behaviours that used to be silent are now written down: sink counters are
  per worker, and clearing the exclude-path list falls back to the shipped
  defaults rather than logging everything.
- The `operator` role's visibility of agent rows is documented, including what
  it deliberately does NOT extend to (anonymous traffic and the usernames in
  failed logins).

The v1.10.4 through v1.10.14 notes are unchanged and still above this in both
files.

.env.template

@@ -127,9 +127,16 @@ REQUEST_LOG_ENABLED=true
 
 # Per-worker in-process queue depth. When it fills, rows are DROPPED and counted
 # rather than blocking the request — the drop count is shown on the Request Log
-# page. Raise it if you see drops under normal load.
+# page (per worker).
 REQUEST_LOG_QUEUE_MAX=2000
 
+# Hard memory ceiling for that same queue, per worker. The row count above does
+# NOT bound memory on its own: `requestlog.max_body_bytes` is editable from
+# Settings up to 256 KB and a row can carry it twice, so at that ceiling a
+# 2000-row queue would hold ~1 GiB. Whichever limit is reached first stops the
+# queue. Raise this if you raise REQUEST_LOG_QUEUE_MAX.
+REQUEST_LOG_QUEUE_MAX_BYTES=67108864
+
 # Rows per batched INSERT. One connection is taken from the pool per batch, not
 # per request.
 REQUEST_LOG_BATCH_SIZE=100

CONFIG.md

@@ -154,7 +154,7 @@ REQUEST_LOG_ENABLED=false
 3. Disk büyümesi asıl operasyonel konudur: sırasıyla `sample_rate`'i düşürün, `capture_get`'i
    kapatın, `capture_bodies`'i kapatın, sonra `success_retention_days`'i kısaltın.
 
-### REQUEST_LOG_QUEUE_MAX / REQUEST_LOG_BATCH_SIZE / REQUEST_LOG_FLUSH_MS (v1.11.0)
+### REQUEST_LOG_QUEUE_MAX / REQUEST_LOG_QUEUE_MAX_BYTES / REQUEST_LOG_BATCH_SIZE / REQUEST_LOG_FLUSH_MS (v1.11.0)
 
 **Ne İşe Yarar**: Log satırlarını yazan arka plan görevinin ayarları. Satırlar sınırlı bir kuyruğa
 konur ve toplu (batch) INSERT ile yazılır; böylece istek yolu asla veritabanını beklemez.
@@ -165,6 +165,12 @@ konur ve toplu (batch) INSERT ile yazılır; böylece istek yolu asla veritaban
 # Request Log sayfasında gösterilir), istek bloklanmaz.
 REQUEST_LOG_QUEUE_MAX=2000
 
+# Aynı kuyruğun BAYT tavanı (worker başına). Satır sayısı tek başına belleği
+# sınırlamaz: `max_body_bytes` Settings'ten 256 KB'a kadar ayarlanabilir ve bir
+# satır bunu iki kez taşıyabilir, o tavanda 2000 satırlık kuyruk ~1 GiB tutar.
+# Hangi sınır önce dolarsa kuyruk orada durur.
+REQUEST_LOG_QUEUE_MAX_BYTES=67108864
+
 # Tek INSERT'te kaç satır yazılacağı (havuzdan istek başına değil, batch başına
 # bir bağlantı alınır)
 REQUEST_LOG_BATCH_SIZE=100
@@ -174,9 +180,16 @@ REQUEST_LOG_FLUSH_MS=500
 ```
 
 **Nasıl Kullanılır**:
-1. Request Log sayfasında "rows dropped" uyarısı görüyorsanız önce `REQUEST_LOG_QUEUE_MAX`'ı
-   artırın; sorun devam ederse `sample_rate`'i düşürün.
-2. Bu üç değer worker başınadır — `UVICORN_WORKERS` arttıkça toplam bellek de o oranda artar.
+1. Request Log sayfasında "rows dropped" uyarısı görüyorsanız önce `requestlog.max_body_bytes`
+   veya `sample_rate`'i düşürün. `REQUEST_LOG_QUEUE_MAX`'ı artırmak bu worker'ın tutabileceği
+   belleği de artırır; artıracaksanız `REQUEST_LOG_QUEUE_MAX_BYTES`'ı da birlikte artırın.
+2. Bu değerler worker başınadır — `UVICORN_WORKERS` arttıkça toplam bellek de o oranda artar.
+   `GET /api/request-logs/stats` içindeki sayaçlar da worker başınadır ve yanıtta öyle
+   etiketlenir; 4 worker'da gördüğünüz düşüş sayısı gerçeğin dörtte biridir.
+3. Büyük filolarda tek en etkili ayar `requestlog.capture_agent_success`'tir (varsayılan kapalı).
+   Açık olsaydı 200 düğümlük bir filo günde ~2M satır yazar ve 500.000 satır tavanına 6 saatte
+   ulaşırdı; yapılandırılmış 7 gün / 30 gün saklama o noktada birkaç saate iner. Başarısız ajan
+   çağrıları bu ayardan bağımsız olarak her zaman loglanır.
 
 ## 🚀 Deployment Senaryoları
 

README.md

@@ -2475,7 +2475,7 @@ Developed with ❤️ for the HAProxy community
 
 ## Release Notes
 
-- **v1.11.0** (2026-08-11) — **Unified request/response log with configurable retention**: until now the only record of what happened was `user_activity_logs`, which stores non-GET **2xx** operations with no bodies — so when something failed you could see *that* the count went up, never *what was sent or what came back*. This release adds one queryable timeline covering **both directions**: every inbound API call (**including GETs and including 4xx/5xx**) with the user, client IP, status, duration and — redacted and size-capped — the request and response bodies; and every **outbound** HTTP call the backend makes, tagged with who it went to (ACME/Let's Encrypt, Cloudflare, GoDaddy, HAProxy stats, agents, the ACME diagnostics probe). Outbound rows **inherit the inbound request's id**, so one operator action and the CA/DNS calls it triggered read as a single trace — opening a failed *Request Certificate* shows the exact `POST /acme/new-order` and the CA's `429` body underneath it. Capture is a **pure-ASGI middleware that tees** the request and response streams rather than draining them, so no downstream handler is affected (notably the raw-body agent heartbeat), and rows are written by a **batching background writer** with a bounded queue, so the request path never waits on the database and a saturated logger drops rows visibly instead of blocking. Secrets never land: headers are an allowlist (`Authorization`/`Cookie` reduced to a presence marker), body keys and value shapes are redacted (passwords, tokens, API keys, private-key PEMs, JWTs), the **ACME JWS request body is never stored** (a stored `protected`+`signature` pair is a replayable credential — a summary is logged instead), DNS-provider errors record only the exception **type**, and the ACME HTTP-01 challenge endpoint is excluded so `key_authorization` is never captured. **Retention is operator-configurable** in *Settings → Request Log*: separate day counts for successful and failed rows (defaults 7 and 30) plus a hard row cap (500 000), whichever is reached first, pruned in **batches** under a Postgres advisory lock so a multi-million-row table cannot time out the delete or have every replica scan it at once. New **Request Log** page (`requestlog.read`) and retention/purge permission (`requestlog.manage`); `super_admin` and `security_admin` get both, `operator` gets read, `viewer` gets neither. Adds one new table (`request_logs`) and its settings seed — SCHEMA_VERSION 10 → 11, auto-migrated, no existing table altered, no agent or rendered-config change. Kill switches: `REQUEST_LOG_ENABLED=false` (environment — the middleware is then never registered and costs nothing) or the `enabled` toggle in Settings (no restart).
+- **v1.11.0** (2026-08-14) — **Unified request/response log with configurable retention**: until now the only record of what happened was `user_activity_logs`, which stores non-GET **2xx** operations with no bodies — so when something failed you could see *that* the count went up, never *what was sent or what came back*. This release adds one queryable timeline covering **both directions**: every inbound API call (**including GETs and including 4xx/5xx**) with the user, client IP, status, duration and — redacted and size-capped — the request and response bodies; and every **outbound** HTTP call the backend makes, tagged with who it went to (ACME/Let's Encrypt, Cloudflare, GoDaddy, HAProxy stats, agents, the ACME diagnostics probe). Outbound rows **inherit the inbound request's id**, so one operator action and the CA/DNS calls it triggered read as a single trace — opening a failed *Request Certificate* shows the exact `POST /acme/new-order` and the CA's `429` body underneath it. Capture is a **pure-ASGI middleware that tees** the request and response streams rather than draining them, so no downstream handler is affected (notably the raw-body agent heartbeat), and rows are written by a **batching background writer** with a bounded queue, so the request path never waits on the database and a saturated logger drops rows visibly instead of blocking. Secrets never land: headers are an allowlist (`Authorization`/`Cookie` reduced to a presence marker), body keys and value shapes are redacted (passwords, tokens, API keys, private-key PEMs, JWTs), the **ACME JWS request body is never stored** (a stored `protected`+`signature` pair is a replayable credential — a summary is logged instead), DNS-provider errors record only the exception **type**, and the ACME HTTP-01 challenge endpoint is excluded so `key_authorization` is never captured. **Retention is operator-configurable** in *Settings → Request Log*: separate day counts for successful and failed rows (defaults 7 and 30) plus a hard row cap (500 000), whichever is reached first, pruned in **batches** under a Postgres advisory lock so a multi-million-row table cannot time out the delete or have every replica scan it at once. New **Request Log** page (`requestlog.read`) and retention/purge permission (`requestlog.manage`); `super_admin` and `security_admin` get both, `operator` gets read, `viewer` gets neither. **Successful agent polls are not logged** (`capture_agent_success`, default off; failures always are), which is what keeps the table's size a function of operator activity rather than of node count: measured at 2 424 bytes/row, a 200-node fleet would otherwise write 2.0M rows/day and reach the row cap in six hours, silently reducing the configured 7-day/30-day retention to a few hours for everything in the table. Cost is measured, not estimated: 27.7 µs per request on the hot path, 18.8 µs per row on the writer task, **0.096 % of one core at 500 nodes**. Adds one new table (`request_logs`) and its settings seed — SCHEMA_VERSION 11 → 12 (not 11: that number was taken by v1.10.4 while this was in review, and the version gate would have skipped the migration entirely on every existing install), auto-migrated, no existing table altered, no agent or rendered-config change. Kill switches: `REQUEST_LOG_ENABLED=false` (environment — the middleware is then never registered and costs nothing) or the `enabled` toggle in Settings (no restart).
 - **v1.10.14** (2026-08-14) — **A converged node keeps acknowledging**: the deploy report is the server's only evidence that a member node applied its `keepalived.conf`, and it was sent on the write path alone. Once the rendered config was on disk the agent took the idempotency early return on every cycle and never reported again, so a **single lost report** — a backend restart, a 5xx, a network blip — left the VIP reading `SYNCING (0/n)` with an empty *Last ack* forever, while the node was demonstrably running the right config. Nothing would ever reconcile the two: the node was correct, the page was not, and the only way out was to change the rendered config so the agent wrote it again. The agent now re-asserts its state on the idempotent path too, which costs one request per node per ~2.5 minutes and touches nothing on the node — keepalived is not reloaded and the file is not rewritten. This is a long-standing gap from the original HA/VIP work, surfaced when acknowledgements were dropped for an unrelated reason in v1.10.12. Agent-script change: sync the script from Agent Management and let the agents upgrade. No schema or API change.
 - **v1.10.13** (2026-08-14) — **Agent deploy acknowledgements were silently dropped** (regression in v1.10.12, fix it before or with that release): the takeover-retirement clause added to `POST /agents/{name}/keepalived-status` in v1.10.12 reused one query placeholder for both the assignment `last_deploy_hash=$n` and the comparison inside its `CASE`. PostgreSQL deduces a type per **use**, so the same placeholder came out as `text` in one and `character varying` in the other, and asyncpg rejected the statement with `AmbiguousParameterError`. The failure was not partial: the whole UPDATE never ran, so **no member ever recorded an acknowledgement**. Every VIP sat at `SYNCING (0/n)` with an empty *Last ack*, even after the nodes had deployed the config successfully, and teardown acknowledgements were lost the same way. The hash is now bound to its own placeholder, which is only ever compared against the column and therefore unambiguous. Verified against a real PostgreSQL: both statements execute, a matching hash retires the takeover authorisation, a non-matching hash and a NULL `applied_config_hash` both leave it in place, and every case records the acknowledgement. A test now asserts every `$n` in these statements is bound exactly once and that the count matches the arguments passed. Backend only: no schema, agent or API-shape change.
 - **v1.10.12** (2026-08-14) — **A valid keepalived config is no longer rejected by its own warning**: before writing a rendered `keepalived.conf` the agent validates it with `keepalived -t` and, on failure, keeps the running config and does not restart keepalived. That fail-safe is right, but it treated **any** non-zero exit as invalid, and keepalived's config-test exit code does not separate fatal from benign. Measured on 2.2.8: a clean config exits 0, but `Truncating auth_pass to 8 characters` exits **5** and so does a missing `}` or an `Unknown keyword`. A VRRP password longer than eight characters was therefore enough to make every apply fail, including on nodes whose own running config produces the same warning and has been serving the VIP for weeks. The gate now judges the **output**: messages known to be benign are dropped and anything that remains still fails, so it fails **closed** and an unrecognised message is treated as fatal. Verified against real keepalived: a truncation warning passes while a missing brace, an unknown keyword and a `SECURITY VIOLATION` are all still refused. The agent also **reports what keepalived said** now, in the log and in the status the HA/VIP page shows; discarding it left a correct refusal with no way to act on it. Agent-script change: sync the script from Agent Management and let the agents upgrade for it to take effect. No schema or API change.

UPGRADE_GUIDE.md

@@ -1,13 +1,39 @@
 # Upgrade Notes — v1.11.0 (Unified request/response log)
 
-**Adds one new table and bumps `SCHEMA_VERSION` 10 → 11. The migration runs automatically on the
+**Adds one new table and bumps `SCHEMA_VERSION` 11 → 12. The migration runs automatically on the
 first backend start.** No agent impact, no rendered-config change, no change to any existing API
 shape or response body.
 
+> **12, not 11.** The feature was developed against v1.10.3, where `SCHEMA_VERSION` was still 10, and
+> proposed 11. v1.10.4 took 11 in the meantime. Shipping it as 11 would have been silently inert:
+> `run_all_migrations()` returns early on `applied_version >= SCHEMA_VERSION`, so every database
+> already at 11 would have skipped the entire sequence and received neither `request_logs` nor the
+> `requestlog.*` permissions, while a fresh install would have received both. If you are coming from
+> a pre-release build that recorded 11 for this feature, no action is needed: the bump to 12 re-runs
+> the (idempotent) sequence and creates whatever is missing.
+
 - **New table `request_logs`** (BIGSERIAL primary key, 9 indexes). Created empty and starts filling
-  immediately. Budget for it as **one row per API call**. The shipped defaults keep 7 days of
-  successful requests, 30 days of failed ones, and at most 500 000 rows — whichever limit is reached
-  first. Change any of it in *Settings → Request Log*.
+  immediately. The shipped defaults keep 7 days of successful requests, 30 days of failed ones, and
+  at most 500 000 rows — whichever limit is reached first. Change any of it in
+  *Settings → Request Log*.
+- **Successful agent polls are NOT logged** (`requestlog.capture_agent_success`, default off).
+  Failed agent calls always are. This is what keeps the table's size a function of operator
+  activity rather than of how many nodes you run. Measured at 2 424 bytes/row on PostgreSQL 15
+  against the real schema and all nine indexes, with each agent issuing ~9 792 logged calls a day:
+
+  | fleet | if successful polls were logged | with the shipped default |
+  |---|---|---|
+  | 20 nodes  | 196k rows/day, 453 MB/day, row cap in 61 h | ~21k rows/day, 48 MB/day, cap in 24 d |
+  | 200 nodes | 2.0M rows/day, 4.5 GB/day, row cap in 6 h  | ~30k rows/day, 69 MB/day, cap in 17 d |
+  | 500 nodes | 4.9M rows/day, 11 GB/day, row cap in 2.5 h | ~44k rows/day, 103 MB/day, cap in 11 d |
+
+  The row cap always holds, but it holds by deleting — so without this default the configured
+  "7 days of successes, 30 days of failures" silently becomes a few hours of both, and the forensic
+  record the feature exists for is evicted by polling noise. Turn it on temporarily when debugging
+  a specific node, then turn it back off.
+- **Runtime cost is measured, not estimated.** The middleware adds 27.7 µs (p50) per request and
+  1.4 µs on an excluded path; redaction runs on the writer task, off the request path, at 18.8 µs
+  per row. At 500 nodes that is **0.096 % of one core** in total.
 - **New permissions `requestlog.read` and `requestlog.manage`.** Both are granted to `super_admin`
   and `security_admin`; `operator` gets `requestlog.read` only; `viewer` gets neither, because
   captured request/response bodies are a broader disclosure surface than the read-only configuration
@@ -31,11 +57,28 @@ shape or response body.
   assets, and the log viewer's own endpoints. The list is editable, except the log viewer itself,
   which is a hard floor so the page cannot end up logging you reading it.
 - **Disk growth is the main operational consideration.** On a busy install, in order of bluntness:
-  lower `sample_rate` (errors are always kept at 100 %), turn off `capture_get`, turn off
-  `capture_bodies`, or shorten `success_retention_days`.
+  leave `capture_agent_success` off (the single biggest lever on a large fleet), lower `sample_rate`
+  (errors are always kept at 100 %), turn off `capture_get`, turn off `capture_bodies`, or shorten
+  `success_retention_days`.
+- **`operator` can see agent rows.** A caller holding only `requestlog.read` sees their own inbound
+  requests plus the fleet's, and nothing belonging to another user. Agent rows are included because
+  an apply fails on the *node* and the node reports it over its own API key — scoping to own-rows
+  only would have hidden the diagnosis from exactly the role the grant exists for. Anonymous traffic
+  (failed logins and the usernames they carry, unauthenticated probes) is **not** agent traffic and
+  remains visible only to `requestlog.manage`.
 - **New environment variables**, all optional: `REQUEST_LOG_ENABLED` (default `true`),
-  `REQUEST_LOG_QUEUE_MAX` (2000), `REQUEST_LOG_BATCH_SIZE` (100), `REQUEST_LOG_FLUSH_MS` (500). See
-  `.env.template`.
+  `REQUEST_LOG_QUEUE_MAX` (2000 rows), `REQUEST_LOG_QUEUE_MAX_BYTES` (64 MiB),
+  `REQUEST_LOG_BATCH_SIZE` (100), `REQUEST_LOG_FLUSH_MS` (500). See `.env.template`.
+  `REQUEST_LOG_QUEUE_MAX_BYTES` is a hard memory ceiling per worker: the row count alone does not
+  bound memory, because `max_body_bytes` is operator-editable up to 256 KB and a row can carry it
+  twice — at the ceiling the default 2 000-row queue would hold ~1 GiB, which is the whole pod
+  limit. Whichever limit binds first stops the queue. If you raise `REQUEST_LOG_QUEUE_MAX`, raise
+  this with it.
+- **`GET /api/request-logs/stats` sink counters are per worker**, labelled as such in the response.
+  With `UVICORN_WORKERS > 1` each process keeps its own queue and its own drop counter.
+- **Clearing the exclude-path list does not mean "log everything."** An empty list falls back to the
+  shipped defaults, so the log viewer and the raw-body agent heartbeat stay excluded; the UI now
+  says so and shows what was actually applied.
 - **To disable entirely:** set `REQUEST_LOG_ENABLED=false` in the backend environment and restart —
   the middleware is then not registered at all and costs nothing, not even a settings lookup. The
   `enabled` toggle in Settings is the no-restart equivalent (it takes effect immediately).