fix(agent): adoption cannot hide a node; Config Import on fresh installs (v1.11.1)

Six fixes to the agent's discovery path and one long-standing parity gap, found
by auditing it in loops against a real fleet.

DISCOVERY REPORTS THAT NEVER REACHED THE SERVER

The agent posts an unmanaged keepalived.conf for adoption and caches the hash of
what it sent, so the file - which carries the VRRP password - is re-posted only
when it changes. Delivery was judged by curl's exit code, and curl without -f
exits 0 on 5xx too, so a report the server REJECTED was recorded as delivered.
Since a hand-maintained config does not change on its own, that node dropped out
of "Unmanaged keepalived detected" permanently; the only cure was deleting a
cache file on the node by hand.

  - the report is cached only on a 2xx;
  - GET /agents/{name}/keepalived-config now reports whether the server actually
    holds a discovery for that agent, and the cache may only suppress while it
    says yes - which is what lets nodes stuck from earlier releases recover on
    their own, with nobody touching them;
  - the flag is parsed with has() + tostring, not `// empty`: jq's alternative
    operator returns the alternative for **false** as well as null, so the naive
    form could not tell "no record" from "older backend" and the recovery would
    have been completely inert;
  - a 400/413/422 records the refusal so identical bytes are not re-posted
    forever - 4xx and 5xx agent calls are never sampled out of the request log,
    so an unattended loop would write a row carrying the whole config every
    cycle - while 401 and 404 keep retrying, because here they mean a token
    rotation or an agent row briefly absent, not a bad payload;
  - the CLEAR path had the same exit-code defect, where it left a stale row
    offering a managed node for adoption with nothing to ever retry it.

CONFIG IMPORT WAS A NO-OP ON FRESHLY INSTALLED AGENTS

check_config_requests uploads a node's live haproxy.cfg on request. It was
defined in the installer body and in the self-upgrade daemon, but not in the
heredoc a fresh install writes, and its call site is guarded by `type` - so on
such a node the operator asked for a config and nothing arrived, with no error
anywhere. Any agent that had self-upgraded at least once already had it, which
is why it went unnoticed. The self-upgrade definition is copied verbatim
(verified line-for-line). A freshly installed agent now polls that endpoint once
per cycle exactly as every upgraded agent already does; no node running today
changes behaviour.

DETERMINISTIC CONFIG PATH

A pool may hold several clusters and the join that resolves keepalived_config_path
was unordered, so the path handed to an agent could differ between polls whenever
two clusters disagreed - the agent would inspect a file that is not there and the
node would never appear, intermittently. A customised path now wins over the
shipped default, then the lowest cluster id. Verified against a real PostgreSQL
over seven arrangements: with one cluster per pool, or when every cluster carries
the default, the value is byte-identical to before.

Verified end to end on a production fleet and, for each decision, against the
real _kp_discover block rather than a paraphrase.

Backend suite: 1674 passed, 152 skipped. bash -n passes on the whole file and on
the fresh-install body in isolation. The keepalived path is logic-identical
across both daemon copies, now pinned by a test.

.env.template

@@ -106,6 +106,44 @@ UVICORN_WORKERS=1
 # Example: http://haproxy-manager.example.com,http://localhost:8080
 CORS_ORIGINS=
 
+# ============================================================================
+# REQUEST / RESPONSE LOG (v1.11.0)
+# ============================================================================
+# Records every inbound API call and every outbound HTTP call the backend makes
+# (ACME, DNS providers, agents) into the `request_logs` table, browsable under
+# "Request Log" in the UI.
+#
+# Only the four settings below are environment-level, because they decide
+# whether the middleware is registered at all and how much memory the writer
+# queue may hold. Everything an operator tunes day to day — retention windows,
+# body capture, sampling, excluded paths — lives in the database and is edited
+# in Settings -> Request Log.
+
+# Hard kill-switch. When false the logging middleware is NEVER added to the ASGI
+# stack and neither the writer nor the retention task starts: zero overhead, not
+# even a settings lookup. Requires a restart to change.
+# (The `enabled` toggle in Settings is the no-restart equivalent.)
+REQUEST_LOG_ENABLED=true
+
+# Per-worker in-process queue depth. When it fills, rows are DROPPED and counted
+# rather than blocking the request — the drop count is shown on the Request Log
+# page (per worker).
+REQUEST_LOG_QUEUE_MAX=2000
+
+# Hard memory ceiling for that same queue, per worker. The row count above does
+# NOT bound memory on its own: `requestlog.max_body_bytes` is editable from
+# Settings up to 256 KB and a row can carry it twice, so at that ceiling a
+# 2000-row queue would hold ~1 GiB. Whichever limit is reached first stops the
+# queue. Raise this if you raise REQUEST_LOG_QUEUE_MAX.
+REQUEST_LOG_QUEUE_MAX_BYTES=67108864
+
+# Rows per batched INSERT. One connection is taken from the pool per batch, not
+# per request.
+REQUEST_LOG_BATCH_SIZE=100
+
+# Maximum wait before a partial batch is flushed, in milliseconds.
+REQUEST_LOG_FLUSH_MS=500
+
 # ============================================================================
 # FRONTEND CONFIGURATION (React)
 # ============================================================================

CONFIG.md

@@ -131,6 +131,66 @@ if (window.location) {
 }
 ```
 
+### REQUEST_LOG_ENABLED (v1.11.0)
+
+**Ne İşe Yarar**: Request/Response Log özelliğinin sert (hard) kill-switch'i. `false` yapıldığında
+loglama middleware'i ASGI zincirine **hiç eklenmez**, yazıcı ve retention görevleri başlatılmaz —
+yani sıfır ek yük, ayar okuması bile yapılmaz. Değişiklik için restart gerekir.
+
+**Örnekler**:
+```bash
+# Varsayılan: açık
+REQUEST_LOG_ENABLED=true
+
+# Tamamen kapat (ör. çok yüksek trafikli kurulum, veya regülasyon gereği)
+REQUEST_LOG_ENABLED=false
+```
+
+**Nasıl Kullanılır**:
+1. Restart gerektirmeden kapatmak isterseniz bunun yerine **Settings → Request Log → Enable request
+   log** anahtarını kullanın; o anında etkili olur.
+2. Retention süreleri, gövde (body) yakalama, örnekleme oranı ve hariç tutulan path'ler bu env
+   değişkeniyle değil, veritabanındaki `requestlog.*` ayarlarıyla yönetilir — arayüzden düzenlenir.
+3. Disk büyümesi asıl operasyonel konudur: sırasıyla `sample_rate`'i düşürün, `capture_get`'i
+   kapatın, `capture_bodies`'i kapatın, sonra `success_retention_days`'i kısaltın.
+
+### REQUEST_LOG_QUEUE_MAX / REQUEST_LOG_QUEUE_MAX_BYTES / REQUEST_LOG_BATCH_SIZE / REQUEST_LOG_FLUSH_MS (v1.11.0)
+
+**Ne İşe Yarar**: Log satırlarını yazan arka plan görevinin ayarları. Satırlar sınırlı bir kuyruğa
+konur ve toplu (batch) INSERT ile yazılır; böylece istek yolu asla veritabanını beklemez.
+
+**Örnekler**:
+```bash
+# Worker başına kuyruk derinliği. Dolduğunda satırlar DÜŞÜRÜLÜR (sayılır ve
+# Request Log sayfasında gösterilir), istek bloklanmaz.
+REQUEST_LOG_QUEUE_MAX=2000
+
+# Aynı kuyruğun BAYT tavanı (worker başına). Satır sayısı tek başına belleği
+# sınırlamaz: `max_body_bytes` Settings'ten 256 KB'a kadar ayarlanabilir ve bir
+# satır bunu iki kez taşıyabilir, o tavanda 2000 satırlık kuyruk ~1 GiB tutar.
+# Hangi sınır önce dolarsa kuyruk orada durur.
+REQUEST_LOG_QUEUE_MAX_BYTES=67108864
+
+# Tek INSERT'te kaç satır yazılacağı (havuzdan istek başına değil, batch başına
+# bir bağlantı alınır)
+REQUEST_LOG_BATCH_SIZE=100
+
+# Yarım dolu bir batch'in en fazla ne kadar bekletileceği (ms)
+REQUEST_LOG_FLUSH_MS=500
+```
+
+**Nasıl Kullanılır**:
+1. Request Log sayfasında "rows dropped" uyarısı görüyorsanız önce `requestlog.max_body_bytes`
+   veya `sample_rate`'i düşürün. `REQUEST_LOG_QUEUE_MAX`'ı artırmak bu worker'ın tutabileceği
+   belleği de artırır; artıracaksanız `REQUEST_LOG_QUEUE_MAX_BYTES`'ı da birlikte artırın.
+2. Bu değerler worker başınadır — `UVICORN_WORKERS` arttıkça toplam bellek de o oranda artar.
+   `GET /api/request-logs/stats` içindeki sayaçlar da worker başınadır ve yanıtta öyle
+   etiketlenir; 4 worker'da gördüğünüz düşüş sayısı gerçeğin dörtte biridir.
+3. Büyük filolarda tek en etkili ayar `requestlog.capture_agent_success`'tir (varsayılan kapalı).
+   Açık olsaydı 200 düğümlük bir filo günde ~2M satır yazar ve 500.000 satır tavanına 6 saatte
+   ulaşırdı; yapılandırılmış 7 gün / 30 gün saklama o noktada birkaç saate iner. Başarısız ajan
+   çağrıları bu ayardan bağımsız olarak her zaman loglanır.
+
 ## 🚀 Deployment Senaryoları
 
 ### Docker Compose

README.md

@@ -116,6 +116,7 @@ This architecture provides better security (no inbound connections to HAProxy se
 ✅ **HA / VIP (Keepalived) Management** - Create virtual IPs from the UI; the agent installs & configures Keepalived (unicast VRRP) with a HAProxy health-check so the VIP fails over automatically; live MASTER/BACKUP detection per node  
 ✅ **Role-Based User Management** - Admin and user roles with granular permissions and access control  
 ✅ **User Activity Audit Logs** - Complete audit trail of all system events  
+✅ **Request/Response Log** *(v1.11.0)* - Every inbound API call (GETs and errors included) and every outbound HTTP call the backend makes (ACME, Cloudflare, GoDaddy, agents) in one filterable timeline, with redacted, size-capped bodies and operator-configurable retention  
 ✅ **REST API** - Full programmatic access for automation and CI/CD integration
 
 
@@ -2474,6 +2475,11 @@ Developed with ❤️ for the HAProxy community
 
 ## Release Notes
 
+- **v1.11.1** (2026-08-15) — **A node can no longer be hidden from adoption for good, and Config Import reaches a freshly installed agent**: the agent posts an unmanaged `keepalived.conf` for adoption and caches the hash of what it sent, so the file (which carries the VRRP password) is re-posted only when it changes. Delivery was judged by `curl`'s exit code, which is **0 for 5xx as well**, so a report the server *rejected* was recorded as delivered — and because a hand-maintained config does not change on its own, that node dropped out of *Unmanaged keepalived detected* permanently, curable only by deleting a cache file on the node by hand. Now: the report is cached only on a 2xx; `GET /agents/{name}/keepalived-config` tells the agent whether the server actually holds a discovery for it, so nodes stuck from earlier releases recover by themselves on the next poll; a **400/413/422** records the refusal so identical bytes are not re-posted forever (4xx and 5xx are never sampled out of the request log, so an unattended loop would write a row carrying the whole config every cycle), while **401 and 404 keep retrying** because in this system they mean a token rotation or an agent row briefly absent, not a bad payload. The same exit-code mistake in the *clear* path is fixed too, where it left a managed node still being offered for adoption. Separately, **Config Import silently did nothing on any agent that had never self-upgraded**: `check_config_requests` was defined in the installer and in the self-upgrade daemon but not in the body a fresh install writes, and its call site is guarded by `type`, so the operator asked a node for its `haproxy.cfg` and nothing arrived, with no error anywhere. And the cluster whose `keepalived_config_path` is handed to an agent is now resolved deterministically — a pool may hold several clusters, and the unordered join could return a different one between polls, pointing the agent at a file that does not exist. Agent-script change: sync the script from Agent Management and let the agents upgrade. No schema change.
+- **v1.11.0** (2026-08-14) — **Unified request/response log with configurable retention**: until now the only record of what happened was `user_activity_logs`, which stores non-GET **2xx** operations with no bodies — so when something failed you could see *that* the count went up, never *what was sent or what came back*. This release adds one queryable timeline covering **both directions**: every inbound API call (**including GETs and including 4xx/5xx**) with the user, client IP, status, duration and — redacted and size-capped — the request and response bodies; and every **outbound** HTTP call the backend makes, tagged with who it went to (ACME/Let's Encrypt, Cloudflare, GoDaddy, HAProxy stats, agents, the ACME diagnostics probe). Outbound rows **inherit the inbound request's id**, so one operator action and the CA/DNS calls it triggered read as a single trace — opening a failed *Request Certificate* shows the exact `POST /acme/new-order` and the CA's `429` body underneath it. Capture is a **pure-ASGI middleware that tees** the request and response streams rather than draining them, so no downstream handler is affected (notably the raw-body agent heartbeat), and rows are written by a **batching background writer** with a bounded queue, so the request path never waits on the database and a saturated logger drops rows visibly instead of blocking. Secrets never land: headers are an allowlist (`Authorization`/`Cookie` reduced to a presence marker), body keys and value shapes are redacted (passwords, tokens, API keys, private-key PEMs, JWTs), the **ACME JWS request body is never stored** (a stored `protected`+`signature` pair is a replayable credential — a summary is logged instead), DNS-provider errors record only the exception **type**, and the ACME HTTP-01 challenge endpoint is excluded so `key_authorization` is never captured. **Retention is operator-configurable** in *Settings → Request Log*: separate day counts for successful and failed rows (defaults 7 and 30) plus a hard row cap (500 000), whichever is reached first, pruned in **batches** under a Postgres advisory lock so a multi-million-row table cannot time out the delete or have every replica scan it at once. New **Request Log** page (`requestlog.read`) and retention/purge permission (`requestlog.manage`); `super_admin` and `security_admin` get both, `operator` gets read, `viewer` gets neither. **Successful agent polls are not logged** (`capture_agent_success`, default off; failures always are), which is what keeps the table's size a function of operator activity rather than of node count: measured at 2 424 bytes/row, a 200-node fleet would otherwise write 2.0M rows/day and reach the row cap in six hours, silently reducing the configured 7-day/30-day retention to a few hours for everything in the table. Cost is measured, not estimated: 27.7 µs per request on the hot path, 18.8 µs per row on the writer task, **0.096 % of one core at 500 nodes**. Adds one new table (`request_logs`) and its settings seed — SCHEMA_VERSION 11 → 12 (not 11: that number was taken by v1.10.4 while this was in review, and the version gate would have skipped the migration entirely on every existing install), auto-migrated, no existing table altered, no agent or rendered-config change. Kill switches: `REQUEST_LOG_ENABLED=false` (environment — the middleware is then never registered and costs nothing) or the `enabled` toggle in Settings (no restart).
+- **v1.10.14** (2026-08-14) — **A converged node keeps acknowledging**: the deploy report is the server's only evidence that a member node applied its `keepalived.conf`, and it was sent on the write path alone. Once the rendered config was on disk the agent took the idempotency early return on every cycle and never reported again, so a **single lost report** — a backend restart, a 5xx, a network blip — left the VIP reading `SYNCING (0/n)` with an empty *Last ack* forever, while the node was demonstrably running the right config. Nothing would ever reconcile the two: the node was correct, the page was not, and the only way out was to change the rendered config so the agent wrote it again. The agent now re-asserts its state on the idempotent path too, which costs one request per node per ~2.5 minutes and touches nothing on the node — keepalived is not reloaded and the file is not rewritten. This is a long-standing gap from the original HA/VIP work, surfaced when acknowledgements were dropped for an unrelated reason in v1.10.12. Agent-script change: sync the script from Agent Management and let the agents upgrade. No schema or API change.
+- **v1.10.13** (2026-08-14) — **Agent deploy acknowledgements were silently dropped** (regression in v1.10.12, fix it before or with that release): the takeover-retirement clause added to `POST /agents/{name}/keepalived-status` in v1.10.12 reused one query placeholder for both the assignment `last_deploy_hash=$n` and the comparison inside its `CASE`. PostgreSQL deduces a type per **use**, so the same placeholder came out as `text` in one and `character varying` in the other, and asyncpg rejected the statement with `AmbiguousParameterError`. The failure was not partial: the whole UPDATE never ran, so **no member ever recorded an acknowledgement**. Every VIP sat at `SYNCING (0/n)` with an empty *Last ack*, even after the nodes had deployed the config successfully, and teardown acknowledgements were lost the same way. The hash is now bound to its own placeholder, which is only ever compared against the column and therefore unambiguous. Verified against a real PostgreSQL: both statements execute, a matching hash retires the takeover authorisation, a non-matching hash and a NULL `applied_config_hash` both leave it in place, and every case records the acknowledgement. A test now asserts every `$n` in these statements is bound exactly once and that the count matches the arguments passed. Backend only: no schema, agent or API-shape change.
+- **v1.10.12** (2026-08-14) — **A valid keepalived config is no longer rejected by its own warning**: before writing a rendered `keepalived.conf` the agent validates it with `keepalived -t` and, on failure, keeps the running config and does not restart keepalived. That fail-safe is right, but it treated **any** non-zero exit as invalid, and keepalived's config-test exit code does not separate fatal from benign. Measured on 2.2.8: a clean config exits 0, but `Truncating auth_pass to 8 characters` exits **5** and so does a missing `}` or an `Unknown keyword`. A VRRP password longer than eight characters was therefore enough to make every apply fail, including on nodes whose own running config produces the same warning and has been serving the VIP for weeks. The gate now judges the **output**: messages known to be benign are dropped and anything that remains still fails, so it fails **closed** and an unrecognised message is treated as fatal. Verified against real keepalived: a truncation warning passes while a missing brace, an unknown keyword and a `SECURITY VIOLATION` are all still refused. The agent also **reports what keepalived said** now, in the log and in the status the HA/VIP page shows; discarding it left a correct refusal with no way to act on it. Agent-script change: sync the script from Agent Management and let the agents upgrade for it to take effect. No schema or API change.
 - **v1.10.11** (2026-08-14) — **The *Adoptable* tag names the problem that actually blocks adoption**: the tag and the disabled *Adopt* button were computed separately and could disagree. A pair blocked because its peer's `keepalived.conf` could not be parsed was labelled **MASTER missing** — technically true, since the unreadable node's `state MASTER` had not been counted, but it pointed the operator at the wrong node while the real reason sat in the button's own tooltip. Both now come from one ordered decision, so the label, its colour and the tooltip always describe the condition that stops the adoption; a group held up by an unreadable or unreachable peer reads **blocked by peer**, and two MASTERs is now distinct from none. Display only: what the endpoint accepts or refuses is unchanged. On the public repo this is the first artifact carrying v1.10.4 through v1.10.10: none was released separately, because VIP adoption did not work end to end until these fixes landed.
 - **v1.10.10** (2026-08-14) — **Adoption blockers are listed once per instance**: with the instance-based panel a two-node pair reported the *same* problems about the *same* shared config twice, once per member, and the line numbers differ between the two files so plain de-duplication did not collapse them. Four issues on a pair read as eight, in both the *Adoptable* tooltip and the adopt dialog. They are now merged on the message text with the leading `line N:` ignored, so each distinct problem appears once. Display only: the endpoint already evaluated the combined set and its refusals are unchanged.
 - **v1.10.9** (2026-08-14) — **Adoption refuses to strand a node or silently normalise a peer's settings**: v1.10.8 adopted the whole VRRP instance, but it could only *match* a node it was able to read, that was enabled, and that sat in the same pool. Each of those was a door a real member of the group left through silently — the nodes that remained were rewritten while the one that left kept serving the same address from an unmanaged config. Seen on a live pool: one node of a pair had an unclosed `vrrp_instance` block, so it parsed to nothing while its partner parsed cleanly. Instead of guarding each door, adoption now asks the question directly — is there **any** reported `keepalived.conf` that mentions this virtual address and is not among the nodes being taken over — and refuses naming the node and the reason (unparseable, agent disabled, different pool). Separately, four VIP-level fields (`prefix_length`, unicast/multicast mode, HAProxy tracking and the VRRP password) are stored once and re-rendered onto **every** member, so taking them from whichever node was clicked imposed its settings on the others; the prefix length is the sharpest, because the design refuses to *guess* a netmask for a live VIP and copying one node's netmask onto another is that same change by another name. Adoption now requires the nodes to agree on all four, and compares the VRRP secret by decrypting each node's token (Fernet is non-deterministic, so the ciphertexts cannot be compared). Finally, the takeover authorisation is genuinely one-shot: `takeover_expected_hash` was written at adoption and never cleared, so it stayed valid for that file content indefinitely — it is now retired the moment a member acknowledges our rendered config, gated on the acked hash matching so a failed deploy never drops it. No schema change, no agent change.

UPGRADE_GUIDE.md

@@ -1,3 +1,211 @@
+# Upgrade Notes — v1.11.1 (adoption cannot hide a node; Config Import on fresh installs)
+
+**Agent-script change, no schema change.** No `SCHEMA_VERSION` bump, so the built-in roles are
+**not** re-seeded. After deploying, sync the Linux agent script from **Agent Management** and let
+the agents upgrade, or none of this reaches the nodes.
+
+- **A node that never appeared under *Unmanaged keepalived detected* now recovers by itself.**
+  The agent caches the hash of its last discovery report and skips re-posting while it matches.
+  Delivery was judged by `curl`'s exit code, which is 0 for 5xx too, so a rejected report was
+  cached as delivered and a hand-maintained config — which never changes on its own — kept that
+  node hidden. The report is now cached only on a 2xx, and the config endpoint reports whether
+  the server actually holds a discovery for that agent, so the cache can only suppress while the
+  server agrees. **No access to the nodes is needed**; affected nodes reappear within one poll
+  cycle (~2.5 min) after the agents pick up the new script.
+- **Permanent refusals do not loop.** A 400, 413 or 422 means the payload itself is unacceptable,
+  so the refusal is recorded and the same bytes are not re-posted; fixing the file releases the
+  brake, because it is keyed to the content hash. **401 and 404 keep retrying** — in this system
+  they mean a token rotation or an agent row briefly absent while it re-registers, and braking on
+  them would have re-created the very failure above. This matters beyond noise: 4xx and 5xx agent
+  calls are never sampled out of the request log, so a loop would write a row carrying the whole
+  `keepalived.conf` every cycle on every affected node.
+- **The clear path had the same defect.** When a node becomes managed the agent tells the server
+  to drop the discovery; that too was judged by the exit code, so a rejected clear left a stale
+  row offering a **managed** node for adoption, with nothing to ever retry it.
+- **Config Import now works on a freshly installed agent.** `check_config_requests` uploads a
+  node's live `haproxy.cfg` when you ask for it. It was defined in the installer and in the
+  self-upgrade daemon but not in the body a fresh install writes, and its call site is guarded by
+  `type`, so on such a node the feature was a silent no-op: the request was made and nothing ever
+  arrived. Any agent that had self-upgraded at least once already had it, which is why it went
+  unnoticed. A freshly installed agent now polls the pending-requests endpoint once per cycle,
+  exactly as every upgraded agent already does — **no node running today changes behaviour**.
+- **The keepalived.conf path is resolved deterministically.** A pool may hold more than one
+  cluster and the join was unordered, so the path handed to an agent could differ between polls
+  whenever two clusters disagreed on it — the agent would inspect a file that is not there and the
+  node would never appear, intermittently. A customised path now wins over the shipped default,
+  then the lowest cluster id. With one cluster per pool, or when every cluster carries the
+  default, the value is byte-identical to before.
+
+**Rollback:** safe. No schema or data change; reverting restores the previous behaviour, in which
+a rejected discovery report is never retried and Config Import is absent on fresh installs.
+
+---
+
+# Upgrade Notes — v1.11.0 (Unified request/response log)
+
+**Adds one new table and bumps `SCHEMA_VERSION` 11 → 12. The migration runs automatically on the
+first backend start.** No agent impact, no rendered-config change, no change to any existing API
+shape or response body.
+
+> **12, not 11.** The feature was developed against v1.10.3, where `SCHEMA_VERSION` was still 10, and
+> proposed 11. v1.10.4 took 11 in the meantime. Shipping it as 11 would have been silently inert:
+> `run_all_migrations()` returns early on `applied_version >= SCHEMA_VERSION`, so every database
+> already at 11 would have skipped the entire sequence and received neither `request_logs` nor the
+> `requestlog.*` permissions, while a fresh install would have received both. If you are coming from
+> a pre-release build that recorded 11 for this feature, no action is needed: the bump to 12 re-runs
+> the (idempotent) sequence and creates whatever is missing.
+
+- **New table `request_logs`** (BIGSERIAL primary key, 9 indexes). Created empty and starts filling
+  immediately. The shipped defaults keep 7 days of successful requests, 30 days of failed ones, and
+  at most 500 000 rows — whichever limit is reached first. Change any of it in
+  *Settings → Request Log*.
+- **Successful agent polls are NOT logged** (`requestlog.capture_agent_success`, default off).
+  Failed agent calls always are. This is what keeps the table's size a function of operator
+  activity rather than of how many nodes you run. Measured at 2 424 bytes/row on PostgreSQL 15
+  against the real schema and all nine indexes, with each agent issuing ~9 792 logged calls a day:
+
+  | fleet | if successful polls were logged | with the shipped default |
+  |---|---|---|
+  | 20 nodes  | 196k rows/day, 453 MB/day, row cap in 61 h | ~21k rows/day, 48 MB/day, cap in 24 d |
+  | 200 nodes | 2.0M rows/day, 4.5 GB/day, row cap in 6 h  | ~30k rows/day, 69 MB/day, cap in 17 d |
+  | 500 nodes | 4.9M rows/day, 11 GB/day, row cap in 2.5 h | ~44k rows/day, 103 MB/day, cap in 11 d |
+
+  The row cap always holds, but it holds by deleting — so without this default the configured
+  "7 days of successes, 30 days of failures" silently becomes a few hours of both, and the forensic
+  record the feature exists for is evicted by polling noise. Turn it on temporarily when debugging
+  a specific node, then turn it back off.
+- **Runtime cost is measured, not estimated.** The middleware adds 27.7 µs (p50) per request and
+  1.4 µs on an excluded path; redaction runs on the writer task, off the request path, at 18.8 µs
+  per row. At 500 nodes that is **0.096 % of one core** in total.
+- **New permissions `requestlog.read` and `requestlog.manage`.** Both are granted to `super_admin`
+  and `security_admin`; `operator` gets `requestlog.read` only; `viewer` gets neither, because
+  captured request/response bodies are a broader disclosure surface than the read-only configuration
+  views a viewer is meant to have. Custom roles can be granted either from **Users → Roles**.
+- **⚠️ Built-in roles are re-seeded to their defaults.** This is the pre-existing behaviour of every
+  `SCHEMA_VERSION` bump, not something new in this release, but it bites here because this release
+  bumps: the version gate re-runs the whole sequence and
+  `update_system_roles_to_enterprise_rbac()` issues an unconditional
+  `UPDATE roles SET … permissions = <defaults> WHERE name = …` for the four **built-in** roles
+  (`super_admin`, `operator`, `security_admin`, `viewer`). **Any customisation you made to a
+  built-in role is reverted.** Roles you created yourself are untouched (the update matches on
+  name). To preserve customisation, export with `GET /api/roles` before upgrading and re-apply with
+  `PUT /api/roles/{id}`, or move the customisation into a custom role.
+- **Bodies are captured, redacted and capped at 8 KB.** Passwords, tokens, API keys, private-key
+  PEMs, JWT-shaped values, `Authorization` / `Cookie` headers, ACME JWS payloads and DNS-provider
+  credentials are never stored. Headers use an allowlist — anything not on it is dropped rather than
+  saved. Review *Settings → Request Log* before enabling body capture in a regulated environment;
+  `capture_bodies` can be turned off while still recording who called what, with what result.
+- **Excluded by default:** health checks, the API docs, the ACME HTTP-01 challenge endpoint (it
+  returns `key_authorization`), the agent heartbeat (the highest-volume POST in the system), static
+  assets, and the log viewer's own endpoints. The list is editable, except the log viewer itself,
+  which is a hard floor so the page cannot end up logging you reading it.
+- **Disk growth is the main operational consideration.** On a busy install, in order of bluntness:
+  leave `capture_agent_success` off (the single biggest lever on a large fleet), lower `sample_rate`
+  (errors are always kept at 100 %), turn off `capture_get`, turn off `capture_bodies`, or shorten
+  `success_retention_days`.
+- **`operator` can see agent rows.** A caller holding only `requestlog.read` sees their own inbound
+  requests plus the fleet's, and nothing belonging to another user. Agent rows are included because
+  an apply fails on the *node* and the node reports it over its own API key — scoping to own-rows
+  only would have hidden the diagnosis from exactly the role the grant exists for. Anonymous traffic
+  (failed logins and the usernames they carry, unauthenticated probes) is **not** agent traffic and
+  remains visible only to `requestlog.manage`.
+- **New environment variables**, all optional: `REQUEST_LOG_ENABLED` (default `true`),
+  `REQUEST_LOG_QUEUE_MAX` (2000 rows), `REQUEST_LOG_QUEUE_MAX_BYTES` (64 MiB),
+  `REQUEST_LOG_BATCH_SIZE` (100), `REQUEST_LOG_FLUSH_MS` (500). See `.env.template`.
+  `REQUEST_LOG_QUEUE_MAX_BYTES` is a hard memory ceiling per worker: the row count alone does not
+  bound memory, because `max_body_bytes` is operator-editable up to 256 KB and a row can carry it
+  twice — at the ceiling the default 2 000-row queue would hold ~1 GiB, which is the whole pod
+  limit. Whichever limit binds first stops the queue. If you raise `REQUEST_LOG_QUEUE_MAX`, raise
+  this with it.
+- **`GET /api/request-logs/stats` sink counters are per worker**, labelled as such in the response.
+  With `UVICORN_WORKERS > 1` each process keeps its own queue and its own drop counter.
+- **Clearing the exclude-path list does not mean "log everything."** An empty list falls back to the
+  shipped defaults, so the log viewer and the raw-body agent heartbeat stay excluded; the UI now
+  says so and shows what was actually applied.
+- **To disable entirely:** set `REQUEST_LOG_ENABLED=false` in the backend environment and restart —
+  the middleware is then not registered at all and costs nothing, not even a settings lookup. The
+  `enabled` toggle in Settings is the no-restart equivalent (it takes effect immediately).
+- **Default admin password is not reset** by this bump; user seeding is guarded by an existence
+  check, not an upsert.
+- **Rollback:** downgrade the backend image freely. `request_logs` is purely additive and is simply
+  ignored by v1.10.x. Drop the table manually if you want the space back:
+  `DROP TABLE IF EXISTS request_logs;`
+
+---
+
+# Upgrade Notes — v1.10.14 (a converged node keeps acknowledging)
+
+**Agent-script change, no schema change.** No `SCHEMA_VERSION` bump. After deploying, sync the
+Linux agent script from **Agent Management** and let the agents upgrade, or the fix does not
+reach the nodes.
+
+- **Symptom:** a VIP shows `SYNCING (0/n)` with an empty *Last ack* even though every member node
+  has the rendered `keepalived.conf` on disk, keepalived is running and the VIP is held.
+- **Cause:** the deploy report was sent only when the agent actually wrote the config. Once the
+  node matched, it took the idempotency early return every cycle and never reported again, so any
+  report lost in transit was never retried and the server's view stayed stale permanently.
+- **Fix:** the agent re-asserts its state on the idempotent path as well. One request per node
+  per poll cycle (~2.5 min); nothing is written and keepalived is not reloaded.
+- **Recovery is automatic.** A VIP stuck at SYNCING converges on the first poll after the agents
+  pick up the new script. No action on the nodes, no re-apply, no edit to force a rewrite.
+- **This is not new in 1.10.12.** The gap dates from the original HA/VIP work; it only became
+  visible when acknowledgements were dropped for an unrelated reason.
+
+**Rollback:** safe. Reverting restores the previous behaviour, in which a lost acknowledgement is
+never recovered.
+
+---
+
+# Upgrade Notes — v1.10.13 (deploy acknowledgements were dropped)
+
+**Backend only, no schema change.** No `SCHEMA_VERSION` bump, no agent change. If you deployed
+v1.10.12, deploy this one too.
+
+- **Regression in v1.10.12.** The takeover-retirement clause added to the keepalived status
+  endpoint reused a query placeholder for both an assignment and a comparison. PostgreSQL types a
+  placeholder per use, so it was deduced as `text` in one place and `character varying` in the
+  other, and asyncpg refused the statement outright.
+- **Symptom:** a VIP stayed at `SYNCING (0/n)` with an empty *Last ack* even though the agent log
+  showed `applied config for VIP <id>` on every member. Teardown acknowledgements were lost the
+  same way, so a deletion never showed as complete.
+- **Nothing was damaged.** The failure was on the write of the acknowledgement, not on the node.
+  Configs were deployed correctly throughout; only the reporting was lost. Existing VIPs converge
+  on the next poll once this is deployed, with no action on the nodes.
+- **Verified against a real PostgreSQL**, not by inspection: both statements execute, a matching
+  hash retires the takeover authorisation, a non-matching hash and a NULL `applied_config_hash`
+  leave it in place, and the acknowledgement is recorded in every case.
+
+**Rollback:** do not roll back to v1.10.12; roll back to v1.10.11 instead, which predates the
+clause entirely.
+
+---
+
+# Upgrade Notes — v1.10.12 (valid config rejected by its own warning)
+
+**Agent-script change, no schema change.** No `SCHEMA_VERSION` bump. After deploying, sync the
+Linux agent script from **Agent Management** and let the agents upgrade, or the fix does not
+reach the nodes.
+
+- **Symptom:** applying a VIP left it stuck at `SYNCING`, the node kept its previous config and
+  the agent logged only `config validation failed (keepalived -t)`.
+- **Cause:** the agent treated any non-zero exit from `keepalived -t` as invalid. keepalived's
+  config-test exit code does not separate fatal from benign: on 2.2.8 a clean config exits 0,
+  while `Truncating auth_pass to 8 characters` exits 5 and so do a missing `}` and an unknown
+  keyword. A VRRP password longer than eight characters was enough to block every apply, even on
+  a node whose own running config emits the same warning.
+- **Fix:** the gate judges the output instead. Known-benign messages are dropped and anything
+  left still fails, so it fails closed. Verified against real keepalived: the truncation warning
+  passes; a missing brace, an unknown keyword and a `SECURITY VIOLATION` are refused.
+- **Also:** the agent now reports what keepalived actually said, in its log and in the status the
+  HA/VIP page shows. The refusal was correct but unactionable without reproducing it by hand.
+- **The fail-safe itself is unchanged:** a config that genuinely fails validation is never
+  written and keepalived is never restarted.
+
+**Rollback:** safe. Reverting restores the stricter gate, which rejects valid configs whose
+password exceeds eight characters.
+
+---
+
 # Upgrade Notes — v1.10.11 (Adoptable tag names the real blocker)
 
 **Frontend only, no schema change.** No `SCHEMA_VERSION` bump, no API change, no agent change.

backend/agent_notifications.py

@@ -25,31 +25,42 @@ async def notify_agents_config_change(cluster_id: int, version_name: str) -> Lis
         for agent in agents:
             try:
                 agent_url = f"http://{agent['ip_address']}:8081"  # Agent default port
-                
+
                 async with aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=10)) as session:
                     payload = {
                         "cluster_id": cluster_id,
                         "version_name": version_name,
                         "action": "config_update"
                     }
-                    
-                    async with session.post(f"{agent_url}/api/config/update", json=payload) as response:
-                        if response.status == 200:
-                            results.append({
-                                'node': agent['name'],
-                                'success': True,
-                                'message': f'Configuration updated successfully',
-                                'version': version_name
-                            })
-                            logger.info(f"✅ Agent {agent['name']} notified successfully")
-                        else:
-                            error_text = await response.text()
-                            results.append({
-                                'node': agent['name'],
-                                'success': False,
-                                'error': f'HTTP {response.status}: {error_text}'
-                            })
-                            logger.error(f"❌ Agent {agent['name']} notification failed: {response.status}")
+
+                    # v1.11.0: instrumented so the code stays correct if the push
+                    # architecture is ever reverted. Unreachable today — see the
+                    # unconditional early return above.
+                    from utils.http_instrumentation import outbound_span, TARGET_AGENT
+
+                    push_url = f"{agent_url}/api/config/update"
+                    async with outbound_span(
+                        target=TARGET_AGENT, method="POST", url=push_url, request_body=payload
+                    ) as span:
+                        async with session.post(push_url, json=payload) as response:
+                            if response.status == 200:
+                                span.set_response(response.status, getattr(response, "headers", None))
+                                results.append({
+                                    'node': agent['name'],
+                                    'success': True,
+                                    'message': f'Configuration updated successfully',
+                                    'version': version_name
+                                })
+                                logger.info(f"✅ Agent {agent['name']} notified successfully")
+                            else:
+                                error_text = await response.text()
+                                span.set_response(response.status, getattr(response, "headers", None), error_text)
+                                results.append({
+                                    'node': agent['name'],
+                                    'success': False,
+                                    'error': f'HTTP {response.status}: {error_text}'
+                                })
+                                logger.error(f"❌ Agent {agent['name']} notification failed: {response.status}")
                             
             except asyncio.TimeoutError:
                 results.append({

backend/config.py

@@ -37,4 +37,65 @@ AGENT_CONFIG_SYNC_INTERVAL_SECONDS = 30
 
 # Entity snapshot enabled by default (rollback functionality)
 # Set to "false" only if you need to disable snapshot temporarily
-ENTITY_SNAPSHOT_ENABLED = os.getenv("ENTITY_SNAPSHOT_ENABLED", "true").lower() == "true" 
+ENTITY_SNAPSHOT_ENABLED = os.getenv("ENTITY_SNAPSHOT_ENABLED", "true").lower() == "true"
+
+
+# ---------------------------------------------------------------------------
+# v1.11.0 — unified request/response log
+# ---------------------------------------------------------------------------
+# These four are deliberately ENV-only (not database settings): they decide
+# whether the middleware is even registered and how much memory the writer
+# queue may hold, so they must be resolvable before the DB pool exists.
+# Everything the operator tunes at runtime (retention, body capture, sampling,
+# excluded paths) lives in `system_settings` under the `requestlog.` category
+# and is editable from Settings → Request Log.
+
+def _bool_env(name: str, default: bool) -> bool:
+    raw = os.getenv(name)
+    if raw is None:
+        return default
+    return raw.strip().lower() not in ("0", "false", "no", "off", "")
+
+
+def _int_env(name: str, default: int, minimum: int, maximum: int) -> int:
+    """Read an int env var, clamped. A malformed value falls back to the
+    default rather than crashing the process at import time."""
+    raw = os.getenv(name)
+    if raw is None or not raw.strip():
+        return default
+    try:
+        value = int(raw.strip())
+    except (TypeError, ValueError):
+        return default
+    return max(minimum, min(maximum, value))
+
+
+# Hard kill-switch. When false the logging middleware is never added to the
+# ASGI stack and neither the writer nor the prune task is started — literally
+# zero overhead, not even a settings lookup.
+REQUEST_LOG_ENABLED = _bool_env("REQUEST_LOG_ENABLED", True)
+# Per-worker in-process queue depth. When full, rows are DROPPED (counted, and
+# reported through GET /api/request-logs/stats) — the request path never blocks
+# on the database.
+REQUEST_LOG_QUEUE_MAX = _int_env("REQUEST_LOG_QUEUE_MAX", 2000, 100, 100000)
+# HARD memory ceiling for the same queue, per worker. The row count above does
+# NOT bound memory on its own, because how much a row weighs is an operator
+# setting: `requestlog.max_body_bytes` is editable from Settings and its stated
+# ceiling is 256 KB, which a row can carry twice (request + response). Measured
+# on the real dataclass, the two limits multiply out to:
+#
+#     defaults (2 000 rows x 8 KB)          33.9 MiB   3.3% of the 1 GiB pod limit
+#     max_body_bytes at its 256 KB ceiling   1003 MiB   at the pod limit
+#     REQUEST_LOG_QUEUE_MAX at its ceiling   1695 MiB   over the pod limit
+#
+# Both of those are reachable from documented, in-range values, and the drop
+# warning used to advise raising the queue - so following the tool's own advice
+# could OOM the worker. Whichever limit is hit FIRST now stops the queue, so
+# memory stays bounded no matter what the other is set to.
+REQUEST_LOG_QUEUE_MAX_BYTES = _int_env(
+    "REQUEST_LOG_QUEUE_MAX_BYTES", 64 * 1024 * 1024, 1024 * 1024, 1024 * 1024 * 1024
+)
+# Rows per batched INSERT: one pool acquire per batch, not per request.
+REQUEST_LOG_BATCH_SIZE = _int_env("REQUEST_LOG_BATCH_SIZE", 100, 1, 1000)
+# Max wait before a partial batch is flushed (milliseconds).
+REQUEST_LOG_FLUSH_MS = _int_env("REQUEST_LOG_FLUSH_MS", 500, 50, 10000)

backend/database/migrations.py

@@ -1365,6 +1365,10 @@ async def update_system_roles_to_enterprise_rbac():
                     'roles.read', 'roles.create', 'roles.update', 'roles.delete', 'roles.permissions',
                     'statistics.read', 'statistics.performance', 'statistics.agents', 'statistics.health', 'statistics.export',
                     'activity.read', 'activity.all', 'activity.export',
+                    # v1.11.0 — request/response log. `read` browses the log,
+                    # `manage` edits retention/capture settings and triggers a
+                    # manual purge.
+                    'requestlog.read', 'requestlog.manage',
                     'settings.read', 'settings.update', 'settings.system', 'settings.security',
                     'system.restart', 'system.logs', 'system.database', 'system.services', 'system.emergency'
                 ]
@@ -1384,7 +1388,11 @@ async def update_system_roles_to_enterprise_rbac():
                     'vip.read', 'vip.create', 'vip.update', 'vip.delete', 'vip.apply',
                     'config.read', 'config.update', 'config.download', 'config.history', 'config.bulk_import', 'config.view_request', 'config.download_request',
                     'statistics.read', 'statistics.performance', 'statistics.agents', 'statistics.health',
-                    'activity.read'
+                    'activity.read',
+                    # v1.11.0 — operators debug failing applies and ACME orders,
+                    # so they get read access to the request log; retention and
+                    # purge stay with the admins.
+                    'requestlog.read'
                 ]
             },
             'security_admin': {
@@ -1403,9 +1411,16 @@ async def update_system_roles_to_enterprise_rbac():
                     'config.read', 'config.history', 'config.view_request', 'config.download_request',
                     'statistics.read', 'statistics.performance', 'statistics.agents', 'statistics.health',
                     'activity.read', 'activity.all', 'activity.export',
+                    # v1.11.0 — the request log is a security-forensics surface,
+                    # so the security admin gets both read and retention control.
+                    'requestlog.read', 'requestlog.manage',
                     'settings.read', 'settings.security'
                 ]
             },
+            # NOTE (v1.11.0): `viewer` deliberately gets NEITHER requestlog
+            # permission. Even redacted, captured request/response bodies are a
+            # far broader disclosure surface than the read-only configuration
+            # views a viewer is meant to have.
             'viewer': {
                 'display_name': 'Viewer',
                 'description': 'Read-only access to view configurations, statistics, and monitor system status',
@@ -1764,7 +1779,24 @@ async def ensure_agent_activity_logs_table():
 # adopted instead of retyped. Additive + idempotent; no existing table is altered and no
 # existing row changes. NOTE for the upgrade notes: a SCHEMA_VERSION bump re-seeds the four
 # built-in roles to their defaults, so role customizations are lost on this upgrade.
-SCHEMA_VERSION = 11
+# v1.11.0 (unified request/response log): bumped 11 -> 12 for the brand-new
+# `request_logs` table (ensure_request_logs_table), its retention-settings seed
+# (ensure_request_log_settings), and the new `requestlog.read` /
+# `requestlog.manage` permissions added to the built-in roles in
+# update_system_roles_to_enterprise_rbac().
+#
+# 12, NOT 11. The feature branch was cut when this constant was still 10 and
+# proposed 11, but 11 was taken in the meantime by v1.10.4 (vip_discoveries)
+# above. Landing it as 11 would be silently inert: run_all_migrations() returns
+# early on `applied_version >= SCHEMA_VERSION`, so every database already at 11
+# would skip the whole sequence and get neither the table nor the permissions,
+# while a fresh install would get both. Same class of bug the bump exists to
+# prevent, one number later.
+#
+# Additive + idempotent; no existing table is altered, agents never read this
+# table. Same caveat as v1.10.4: this bump re-seeds the four built-in roles to
+# their defaults, so export role customizations before upgrading.
+SCHEMA_VERSION = 12
 
 
 async def run_all_migrations():
@@ -1908,6 +1940,12 @@ async def _run_all_migrations_inner():
     # ssl_certificates/users, both created above.
     await ensure_ssl_csrs_table()
 
+    # v1.11.0 — unified request/response log: brand-new request_logs table
+    # (no FK targets) plus the seed for its operator-tunable retention
+    # settings. Both are additive and idempotent.
+    await ensure_request_logs_table()
+    await ensure_request_log_settings()
+
     logger.info("Database migrations completed successfully.")
 
 
@@ -1979,6 +2017,162 @@ async def ensure_ssl_csrs_table():
             await close_database_connection(conn)
 
 
+async def ensure_request_logs_table():
+    """v1.11.0 — unified inbound/outbound request/response log.
+
+    Additive only: one brand-new table (request_logs) + indexes. No ALTER of
+    any existing table; agents never read this table.
+
+    Deliberately has NO foreign key on user_id. This is the highest-volume
+    table in the system — one row per API call — and per-insert FK validation
+    is not worth it here; `username` is a denormalized snapshot so a row stays
+    readable after the user who made the request is deleted. That is also the
+    correct audit semantics: the record should outlive the account.
+
+    Fully idempotent (CREATE TABLE/INDEX IF NOT EXISTS). Uses only PostgreSQL
+    9.5+ features (BIGSERIAL, JSONB, partial indexes, varchar_pattern_ops) so
+    there is no server-version floor beyond what the rest of the schema needs.
+    """
+    conn = None
+    try:
+        conn = await get_database_connection()
+
+        await conn.execute("""
+            CREATE TABLE IF NOT EXISTS request_logs (
+                id                  BIGSERIAL PRIMARY KEY,
+                request_id          VARCHAR(64) NOT NULL,
+                direction           VARCHAR(8)  NOT NULL,
+                target              VARCHAR(32),
+                method              VARCHAR(10) NOT NULL,
+                url                 TEXT        NOT NULL,
+                path                VARCHAR(512),
+                query_params        JSONB,
+                status_code         INTEGER,
+                status_class        SMALLINT    NOT NULL DEFAULT 0,
+                duration_ms         INTEGER     NOT NULL DEFAULT 0,
+                user_id             INTEGER,
+                username            VARCHAR(50),
+                client_ip           INET,
+                user_agent          TEXT,
+                request_headers     JSONB,
+                request_body        JSONB,
+                request_body_bytes  INTEGER     NOT NULL DEFAULT 0,
+                response_headers    JSONB,
+                response_body       JSONB,
+                response_body_bytes INTEGER     NOT NULL DEFAULT 0,
+                error               TEXT,
+                truncated           BOOLEAN     NOT NULL DEFAULT FALSE,
+                created_at          TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+                CONSTRAINT request_logs_direction_check
+                    CHECK (direction IN ('inbound', 'outbound'))
+            );
+        """)
+
+        # Indexes run UNCONDITIONALLY on every startup, not only on first
+        # creation (the R16-2 rule established for acme_order_events): an older
+        # deploy that raced ahead of an index would otherwise be stuck doing
+        # sequential scans forever. All are IF NOT EXISTS, so re-running is free.
+
+        # --- read paths: the filters the log viewer actually issues ---
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_created_at "
+            "ON request_logs(created_at DESC);"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_dir_created "
+            "ON request_logs(direction, created_at DESC);"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_status_created "
+            "ON request_logs(status_class, created_at DESC);"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_user_created "
+            "ON request_logs(user_id, created_at DESC) WHERE user_id IS NOT NULL;"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_target_created "
+            "ON request_logs(target, created_at DESC) WHERE target IS NOT NULL;"
+        )
+        # Correlates one inbound row with the outbound calls it caused — this is
+        # what makes "which request went where" readable as a single trace.
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_request_id "
+            "ON request_logs(request_id);"
+        )
+        # Prefix search on path (LIKE 'x%') needs pattern_ops to be usable under
+        # a non-C collation.
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_path_prefix "
+            "ON request_logs(path varchar_pattern_ops);"
+        )
+
+        # --- prune paths: the TTL delete is split by outcome, so a plain
+        # (status_class, created_at) index would still range-scan the half it
+        # is not interested in.
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_prune_ok "
+            "ON request_logs(created_at) WHERE status_class BETWEEN 1 AND 3;"
+        )
+        await conn.execute(
+            "CREATE INDEX IF NOT EXISTS idx_request_logs_prune_err "
+            "ON request_logs(created_at) WHERE status_class = 0 OR status_class >= 4;"
+        )
+
+        logger.info("request_logs table ensured (v1.11.0 request/response log)")
+    except Exception as e:
+        logger.error(f"Error ensuring request_logs table: {e}")
+        # Re-raise (ssl_csrs precedent): this step is part of the
+        # SCHEMA_VERSION=11 bump and the version marker is written only after
+        # the inner sequence completes cleanly. Swallowing here would stamp
+        # version 11 with no request_logs table, and the version gate would
+        # then skip every future retry — permanently.
+        raise
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
+async def ensure_request_log_settings():
+    """v1.11.0 — seed the request/response-log retention defaults.
+
+    Runs UNCONDITIONALLY rather than inside an `if not table_exists:` branch,
+    so an install that already has `system_settings` picks the rows up too.
+    ON CONFLICT DO NOTHING means an operator's tuning is never overwritten by a
+    later upgrade.
+
+    Defaults are mirrored in utils/request_log_settings.py; the pair is pinned
+    by backend/tests/test_request_log_settings.py so they cannot drift apart.
+    """
+    conn = None
+    try:
+        conn = await get_database_connection()
+        await conn.execute("""
+            INSERT INTO system_settings (key, value, category, description) VALUES
+            ('requestlog.enabled', 'true', 'requestlog', 'Master switch for the request/response log'),
+            ('requestlog.capture_inbound', 'true', 'requestlog', 'Log inbound API calls'),
+            ('requestlog.capture_outbound', 'true', 'requestlog', 'Log outbound HTTP calls made by the backend'),
+            ('requestlog.capture_bodies', 'true', 'requestlog', 'Capture redacted, size-capped request/response bodies'),
+            ('requestlog.capture_get', 'true', 'requestlog', 'Log inbound GET requests'),
+            ('requestlog.capture_agent_success', 'false', 'requestlog', 'Log SUCCESSFUL agent polls too (failures are always logged); off by default because the row rate scales with fleet size, not operator activity'),
+            ('requestlog.max_body_bytes', '8192', 'requestlog', 'Per-body capture cap in bytes'),
+            ('requestlog.sample_rate', '1.0', 'requestlog', 'Sampling rate for successful inbound requests (errors always 1.0)'),
+            ('requestlog.exclude_paths', '["/api/request-logs","/api/health","/api/docs","/api/redoc","/api/openapi.json","/.well-known/acme-challenge","/api/agents/heartbeat","/static","/favicon.ico"]', 'requestlog', 'Path prefixes that are never logged'),
+            ('requestlog.success_retention_days', '7', 'requestlog', 'Retention for 1xx/2xx/3xx rows, in days'),
+            ('requestlog.error_retention_days', '30', 'requestlog', 'Retention for 4xx/5xx/transport-error rows, in days'),
+            ('requestlog.max_rows', '500000', 'requestlog', 'Hard row cap; oldest rows are pruned beyond this'),
+            ('requestlog.prune_interval_minutes', '60', 'requestlog', 'Minimum interval between retention prune passes')
+            ON CONFLICT (key) DO NOTHING
+        """)
+        logger.info("request_log retention settings seeded (v1.11.0)")
+    except Exception as e:
+        logger.error(f"Error seeding request_log settings: {e}")
+        raise
+    finally:
+        if conn:
+            await close_database_connection(conn)
+
+
 async def ensure_mfa_columns():
     """Issue #18 — TOTP MFA (v1.6.0): additive columns on users + 3 new tables.
 
@@ -2348,19 +2542,19 @@ async def create_initial_system_data(conn):
                 'name': 'super_admin',
                 'display_name': 'Super Administrator', 
                 'description': 'Full system access with all permissions',
-                'permissions': ["dashboard.read","dashboard.statistics","frontends.read","frontends.create","frontends.update","frontends.delete","backends.read","backends.create","backends.update","backends.delete","waf.read","waf.create","waf.update","waf.delete","ssl.read","ssl.create","ssl.update","ssl.delete","apply.read","apply.execute","agents.read","agents.create","agents.update","agents.delete","clusters.read","clusters.create","clusters.update","clusters.delete","config.read","config.update","config.bulk_import","config.view_request","config.download_request","users.read","users.create","users.update","users.delete","roles.read","roles.create","roles.update","roles.delete"]
+                'permissions': ["dashboard.read","dashboard.statistics","frontends.read","frontends.create","frontends.update","frontends.delete","backends.read","backends.create","backends.update","backends.delete","waf.read","waf.create","waf.update","waf.delete","ssl.read","ssl.create","ssl.update","ssl.delete","apply.read","apply.execute","agents.read","agents.create","agents.update","agents.delete","clusters.read","clusters.create","clusters.update","clusters.delete","config.read","config.update","config.bulk_import","config.view_request","config.download_request","users.read","users.create","users.update","users.delete","roles.read","roles.create","roles.update","roles.delete","requestlog.read","requestlog.manage"]
             },
             {
                 'name': 'operator',
                 'display_name': 'Operator',
                 'description': 'Daily operational access for managing HAProxy configurations',
-                'permissions': ["dashboard.read","dashboard.statistics","frontends.read","frontends.create","frontends.update","backends.read","backends.create","backends.update","waf.read","waf.create","waf.update","ssl.read","ssl.create","ssl.update","apply.read","apply.execute","agents.read","clusters.read","config.read","config.update","config.bulk_import","config.view_request","config.download_request"]
+                'permissions': ["dashboard.read","dashboard.statistics","frontends.read","frontends.create","frontends.update","backends.read","backends.create","backends.update","waf.read","waf.create","waf.update","ssl.read","ssl.create","ssl.update","apply.read","apply.execute","agents.read","clusters.read","config.read","config.update","config.bulk_import","config.view_request","config.download_request","requestlog.read"]
             },
             {
                 'name': 'security_admin',
                 'display_name': 'Security Administrator',
                 'description': 'Security-focused access for WAF rules and SSL certificates',
-                'permissions': ["dashboard.read","frontends.read","backends.read","waf.read","waf.create","waf.update","waf.delete","ssl.read","ssl.create","ssl.update","ssl.delete","apply.read","apply.execute","agents.read","clusters.read","config.read","config.view_request","config.download_request"]
+                'permissions': ["dashboard.read","frontends.read","backends.read","waf.read","waf.create","waf.update","waf.delete","ssl.read","ssl.create","ssl.update","ssl.delete","apply.read","apply.execute","agents.read","clusters.read","config.read","config.view_request","config.download_request","requestlog.read","requestlog.manage"]
             },
             {
                 'name': 'viewer',

backend/haproxy_client.py

@@ -61,14 +61,27 @@ class HAProxyClient:
             if self.stats_username and self.stats_password:
                 auth = aiohttp.BasicAuth(self.stats_username, self.stats_password)
             
+            # v1.11.0: instrumented for completeness. NOTE the CSV body is
+            # deliberately NOT handed to the span — a full stats dump is large,
+            # changes every poll, and has no diagnostic value in an audit row;
+            # status + duration is what matters. `auth` is likewise never logged:
+            # aiohttp.BasicAuth is a NamedTuple whose repr contains the cleartext
+            # password.
+            from utils.http_instrumentation import outbound_span, TARGET_HAPROXY_STATS
+
             async with aiohttp.ClientSession() as session:
-                async with session.get(url, auth=auth, timeout=aiohttp.ClientTimeout(total=10)) as response:
-                    if response.status == 200:
-                        csv_data = await response.text()
-                        return self._parse_csv_stats(csv_data)
-                    else:
-                        logger.warning(f"HTTP stats request failed with status {response.status}")
-                        return self._get_fallback_stats()
+                async with outbound_span(
+                    target=TARGET_HAPROXY_STATS, method="GET", url=url,
+                    capture_body=False, capture_response_body=False,
+                ) as span:
+                    async with session.get(url, auth=auth, timeout=aiohttp.ClientTimeout(total=10)) as response:
+                        span.set_response(response.status, getattr(response, "headers", None))
+                        if response.status == 200:
+                            csv_data = await response.text()
+                            return self._parse_csv_stats(csv_data)
+                        else:
+                            logger.warning(f"HTTP stats request failed with status {response.status}")
+                            return self._get_fallback_stats()
         except Exception as e:
             logger.error(f"HTTP stats request failed: {e}")
             return self._get_fallback_stats()

backend/main.py

@@ -29,7 +29,7 @@ for _vpath in [os.path.join(os.path.dirname(__file__), "version.json"), "/app/ve
 
 # Import configurations and database
 
-from config import CORS_ORIGINS, REDIS_URL, LOG_LEVEL
+from config import CORS_ORIGINS, REDIS_URL, LOG_LEVEL, REQUEST_LOG_ENABLED
 from database.connection import redis_client, get_database_connection, close_database_connection, init_database_pool, close_database_pool
 from database.migrations import run_all_migrations
 
@@ -48,6 +48,7 @@ from routers.site_wizard import router as site_wizard_router
 from routers.mfa import router as mfa_router
 from routers.vip import router as vip_router  # Issue #27 — HA/VIP (Keepalived) management
 from routers.csr import router as csr_router  # v1.9.0 — CSR creation (in-app key+CSR generation, signed-cert import)
+from routers.request_logs import router as request_logs_router  # v1.11.0 — unified request/response log
 
 # Production logging configuration
 from utils.logging_config import setup_production_logging
@@ -56,6 +57,10 @@ from middleware.error_handler import (
     GlobalExceptionHandler, get_error_statistics
 )
 from middleware.activity_logger import log_activity_middleware
+from middleware.request_logger import RequestResponseLogMiddleware  # v1.11.0
+from utils.http_instrumentation import begin_background_trace  # v1.11.0
+from utils.request_log_settings import refresh_config as refresh_request_log_config
+from utils.request_log_sink import request_log_sink
 
 # Setup structured logging
 logger = setup_production_logging(LOG_LEVEL)
@@ -232,6 +237,8 @@ Most operations are **cluster-scoped**:
 async def monitor_agent_status():
     """Background task to monitor agent status and mark offline agents"""
     while True:
+        # v1.11.0: see complete_pending_acme_orders — one id per tick.
+        begin_background_trace("agent_status_monitor")
         try:
             conn = await get_database_connection()
             
@@ -276,6 +283,10 @@ async def complete_pending_acme_orders():
     """
     await asyncio.sleep(60)
     while True:
+        # v1.11.0: one correlation id per TICK, so the outbound rows for this
+        # pass group together and do not merge with every other pass this
+        # process has ever run.
+        begin_background_trace("acme_complete_orders")
         try:
             conn_check = await get_database_connection()
             try:
@@ -645,6 +656,8 @@ async def check_letsencrypt_renewals():
     """
     await asyncio.sleep(120)
     while True:
+        # v1.11.0: see complete_pending_acme_orders — one id per tick.
+        begin_background_trace("acme_renewals")
         conn = None
         try:
             conn = await get_database_connection()
@@ -842,6 +855,41 @@ async def cleanup_stuck_agent_upgrades():
         # Wait 120 seconds (2 minutes) before next check
         await asyncio.sleep(120)
 
+async def prune_request_logs_loop():
+    """v1.11.0 — retention prune for `request_logs`.
+
+    Kept independent of the ACME prune loop on purpose: that one is gated on
+    the `letsencrypt_orders` table existing, which would silently disable this
+    prune on an install that never uses ACME.
+
+    The 5-minute tick is only a heartbeat — the real gate is the DB watermark
+    plus `requestlog.prune_interval_minutes`, so N replicas ticking every 5
+    minutes still produce one pass per configured interval.
+    """
+    # Stagger past startup so migrations and the first request burst are done.
+    await asyncio.sleep(180)
+
+    while True:
+        try:
+            conn = await get_database_connection()
+            try:
+                table_exists = await conn.fetchval("""
+                    SELECT EXISTS (
+                        SELECT 1 FROM information_schema.tables
+                        WHERE table_name = 'request_logs'
+                    )
+                """)
+            finally:
+                await close_database_connection(conn)
+
+            if table_exists:
+                from utils.request_log_prune import prune_request_logs_if_due
+                await prune_request_logs_if_due()
+        except Exception as e:
+            logger.error(f"Error in request_logs prune loop: {e}")
+
+        await asyncio.sleep(300)
+
 # Production middleware stack (order matters!)
 app.add_middleware(PerformanceMonitoringMiddleware, slow_request_threshold_ms=1000)
 app.add_middleware(RequestLoggingMiddleware, exclude_paths=["/api/health/", "/docs", "/redoc"])
@@ -849,15 +897,35 @@ app.add_middleware(RequestLoggingMiddleware, exclude_paths=["/api/health/", "/do
 # Activity logging middleware - must be before CORS
 app.middleware("http")(log_activity_middleware)
 
-# CORS middleware  
+# CORS middleware
 app.add_middleware(
     CORSMiddleware,
     allow_origins=CORS_ORIGINS,
     allow_credentials=True,
     allow_methods=["*"],
     allow_headers=["*"],
+    # v1.11.0: without an explicit expose list, browser JS on a cross-origin
+    # deployment cannot read ANY of these — so an operator could see the
+    # X-Request-ID in devtools but the app could never quote it back. Same-origin
+    # (nginx) deployments were already fine; this fixes the split-origin case.
+    expose_headers=["X-Correlation-ID", "X-Response-Time", "X-Request-ID"],
 )
 
+# v1.11.0 — unified request/response log.
+#
+# MUST be the LAST add_middleware call: Starlette inserts each new middleware at
+# index 0, so the last registration ends up OUTERMOST. Outermost is what we want:
+#   (a) we see the exact status/headers/body the client receives, including the
+#       JSONResponse that RequestLoggingMiddleware fabricates from an exception
+#       it swallowed, and
+#   (b) we seed correlation_id_context BEFORE RequestLoggingMiddleware calls
+#       get_correlation_id(), so X-Correlation-ID matches request_logs.request_id.
+#
+# REQUEST_LOG_ENABLED=false keeps it out of the ASGI stack entirely — not a
+# runtime branch, genuinely zero overhead.
+if REQUEST_LOG_ENABLED:
+    app.add_middleware(RequestResponseLogMiddleware)
+
 # Global exception handlers
 from fastapi.exceptions import RequestValidationError
 from starlette.exceptions import HTTPException as StarletteHTTPException
@@ -894,6 +962,7 @@ app.include_router(agent_router)
 app.include_router(waf_router)
 app.include_router(ssl_router)
 app.include_router(csr_router)  # v1.9.0: CSR creation (in-app key+CSR generation, signed-cert import)
+app.include_router(request_logs_router)  # v1.11.0: unified request/response log
 app.include_router(security_router)
 app.include_router(configuration_router)
 app.include_router(settings_router)
@@ -1031,6 +1100,17 @@ async def startup_event():
         # Decoupled from auto_renew_enabled flag so user-initiated orders also complete.
         asyncio.create_task(complete_pending_acme_orders())
         logger.info("ACME order auto-completion task started (60s checks, replica-safe)")
+
+        # v1.11.0 — request/response log: load the operator's capture/retention
+        # policy, then start the batching writer and the retention prune.
+        # Guarded by the env kill-switch so a deployment that turned the log off
+        # pays for neither task.
+        if REQUEST_LOG_ENABLED:
+            await refresh_request_log_config()
+            asyncio.create_task(request_log_sink.run())
+            logger.info("Request/response log sink started (batching writer)")
+            asyncio.create_task(prune_request_logs_loop())
+            logger.info("Request/response log retention prune task started")
         
         # Create test activity log entry to verify system is working
         try:
@@ -1059,6 +1139,16 @@ async def shutdown_event():
     """Cleanup on shutdown"""
     logger.info("HAProxy OpenManager API shutting down...")
 
+    # v1.11.0: flush queued request-log rows FIRST. The sink's writer is a
+    # `while True` loop, so it can never satisfy the asyncio.wait below — the
+    # rows still sitting in its queue would be lost when the pool closes.
+    try:
+        flushed = await request_log_sink.flush(timeout=3.0)
+        if flushed:
+            logger.info(f"Flushed {flushed} queued request-log row(s)")
+    except Exception as flush_err:
+        logger.warning(f"request-log flush skipped: {flush_err}")
+
     # R18c audit fix (round 3 #5): drain pending fire-and-forget
     # background tasks BEFORE closing the DB pool. The audit
     # logger middleware (`activity_logger.py`) and the wizard

backend/middleware/request_logger.py

@@ -0,0 +1,321 @@
+"""v1.11.0 — inbound half of the unified request/response log.
+
+Pure ASGI on purpose, NOT BaseHTTPMiddleware:
+
+  * `BaseHTTPMiddleware` hands the response back as a
+    `starlette.middleware.base._StreamingResponse`, which has no `.body` to
+    read, and
+  * `await request.body()` inside a `dispatch()` DRAINS the receive channel.
+    `POST /api/agents/heartbeat` (routers/agent.py) reads the raw stream
+    itself, as does the validation-error body preview in
+    middleware/error_handler.py — draining it here would break both.
+
+So we never consume anything: we TEE. `receive` and `send` are wrapped, every
+message is forwarded verbatim, and a size-capped copy is kept for the log row.
+Cost per in-flight request is therefore bounded at ~2 × max_body_bytes (8 KB
+by default), not the size of the upload.
+
+Registration: this MUST be the LAST `app.add_middleware(...)` call, because
+Starlette inserts at index 0 — the last registration is the OUTERMOST
+middleware. Outermost is what we want: we then see the exact status and body
+the client receives (including the JSONResponse that RequestLoggingMiddleware
+fabricates out of a swallowed exception), and we can seed
+`correlation_id_context` before anything downstream reads it.
+"""
+import logging
+import time
+import uuid
+from typing import Any, Dict, List, Optional, Tuple
+
+from starlette.types import ASGIApp, Receive, Scope, Send
+
+from utils.logging_config import correlation_id_context
+from utils.request_log_redaction import is_capturable_content_type, scrub_query_string
+from utils.request_log_settings import get_config
+from utils.request_log_sink import (
+    TARGET_INBOUND_AGENT,
+    RequestLogRow,
+    request_id_context,
+    request_log_sink,
+)
+
+logger = logging.getLogger("haproxy_openmanager.request_log")
+
+# Hard floor, NOT settable away through `requestlog.exclude_paths`. Without it
+# an operator who clears the exclude list turns the log viewer into a machine
+# that logs itself reading its own logs.
+_ALWAYS_EXCLUDED: Tuple[str, ...] = ("/api/request-logs",)
+
+
+def _header(scope: Scope, name: bytes) -> Optional[str]:
+    for key, value in scope.get("headers") or ():
+        if key == name:
+            try:
+                return value.decode("latin-1")
+            except Exception:
+                return None
+    return None
+
+
+def _identify(scope: Scope) -> Tuple[Optional[int], Optional[str]]:
+    """Resolve the caller from the JWT locally — NO database round-trip.
+
+    `log_activity_middleware` already pays a `SELECT ... FROM users` per
+    non-GET request; this middleware runs on every request including GETs, so a
+    second lookup per call is not acceptable. The token issued at
+    routers/auth.py carries both `user_id` and `username`, which is everything
+    the log row needs.
+
+    A token that fails to decode simply yields (None, None): this is a logging
+    path, not an authorization path — the real auth check still runs
+    downstream.
+    """
+    raw = _header(scope, b"authorization")
+    if not raw:
+        return None, None
+    token = raw[7:].strip() if raw.lower().startswith("bearer ") else raw.strip()
+    if not token or token in ("null", "undefined") or token.count(".") != 2:
+        return None, None
+    try:
+        from jose import jwt
+        from config import JWT_SECRET_KEY, JWT_ALGORITHM
+
+        payload = jwt.decode(token, JWT_SECRET_KEY, algorithms=[JWT_ALGORITHM])
+    except Exception:
+        return None, None
+
+    raw_uid = payload.get("user_id") or payload.get("sub")
+    try:
+        user_id = int(raw_uid) if raw_uid is not None else None
+    except (TypeError, ValueError):
+        user_id = None
+    username = payload.get("username")
+    return user_id, (str(username) if username else None)
+
+
+def _is_agent_call(scope: Scope) -> bool:
+    """True when the caller authenticated as an AGENT rather than as a user.
+
+    Every call the installed agent makes carries `X-API-Key` and never an
+    `Authorization` header (linux_install.sh / macos_install.sh: heartbeat,
+    config, pending-requests, upgrade-status, keepalived-*, config-response are
+    all `-H "X-API-Key: $AGENT_TOKEN"`), while the UI carries a JWT and never an
+    agent key. The one endpoint that accepts either -
+    `POST /api/agents/generate-install-script`, used by agent self-upgrade -
+    is correctly classified by the same rule: an operator generating a script
+    sends Authorization, the self-upgrading agent sends only the key.
+
+    Header-only, so it costs two scope reads and no database round-trip.
+    """
+    if _header(scope, b"authorization"):
+        return False
+    return bool(_header(scope, b"x-api-key"))
+
+
+def _client_ip(scope: Scope) -> Optional[str]:
+    """The peer address only.
+
+    `request_logs.client_ip` is an INET column, so a comma-joined
+    X-Forwarded-For string would raise on INSERT (the same trap as
+    `user_activity_logs.ip_address`). The XFF header is still captured — it is
+    on the header allowlist — so the original client is not lost behind a proxy.
+    """
+    client = scope.get("client")
+    if not client:
+        return None
+    try:
+        return str(client[0])
+    except Exception:
+        return None
+
+
+class RequestResponseLogMiddleware:
+    def __init__(self, app: ASGIApp):
+        self.app = app
+
+    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
+        if scope.get("type") != "http":
+            await self.app(scope, receive, send)
+            return
+
+        cfg = get_config()
+        path = scope.get("path", "") or ""
+        method = scope.get("method", "") or ""
+
+        if (
+            not cfg.enabled
+            or not cfg.capture_inbound
+            # OPTIONS never reaches a handler — CORSMiddleware short-circuits
+            # it below us — and a preflight carries no information worth a row.
+            or method == "OPTIONS"
+            or (method == "GET" and not cfg.capture_get)
+            or any(path.startswith(prefix) for prefix in _ALWAYS_EXCLUDED)
+            or any(path.startswith(prefix) for prefix in cfg.exclude_paths)
+        ):
+            await self.app(scope, receive, send)
+            return
+
+        request_id = uuid.uuid4().hex
+        # Seed the id BEFORE the downstream app runs so error_handler's
+        # get_correlation_id() adopts ours instead of minting a second one; the
+        # X-Correlation-ID header then matches request_logs.request_id.
+        cid_token = correlation_id_context.set(request_id[:8])
+        rid_token = request_id_context.set(request_id)
+
+        cap = cfg.max_body_bytes if cfg.capture_bodies else 0
+        req_ctype = _header(scope, b"content-type")
+        req_capturable = is_capturable_content_type(req_ctype)
+
+        req_buf = bytearray()
+        res_buf = bytearray()
+        state = {
+            "req_bytes": 0,
+            "res_bytes": 0,
+            "status": None,
+            "res_headers": {},
+            "res_ctype": None,
+            "res_capturable": True,
+        }
+
+        async def tee_receive() -> Dict[str, Any]:
+            message = await receive()
+            try:
+                if message.get("type") == "http.request":
+                    chunk = message.get("body", b"") or b""
+                    state["req_bytes"] += len(chunk)
+                    if cap and req_capturable and len(req_buf) < cap:
+                        req_buf.extend(chunk[: cap - len(req_buf)])
+            except Exception:
+                pass
+            return message  # forwarded verbatim, always
+
+        async def tee_send(message: Dict[str, Any]) -> None:
+            try:
+                mtype = message.get("type")
+                if mtype == "http.response.start":
+                    state["status"] = message.get("status")
+                    raw_headers: List[Tuple[bytes, bytes]] = message.get("headers") or []
+                    headers = {}
+                    for key, value in raw_headers:
+                        try:
+                            headers[key.decode("latin-1").lower()] = value.decode("latin-1")
+                        except Exception:
+                            continue
+                    state["res_headers"] = headers
+                    state["res_ctype"] = headers.get("content-type")
+                    state["res_capturable"] = is_capturable_content_type(state["res_ctype"])
+                    # Hand the id to the client so a user reporting a problem can
+                    # quote it and an operator can find the exact row.
+                    if isinstance(raw_headers, list):
+                        raw_headers.append((b"x-request-id", request_id.encode("latin-1")))
+                elif mtype == "http.response.body":
+                    chunk = message.get("body", b"") or b""
+                    state["res_bytes"] += len(chunk)
+                    if cap and state["res_capturable"] and len(res_buf) < cap:
+                        res_buf.extend(chunk[: cap - len(res_buf)])
+            except Exception:
+                pass
+            await send(message)  # forwarded verbatim, always
+
+        started = time.perf_counter()
+        error_text: Optional[str] = None
+        try:
+            await self.app(scope, tee_receive, tee_send)
+        except Exception as exc:
+            # Almost never taken: RequestLoggingMiddleware sits below us and
+            # converts exceptions into a JSONResponse first. It IS taken for
+            # paths on that middleware's own exclude list, so the row still has
+            # to be recorded before the exception continues upward.
+            error_text = f"{type(exc).__name__}: {exc}"[:2000]
+            raise
+        finally:
+            duration_ms = int((time.perf_counter() - started) * 1000)
+            try:
+                self._record(
+                    scope=scope,
+                    request_id=request_id,
+                    method=method,
+                    path=path,
+                    duration_ms=duration_ms,
+                    status=state["status"],
+                    req_buf=bytes(req_buf),
+                    req_bytes=state["req_bytes"],
+                    req_ctype=req_ctype,
+                    res_buf=bytes(res_buf),
+                    res_bytes=state["res_bytes"],
+                    res_ctype=state["res_ctype"],
+                    res_headers=state["res_headers"],
+                    error_text=error_text,
+                )
+            except Exception as exc:  # pragma: no cover - defensive
+                logger.debug(f"request_log: failed to record inbound row: {exc}")
+            try:
+                correlation_id_context.reset(cid_token)
+                request_id_context.reset(rid_token)
+            except Exception:
+                pass
+
+    @staticmethod
+    def _record(
+        *,
+        scope: Scope,
+        request_id: str,
+        method: str,
+        path: str,
+        duration_ms: int,
+        status: Optional[int],
+        req_buf: bytes,
+        req_bytes: int,
+        req_ctype: Optional[str],
+        res_buf: bytes,
+        res_bytes: int,
+        res_ctype: Optional[str],
+        res_headers: Dict[str, str],
+        error_text: Optional[str],
+    ) -> None:
+        raw_query = scope.get("query_string") or b""
+        try:
+            query = raw_query.decode("latin-1")
+        except Exception:
+            query = ""
+        scrubbed_query, query_params = scrub_query_string(query)
+
+        user_id, username = _identify(scope)
+
+        req_headers = {}
+        for key, value in scope.get("headers") or ():
+            try:
+                req_headers[key.decode("latin-1").lower()] = value.decode("latin-1")
+            except Exception:
+                continue
+
+        request_log_sink.offer(
+            RequestLogRow(
+                request_id=request_id,
+                direction="inbound",
+                # Who was on the other end. `offer()` uses this to drop
+                # SUCCESSFUL agent polls, which are ~9 800 rows/day per node and
+                # would otherwise make the table's size a function of fleet size.
+                target=TARGET_INBOUND_AGENT if _is_agent_call(scope) else None,
+                method=method,
+                url=path + (("?" + scrubbed_query) if scrubbed_query else ""),
+                path=path,
+                query_string=scrubbed_query or None,
+                query_params=query_params,
+                status_code=status,
+                duration_ms=duration_ms,
+                user_id=user_id,
+                username=username,
+                client_ip=_client_ip(scope),
+                user_agent=req_headers.get("user-agent"),
+                request_headers=req_headers or None,
+                response_headers=res_headers or None,
+                request_body_raw=req_buf or None,
+                request_body_bytes=req_bytes,
+                request_content_type=req_ctype,
+                response_body_raw=res_buf or None,
+                response_body_bytes=res_bytes,
+                response_content_type=res_ctype,
+                error=error_text,
+            )
+        )

backend/routers/agent.py

@@ -2325,16 +2325,41 @@ async def get_agent_keepalived_config(agent_name: str, x_api_key: Optional[str]
         # to write/own-marker-check even on not_configured/teardown.
         agent = await conn.fetchrow("""
             SELECT a.id, a.name, COALESCE(a.enabled, TRUE) AS enabled,
-                   hc.keepalived_config_path
+                   hc.keepalived_config_path,
+                   -- v1.11.1: does the server already hold a discovery for this node? The agent
+                   -- caches the hash of its last discovery report next to the config and skips
+                   -- re-posting while it matches. That cache used to be written even when the
+                   -- POST was REJECTED, so a node could be hidden from the adoption panel for
+                   -- good: the file never changes, so the agent never speaks again. Telling it
+                   -- what we actually hold lets it recover on its own, with no extra request and
+                   -- no one having to touch the node.
+                   EXISTS (SELECT 1 FROM vip_discoveries vd WHERE vd.agent_id = a.id)
+                       AS discovery_known
             FROM agents a
             LEFT JOIN haproxy_clusters hc ON hc.pool_id = a.pool_id
             WHERE a.name = $1
+            -- A pool may hold more than one cluster, and the join then multiplies this row. With
+            -- no ordering the fetch took an arbitrary one, so the keepalived.conf PATH handed to
+            -- the agent was non-deterministic whenever two clusters in a pool disagreed on it:
+            -- the agent would look at the wrong file, find nothing there, and the node would
+            -- never appear for adoption — intermittently, which is the worst way to fail.
+            --
+            -- A CUSTOMISED path wins over the shipped default, then the lowest cluster id. The
+            -- column defaults to '/etc/keepalived/keepalived.conf' rather than NULL, so ordering
+            -- by id alone could have picked a default-valued row over one the operator had
+            -- deliberately set — turning "undefined" into "reliably wrong" for that install.
+            -- When every cluster in the pool carries the default the string is identical, so the
+            -- ordering cannot change what any working deployment already receives.
+            ORDER BY (hc.keepalived_config_path IS NULL
+                      OR hc.keepalived_config_path = '/etc/keepalived/keepalived.conf'),
+                     hc.id
+            LIMIT 1
         """, agent_name)
         if not agent:
             raise HTTPException(status_code=404, detail=f"Agent '{agent_name}' not found")
         config_path = agent['keepalived_config_path'] or '/etc/keepalived/keepalived.conf'
         if not agent['enabled']:
-            return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "keepalived": None}
+            return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "discovery_known": bool(agent["discovery_known"]), "keepalived": None}
 
         row = await conn.fetchrow("""
             SELECT v.id AS vip_id, v.name AS vip_name, v.is_active, v.track_haproxy,
@@ -2350,22 +2375,22 @@ async def get_agent_keepalived_config(agent_name: str, x_api_key: Optional[str]
         """, agent['id'])
 
         if not row:
-            return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "keepalived": None}
+            return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "discovery_known": bool(agent["discovery_known"]), "keepalived": None}
         if not row['is_active']:
             # Soft-deleted VIP → teardown. purge carries the operator's opt-in package removal;
             # the agent still only purges on nodes where IT installed keepalived (install marker).
             return {"agent_name": agent_name, "status": "teardown", "vip_id": row['vip_id'],
-                    "config_path": config_path, "keepalived": None,
+                    "config_path": config_path, "discovery_known": bool(agent["discovery_known"]), "keepalived": None,
                     "purge": bool(row['purge_on_teardown'])}
         if not row['applied_config_content']:
-            return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "keepalived": None}
+            return {"agent_name": agent_name, "status": "not_configured", "config_path": config_path, "discovery_known": bool(agent["discovery_known"]), "keepalived": None}
 
         from services.keepalived_config import build_haproxy_check_script
         check_script = build_haproxy_check_script() if row['track_haproxy'] else ""
         return {
             "agent_name": agent_name,
             "status": "available",
-            "config_path": config_path,
+            "config_path": config_path, "discovery_known": bool(agent["discovery_known"]),
             "keepalived": {
                 "desired_state": "enabled",
                 "install_if_missing": True,
@@ -2433,23 +2458,30 @@ async def agent_keepalived_status(agent_name: str, status_data: dict, x_api_key:
         #
         # Gated on the acked hash MATCHING applied_config_hash, so a partial or failed deploy
         # never drops the authorisation and leaves the VIP unable to converge.
+        # The hash is passed TWICE on purpose. Reusing one placeholder for both the assignment
+        # (`last_deploy_hash=$n`, a VARCHAR column) and the comparison inside the CASE made
+        # PostgreSQL deduce two different types for it and asyncpg refused the whole statement
+        # with AmbiguousParameterError ("text versus character varying"). Because the failure is
+        # in the UPDATE itself, not in one column, EVERY status ack was lost and every VIP sat at
+        # SYNCING forever — including teardown acks. A separate placeholder is only ever compared
+        # against the column, so its type is unambiguous.
         _retire_takeover = ("takeover_expected_hash = CASE WHEN applied_config_hash IS NOT NULL "
-                            "AND {p} = applied_config_hash THEN NULL ELSE takeover_expected_hash END")
+                            "AND applied_config_hash = {p} THEN NULL ELSE takeover_expected_hash END")
         if vip_id is None:
             # No specific VIP (e.g. a teardown ack) — update all this agent's memberships.
             await conn.execute(f"""
                 UPDATE vip_members SET last_deploy_state=$2, last_deploy_message=$3,
                     last_deploy_hash=$4, last_deploy_at=CURRENT_TIMESTAMP, updated_at=CURRENT_TIMESTAMP,
-                    {_retire_takeover.format(p="$4")}
+                    {_retire_takeover.format(p="$5")}
                 WHERE agent_id=$1
-            """, agent['id'], state, message, config_hash)
+            """, agent['id'], state, message, config_hash, config_hash)
         else:
             await conn.execute(f"""
                 UPDATE vip_members SET last_deploy_state=$3, last_deploy_message=$4,
                     last_deploy_hash=$5, last_deploy_at=CURRENT_TIMESTAMP, updated_at=CURRENT_TIMESTAMP,
-                    {_retire_takeover.format(p="$5")}
+                    {_retire_takeover.format(p="$6")}
                 WHERE agent_id=$1 AND vip_id=$2
-            """, agent['id'], int(vip_id), state, message, config_hash)
+            """, agent['id'], int(vip_id), state, message, config_hash, config_hash)
         return {"status": "ok"}
     except HTTPException:
         raise

backend/routers/letsencrypt.py

@@ -926,12 +926,22 @@ async def import_le_ca_chain(authorization: str = Header(None)):
         ("https://letsencrypt.org/certs/r11.pem", "R11 Intermediate"),
     ]
     chain_parts = []
+    # v1.11.0: each download is recorded as an outbound row. The bodies are
+    # public CA certificates, not secrets, and the 8 KB body cap truncates them —
+    # what matters here is which URL failed, with what status.
+    from utils.http_instrumentation import outbound_span, TARGET_LETSENCRYPT_CA
+
     async with aiohttp.ClientSession() as session:
         for url, name in ca_urls:
             try:
-                async with session.get(url, timeout=aiohttp.ClientTimeout(total=10)) as resp:
-                    if resp.status == 200:
-                        chain_parts.append(await resp.text())
+                async with outbound_span(
+                    target=TARGET_LETSENCRYPT_CA, method="GET", url=url
+                ) as span:
+                    async with session.get(url, timeout=aiohttp.ClientTimeout(total=10)) as resp:
+                        text = await resp.text() if resp.status == 200 else None
+                        span.set_response(resp.status, getattr(resp, "headers", None), text)
+                        if resp.status == 200:
+                            chain_parts.append(text)
             except Exception as e:
                 logger.warning(f"Failed to download {name}: {e}")
 

backend/routers/request_logs.py

@@ -0,0 +1,514 @@
+"""v1.11.0 — the read/administration API for the unified request/response log.
+
+Endpoints (declaration order matters — see below):
+
+    GET    /api/request-logs/settings   requestlog.manage
+    PUT    /api/request-logs/settings   requestlog.manage
+    GET    /api/request-logs/stats      requestlog.read
+    POST   /api/request-logs/purge      requestlog.manage
+    GET    /api/request-logs            requestlog.read
+    GET    /api/request-logs/{log_id}   requestlog.read
+
+`/{log_id}` is a single-segment path, so FastAPI — which matches in declaration
+order — would shadow `/settings`, `/stats` and `/purge` if it came first. The
+literals are therefore declared before it. (This is the mirror image of the
+trap in routers/settings.py, where `GET /{category}` sits at the top of the
+file and swallows every literal route added after it.)
+
+Settings are stored in `system_settings` under the `requestlog` category, so
+`GET /api/settings/requestlog` still reads them, but writes go through THIS
+router: the generic `PUT /api/settings/{category}` stringifies values with
+`str(value)`, which turns `True` into `'True'` — not valid JSON, and the
+`::jsonb` cast then fails.
+"""
+import json
+import logging
+from datetime import datetime
+from typing import Any, Dict, List, Optional
+
+from fastapi import APIRouter, Header, HTTPException, Query
+from pydantic import BaseModel, Field, field_validator
+
+from auth_middleware import check_user_permission, get_current_user_from_token
+from database.connection import get_database_connection, close_database_connection
+from utils.request_log_settings import (
+    DEFAULT_CONFIG,
+    DEFAULT_EXCLUDE_PATHS,
+    MAX_EXCLUDE_PATHS,
+    MAX_EXCLUDE_PATH_LENGTH,
+    SETTINGS_CATEGORY,
+    config_from_mapping,
+    get_config,
+    load_settings_rows,
+    refresh_config,
+    set_config,
+)
+from utils.request_log_sink import TARGET_INBOUND_AGENT, request_log_sink
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/api/request-logs", tags=["Request Logs"])
+
+# Columns returned by the list endpoint. Bodies and headers are detail-only:
+# a 200-row page carrying two 8 KB JSONB blobs per row is a 3 MB response.
+_LIST_COLUMNS = """
+    id, request_id, direction, target, method, url, path, status_code,
+    status_class, duration_ms, user_id, username, host(client_ip) AS client_ip,
+    error, request_body_bytes, response_body_bytes, truncated, created_at
+"""
+
+_JSONB_COLUMNS = ("query_params", "request_headers", "request_body",
+                  "response_headers", "response_body")
+
+
+class RequestLogSettings(BaseModel):
+    """Operator-tunable capture + retention policy."""
+
+    enabled: bool = True
+    capture_inbound: bool = True
+    capture_outbound: bool = True
+    capture_bodies: bool = True
+    capture_get: bool = True
+    capture_agent_success: bool = False
+    max_body_bytes: int = Field(8192, ge=0, le=262144)
+    sample_rate: float = Field(1.0, ge=0.0, le=1.0)
+    exclude_paths: List[str] = Field(
+        default_factory=lambda: list(DEFAULT_EXCLUDE_PATHS),
+        max_length=MAX_EXCLUDE_PATHS,
+    )
+    success_retention_days: int = Field(7, ge=1, le=365)
+    error_retention_days: int = Field(30, ge=1, le=365)
+    max_rows: int = Field(500000, ge=1000, le=50_000_000)
+    prune_interval_minutes: int = Field(60, ge=5, le=1440)
+
+    @field_validator("exclude_paths")
+    @classmethod
+    def _validate_paths(cls, value: List[str]) -> List[str]:
+        for entry in value:
+            if not entry.startswith("/"):
+                raise ValueError("exclude_paths entries must start with '/'")
+            if len(entry) > MAX_EXCLUDE_PATH_LENGTH:
+                raise ValueError(
+                    f"exclude_paths entries must be <= {MAX_EXCLUDE_PATH_LENGTH} characters"
+                )
+        return value
+
+
+async def _require(authorization: Optional[str], action: str) -> Dict[str, Any]:
+    """Authenticate, then enforce `requestlog.<action>`.
+
+    `current_user=` is passed through so the admin bypass in
+    check_user_permission short-circuits without a second DB round-trip.
+    """
+    current_user = await get_current_user_from_token(authorization)
+    allowed = await check_user_permission(
+        current_user["id"], "requestlog", action, current_user=current_user
+    )
+    if not allowed:
+        raise HTTPException(
+            status_code=403,
+            detail=f"Insufficient permissions: requestlog.{action} required",
+        )
+    return current_user
+
+
+async def _can_manage(current_user: Dict[str, Any]) -> bool:
+    return await check_user_permission(
+        current_user["id"], "requestlog", "manage", current_user=current_user
+    )
+
+
+def _parse_jsonb(value: Any) -> Any:
+    """asyncpg has no JSONB codec on this pool, so JSONB comes back as raw
+    text. This router is a new contract, so it parses server-side and returns
+    real JSON rather than pushing a JSON.parse() into the UI."""
+    if isinstance(value, str):
+        try:
+            return json.loads(value)
+        except (json.JSONDecodeError, ValueError):
+            return value
+    return value
+
+
+def _row_to_dict(row) -> Dict[str, Any]:
+    out = dict(row)
+    for key in _JSONB_COLUMNS:
+        if key in out:
+            out[key] = _parse_jsonb(out[key])
+    created = out.get("created_at")
+    if isinstance(created, datetime):
+        out["created_at"] = created.isoformat()
+    return out
+
+
+# ---------------------------------------------------------------------------
+# Literal paths FIRST — see the module docstring.
+# ---------------------------------------------------------------------------
+
+
+@router.get("/settings")
+async def get_request_log_settings(authorization: Optional[str] = Header(None)):
+    """Current capture + retention policy, plus the shipped defaults so the UI
+    can offer a 'reset' without hardcoding them."""
+    await _require(authorization, "manage")
+
+    conn = None
+    try:
+        conn = await get_database_connection()
+        values = await load_settings_rows(conn)
+        config = config_from_mapping(values) if values else DEFAULT_CONFIG
+        return {
+            "settings": config.as_dict(),
+            "defaults": DEFAULT_CONFIG.as_dict(),
+            "category": SETTINGS_CATEGORY,
+        }
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error fetching request log settings: {e}")
+        raise HTTPException(status_code=500, detail="Failed to fetch request log settings")
+    finally:
+        if conn is not None:
+            await close_database_connection(conn)
+
+
+@router.put("/settings")
+async def update_request_log_settings(
+    body: RequestLogSettings,
+    authorization: Optional[str] = Header(None),
+):
+    """Persist the policy and apply it immediately.
+
+    `refresh_config()` at the end is what makes an operator's change take
+    effect on the very next request instead of up to 30 seconds later, when
+    the writer loop would otherwise pick it up.
+    """
+    current_user = await _require(authorization, "manage")
+
+    conn = None
+    try:
+        conn = await get_database_connection()
+        updated = []
+        for suffix, value in body.model_dump().items():
+            await conn.execute(
+                """
+                INSERT INTO system_settings (key, value, category, updated_at, updated_by)
+                VALUES ($1, $2::jsonb, $3, $4, $5)
+                ON CONFLICT (key) DO UPDATE SET
+                    value = EXCLUDED.value,
+                    updated_at = EXCLUDED.updated_at,
+                    updated_by = EXCLUDED.updated_by
+                """,
+                f"{SETTINGS_CATEGORY}.{suffix}",
+                json.dumps(value),
+                SETTINGS_CATEGORY,
+                datetime.utcnow(),
+                current_user.get("id"),
+            )
+            updated.append(suffix)
+
+        # Apply in-process right away, then re-read so this worker's snapshot
+        # is exactly what is on disk.
+        set_config(config_from_mapping(body.model_dump()))
+        await refresh_config()
+
+        logger.info(
+            f"Request log settings updated by {current_user.get('username')}: {len(updated)} keys"
+        )
+        return {"message": f"Updated {len(updated)} settings", "settings": get_config().as_dict()}
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error updating request log settings: {e}")
+        raise HTTPException(status_code=500, detail="Failed to update request log settings")
+    finally:
+        if conn is not None:
+            await close_database_connection(conn)
+
+
+@router.get("/stats")
+async def get_request_log_stats(
+    authorization: Optional[str] = Header(None),
+    hours: int = Query(24, ge=1, le=720),
+):
+    """Volume and error breakdown over a window, plus table-level totals and
+    this worker's sink counters (so a saturated queue is visible)."""
+    await _require(authorization, "read")
+
+    conn = None
+    try:
+        conn = await get_database_connection()
+
+        by_direction = await conn.fetch(
+            """
+            SELECT direction,
+                   COUNT(*)                                                    AS total,
+                   COUNT(*) FILTER (WHERE status_class = 0 OR status_class >= 4) AS errors,
+                   COALESCE(ROUND(AVG(duration_ms))::int, 0)                   AS avg_duration_ms,
+                   COALESCE(MAX(duration_ms), 0)                               AS max_duration_ms
+            FROM request_logs
+            WHERE created_at > NOW() - ($1 || ' hours')::INTERVAL
+            GROUP BY direction
+            """,
+            str(hours),
+        )
+
+        by_status = await conn.fetch(
+            """
+            SELECT status_class, COUNT(*) AS total
+            FROM request_logs
+            WHERE created_at > NOW() - ($1 || ' hours')::INTERVAL
+            GROUP BY status_class
+            ORDER BY status_class
+            """,
+            str(hours),
+        )
+
+        by_target = await conn.fetch(
+            """
+            SELECT target,
+                   COUNT(*)                                                    AS total,
+                   COUNT(*) FILTER (WHERE status_class = 0 OR status_class >= 4) AS errors
+            FROM request_logs
+            WHERE target IS NOT NULL
+              AND created_at > NOW() - ($1 || ' hours')::INTERVAL
+            GROUP BY target
+            ORDER BY total DESC
+            LIMIT 20
+            """,
+            str(hours),
+        )
+
+        totals = await conn.fetchrow(
+            "SELECT COUNT(*) AS total_rows, MIN(created_at) AS oldest_at, "
+            "MAX(created_at) AS newest_at FROM request_logs"
+        )
+
+        return {
+            "window_hours": hours,
+            "by_direction": [dict(r) for r in by_direction],
+            "by_status_class": [dict(r) for r in by_status],
+            "by_target": [dict(r) for r in by_target],
+            "total_rows": (totals or {}).get("total_rows", 0),
+            "oldest_at": totals["oldest_at"].isoformat() if totals and totals["oldest_at"] else None,
+            "newest_at": totals["newest_at"].isoformat() if totals and totals["newest_at"] else None,
+            # THIS WORKER only. The sink is a module global, so with
+            # UVICORN_WORKERS > 1 each process keeps its own queue and its own
+            # counters, and whichever worker happens to serve this request is
+            # the one being reported. Labelled rather than aggregated: there is
+            # no cross-process channel here, and a number that looks fleet-wide
+            # but is not would understate drops by exactly the worker count.
+            "sink": {**request_log_sink.stats, "scope": "this worker only"},
+            "retention": {
+                "success_retention_days": get_config().success_retention_days,
+                "error_retention_days": get_config().error_retention_days,
+                "max_rows": get_config().max_rows,
+            },
+        }
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error fetching request log stats: {e}")
+        raise HTTPException(status_code=500, detail="Failed to fetch request log stats")
+    finally:
+        if conn is not None:
+            await close_database_connection(conn)
+
+
+@router.post("/purge")
+async def purge_request_logs(authorization: Optional[str] = Header(None)):
+    """Run a retention pass now, ignoring the watermark.
+
+    This applies the CONFIGURED retention — it is not a 'delete everything'
+    button. It exists so an operator who has just lowered the retention does
+    not have to wait for the next scheduled pass to reclaim the space.
+    """
+    current_user = await _require(authorization, "manage")
+    from utils.request_log_prune import prune_request_logs_if_due
+
+    counts = await prune_request_logs_if_due(force=True)
+    logger.info(f"Manual request log purge by {current_user.get('username')}: {counts}")
+    return {
+        "message": "Retention pass completed",
+        "removed": {
+            "success": counts.get("success", 0),
+            "error": counts.get("error", 0),
+            "overflow": counts.get("overflow", 0),
+        },
+        "ran": bool(counts.get("ran")),
+    }
+
+
+# ---------------------------------------------------------------------------
+# List, then the catch-all detail route LAST.
+# ---------------------------------------------------------------------------
+
+
+@router.get("")
+async def list_request_logs(
+    authorization: Optional[str] = Header(None),
+    direction: Optional[str] = Query(None, pattern="^(inbound|outbound)$"),
+    status_class: Optional[int] = Query(None, ge=0, le=5),
+    method: Optional[str] = Query(None, max_length=10),
+    target: Optional[str] = Query(None, max_length=32),
+    user_id: Optional[int] = Query(None, ge=1),
+    path_prefix: Optional[str] = Query(None, max_length=200),
+    q: Optional[str] = Query(None, max_length=200),
+    request_id: Optional[str] = Query(None, max_length=64),
+    errors_only: bool = Query(False),
+    since: Optional[datetime] = Query(None),
+    until: Optional[datetime] = Query(None),
+    min_duration_ms: Optional[int] = Query(None, ge=0),
+    limit: int = Query(50, ge=1, le=500),
+    offset: int = Query(0, ge=0),
+):
+    """Filtered, server-paginated list. Bodies are not included — use the
+    detail endpoint for those."""
+    current_user = await _require(authorization, "read")
+    can_manage = await _can_manage(current_user)
+
+    where: List[str] = []
+    params: List[Any] = []
+
+    def add(clause_template: str, value: Any) -> None:
+        params.append(value)
+        where.append(clause_template.format(n=len(params)))
+
+    if direction:
+        add("direction = ${n}", direction)
+    if status_class is not None:
+        add("status_class = ${n}", status_class)
+    if method:
+        add("method = ${n}", method.upper())
+    if target:
+        add("target = ${n}", target)
+    if user_id is not None:
+        add("user_id = ${n}", user_id)
+    if path_prefix:
+        add("path LIKE ${n} || '%'", path_prefix)
+    if q:
+        # Substring search has no index to lean on; it is the deliberately slow
+        # filter and should be combined with a time window.
+        add("url ILIKE '%' || ${n} || '%'", q)
+    if request_id:
+        add("request_id = ${n}", request_id)
+    if errors_only:
+        where.append("(status_class = 0 OR status_class >= 4)")
+    if since:
+        add("created_at >= ${n}", since)
+    if until:
+        add("created_at <= ${n}", until)
+    if min_duration_ms is not None:
+        add("duration_ms >= ${n}", min_duration_ms)
+
+    # Self-scoping. Captured bodies are a broader disclosure surface than the
+    # existing activity log, so a caller holding only `requestlog.read` sees
+    # their OWN inbound requests, plus the fleet's. `requestlog.manage` (and the
+    # is_admin bypass inside it) lifts the restriction.
+    #
+    # The agent clause is not a widening for its own sake, it is what makes the
+    # `operator` grant do what the migration says it is for: "operators debug
+    # failing applies and ACME orders, so they get read access to the request
+    # log". An apply fails on the NODE, and the node reports that back over its
+    # own API key - so the row carrying the diagnosis is an agent row with
+    # `user_id IS NULL`, which own-rows-only scoping hid from exactly the role
+    # the grant was written for. Scoped on `target`, not on `user_id IS NULL`:
+    # anonymous inbound traffic (failed logins and their usernames, unauthorised
+    # probes) is NOT agent traffic and stays admin-only.
+    if not can_manage:
+        params.append(current_user["id"])
+        own = f"user_id = ${len(params)}"
+        params.append(TARGET_INBOUND_AGENT)
+        where.append(
+            f"(direction = 'inbound' AND ({own} OR target = ${len(params)}))"
+        )
+
+    where_sql = (" WHERE " + " AND ".join(where)) if where else ""
+
+    conn = None
+    try:
+        conn = await get_database_connection()
+
+        rows = await conn.fetch(
+            f"SELECT {_LIST_COLUMNS} FROM request_logs{where_sql} "
+            f"ORDER BY id DESC LIMIT ${len(params) + 1} OFFSET ${len(params) + 2}",
+            *params, limit, offset,
+        )
+
+        # Bounded count: an unfiltered COUNT(*) over a multi-million-row table
+        # is a sequential scan on every page change. Cap it and tell the client
+        # the number is a floor.
+        count_cap = 10001
+        counted = await conn.fetchval(
+            f"SELECT COUNT(*) FROM (SELECT 1 FROM request_logs{where_sql} LIMIT {count_cap}) t",
+            *params,
+        )
+        total = int(counted or 0)
+
+        return {
+            "logs": [_row_to_dict(r) for r in rows],
+            "total": total,
+            "total_is_estimate": total >= count_cap,
+            "limit": limit,
+            "offset": offset,
+            "scoped_to_self": not can_manage,
+        }
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error listing request logs: {e}")
+        raise HTTPException(status_code=500, detail="Failed to list request logs")
+    finally:
+        if conn is not None:
+            await close_database_connection(conn)
+
+
+@router.get("/{log_id}")
+async def get_request_log(log_id: int, authorization: Optional[str] = Header(None)):
+    """One exchange in full, plus every other row sharing its `request_id`.
+
+    That `related` list is the point of the feature: one inbound API call and
+    the ACME / DNS / agent calls it triggered read as a single trace.
+    """
+    current_user = await _require(authorization, "read")
+    can_manage = await _can_manage(current_user)
+
+    conn = None
+    try:
+        conn = await get_database_connection()
+        row = await conn.fetchrow(
+            "SELECT *, host(client_ip) AS client_ip_text FROM request_logs WHERE id = $1",
+            log_id,
+        )
+        if not row:
+            raise HTTPException(status_code=404, detail="Request log entry not found")
+
+        record = _row_to_dict(row)
+        record["client_ip"] = record.pop("client_ip_text", None)
+
+        if not can_manage and not (
+            record.get("direction") == "inbound"
+            and (
+                record.get("user_id") == current_user["id"]
+                or record.get("target") == TARGET_INBOUND_AGENT
+            )
+        ):
+            # Same self-scoping rule as the list endpoint. 404 rather than 403
+            # so the endpoint does not confirm that a given id exists.
+            raise HTTPException(status_code=404, detail="Request log entry not found")
+
+        related = await conn.fetch(
+            f"SELECT {_LIST_COLUMNS} FROM request_logs "
+            "WHERE request_id = $1 AND id <> $2 ORDER BY id ASC LIMIT 100",
+            record["request_id"], log_id,
+        )
+
+        return {"log": record, "related": [_row_to_dict(r) for r in related]}
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error fetching request log {log_id}: {e}")
+        raise HTTPException(status_code=500, detail="Failed to fetch request log entry")
+    finally:
+        if conn is not None:
+            await close_database_connection(conn)

backend/routers/settings.py

@@ -163,26 +163,35 @@ async def test_acme_connection(authorization: str = Header(None), directory_url:
     _KNOWN_ACME_FIELDS = ["newNonce", "newAccount", "newOrder", "newAuthz", "revokeCert", "keyChange"]
     try:
         import aiohttp
+        # v1.11.0: this handler returns str(e) to the caller and logs nothing —
+        # the span gives the failed probe a durable record.
+        from utils.http_instrumentation import outbound_span, TARGET_SETTINGS_PROBE
+
         async with aiohttp.ClientSession(connector=safe_connector()) as session:
-            async with session.get(
-                directory_url,
-                timeout=aiohttp.ClientTimeout(total=10),
-                allow_redirects=False,
-            ) as resp:
-                if resp.status == 200:
-                    data = await resp.json(content_type=None)
-                    if not isinstance(data, dict):
-                        return {"success": False, "error": "Directory URL did not return a JSON object"}
-                    present = [k for k in _KNOWN_ACME_FIELDS if k in data]
-                    if not present:
-                        return {"success": False, "error": "Response is not a valid ACME directory"}
-                    return {
-                        "success": True,
-                        "directory": directory_url,
-                        "endpoints": present,
-                    }
-                else:
-                    return {"success": False, "error": f"HTTP {resp.status} from directory URL"}
+            async with outbound_span(
+                target=TARGET_SETTINGS_PROBE, method="GET", url=directory_url
+            ) as span:
+                async with session.get(
+                    directory_url,
+                    timeout=aiohttp.ClientTimeout(total=10),
+                    allow_redirects=False,
+                ) as resp:
+                    if resp.status == 200:
+                        data = await resp.json(content_type=None)
+                        span.set_response(resp.status, getattr(resp, "headers", None), data)
+                        if not isinstance(data, dict):
+                            return {"success": False, "error": "Directory URL did not return a JSON object"}
+                        present = [k for k in _KNOWN_ACME_FIELDS if k in data]
+                        if not present:
+                            return {"success": False, "error": "Response is not a valid ACME directory"}
+                        return {
+                            "success": True,
+                            "directory": directory_url,
+                            "endpoints": present,
+                        }
+                    else:
+                        span.set_response(resp.status, getattr(resp, "headers", None))
+                        return {"success": False, "error": f"HTTP {resp.status} from directory URL"}
     except HTTPException:
         raise
     except Exception as e:

backend/services/acme_diagnostics.py

@@ -433,53 +433,84 @@ async def check_port80(domains: List[str], *, http_timeout: float = 5.0) -> Dict
                 continue
             url = f"http://{d}/.well-known/acme-challenge/diagnostic-probe"
             try:
+                # v1.11.0: recorded as an outbound row so a failing port-80 probe is
+                # diagnosable after the fact, not only while the panel is open.
+                #
+                # MERGE NOTE: the feature branch instrumented a `session.head(...)`
+                # probe, which is what this function did when that branch was cut.
+                # It has since become a GET, because the status code alone cannot
+                # tell a working challenge endpoint from a SPA catch-all serving
+                # index.html with HTTP 200 (v1.10.x). Reverting to HEAD to gain the
+                # log row would put that bug straight back, so the GET probe below
+                # is the one that is wrapped.
+                #
                 # GET, not HEAD: the status code alone cannot tell a working challenge
                 # endpoint from a SPA. A reverse proxy that has lost its
                 # /.well-known/acme-challenge/ location falls through to its catch-all
                 # and serves index.html with HTTP 200 — which the old
                 # `status in (200, 404)` rule accepted as healthy while every real
                 # validation failed. Only the body distinguishes them.
-                async with session.get(url, allow_redirects=False) as resp:
-                    # The body is EVIDENCE, not a precondition. If it cannot be read —
-                    # connection reset mid-response, a server that hangs after headers —
-                    # fall back to the status-only semantics this check has always had
-                    # rather than turning a healthy 404 into a hard failure. The stricter
-                    # rule below applies only when there is something to judge.
-                    try:
-                        body = await resp.content.read(_PROBE_BODY_LIMIT)
-                    except Exception:
-                        body = None
-                    content_type = (resp.headers.get("content-type") or "").split(";")[0].strip()
-                    body_class = (
-                        _classify_probe_body(body, content_type) if body is not None else "unread"
-                    )
-                    target = {
-                        "domain": d,
-                        "status": resp.status,
-                        "content_type": content_type or None,
-                        "body_len": len(body) if body is not None else None,
-                        "body_class": body_class,
-                    }
-                    if resp.status == 200 and body_class == "html":
-                        # Reachable, wrong responder. Reported as a warning rather than
-                        # a failure: this check probes the PUBLIC domain and cannot see
-                        # the challenge backend, so it is evidence, not a verdict — and
-                        # a new `fail` here would block the site wizard on upgrade day
-                        # for every install.
-                        target["warn"] = True
-                        target["diagnosis"] = (
-                            "responded 200 with an HTML page, not a challenge token — "
-                            "the request is reaching a web UI instead of the ACME endpoint"
+                from utils.http_instrumentation import outbound_span, TARGET_ACME_DIAG
+
+                async with outbound_span(
+                    target=TARGET_ACME_DIAG, method="GET", url=url, capture_body=False
+                ) as span:
+                    async with session.get(url, allow_redirects=False) as resp:
+                        # The body is EVIDENCE, not a precondition. If it cannot be read —
+                        # connection reset mid-response, a server that hangs after headers —
+                        # fall back to the status-only semantics this check has always had
+                        # rather than turning a healthy 404 into a hard failure. The stricter
+                        # rule below applies only when there is something to judge.
+                        try:
+                            body = await resp.content.read(_PROBE_BODY_LIMIT)
+                        except Exception:
+                            body = None
+                        content_type = (resp.headers.get("content-type") or "").split(";")[0].strip()
+                        body_class = (
+                            _classify_probe_body(body, content_type) if body is not None else "unread"
                         )
-                    elif resp.status in (301, 302, 303, 307, 308):
-                        target["warn"] = True
-                        target["redirect_location"] = resp.headers.get("location")
-                        target["diagnosis"] = (
-                            "redirected instead of serving the challenge path"
+                        target = {
+                            "domain": d,
+                            "status": resp.status,
+                            "content_type": content_type or None,
+                            "body_len": len(body) if body is not None else None,
+                            "body_class": body_class,
+                        }
+                        # The VERDICT goes in the log row, not the body. `body` here is
+                        # up to 64 KB of a third party's page (`_PROBE_BODY_LIMIT`);
+                        # storing it would put an arbitrary remote document into the
+                        # audit table for every probed domain. The classification is
+                        # what an operator reads back anyway.
+                        span.set_response(
+                            resp.status,
+                            getattr(resp, "headers", None),
+                            {
+                                "probe": "acme-http01",
+                                "content_type": content_type or None,
+                                "body_len": len(body) if body is not None else None,
+                                "body_class": body_class,
+                            },
                         )
-                    else:
-                        target["ok"] = resp.status in (200, 404)
-                    targets.append(target)
+                        if resp.status == 200 and body_class == "html":
+                            # Reachable, wrong responder. Reported as a warning rather than
+                            # a failure: this check probes the PUBLIC domain and cannot see
+                            # the challenge backend, so it is evidence, not a verdict — and
+                            # a new `fail` here would block the site wizard on upgrade day
+                            # for every install.
+                            target["warn"] = True
+                            target["diagnosis"] = (
+                                "responded 200 with an HTML page, not a challenge token — "
+                                "the request is reaching a web UI instead of the ACME endpoint"
+                            )
+                        elif resp.status in (301, 302, 303, 307, 308):
+                            target["warn"] = True
+                            target["redirect_location"] = resp.headers.get("location")
+                            target["diagnosis"] = (
+                                "redirected instead of serving the challenge path"
+                            )
+                        else:
+                            target["ok"] = resp.status in (200, 404)
+                        targets.append(target)
             except asyncio.TimeoutError:
                 targets.append({"domain": d, "error": "egress timeout", "warn": True})
                 skip_reason = "egress timeout"

backend/services/acme_service.py

@@ -75,16 +75,23 @@ class ACMEService:
         from utils.ssrf_guard import assert_public_url, safe_connector
         await assert_public_url(directory_url)
 
-        async with aiohttp.ClientSession(connector=safe_connector()) as session:
-            async with session.get(directory_url, timeout=aiohttp.ClientTimeout(total=15), allow_redirects=False) as resp:
-                if resp.status != 200:
-                    raise Exception(f"Failed to fetch ACME directory: HTTP {resp.status}")
-                data = await resp.json()
-                if 'Replay-Nonce' in resp.headers:
-                    self._nonce_by_dir[directory_url] = resp.headers['Replay-Nonce']
-                data['_fetched_at'] = time.time()
-                self._directory_cache[directory_url] = data
-                return data
+        # v1.11.0: recorded in request_logs as an outbound call so an operator
+        # can see exactly which CA was contacted and what it answered.
+        from utils.http_instrumentation import outbound_span, TARGET_ACME
+
+        async with outbound_span(target=TARGET_ACME, method="GET", url=directory_url) as span:
+            async with aiohttp.ClientSession(connector=safe_connector()) as session:
+                async with session.get(directory_url, timeout=aiohttp.ClientTimeout(total=15), allow_redirects=False) as resp:
+                    if resp.status != 200:
+                        span.set_response(resp.status, dict(resp.headers))
+                        raise Exception(f"Failed to fetch ACME directory: HTTP {resp.status}")
+                    data = await resp.json()
+                    span.set_response(resp.status, dict(resp.headers), data)
+                    if 'Replay-Nonce' in resp.headers:
+                        self._nonce_by_dir[directory_url] = resp.headers['Replay-Nonce']
+                    data['_fetched_at'] = time.time()
+                    self._directory_cache[directory_url] = data
+                    return data
 
     async def _get_nonce(self, directory_url: str) -> str:
         # Use a cached nonce for THIS CA only; otherwise fetch a fresh one from THIS CA's newNonce.
@@ -104,9 +111,19 @@ class ACMEService:
         from utils.ssrf_guard import assert_public_url, safe_connector
         nonce_url = directory['newNonce']
         await assert_public_url(nonce_url)
-        async with aiohttp.ClientSession(connector=safe_connector()) as session:
-            async with session.head(nonce_url, timeout=aiohttp.ClientTimeout(total=15), allow_redirects=False) as resp:
-                return resp.headers['Replay-Nonce']
+
+        # v1.11.0: a HEAD with no body and no status check — capture the status
+        # and the allowlisted headers only. `Replay-Nonce` itself is redacted by
+        # the header rules: it is a single-use credential.
+        from utils.http_instrumentation import outbound_span, TARGET_ACME
+
+        async with outbound_span(
+            target=TARGET_ACME, method="HEAD", url=nonce_url, capture_body=False
+        ) as span:
+            async with aiohttp.ClientSession(connector=safe_connector()) as session:
+                async with session.head(nonce_url, timeout=aiohttp.ClientTimeout(total=15), allow_redirects=False) as resp:
+                    span.set_response(resp.status, dict(resp.headers))
+                    return resp.headers['Replay-Nonce']
 
     def _generate_account_key(self) -> Tuple[str, dict]:
         private_key = rsa.generate_private_key(
@@ -208,46 +225,75 @@ class ACMEService:
         from utils.ssrf_guard import assert_public_url, safe_connector
         await assert_public_url(url)
 
+        # v1.11.0: instrument each ATTEMPT separately (the span goes inside the
+        # retry loop, the session stays outside it) so a badNonce retry shows up
+        # as its own row instead of being folded into the successful one.
+        #
+        # capture_body=False is mandatory here. The JWS body is
+        # {protected, payload, signature}: `protected` carries the nonce and the
+        # account kid/jwk, and `signature` is made with the account private key.
+        # The key itself never crosses the wire, but a stored (protected,
+        # signature) pair is a REPLAYABLE ACME credential for the lifetime of the
+        # nonce. We log a description of the request instead of the request.
+        from utils.http_instrumentation import outbound_span, TARGET_ACME
+
         async with aiohttp.ClientSession(connector=safe_connector()) as session:
             for attempt in range(3):
-                async with session.post(
-                    url,
-                    json=body,
-                    headers={"Content-Type": "application/jose+json"},
-                    timeout=aiohttp.ClientTimeout(total=30),
-                    allow_redirects=False,
-                ) as resp:
-                    if 'Replay-Nonce' in resp.headers:
-                        self._nonce_by_dir[directory_url] = resp.headers['Replay-Nonce']
+                jws_summary = {
+                    "jws": True,
+                    "acme_url": protected.get("url"),
+                    "kid_present": bool(protected.get("kid")),
+                    "jwk_present": bool(protected.get("jwk")),
+                    "payload_empty": payload == "",
+                    "attempt": attempt + 1,
+                }
+                async with outbound_span(
+                    target=TARGET_ACME,
+                    method="POST",
+                    url=url,
+                    request_body=jws_summary,
+                    capture_body=False,
+                ) as span:
+                    async with session.post(
+                        url,
+                        json=body,
+                        headers={"Content-Type": "application/jose+json"},
+                        timeout=aiohttp.ClientTimeout(total=30),
+                        allow_redirects=False,
+                    ) as resp:
+                        if 'Replay-Nonce' in resp.headers:
+                            self._nonce_by_dir[directory_url] = resp.headers['Replay-Nonce']
 
-                    if resp.status == 400 and attempt < 2:
-                        err = await resp.json()
-                        etype = (err.get('type') or '')
-                        edetail = (err.get('detail') or '').lower()
-                        # Retry on badNonce, and on any nonce-related malformed rejection (e.g.
-                        # "The Replay Nonce could not be base64url-decoded") — refetch a FRESH nonce
-                        # from the target CA and resign. With per-CA scoping the cross-CA cause is gone;
-                        # this is defense-in-depth so a stale/rejected nonce always self-heals.
-                        if etype.endswith('badNonce') or 'nonce' in edetail:
-                            nonce = resp.headers.get('Replay-Nonce') or await self._get_nonce(directory_url)
-                            protected['nonce'] = nonce
-                            body = self._sign_jws(private_key, protected, payload)
-                            continue
+                        if resp.status == 400 and attempt < 2:
+                            err = await resp.json()
+                            etype = (err.get('type') or '')
+                            edetail = (err.get('detail') or '').lower()
+                            # Retry on badNonce, and on any nonce-related malformed rejection (e.g.
+                            # "The Replay Nonce could not be base64url-decoded") — refetch a FRESH nonce
+                            # from the target CA and resign. With per-CA scoping the cross-CA cause is gone;
+                            # this is defense-in-depth so a stale/rejected nonce always self-heals.
+                            if etype.endswith('badNonce') or 'nonce' in edetail:
+                                span.set_response(resp.status, dict(resp.headers), err)
+                                nonce = resp.headers.get('Replay-Nonce') or await self._get_nonce(directory_url)
+                                protected['nonce'] = nonce
+                                body = self._sign_jws(private_key, protected, payload)
+                                continue
 
-                    resp_data = {}
-                    content_type = resp.headers.get('Content-Type', '')
-                    if 'json' in content_type:
-                        resp_data = await resp.json()
-                    elif resp.status < 300:
-                        text = await resp.text()
-                        if text:
-                            try:
-                                resp_data = json.loads(text)
-                            except json.JSONDecodeError:
-                                resp_data = {"raw": text}
+                        resp_data = {}
+                        content_type = resp.headers.get('Content-Type', '')
+                        if 'json' in content_type:
+                            resp_data = await resp.json()
+                        elif resp.status < 300:
+                            text = await resp.text()
+                            if text:
+                                try:
+                                    resp_data = json.loads(text)
+                                except json.JSONDecodeError:
+                                    resp_data = {"raw": text}
 
-                    headers = dict(resp.headers)
-                    return resp.status, resp_data, headers
+                        headers = dict(resp.headers)
+                        span.set_response(resp.status, headers, resp_data)
+                        return resp.status, resp_data, headers
 
         raise Exception(f"ACME request to {url} failed after retries")
 

backend/services/dns_providers/cloudflare.py

@@ -73,26 +73,42 @@ class CloudflareDNSProvider(DnsProvider):
         """One Cloudflare API call. Returns the parsed JSON body. Raises a SANITIZED
         DnsProviderError on transport/HTTP/API error (never echoes the token or raw headers)."""
         url = f"{CLOUDFLARE_API_BASE}{path}"
+        # v1.11.0: single funnel for every Cloudflare call, so instrumenting here
+        # covers all five logical endpoints. `safe_error_only=True` records only
+        # the exception TYPE — the same stance the handlers below already take,
+        # because a raw message can carry the request URL and through it the zone
+        # identifier. The Authorization header is dropped to a presence marker by
+        # the header allowlist.
+        from utils.http_instrumentation import outbound_span, TARGET_DNS_CLOUDFLARE
+
         try:
-            async with session.request(
-                method, url, headers=self._headers(), allow_redirects=False, **kwargs
-            ) as resp:
-                try:
-                    body = await resp.json()
-                except Exception:  # noqa: BLE001
-                    body = {}
-                if resp.status in (401, 403):
-                    raise DnsProviderError("Cloudflare rejected the API token (check it has Zone:DNS:Edit + Zone:Read).")
-                if resp.status >= 400 or not body.get("success", False):
-                    # Cloudflare returns {"errors":[{"code":..,"message":..}]} — surface only the
-                    # human message text, never the request (which carries the token header).
-                    msgs = "; ".join(
-                        str(e.get("message")) for e in (body.get("errors") or []) if e.get("message")
-                    )
-                    raise DnsProviderError(
-                        f"Cloudflare API error (HTTP {resp.status}){': ' + msgs if msgs else ''}"
-                    )
-                return body
+            async with outbound_span(
+                target=TARGET_DNS_CLOUDFLARE,
+                method=method,
+                url=url,
+                request_body=kwargs.get("json"),
+                safe_error_only=True,
+            ) as span:
+                async with session.request(
+                    method, url, headers=self._headers(), allow_redirects=False, **kwargs
+                ) as resp:
+                    try:
+                        body = await resp.json()
+                    except Exception:  # noqa: BLE001
+                        body = {}
+                    span.set_response(resp.status, getattr(resp, "headers", None), body)
+                    if resp.status in (401, 403):
+                        raise DnsProviderError("Cloudflare rejected the API token (check it has Zone:DNS:Edit + Zone:Read).")
+                    if resp.status >= 400 or not body.get("success", False):
+                        # Cloudflare returns {"errors":[{"code":..,"message":..}]} — surface only the
+                        # human message text, never the request (which carries the token header).
+                        msgs = "; ".join(
+                            str(e.get("message")) for e in (body.get("errors") or []) if e.get("message")
+                        )
+                        raise DnsProviderError(
+                            f"Cloudflare API error (HTTP {resp.status}){': ' + msgs if msgs else ''}"
+                        )
+                    return body
         except DnsProviderError:
             raise
         except aiohttp.ClientError as exc:

backend/services/dns_providers/godaddy.py

@@ -343,30 +343,45 @@ class GoDaddyDNSProvider(DnsProvider):
         request, never a response body verbatim.
         """
         url = f"{GODADDY_API_BASE}{path}"
+        # v1.11.0: single funnel for every GoDaddy call. `safe_error_only=True`
+        # keeps the recorded error to the exception TYPE, matching the stance the
+        # handlers below already take — a raw message can carry the request URL.
+        # The `Authorization: sso-key <key>:<secret>` header never reaches the log:
+        # the header allowlist reduces it to a presence marker.
+        from utils.http_instrumentation import outbound_span, TARGET_DNS_GODADDY
+
         try:
-            async with session.request(
-                method, url, headers=self._headers(), allow_redirects=False, **kwargs
-            ) as resp:
-                try:
-                    # content_type=None: every GoDaddy write answers 200/204 with an EMPTY body, and
-                    # aiohttp would otherwise raise on the missing/other content type before parsing.
-                    body = await resp.json(content_type=None)
-                except ValueError:
-                    # ONLY a decode failure (JSONDecodeError subclasses ValueError) is swallowed —
-                    # an empty write body, or an HTML error page on a >=400. A transport failure
-                    # mid-read (ClientPayloadError, TimeoutError) must NOT land here: it would look
-                    # identical to "empty body", and a caller that reads an RRset would then see
-                    # None and could mistake it for an empty RRset. Those propagate to the handlers
-                    # below and become a real DnsProviderError.
-                    body = None
-                # 2xx only. Redirects are deliberately not followed (aiohttp would forward the
-                # Authorization header), so a 3xx is a failed call — treating `< 400` as success
-                # would report a redirected write as a silent no-op.
-                if 200 <= resp.status < 300:
-                    return body
-                code, message = _error_fields(body)
-                retry_after = _retry_after_seconds(resp.headers, body) if resp.status == 429 else None
-                raise self._http_error(resp.status, code, message, retry_after)
+            async with outbound_span(
+                target=TARGET_DNS_GODADDY,
+                method=method,
+                url=url,
+                request_body=kwargs.get("json"),
+                safe_error_only=True,
+            ) as span:
+                async with session.request(
+                    method, url, headers=self._headers(), allow_redirects=False, **kwargs
+                ) as resp:
+                    try:
+                        # content_type=None: every GoDaddy write answers 200/204 with an EMPTY body, and
+                        # aiohttp would otherwise raise on the missing/other content type before parsing.
+                        body = await resp.json(content_type=None)
+                    except ValueError:
+                        # ONLY a decode failure (JSONDecodeError subclasses ValueError) is swallowed —
+                        # an empty write body, or an HTML error page on a >=400. A transport failure
+                        # mid-read (ClientPayloadError, TimeoutError) must NOT land here: it would look
+                        # identical to "empty body", and a caller that reads an RRset would then see
+                        # None and could mistake it for an empty RRset. Those propagate to the handlers
+                        # below and become a real DnsProviderError.
+                        body = None
+                    span.set_response(resp.status, getattr(resp, "headers", None), body)
+                    # 2xx only. Redirects are deliberately not followed (aiohttp would forward the
+                    # Authorization header), so a 3xx is a failed call — treating `< 400` as success
+                    # would report a redirected write as a silent no-op.
+                    if 200 <= resp.status < 300:
+                        return body
+                    code, message = _error_fields(body)
+                    retry_after = _retry_after_seconds(resp.headers, body) if resp.status == 429 else None
+                    raise self._http_error(resp.status, code, message, retry_after)
         except DnsProviderError:
             raise
         except aiohttp.ClientError as exc:

backend/tests/test_outbound_instrumentation.py

@@ -0,0 +1,327 @@
+"""v1.11.0: every outbound HTTP call is recorded, and instrumentation can never
+become the failure.
+
+Two independent risks:
+
+**Secrets.** The outbound calls carry the most sensitive material in the
+system: the ACME JWS (a replayable signed capability for the lifetime of its
+nonce) and the DNS provider API credentials. Those call sites must opt out of
+request-body capture and out of verbatim error text — the tests below assert
+that at the call site, not just in the helper.
+
+**Availability.** Both DNS provider funnels end in
+`except Exception: raise DnsProviderError("Unexpected ... failure")`, and in
+GoDaddy's publish path that reverts `dns_record_published` and stalls the ACME
+order. So an exception escaping `outbound_span` would be reported to the
+operator as a provider outage. It must never raise — and it must never swallow.
+"""
+import asyncio
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from dataclasses import replace  # noqa: E402
+from unittest.mock import patch  # noqa: E402
+
+from utils import http_instrumentation  # noqa: E402
+from utils import request_log_settings  # noqa: E402
+from utils.http_instrumentation import (  # noqa: E402
+    TARGET_ACME,
+    TARGET_DNS_CLOUDFLARE,
+    TARGET_DNS_GODADDY,
+    outbound_span,
+)
+from utils.request_log_settings import DEFAULT_CONFIG  # noqa: E402
+
+_BACKEND = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _read(*parts):
+    with open(os.path.join(_BACKEND, *parts), encoding="utf-8") as f:
+        return f.read()
+
+
+def _function_body(src, signature):
+    start = src.index(signature)
+    rest = src[start:]
+    # Next def at the same or lower indentation ends the body.
+    end = rest.find("\n    async def ", 1)
+    alt = rest.find("\n    def ", 1)
+    if alt != -1 and (end == -1 or alt < end):
+        end = alt
+    return rest if end == -1 else rest[:end]
+
+
+@pytest.fixture
+def captured(monkeypatch):
+    rows = []
+    monkeypatch.setattr(http_instrumentation.request_log_sink, "offer", rows.append)
+    monkeypatch.setattr(request_log_settings, "_CACHE", DEFAULT_CONFIG)
+    monkeypatch.setattr(http_instrumentation, "get_config", lambda: request_log_settings._CACHE)
+    return rows
+
+
+# --------------------------------------------------------------------------
+# outbound_span behaviour
+# --------------------------------------------------------------------------
+
+def test_records_a_successful_call(captured):
+    async def run():
+        async with outbound_span(target=TARGET_ACME, method="POST",
+                                 url="https://acme-v02.api.letsencrypt.org/acme/new-order") as span:
+            span.set_response(201, {"content-type": "application/json"}, {"status": "pending"})
+
+    asyncio.run(run())
+
+    row = captured[0]
+    assert row.direction == "outbound"
+    assert row.target == TARGET_ACME
+    assert row.method == "POST"
+    assert row.status_code == 201
+    assert row.status_class == 2
+    assert row.response_body_value == {"status": "pending"}
+
+
+def test_exception_is_recorded_and_reraised_unchanged(captured):
+    async def run():
+        async with outbound_span(target=TARGET_ACME, method="GET", url="https://example.com/x"):
+            raise ValueError("connection reset")
+
+    with pytest.raises(ValueError, match="connection reset"):
+        asyncio.run(run())
+
+    row = captured[0]
+    assert row.status_code is None
+    assert row.status_class == 0, (
+        "a call that never got a response must be status_class 0 — the sentinel the "
+        "error-retention window keys off"
+    )
+    assert row.error.startswith("ValueError")
+
+
+def test_safe_error_only_records_the_type_not_the_message(captured):
+    async def run():
+        async with outbound_span(target=TARGET_DNS_GODADDY, method="PUT",
+                                 url="https://api.godaddy.com/v1/domains/example.com/records/TXT/_acme-challenge",
+                                 safe_error_only=True):
+            raise RuntimeError("failed talking to https://api.godaddy.com/v1/domains/secret-zone")
+
+    with pytest.raises(RuntimeError):
+        asyncio.run(run())
+
+    assert captured[0].error == "RuntimeError"
+    assert "secret-zone" not in (captured[0].error or "")
+
+
+def test_instrumentation_failure_never_becomes_a_provider_failure(captured, monkeypatch):
+    """A bug in row construction must not surface to the operator as
+    'Unexpected GoDaddy API failure' and stall an ACME order."""
+    def explode(row):
+        raise RuntimeError("sink is broken")
+
+    monkeypatch.setattr(http_instrumentation.request_log_sink, "offer", explode)
+
+    async def run():
+        async with outbound_span(target=TARGET_DNS_CLOUDFLARE, method="GET",
+                                 url="https://api.cloudflare.com/client/v4/zones") as span:
+            span.set_response(200, {}, {"success": True})
+            return "provider-result"
+
+    assert asyncio.run(run()) == "provider-result", (
+        "a broken sink propagated out of outbound_span; both DNS funnels would convert "
+        "that into DnsProviderError('Unexpected ... failure'), and in GoDaddy's publish "
+        "path that reverts dns_record_published and stalls the ACME order"
+    )
+
+
+def test_block_exception_still_propagates_when_the_sink_is_broken(monkeypatch):
+    monkeypatch.setattr(http_instrumentation.request_log_sink, "offer",
+                        lambda row: (_ for _ in ()).throw(RuntimeError("sink is broken")))
+
+    async def run():
+        async with outbound_span(target=TARGET_ACME, method="GET", url="https://example.com"):
+            raise KeyError("original")
+
+    with pytest.raises(KeyError, match="original"):
+        asyncio.run(run())
+
+
+def test_capture_body_false_stores_the_summary_not_the_payload(captured):
+    async def run():
+        async with outbound_span(
+            target=TARGET_ACME, method="POST", url="https://acme/new-order",
+            request_body={"jws": True, "kid_present": True, "payload_empty": False},
+            capture_body=False,
+        ) as span:
+            span.set_response(200, {}, {"status": "valid"})
+
+    asyncio.run(run())
+
+    row = captured[0]
+    assert row.request_body_value == {"jws": True, "kid_present": True, "payload_empty": False}
+    assert row.request_body_raw is None
+    # The CA's RESPONSE is still captured — that is the half operators need.
+    assert row.response_body_value == {"status": "valid"}
+
+
+def test_urls_are_scrubbed_before_storage(captured):
+    async def run():
+        async with outbound_span(
+            target=TARGET_DNS_CLOUDFLARE, method="GET",
+            url="https://user:hunter2@api.cloudflare.com/client/v4/zones?api_key=abc&page=1",
+        ) as span:
+            span.set_response(200, {}, {})
+
+    asyncio.run(run())
+
+    url = captured[0].url
+    assert "hunter2" not in url
+    assert "abc" not in url
+    assert "page=1" in url
+
+
+def test_outbound_rows_inherit_the_inbound_request_id(captured):
+    from utils.request_log_sink import request_id_context
+
+    async def run():
+        token = request_id_context.set("abc123def456")
+        try:
+            async with outbound_span(target=TARGET_ACME, method="GET", url="https://acme/dir") as span:
+                span.set_response(200, {}, {})
+        finally:
+            request_id_context.reset(token)
+
+    asyncio.run(run())
+
+    assert captured[0].request_id == "abc123def456", (
+        "an outbound call must carry the inbound request's id, otherwise the detail view "
+        "cannot show which API call triggered which CA/DNS call"
+    )
+
+
+def test_background_calls_get_a_task_scoped_id(captured):
+    async def run():
+        async with outbound_span(target=TARGET_ACME, method="GET", url="https://acme/dir") as span:
+            span.set_response(200, {}, {})
+
+    asyncio.run(run())
+    assert captured[0].request_id.startswith("bg:")
+
+
+def test_disabled_outbound_capture_produces_no_row(captured, monkeypatch):
+    monkeypatch.setattr(request_log_settings, "_CACHE",
+                        replace(DEFAULT_CONFIG, capture_outbound=False))
+
+    async def run():
+        async with outbound_span(target=TARGET_ACME, method="GET", url="https://acme/dir") as span:
+            # The call site keeps working — set_response must still be callable.
+            span.set_response(200, {}, {})
+
+    asyncio.run(run())
+    assert captured == []
+
+
+def test_set_response_tolerates_a_response_without_headers(captured):
+    """Some call sites are driven in tests by minimal fakes exposing only
+    `.status`."""
+    async def run():
+        async with outbound_span(target=TARGET_ACME, method="HEAD", url="https://acme/nonce") as span:
+            span.set_response(200, None)
+
+    asyncio.run(run())
+    assert captured[0].status_code == 200
+
+
+# --------------------------------------------------------------------------
+# Call-site coverage
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("path,target", [
+    (("services", "acme_service.py"), "TARGET_ACME"),
+    (("services", "acme_diagnostics.py"), "TARGET_ACME_DIAG"),
+    (("services", "dns_providers", "cloudflare.py"), "TARGET_DNS_CLOUDFLARE"),
+    (("services", "dns_providers", "godaddy.py"), "TARGET_DNS_GODADDY"),
+    (("routers", "letsencrypt.py"), "TARGET_LETSENCRYPT_CA"),
+    (("routers", "settings.py"), "TARGET_SETTINGS_PROBE"),
+    (("haproxy_client.py",), "TARGET_HAPROXY_STATS"),
+    (("agent_notifications.py",), "TARGET_AGENT"),
+])
+def test_every_outbound_module_is_instrumented(path, target):
+    src = _read(*path)
+    assert "outbound_span(" in src, f"{'/'.join(path)} makes HTTP calls but records nothing"
+    assert target in src, f"{'/'.join(path)} does not tag its rows with {target}"
+
+
+def test_acme_signed_request_never_captures_the_jws_body():
+    """The JWS body is {protected, payload, signature}: `protected` carries the
+    nonce and account kid, `signature` is made with the account private key. A
+    stored (protected, signature) pair is a replayable ACME credential."""
+    src = _read("services", "acme_service.py")
+    body = _function_body(src, "    async def _signed_request(")
+
+    assert "capture_body=False" in body, (
+        "the ACME JWS request body would be written to request_logs verbatim — that is a "
+        "replayable signed credential sitting in an audit table"
+    )
+    assert '"jws": True' in body, "no synthetic summary replaces the suppressed JWS body"
+
+
+def test_acme_span_is_inside_the_retry_loop():
+    """The session is built outside `for attempt in range(3)`; the span must be
+    inside it, so a badNonce retry is its own row rather than being folded into
+    the successful attempt."""
+    src = _read("services", "acme_service.py")
+    body = _function_body(src, "    async def _signed_request(")
+
+    loop_at = body.index("for attempt in range(3):")
+    span_at = body.index("async with outbound_span(")
+    assert loop_at < span_at, (
+        "outbound_span wraps the retry loop instead of sitting inside it, so three "
+        "attempts collapse into one log row and a nonce retry becomes invisible"
+    )
+
+
+@pytest.mark.parametrize("path", [
+    ("services", "dns_providers", "cloudflare.py"),
+    ("services", "dns_providers", "godaddy.py"),
+])
+def test_dns_providers_record_error_types_only(path):
+    src = _read(*path)
+    body = _function_body(src, "    async def _request(")
+    assert "safe_error_only=True" in body, (
+        f"{'/'.join(path)} would record the full exception text, which can carry the "
+        f"request URL and through it the tenant/zone identifier"
+    )
+
+
+def test_godaddy_narrow_value_error_handling_is_preserved():
+    """R-round hardening: only a JSON decode failure may be swallowed. Widening
+    it would make a mid-read transport failure look like an empty RRset, and the
+    follow-up full-RRset PUT would then destroy coexisting TXT values."""
+    src = _read("services", "dns_providers", "godaddy.py")
+    body = _function_body(src, "    async def _request(")
+    assert "except ValueError:" in body
+    assert "except Exception:\n                        body = None" not in body
+
+
+def test_acme_diagnostics_keeps_its_ipv4_pinned_connector():
+    """Duplicates an existing assertion on purpose: instrumenting this module
+    must not have refactored the SSRF-guard connector away."""
+    src = _read("services", "acme_diagnostics.py")
+    assert "TCPConnector(family=socket.AF_INET" in src, (
+        "the IPv4 pin was removed from the port-80 probe — that reopens the dual-stack "
+        "AAAA bypass the SSRF guard closes"
+    )
+
+
+def test_haproxy_stats_never_logs_basic_auth_or_the_csv():
+    """aiohttp.BasicAuth is a NamedTuple whose repr contains the cleartext
+    password, and a full stats CSV has no audit value."""
+    src = _read("haproxy_client.py")
+    body = _function_body(src, "    async def _get_stats_via_http(")
+    assert "capture_body=False" in body
+    assert "capture_response_body=False" in body
+    assert "auth=auth" in body and "request_body=auth" not in body

backend/tests/test_request_log_fleet_scale.py

@@ -0,0 +1,327 @@
+"""v1.11.0: the log's cost must follow operator activity, not fleet size.
+
+Every property here was a real defect measured on the feature branch, and each
+one only shows up at scale or at the edge of a setting's documented range, which
+is why none of them were caught by the rule-level tests.
+
+  * one row per API call becomes millions per day once the fleet is a few
+    hundred nodes, and the row cap then evicts the forensic history the feature
+    exists for;
+  * the operator role could not see the rows its grant was written for;
+  * every background call ever made shared one correlation id;
+  * queue memory was a function of an operator-editable setting, not a limit.
+"""
+import asyncio
+import os
+import re
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from dataclasses import replace  # noqa: E402
+
+from utils.http_instrumentation import _correlation_id, begin_background_trace  # noqa: E402
+from utils.request_log_settings import (  # noqa: E402
+    DEFAULT_CONFIG,
+    get_config,
+    set_config,
+)
+from utils.request_log_sink import (  # noqa: E402
+    TARGET_INBOUND_AGENT,
+    RequestLogRow,
+    RequestLogSink,
+    request_id_context,
+)
+
+_BACKEND = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+_ROUTER = os.path.join(_BACKEND, "routers", "request_logs.py")
+_MIDDLEWARE = os.path.join(_BACKEND, "middleware", "request_logger.py")
+
+
+@pytest.fixture(autouse=True)
+def _restore_config():
+    """These tests mutate the module-global snapshot; put it back."""
+    before = get_config()
+    yield
+    set_config(before)
+
+
+def _row(**kw):
+    kw.setdefault("request_id", "a" * 32)
+    kw.setdefault("direction", "inbound")
+    kw.setdefault("method", "GET")
+    kw.setdefault("url", "/api/agents/prod-lb-1/config")
+    kw.setdefault("status_code", 200)
+    return RequestLogRow(**kw)
+
+
+class _CountingSink(RequestLogSink):
+    """Counts what survives `offer()` without needing an event loop."""
+
+    def __init__(self, **kw):
+        super().__init__(kw.pop("maxsize", 10000), 100, 500, **kw)
+        self.accepted = []
+
+    def _ensure_queue(self):
+        sink = self
+
+        class _Q:
+            def put_nowait(self, row):
+                sink.accepted.append(row)
+
+            def qsize(self):
+                return len(sink.accepted)
+
+        return _Q()
+
+
+# --------------------------------------------------------------------------
+# Volume: successful agent polls are not rows
+# --------------------------------------------------------------------------
+
+def test_successful_agent_polls_are_dropped_by_default():
+    """~9 800 rows/day PER AGENT, all of them 200s meaning "nothing changed".
+
+    At 200 nodes that is ~2M rows/day and the 500 000 row cap is reached in
+    about six hours, so the configured "7 days of successes, 30 days of
+    failures" silently becomes about six hours of each — for everything in the
+    table, not just for the agent rows.
+    """
+    assert DEFAULT_CONFIG.capture_agent_success is False, (
+        "the default must be off; on, the table's size is a function of node "
+        "count rather than of anything anyone did"
+    )
+    sink = _CountingSink()
+    for _ in range(100):
+        sink.offer(_row(target=TARGET_INBOUND_AGENT, status_code=200))
+    assert sink.accepted == []
+
+
+@pytest.mark.parametrize("status", [401, 422, 500, None])
+def test_failed_agent_calls_are_always_kept(status):
+    """The half an operator actually needs, and rare enough to be free.
+
+    `None` is a transport error with no HTTP response at all, which
+    status_class reports as 0.
+    """
+    sink = _CountingSink()
+    sink.offer(_row(target=TARGET_INBOUND_AGENT, status_code=status))
+    assert len(sink.accepted) == 1, f"a {status} agent call must be recorded"
+
+
+def test_operator_traffic_is_unaffected_by_the_agent_gate():
+    sink = _CountingSink()
+    sink.offer(_row(target=None, status_code=200, user_id=7))
+    assert len(sink.accepted) == 1
+
+
+def test_the_gate_can_be_turned_on_for_debugging():
+    set_config(replace(get_config(), capture_agent_success=True))
+    sink = _CountingSink()
+    sink.offer(_row(target=TARGET_INBOUND_AGENT, status_code=200))
+    assert len(sink.accepted) == 1
+
+
+def test_agent_traffic_is_identified_by_headers_not_by_a_database_lookup():
+    """The hot path runs on every request; a lookup per call is not affordable.
+
+    The installed agent sends `X-API-Key` and never `Authorization`; the UI
+    sends a JWT and never an agent key.
+    """
+    from middleware.request_logger import _is_agent_call
+
+    def scope(headers):
+        return {"type": "http", "headers": [(k.encode(), v.encode()) for k, v in headers.items()]}
+
+    assert _is_agent_call(scope({"x-api-key": "agt_x"})) is True
+    assert _is_agent_call(scope({"authorization": "Bearer x.y.z"})) is False
+    # generate-install-script accepts either; self-upgrade sends only the key.
+    assert _is_agent_call(scope({"authorization": "Bearer x.y.z", "x-api-key": "agt_x"})) is False
+    assert _is_agent_call(scope({})) is False
+
+
+def test_agent_gate_does_not_reach_for_a_connection():
+    """`offer()` is called from the request coroutine and must stay pure."""
+    src = open(os.path.join(_BACKEND, "utils", "request_log_sink.py"), encoding="utf-8").read()
+    body = src.split("def offer(", 1)[1].split("\n    # -- consumer", 1)[0]
+    for forbidden in ("await ", "get_database_connection", "fetch"):
+        assert forbidden not in body, f"offer() must not {forbidden.strip()!r} — it runs on the hot path"
+
+
+# --------------------------------------------------------------------------
+# Visibility: the operator grant has to mean something
+# --------------------------------------------------------------------------
+
+def test_read_only_scoping_admits_agent_rows_but_not_other_users():
+    """`operator` holds requestlog.read to "debug failing applies" — but an
+    apply fails on the NODE, and the node reports over its own API key, so that
+    row has user_id NULL and own-rows-only scoping hid it.
+
+    Keyed on `target`, NOT on `user_id IS NULL`: anonymous traffic (failed
+    logins and their usernames, unauthenticated probes) is not agent traffic
+    and must stay admin-only.
+    """
+    src = open(_ROUTER, encoding="utf-8").read()
+    clause = re.search(r"if not can_manage:(.*?)where_sql =", src, re.S)
+    assert clause, "the self-scoping block moved; re-check this test"
+    # Code only: the comment above the clause explains what it deliberately
+    # does NOT do, and would otherwise match the negative assertion below.
+    body = "\n".join(
+        line for line in clause.group(1).splitlines()
+        if not line.lstrip().startswith("#")
+    )
+    assert "TARGET_INBOUND_AGENT" in body, "agent rows are still hidden from requestlog.read"
+    assert "user_id IS NULL" not in body, (
+        "scoping on NULL would also expose anonymous traffic, including failed "
+        "logins and the usernames they carry"
+    )
+
+
+def test_detail_endpoint_uses_the_same_scoping_rule_as_the_list():
+    src = open(_ROUTER, encoding="utf-8").read()
+    detail = src.split('@router.get("/{log_id}")', 1)[1]
+    assert "TARGET_INBOUND_AGENT" in detail, (
+        "the detail endpoint would 404 on the very rows the list now shows"
+    )
+
+
+@pytest.mark.parametrize("decorator", [
+    '@router.get("/settings")', '@router.put("/settings")',
+    '@router.get("/stats")', '@router.post("/purge")',
+    '@router.get("")', '@router.get("/{log_id}")',
+])
+def test_permission_is_enforced_before_the_try_block(decorator):
+    """The repo's GHSA-3p5c pattern: a permission check inside `try` gets
+    swallowed by the handler's own `except Exception -> 500`, turning a 403
+    into a server error and, worse, hiding that the check ran at all."""
+    src = open(_ROUTER, encoding="utf-8").read()
+    body = src.split(decorator, 1)[1]
+    body = body.split("\n@router.")[0]
+    require_at = body.find("_require(authorization")
+    try_at = body.find("\n    try:")
+    assert require_at != -1, f"{decorator} does not call _require at all"
+    assert try_at == -1 or require_at < try_at, (
+        f"{decorator} checks permissions INSIDE its try block"
+    )
+
+
+# --------------------------------------------------------------------------
+# Correlation: a trace that groups the wrong rows is worse than no trace
+# --------------------------------------------------------------------------
+
+def test_each_background_pass_gets_its_own_correlation_id():
+    """Nothing in main.py names its tasks, so the old `bg:<task name>` fallback
+    gave one long-lived loop a single id for its entire life — measured, 15
+    ACME calls across 5 ticks came out as 1 id. `related` (LIMIT 100) then
+    presents up to a hundred unrelated calls as this request's trace.
+    """
+    async def loop():
+        per_tick = []
+        for _ in range(5):
+            begin_background_trace("acme_renewals")
+            per_tick.append([_correlation_id() for _ in range(3)])
+            await asyncio.sleep(0)
+        return per_tick
+
+    ticks = asyncio.run(loop())
+    for tick in ticks:
+        assert len(set(tick)) == 1, "calls within one pass must share an id"
+    ids = [t[0] for t in ticks]
+    assert len(set(ids)) == 5, f"passes must not share an id, got {ids}"
+
+
+def test_unwrapped_background_code_does_not_collapse_onto_one_id():
+    """Erring toward too little grouping: a row that stands alone is honest, a
+    row falsely grouped with a hundred others is not."""
+    async def unwrapped():
+        request_id_context.set(None)
+        return [_correlation_id() for _ in range(4)]
+
+    ids = asyncio.run(unwrapped())
+    assert len(set(ids)) == 4
+
+
+def test_the_background_loops_that_make_outbound_calls_open_a_trace():
+    src = open(os.path.join(_BACKEND, "main.py"), encoding="utf-8").read()
+    for loop_name in ("complete_pending_acme_orders", "check_letsencrypt_renewals",
+                      "monitor_agent_status"):
+        body = src.split(f"async def {loop_name}", 1)[1].split("\nasync def ")[0]
+        assert "begin_background_trace(" in body, (
+            f"{loop_name} makes outbound calls but never opens a per-pass trace"
+        )
+
+
+# --------------------------------------------------------------------------
+# Memory: a limit, not a setting
+# --------------------------------------------------------------------------
+
+def test_queue_memory_is_bounded_even_at_the_max_body_size_ceiling():
+    """`max_body_bytes` is editable from Settings and its documented ceiling is
+    256 KB, which a row carries twice. Against the default 2 000-row queue that
+    is ~1 GiB — the entire pod limit — reachable from in-range values.
+    """
+    set_config(replace(get_config(), max_body_bytes=262144, capture_agent_success=True))
+    budget = 8 * 1024 * 1024
+    sink = _CountingSink(maxsize=2000, max_bytes=budget)
+    blob = b"x" * 262144
+    for _ in range(2000):
+        sink.offer(_row(target=None, request_body_raw=blob, response_body_raw=blob))
+
+    held = sum(r.queue_weight() for r in sink.accepted)
+    assert held <= budget, f"queue held {held} bytes against a {budget} byte budget"
+    assert sink.stats["dropped"] > 0, "over-budget rows must be dropped, and counted"
+    unbounded = 2000 * (2 * 262144 + 1400)
+    assert held < unbounded / 10, (
+        f"without the byte budget this queue would hold {unbounded // 1024 // 1024} MiB"
+    )
+
+
+_DRAIN_ROWS = 50
+
+
+def test_the_byte_budget_is_released_as_rows_drain():
+    """A budget that only ever counts up is a slow leak, not a limit.
+
+    Deliberately time-INDEPENDENT. `_collect()` stops at whichever comes first,
+    `batch_size` rows or the `flush_ms` deadline, so a batch size larger than
+    the row count makes the result a function of how fast the runner happens to
+    be. The first version of this test used batch=100/flush=10ms for 50 rows and
+    passed on a native build while failing in CI, which builds
+    linux/amd64 + linux/arm64 and therefore runs one of them under qemu
+    emulation: 25 iterations of `asyncio.wait_for` were enough to exhaust 10 ms
+    there, `_collect()` returned half a batch, and the assertion read
+    `239800 == 0` - measuring the scheduler, not the sink.
+
+    So: batch size EQUAL to the row count, so the loop exits on the count and
+    never consults the deadline; a generous flush window in case it somehow
+    does; and a drain loop rather than a single call. Nothing here depends on
+    wall-clock speed.
+    """
+    async def drain():
+        sink = RequestLogSink(2000, _DRAIN_ROWS, 5000, max_bytes=8 * 1024 * 1024)
+        blob = b"x" * 4096
+        set_config(replace(get_config(), capture_agent_success=True))
+        for _ in range(_DRAIN_ROWS):
+            sink.offer(_row(target=None, request_body_raw=blob, response_body_raw=blob))
+        assert sink.stats["queued_bytes"] > 0, "nothing was queued, so nothing is being measured"
+        assert sink._queue.qsize() == _DRAIN_ROWS, "the queue did not take every row"
+
+        drained = 0
+        while not sink._queue.empty():
+            drained += len(await sink._collect())
+        assert drained == _DRAIN_ROWS, f"drained {drained} of {_DRAIN_ROWS} rows"
+        return sink.stats["queued_bytes"]
+
+    assert asyncio.run(drain()) == 0
+
+
+def test_stats_say_the_sink_counters_are_per_worker():
+    """The sink is a module global; with UVICORN_WORKERS > 1 each process keeps
+    its own. A number that looks fleet-wide but is not understates drops by
+    exactly the worker count."""
+    src = open(_ROUTER, encoding="utf-8").read()
+    assert '"scope"' in src.split('"sink"', 1)[1][:400], (
+        "the stats response must label the sink counters as this-worker-only"
+    )

backend/tests/test_request_log_middleware.py

@@ -0,0 +1,432 @@
+"""v1.11.0: the request/response logger must be invisible to everything below it.
+
+This is the highest-risk piece of the feature. A logging middleware that reads
+the request body the naive way DRAINS the ASGI receive channel, and the handler
+underneath then sees an empty body — `POST /api/agents/heartbeat` reads the raw
+stream itself, so every agent in the fleet would start failing its heartbeat
+because someone wanted nicer logs.
+
+The implementation therefore TEES rather than consumes. These tests drive the
+middleware over a stub ASGI app and assert that property directly: the
+downstream app sees the full body, the client sees the full response, and only
+a capped copy is kept.
+"""
+import asyncio
+import json
+import os
+import re
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from dataclasses import replace  # noqa: E402
+
+from middleware.request_logger import RequestResponseLogMiddleware  # noqa: E402
+from utils.logging_config import correlation_id_context  # noqa: E402
+from utils import request_log_settings  # noqa: E402
+from utils.request_log_settings import DEFAULT_CONFIG  # noqa: E402
+from utils import request_log_sink as sink_module  # noqa: E402
+
+
+_MAIN = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "main.py")
+
+
+@pytest.fixture
+def captured(monkeypatch):
+    """Collect the rows the middleware hands to the sink, instead of writing them."""
+    rows = []
+    monkeypatch.setattr(sink_module.request_log_sink, "offer", rows.append)
+    # The middleware imports `request_log_sink` by value, so patch there too.
+    import middleware.request_logger as rl
+    monkeypatch.setattr(rl.request_log_sink, "offer", rows.append)
+    return rows
+
+
+@pytest.fixture(autouse=True)
+def default_config(monkeypatch):
+    """Every test starts from the shipped defaults, with a small body cap so the
+    truncation paths are exercised without megabyte fixtures."""
+    cfg = replace(DEFAULT_CONFIG, max_body_bytes=1024)
+    monkeypatch.setattr(request_log_settings, "_CACHE", cfg)
+    import middleware.request_logger as rl
+    monkeypatch.setattr(rl, "get_config", lambda: request_log_settings._CACHE)
+    return cfg
+
+
+def set_config(monkeypatch, **overrides):
+    cfg = replace(request_log_settings._CACHE, **overrides)
+    monkeypatch.setattr(request_log_settings, "_CACHE", cfg)
+    return cfg
+
+
+# --------------------------------------------------------------------------
+# A minimal ASGI harness — no TestClient, no HTTP stack, just the protocol.
+# --------------------------------------------------------------------------
+
+async def drive(app, *, method="POST", path="/api/backends", body=b"", query=b"",
+                headers=None, content_type="application/json"):
+    """Run one request through `app` and return (status, headers, body)."""
+    raw_headers = [(b"host", b"testserver")]
+    if content_type:
+        raw_headers.append((b"content-type", content_type.encode()))
+    for k, v in (headers or {}).items():
+        raw_headers.append((k.encode().lower(), v.encode()))
+
+    scope = {
+        "type": "http",
+        "asgi": {"version": "3.0"},
+        "http_version": "1.1",
+        "method": method,
+        "scheme": "http",
+        "path": path,
+        "raw_path": path.encode(),
+        "query_string": query,
+        "root_path": "",
+        "headers": raw_headers,
+        "client": ("10.1.2.3", 51234),
+        "server": ("testserver", 80),
+    }
+
+    # Deliver the body in three chunks so the tee is exercised across messages.
+    chunks = [body[i:i + max(1, len(body) // 3 or 1)] for i in range(0, len(body), max(1, len(body) // 3 or 1))] or [b""]
+    pending = list(chunks)
+
+    async def receive():
+        if pending:
+            chunk = pending.pop(0)
+            return {"type": "http.request", "body": chunk, "more_body": bool(pending)}
+        return {"type": "http.request", "body": b"", "more_body": False}
+
+    sent = {"status": None, "headers": [], "body": b""}
+
+    async def send(message):
+        if message["type"] == "http.response.start":
+            sent["status"] = message["status"]
+            sent["headers"] = message.get("headers", [])
+        elif message["type"] == "http.response.body":
+            sent["body"] += message.get("body", b"") or b""
+
+    await app(scope, receive, send)
+    return sent
+
+
+def echo_length_app(status=200, content_type=b"application/json"):
+    """Stub app that CONSUMES the whole request body and reports its length.
+
+    This is the regression shape: if the middleware drained the stream, the app
+    below it would see 0 bytes.
+    """
+    async def app(scope, receive, send):
+        total = 0
+        while True:
+            message = await receive()
+            total += len(message.get("body", b"") or b"")
+            if not message.get("more_body"):
+                break
+        payload = json.dumps({"received_bytes": total}).encode()
+        await send({"type": "http.response.start", "status": status,
+                    "headers": [(b"content-type", content_type)]})
+        await send({"type": "http.response.body", "body": payload})
+    return app
+
+
+def chunked_app(chunks, content_type=b"application/json"):
+    async def app(scope, receive, send):
+        await send({"type": "http.response.start", "status": 200,
+                    "headers": [(b"content-type", content_type)]})
+        for i, chunk in enumerate(chunks):
+            await send({"type": "http.response.body", "body": chunk,
+                        "more_body": i < len(chunks) - 1})
+    return app
+
+
+# --------------------------------------------------------------------------
+# The transparency guarantees
+# --------------------------------------------------------------------------
+
+def test_request_body_reaches_downstream_intact(captured):
+    """THE regression guard: draining the receive channel would break the raw-body
+    agent heartbeat handler."""
+    body = b"x" * 100_000
+    app = RequestResponseLogMiddleware(echo_length_app())
+
+    sent = asyncio.run(drive(app, body=body))
+
+    assert json.loads(sent["body"])["received_bytes"] == 100_000, (
+        "the handler below the logger saw a different body length than the client sent — "
+        "the middleware consumed the receive channel instead of teeing it"
+    )
+
+
+def test_response_body_reaches_client_intact(captured):
+    chunks = [b'{"part":', b'"one",', b'"n":2}']
+    app = RequestResponseLogMiddleware(chunked_app(chunks))
+
+    sent = asyncio.run(drive(app, method="GET", body=b""))
+
+    assert sent["body"] == b"".join(chunks), "a response chunk was swallowed by the logger"
+    assert sent["status"] == 200
+
+
+def test_only_the_capped_prefix_is_captured(captured):
+    body = b"y" * 100_000
+    app = RequestResponseLogMiddleware(echo_length_app())
+
+    asyncio.run(drive(app, body=body))
+
+    row = captured[0]
+    assert row.request_body_bytes == 100_000, "the on-the-wire size must be recorded in full"
+    assert len(row.request_body_raw) <= 1024, (
+        "the middleware buffered more than max_body_bytes — memory is unbounded per request"
+    )
+
+
+def test_non_capturable_content_type_is_counted_but_not_buffered(captured):
+    app = RequestResponseLogMiddleware(chunked_app([b"\x00\x01\x02" * 500],
+                                                   content_type=b"application/octet-stream"))
+
+    asyncio.run(drive(app, method="GET", content_type=None))
+
+    row = captured[0]
+    assert row.response_body_bytes == 1500
+    assert row.response_body_raw is None, (
+        "a binary response body was buffered — this is what keeps streaming/file "
+        "responses safe"
+    )
+
+
+# --------------------------------------------------------------------------
+# What gets logged, and what does not
+# --------------------------------------------------------------------------
+
+def test_basic_row_fields(captured):
+    app = RequestResponseLogMiddleware(echo_length_app())
+
+    asyncio.run(drive(app, method="POST", path="/api/backends",
+                      body=b'{"name":"web"}', query=b"cluster_id=2&token=secret"))
+
+    row = captured[0]
+    assert row.direction == "inbound"
+    assert row.method == "POST"
+    assert row.path == "/api/backends"
+    assert row.status_code == 200
+    assert row.status_class == 2
+    assert row.client_ip == "10.1.2.3"
+    assert row.duration_ms >= 0
+    # The query string is scrubbed before it is stored, in the URL and the dict.
+    assert "secret" not in row.url
+    assert row.query_params["token"] == "***REDACTED***"
+    assert row.query_params["cluster_id"] == "2"
+
+
+@pytest.mark.parametrize("path", [
+    "/api/health",
+    "/api/health/deep",
+    "/api/docs",
+    "/api/openapi.json",
+    "/.well-known/acme-challenge/abc123",
+    "/api/agents/heartbeat",
+    "/favicon.ico",
+])
+def test_excluded_paths_produce_no_row(captured, path):
+    app = RequestResponseLogMiddleware(echo_length_app())
+    asyncio.run(drive(app, method="GET", path=path))
+    assert captured == [], f"{path} must not be logged by default"
+
+
+def test_log_viewer_path_cannot_be_un_excluded(captured, monkeypatch):
+    """`exclude_paths` is operator-editable, so the viewer's own endpoints have a
+    hard floor — otherwise reading the log generates log entries about reading
+    the log."""
+    set_config(monkeypatch, exclude_paths=())
+
+    app = RequestResponseLogMiddleware(echo_length_app())
+    asyncio.run(drive(app, method="GET", path="/api/request-logs?limit=50"))
+
+    assert captured == [], (
+        "clearing exclude_paths re-enabled logging of the log viewer itself"
+    )
+
+
+def test_options_preflight_is_skipped(captured):
+    app = RequestResponseLogMiddleware(echo_length_app())
+    asyncio.run(drive(app, method="OPTIONS", path="/api/backends"))
+    assert captured == []
+
+
+def test_get_can_be_turned_off(captured, monkeypatch):
+    set_config(monkeypatch, capture_get=False)
+    app = RequestResponseLogMiddleware(echo_length_app())
+
+    asyncio.run(drive(app, method="GET", path="/api/backends"))
+    assert captured == []
+
+    asyncio.run(drive(app, method="POST", path="/api/backends", body=b"{}"))
+    assert len(captured) == 1, "turning GETs off must not silence writes"
+
+
+def test_disabled_config_short_circuits_but_still_serves(captured, monkeypatch):
+    set_config(monkeypatch, enabled=False)
+    app = RequestResponseLogMiddleware(echo_length_app())
+
+    sent = asyncio.run(drive(app, body=b"hello"))
+
+    assert captured == []
+    assert sent["status"] == 200, "the kill-switch must not break request serving"
+
+
+def test_capture_bodies_off_keeps_sizes_but_drops_content(captured, monkeypatch):
+    set_config(monkeypatch, capture_bodies=False)
+    app = RequestResponseLogMiddleware(echo_length_app())
+
+    asyncio.run(drive(app, body=b'{"secret":"x"}'))
+
+    row = captured[0]
+    assert row.request_body_raw is None
+    assert row.request_body_bytes == 14, "size accounting must survive with bodies off"
+
+
+# --------------------------------------------------------------------------
+# Errors and correlation
+# --------------------------------------------------------------------------
+
+def test_exception_is_recorded_as_status_class_zero_and_reraised(captured):
+    async def boom(scope, receive, send):
+        raise RuntimeError("handler exploded")
+
+    app = RequestResponseLogMiddleware(boom)
+
+    with pytest.raises(RuntimeError):
+        asyncio.run(drive(app, method="GET"))
+
+    row = captured[0]
+    assert row.status_code is None
+    assert row.status_class == 0, (
+        "a request that produced no HTTP response must be status_class 0 — that is the "
+        "sentinel the error-retention prune keys off"
+    )
+    assert row.error.startswith("RuntimeError")
+
+
+def test_correlation_id_is_seeded_before_downstream_and_reset_after(captured):
+    seen = {}
+
+    async def app(scope, receive, send):
+        seen["cid"] = correlation_id_context.get()
+        await send({"type": "http.response.start", "status": 204, "headers": []})
+        await send({"type": "http.response.body", "body": b""})
+
+    wrapped = RequestResponseLogMiddleware(app)
+    asyncio.run(drive(wrapped, method="GET"))
+
+    row = captured[0]
+    assert seen["cid"] == row.request_id[:8], (
+        "the downstream error handler would mint its own id, so X-Correlation-ID would "
+        "not match request_logs.request_id"
+    )
+    assert correlation_id_context.get() is None, (
+        "the ContextVar token was not reset — the next request on this task would inherit "
+        "a stale correlation id"
+    )
+
+
+def test_x_request_id_header_is_returned_to_the_client(captured):
+    app = RequestResponseLogMiddleware(echo_length_app())
+    sent = asyncio.run(drive(app, method="GET"))
+
+    names = {k.decode().lower() for k, _ in sent["headers"]}
+    assert "x-request-id" in names, (
+        "without this header a user reporting a problem has no id to quote"
+    )
+
+
+def test_error_responses_are_logged_with_their_status(captured):
+    app = RequestResponseLogMiddleware(echo_length_app(status=422))
+    asyncio.run(drive(app, body=b'{"bad":true}'))
+
+    row = captured[0]
+    assert row.status_code == 422
+    assert row.status_class == 4, "4xx must be classed as an error for retention purposes"
+
+
+def test_jwt_identity_is_resolved_without_a_database(captured):
+    """The middleware runs on every request; a DB lookup per call is not
+    acceptable, so the user is read straight out of the token claims."""
+    from datetime import datetime, timedelta
+
+    from jose import jwt
+    from config import JWT_ALGORITHM, JWT_SECRET_KEY
+
+    token = jwt.encode(
+        {"user_id": 42, "username": "ops", "exp": datetime.utcnow() + timedelta(minutes=10)},
+        JWT_SECRET_KEY, algorithm=JWT_ALGORITHM,
+    )
+
+    app = RequestResponseLogMiddleware(echo_length_app())
+    asyncio.run(drive(app, body=b"{}", headers={"authorization": f"Bearer {token}"}))
+
+    row = captured[0]
+    assert row.user_id == 42
+    assert row.username == "ops"
+
+
+def test_malformed_token_yields_an_anonymous_row(captured):
+    app = RequestResponseLogMiddleware(echo_length_app())
+    asyncio.run(drive(app, body=b"{}", headers={"authorization": "Bearer not.a.jwt"}))
+
+    row = captured[0]
+    assert row.user_id is None
+    assert row.username is None
+    # Logging is not an auth path — a bad token must not turn into an exception.
+
+
+def test_authorization_header_is_never_stored_verbatim(captured):
+    app = RequestResponseLogMiddleware(echo_length_app())
+    asyncio.run(drive(app, body=b"{}", headers={"authorization": "Bearer super-secret"}))
+
+    params = captured[0].to_params()
+    assert "super-secret" not in json.dumps(params, default=str)
+
+
+# --------------------------------------------------------------------------
+# Registration order in main.py
+# --------------------------------------------------------------------------
+
+def test_middleware_is_registered_last_so_it_is_outermost():
+    with open(_MAIN, encoding="utf-8") as f:
+        src = f.read()
+
+    log_at = src.index("app.add_middleware(RequestResponseLogMiddleware)")
+    cors_at = src.index("    CORSMiddleware,")
+
+    assert log_at > cors_at, (
+        "Starlette's add_middleware inserts at index 0, so the LAST registration is the "
+        "OUTERMOST middleware. Registering the request logger before CORS would put it "
+        "inside the stack, where it can no longer see the final client-visible response "
+        "and can no longer seed the correlation id before the error handler reads it."
+    )
+
+
+def test_env_kill_switch_guards_the_registration():
+    with open(_MAIN, encoding="utf-8") as f:
+        src = f.read()
+
+    assert re.search(
+        r"if REQUEST_LOG_ENABLED:\s*\n\s*app\.add_middleware\(RequestResponseLogMiddleware\)",
+        src,
+    ), (
+        "REQUEST_LOG_ENABLED must gate the add_middleware call itself, not a branch inside "
+        "the middleware — the whole point is that a disabled log costs nothing"
+    )
+
+
+def test_cors_exposes_the_request_id_header():
+    with open(_MAIN, encoding="utf-8") as f:
+        src = f.read()
+
+    assert "expose_headers=" in src and "X-Request-ID" in src, (
+        "without expose_headers the browser cannot read X-Request-ID on a cross-origin "
+        "deployment, so the id is unusable from the app"
+    )

backend/tests/test_request_log_migration.py

@@ -0,0 +1,251 @@
+"""v1.11.0: the request_logs migration actually runs on existing installs.
+
+Source-scan tests (the sanctioned pattern here — there is no database in this
+suite). The failure mode being pinned is specific and silent: migrations are
+gated on `applied_version >= SCHEMA_VERSION`, so forgetting the bump means the
+whole sequence is skipped on every already-deployed database and neither the
+table nor the new permissions ever appear — while a fresh install works fine,
+so it looks correct in development.
+"""
+import os
+import re
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+_BACKEND = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+_MIGRATIONS = os.path.join(_BACKEND, "database", "migrations.py")
+_MAIN = os.path.join(_BACKEND, "main.py")
+
+
+@pytest.fixture(scope="module")
+def src():
+    with open(_MIGRATIONS, encoding="utf-8") as f:
+        return f.read()
+
+
+@pytest.fixture(scope="module")
+def runner_body(src):
+    """The body of _run_all_migrations_inner, where steps are registered."""
+    assert "async def _run_all_migrations_inner" in src
+    return src.split("async def _run_all_migrations_inner", 1)[1].split("\nasync def ", 1)[0]
+
+
+@pytest.fixture(scope="module")
+def rbac_body(src):
+    return src.split("async def update_system_roles_to_enterprise_rbac", 1)[1].split("\nasync def ", 1)[0]
+
+
+def _role_block(rbac_body, role):
+    """Slice one role's permission list out of the enterprise_roles literal."""
+    start = rbac_body.index(f"'{role}': {{")
+    end = rbac_body.index("]", rbac_body.index("'permissions': [", start))
+    return rbac_body[start:end]
+
+
+# --------------------------------------------------------------------------
+# The version gate
+# --------------------------------------------------------------------------
+
+def test_schema_version_bumped_to_at_least_11(src):
+    match = re.search(r"^SCHEMA_VERSION\s*=\s*(\d+)", src, re.MULTILINE)
+    assert match, "SCHEMA_VERSION assignment not found in migrations.py"
+    assert int(match.group(1)) >= 11, (
+        "SCHEMA_VERSION was not bumped for the request_logs table. run_all_migrations() "
+        "returns early when the recorded version is already >= SCHEMA_VERSION, so every "
+        "existing deployment would skip the whole run: no request_logs table, no "
+        "requestlog.* permissions, and the feature would silently do nothing in production "
+        "while working perfectly on a fresh database."
+    )
+
+
+# --------------------------------------------------------------------------
+# Registration
+# --------------------------------------------------------------------------
+
+def test_both_migration_steps_are_registered(runner_body):
+    assert "await ensure_request_logs_table()" in runner_body, (
+        "ensure_request_logs_table is defined but never called from the migration runner"
+    )
+    assert "await ensure_request_log_settings()" in runner_body, (
+        "the retention defaults are never seeded, so an upgraded install has no "
+        "requestlog.* rows and Settings shows blanks"
+    )
+
+
+def test_table_is_created_before_its_settings_are_seeded(runner_body):
+    table_at = runner_body.index("await ensure_request_logs_table()")
+    seed_at = runner_body.index("await ensure_request_log_settings()")
+    assert table_at < seed_at, (
+        "the settings seed runs before the table step; if the table step then raises, the "
+        "run aborts with settings but no table"
+    )
+
+
+# --------------------------------------------------------------------------
+# The DDL itself
+# --------------------------------------------------------------------------
+
+@pytest.fixture(scope="module")
+def ddl_body(src):
+    return src.split("async def ensure_request_logs_table", 1)[1].split("\nasync def ", 1)[0]
+
+
+@pytest.mark.parametrize("fragment", [
+    "CREATE TABLE IF NOT EXISTS request_logs",
+    "id                  BIGSERIAL PRIMARY KEY",
+    "request_id          VARCHAR(64) NOT NULL",
+    "direction           VARCHAR(8)  NOT NULL",
+    "status_class        SMALLINT",
+    "created_at          TIMESTAMPTZ NOT NULL DEFAULT NOW()",
+    "request_logs_direction_check",
+    "client_ip           INET",
+])
+def test_ddl_essentials(ddl_body, fragment):
+    assert fragment in ddl_body, f"request_logs DDL is missing {fragment!r}"
+
+
+def test_ddl_is_idempotent(ddl_body):
+    assert "CREATE TABLE IF NOT EXISTS" in ddl_body
+    creates = re.findall(r"CREATE INDEX(?: IF NOT EXISTS)?", ddl_body)
+    assert creates, "no indexes are created for request_logs"
+    assert all(c == "CREATE INDEX IF NOT EXISTS" for c in creates), (
+        "an index is created without IF NOT EXISTS — the second startup would raise and "
+        "abort the whole migration run"
+    )
+
+
+def test_prune_partial_indexes_are_present(ddl_body):
+    """The retention delete is split by outcome, so a plain
+    (status_class, created_at) index would still range-scan the half it does not
+    want."""
+    assert "idx_request_logs_prune_ok" in ddl_body
+    assert "idx_request_logs_prune_err" in ddl_body
+    assert "WHERE status_class BETWEEN 1 AND 3" in ddl_body
+    assert "WHERE status_class = 0 OR status_class >= 4" in ddl_body
+
+
+def test_request_id_index_exists_for_the_trace_view(ddl_body):
+    assert "idx_request_logs_request_id" in ddl_body, (
+        "without this index, opening one request to see the outbound calls it triggered "
+        "is a sequential scan"
+    )
+
+
+def test_no_foreign_key_on_user_id(ddl_body):
+    """Deliberate deviation from the house style — see the docstring in
+    migrations.py. Pinned so it is not 'fixed' back into an FK later."""
+    user_id_line = [line for line in ddl_body.splitlines() if "user_id " in line and "INTEGER" in line]
+    assert user_id_line, "user_id column not found"
+    assert "REFERENCES" not in user_id_line[0], (
+        "an FK was added to request_logs.user_id — per-insert FK validation on the "
+        "highest-volume table in the system, and audit rows must outlive the account"
+    )
+
+
+def test_migration_step_reraises_on_failure(ddl_body):
+    """The version marker is written only after the inner sequence completes, so
+    swallowing here would stamp version 11 with no table and the gate would then
+    skip every retry, permanently."""
+    assert re.search(r"\n\s+raise\n", ddl_body), (
+        "ensure_request_logs_table swallows its exception instead of re-raising"
+    )
+
+
+def test_settings_seed_does_not_overwrite_operator_tuning(src):
+    seed_body = src.split("async def ensure_request_log_settings", 1)[1].split("\nasync def ", 1)[0]
+    assert "ON CONFLICT (key) DO NOTHING" in seed_body, (
+        "the seed uses DO UPDATE, so every upgrade would reset the operator's retention "
+        "settings back to the defaults"
+    )
+
+
+# --------------------------------------------------------------------------
+# Permission seeding
+# --------------------------------------------------------------------------
+
+def test_super_admin_gets_both_permissions(rbac_body):
+    block = _role_block(rbac_body, "super_admin")
+    assert "'requestlog.read'" in block
+    assert "'requestlog.manage'" in block
+
+
+def test_security_admin_gets_both_permissions(rbac_body):
+    block = _role_block(rbac_body, "security_admin")
+    assert "'requestlog.read'" in block
+    assert "'requestlog.manage'" in block
+
+
+def test_operator_gets_read_only(rbac_body):
+    block = _role_block(rbac_body, "operator")
+    assert "'requestlog.read'" in block
+    assert "'requestlog.manage'" not in block, (
+        "operators should be able to read the log to debug an apply or an ACME order, but "
+        "retention policy and purge belong to the admins"
+    )
+
+
+def test_viewer_gets_neither(rbac_body):
+    block = _role_block(rbac_body, "viewer")
+    assert "requestlog" not in block, (
+        "viewer was granted a requestlog permission. Even redacted, captured request and "
+        "response bodies are a far broader disclosure surface than the read-only config "
+        "views a viewer is meant to have."
+    )
+
+
+def test_permission_strings_have_exactly_one_dot(rbac_body):
+    """get_user_permissions splits on the FIRST dot and silently drops any
+    string without one."""
+    for perm in re.findall(r"'(requestlog[^']*)'", rbac_body):
+        assert perm.count(".") == 1, f"{perm!r} is not a <resource>.<action> pair"
+
+
+def test_initial_seed_lists_stay_in_sync(src):
+    """create_initial_system_data() is overwritten by the enterprise seeder on
+    every run, but that seeder swallows all exceptions — keeping the two in sync
+    is the safety net."""
+    initial = src.split("system_roles = [", 1)[1].split("\n        ]", 1)[0]
+    assert '"requestlog.read"' in initial
+    assert '"requestlog.manage"' in initial
+
+
+# --------------------------------------------------------------------------
+# Runtime wiring
+# --------------------------------------------------------------------------
+
+def test_prune_loop_is_started_and_independent_of_the_acme_loop():
+    with open(_MAIN, encoding="utf-8") as f:
+        main_src = f.read()
+
+    assert "async def prune_request_logs_loop" in main_src
+    assert "asyncio.create_task(prune_request_logs_loop())" in main_src, (
+        "the retention prune task is defined but never started, so request_logs grows "
+        "without bound"
+    )
+    loop_body = main_src.split("async def prune_request_logs_loop", 1)[1].split("\n# Production middleware", 1)[0]
+    assert "table_name = 'request_logs'" in loop_body, (
+        "the prune loop does not check for its own table, so it would log an error every "
+        "tick on a database where the migration has not run yet"
+    )
+    assert "table_name = 'letsencrypt_orders'" not in loop_body, (
+        "the prune loop was gated on the ACME table, which would disable retention "
+        "entirely on an install that never uses ACME"
+    )
+
+
+def test_sink_is_flushed_before_the_pool_closes():
+    with open(_MAIN, encoding="utf-8") as f:
+        main_src = f.read()
+
+    body = main_src.split("async def shutdown_event", 1)[1]
+    flush_at = body.find("request_log_sink.flush")
+    close_at = body.find("close_database_pool()")
+    assert flush_at != -1, "queued request-log rows are never flushed on shutdown"
+    assert flush_at < close_at, (
+        "the sink is flushed after the pool is closed, so the queued rows are lost. The "
+        "sink's writer is a `while True` loop and can never satisfy the generic "
+        "asyncio.wait drain, so it needs its own explicit flush first."
+    )

backend/tests/test_request_log_prune.py

@@ -0,0 +1,261 @@
+"""v1.11.0: retention actually reclaims space, and cannot be turned into an
+injection point or a 60-second lock.
+
+`request_logs` is the highest-volume table in the system, so the prune has
+three properties that are easy to get wrong and expensive to get wrong:
+
+  * the operator-supplied retention day counts are BIND PARAMETERS, never
+    string-interpolated into the SQL;
+  * deletes are BATCHED, because the pool's command_timeout is 60s and an
+    unbounded DELETE over millions of rows raises and then nothing is ever
+    pruned;
+  * the watermark is stamped only after a COMPLETE pass, so a pass that dies
+    half-way is retried instead of being recorded as done.
+
+The fake connection dispatches on the SQL text rather than on call order — an
+ordered side_effect list silently passes tests for the wrong reason as soon as
+the number of statements changes.
+"""
+import asyncio
+import json
+import os
+import sys
+from datetime import datetime, timedelta
+from unittest.mock import AsyncMock, patch
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from dataclasses import replace  # noqa: E402
+
+from utils import request_log_prune  # noqa: E402
+from utils.request_log_prune import (  # noqa: E402
+    BATCH_SIZE,
+    MAX_BATCHES,
+    PRUNE_LOCK_KEY,
+    prune_request_logs_if_due,
+)
+from utils.request_log_settings import DEFAULT_CONFIG  # noqa: E402
+
+_SUCCESS_MARKER = "status_class BETWEEN 1 AND 3"
+_ERROR_MARKER = "status_class = 0 OR status_class >= 4"
+_CAP_MARKER = "id <= $1"
+
+
+def _conn(*, lock=True, watermark_age_minutes=None, cutoff_id=None,
+          success_batches=None, error_batches=None, cap_batches=None,
+          fail_on=None):
+    """A fake asyncpg connection that answers by SQL shape."""
+    conn = AsyncMock()
+
+    def fetchval(sql, *args):
+        text = str(sql)
+        if "pg_try_advisory_lock" in text:
+            return lock
+        if "ORDER BY id DESC OFFSET" in text:
+            return cutoff_id
+        return None
+
+    conn.fetchval = AsyncMock(side_effect=fetchval)
+
+    if watermark_age_minutes is None:
+        conn.fetchrow = AsyncMock(return_value=None)
+    else:
+        stamp = (datetime.utcnow() - timedelta(minutes=watermark_age_minutes)).isoformat() + "Z"
+        conn.fetchrow = AsyncMock(return_value={"value": json.dumps(stamp)})
+
+    queues = {
+        _SUCCESS_MARKER: list(success_batches or ["DELETE 0"]),
+        _ERROR_MARKER: list(error_batches or ["DELETE 0"]),
+        _CAP_MARKER: list(cap_batches or ["DELETE 0"]),
+    }
+
+    def execute(sql, *args):
+        text = str(sql)
+        if fail_on and fail_on in text:
+            raise RuntimeError("statement timeout")
+        for marker, queue in queues.items():
+            if marker in text:
+                return queue.pop(0) if queue else "DELETE 0"
+        return ""
+
+    conn.execute = AsyncMock(side_effect=execute)
+    return conn
+
+
+def _run(conn, *, force=False, **cfg_overrides):
+    cfg = replace(DEFAULT_CONFIG, **cfg_overrides)
+    with patch.object(request_log_prune, "get_config", lambda: cfg), \
+         patch.object(request_log_prune, "get_database_connection", AsyncMock(return_value=conn)), \
+         patch.object(request_log_prune, "close_database_connection", AsyncMock()):
+        return asyncio.run(prune_request_logs_if_due(force=force))
+
+
+def _delete_sql(conn):
+    return [str(c.args[0]) for c in conn.execute.call_args_list
+            if "DELETE FROM request_logs" in str(c.args[0])]
+
+
+def test_skips_entirely_when_another_replica_holds_the_lock():
+    conn = _conn(lock=False)
+    counts = _run(conn)
+
+    assert counts == {"success": 0, "error": 0, "overflow": 0, "ran": 0}
+    assert _delete_sql(conn) == [], (
+        "a second replica ran the prune concurrently — pg_try_advisory_lock is what keeps "
+        "N pods from all scanning the same table at once"
+    )
+
+
+def test_skips_when_the_watermark_is_still_fresh():
+    conn = _conn(watermark_age_minutes=10)
+    counts = _run(conn, prune_interval_minutes=60)
+
+    assert counts["ran"] == 0
+    assert _delete_sql(conn) == []
+
+
+def test_runs_all_three_limits_when_due():
+    conn = _conn(
+        watermark_age_minutes=120, cutoff_id=999,
+        success_batches=["DELETE 3"], error_batches=["DELETE 4"], cap_batches=["DELETE 5"],
+    )
+    counts = _run(conn, prune_interval_minutes=60, success_retention_days=7,
+                  error_retention_days=30, max_rows=500000)
+
+    sqls = _delete_sql(conn)
+    assert len(sqls) == 3, f"expected success TTL + error TTL + row cap, got {len(sqls)}"
+    assert _SUCCESS_MARKER in sqls[0]
+    assert _ERROR_MARKER in sqls[1]
+    assert _CAP_MARKER in sqls[2]
+
+    assert counts["success"] == 3
+    assert counts["error"] == 4
+    assert counts["overflow"] == 5
+    assert counts["ran"] == 1
+
+
+def test_retention_days_travel_as_bind_parameters():
+    """Injection guard: the day counts come straight from an operator-editable
+    setting, so they must never be formatted into the SQL text."""
+    conn = _conn(watermark_age_minutes=120)
+    _run(conn, prune_interval_minutes=60, success_retention_days=7, error_retention_days=30)
+
+    ttl_calls = [c for c in conn.execute.call_args_list
+                 if "created_at < NOW()" in str(c.args[0])]
+    assert len(ttl_calls) == 2
+
+    for call in ttl_calls:
+        assert "($1 || ' days')::INTERVAL" in str(call.args[0]), (
+            "the retention window is interpolated into the SQL string instead of bound — "
+            "an operator-supplied value reaching the parser is an injection point"
+        )
+
+    assert ttl_calls[0].args[1] == "7"
+    assert ttl_calls[1].args[1] == "30"
+    assert ttl_calls[0].args[2] == BATCH_SIZE
+
+
+def test_deletes_are_batched_until_a_short_batch():
+    conn = _conn(
+        watermark_age_minutes=120,
+        success_batches=[f"DELETE {BATCH_SIZE}", f"DELETE {BATCH_SIZE}", "DELETE 12"],
+    )
+    counts = _run(conn, prune_interval_minutes=60)
+
+    assert counts["success"] == BATCH_SIZE * 2 + 12, (
+        "the batch loop stopped early or double-counted"
+    )
+    success_calls = [s for s in _delete_sql(conn) if _SUCCESS_MARKER in s]
+    assert len(success_calls) == 3, "the loop must stop on the first short batch"
+
+
+def test_batch_loop_respects_the_ceiling():
+    """A table so far behind that every batch comes back full must still hand the
+    connection back rather than looping forever."""
+    conn = _conn(
+        watermark_age_minutes=120,
+        success_batches=[f"DELETE {BATCH_SIZE}"] * (MAX_BATCHES * 3),
+    )
+    counts = _run(conn, prune_interval_minutes=60)
+
+    assert counts["success"] == BATCH_SIZE * MAX_BATCHES
+    success_calls = [s for s in _delete_sql(conn) if _SUCCESS_MARKER in s]
+    assert len(success_calls) == MAX_BATCHES
+
+
+def test_watermark_is_not_stamped_when_a_step_fails():
+    conn = _conn(watermark_age_minutes=120, cutoff_id=42, fail_on=_CAP_MARKER)
+    counts = _run(conn, prune_interval_minutes=60)
+
+    stamps = [c for c in conn.execute.call_args_list
+              if "INSERT INTO system_settings" in str(c.args[0])]
+    assert stamps == [], (
+        "a partially-completed pass stamped the watermark, so the remainder would not be "
+        "retried until the next interval"
+    )
+    assert counts["ran"] == 0
+
+
+def test_watermark_is_stamped_after_a_complete_pass():
+    conn = _conn(watermark_age_minutes=120, cutoff_id=None)
+    counts = _run(conn, prune_interval_minutes=60)
+
+    stamps = [c for c in conn.execute.call_args_list
+              if "INSERT INTO system_settings" in str(c.args[0])]
+    assert len(stamps) == 1
+    # args = (sql, key, json_value)
+    assert stamps[0].args[1] == "requestlog.last_pruned_at"
+    assert stamps[0].args[2].startswith('"'), (
+        "the watermark must be stored as a JSON string — the ::jsonb cast rejects a bare "
+        "timestamp, and the reader json.loads() it back"
+    )
+    assert counts["ran"] == 1
+
+
+def test_advisory_lock_is_released_even_on_failure():
+    conn = _conn(watermark_age_minutes=120, fail_on=_SUCCESS_MARKER)
+    _run(conn, prune_interval_minutes=60)
+
+    unlocks = [c for c in conn.execute.call_args_list if "pg_advisory_unlock" in str(c.args[0])]
+    assert unlocks, "the advisory lock was leaked — every later pass on any replica would skip"
+    assert unlocks[0].args[1] == PRUNE_LOCK_KEY
+
+
+def test_never_raises_when_the_pool_is_exhausted():
+    with patch.object(request_log_prune, "get_database_connection",
+                      AsyncMock(side_effect=RuntimeError("pool exhausted"))), \
+         patch.object(request_log_prune, "close_database_connection", AsyncMock()):
+        counts = asyncio.run(prune_request_logs_if_due())
+
+    assert counts == {"success": 0, "error": 0, "overflow": 0, "ran": 0}
+
+
+def test_row_cap_is_a_noop_when_the_table_is_smaller_than_the_cap():
+    conn = _conn(watermark_age_minutes=120, cutoff_id=None, cap_batches=["DELETE 77"])
+    counts = _run(conn, prune_interval_minutes=60)
+
+    assert counts["overflow"] == 0, (
+        "the cap deleted rows even though OFFSET max_rows found no cutoff — that would "
+        "truncate a table that is under the limit"
+    )
+    assert not any(_CAP_MARKER in s for s in _delete_sql(conn))
+
+
+def test_force_bypasses_the_watermark():
+    """The manual purge button must not be a no-op just because the scheduled
+    pass ran a minute ago."""
+    conn = _conn(watermark_age_minutes=1, cutoff_id=None,
+                 success_batches=["DELETE 1"], error_batches=["DELETE 2"])
+    counts = _run(conn, force=True, prune_interval_minutes=1440)
+
+    assert counts["ran"] == 1
+    assert counts["success"] == 1
+    assert counts["error"] == 2
+
+
+def test_lock_key_does_not_collide_with_the_existing_ones():
+    # 18181818 drafts cap, 18181819 wizard create, 18181820 apply,
+    # 0x41434D45 per-ACME-order, 1836016242 migration.
+    assert PRUNE_LOCK_KEY not in (18181818, 18181819, 18181820, 0x41434D45, 1836016242)

backend/tests/test_request_log_real_payloads.py

@@ -0,0 +1,289 @@
+"""v1.11.0: redaction pinned against THIS codebase's real payloads.
+
+test_request_log_redaction.py pins the RULES — which key names match, which
+value shapes fire. It passed 72/72 while six real endpoints of this application
+still wrote secrets to `request_logs`, because a rule test proves the rule, not
+the coverage. Every case here is built from an actual handler's request or
+response shape, with the field names taken from the source and named in the
+docstring, so a future change to redaction is measured against what this system
+actually sends rather than against what someone remembered to imagine.
+
+Method note: the payloads go through `decode_body()`, the same entry point the
+writer task uses, rather than calling `redact()` directly. That is deliberate —
+two of the findings below only appear on the way in (an oversized body never
+reaches `redact()` as a dict at all, it arrives as one `_raw` string), so a test
+that starts from a dict would report a pass on a payload that leaks in
+production.
+"""
+import json
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from utils.request_log_redaction import (  # noqa: E402
+    decode_body,
+    is_secret_key,
+    redact_headers,
+)
+
+VRRP_SECRET = "S3cr3tVrrpPass!"
+TOTP_SECRET = "JBSWY3DPEHPK3PXP"
+STATS_PASSWORD = "StatsPa55word"
+USERLIST_HASH = "$6$rounds=5000$abcdefgh$XyZ"
+JWT = (
+    "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9"
+    ".eyJzdWIiOiIxIiwidXNlcm5hbWUiOiJhZG1pbiJ9"
+    ".dQw4w9WgXcQdQw4w9WgXcQdQw4w9WgXcQ"
+)
+PEM_KEY = (
+    "-----BEGIN RSA PRIVATE KEY-----\n"
+    + "MIIEowIBAAKCAQEA" + "A" * 200 + "\n"
+    + "-----END RSA PRIVATE KEY-----\n"
+)
+
+# The rendered file `GET /api/agents/{n}/keepalived-config` hands to an agent,
+# and the one `POST /api/agents/{n}/keepalived-discovery` sends back.
+KEEPALIVED_CONF = f"""! Managed by HAProxy OpenManager
+vrrp_instance VI_1 {{
+    state MASTER
+    interface eth0
+    virtual_router_id 51
+    priority 200
+    advert_int 1
+    authentication {{
+        auth_type PASS
+        auth_pass {VRRP_SECRET}
+    }}
+    virtual_ipaddress {{
+        10.20.30.40/24
+    }}
+}}
+"""
+
+# A production haproxy.cfg as the agent uploads it verbatim from the node
+# (`config_content=$(cat "$config_path")` -> POST .../config-response).
+HAPROXY_CFG = f"""global
+    log stdout local0
+    stats socket /var/run/haproxy.sock mode 660
+
+userlist admins
+    user ops password {USERLIST_HASH}
+    user dev insecure-password Hunter2Plain
+
+listen stats
+    bind *:8404
+    stats enable
+    stats auth admin:{STATS_PASSWORD}
+    stats uri /stats
+
+backend web
+    server web1 10.0.0.1:80 check
+"""
+
+
+def _capture(payload, *, cap=8192, content_type="application/json"):
+    """Run a payload through the capture path exactly as the writer does.
+
+    `cap` is `requestlog.max_body_bytes`. Bodies larger than it arrive
+    truncated, do not parse as JSON, and land in the `{"_raw": ...}` fallback —
+    which is the common case for config uploads and the case a dict-based test
+    never exercises.
+    """
+    body = json.dumps(payload).encode()
+    value, truncated = decode_body(body[:cap], content_type, len(body))
+    return json.dumps(value, default=str), truncated
+
+
+def _assert_absent(rendered, *secrets):
+    for secret in secrets:
+        assert secret not in rendered, (
+            f"{secret!r} reached request_logs. Rendered row: {rendered[:400]}"
+        )
+
+
+# --------------------------------------------------------------------------
+# The VRRP password. routers/vip.py: "the secret never leaves the server in
+# cleartext ... only the at-rest Fernet token and the agent-delivery endpoint
+# ever see the real value."
+# --------------------------------------------------------------------------
+
+def test_vip_create_body_does_not_store_auth_pass():
+    """POST/PUT /api/vip — `payload.auth_pass`, routers/vip.py:585,679."""
+    rendered, _ = _capture({
+        "name": "vip-prod", "virtual_ip": "10.20.30.40", "interface": "eth0",
+        "virtual_router_id": 51, "auth_pass": VRRP_SECRET,
+    })
+    _assert_absent(rendered, VRRP_SECRET)
+
+
+def test_keepalived_config_delivery_does_not_store_the_rendered_secret():
+    """GET /api/agents/{n}/keepalived-config — `keepalived.config_content`.
+
+    Polled on the SSL cadence, so an unmasked capture rewrites the secret to the
+    audit table roughly 576 times a day per member node.
+    """
+    rendered, _ = _capture({
+        "agent_name": "prod-lb-01", "status": "available",
+        "config_path": "/etc/keepalived/keepalived.conf",
+        "keepalived": {
+            "vip_id": 3, "vip_name": "vip-prod",
+            "config_content": KEEPALIVED_CONF, "config_hash": "abc123",
+        },
+    })
+    _assert_absent(rendered, VRRP_SECRET)
+    assert "auth_pass" in rendered, "the directive should stay visible, only its value masked"
+    assert "vrrp_instance VI_1" in rendered, "masking must not destroy the rest of the config"
+
+
+def test_keepalived_discovery_body_does_not_store_the_found_secret():
+    """POST /api/agents/{n}/keepalived-discovery — `config_content`.
+
+    routers/agent.py already pops auth_pass out of the parsed analysis,
+    Fernet-encrypts it into its own column and stores only
+    `vip_discoveries.raw_config_masked`. Capturing the request that produced all
+    that, unmasked, would put the plaintext straight back next to it.
+    """
+    rendered, _ = _capture({
+        "agent_name": "prod-lb-01", "exists": True, "is_managed": False,
+        "config_path": "/etc/keepalived/keepalived.conf",
+        "config_content": KEEPALIVED_CONF,
+    })
+    _assert_absent(rendered, VRRP_SECRET)
+
+
+def test_auth_pass_is_masked_when_the_body_is_too_large_to_parse():
+    """The truncated `_raw` path, where line breaks are the escape `\\n`.
+
+    A value pattern that stops only at a REAL newline runs to the end of the
+    string here: no leak, but the whole remainder of the config is masked and
+    the row is useless. Both properties are asserted.
+    """
+    payload = {"config_content": KEEPALIVED_CONF + "backend b\n    server s1 10.0.0.1:80 check\n" * 400}
+    rendered, truncated = _capture(payload)
+    assert truncated, "this fixture must exercise the truncated path"
+    _assert_absent(rendered, VRRP_SECRET)
+    assert "server s1 10.0.0.1:80" in rendered, (
+        "masking ran past the end of the auth_pass line and ate the rest of the config"
+    )
+
+
+# --------------------------------------------------------------------------
+# TOTP. routers/mfa.py logs `{"secret_len": ...}` with the comment
+# "NEVER log the secret itself".
+# --------------------------------------------------------------------------
+
+def test_mfa_enroll_response_does_not_store_the_totp_secret_in_either_field():
+    """POST /api/mfa/enroll — returns `secret` AND `otpauth_uri`.
+
+    Redacting one while the same value sits in the other is not redaction.
+    """
+    rendered, _ = _capture({
+        "secret": TOTP_SECRET,
+        "otpauth_uri": f"otpauth://totp/OpenManager:admin?secret={TOTP_SECRET}&issuer=OpenManager",
+        "qr_size": 256,
+    })
+    _assert_absent(rendered, TOTP_SECRET)
+
+
+def test_userinfo_credentials_in_a_url_are_dropped():
+    rendered, _ = _capture({"webhook": "https://svc:Sup3rSecret@hooks.example.com/notify?api_key=abc123"})
+    _assert_absent(rendered, "Sup3rSecret", "abc123")
+    assert "hooks.example.com" in rendered, "the host is the diagnostic value; keep it"
+
+
+# --------------------------------------------------------------------------
+# HAProxy config. We never RENDER credentials into one, but the agent uploads
+# the node's real file and the operator can paste one.
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("cap,label", [(8192, "truncated _raw path"), (10 ** 6, "parsed path")])
+def test_uploaded_haproxy_config_masks_credentials_on_both_paths(cap, label):
+    """POST /api/configuration/agents/{n}/config-response, and
+    POST /api/config/validate."""
+    rendered, _ = _capture({"config_content": HAPROXY_CFG, "config_path": "/etc/haproxy/haproxy.cfg"}, cap=cap)
+    _assert_absent(rendered, STATS_PASSWORD, USERLIST_HASH, "Hunter2Plain")
+    assert "stats auth admin:" in rendered, f"[{label}] the account name is diagnostic; keep it"
+    assert "server web1 10.0.0.1:80" in rendered, f"[{label}] the rest of the config must survive"
+
+
+@pytest.mark.parametrize("prose", [
+    "invalid password format",
+    "the password must be at least 8 characters",
+    "authentication failed for user admin",
+])
+def test_ordinary_prose_is_not_mangled(prose):
+    """Over-matching would blank the messages the log exists to show."""
+    rendered, _ = _capture({"detail": prose})
+    assert prose in rendered, f"redaction damaged an ordinary message: {rendered}"
+
+
+# --------------------------------------------------------------------------
+# Regressions guarding what already worked, so a later rule change cannot
+# quietly trade one of these away for one of the above.
+# --------------------------------------------------------------------------
+
+def test_login_exchange_stores_neither_the_password_nor_the_token():
+    req, _ = _capture({"username": "admin", "password": "hunter2hunter2"})
+    _assert_absent(req, "hunter2hunter2")
+    res, _ = _capture({"access_token": JWT, "token_type": "bearer", "user": {"id": 1}})
+    _assert_absent(res, JWT)
+
+
+def test_private_key_is_redacted_even_under_an_innocent_key_name():
+    """The value-shape guard is the net under the key-name rules."""
+    rendered, _ = _capture({"blob": PEM_KEY, "note": "backup"})
+    _assert_absent(rendered, "MIIEowIBAAKCAQEA")
+
+
+def test_dns_provider_credentials_are_redacted():
+    cf, _ = _capture({"provider": "cloudflare", "api_token": "cf_live_abcdefghijklmnop", "zone_id": "z1"})
+    _assert_absent(cf, "cf_live_abcdefghijklmnop")
+    gd, _ = _capture({"provider": "godaddy", "api_key": "gd_key_1234567890", "api_secret": "gd_secret_098"})
+    _assert_absent(gd, "gd_key_1234567890", "gd_secret_098")
+
+
+def test_innocent_urls_survive_untouched():
+    """Scrubbing must not rewrite the ACME URLs an operator reads back."""
+    for url in (
+        "https://acme-v02.api.letsencrypt.org/directory",
+        "https://acme-v02.api.letsencrypt.org/acme/acct/12345",
+    ):
+        rendered, _ = _capture({"directory_url": url})
+        assert url in rendered, f"an innocent URL was rewritten: {rendered}"
+
+
+def test_credential_headers_are_presence_only_and_the_rest_are_dropped():
+    out = redact_headers({
+        "authorization": f"Bearer {JWT}",
+        "x-api-key": "agt_" + "a" * 32,
+        "cookie": "session=abc123",
+        "user-agent": "curl/8.4.0",
+        "x-forwarded-for": "10.20.30.5",
+        "x-internal-secret": "not-on-the-allowlist",
+    })
+    rendered = json.dumps(out)
+    _assert_absent(rendered, JWT, "agt_" + "a" * 32, "abc123", "not-on-the-allowlist")
+    assert out["user-agent"] == "curl/8.4.0"
+    assert out["x-forwarded-for"] == "10.20.30.5"
+    assert "x-internal-secret" not in out, "an unlisted header must be dropped, not kept"
+
+
+@pytest.mark.parametrize("key", ["auth_pass", "authPass", "auth-pass", "AUTH_PASS"])
+def test_auth_pass_key_matches_in_every_spelling(key):
+    assert is_secret_key(key), (
+        f"{key!r} normalizes to something no rule matches. 'password' is not a "
+        f"substring of 'authpass' and the bare 'auth' entry is an exact match."
+    )
+
+
+@pytest.mark.parametrize("key", [
+    "monkey", "key_suffix", "payload_size", "nonce_count", "keyboard_layout",
+    "config_path", "authenticated", "author",
+])
+def test_innocent_field_names_are_still_kept(key):
+    """The other half of the trade: over-redaction blanks the fields the
+    feature exists to show."""
+    assert not is_secret_key(key)

backend/tests/test_request_log_redaction.py

@@ -0,0 +1,262 @@
+"""v1.11.0: nothing secret reaches request_logs.
+
+The request/response log stores bodies and headers, so redaction is the single
+control standing between "operators can debug a failing ACME order" and "the
+audit table is a credential store". These tests pin both halves of that: the
+things that MUST be redacted, and the innocent field names that must NOT be
+(over-matching would silently blank out the fields the feature exists to show).
+"""
+import json
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from utils.request_log_redaction import (  # noqa: E402
+    REDACTED,
+    decode_body,
+    is_capturable_content_type,
+    is_secret_key,
+    redact,
+    redact_headers,
+    safe_error_text,
+    scrub_query_string,
+    scrub_url,
+)
+
+
+# --------------------------------------------------------------------------
+# Key matching
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("key", [
+    "password", "PASSWORD", "Pass_Word", "passwd", "pwd",
+    # api_token is the literal field name of the Cloudflare provider credential
+    # (services/dns_providers/cloudflare.py) — it must never survive a round trip.
+    "token", "access_token", "refreshToken", "MFA_TOKEN",
+    "api_token", "agent_token", "csrf_token", "session_token",
+    "api_key", "API-KEY", "apiKey", "x-api-key",
+    "secret", "client_secret", "eab_hmac_key",
+    "private_key", "cert_private_key", "csr_private_key", "jwk_private_key",
+    "authorization", "cookie", "set-cookie",
+    "signature", "protected", "payload", "nonce", "replay-nonce",
+    "key_authorization", "backup_codes", "totp_secret",
+    "stats_password", "credentials", "encryption_key",
+])
+def test_secret_keys_are_detected(key):
+    assert is_secret_key(key), f"{key!r} must be treated as a secret field name"
+
+
+@pytest.mark.parametrize("key", [
+    # Every one of these has a secret-looking substring but is innocent. If any
+    # starts redacting, the log stops being useful for the exact debugging it
+    # was built for.
+    "key_suffix", "monkey", "keyboard", "turkey",
+    "payload_size", "nonce_count",
+    "public_key_id", "keys_total",
+    "name", "status_code", "duration_ms", "domain", "directory_url",
+])
+def test_innocent_keys_are_not_redacted(key):
+    assert not is_secret_key(key), (
+        f"{key!r} was redacted by over-matching — the log would blank out a field "
+        f"operators need"
+    )
+
+
+# --------------------------------------------------------------------------
+# Recursive body redaction
+# --------------------------------------------------------------------------
+
+def test_nested_dicts_and_lists_are_redacted_recursively():
+    body = {
+        "user": {"username": "admin", "password": "hunter2"},
+        "accounts": [
+            {"email": "a@example.com", "eab_hmac_key": "s3cr3t"},
+            {"email": "b@example.com", "api_token": "cf-token"},
+        ],
+        "cluster_id": 3,
+    }
+    out = redact(body)
+
+    assert out["user"]["username"] == "admin"
+    assert out["user"]["password"] == REDACTED
+    assert out["accounts"][0]["email"] == "a@example.com"
+    assert out["accounts"][0]["eab_hmac_key"] == REDACTED
+    assert out["accounts"][1]["api_token"] == REDACTED
+    assert out["cluster_id"] == 3
+
+
+def test_depth_limit_stops_runaway_nesting():
+    deep = current = {}
+    for _ in range(20):
+        current["child"] = {}
+        current = current["child"]
+    current["password"] = "leak"
+
+    out = redact(deep)
+    flattened = json.dumps(out)
+    assert "***DEPTH_LIMIT***" in flattened
+    assert "leak" not in flattened
+
+
+def test_node_budget_bounds_a_very_wide_body():
+    wide = {f"field_{i}": i for i in range(5000)}
+    out = redact(wide)
+    assert out.get("_node_limit") is True
+    assert len(out) < 5000, "node budget did not bound a pathologically wide body"
+
+
+def test_pem_private_key_is_redacted_by_value_shape():
+    body = {"blob": "-----BEGIN RSA PRIVATE KEY-----\n" + "A" * 200 + "\n-----END RSA PRIVATE KEY-----"}
+    out = redact(body)
+    assert out["blob"] == REDACTED, (
+        "a PEM private key under an innocent key name was stored verbatim"
+    )
+
+
+def test_jwt_shaped_string_is_redacted_by_value_shape():
+    jwt_like = "eyJhbGciOiJIUzI1NiJ9." + "a" * 40 + "." + "b" * 40
+    out = redact({"data": jwt_like})
+    assert out["data"] == REDACTED
+
+
+def test_long_strings_are_truncated_with_a_marker():
+    out = redact({"note": "x" * 9000})
+    assert out["note"].endswith("chars]")
+    assert len(out["note"]) < 9000
+
+
+def test_redact_never_raises_on_odd_input():
+    class Weird:
+        def __repr__(self):
+            raise RuntimeError("boom")
+
+    # Non-serializable leaf values must pass straight through, not explode.
+    assert redact({"x": Weird()}) is not None
+
+
+# --------------------------------------------------------------------------
+# Headers (allowlist)
+# --------------------------------------------------------------------------
+
+def test_headers_use_an_allowlist_with_presence_markers():
+    out = redact_headers({
+        "Content-Type": "application/json",
+        "User-Agent": "curl/8.0",
+        "Authorization": "Bearer super-secret-token",
+        "Cookie": "session=abc",
+        "X-Custom-Internal": "some value",
+    })
+
+    assert out["content-type"] == "application/json"
+    assert out["user-agent"] == "curl/8.0"
+    # Presence is useful when debugging a 401; the value is not.
+    assert out["authorization"] == REDACTED
+    assert out["cookie"] == REDACTED
+    # Not on the allowlist and not a known credential header -> dropped entirely.
+    assert "x-custom-internal" not in out
+
+
+def test_redact_headers_handles_none():
+    assert redact_headers(None) is None
+    assert redact_headers({}) is None
+
+
+# --------------------------------------------------------------------------
+# URLs and query strings
+# --------------------------------------------------------------------------
+
+def test_query_string_secrets_are_scrubbed():
+    scrubbed, as_dict = scrub_query_string("token=abc123&page=2&api_key=xyz")
+    assert "abc123" not in scrubbed
+    assert "xyz" not in scrubbed
+    assert "page=2" in scrubbed
+    assert as_dict["token"] == REDACTED
+    assert as_dict["page"] == "2"
+
+
+def test_scrub_url_strips_userinfo_and_query_secrets():
+    out = scrub_url("https://user:hunter2@api.example.com:8443/v1/zones?api_key=abc&page=1")
+    assert "hunter2" not in out
+    assert "user" not in out.split("/v1")[0].replace("api.example.com", "")
+    assert "abc" not in out
+    assert "api.example.com:8443" in out
+    assert "page=1" in out
+
+
+def test_scrub_url_drops_the_fragment():
+    # Fragments never reach a server, and they are a classic token carrier.
+    assert "#" not in scrub_url("https://example.com/x?a=1#access_token=leak")
+
+
+# --------------------------------------------------------------------------
+# Body decoding, capping, truncation marker
+# --------------------------------------------------------------------------
+
+def test_decode_body_parses_and_redacts_json():
+    raw = json.dumps({"username": "admin", "password": "hunter2"}).encode()
+    value, truncated = decode_body(raw, "application/json", len(raw))
+    assert value["username"] == "admin"
+    assert value["password"] == REDACTED
+    assert truncated is False
+
+
+def test_decode_body_marks_truncation_with_the_original_size():
+    full = b"x" * 20000
+    captured = full[:1024]
+    value, truncated = decode_body(captured, "text/plain", len(full))
+    assert truncated is True
+    assert value["_truncated"] is True
+    assert value["_original_bytes"] == 20000
+
+
+def test_decode_body_wraps_non_json_as_raw_object():
+    value, _ = decode_body(b"plain text response", "text/plain", 19)
+    assert value == {"_raw": "plain text response"}
+
+
+def test_decode_body_survives_truncated_json():
+    # A JSON body cut off at the cap will not parse — keep the prefix rather
+    # than losing the field entirely.
+    value, truncated = decode_body(b'{"a": "bb', "application/json", 500)
+    assert truncated is True
+    assert "_raw" in value
+
+
+def test_decode_body_parses_form_encoded():
+    value, _ = decode_body(b"username=admin&password=hunter2",
+                           "application/x-www-form-urlencoded", 30)
+    assert value["username"] == "admin"
+    assert value["password"] == REDACTED
+
+
+def test_decode_body_returns_none_for_empty():
+    assert decode_body(b"", "application/json", 0) == (None, False)
+    assert decode_body(None, "application/json", 0) == (None, False)
+
+
+def test_binary_content_types_are_not_capturable():
+    assert is_capturable_content_type("application/json")
+    assert is_capturable_content_type("application/json; charset=utf-8")
+    assert is_capturable_content_type("text/plain")
+    assert not is_capturable_content_type("application/octet-stream")
+    assert not is_capturable_content_type("image/png")
+    assert not is_capturable_content_type("text/event-stream")
+
+
+# --------------------------------------------------------------------------
+# Error rendering
+# --------------------------------------------------------------------------
+
+def test_safe_error_text_type_only_hides_the_message():
+    exc = ValueError("https://api.godaddy.com/v1/domains/secret-zone/records failed")
+    assert safe_error_text(exc, type_only=True) == "ValueError"
+    assert "godaddy" not in safe_error_text(exc, type_only=True)
+
+
+def test_safe_error_text_includes_the_message_when_allowed():
+    text = safe_error_text(RuntimeError("connection refused"))
+    assert text.startswith("RuntimeError")
+    assert "connection refused" in text

backend/tests/test_request_log_router_auth.py

@@ -0,0 +1,205 @@
+"""v1.11.0: the request log API is gated, and its routes resolve.
+
+Two distinct failure modes are pinned here.
+
+**Auth.** The table holds redacted-but-real request and response bodies for
+every user, so an unauthenticated or under-privileged caller must never get a
+row. There is no database in this suite, so the behavioural checks assert only
+that an anonymous call is rejected before any DB work — which is exactly the
+property that matters — and a source scan covers the per-endpoint permission.
+
+**Route order.** `/{log_id}` is a single-segment path and FastAPI matches in
+declaration order, so declaring it before `/settings`, `/stats` or `/purge`
+makes those three unreachable (they parse as a log id and 422). This is the
+mirror image of the shadowing trap already present in routers/settings.py.
+"""
+import os
+import re
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+_BACKEND = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+_ROUTER = os.path.join(_BACKEND, "routers", "request_logs.py")
+_MAIN = os.path.join(_BACKEND, "main.py")
+
+REJECT = (401, 403, 422)
+
+
+@pytest.fixture(scope="module")
+def src():
+    with open(_ROUTER, encoding="utf-8") as f:
+        return f.read()
+
+
+# --------------------------------------------------------------------------
+# Behavioural: nothing is readable without credentials
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("method,path", [
+    ("get", "/api/request-logs"),
+    ("get", "/api/request-logs/1"),
+    ("get", "/api/request-logs/stats"),
+    ("get", "/api/request-logs/settings"),
+    ("put", "/api/request-logs/settings"),
+    ("post", "/api/request-logs/purge"),
+])
+def test_anonymous_access_is_rejected(client, method, path):
+    res = getattr(client, method)(path) if method != "put" else client.put(path, json={})
+    assert res.status_code in REJECT, (
+        f"{method.upper()} {path} returned {res.status_code} without an Authorization "
+        f"header — the request log contains captured bodies for every user"
+    )
+
+
+def test_a_garbage_token_is_rejected(client):
+    res = client.get("/api/request-logs", headers={"authorization": "Bearer not-a-token"})
+    assert res.status_code in REJECT
+
+
+# --------------------------------------------------------------------------
+# Source scan: per-endpoint permission
+# --------------------------------------------------------------------------
+
+def _handler_body(src, decorator):
+    start = src.index(decorator)
+    rest = src[start + len(decorator):]
+    end = rest.find("\n@router.")
+    return rest if end == -1 else rest[:end]
+
+
+@pytest.mark.parametrize("decorator,action", [
+    ('@router.get("/settings")', "manage"),
+    ('@router.put("/settings")', "manage"),
+    ('@router.get("/stats")', "read"),
+    ('@router.post("/purge")', "manage"),
+    ('@router.get("")', "read"),
+    ('@router.get("/{log_id}")', "read"),
+])
+def test_every_endpoint_enforces_its_permission(src, decorator, action):
+    body = _handler_body(src, decorator)
+    assert f'_require(authorization, "{action}")' in body, (
+        f"{decorator} does not enforce requestlog.{action}"
+    )
+
+
+def test_require_helper_raises_403_not_a_silent_pass(src):
+    helper = src.split("async def _require", 1)[1].split("\nasync def ", 1)[0]
+    assert "check_user_permission" in helper
+    assert "status_code=403" in helper
+    assert "current_user=current_user" in helper, (
+        "the admin bypass is skipped, so every call pays an extra SELECT on users"
+    )
+
+
+# --------------------------------------------------------------------------
+# Route declaration order
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("literal", ['@router.get("/settings")', '@router.put("/settings")',
+                                     '@router.get("/stats")', '@router.post("/purge")'])
+def test_literal_routes_are_declared_before_the_catch_all(src, literal):
+    catch_all = src.index('@router.get("/{log_id}")')
+    assert src.index(literal) < catch_all, (
+        f"{literal} is declared after GET /{{log_id}}. FastAPI matches in declaration "
+        f"order and /{{log_id}} is a single-segment path, so it would swallow this route "
+        f"and the request would fail parsing 'settings' as an int."
+    )
+
+
+def test_list_route_is_declared_before_the_catch_all(src):
+    assert src.index('@router.get("")') < src.index('@router.get("/{log_id}")')
+
+
+# --------------------------------------------------------------------------
+# Query construction
+# --------------------------------------------------------------------------
+
+def test_filters_are_bound_never_interpolated(src):
+    """User-supplied filters reach the WHERE clause; they must arrive as $n
+    parameters."""
+    body = _handler_body(src, '@router.get("")')
+    # The only f-string interpolation allowed into SQL is the placeholder index
+    # and the assembled clause list, never a raw value.
+    for match in re.findall(r'add\("([^"]+)"', body):
+        assert "{n}" in match, f"filter clause {match!r} does not use a bound placeholder"
+
+
+def test_list_endpoint_scopes_non_privileged_callers(src):
+    """A caller with only `requestlog.read` sees their own rows plus the fleet's.
+
+    Widened from own-rows-only during review, deliberately. The `operator` role
+    is granted requestlog.read to "debug failing applies and ACME orders", but
+    an apply fails on the NODE and the node reports it over its own API key, so
+    the row carrying the diagnosis has `user_id IS NULL` — own-rows-only hid it
+    from exactly the role the grant was written for.
+
+    What must NOT widen is the part this test was written to protect: another
+    USER's captured bodies. Both halves are asserted below.
+    """
+    # Comments explain what the clause deliberately does NOT do, so match on
+    # code only — otherwise the prose describing the rule fails the test for it.
+    body = "\n".join(
+        line for line in _handler_body(src, '@router.get("")').splitlines()
+        if not line.lstrip().startswith("#")
+    )
+    assert "if not can_manage:" in body
+    assert "direction = 'inbound'" in body, (
+        "outbound rows are not scoped at all, so a caller with only "
+        "requestlog.read would see every CA and DNS call the backend ever made"
+    )
+    assert "user_id = $" in body, (
+        "a caller with only requestlog.read can see every other user's captured "
+        "request bodies"
+    )
+    assert "TARGET_INBOUND_AGENT" in body, (
+        "agent rows are hidden from requestlog.read, which is the one thing the "
+        "operator grant exists for"
+    )
+    assert "user_id IS NULL" not in body, (
+        "scoping on NULL rather than on target would also expose anonymous "
+        "traffic — failed logins and the usernames they carry, unauthenticated "
+        "probes — to any requestlog.read holder"
+    )
+
+
+def test_detail_endpoint_applies_the_same_scoping(src):
+    body = _handler_body(src, '@router.get("/{log_id}")')
+    assert "can_manage" in body
+    assert "404" in body, (
+        "the detail endpoint should 404 rather than 403 for a row the caller may not see, "
+        "so it does not confirm which ids exist"
+    )
+
+
+def test_list_response_omits_bodies(src):
+    """A 200-row page carrying two 8 KB JSONB blobs per row is a multi-megabyte
+    response; bodies belong to the detail endpoint."""
+    columns = src.split("_LIST_COLUMNS = ", 1)[1].split('"""', 2)[1]
+    assert "request_body," not in columns
+    assert "response_body," not in columns
+    assert "request_body_bytes" in columns, "the size is still useful in the list"
+
+
+def test_count_is_bounded(src):
+    body = _handler_body(src, '@router.get("")')
+    assert "LIMIT {count_cap}" in body or "count_cap" in body, (
+        "an unbounded COUNT(*) over request_logs is a sequential scan on every page change"
+    )
+    assert "total_is_estimate" in body
+
+
+# --------------------------------------------------------------------------
+# Registration
+# --------------------------------------------------------------------------
+
+def test_router_is_registered_in_main():
+    with open(_MAIN, encoding="utf-8") as f:
+        main_src = f.read()
+
+    assert "from routers.request_logs import router as request_logs_router" in main_src
+    assert "app.include_router(request_logs_router)" in main_src, (
+        "the router is imported but never mounted, so every endpoint 404s"
+    )

backend/tests/test_request_log_settings.py

@@ -0,0 +1,258 @@
+"""v1.11.0: the retention policy the operator sees is the policy that runs.
+
+Two things drift silently and are caught here:
+
+  1. The defaults live in TWO places — the seed SQL in migrations.py and the
+     dataclass in utils/request_log_settings.py. If they disagree, a fresh
+     install and an upgraded install behave differently, which is the worst
+     kind of bug to chase.
+  2. Values in `system_settings` are operator-editable and arrive from asyncpg
+     as raw JSON *strings*. Anything out of range, mistyped or hand-edited must
+     be clamped rather than crash the writer loop.
+"""
+import asyncio
+import json
+import os
+import re
+import sys
+from unittest.mock import AsyncMock, patch
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from utils import request_log_settings  # noqa: E402
+from utils.request_log_settings import (  # noqa: E402
+    DEFAULT_CONFIG,
+    DEFAULT_EXCLUDE_PATHS,
+    RequestLogConfig,
+    config_from_mapping,
+    get_config,
+    normalize_exclude_paths,
+    refresh_config,
+    set_config,
+)
+
+_MIGRATIONS = os.path.join(
+    os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "database", "migrations.py"
+)
+
+
+def _seeded_defaults():
+    """Parse the ('requestlog.x', 'value', ...) tuples out of the seed SQL."""
+    with open(_MIGRATIONS, encoding="utf-8") as f:
+        src = f.read()
+
+    body = src.split("async def ensure_request_log_settings", 1)[1].split("\nasync def ", 1)[0]
+    out = {}
+    for key, raw in re.findall(r"\('requestlog\.(\w+)', '(.*?)', 'requestlog'", body):
+        try:
+            out[key] = json.loads(raw)
+        except json.JSONDecodeError:
+            out[key] = raw
+    return out
+
+
+# --------------------------------------------------------------------------
+# Defaults must not drift between the seed and the code
+# --------------------------------------------------------------------------
+
+def test_seed_and_dataclass_defaults_agree():
+    seeded = _seeded_defaults()
+    assert seeded, "could not parse the requestlog seed rows out of migrations.py"
+
+    code = DEFAULT_CONFIG.as_dict()
+    for key, seed_value in seeded.items():
+        assert key in code, f"migrations seeds requestlog.{key} but RequestLogConfig has no such field"
+        assert code[key] == seed_value, (
+            f"requestlog.{key} default drifted: migrations.py seeds {seed_value!r} but "
+            f"RequestLogConfig has {code[key]!r}. A fresh install and an upgraded install "
+            f"would then behave differently."
+        )
+
+    for key in code:
+        assert key in seeded, (
+            f"RequestLogConfig has {key!r} but migrations.py does not seed requestlog.{key} — "
+            f"existing installs would silently fall back to the in-code default"
+        )
+
+
+def test_log_viewer_is_excluded_by_default():
+    assert "/api/request-logs" in DEFAULT_EXCLUDE_PATHS
+    assert "/api/health" in DEFAULT_EXCLUDE_PATHS
+    assert "/.well-known/acme-challenge" in DEFAULT_EXCLUDE_PATHS, (
+        "the ACME challenge endpoint returns key_authorization — logging it would store "
+        "the challenge secret"
+    )
+    assert "/api/agents/heartbeat" in DEFAULT_EXCLUDE_PATHS, (
+        "the agent heartbeat is the highest-volume POST in the system; logging it by "
+        "default would dominate the table"
+    )
+
+
+def test_error_retention_defaults_longer_than_success_retention():
+    assert DEFAULT_CONFIG.error_retention_days > DEFAULT_CONFIG.success_retention_days, (
+        "the whole point of splitting the two is to keep failures around after the "
+        "ordinary traffic has aged out"
+    )
+
+
+# --------------------------------------------------------------------------
+# Coercion and clamping
+# --------------------------------------------------------------------------
+
+def test_raw_json_strings_from_asyncpg_are_parsed():
+    cfg = config_from_mapping({
+        "enabled": True,
+        "max_body_bytes": 4096,
+        "sample_rate": 0.25,
+        "success_retention_days": 3,
+        "exclude_paths": ["/api/health", "/metrics"],
+    })
+    assert cfg.enabled is True
+    assert cfg.max_body_bytes == 4096
+    assert cfg.sample_rate == 0.25
+    assert cfg.success_retention_days == 3
+    assert cfg.exclude_paths == ("/api/health", "/metrics")
+
+
+@pytest.mark.parametrize("raw,expected", [
+    ("true", True), ("false", False), ("1", True), ("0", False),
+    ("on", True), ("off", False), (1, True), (0, False), (True, True),
+])
+def test_boolean_coercion_accepts_hand_written_values(raw, expected):
+    cfg = config_from_mapping({"enabled": raw})
+    assert cfg.enabled is expected
+
+
+@pytest.mark.parametrize("field,value,expected", [
+    ("max_body_bytes", 10_000_000, 262144),
+    ("max_body_bytes", -5, 0),
+    ("success_retention_days", 0, 1),
+    ("success_retention_days", 9999, 365),
+    ("error_retention_days", 0, 1),
+    ("max_rows", 10, 1000),
+    ("prune_interval_minutes", 1, 5),
+    ("prune_interval_minutes", 99999, 1440),
+])
+def test_out_of_range_values_are_clamped_not_rejected(field, value, expected):
+    """A bad value in the table must not disable logging or crash the writer —
+    it is clamped to the nearest sane bound."""
+    cfg = config_from_mapping({field: value})
+    assert getattr(cfg, field) == expected
+
+
+@pytest.mark.parametrize("value,expected", [(1.5, 1.0), (-0.2, 0.0), ("0.4", 0.4)])
+def test_sample_rate_is_clamped(value, expected):
+    assert config_from_mapping({"sample_rate": value}).sample_rate == expected
+
+
+def test_garbage_values_fall_back_to_the_default():
+    cfg = config_from_mapping({"max_body_bytes": "not-a-number", "sample_rate": "abc"})
+    assert cfg.max_body_bytes == DEFAULT_CONFIG.max_body_bytes
+    assert cfg.sample_rate == DEFAULT_CONFIG.sample_rate
+
+
+def test_exclude_paths_shape_is_enforced():
+    out = normalize_exclude_paths(
+        ["/good", "no-leading-slash", "/" + "x" * 500, 42, "/also-good"],
+        DEFAULT_EXCLUDE_PATHS,
+    )
+    assert out == ("/good", "/also-good")
+
+
+def test_exclude_paths_count_is_bounded():
+    out = normalize_exclude_paths([f"/p{i}" for i in range(500)], DEFAULT_EXCLUDE_PATHS)
+    assert len(out) <= 64
+
+
+def test_empty_exclude_paths_falls_back_rather_than_logging_everything():
+    """An empty list would re-enable logging of health checks and the docs, and
+    flood the table — treat it as 'not configured'."""
+    assert normalize_exclude_paths([], DEFAULT_EXCLUDE_PATHS) == DEFAULT_EXCLUDE_PATHS
+    assert normalize_exclude_paths(None, DEFAULT_EXCLUDE_PATHS) == DEFAULT_EXCLUDE_PATHS
+
+
+def test_partial_mapping_keeps_the_other_defaults():
+    cfg = config_from_mapping({"sample_rate": 0.5})
+    assert cfg.sample_rate == 0.5
+    assert cfg.success_retention_days == DEFAULT_CONFIG.success_retention_days
+    assert cfg.enabled is DEFAULT_CONFIG.enabled
+
+
+# --------------------------------------------------------------------------
+# refresh_config
+# --------------------------------------------------------------------------
+
+def test_refresh_config_parses_the_raw_jsonb_strings_asyncpg_returns():
+    conn = AsyncMock()
+    conn.fetch = AsyncMock(return_value=[
+        {"key": "requestlog.enabled", "value": "false"},
+        {"key": "requestlog.max_body_bytes", "value": "4096"},
+        {"key": "requestlog.sample_rate", "value": "0.5"},
+        {"key": "requestlog.exclude_paths", "value": '["/api/health","/metrics"]'},
+    ])
+
+    with patch.object(request_log_settings, "get_database_connection", AsyncMock(return_value=conn)), \
+         patch.object(request_log_settings, "close_database_connection", AsyncMock()):
+        cfg = asyncio.run(refresh_config())
+
+    assert cfg.enabled is False
+    assert cfg.max_body_bytes == 4096
+    assert cfg.sample_rate == 0.5
+    assert cfg.exclude_paths == ("/api/health", "/metrics")
+
+    set_config(DEFAULT_CONFIG)
+
+
+def test_refresh_config_keeps_the_previous_snapshot_on_db_failure():
+    """A transient pool error must not silently flip logging on or off."""
+    known = RequestLogConfig(enabled=False, sample_rate=0.1)
+    set_config(known)
+
+    with patch.object(request_log_settings, "get_database_connection",
+                      AsyncMock(side_effect=RuntimeError("pool exhausted"))), \
+         patch.object(request_log_settings, "close_database_connection", AsyncMock()):
+        cfg = asyncio.run(refresh_config())
+
+    assert cfg.enabled is False
+    assert cfg.sample_rate == 0.1
+    set_config(DEFAULT_CONFIG)
+
+
+def test_get_config_is_synchronous_and_needs_no_database():
+    """The middleware calls this on every request; it must never await."""
+    assert not asyncio.iscoroutinefunction(get_config)
+    assert isinstance(get_config(), RequestLogConfig)
+
+
+# --------------------------------------------------------------------------
+# The Pydantic model the API exposes
+# --------------------------------------------------------------------------
+
+def test_api_model_defaults_match_the_dataclass():
+    from routers.request_logs import RequestLogSettings
+
+    model = RequestLogSettings().model_dump()
+    code = DEFAULT_CONFIG.as_dict()
+    for key, value in code.items():
+        assert model[key] == value, f"API model default for {key} disagrees with RequestLogConfig"
+
+
+@pytest.mark.parametrize("payload", [
+    {"max_body_bytes": 999999},
+    {"success_retention_days": 0},
+    {"error_retention_days": 400},
+    {"sample_rate": 1.5},
+    {"max_rows": 10},
+    {"prune_interval_minutes": 1},
+    {"exclude_paths": ["no-slash"]},
+    {"exclude_paths": ["/" + "x" * 300]},
+])
+def test_api_model_rejects_out_of_range_input(payload):
+    from pydantic import ValidationError
+
+    from routers.request_logs import RequestLogSettings
+
+    with pytest.raises(ValidationError):
+        RequestLogSettings(**payload)

backend/tests/test_request_log_sink.py

@@ -0,0 +1,276 @@
+"""v1.11.0: the batching writer must never slow down or break a request.
+
+One row per API call is the highest write volume in the system and the asyncpg
+pool (min=10/max=50) is shared with every handler and four background loops. So
+the hot path enqueues and returns; a single writer task batches and inserts.
+The properties pinned here:
+
+  * `offer()` never blocks and never raises — a full queue drops and counts;
+  * the parameter list stays aligned with the INSERT placeholders (a column
+    added to one and not the other would fail every write at runtime, in
+    production, with the migration already applied);
+  * a failed batch is dropped with a warning rather than killing the loop.
+"""
+import asyncio
+import json
+import os
+import re
+import sys
+from datetime import datetime, timezone
+from unittest.mock import AsyncMock, patch
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from dataclasses import replace  # noqa: E402
+
+from utils import request_log_settings  # noqa: E402
+from utils import request_log_sink as sink_module  # noqa: E402
+from utils.request_log_sink import (  # noqa: E402
+    RequestLogRow,
+    RequestLogSink,
+    _INSERT_SQL,
+)
+from utils.request_log_settings import DEFAULT_CONFIG  # noqa: E402
+
+
+def _row(**overrides):
+    base = dict(
+        request_id="abc123",
+        direction="inbound",
+        method="POST",
+        url="/api/backends",
+        path="/api/backends",
+        status_code=200,
+        duration_ms=12,
+        created_at=datetime(2026, 8, 11, 9, 0, tzinfo=timezone.utc),
+    )
+    base.update(overrides)
+    return RequestLogRow(**base)
+
+
+@pytest.fixture(autouse=True)
+def defaults(monkeypatch):
+    monkeypatch.setattr(request_log_settings, "_CACHE", DEFAULT_CONFIG)
+    monkeypatch.setattr(sink_module, "get_config", lambda: request_log_settings._CACHE)
+
+
+def _set(monkeypatch, **overrides):
+    monkeypatch.setattr(request_log_settings, "_CACHE", replace(DEFAULT_CONFIG, **overrides))
+
+
+# --------------------------------------------------------------------------
+# SQL / parameter alignment
+# --------------------------------------------------------------------------
+
+def test_insert_placeholders_match_the_column_list():
+    columns = _INSERT_SQL.split("(", 1)[1].split(")", 1)[0]
+    n_columns = len([c for c in columns.split(",") if c.strip()])
+    n_placeholders = len(set(re.findall(r"\$(\d+)", _INSERT_SQL)))
+
+    assert n_columns == n_placeholders, (
+        f"the INSERT names {n_columns} columns but binds {n_placeholders} placeholders — "
+        f"every write would fail at runtime, on a database where the migration has "
+        f"already succeeded"
+    )
+
+
+def test_row_produces_exactly_as_many_params_as_the_insert_binds():
+    n_placeholders = len(set(re.findall(r"\$(\d+)", _INSERT_SQL)))
+    assert len(_row().to_params()) == n_placeholders, (
+        "RequestLogRow.to_params() drifted from _INSERT_SQL"
+    )
+
+
+def test_jsonb_params_are_serialized_strings_not_dicts():
+    """No JSONB codec is registered on this pool, so JSONB values travel as text
+    and are cast in SQL — handing asyncpg a dict raises."""
+    row = _row(
+        query_params={"page": "2"},
+        request_headers={"content-type": "application/json"},
+        request_body_value={"name": "web"},
+    )
+    params = row.to_params()
+
+    for value in params:
+        assert not isinstance(value, (dict, list)), (
+            f"{value!r} was passed as a Python container; asyncpg cannot bind it to a "
+            f"jsonb parameter"
+        )
+
+    assert json.loads(params[6]) == {"page": "2"}
+
+
+def test_client_ip_is_never_a_placeholder_string():
+    """client_ip is an INET column: 'unknown' or a comma-joined X-Forwarded-For
+    raises on INSERT."""
+    params = _row(client_ip=None).to_params()
+    assert params[12] is None
+
+
+def test_status_class_is_zero_when_there_was_no_response():
+    assert _row(status_code=None).status_class == 0
+    assert _row(status_code=204).status_class == 2
+    assert _row(status_code=503).status_class == 5
+
+
+# --------------------------------------------------------------------------
+# offer(): the hot path
+# --------------------------------------------------------------------------
+
+def test_offer_drops_and_counts_when_the_queue_is_full():
+    sink = RequestLogSink(maxsize=3, batch_size=10, flush_ms=10)
+
+    async def run():
+        for _ in range(10):
+            sink.offer(_row())
+
+    asyncio.run(run())
+
+    assert sink.stats["queued"] == 3
+    assert sink.stats["dropped"] == 7, (
+        "a full queue must drop and count, never block the request or raise"
+    )
+
+
+def test_offer_never_raises_on_a_broken_row():
+    sink = RequestLogSink(maxsize=10, batch_size=10, flush_ms=10)
+
+    async def run():
+        sink.offer(None)  # not a RequestLogRow at all
+
+    asyncio.run(run())  # must not raise
+
+
+def test_offer_respects_the_kill_switch(monkeypatch):
+    _set(monkeypatch, enabled=False)
+    sink = RequestLogSink(maxsize=10, batch_size=10, flush_ms=10)
+
+    asyncio.run(_offer(sink, _row()))
+    assert sink.stats["queued"] == 0
+
+
+def test_offer_respects_the_per_direction_switches(monkeypatch):
+    _set(monkeypatch, capture_outbound=False)
+    sink = RequestLogSink(maxsize=10, batch_size=10, flush_ms=10)
+
+    async def run():
+        sink.offer(_row(direction="outbound", target="acme"))
+        sink.offer(_row(direction="inbound"))
+
+    asyncio.run(run())
+    assert sink.stats["queued"] == 1
+
+
+def test_sampling_never_drops_errors(monkeypatch):
+    """A sample rate of zero must still capture every failure — that is the whole
+    point of sampling successes only."""
+    _set(monkeypatch, sample_rate=0.0)
+    sink = RequestLogSink(maxsize=100, batch_size=10, flush_ms=10)
+
+    async def run():
+        for _ in range(20):
+            sink.offer(_row(status_code=200))
+        for _ in range(5):
+            sink.offer(_row(status_code=500))
+        for _ in range(5):
+            sink.offer(_row(status_code=None))
+
+    asyncio.run(run())
+    assert sink.stats["queued"] == 10, (
+        "sampling removed error rows; only 1xx/2xx/3xx inbound traffic may be sampled out"
+    )
+
+
+def test_sampling_does_not_touch_outbound_rows(monkeypatch):
+    _set(monkeypatch, sample_rate=0.0)
+    sink = RequestLogSink(maxsize=100, batch_size=10, flush_ms=10)
+
+    async def run():
+        for _ in range(5):
+            sink.offer(_row(direction="outbound", target="acme", status_code=200))
+
+    asyncio.run(run())
+    assert sink.stats["queued"] == 5, (
+        "outbound calls are low-volume and high-value; sampling them away hides which CA "
+        "or DNS call was made"
+    )
+
+
+def test_capture_bodies_off_strips_the_payload_before_queueing(monkeypatch):
+    _set(monkeypatch, capture_bodies=False)
+    sink = RequestLogSink(maxsize=10, batch_size=10, flush_ms=10)
+    row = _row(request_body_raw=b'{"a":1}', request_body_bytes=7)
+
+    asyncio.run(_offer(sink, row))
+
+    assert row.request_body_raw is None
+    assert row.request_body_bytes == 7, "the size must survive so growth is still measurable"
+
+
+async def _offer(sink, row):
+    sink.offer(row)
+
+
+# --------------------------------------------------------------------------
+# The writer
+# --------------------------------------------------------------------------
+
+def test_a_batch_is_written_with_one_executemany():
+    conn = AsyncMock()
+    sink = RequestLogSink(maxsize=100, batch_size=10, flush_ms=10)
+
+    async def run():
+        for _ in range(5):
+            sink.offer(_row())
+        with patch.object(sink_module, "get_database_connection", AsyncMock(return_value=conn)), \
+             patch.object(sink_module, "close_database_connection", AsyncMock()):
+            return await sink.flush(timeout=1.0)
+
+    written = asyncio.run(run())
+
+    assert written == 5
+    assert conn.executemany.await_count == 1, (
+        "rows were inserted one at a time; that is one pool acquire per API call and the "
+        "pool has 50 connections"
+    )
+    sql, params = conn.executemany.await_args.args
+    assert "INSERT INTO request_logs" in sql
+    assert len(params) == 5
+
+
+def test_a_failed_batch_does_not_kill_the_writer():
+    conn = AsyncMock()
+    conn.executemany = AsyncMock(side_effect=RuntimeError("relation does not exist"))
+    sink = RequestLogSink(maxsize=100, batch_size=10, flush_ms=10)
+
+    async def run():
+        sink.offer(_row())
+        with patch.object(sink_module, "get_database_connection", AsyncMock(return_value=conn)), \
+             patch.object(sink_module, "close_database_connection", AsyncMock()):
+            await sink.flush(timeout=1.0)
+
+    asyncio.run(run())  # must not raise
+    assert sink.stats["failed_batches"] == 1
+
+
+def test_the_connection_is_released_even_when_the_write_fails():
+    conn = AsyncMock()
+    conn.executemany = AsyncMock(side_effect=RuntimeError("boom"))
+    release = AsyncMock()
+    sink = RequestLogSink(maxsize=100, batch_size=10, flush_ms=10)
+
+    async def run():
+        sink.offer(_row())
+        with patch.object(sink_module, "get_database_connection", AsyncMock(return_value=conn)), \
+             patch.object(sink_module, "close_database_connection", release):
+            await sink.flush(timeout=1.0)
+
+    asyncio.run(run())
+    assert release.await_count == 1, "a failed batch leaked a pooled connection"
+
+
+def test_flush_on_an_empty_queue_is_a_noop():
+    sink = RequestLogSink(maxsize=10, batch_size=10, flush_ms=10)
+    assert asyncio.run(sink.flush(timeout=0.1)) == 0

backend/tests/test_vip_adoption_instance.py

@@ -261,6 +261,144 @@ def test_takeover_authorisation_is_retired_once_the_node_acks_our_config():
     )
 
 
+def test_validation_judges_the_output_not_the_exit_code():
+    """keepalived's config-test exit code cannot separate fatal from benign.
+
+    Measured on keepalived 2.2.8 rather than assumed:
+
+        clean config ..................... 0
+        auth_pass longer than 8 chars .... 5   "Truncating auth_pass to 8 characters"
+        missing '}' ...................... 5   "There are 1 missing '}'s"
+        unknown keyword .................. 5   "Unknown keyword '...'"
+        script without script_security ... 6   "SECURITY VIOLATION ..."
+
+    So 5 covers both a harmless truncation and a broken file. Treating any non-zero exit as
+    invalid rejected VALID configs: a VRRP password over 8 characters is enough, and keepalived
+    truncates it to 8 anyway, exactly as it does for the file the operator already runs. Found
+    on the first live adoption, where the node's OWN running config also exited non-zero.
+
+    The gate must therefore judge the output, and it must fail CLOSED: anything not on the
+    benign list still counts as fatal.
+    """
+    script = (BACKEND / "utils" / "agent_scripts" / "linux_install.sh").read_text()
+    assert script.count("grep -v 'Truncating auth_pass to 8 characters'") == 2, (
+        "both daemon copies must drop the known-benign truncation warning before judging"
+    )
+    assert script.count('if [[ -n "$kp_fatal" ]]; then') == 2, (
+        "the decision must be made on what REMAINS after the benign lines are dropped"
+    )
+    # Fail-closed: the benign list is an allowlist, never a denylist of fatal messages. The
+    # measured table above is quoted in a comment inside the script, so assert on what is used
+    # as a grep PATTERN rather than on the text appearing anywhere.
+    patterns = re.findall(r"grep -v '([^']*)'", script)
+    assert set(patterns) == {"Truncating auth_pass to 8 characters", "^[[:space:]]*$"}, (
+        f"the validation filter greps for {sorted(set(patterns))}. It must drop only messages "
+        f"known to be benign; matching on fatal messages instead would let an unrecognised "
+        f"error through."
+    )
+
+
+def test_validation_fails_closed_when_keepalived_says_nothing():
+    """A non-zero exit with no output must stay fatal.
+
+    Filtering the output introduces a way to reach the accept path with an EMPTY filter result,
+    and some keepalived builds log to syslog rather than stderr — on such a host every config
+    would then be accepted regardless of what is wrong with it. Verified against a stub that
+    exits non-zero silently, and against one that emits only whitespace.
+    """
+    script = (BACKEND / "utils" / "agent_scripts" / "linux_install.sh").read_text()
+    assert script.count('if [[ -z "${kp_out//[[:space:]]/}" ]]; then') == 2, (
+        "both daemon copies must treat a non-zero exit with no readable output as fatal"
+    )
+    assert script.count('kp_fatal="keepalived -t exited non-zero without output"') == 2
+
+
+def test_validation_failure_reports_what_keepalived_said():
+    """The fail-safe protected the node correctly on a live adoption but logged only
+    "config validation failed", with keepalived's own output sent to /dev/null. The operator
+    had no way to act on it without reproducing the check by hand on the node."""
+    script = (BACKEND / "utils" / "agent_scripts" / "linux_install.sh").read_text()
+    assert 'keepalived -t -f "$tmp_conf" >/dev/null 2>&1' not in script, (
+        "keepalived's output must not be discarded — a fail-safe that cannot say why it fired "
+        "is only half a safety feature"
+    )
+    assert script.count('kp_out=$(keepalived -t -f "$tmp_conf" 2>&1)') == 2, (
+        "both daemon copies must capture the validation output"
+    )
+    # The text is interpolated into JSON by both `log` and _kp_report, so it must be sanitised.
+    assert script.count(r"""tr -d '"\\'""") == 2, (
+        "captured output must have quotes and backslashes stripped before it reaches the JSON "
+        "log line and the status report"
+    )
+    assert script.count('keepalived -t failed: ${kp_err}') == 2, (
+        "the reason must also travel to the server so the UI can show it"
+    )
+
+
+def test_converged_node_keeps_acknowledging():
+    """The idempotent path must still report, or a lost ack is never recovered.
+
+    The status report is the server's ONLY evidence that a member converged, and it used to be
+    sent solely on the write path. Once the rendered config was on disk the agent took the
+    idempotency early return every cycle and never spoke again, so a single lost report - a
+    backend restart, a 5xx, a network blip - left the VIP reading SYNCING forever with an empty
+    "Last ack" while the node was demonstrably running the right config. Seen in the field after
+    acks were dropped for an unrelated reason: the node was correct, the page was not, and
+    nothing would ever reconcile them.
+    """
+    script = (BACKEND / "utils" / "agent_scripts" / "linux_install.sh").read_text()
+    assert script.count('_kp_report "enabled" "$vip_id" "$new_hash" "already converged"') == 2, (
+        "both daemon copies must re-assert the deploy state on the idempotent path; without it "
+        "the server can never recover a lost acknowledgement"
+    )
+    # The report has to come BEFORE the early return in both copies.
+    for m in re.finditer(r'if \[\[ -n "\$cur_hash" && "\$cur_hash" == "\$would_hash" \]\]; then(.*?)fi',
+                         script, re.S):
+        body = m.group(1)
+        assert body.index("_kp_report") < body.index("return 0"), (
+            "the acknowledgement must be sent before returning, or the early return skips it"
+        )
+
+
+def test_status_ack_statements_bind_each_placeholder_once():
+    """Every `$n` in the keepalived-status UPDATEs must be used exactly once, and the count must
+    match the arguments passed.
+
+    Reusing one placeholder for both the assignment (`last_deploy_hash=$n`, a VARCHAR column)
+    and the comparison inside the takeover-retirement CASE made PostgreSQL deduce two types for
+    it, and asyncpg rejected the whole statement with AmbiguousParameterError. The failure was
+    not partial: no ack was written at all, so every VIP sat at SYNCING forever and teardown acks
+    were lost too. Shipped in v1.10.12 and caught in the field.
+
+    The suite has no database, so this pins the shape that made it possible rather than the SQL
+    behaviour: one placeholder, one binding site.
+    """
+    src = (BACKEND / "routers" / "agent.py").read_text()
+    start = src.index("async def agent_keepalived_status")
+    seg = src[start:src.index('return {"status": "ok"}', start)]
+
+    retire = "".join(re.findall(r'"([^"]*)"',
+                                re.search(r"_retire_takeover = \((.*?)\)\n", seg, re.S).group(1)))
+    calls = re.findall(r'await conn\.execute\(f"""(.*?)""",\s*(.*?)\)\n', seg, re.S)
+    assert len(calls) == 2, f"expected the two ack UPDATEs, found {len(calls)}"
+
+    for sql, args in calls:
+        placeholder = re.search(r'_retire_takeover\.format\(p="(\$\d+)"\)', sql).group(1)
+        rendered = re.sub(r"\{_retire_takeover\.format\(p=\"\$\d+\"\)\}",
+                          retire.replace("{p}", placeholder), sql)
+        used = re.findall(r"\$(\d+)", rendered)
+        dupes = {n for n in used if used.count(n) > 1}
+        assert not dupes, (
+            f"placeholder(s) {sorted('$'+d for d in dupes)} are bound more than once. PostgreSQL "
+            f"deduces a type per USE, so a placeholder that is both assigned to a column and "
+            f"compared against one is ambiguous and the whole UPDATE is rejected."
+        )
+        n_args = len([a for a in args.split(",") if a.strip()])
+        assert max(int(n) for n in used) == n_args, (
+            f"the statement uses ${max(int(n) for n in used)} but {n_args} arguments are passed"
+        )
+
+
 def test_takeover_still_requires_the_pinned_hash_to_match_on_disk():
     """The guard that stops an edit between adoption and Apply from being overwritten."""
     script = (BACKEND / "utils" / "agent_scripts" / "linux_install.sh").read_text()
@@ -291,3 +429,213 @@ def test_vip_version_transition_matches_any_action():
     assert 'f"vip-{vip_id}-%"' in VIP_ROUTER, (
         "the PENDING -> APPLIED/REJECTED transition must match every action for the VIP"
     )
+
+
+# ----------------------------------------------------------------------------
+# v1.11.1 — a discovery report that was never accepted must be retried
+# ----------------------------------------------------------------------------
+
+def test_discovery_is_cached_only_when_the_server_accepted_it():
+    """`curl` without -f exits 0 on 500/403/404, so the previous `if curl ...` recorded a
+    REJECTED discovery as delivered. The cache then suppressed every later attempt, and since
+    the file never changes on its own the node stayed out of the adoption panel permanently:
+    the only cure was deleting the cache on the node by hand."""
+    script = (BACKEND / "utils" / "agent_scripts" / "linux_install.sh").read_text()
+    assert "if curl -k -s --connect-timeout 10 --max-time 30 -o /dev/null -X POST" not in script, (
+        "the discovery POST must not be judged by curl's exit code; it is 0 for 5xx as well"
+    )
+    assert script.count('if [[ "$disc_code" =~ ^2[0-9][0-9]$ ]]; then') == 2, (
+        "both daemon copies must cache only on a 2xx"
+    )
+
+
+def test_discovery_cache_defers_to_the_server():
+    """Recovery without touching the node. The server reports whether it actually holds a
+    discovery for this agent; only an explicit `false` overrides the cache, so a backend older
+    than v1.11.1 (which omits the field) keeps the previous behaviour instead of being flooded
+    with re-posts."""
+    script = (BACKEND / "utils" / "agent_scripts" / "linux_install.sh").read_text()
+    agent_router = (BACKEND / "routers" / "agent.py").read_text()
+
+    assert agent_router.count('AS discovery_known') == 1, (
+        "the keepalived-config query must report whether a discovery row exists"
+    )
+    assert agent_router.count('"discovery_known": bool(agent["discovery_known"])') == 5, (
+        "every response path that knows the agent must carry the flag — the adoptable nodes are "
+        "precisely the not_configured ones"
+    )
+    assert script.count('jq -r \'if has("discovery_known")') == 2
+    assert script.count('[[ "$disc_known" != "false" ]] && return 0') == 2, (
+        "only an explicit false may override the cache, or an older backend — which omits the "
+        "field — would cause a re-post on every cycle"
+    )
+    assert script.count('conf chk disc_known=""') == 2, (
+        "disc_known must be function-local; in the in-script daemon the enclosing scope is the "
+        "poll loop, so a stale value would outlive the response it came from"
+    )
+
+
+def test_keepalived_config_path_is_resolved_deterministically():
+    """A pool may hold more than one cluster, and the join multiplies the agent row. Without an
+    ordering the fetch took an arbitrary cluster, so the keepalived.conf PATH handed to the agent
+    was non-deterministic whenever two clusters in a pool disagreed on it: the agent would look at
+    the wrong file, find nothing, and the node would never appear for adoption."""
+    src = (BACKEND / "routers" / "agent.py").read_text()
+    start = src.index("async def get_agent_keepalived_config")
+    q_start = src.index('agent = await conn.fetchrow("""', start)
+    query = src[q_start:src.index('""", agent_name)', q_start)]
+    assert "LEFT JOIN haproxy_clusters" in query, "re-point this test; the join moved"
+    assert "ORDER BY" in query and "LIMIT 1" in query, (
+        "the cluster row must be picked deterministically, or the config path the agent is told "
+        "to inspect can change between polls"
+    )
+    assert "hc.keepalived_config_path = '/etc/keepalived/keepalived.conf'" in query, (
+        "the ordering must prefer a CUSTOMISED path over the shipped default. The column defaults "
+        "to that path rather than NULL, so ordering by id alone could pick a default-valued row "
+        "over one the operator deliberately set, turning 'undefined' into 'reliably wrong'."
+    )
+
+
+def test_daemon_copies_agree_on_the_whole_keepalived_path():
+    """Everything the adoption flow depends on must behave identically on BOTH install routes:
+    a freshly installed agent runs the heredoc body, a self-upgraded one runs the in-script
+    daemon. Comments may differ; logic may not."""
+    import difflib
+    lines = (BACKEND / "utils" / "agent_scripts" / "linux_install.sh").read_text().splitlines()
+    term = next(i for i, l in enumerate(lines) if l.strip() == "AGENT_SCRIPT")
+
+    def strip_comment(s):
+        out, q, esc = [], None, False
+        for ch in s:
+            if esc:
+                out.append(ch); esc = False; continue
+            if ch == "\\":
+                out.append(ch); esc = True; continue
+            if q:
+                out.append(ch)
+                if ch == q:
+                    q = None
+                continue
+            if ch in ('"', "'"):
+                q = ch; out.append(ch); continue
+            if ch == "#":
+                break
+            out.append(ch)
+        return "".join(out).rstrip()
+
+    def funcs(block):
+        found = {}
+        for idx, l in enumerate(block):
+            m = re.match(r"^(\s*)([a-zA-Z_][a-zA-Z0-9_]*)\(\)\s*\{\s*(#.*)?$", l)
+            if not m:
+                continue
+            close = m.group(1) + "}"
+            end = next((j for j in range(idx + 1, len(block)) if block[j].rstrip() == close), None)
+            if end is None:
+                continue
+            found[m.group(2)] = [re.sub(r"\s+", " ", strip_comment(x).strip())
+                                 for x in block[idx + 1:end] if strip_comment(x).strip()]
+        return found
+
+    here, insc = funcs(lines[923:term]), funcs(lines[term + 1:])
+    for name in ("_kp_discover", "_kp_report", "_kp_teardown",
+                 "fetch_and_deploy_keepalived_config", "get_keepalive_state"):
+        assert name in here and name in insc, f"{name} is missing from one daemon copy"
+        if here[name] != insc[name]:
+            d = "\n".join(x for x in difflib.unified_diff(here[name], insc[name], lineterm="")
+                          if x[:1] in "+-" and x[:3] not in ("+++", "---"))
+            raise AssertionError(
+                f"{name}() differs between the daemon copies, so a self-upgraded agent would "
+                f"behave differently from a freshly installed one:\n{d}"
+            )
+
+
+def test_discovery_flag_distinguishes_false_from_absent():
+    """jq's `//` returns the alternative for **false** as well as null.
+
+    `.discovery_known // empty` therefore yields an empty string both when the backend omits the
+    field (older release) and when it explicitly says `false` (no discovery on record) — the one
+    case the recovery exists for. Written that way the fix is inert: the cache is never overridden
+    and a stuck node stays hidden. Caught in review, before it shipped, by parsing a real response
+    rather than passing the value in by hand.
+    """
+    script = (BACKEND / "utils" / "agent_scripts" / "linux_install.sh").read_text()
+    assert "'.discovery_known // empty'" not in script, (
+        "jq's // treats false like null, so this cannot tell 'no record' from 'old backend'"
+    )
+    expected = ('disc_known=$(echo "$resp" | jq -r \'if has("discovery_known") '
+                'then (.discovery_known|tostring) else "" end\' 2>/dev/null)')
+    assert script.count(expected) == 2, (
+        "both daemon copies must distinguish an explicit false from an absent field"
+    )
+
+
+def test_discovery_backs_off_on_a_permanent_rejection():
+    """A 4xx means the payload itself is unacceptable, so re-posting the same bytes cannot help.
+
+    Retrying forever is not free here: 4xx and 5xx agent calls are never sampled out of the
+    request log (see request_log_sink), so an unattended loop writes a row carrying the whole
+    keepalived.conf every poll cycle, on every affected node — the exact "polling noise evicts
+    the forensic record" failure the log's own defaults exist to prevent. The cache therefore
+    records the rejection and stays quiet until the file changes; a 5xx or a transport failure
+    is still retried, which is what the recovery depends on.
+    """
+    script = (BACKEND / "utils" / "agent_scripts" / "linux_install.sh").read_text()
+    brake = ('elif [[ "$disc_code" == "400" || "$disc_code" == "413" '
+             '|| "$disc_code" == "422" ]]; then')
+    assert script.count(brake) == 2, (
+        "the brake must be limited to the codes that mean 'these bytes are unacceptable'"
+    )
+    # 401 and 404 are 4xx but TRANSIENT here: a token rotation, or an agent row briefly absent
+    # while it re-registers. Braking on them would silence discovery for every affected node
+    # until its keepalived.conf changed, which for a hand-maintained file may be never — the
+    # exact failure this release removes.
+    for transient in ('"401"', '"404"'):
+        assert transient not in brake, (
+            f"{transient} must stay in the retry class; it does not mean the payload is bad"
+        )
+    assert script.count("""printf '%s rejected' "$cur_hash" > "$cache" 2>/dev/null""") == 2
+    assert script.count('[[ "$cached_state" == "rejected" ]] && return 0') == 2, (
+        "a recorded rejection must suppress the post even when the server reports no record, "
+        "or the flag override turns into an unbounded retry loop"
+    )
+    # The rejection must be keyed to the CONTENT, so a fixed config is retried.
+    assert script.count('cached_hash="${cached_line%% *}"') == 2, (
+        "the rejection is stored against the hash; changing the file must clear the brake"
+    )
+
+
+def test_config_import_reaches_a_freshly_installed_agent():
+    """`check_config_requests` uploads the node's live haproxy.cfg when the operator asks for it.
+
+    It was defined in the installer body and in the in-script daemon, but NOT in the heredoc a
+    fresh install writes to /usr/local/bin/haproxy-agent. Its call site is guarded by
+    `type check_config_requests`, so on a freshly installed agent the whole feature was a silent
+    no-op: the operator requested a config from the node and nothing ever arrived, with no error.
+    Agents that had self-upgraded at least once did have it, which is why it went unnoticed.
+    """
+    lines = (BACKEND / "utils" / "agent_scripts" / "linux_install.sh").read_text().splitlines()
+    term = next(i for i, l in enumerate(lines) if l.strip() == "AGENT_SCRIPT")
+    pattern = re.compile(r"^\s*check_config_requests\(\)\s*\{")
+
+    in_heredoc = sum(1 for l in lines[923:term] if pattern.match(l))
+    in_daemon = sum(1 for l in lines[term + 1:] if pattern.match(l))
+    assert in_heredoc == 1, (
+        "the heredoc a fresh install writes must define check_config_requests, or Config Import "
+        "silently does nothing on any node that has never self-upgraded"
+    )
+    assert in_daemon == 1, "the self-upgrade daemon must keep its definition"
+
+    # And the two must be the same function, not two drifting implementations.
+    def body(block):
+        idx = next(i for i, l in enumerate(block) if pattern.match(l))
+        out = []
+        for l in block[idx:]:
+            out.append(l.strip())
+            if l.strip() == "}" and len(out) > 5:
+                break
+        return out
+
+    assert body(lines[923:term]) == body(lines[term + 1:]), (
+        "the two copies of check_config_requests have drifted"
+    )

backend/utils/agent_scripts/linux_install.sh

@@ -1691,7 +1691,7 @@ check_ssl_updates() {
 # (kept in sync with the live token in both daemon loops).
 fetch_and_deploy_keepalived_config() {
     local marker="# Managed by HAProxy OpenManager"
-    local resp status http_code we_own="false" conf chk
+    local resp status http_code we_own="false" conf chk disc_known=""
 
     # Timeouts so a hung management server can never stall the daemon loop.
     resp=$(curl -k -s --connect-timeout 10 --max-time 30 -w '\n%{http_code}' -X GET \
@@ -1710,6 +1710,9 @@ fetch_and_deploy_keepalived_config() {
     [[ -z "$conf" || "$conf" == "null" ]] && conf="/etc/keepalived/keepalived.conf"
     chk="$(dirname "$conf")/check_haproxy.sh"
     if [[ -f "$conf" ]] && grep -q "$marker" "$conf" 2>/dev/null; then we_own="true"; fi
+    # Whether the SERVER already holds a discovery for this node. Absent on backends older than
+    # v1.11.1, in which case the discovery cache keeps its previous meaning.
+    disc_known=$(echo "$resp" | jq -r 'if has("discovery_known") then (.discovery_known|tostring) else "" end' 2>/dev/null)
 
     # v1.10.4 — VIP adoption discovery. Report a keepalived.conf we do NOT own so an existing
     # VIP can be adopted from the UI instead of retyped. STRICTLY READ-ONLY: this never writes
@@ -1724,28 +1727,75 @@ fetch_and_deploy_keepalived_config() {
         if [[ "$we_own" == "true" || ! -f "$conf" ]]; then
             # Nothing adoptable here. Clear a previous report exactly once.
             [[ -f "$cache" ]] || return 0
-            curl -k -s --connect-timeout 10 --max-time 30 -X POST \
-                "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-discovery" \
+            # Clear ONLY when the server accepted it. curl's exit code is 0 for 5xx too, so
+            # dropping the cache on a rejected clear lost the fact that this node is ours: the
+            # stale discovery row would keep offering a MANAGED node for adoption, and with the
+            # cache gone nothing would ever send the clear again.
+            local clr_code
+            clr_code=$(curl -k -s --connect-timeout 10 --max-time 30 -o /dev/null -w '%{http_code}' \
+                -X POST "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-discovery" \
                 -H "X-API-Key: $AGENT_TOKEN" -H "Content-Type: application/json" \
-                -d "{\"config_path\":\"$conf\",\"exists\":false}" >/dev/null 2>&1 || return 0
+                -d "{\"config_path\":\"$conf\",\"exists\":false}" 2>/dev/null)
+            [[ "$clr_code" =~ ^2[0-9][0-9]$ ]] || return 0
             rm -f "$cache"
             return 0
         fi
         cur_hash=$(md5sum "$conf" 2>/dev/null | awk '{print $1}')
         [[ -z "$cur_hash" ]] && return 0
-        [[ -f "$cache" && "$(cat "$cache" 2>/dev/null)" == "$cur_hash" ]] && return 0
+        # The cache may only suppress a report while the SERVER agrees it already holds one.
+        # v1.11.1: it suppressed unconditionally, so a report the server REJECTED was recorded
+        # as delivered and the node stayed out of the adoption panel for good — the file never
+        # changes, so nothing ever triggered another attempt and the only cure was deleting this
+        # file on the node by hand. `discovery_known` is absent on older backends, and only an
+        # explicit "false" overrides the cache, so an old server keeps the previous behaviour
+        # rather than being flooded with re-posts.
+        # Cache line is "<hash>" after a success, or "<hash> rejected" after a 4xx.
+        local cached_line cached_hash cached_state=""
+        cached_line=$(cat "$cache" 2>/dev/null)
+        cached_hash="${cached_line%% *}"
+        [[ "$cached_line" == *" "* ]] && cached_state="${cached_line#* }"
+        if [[ -n "$cached_hash" && "$cached_hash" == "$cur_hash" ]]; then
+            # A 4xx means the server refuses THESE BYTES. Re-posting them cannot succeed, and
+            # every attempt is stored as a failed agent call (4xx/5xx are never sampled out), so
+            # an unattended loop would write a row carrying the whole config every cycle. Stay
+            # quiet until the file changes — the reason was logged when it was rejected.
+            [[ "$cached_state" == "rejected" ]] && return 0
+            # Otherwise the cache only holds while the SERVER agrees it has the report.
+            [[ "$disc_known" != "false" ]] && return 0
+        fi
         # jq -Rs makes the file a single JSON string with its newlines intact, so the content the
         # server hashes is byte-identical to what is on disk — the takeover authorisation is
         # pinned to that hash.
         content_json=$(jq -Rs . < "$conf" 2>/dev/null) || return 0
         body=$(jq -n --arg p "$conf" --argjson c "$content_json" \
             '{config_path:$p, exists:true, is_managed:false, config_content:$c}' 2>/dev/null) || return 0
-        if curl -k -s --connect-timeout 10 --max-time 30 -o /dev/null -X POST \
-            "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-discovery" \
+        # Cache ONLY on a 2xx. curl without -f exits 0 on 500/403/404 too, so the previous
+        # `if curl ...` recorded a REJECTED report as delivered — and since the cache suppresses
+        # every later attempt until the file itself changes, one server-side error hid the node
+        # from the adoption panel permanently. Same failure shape as the deploy acknowledgement
+        # fixed in v1.10.14, on the discovery path.
+        local disc_code
+        disc_code=$(curl -k -s --connect-timeout 10 --max-time 30 -o /dev/null -w '%{http_code}' \
+            -X POST "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-discovery" \
             -H "X-API-Key: $AGENT_TOKEN" -H "Content-Type: application/json" \
-            --data-binary "$body" 2>/dev/null; then
+            --data-binary "$body" 2>/dev/null)
+        if [[ "$disc_code" =~ ^2[0-9][0-9]$ ]]; then
             printf '%s' "$cur_hash" > "$cache" 2>/dev/null
             log "INFO" "KEEPALIVED: reported an unmanaged keepalived.conf for adoption"
+        elif [[ "$disc_code" == "400" || "$disc_code" == "413" || "$disc_code" == "422" ]]; then
+            # ONLY the codes that mean "these bytes are unacceptable" stop the retry: too large
+            # to analyse, malformed, rejected by validation. Re-posting identical content cannot
+            # change any of those answers.
+            #
+            # 401 and 404 are deliberately NOT here even though they are 4xx. Both are transient
+            # in this system — a token rotation, or an agent row briefly absent while it
+            # re-registers — and treating them as permanent would silence discovery for every
+            # affected node until its keepalived.conf changed, which for a hand-maintained file
+            # may be never. That is the exact failure this release set out to remove.
+            printf '%s rejected' "$cur_hash" > "$cache" 2>/dev/null
+            log "WARN" "KEEPALIVED: discovery report refused (HTTP $disc_code); not retrying until the config changes"
+        else
+            log "WARN" "KEEPALIVED: discovery report failed (HTTP ${disc_code:-none}); will retry next cycle"
         fi
     }
     _kp_discover
@@ -1865,6 +1915,13 @@ fetch_and_deploy_keepalived_config() {
         cur_hash=$(md5sum "$conf" 2>/dev/null | awk '{print $1}')
         would_hash=$(printf '%s' "$new_conf" | md5sum 2>/dev/null | awk '{print $1}')
         if [[ -n "$cur_hash" && "$cur_hash" == "$would_hash" ]]; then
+            # STILL ACK. This report is the server's only evidence that the node converged, and
+            # it used to be sent on the write path alone — so a single lost ack (a backend
+            # restart, a 5xx, a network blip) left the VIP reading SYNCING forever: the node was
+            # already correct on disk, took this early return every cycle, and never spoke again.
+            # Re-asserting the state makes the loop self-healing, costs one request per ~2.5
+            # minutes, and touches nothing on the node.
+            _kp_report "enabled" "$vip_id" "$new_hash" "already converged"
             return 0
         fi
     fi
@@ -1880,11 +1937,43 @@ fetch_and_deploy_keepalived_config() {
     local tmp_conf="${conf}.hom.tmp"
     printf '%s' "$new_conf" > "$tmp_conf"
     chmod 0644 "$tmp_conf"
-    if ! keepalived -t -f "$tmp_conf" >/dev/null 2>&1; then
-        rm -f "$tmp_conf"
-        log "ERROR" "KEEPALIVED: config validation failed (keepalived -t) — keeping current config, not (re)starting"
-        _kp_report "error" "$vip_id" "$new_hash" "keepalived -t failed"
-        return 0
+    # Capture what keepalived actually said. Discarding it made the fail-safe useless in
+    # practice: the node was correctly protected, but neither the log nor the UI could say WHY,
+    # so the only way forward was to reproduce the check by hand on the node. Sanitised hard
+    # (quotes, backslashes and newlines removed, tail kept) because both `log` and _kp_report
+    # embed the text in JSON built by string interpolation.
+    local kp_out kp_err kp_fatal
+    if ! kp_out=$(keepalived -t -f "$tmp_conf" 2>&1); then
+        # keepalived's config-test EXIT CODE does not separate a fatal config error from a
+        # harmless warning. Measured on 2.2.8, not assumed:
+        #     clean config .................. 0
+        #     auth_pass longer than 8 chars . 5   "Truncating auth_pass to 8 characters"
+        #     missing '}' ................... 5   "There are 1 missing '}'s"
+        #     unknown keyword ............... 5   "Unknown keyword '...'"
+        #     script without script_security  6   "SECURITY VIOLATION ..."
+        # So 5 covers both a benign truncation and a broken file, and treating any non-zero
+        # exit as invalid rejected VALID configs: a VRRP password over 8 characters is enough,
+        # and keepalived truncates it to 8 regardless, exactly as it does for the file the
+        # operator already runs. Judge on the OUTPUT instead, dropping only messages known to
+        # be benign; anything unrecognised is still fatal, so this fails CLOSED.
+        if [[ -z "${kp_out//[[:space:]]/}" ]]; then
+            # Non-zero with NOTHING to read. We cannot confirm the reason is benign, and some
+            # builds log to syslog rather than stderr, so proceeding here would silently accept
+            # every config on such a host. Treat as fatal — the gate must fail closed.
+            kp_fatal="keepalived -t exited non-zero without output"
+        else
+            kp_fatal=$(printf '%s\n' "$kp_out" \
+                | grep -v 'Truncating auth_pass to 8 characters' \
+                | grep -v '^[[:space:]]*$' || true)
+        fi
+        if [[ -n "$kp_fatal" ]]; then
+            rm -f "$tmp_conf"
+            kp_err=$(printf '%s' "$kp_fatal" | tr '\n\r\t' '   ' | tr -d '"\\' | tail -c 300)
+            log "ERROR" "KEEPALIVED: config validation failed (keepalived -t): ${kp_err} — keeping current config, not (re)starting"
+            _kp_report "error" "$vip_id" "$new_hash" "keepalived -t failed: ${kp_err}"
+            return 0
+        fi
+        log "WARN" "KEEPALIVED: keepalived -t exited non-zero with only known-benign warnings; proceeding"
     fi
     mv -f "$tmp_conf" "$conf"
     chown root:root "$conf" 2>/dev/null
@@ -2154,6 +2243,95 @@ UPGRADE_EOF
 }
 
 # Check and apply configuration updates (Linux)
+# Config Import: upload this node's live haproxy.cfg when the operator asks for it.
+#
+# This function existed ONLY in the installer body and in the in-script daemon, never in the
+# heredoc that a FRESH install writes to /usr/local/bin/haproxy-agent. The call site below is
+# guarded by `type check_config_requests`, so on a freshly installed agent the whole feature
+# was a silent no-op: the operator requested a config from the node and nothing ever arrived,
+# with no error anywhere. Agents that had self-upgraded at least once did have it, which is why
+# it went unnoticed. Copied verbatim from the in-script daemon so both install routes behave
+# identically (v1.11.1).
+check_config_requests() {
+    local curl_bin="${CURL_BIN:-$(find_binary curl)}"
+    
+    # Get pending config requests from backend
+    log "DEBUG" "CONFIG: Checking for pending config requests from: ${MANAGEMENT_URL}/api/configuration/agents/${AGENT_NAME}/pending-requests"
+    
+    local response=$("$curl_bin" -k -s -X GET "${MANAGEMENT_URL}/api/configuration/agents/${AGENT_NAME}/pending-requests" \
+        -H "Content-Type: application/json" \
+        -H "X-API-Key: ${AGENT_TOKEN}")
+    
+    log "DEBUG" "CONFIG: Response: ${response:0:200}..."
+    
+    # Check if there are pending requests
+    local pending_count=$(echo "$response" | jq -r '.pending_requests | length' 2>/dev/null)
+    
+    log "DEBUG" "CONFIG: Pending count: $pending_count"
+    
+    if [[ "$pending_count" -gt 0 ]]; then
+        log "INFO" "CONFIG: Found $pending_count pending config request(s)"
+        
+        # Process each request
+        echo "$response" | jq -c '.pending_requests[]' 2>/dev/null | while read -r request; do
+            local request_id=$(echo "$request" | jq -r '.request_id')
+            local request_type=$(echo "$request" | jq -r '.request_type')
+            
+            log "INFO" "CONFIG: Processing config request #$request_id (type: $request_type)"
+            
+            # Read haproxy.cfg content
+            local config_path="${HAPROXY_CONFIG_PATH}"
+            if [[ -f "$config_path" ]]; then
+                local file_size=$(wc -c < "$config_path")
+                log "INFO" "CONFIG: Reading config from: $config_path (size: $file_size bytes)"
+                
+                local config_content=$(cat "$config_path")
+                
+                # Escape config content for JSON
+                log "DEBUG" "CONFIG: Escaping config content for JSON..."
+                local escaped_content=$(echo "$config_content" | jq -Rs .)
+                
+                if [[ -z "$escaped_content" ]]; then
+                    log "ERROR" "CONFIG: JSON escaping failed for config content!"
+                    continue
+                fi
+                
+                local escaped_size=${#escaped_content}
+                log "DEBUG" "CONFIG: JSON escaped content size: $escaped_size bytes"
+                
+                # Submit config response
+                local response_payload=$(cat <<CONFIG_RESPONSE_EOF
+{
+"request_id": $request_id,
+"config_content": $escaped_content,
+"config_path": "$config_path"
+}
+CONFIG_RESPONSE_EOF
+)
+                
+                local payload_size=${#response_payload}
+                log "INFO" "CONFIG: Sending config response (payload size: $payload_size bytes)..."
+                
+                local http_response=$("$curl_bin" -k -s -w "\n%{http_code}" -X POST "${MANAGEMENT_URL}/api/configuration/agents/${AGENT_NAME}/config-response" \
+                    -H "Content-Type: application/json" \
+                    -H "X-API-Key: ${AGENT_TOKEN}" \
+                    -d "$response_payload")
+                
+                local http_code=$(echo "$http_response" | tail -n1)
+                local response_body=$(echo "$http_response" | head -n-1)
+                
+                if [[ "$http_code" == "200" || "$http_code" == "201" ]]; then
+                    log "INFO" "CONFIG: Config response sent for request #$request_id (size: $file_size bytes, HTTP $http_code)"
+                else
+                    log "ERROR" "CONFIG: Config response failed for request #$request_id (HTTP $http_code): $response_body"
+                fi
+            else
+                log "ERROR" "CONFIG: Config file not found: $config_path"
+            fi
+        done
+    fi
+}
+
 check_config_updates() {
     # Only log config check in debug mode to reduce log spam
     [[ "${DEBUG_MODE:-0}" == "1" ]] && log "INFO" "Checking for configuration updates..."
@@ -3300,7 +3478,7 @@ CONFIG_RESPONSE_EOF
     # the live token at the top of each loop iteration below.
     fetch_and_deploy_keepalived_config() {
         local marker="# Managed by HAProxy OpenManager"
-        local resp status http_code we_own="false" conf chk
+        local resp status http_code we_own="false" conf chk disc_known=""
 
         # Timeouts so a hung management server can never stall the daemon loop.
         resp=$(curl -k -s --connect-timeout 10 --max-time 30 -w '\n%{http_code}' -X GET \
@@ -3319,6 +3497,8 @@ CONFIG_RESPONSE_EOF
         [[ -z "$conf" || "$conf" == "null" ]] && conf="/etc/keepalived/keepalived.conf"
         chk="$(dirname "$conf")/check_haproxy.sh"
         if [[ -f "$conf" ]] && grep -q "$marker" "$conf" 2>/dev/null; then we_own="true"; fi
+        # See the heredoc copy.
+        disc_known=$(echo "$resp" | jq -r 'if has("discovery_known") then (.discovery_known|tostring) else "" end' 2>/dev/null)
 
         # v1.10.4 — VIP adoption discovery. Report a keepalived.conf we do NOT own so an existing
         # VIP can be adopted from the UI instead of retyped. STRICTLY READ-ONLY: this never writes
@@ -3333,28 +3513,65 @@ CONFIG_RESPONSE_EOF
             if [[ "$we_own" == "true" || ! -f "$conf" ]]; then
                 # Nothing adoptable here. Clear a previous report exactly once.
                 [[ -f "$cache" ]] || return 0
-                curl -k -s --connect-timeout 10 --max-time 30 -X POST \
-                    "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-discovery" \
+                # See the heredoc copy: clear only on a 2xx, or a managed node keeps being
+                # offered for adoption and nothing ever retries the clear.
+                local clr_code
+                clr_code=$(curl -k -s --connect-timeout 10 --max-time 30 -o /dev/null -w '%{http_code}' \
+                    -X POST "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-discovery" \
                     -H "X-API-Key: $AGENT_TOKEN" -H "Content-Type: application/json" \
-                    -d "{\"config_path\":\"$conf\",\"exists\":false}" >/dev/null 2>&1 || return 0
+                    -d "{\"config_path\":\"$conf\",\"exists\":false}" 2>/dev/null)
+                [[ "$clr_code" =~ ^2[0-9][0-9]$ ]] || return 0
                 rm -f "$cache"
                 return 0
             fi
             cur_hash=$(md5sum "$conf" 2>/dev/null | awk '{print $1}')
             [[ -z "$cur_hash" ]] && return 0
-            [[ -f "$cache" && "$(cat "$cache" 2>/dev/null)" == "$cur_hash" ]] && return 0
+            # See the heredoc copy: the cache may only suppress while the server agrees it holds
+            # a discovery for this node, or a rejected report hides the node permanently.
+            # Cache line is "<hash>" after a success, or "<hash> rejected" after a 4xx.
+            local cached_line cached_hash cached_state=""
+            cached_line=$(cat "$cache" 2>/dev/null)
+            cached_hash="${cached_line%% *}"
+            [[ "$cached_line" == *" "* ]] && cached_state="${cached_line#* }"
+            if [[ -n "$cached_hash" && "$cached_hash" == "$cur_hash" ]]; then
+                # A 4xx means the server refuses THESE BYTES. Re-posting them cannot succeed, and
+                # every attempt is stored as a failed agent call (4xx/5xx are never sampled out), so
+                # an unattended loop would write a row carrying the whole config every cycle. Stay
+                # quiet until the file changes — the reason was logged when it was rejected.
+                [[ "$cached_state" == "rejected" ]] && return 0
+                # Otherwise the cache only holds while the SERVER agrees it has the report.
+                [[ "$disc_known" != "false" ]] && return 0
+            fi
             # jq -Rs makes the file a single JSON string with its newlines intact, so the content the
             # server hashes is byte-identical to what is on disk — the takeover authorisation is
             # pinned to that hash.
             content_json=$(jq -Rs . < "$conf" 2>/dev/null) || return 0
             body=$(jq -n --arg p "$conf" --argjson c "$content_json" \
                 '{config_path:$p, exists:true, is_managed:false, config_content:$c}' 2>/dev/null) || return 0
-            if curl -k -s --connect-timeout 10 --max-time 30 -o /dev/null -X POST \
-                "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-discovery" \
+            # See the heredoc copy: cache ONLY on a 2xx, or a rejected report is recorded as
+            # delivered and the node never reappears in the adoption panel.
+            local disc_code
+            disc_code=$(curl -k -s --connect-timeout 10 --max-time 30 -o /dev/null -w '%{http_code}' \
+                -X POST "$MANAGEMENT_URL/api/agents/$AGENT_NAME/keepalived-discovery" \
                 -H "X-API-Key: $AGENT_TOKEN" -H "Content-Type: application/json" \
-                --data-binary "$body" 2>/dev/null; then
+                --data-binary "$body" 2>/dev/null)
+            if [[ "$disc_code" =~ ^2[0-9][0-9]$ ]]; then
                 printf '%s' "$cur_hash" > "$cache" 2>/dev/null
                 log "INFO" "KEEPALIVED: reported an unmanaged keepalived.conf for adoption"
+            elif [[ "$disc_code" == "400" || "$disc_code" == "413" || "$disc_code" == "422" ]]; then
+                # ONLY the codes that mean "these bytes are unacceptable" stop the retry: too large
+                # to analyse, malformed, rejected by validation. Re-posting identical content cannot
+                # change any of those answers.
+                #
+                # 401 and 404 are deliberately NOT here even though they are 4xx. Both are transient
+                # in this system — a token rotation, or an agent row briefly absent while it
+                # re-registers — and treating them as permanent would silence discovery for every
+                # affected node until its keepalived.conf changed, which for a hand-maintained file
+                # may be never. That is the exact failure this release set out to remove.
+                printf '%s rejected' "$cur_hash" > "$cache" 2>/dev/null
+                log "WARN" "KEEPALIVED: discovery report refused (HTTP $disc_code); not retrying until the config changes"
+            else
+                log "WARN" "KEEPALIVED: discovery report failed (HTTP ${disc_code:-none}); will retry next cycle"
             fi
         }
         _kp_discover
@@ -3463,6 +3680,9 @@ CONFIG_RESPONSE_EOF
             cur_hash=$(md5sum "$conf" 2>/dev/null | awk '{print $1}')
             would_hash=$(printf '%s' "$new_conf" | md5sum 2>/dev/null | awk '{print $1}')
             if [[ -n "$cur_hash" && "$cur_hash" == "$would_hash" ]]; then
+                # See the heredoc copy: the ack must be re-asserted here or a single lost report
+                # leaves the VIP reading SYNCING forever.
+                _kp_report "enabled" "$vip_id" "$new_hash" "already converged"
                 return 0
             fi
         fi
@@ -3478,11 +3698,26 @@ CONFIG_RESPONSE_EOF
         local tmp_conf="${conf}.hom.tmp"
         printf '%s' "$new_conf" > "$tmp_conf"
         chmod 0644 "$tmp_conf"
-        if ! keepalived -t -f "$tmp_conf" >/dev/null 2>&1; then
-            rm -f "$tmp_conf"
-            log "ERROR" "KEEPALIVED: config validation failed (keepalived -t) — keeping current config, not (re)starting"
-            _kp_report "error" "$vip_id" "$new_hash" "keepalived -t failed"
-            return 0
+        # See the heredoc copy for why the output is captured rather than discarded.
+        # See the heredoc copy for the measured exit-code table and why the OUTPUT, not the
+        # exit code, decides. Fails closed on anything not known to be benign.
+        local kp_out kp_err kp_fatal
+        if ! kp_out=$(keepalived -t -f "$tmp_conf" 2>&1); then
+            if [[ -z "${kp_out//[[:space:]]/}" ]]; then
+                kp_fatal="keepalived -t exited non-zero without output"
+            else
+                kp_fatal=$(printf '%s\n' "$kp_out" \
+                    | grep -v 'Truncating auth_pass to 8 characters' \
+                    | grep -v '^[[:space:]]*$' || true)
+            fi
+            if [[ -n "$kp_fatal" ]]; then
+                rm -f "$tmp_conf"
+                kp_err=$(printf '%s' "$kp_fatal" | tr '\n\r\t' '   ' | tr -d '"\\' | tail -c 300)
+                log "ERROR" "KEEPALIVED: config validation failed (keepalived -t): ${kp_err} — keeping current config, not (re)starting"
+                _kp_report "error" "$vip_id" "$new_hash" "keepalived -t failed: ${kp_err}"
+                return 0
+            fi
+            log "WARN" "KEEPALIVED: keepalived -t exited non-zero with only known-benign warnings; proceeding"
         fi
         mv -f "$tmp_conf" "$conf"
         chown root:root "$conf" 2>/dev/null

backend/utils/http_instrumentation.py

@@ -0,0 +1,328 @@
+"""v1.11.0 — outbound half of the unified request/response log.
+
+This is deliberately NOT a session or connector factory. Three incompatible
+connector policies coexist in this codebase:
+
+  * `utils.ssrf_guard.safe_connector()` — IPv4-pinned, TLS verification on;
+    returns a NEW connector per call because `ClientSession` closes the one it
+    owns, so a shared long-lived connector would raise "Connector is closed".
+  * `services/acme_diagnostics.py` — IPv4-pinned with `ssl=False` for the
+    plain-HTTP port-80 probe.
+  * the DNS providers and the CA-chain import — the default dual-stack
+    connector.
+
+On top of that, `backend/tests/test_acme_diagnostics.py` monkeypatches
+`aiohttp.ClientSession` globally with fakes that implement only
+`__aenter__/__aexit__/head(...)`. Centralising session construction would break
+all of it. So this module wraps the CALL, never the session.
+
+Two hard rules, both load-bearing:
+
+1.  `outbound_span` NEVER raises. Both DNS provider funnels end in
+    `except Exception: raise DnsProviderError("Unexpected ... failure")`, and in
+    GoDaddy's publish path that reverts `dns_record_published` and stalls the
+    ACME order — an instrumentation bug must not masquerade as a provider
+    outage.
+2.  `outbound_span` NEVER swallows. An exception raised inside the block is
+    recorded (status_class 0) and re-raised unchanged.
+"""
+import asyncio
+import logging
+import time
+import uuid
+from contextlib import asynccontextmanager
+from typing import Any, Dict, Optional
+
+from utils.request_log_redaction import safe_error_text, scrub_query_string, scrub_url
+from utils.request_log_settings import get_config
+from utils.request_log_sink import RequestLogRow, request_id_context, request_log_sink
+
+logger = logging.getLogger("haproxy_openmanager.request_log")
+
+# Stable identifiers for the `request_logs.target` column — this is the
+# "kime gitti" (who did we call) axis of the log.
+TARGET_ACME = "acme"
+TARGET_ACME_DIAG = "acme_diag"
+TARGET_LETSENCRYPT_CA = "letsencrypt_ca"
+TARGET_DNS_CLOUDFLARE = "dns_cloudflare"
+TARGET_DNS_GODADDY = "dns_godaddy"
+TARGET_AGENT = "agent"
+TARGET_HAPROXY_STATS = "haproxy_stats"
+TARGET_SETTINGS_PROBE = "settings_probe"
+
+
+def begin_background_trace(label: str) -> str:
+    """Open a fresh correlation id for ONE iteration of a background loop.
+
+    Without this, background outbound rows fell back to `bg:<asyncio task
+    name>`. Nothing in main.py passes `name=` to `create_task`, so a loop is
+    `Task-5` for its entire life and EVERY call it ever makes carries the same
+    `request_id` — measured: fifteen ACME calls across five renewal ticks came
+    out as one id. `GET /api/request-logs/{id}` then answers with up to 100 rows
+    under `related`, presented as "the calls this request made", which in a
+    forensics tool is worse than having no trace: an operator reading a failed
+    renewal is shown a hundred unrelated calls spanning days. Task numbers are
+    also reused across restarts, so `bg:Task-5` can mean a different loop after
+    a redeploy.
+
+    Called at the top of each iteration; the loop task is dedicated, so the next
+    iteration simply overwrites it and there is nothing to reset.
+    """
+    trace_id = f"bg:{label}:{uuid.uuid4().hex[:12]}"[:64]
+    request_id_context.set(trace_id)
+    return trace_id
+
+
+def _correlation_id() -> str:
+    """Inherit the inbound request's id when there is one, so an API call and
+    the CA/DNS calls it triggered share a trace. Background work gets the id
+    opened by begin_background_trace() for the current iteration."""
+    existing = request_id_context.get()
+    if existing:
+        return existing
+    # No inbound request and no iteration trace: background code that has not
+    # been wrapped. Mint a unique id rather than falling back to the task name,
+    # which would silently re-collapse every such call into one row group.
+    try:
+        task = asyncio.current_task()
+        name = task.get_name() if task else "unknown"
+    except Exception:
+        name = "unknown"
+    return f"bg:{name}:{uuid.uuid4().hex[:12]}"[:64]
+
+
+class OutboundSpan:
+    """Handle passed to the `async with` body so the call site can attach the
+    response it just read."""
+
+    __slots__ = (
+        "target", "method", "url", "capture_request_body", "capture_response_body",
+        "safe_error_only",
+        "_status", "_response_headers", "_response_body", "_response_bytes",
+        "_response_content_type", "_request_body", "_request_headers", "_error",
+    )
+
+    def __init__(
+        self,
+        *,
+        target: str,
+        method: str,
+        url: str,
+        capture_request_body: bool,
+        capture_response_body: bool,
+        safe_error_only: bool,
+        request_body: Any = None,
+        request_headers: Optional[Dict[str, str]] = None,
+    ):
+        self.target = target
+        self.method = (method or "GET").upper()
+        self.url = url
+        # Two independent switches on purpose: the ACME JWS request body is a
+        # replayable credential and must never be stored, but the CA's RESPONSE
+        # (problem JSON, order state) is exactly what an operator needs to see.
+        self.capture_request_body = capture_request_body
+        self.capture_response_body = capture_response_body
+        self.safe_error_only = safe_error_only
+        self._request_body = request_body
+        self._request_headers = request_headers
+        self._status: Optional[int] = None
+        self._response_headers: Optional[Dict[str, str]] = None
+        self._response_body: Any = None
+        self._response_bytes: int = 0
+        self._response_content_type: Optional[str] = None
+        self._error: Optional[str] = None
+
+    def set_response(
+        self,
+        status: Optional[int],
+        headers: Optional[Dict[str, str]] = None,
+        body: Any = None,
+    ) -> None:
+        """Record what came back. Safe to call with a partially-read response;
+        never raises, so a call site can hand us whatever it happens to have."""
+        try:
+            self._status = int(status) if status is not None else None
+        except (TypeError, ValueError):
+            self._status = None
+        try:
+            if headers:
+                self._response_headers = {str(k).lower(): str(v) for k, v in dict(headers).items()}
+                self._response_content_type = self._response_headers.get("content-type")
+        except Exception:
+            self._response_headers = None
+
+        if body is None or not self.capture_response_body:
+            return
+        try:
+            if isinstance(body, (bytes, bytearray)):
+                self._response_bytes = len(body)
+                cap = get_config().max_body_bytes
+                self._response_body = bytes(body[:cap]) if cap else None
+            elif isinstance(body, str):
+                encoded = body.encode("utf-8", "replace")
+                self._response_bytes = len(encoded)
+                cap = get_config().max_body_bytes
+                self._response_body = encoded[:cap] if cap else None
+            else:
+                # Already-decoded JSON (the common case: `await resp.json()`).
+                self._response_body = body
+        except Exception:
+            self._response_body = None
+
+    def set_error(self, exc: BaseException, *, type_only: Optional[bool] = None) -> None:
+        try:
+            only = self.safe_error_only if type_only is None else type_only
+            self._error = safe_error_text(exc, type_only=only)
+        except Exception:
+            self._error = "UnknownError"
+
+    def to_row(self, duration_ms: int) -> RequestLogRow:
+        scrubbed = scrub_url(self.url)
+        path = None
+        query_params = None
+        try:
+            import urllib.parse
+
+            parts = urllib.parse.urlsplit(self.url)
+            path = parts.path or "/"
+            _, query_params = scrub_query_string(parts.query)
+        except Exception:
+            pass
+
+        row = RequestLogRow(
+            request_id=_correlation_id(),
+            direction="outbound",
+            target=self.target,
+            method=self.method,
+            url=scrubbed,
+            path=path,
+            query_params=query_params,
+            status_code=self._status,
+            duration_ms=duration_ms,
+            request_headers=self._request_headers,
+            response_headers=self._response_headers,
+            error=self._error,
+        )
+
+        if self._request_body is not None:
+            if not self.capture_request_body:
+                # The call site handed us a synthetic SUMMARY instead of the real
+                # payload (the ACME JWS case) — store the summary as-is.
+                row.request_body_value = _redacted_value(self._request_body)
+            elif isinstance(self._request_body, (bytes, bytearray)):
+                row.request_body_bytes = len(self._request_body)
+                cap = get_config().max_body_bytes
+                row.request_body_raw = bytes(self._request_body[:cap]) if cap else None
+            else:
+                row.request_body_value = _redacted_value(self._request_body)
+
+        if isinstance(self._response_body, (bytes, bytearray)):
+            row.response_body_raw = bytes(self._response_body)
+            row.response_body_bytes = self._response_bytes or len(self._response_body)
+            row.response_content_type = self._response_content_type
+        elif self._response_body is not None:
+            row.response_body_value = _redacted_value(self._response_body)
+
+        return row
+
+
+def _redacted_value(value: Any) -> Any:
+    from utils.request_log_redaction import redact
+
+    return redact(value)
+
+
+@asynccontextmanager
+async def outbound_span(
+    *,
+    target: str,
+    method: str,
+    url: str,
+    request_body: Any = None,
+    request_headers: Optional[Dict[str, str]] = None,
+    capture_body: bool = True,
+    capture_response_body: bool = True,
+    safe_error_only: bool = False,
+):
+    """Time an outbound HTTP call and record one `direction='outbound'` row.
+
+    `capture_body=False` applies to the REQUEST body only, for payloads that
+    are themselves credentials — the ACME JWS body is a replayable, signed
+    capability for the lifetime of its nonce, so the call site passes a
+    description of it instead. The CA's response is still captured, because
+    that is the half an operator actually needs when an order fails.
+
+    `safe_error_only=True` reduces a recorded exception to its type name, for
+    the DNS providers whose own error handling already refuses to surface
+    `str(exc)` (it can carry the request URL and, through it, zone identifiers).
+    """
+    span: Optional[OutboundSpan] = None
+    started = time.perf_counter()
+    try:
+        cfg = get_config()
+        if cfg.enabled and cfg.capture_outbound:
+            span = OutboundSpan(
+                target=target,
+                method=method,
+                url=url,
+                capture_request_body=capture_body and cfg.capture_bodies,
+                capture_response_body=capture_response_body and cfg.capture_bodies,
+                safe_error_only=safe_error_only,
+                request_body=request_body,
+                request_headers=request_headers,
+            )
+    except Exception as exc:  # pragma: no cover - defensive
+        logger.debug(f"outbound_span: could not start span for {target}: {exc}")
+        span = None
+
+    if span is None:
+        # Logging is off (or failed to initialise) — yield a throwaway span so
+        # the call site's `span.set_response(...)` still works.
+        span = OutboundSpan(
+            target=target, method=method, url=url,
+            capture_request_body=False, capture_response_body=False,
+            safe_error_only=safe_error_only,
+        )
+        try:
+            yield span
+        finally:
+            pass
+        return
+
+    try:
+        yield span
+    except BaseException as exc:
+        try:
+            span.set_error(exc)
+        except Exception:
+            pass
+        raise
+    finally:
+        try:
+            duration_ms = int((time.perf_counter() - started) * 1000)
+            request_log_sink.offer(span.to_row(duration_ms))
+        except Exception as exc:  # pragma: no cover - defensive
+            logger.debug(f"outbound_span: failed to record row for {target}: {exc}")
+
+
+async def instrumented_request(session, method: str, url: str, *, target: str,
+                               safe_error_only: bool = True, capture_body: bool = True,
+                               **kwargs):
+    """Convenience wrapper for the call sites that already funnel through
+    `session.request(...)` (the two DNS providers).
+
+    Returns `(status, headers, text)` and leaves error handling entirely to the
+    caller — this helper only adds the log row.
+    """
+    async with outbound_span(
+        target=target,
+        method=method,
+        url=url,
+        request_body=kwargs.get("json"),
+        capture_body=capture_body,
+        safe_error_only=safe_error_only,
+    ) as span:
+        async with session.request(method, url, **kwargs) as resp:
+            text = await resp.text()
+            span.set_response(resp.status, dict(resp.headers), text)
+            return resp.status, dict(resp.headers), text

backend/utils/request_log_prune.py

@@ -0,0 +1,201 @@
+"""v1.11.0 — retention prune for `request_logs`.
+
+Three independent limits, applied in order:
+
+  1. successful rows (`status_class` 1..3) older than `success_retention_days`
+  2. errored rows (`status_class` 0, 4, 5 — 0 meaning "no HTTP response at
+     all") older than `error_retention_days`
+  3. a hard row cap: anything below the `max_rows`-th newest id
+
+Splitting success from error is the point of the design: a busy install can
+keep a week of ordinary traffic while still holding three months of failures
+for forensics, without paying for both.
+
+Deliberately NOT folded into `utils/activity_log.prune_acme_events_and_drafts_if_due`:
+that function is driven by tests with fixed `execute.side_effect` lists and an
+exact return dict, and it is gated behind a `letsencrypt_orders`-exists check
+that would silently disable this prune on an ACME-free install.
+
+Three safety properties, all of which matter at scale:
+
+  * **Batched deletes.** The pool sets `command_timeout=60`; an unbounded
+    DELETE over a multi-million-row table raises `asyncpg.TimeoutError` and
+    then nothing is ever pruned.
+  * **Advisory lock.** `pg_try_advisory_lock` (try, never block) so N replicas
+    × M uvicorn workers do not all scan at once.
+  * **Watermark stamped only after a complete pass.** A pass that times out
+    mid-way is retried at the next tick instead of being recorded as done.
+"""
+import json
+import logging
+from datetime import datetime
+from typing import Dict, Optional
+
+from database.connection import get_database_connection, close_database_connection
+from utils.request_log_settings import get_config
+
+logger = logging.getLogger("haproxy_openmanager.request_log")
+
+# Fresh namespace. Already taken in this codebase: 18181818 (draft cap),
+# 18181819 (wizard create), 18181820 (apply), 0x41434D45 (per-ACME-order),
+# 1836016242 (migration lock).
+PRUNE_LOCK_KEY = 18181821
+
+WATERMARK_KEY = "requestlog.last_pruned_at"
+
+BATCH_SIZE = 5000
+MAX_BATCHES = 40  # ceiling of 200k rows removed per pass
+
+# Retention days ALWAYS travel as a bind parameter. They are operator-supplied,
+# so interpolating them into the SQL string would be an injection point.
+_SQL_TTL_SUCCESS = """
+DELETE FROM request_logs
+WHERE ctid IN (
+    SELECT ctid FROM request_logs
+    WHERE status_class BETWEEN 1 AND 3
+      AND created_at < NOW() - ($1 || ' days')::INTERVAL
+    LIMIT $2
+)
+"""
+
+_SQL_TTL_ERROR = """
+DELETE FROM request_logs
+WHERE ctid IN (
+    SELECT ctid FROM request_logs
+    WHERE (status_class = 0 OR status_class >= 4)
+      AND created_at < NOW() - ($1 || ' days')::INTERVAL
+    LIMIT $2
+)
+"""
+
+_SQL_CAP_CUTOFF = "SELECT id FROM request_logs ORDER BY id DESC OFFSET $1 LIMIT 1"
+
+_SQL_CAP_DELETE = """
+DELETE FROM request_logs
+WHERE ctid IN (
+    SELECT ctid FROM request_logs WHERE id <= $1 LIMIT $2
+)
+"""
+
+
+def _deleted_count(result) -> int:
+    """asyncpg returns the command tag ('DELETE 42') from execute()."""
+    if isinstance(result, str) and result.startswith("DELETE "):
+        try:
+            return int(result.split()[-1])
+        except (ValueError, IndexError):
+            return 0
+    return 0
+
+
+async def _batched_delete(conn, sql: str, first_param) -> int:
+    """Run `sql` repeatedly until a short batch comes back or the ceiling hits."""
+    total = 0
+    for _ in range(MAX_BATCHES):
+        result = await conn.execute(sql, first_param, BATCH_SIZE)
+        count = _deleted_count(result)
+        total += count
+        if count < BATCH_SIZE:
+            break
+    else:
+        logger.info(
+            f"request_logs prune hit the {MAX_BATCHES}-batch ceiling "
+            f"({total} rows this pass); the remainder is removed on the next run"
+        )
+    return total
+
+
+async def _is_due(conn, key: str, min_interval_seconds: int) -> bool:
+    """Watermark gate. Unlike the hardcoded 24h in utils/activity_log.py the
+    interval here is operator-configurable."""
+    row = await conn.fetchrow("SELECT value FROM system_settings WHERE key = $1", key)
+    if not row or row["value"] is None:
+        return True
+    raw = row["value"]
+    if isinstance(raw, str):
+        try:
+            raw = json.loads(raw)
+        except (json.JSONDecodeError, ValueError):
+            return True
+    if not isinstance(raw, str):
+        return True
+    try:
+        last = datetime.fromisoformat(raw.replace("Z", "+00:00"))
+    except ValueError:
+        return True
+    age = (datetime.utcnow() - last.replace(tzinfo=None)).total_seconds()
+    return age >= min_interval_seconds
+
+
+async def _stamp(conn, key: str) -> None:
+    await conn.execute(
+        """
+        INSERT INTO system_settings (key, value, category, description)
+        VALUES ($1, $2::jsonb, 'requestlog', 'Internal: last request_logs prune timestamp')
+        ON CONFLICT (key) DO UPDATE
+        SET value = EXCLUDED.value, updated_at = CURRENT_TIMESTAMP
+        """,
+        key,
+        json.dumps(datetime.utcnow().isoformat() + "Z"),
+    )
+
+
+async def _prune_row_cap(conn, max_rows: int) -> int:
+    """Delete everything below the `max_rows`-th newest id."""
+    cutoff: Optional[int] = await conn.fetchval(_SQL_CAP_CUTOFF, max_rows)
+    if cutoff is None:
+        return 0  # fewer rows than the cap — nothing to do
+    return await _batched_delete(conn, _SQL_CAP_DELETE, cutoff)
+
+
+async def prune_request_logs_if_due(force: bool = False) -> Dict[str, int]:
+    """Run one retention pass if the watermark says it is due.
+
+    Never raises: a prune failure must not take down the loop that calls it.
+    `force=True` skips the watermark gate (used by the manual purge endpoint).
+    """
+    counts = {"success": 0, "error": 0, "overflow": 0, "ran": 0}
+    cfg = get_config()
+
+    conn = None
+    locked = False
+    try:
+        conn = await get_database_connection()
+
+        # One replica only. try-lock: never block a pod waiting on another's pass.
+        locked = await conn.fetchval("SELECT pg_try_advisory_lock($1)", PRUNE_LOCK_KEY)
+        if not locked:
+            return counts
+
+        if not force and not await _is_due(conn, WATERMARK_KEY, cfg.prune_interval_minutes * 60):
+            return counts
+
+        counts["success"] = await _batched_delete(conn, _SQL_TTL_SUCCESS, str(cfg.success_retention_days))
+        counts["error"] = await _batched_delete(conn, _SQL_TTL_ERROR, str(cfg.error_retention_days))
+        counts["overflow"] = await _prune_row_cap(conn, cfg.max_rows)
+        counts["ran"] = 1
+
+        # Only after all three steps completed — a partial pass must be retried,
+        # not recorded as done.
+        await _stamp(conn, WATERMARK_KEY)
+
+        if counts["success"] or counts["error"] or counts["overflow"]:
+            logger.info(
+                f"request_logs prune: {counts['success']} successful, {counts['error']} errored, "
+                f"{counts['overflow']} over-cap row(s) removed"
+            )
+        return counts
+    except Exception as exc:
+        logger.warning(f"prune_request_logs_if_due: {exc}")
+        return counts
+    finally:
+        if conn is not None:
+            if locked:
+                try:
+                    await conn.execute("SELECT pg_advisory_unlock($1)", PRUNE_LOCK_KEY)
+                except Exception:
+                    pass
+            try:
+                await close_database_connection(conn)
+            except Exception:
+                pass

backend/utils/request_log_redaction.py

@@ -0,0 +1,517 @@
+"""v1.11.0 — redaction for the unified request/response log.
+
+Everything that lands in `request_logs.request_body` / `response_body` /
+`request_headers` / `response_headers` / `query_params` passes through here
+first. The rules, in order of how much they are trusted:
+
+1.  **Headers are an ALLOWLIST.** Anything not explicitly listed is dropped.
+    A small set of high-signal headers (`Authorization`, `Cookie`, …) is kept
+    as a presence marker with the value replaced, so an operator debugging a
+    401 can still see *that* a credential was sent.
+2.  **Body keys are matched by a normalized name** (lowercased, punctuation
+    stripped), against an exact set for short generic names that would
+    over-match as substrings (`key`, `payload`) and a contains set for the
+    compound ones (`cert_private_key`, `eab_hmac_key`, …).
+3.  **Values are shape-checked too.** A PEM private key or a JWT-shaped string
+    is redacted no matter what key it arrived under — this is the net that
+    catches a route echoing a secret under a renamed field.
+
+None of these functions raise: a redaction failure must never turn into a
+failed request or a failed provider call, so callers get a safe placeholder
+instead of an exception.
+"""
+import json
+import logging
+import re
+import urllib.parse
+from typing import Any, Dict, Optional, Tuple
+
+logger = logging.getLogger("haproxy_openmanager.request_log")
+
+REDACTED = "***REDACTED***"
+
+# Short, generic names. Matched EXACTLY after normalization, because as
+# substrings they would swallow innocent fields (`key_suffix`, `monkey`,
+# `payload_size`, `keyboard`, `nonce_count`).
+REDACT_EXACT = {
+    "password", "passwd", "pwd", "secret", "token", "key", "auth",
+    "authorization", "cookie", "signature", "protected", "payload",
+    "nonce", "credentials", "credential", "otp", "pin", "jwk", "csr",
+    # `auth_pass` is keepalived's VRRP password and it is the plaintext field
+    # name on `POST/PUT /api/vip` (routers/vip.py binds `payload.auth_pass`
+    # straight into encrypt_vrrp_secret). It normalizes to "authpass", which
+    # matches NOTHING above: "password" is not a substring of "authpass", and
+    # the bare "auth" entry is an EXACT match, not a prefix. Without this line
+    # the VRRP secret is written to request_logs in cleartext on every VIP
+    # create and edit.
+    "authpass",
+}
+
+# Compound names. Matched as SUBSTRINGS of the normalized key.
+#
+# `token` is in here on purpose, not just its compounds. In this domain EVERY
+# field whose name contains "token" is a credential — api_token (the Cloudflare
+# provider credential), agent_token, access_token, session_token — and the cost
+# of over-redacting a hypothetical innocent one is a blanked field, while the
+# cost of under-redacting is a live credential sitting in an audit table.
+REDACT_CONTAINS = {
+    "password", "passwordhash", "secret", "apisecret", "clientsecret",
+    "token", "accesstoken", "refreshtoken", "mfatoken", "resettoken",
+    "sessiontoken", "apitoken", "agenttoken", "csrftoken",
+    "apikey", "xapikey", "privatekey", "publicprivate", "jwkprivatekey",
+    "certprivatekey", "csrprivatekey", "keypem", "privkey",
+    "hmac", "eabhmackey", "eabkid",
+    "credentialsencrypted", "encryptedcredentials", "dnscredentials",
+    "authorization", "cookie", "setcookie", "keyauthorization",
+    "backupcode", "backupcodes", "totp", "totpcode", "totpsecret",
+    "replaynonce", "sessionid", "statspassword", "encryptionkey",
+    "bearer", "signature",
+}
+
+_NORMALIZE_RE = re.compile(r"[^a-z0-9]")
+
+# Value-shaped guards — these fire regardless of the key name.
+_PEM_RE = re.compile(r"-----BEGIN [A-Z0-9 ]*PRIVATE KEY-----")
+_JWT_RE = re.compile(r"^[A-Za-z0-9_-]{16,}\.[A-Za-z0-9_-]{16,}\.[A-Za-z0-9_-]{16,}$")
+
+# ---------------------------------------------------------------------------
+# Secrets embedded INSIDE a value, not carried as their own field.
+# ---------------------------------------------------------------------------
+# Key-name and whole-value matching both miss the biggest source of secrets in
+# this system: a rendered config file handed around as one long string under an
+# innocent key.
+#
+#   GET  /api/agents/{n}/keepalived-config   -> keepalived.config_content
+#   POST /api/agents/{n}/keepalived-discovery -> config_content
+#
+# Both carry a full keepalived.conf whose `auth_pass <secret>` line is the VRRP
+# password in cleartext, and the delivery endpoint is polled every ~2.5 minutes
+# per member node. routers/vip.py states the rule this restores: "the secret
+# never leaves the server in cleartext ... only the at-rest Fernet token and the
+# agent-delivery endpoint ever see the real value" — and routers/agent.py's
+# discovery handler already masks the very same text with the very same pattern
+# before storing it in `vip_discoveries.raw_config_masked`. Capturing the
+# unmasked original into request_logs would sit that plaintext right next to the
+# masked copy the codebase went to three separate lengths to produce.
+#
+# ONE compiled alternation, applied in a single pass over each string value:
+# `re.sub` with no match costs one scan, whereas a pre-filter plus N patterns
+# costs a scan each. Mask the WHOLE remainder of the line (vip.py's reasoning),
+# so a secret containing whitespace cannot partially leak.
+#
+# The `uri` branch is the same class of miss in a different shape. Redacting a
+# key called `secret` does nothing when the SAME secret is also handed back
+# inside a URI under a key called `otpauth_uri`:
+#
+#   POST /api/mfa/enroll -> {"secret": "***REDACTED***",
+#                            "otpauth_uri": "otpauth://totp/X?secret=JBSWY3DP..."}
+#
+# routers/mfa.py's own activity log says "NEVER log the secret itself" and
+# records only `secret_len`. scrub_query_string already knows `secret` is a
+# credential; it was simply never pointed at query strings that arrive inside a
+# body value rather than on the request line. Any URI in any captured string is
+# now run through scrub_url, which also strips userinfo (`https://u:p@host`)
+# and drops the fragment.
+#
+# The `statsauth` / `userlist` branches cover the same shape for HAProxy. This
+# application never RENDERS a credential into a haproxy.cfg, so nothing we
+# generate is at risk - but the agent uploads the node's REAL on-disk file:
+#
+#   linux_install.sh: config_content=$(cat "$config_path")
+#                     -> POST /api/configuration/agents/{n}/config-response
+#
+# plus `POST /api/config/validate` and the bulk import, which take whatever the
+# operator pastes. A production haproxy.cfg routinely carries `stats auth
+# admin:<password>` and a `userlist` block, so the content is credential-bearing
+# even though our generator's output is not. Patterns are anchored to HAProxy
+# keyword syntax rather than the bare word "password", so ordinary prose in an
+# error message ("invalid password format") is left alone.
+#
+# `_EOL` instead of a plain `[^\r\n]` / `\S`: a captured body is only decoded
+# from JSON when it PARSES, and a config upload is routinely larger than the
+# capture cap, so the common case for exactly these payloads is the truncated
+# `{"_raw": ...}` fallback - where the line breaks are still the two-character
+# escape `\n`, not real newlines. Matching to "end of line" without knowing that
+# makes `auth_pass ...` swallow the entire rest of the string: no leak, but the
+# whole remainder of the config is masked and the row becomes useless. So every
+# value here stops at a real newline OR at a literal backslash-n.
+#
+# PERFORMANCE. These patterns run on the writer task, on every string value of
+# every captured body, so their cost is paid per row forever. Measured on an
+# 8 KB config body, 300 iterations:
+#
+#   one combined alternation, IGNORECASE, \b-anchored     308.2 us
+#   the same four patterns run separately (sum)           219.7 us
+#   text.lower() once + four substring pre-checks           5.3 us
+#
+# Two things make the naive version expensive, and neither is the matching:
+# `\b` and IGNORECASE both defeat the regex engine's literal-prefix scan, so
+# every alphanumeric position in 8 KB becomes a candidate start. A body with
+# none of these keywords - which is almost every body - paid the full 308 us to
+# find nothing.
+#
+# So: split the alternation, and gate each pattern behind a substring test on
+# one lowercased copy. A `str.lower()` and an `in` are C-level scans; the regex
+# only runs when its keyword is actually present, and then it runs on text that
+# genuinely contains it. Cost becomes O(total string bytes) rather than
+# O(bytes x patterns), and the common case is ~5 us instead of ~308 us.
+#
+# The pre-checks are on the LOWERCASED copy, so the patterns must stay
+# IGNORECASE: the marker may well have been `AUTH_PASS` in the original.
+_EOL = r"(?:(?!\\n)[^\r\n])"
+_NON_SPACE = r"(?:(?!\\n)\S)"
+
+_RE_AUTH_PASS = re.compile(r"(auth_pass[ \t]+)\S" + _EOL + r"*", re.IGNORECASE)
+# `stats auth <user>:<passwd>` - keep the user, mask the password half, so an
+# operator can still tell WHICH account a 401 was about.
+_RE_STATS_AUTH = re.compile(
+    r"(stats[ \t]+auth[ \t]+[^\s:]*:)" + _NON_SPACE + r"+", re.IGNORECASE
+)
+# `user <name> password <hash>` / `user <name> insecure-password <plain>`
+_RE_USERLIST = re.compile(
+    r"(user[ \t]+" + _NON_SPACE + r"+[ \t]+(?:insecure-)?password[ \t]+)" + _NON_SPACE + r"+",
+    re.IGNORECASE,
+)
+# No IGNORECASE and no `\b`: the character class already covers both cases, and
+# `://` gives the engine a literal to scan for.
+_RE_URI = re.compile(r"[a-zA-Z][a-zA-Z0-9+.\-]*://[^\s\"'<>\\]+")
+
+_EMBEDDED_MASK = "********"
+
+# Only strings long enough to hold `auth_pass ` plus a value are worth scanning.
+_EMBEDDED_MIN_LENGTH = 11
+
+def _keep_prefix(match: "re.Match") -> str:
+    """Keep group 1 (the directive, and for `stats auth` the account name too),
+    replace the value that follows it."""
+    return f"{match.group(1)}{_EMBEDDED_MASK}"
+
+
+def _mask_uri(match: "re.Match") -> str:
+    uri = match.group(0)
+    # A URI with neither a query nor userinfo cannot carry a credential in the
+    # place we scrub, and rebuilding it would only risk changing a value for no
+    # benefit.
+    if "?" not in uri and "@" not in uri:
+        return uri
+    try:
+        scrubbed = scrub_url(uri)
+    except Exception:  # pragma: no cover - scrub_url already swallows
+        return uri
+    # scrub_url reports its own failure as a placeholder string. Inside a larger
+    # text value that would corrupt the surrounding sentence, so keep the
+    # original: it parsed badly enough that urlsplit found no query to scrub.
+    if not scrubbed or scrubbed == "***URL_PARSE_ERROR***":
+        return uri
+    return scrubbed
+
+
+def scrub_embedded_secrets(text: str) -> str:
+    """Mask credentials that live inside a larger text value (config blobs).
+
+    Each pattern is gated behind a substring test on one lowercased copy: see
+    the block comment above for the measurements. Never raises - a scrub
+    failure must not turn into a failed log write, and returning the input
+    unchanged would be the wrong failure direction, so the whole value is
+    replaced instead.
+    """
+    if not text or len(text) < _EMBEDDED_MIN_LENGTH:
+        return text
+    try:
+        lowered = text.lower()
+        if "auth_pass" in lowered:
+            text = _RE_AUTH_PASS.sub(_keep_prefix, text)
+        # Not "stats auth": the directive may be separated by tabs or by more
+        # than one space, which the pattern allows and a substring test does not.
+        if "stats" in lowered and "auth" in lowered:
+            text = _RE_STATS_AUTH.sub(_keep_prefix, text)
+        if "password" in lowered:
+            text = _RE_USERLIST.sub(_keep_prefix, text)
+        if "://" in text:
+            text = _RE_URI.sub(_mask_uri, text)
+        return text
+    except Exception as exc:  # pragma: no cover - defensive
+        logger.debug(f"scrub_embedded_secrets() failed, blanking value: {exc}")
+        return REDACTED
+
+
+_MAX_DEPTH = 6
+_MAX_NODES = 2000
+_MAX_STRING = 4096
+_MAX_LIST_ITEMS = 200
+
+# Header handling. Allowlist wins; presence-only names are emitted with the
+# value replaced so the operator knows the header was there.
+HEADER_ALLOWLIST = {
+    "content-type", "content-length", "content-encoding", "accept",
+    "accept-encoding", "accept-language", "user-agent", "referer", "origin",
+    "host", "connection", "cache-control", "pragma", "date", "server",
+    "x-correlation-id", "x-request-id", "x-response-time",
+    "x-forwarded-for", "x-forwarded-proto", "x-forwarded-host", "x-real-ip",
+    "location", "retry-after", "ratelimit-reset", "ratelimit-remaining",
+    "link", "etag", "vary",
+}
+
+HEADER_PRESENCE_ONLY = {
+    "authorization", "cookie", "set-cookie", "x-api-key", "api-key",
+    "proxy-authorization", "replay-nonce", "www-authenticate",
+    "x-auth-token", "x-agent-token", "x-agent-api-key",
+}
+
+_MAX_HEADERS = 40
+
+
+class _NodeBudget:
+    """Shared mutable counter so a single body can't blow the CPU budget by
+    being wide as well as deep."""
+
+    __slots__ = ("remaining",)
+
+    def __init__(self, remaining: int = _MAX_NODES):
+        self.remaining = remaining
+
+    def spend(self) -> int:
+        self.remaining -= 1
+        return self.remaining
+
+
+def _normalize_key(key: Any) -> str:
+    try:
+        return _NORMALIZE_RE.sub("", str(key).lower())
+    except Exception:
+        return ""
+
+
+def is_secret_key(key: Any) -> bool:
+    """True when a dict key / query param name names a secret."""
+    norm = _normalize_key(key)
+    if not norm:
+        return False
+    if norm in REDACT_EXACT:
+        return True
+    return any(needle in norm for needle in REDACT_CONTAINS)
+
+
+def _is_secret_value(value: str) -> bool:
+    """Shape-based guard for secrets that arrive under an innocent key."""
+    if len(value) < 32:
+        # Neither a PEM block nor a JWT fits in less than this; skip the
+        # regex work on the overwhelmingly common short-string case.
+        return False
+    if _PEM_RE.search(value):
+        return True
+    return bool(_JWT_RE.match(value.strip()))
+
+
+def redact(value: Any, *, depth: int = 0, budget: Optional[_NodeBudget] = None) -> Any:
+    """Recursively redact a decoded body.
+
+    Depth- and node-capped so a hostile or merely pathological payload cannot
+    burn CPU on the writer task. Never raises.
+    """
+    if budget is None:
+        budget = _NodeBudget()
+
+    try:
+        if depth > _MAX_DEPTH:
+            return "***DEPTH_LIMIT***"
+
+        if isinstance(value, dict):
+            out: Dict[str, Any] = {}
+            for k, v in value.items():
+                if budget.spend() <= 0:
+                    out["_node_limit"] = True
+                    break
+                if is_secret_key(k):
+                    out[str(k)] = REDACTED
+                else:
+                    out[str(k)] = redact(v, depth=depth + 1, budget=budget)
+            return out
+
+        if isinstance(value, (list, tuple)):
+            out_list = []
+            for item in list(value)[:_MAX_LIST_ITEMS]:
+                if budget.spend() <= 0:
+                    out_list.append("_node_limit")
+                    break
+                out_list.append(redact(item, depth=depth + 1, budget=budget))
+            if len(value) > _MAX_LIST_ITEMS:
+                out_list.append(f"…[{len(value) - _MAX_LIST_ITEMS} more items]")
+            return out_list
+
+        if isinstance(value, str):
+            if _is_secret_value(value):
+                return REDACTED
+            # Scrub BEFORE truncating. Truncation is not a security control:
+            # `auth_pass` sits inside the first 400 bytes of a rendered
+            # keepalived.conf, well under _MAX_STRING, so relying on the cut to
+            # drop it would be relying on luck about where the secret happens
+            # to fall in the file.
+            value = scrub_embedded_secrets(value)
+            if len(value) > _MAX_STRING:
+                return value[:_MAX_STRING] + f"…[truncated {len(value) - _MAX_STRING} chars]"
+            return value
+
+        return value
+    except Exception as exc:  # pragma: no cover - defensive
+        logger.debug(f"redact() failed, substituting placeholder: {exc}")
+        return "***REDACTION_ERROR***"
+
+
+def redact_headers(headers: Optional[Dict[str, str]]) -> Optional[Dict[str, str]]:
+    """Allowlist-filter a header mapping.
+
+    Allowlisted headers keep their value, `HEADER_PRESENCE_ONLY` headers keep
+    only the fact they were present, everything else is dropped silently.
+    """
+    if not headers:
+        return None
+    try:
+        out: Dict[str, str] = {}
+        for raw_name, raw_value in headers.items():
+            name = str(raw_name).lower()
+            if name in HEADER_PRESENCE_ONLY:
+                out[name] = REDACTED
+            elif name in HEADER_ALLOWLIST:
+                value = str(raw_value)
+                out[name] = value[:1024]
+            if len(out) >= _MAX_HEADERS:
+                break
+        return out or None
+    except Exception as exc:  # pragma: no cover - defensive
+        logger.debug(f"redact_headers() failed: {exc}")
+        return None
+
+
+def scrub_query_string(query: Optional[str]) -> Tuple[str, Optional[Dict[str, str]]]:
+    """Return (scrubbed_query_string, scrubbed_dict) for a raw query string."""
+    if not query:
+        return "", None
+    try:
+        pairs = urllib.parse.parse_qsl(query, keep_blank_values=True)
+        scrubbed = [(k, REDACTED if is_secret_key(k) else v) for k, v in pairs]
+        return urllib.parse.urlencode(scrubbed), dict(scrubbed)
+    except Exception as exc:  # pragma: no cover - defensive
+        logger.debug(f"scrub_query_string() failed: {exc}")
+        return "", None
+
+
+def scrub_url(url: str) -> str:
+    """Strip userinfo and scrub the query string of an absolute URL.
+
+    `https://user:pass@api.example.com/v1?api_key=x`
+      → `https://api.example.com/v1?api_key=***REDACTED***`
+    """
+    if not url:
+        return ""
+    try:
+        parts = urllib.parse.urlsplit(url)
+        netloc = parts.hostname or ""
+        if parts.port:
+            netloc = f"{netloc}:{parts.port}"
+        query, _ = scrub_query_string(parts.query)
+        # Fragments are dropped: they never reach a server and can carry tokens.
+        return urllib.parse.urlunsplit((parts.scheme, netloc, parts.path, query, ""))
+    except Exception as exc:  # pragma: no cover - defensive
+        logger.debug(f"scrub_url() failed: {exc}")
+        return "***URL_PARSE_ERROR***"
+
+
+# Content types whose bodies are worth buffering. Anything else (octet-stream,
+# images, text/event-stream) is size-counted but never copied, which is what
+# keeps streaming and file responses safe.
+CAPTURABLE_CONTENT_TYPES = (
+    "application/json",
+    "application/problem+json",
+    "application/jose+json",
+    "application/x-www-form-urlencoded",
+    "text/plain",
+    "text/html",
+    "text/xml",
+    "application/xml",
+)
+
+
+def is_capturable_content_type(content_type: Optional[str]) -> bool:
+    if not content_type:
+        # No Content-Type on a body-bearing message is rare; assume JSON-ish
+        # rather than dropping the one field the operator wanted to see.
+        return True
+    ct = content_type.split(";")[0].strip().lower()
+    return any(ct.startswith(prefix) for prefix in CAPTURABLE_CONTENT_TYPES)
+
+
+def decode_body(
+    raw: Optional[bytes],
+    content_type: Optional[str],
+    total_bytes: int = 0,
+) -> Tuple[Optional[Any], bool]:
+    """Decode + redact a captured body fragment.
+
+    `raw` is what the middleware managed to buffer (already capped);
+    `total_bytes` is how large the body actually was on the wire. Returns
+    `(jsonb_value, truncated)`. Non-JSON payloads are wrapped as
+    `{"_raw": "..."}` so the column stays a uniform JSONB object that the
+    detail view and any future `->>` query can rely on.
+    """
+    if not raw:
+        return None, False
+
+    truncated = total_bytes > len(raw)
+    ct = (content_type or "").split(";")[0].strip().lower()
+
+    try:
+        text = raw.decode("utf-8", "replace")
+    except Exception:  # pragma: no cover - decode with 'replace' can't raise
+        return {"_raw": "***DECODE_ERROR***"}, truncated
+
+    value: Any
+    if ct in ("application/json", "application/problem+json", "application/jose+json") or (
+        not ct and text[:1] in ("{", "[")
+    ):
+        try:
+            value = redact(json.loads(text))
+        except Exception:
+            # A truncated JSON body will not parse — keep the raw prefix so the
+            # operator still sees what was sent.
+            value = {"_raw": redact(text)}
+    elif ct == "application/x-www-form-urlencoded":
+        try:
+            value = redact(dict(urllib.parse.parse_qsl(text, keep_blank_values=True)))
+        except Exception:
+            value = {"_raw": redact(text)}
+    else:
+        value = {"_raw": redact(text)}
+
+    if truncated:
+        if isinstance(value, dict):
+            value["_truncated"] = True
+            value["_original_bytes"] = total_bytes
+        else:
+            value = {
+                "_value": value,
+                "_truncated": True,
+                "_original_bytes": total_bytes,
+            }
+
+    return value, truncated
+
+
+def safe_error_text(exc: BaseException, *, type_only: bool = False, limit: int = 2000) -> str:
+    """Render an exception for the `error` column.
+
+    `type_only=True` is used for the DNS providers, whose own error paths
+    deliberately never surface `str(exc)` — it can carry the request URL and,
+    through it, tenant/zone identifiers (see services/dns_providers/*.py).
+    """
+    try:
+        name = type(exc).__name__
+        if type_only:
+            return name
+        text = f"{name}: {exc}"
+        redacted = redact(text)
+        if not isinstance(redacted, str):
+            return name
+        return redacted[:limit]
+    except Exception:  # pragma: no cover - defensive
+        return "UnknownError"

backend/utils/request_log_settings.py

@@ -0,0 +1,272 @@
+"""v1.11.0 — operator-tunable settings for the request/response log.
+
+The middleware runs on EVERY request, so the hot path must not touch the
+database. `get_config()` returns a module-global immutable snapshot with no
+`await`; `refresh_config()` reloads it from `system_settings` and is called
+
+  * once at startup, right after migrations,
+  * every `_TTL_SECONDS` from the sink's writer loop (off the request path),
+  * synchronously at the end of `PUT /api/request-logs/settings`, so an
+    operator's change takes effect immediately instead of up to 30s later.
+
+asyncpg has no JSONB codec registered on this pool (see
+database/connection.py), so every value comes back as a raw JSON *string* and
+needs the `isinstance(v, str)` + `json.loads` guard used elsewhere in this
+codebase (services/acme_service.py, utils/activity_log.py).
+"""
+import json
+import logging
+import time
+from dataclasses import dataclass, replace
+from typing import Any, Dict, Optional, Tuple
+
+from database.connection import get_database_connection, close_database_connection
+
+logger = logging.getLogger("haproxy_openmanager.request_log")
+
+SETTINGS_CATEGORY = "requestlog"
+
+# Kept in sync with the seed in database/migrations.ensure_request_log_settings().
+# backend/tests/test_request_log_settings.py asserts the two agree, so a change
+# here without a change there fails the suite rather than drifting silently.
+DEFAULT_EXCLUDE_PATHS = (
+    "/api/request-logs",
+    "/api/health",
+    "/api/docs",
+    "/api/redoc",
+    "/api/openapi.json",
+    "/.well-known/acme-challenge",
+    "/api/agents/heartbeat",
+    "/static",
+    "/favicon.ico",
+)
+
+
+@dataclass(frozen=True)
+class RequestLogConfig:
+    enabled: bool = True
+    capture_inbound: bool = True
+    capture_outbound: bool = True
+    capture_bodies: bool = True
+    capture_get: bool = True
+    # SUCCESSFUL agent polls only. Off by default because the row rate of this
+    # table is otherwise a linear function of fleet size, not of operator
+    # activity: each agent runs a 30s cycle that issues three logged calls
+    # (config, pending-requests, upgrade-status; the heartbeat is already
+    # excluded) plus two more every fifth cycle. Measured, that is ~9 800 rows
+    # per day PER AGENT, so a 200-node fleet writes ~2M rows/day and reaches the
+    # 500 000 max_rows cap in about six hours - at which point the shipped
+    # "7 days of successes, 30 days of failures" is not 7 and 30, it is 0.25.
+    # FAILED agent calls are always kept regardless of this flag: they are the
+    # half an operator actually needs, and they are rare.
+    capture_agent_success: bool = False
+    max_body_bytes: int = 8192
+    sample_rate: float = 1.0
+    exclude_paths: Tuple[str, ...] = DEFAULT_EXCLUDE_PATHS
+    success_retention_days: int = 7
+    error_retention_days: int = 30
+    max_rows: int = 500000
+    prune_interval_minutes: int = 60
+
+    def as_dict(self) -> Dict[str, Any]:
+        return {
+            "enabled": self.enabled,
+            "capture_inbound": self.capture_inbound,
+            "capture_outbound": self.capture_outbound,
+            "capture_bodies": self.capture_bodies,
+            "capture_get": self.capture_get,
+            "capture_agent_success": self.capture_agent_success,
+            "max_body_bytes": self.max_body_bytes,
+            "sample_rate": self.sample_rate,
+            "exclude_paths": list(self.exclude_paths),
+            "success_retention_days": self.success_retention_days,
+            "error_retention_days": self.error_retention_days,
+            "max_rows": self.max_rows,
+            "prune_interval_minutes": self.prune_interval_minutes,
+        }
+
+
+DEFAULT_CONFIG = RequestLogConfig()
+
+_CACHE: RequestLogConfig = DEFAULT_CONFIG
+_CACHE_AT: float = 0.0
+_TTL_SECONDS: float = 30.0
+
+# Bounds, mirrored by the Pydantic model in routers/request_logs.py. Kept here
+# too because refresh_config() reads whatever is in the table, which may have
+# been written by an older build or by hand.
+_BOUNDS = {
+    "max_body_bytes": (0, 262144),
+    "success_retention_days": (1, 365),
+    "error_retention_days": (1, 365),
+    "max_rows": (1000, 50_000_000),
+    "prune_interval_minutes": (5, 1440),
+}
+
+MAX_EXCLUDE_PATHS = 64
+MAX_EXCLUDE_PATH_LENGTH = 200
+
+
+def get_config() -> RequestLogConfig:
+    """Hot-path read: no await, no DB, no lock. Returns the last snapshot."""
+    return _CACHE
+
+
+def set_config(config: RequestLogConfig) -> None:
+    """Replace the snapshot directly. Used by the settings PUT handler (which
+    already has the validated values) and by tests."""
+    global _CACHE, _CACHE_AT
+    _CACHE = config
+    _CACHE_AT = time.monotonic()
+
+
+def _clamp_int(raw: Any, field: str, fallback: int) -> int:
+    try:
+        value = int(raw)
+    except (TypeError, ValueError):
+        return fallback
+    low, high = _BOUNDS[field]
+    return max(low, min(high, value))
+
+
+def _clamp_float(raw: Any, fallback: float, low: float, high: float) -> float:
+    try:
+        value = float(raw)
+    except (TypeError, ValueError):
+        return fallback
+    return max(low, min(high, value))
+
+
+def _as_bool(raw: Any, fallback: bool) -> bool:
+    if isinstance(raw, bool):
+        return raw
+    if isinstance(raw, (int, float)):
+        return bool(raw)
+    if isinstance(raw, str):
+        lowered = raw.strip().lower()
+        if lowered in ("true", "1", "yes", "on"):
+            return True
+        if lowered in ("false", "0", "no", "off"):
+            return False
+    return fallback
+
+
+def normalize_exclude_paths(raw: Any, fallback: Tuple[str, ...]) -> Tuple[str, ...]:
+    """Coerce whatever is stored into a bounded tuple of path prefixes."""
+    if not isinstance(raw, (list, tuple)):
+        return fallback
+    out = []
+    for item in raw:
+        if not isinstance(item, str):
+            continue
+        candidate = item.strip()
+        if not candidate.startswith("/") or len(candidate) > MAX_EXCLUDE_PATH_LENGTH:
+            continue
+        out.append(candidate)
+        if len(out) >= MAX_EXCLUDE_PATHS:
+            break
+    return tuple(out) if out else fallback
+
+
+def config_from_mapping(values: Dict[str, Any], base: Optional[RequestLogConfig] = None) -> RequestLogConfig:
+    """Build a config from a plain suffix→value mapping, clamping every field.
+
+    Unknown keys are ignored and missing keys keep the value from `base`
+    (default: the shipped defaults), so a partially-seeded table still yields a
+    complete, usable config.
+    """
+    base = base or DEFAULT_CONFIG
+    return replace(
+        base,
+        enabled=_as_bool(values.get("enabled", base.enabled), base.enabled),
+        capture_inbound=_as_bool(values.get("capture_inbound", base.capture_inbound), base.capture_inbound),
+        capture_outbound=_as_bool(values.get("capture_outbound", base.capture_outbound), base.capture_outbound),
+        capture_bodies=_as_bool(values.get("capture_bodies", base.capture_bodies), base.capture_bodies),
+        capture_get=_as_bool(values.get("capture_get", base.capture_get), base.capture_get),
+        capture_agent_success=_as_bool(
+            values.get("capture_agent_success", base.capture_agent_success),
+            base.capture_agent_success,
+        ),
+        max_body_bytes=_clamp_int(values.get("max_body_bytes", base.max_body_bytes), "max_body_bytes", base.max_body_bytes),
+        sample_rate=_clamp_float(values.get("sample_rate", base.sample_rate), base.sample_rate, 0.0, 1.0),
+        exclude_paths=normalize_exclude_paths(values.get("exclude_paths"), base.exclude_paths),
+        success_retention_days=_clamp_int(
+            values.get("success_retention_days", base.success_retention_days),
+            "success_retention_days", base.success_retention_days,
+        ),
+        error_retention_days=_clamp_int(
+            values.get("error_retention_days", base.error_retention_days),
+            "error_retention_days", base.error_retention_days,
+        ),
+        max_rows=_clamp_int(values.get("max_rows", base.max_rows), "max_rows", base.max_rows),
+        prune_interval_minutes=_clamp_int(
+            values.get("prune_interval_minutes", base.prune_interval_minutes),
+            "prune_interval_minutes", base.prune_interval_minutes,
+        ),
+    )
+
+
+def _decode_setting_value(raw: Any) -> Any:
+    """JSONB comes back as a raw string on this pool — parse it, but keep the
+    original text if it is not valid JSON (an operator may have hand-written
+    `7` or `seven`)."""
+    if isinstance(raw, str):
+        try:
+            return json.loads(raw)
+        except (json.JSONDecodeError, ValueError):
+            return raw
+    return raw
+
+
+async def load_settings_rows(conn) -> Dict[str, Any]:
+    """Read the `requestlog.*` rows into a suffix→value mapping."""
+    rows = await conn.fetch(
+        "SELECT key, value FROM system_settings WHERE category = $1",
+        SETTINGS_CATEGORY,
+    )
+    values: Dict[str, Any] = {}
+    for row in rows:
+        key = row["key"]
+        suffix = key.split(".", 1)[1] if "." in key else key
+        values[suffix] = _decode_setting_value(row["value"])
+    return values
+
+
+async def refresh_config(force: bool = True) -> RequestLogConfig:
+    """Reload the snapshot from the database.
+
+    Never raises and never leaves a half-built config behind: on any failure
+    the previous snapshot is kept, so a transient DB blip cannot silently turn
+    logging off (or on).
+    """
+    global _CACHE_AT
+    if not force and (time.monotonic() - _CACHE_AT) < _TTL_SECONDS:
+        return _CACHE
+
+    conn = None
+    try:
+        conn = await get_database_connection()
+        values = await load_settings_rows(conn)
+        if values:
+            set_config(config_from_mapping(values))
+        else:
+            # Table not seeded yet (fresh install mid-migration) — keep the
+            # in-code defaults but stamp the timestamp so we don't re-query
+            # every tick.
+            _CACHE_AT = time.monotonic()
+        return _CACHE
+    except Exception as exc:
+        logger.debug(f"refresh_config: keeping previous snapshot ({exc})")
+        _CACHE_AT = time.monotonic()
+        return _CACHE
+    finally:
+        if conn is not None:
+            try:
+                await close_database_connection(conn)
+            except Exception:
+                pass
+
+
+async def maybe_refresh_config() -> RequestLogConfig:
+    """TTL-gated refresh, called from the sink's writer loop."""
+    return await refresh_config(force=False)

backend/utils/request_log_sink.py

@@ -0,0 +1,388 @@
+"""v1.11.0 — batching writer for the unified request/response log.
+
+One row per API call is the highest write volume in this system, and the
+asyncpg pool (min=10/max=50, see database/connection.py) is shared with every
+request handler and four background loops. Acquiring a connection per logged
+request would exhaust it under any real load, so instead:
+
+    hot path  ──offer(row)──▶  bounded asyncio.Queue  ──▶  single writer task
+                                (drops when full)          (executemany batches)
+
+The hot path never awaits I/O and never raises. When the queue is full rows are
+counted as dropped and reported through `GET /api/request-logs/stats`, so a
+saturated logger is visible rather than silent.
+
+Redaction deliberately happens HERE, on the writer task, not in the middleware:
+the recursive walk is the most expensive part of building a row and it has no
+business running inside the request coroutine.
+"""
+import asyncio
+import json
+import logging
+import random
+from contextvars import ContextVar
+from dataclasses import dataclass, field
+from datetime import datetime, timezone
+from typing import Any, Dict, List, Optional
+
+from config import (
+    REQUEST_LOG_BATCH_SIZE,
+    REQUEST_LOG_FLUSH_MS,
+    REQUEST_LOG_QUEUE_MAX,
+    REQUEST_LOG_QUEUE_MAX_BYTES,
+)
+from database.connection import get_database_connection, close_database_connection
+from utils.request_log_redaction import decode_body, redact_headers
+from utils.request_log_settings import get_config, maybe_refresh_config
+
+logger = logging.getLogger("haproxy_openmanager.request_log")
+
+# Set by the inbound middleware; read by outbound_span so an outbound call
+# inherits the id of the inbound request that caused it. That is what turns
+# "operator clicked Issue Certificate" and "we POSTed to Let's Encrypt" into
+# one readable trace.
+request_id_context: ContextVar[Optional[str]] = ContextVar(
+    "request_log_request_id", default=None
+)
+
+# `target` on an INBOUND row means the same thing it means on an outbound one:
+# who was on the other end. Set by the middleware when the caller authenticated
+# with an agent API key rather than a user JWT, which is how agent traffic is
+# told apart from operator traffic without a database lookup on the hot path.
+# Deliberately the same literal as http_instrumentation.TARGET_AGENT, so
+# `target = 'agent'` selects the whole conversation with the fleet in both
+# directions; `direction` separates them when that matters.
+TARGET_INBOUND_AGENT = "agent"
+
+_INSERT_SQL = """
+INSERT INTO request_logs (
+    request_id, direction, target, method, url, path, query_params,
+    status_code, status_class, duration_ms, user_id, username, client_ip,
+    user_agent, request_headers, request_body, request_body_bytes,
+    response_headers, response_body, response_body_bytes, error, truncated,
+    created_at
+) VALUES (
+    $1, $2, $3, $4, $5, $6, $7::jsonb,
+    $8, $9, $10, $11, $12, $13::inet,
+    $14, $15::jsonb, $16::jsonb, $17,
+    $18::jsonb, $19::jsonb, $20, $21, $22,
+    $23
+)
+"""
+
+
+def _jsonb(value: Any) -> Optional[str]:
+    """asyncpg has no JSONB codec on this pool, so JSONB params travel as text
+    and are cast in SQL — the house idiom (utils/activity_log.py)."""
+    if value is None:
+        return None
+    try:
+        return json.dumps(value, default=str)
+    except Exception:
+        return json.dumps({"_serialize_error": True})
+
+
+@dataclass
+class RequestLogRow:
+    """One captured exchange, still holding RAW body bytes.
+
+    Decoding and redaction run in `to_params()` on the writer task.
+    """
+
+    request_id: str
+    direction: str
+    method: str
+    url: str
+    target: Optional[str] = None
+    path: Optional[str] = None
+    query_string: Optional[str] = None
+    query_params: Optional[Dict[str, Any]] = None
+    status_code: Optional[int] = None
+    duration_ms: int = 0
+    user_id: Optional[int] = None
+    username: Optional[str] = None
+    client_ip: Optional[str] = None
+    user_agent: Optional[str] = None
+    request_headers: Optional[Dict[str, str]] = None
+    response_headers: Optional[Dict[str, str]] = None
+    request_body_raw: Optional[bytes] = None
+    request_body_bytes: int = 0
+    request_content_type: Optional[str] = None
+    response_body_raw: Optional[bytes] = None
+    response_body_bytes: int = 0
+    response_content_type: Optional[str] = None
+    # Pre-decoded body override, used by outbound spans that hold a dict/str
+    # rather than wire bytes (e.g. the synthetic JWS summary).
+    request_body_value: Optional[Any] = None
+    response_body_value: Optional[Any] = None
+    error: Optional[str] = None
+    created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
+
+    def queue_weight(self) -> int:
+        """Approximate bytes this row holds while it waits in the queue.
+
+        The buffered bodies are the only part that varies by orders of
+        magnitude (0 to 2 x max_body_bytes); everything else is a handful of
+        short strings and two small header dicts, measured at ~1.4 KB per row.
+        Used to enforce a byte budget alongside the row count, so memory does
+        not become a function of an operator-editable setting.
+        """
+        weight = 1400
+        if self.request_body_raw:
+            weight += len(self.request_body_raw)
+        if self.response_body_raw:
+            weight += len(self.response_body_raw)
+        return weight
+
+    @property
+    def status_class(self) -> int:
+        """`status_code // 100`, or 0 when there was no HTTP response at all
+        (transport error / unhandled exception). 0 is what the error-retention
+        prune treats as an error alongside >= 4."""
+        if not self.status_code:
+            return 0
+        return int(self.status_code) // 100
+
+    def to_params(self) -> List[Any]:
+        req_body, req_truncated = (self.request_body_value, False)
+        if req_body is None:
+            req_body, req_truncated = decode_body(
+                self.request_body_raw, self.request_content_type, self.request_body_bytes
+            )
+
+        res_body, res_truncated = (self.response_body_value, False)
+        if res_body is None:
+            res_body, res_truncated = decode_body(
+                self.response_body_raw, self.response_content_type, self.response_body_bytes
+            )
+
+        return [
+            self.request_id[:64],
+            self.direction,
+            self.target[:32] if self.target else None,
+            (self.method or "")[:10],
+            self.url or "",
+            self.path[:512] if self.path else None,
+            _jsonb(self.query_params),
+            self.status_code,
+            self.status_class,
+            max(0, int(self.duration_ms)),
+            self.user_id,
+            self.username[:50] if self.username else None,
+            self.client_ip,
+            self.user_agent[:1024] if self.user_agent else None,
+            _jsonb(redact_headers(self.request_headers)),
+            _jsonb(req_body),
+            max(0, int(self.request_body_bytes)),
+            _jsonb(redact_headers(self.response_headers)),
+            _jsonb(res_body),
+            max(0, int(self.response_body_bytes)),
+            self.error[:4000] if self.error else None,
+            bool(req_truncated or res_truncated),
+            self.created_at,
+        ]
+
+
+class RequestLogSink:
+    """Bounded queue + single batching writer task (one per uvicorn worker)."""
+
+    def __init__(self, maxsize: int, batch_size: int, flush_ms: int,
+                 max_bytes: int = REQUEST_LOG_QUEUE_MAX_BYTES):
+        self._maxsize = maxsize
+        self._max_bytes = max_bytes
+        self._queued_bytes = 0
+        self._batch_size = batch_size
+        self._flush_seconds = flush_ms / 1000.0
+        self._queue: Optional[asyncio.Queue] = None
+        self._dropped = 0
+        self._written = 0
+        self._failed = 0
+        self._running = False
+
+    # -- lifecycle ---------------------------------------------------------
+
+    def _ensure_queue(self) -> asyncio.Queue:
+        # Created lazily so importing this module never needs a running loop
+        # (matters for the test suite, which imports main.py without one).
+        if self._queue is None:
+            self._queue = asyncio.Queue(maxsize=self._maxsize)
+        return self._queue
+
+    @property
+    def stats(self) -> Dict[str, int]:
+        return {
+            "queued": self._queue.qsize() if self._queue is not None else 0,
+            "queue_capacity": self._maxsize,
+            "queued_bytes": self._queued_bytes,
+            "queue_capacity_bytes": self._max_bytes,
+            "written": self._written,
+            "dropped": self._dropped,
+            "failed_batches": self._failed,
+            "running": 1 if self._running else 0,
+        }
+
+    # -- producer side (hot path) -----------------------------------------
+
+    def offer(self, row: RequestLogRow) -> None:
+        """Enqueue a row. NEVER blocks, NEVER raises.
+
+        Sampling is applied here rather than in the middleware so both
+        directions go through one policy: successful *inbound* traffic can be
+        sampled down, errors never are.
+        """
+        try:
+            cfg = get_config()
+            if not cfg.enabled:
+                return
+            if row.direction == "inbound" and not cfg.capture_inbound:
+                return
+            if row.direction == "outbound" and not cfg.capture_outbound:
+                return
+            # Fleet-scale gate, and the reason this table's size is a function of
+            # operator activity rather than of node count. An agent's 30s cycle
+            # issues three logged calls, plus two more every fifth cycle: ~9 800
+            # rows/day PER AGENT, all of them 200s saying "nothing changed". At a
+            # few hundred nodes that is millions of rows a day, and the row cap is
+            # then reached in hours, which silently shortens the configured
+            # retention for EVERYTHING else in the table - including the failures
+            # the log exists for. Successes are dropped, failures never are.
+            if (
+                row.direction == "inbound"
+                and row.target == TARGET_INBOUND_AGENT
+                and not cfg.capture_agent_success
+                and row.status_class in (1, 2, 3)
+            ):
+                return
+            if (
+                row.direction == "inbound"
+                and cfg.sample_rate < 1.0
+                and row.status_class in (1, 2, 3)
+                and random.random() > cfg.sample_rate
+            ):
+                return
+            if not cfg.capture_bodies:
+                row.request_body_raw = None
+                row.response_body_raw = None
+                row.request_body_value = None
+                row.response_body_value = None
+
+            # Byte budget, checked BEFORE the row count. The count alone does
+            # not bound memory: how much a row weighs is an operator setting,
+            # and `max_body_bytes` at its documented 256 KB ceiling puts the
+            # default 2 000-row queue at ~1 GiB, which is the whole pod limit.
+            # Dropping here is the same visible, counted drop as a full queue.
+            weight = row.queue_weight()
+            if self._queued_bytes + weight > self._max_bytes:
+                raise asyncio.QueueFull
+
+            self._ensure_queue().put_nowait(row)
+            self._queued_bytes += weight
+        except asyncio.QueueFull:
+            self._dropped += 1
+            if self._dropped % 500 == 1:
+                logger.warning(
+                    f"request_log: queue full, {self._dropped} row(s) dropped so far "
+                    f"({self._queue.qsize() if self._queue is not None else 0}/{self._maxsize} rows, "
+                    f"{self._queued_bytes // 1024} KiB/{self._max_bytes // 1024} KiB). "
+                    f"Lower requestlog.max_body_bytes or requestlog.sample_rate. "
+                    f"Raising REQUEST_LOG_QUEUE_MAX also raises the memory this "
+                    f"worker can hold, so raise REQUEST_LOG_QUEUE_MAX_BYTES with it."
+                )
+        except Exception as exc:
+            # Instrumentation must never break the thing it instruments.
+            logger.debug(f"request_log: offer() failed: {exc}")
+
+    # -- consumer side (writer task) --------------------------------------
+
+    async def _collect(self) -> List[RequestLogRow]:
+        """Wait for at least one row, then drain up to batch_size or flush_ms."""
+        queue = self._ensure_queue()
+        first = await queue.get()
+        self._queued_bytes = max(0, self._queued_bytes - first.queue_weight())
+        batch = [first]
+        loop = asyncio.get_running_loop()
+        deadline = loop.time() + self._flush_seconds
+        while len(batch) < self._batch_size:
+            remaining = deadline - loop.time()
+            if remaining <= 0:
+                break
+            try:
+                nxt = await asyncio.wait_for(queue.get(), timeout=remaining)
+                self._queued_bytes = max(0, self._queued_bytes - nxt.queue_weight())
+                batch.append(nxt)
+            except asyncio.TimeoutError:
+                break
+        return batch
+
+    async def _write(self, batch: List[RequestLogRow]) -> None:
+        if not batch:
+            return
+        params = []
+        for row in batch:
+            try:
+                params.append(row.to_params())
+            except Exception as exc:
+                logger.debug(f"request_log: row serialization failed, skipped: {exc}")
+        if not params:
+            return
+
+        conn = None
+        try:
+            conn = await get_database_connection()
+            await conn.executemany(_INSERT_SQL, params)
+            self._written += len(params)
+        except Exception as exc:
+            self._failed += 1
+            # A missing table (pre-migration) or a transient pool error must not
+            # take the writer loop down — drop the batch and carry on.
+            logger.warning(f"request_log: batch write failed ({len(params)} rows): {exc}")
+        finally:
+            if conn is not None:
+                try:
+                    await close_database_connection(conn)
+                except Exception:
+                    pass
+
+    async def run(self) -> None:
+        """Writer loop. Started once per worker from startup_event()."""
+        self._running = True
+        logger.info(
+            f"request_log sink started (queue={self._maxsize}, batch={self._batch_size}, "
+            f"flush={int(self._flush_seconds * 1000)}ms)"
+        )
+        try:
+            while True:
+                try:
+                    batch = await self._collect()
+                    await self._write(batch)
+                    await maybe_refresh_config()
+                except asyncio.CancelledError:
+                    raise
+                except Exception as exc:  # pragma: no cover - defensive
+                    logger.error(f"request_log sink loop error: {exc}")
+                    await asyncio.sleep(1)
+        finally:
+            self._running = False
+
+    async def flush(self, timeout: float = 3.0) -> int:
+        """Drain and persist whatever is queued. Called on shutdown."""
+        queue = self._queue
+        if queue is None or queue.empty():
+            return 0
+        written = 0
+        loop = asyncio.get_running_loop()
+        deadline = loop.time() + timeout
+        while not queue.empty() and loop.time() < deadline:
+            batch: List[RequestLogRow] = []
+            while not queue.empty() and len(batch) < self._batch_size:
+                row = queue.get_nowait()
+                self._queued_bytes = max(0, self._queued_bytes - row.queue_weight())
+                batch.append(row)
+            await self._write(batch)
+            written += len(batch)
+        return written
+
+
+request_log_sink = RequestLogSink(
+    REQUEST_LOG_QUEUE_MAX, REQUEST_LOG_BATCH_SIZE, REQUEST_LOG_FLUSH_MS
+)

backend/version.json

@@ -1,5 +1,5 @@
 {
-  "version": "1.10.11",
-  "releaseName": "The Adoptable tag names the problem that actually blocks adoption",
-  "releaseDate": "2026-08-14"
+  "version": "1.11.1",
+  "releaseName": "Adoption cannot hide a node; Config Import on fresh installs",
+  "releaseDate": "2026-08-15"
 }

frontend/package.json

@@ -1,6 +1,6 @@
 {
   "name": "haproxy-openmanager-frontend",
-  "version": "1.10.11",
+  "version": "1.11.1",
   "description": "HAProxy Load Balancer Management UI",
   "license": "AGPL-3.0-or-later",
   "dependencies": {

frontend/src/App.css

@@ -209,4 +209,24 @@
 
 [data-theme='dark'] .agent-offline:hover > td {
   background-color: #321518 !important;
+}
+
+/* v1.11.0 — Request Log: failed exchanges (4xx/5xx and transport errors) are
+   tinted so a page of traffic reads at a glance. Defined for BOTH themes here
+   rather than as an inline style, so the dark variant is not forgotten (the
+   v1.10.2 regression class). */
+.request-log-error-row > td {
+  background-color: #fff2f0;
+}
+
+.request-log-error-row:hover > td {
+  background-color: #ffe7e5 !important;
+}
+
+[data-theme='dark'] .request-log-error-row > td {
+  background-color: #2a1215 !important;
+}
+
+[data-theme='dark'] .request-log-error-row:hover > td {
+  background-color: #321518 !important;
 }

frontend/src/App.js

@@ -23,7 +23,8 @@ import {
   BulbOutlined,
   BulbFilled,
   PlusOutlined,
-  ThunderboltOutlined
+  ThunderboltOutlined,
+  FileSearchOutlined
 } from '@ant-design/icons';
 
 import Dashboard from './components/DashboardV2';
@@ -49,6 +50,7 @@ import APIDocumentation from './components/APIDocumentation';
 import IPInventory from './components/IPInventory';
 import SiteWizard from './components/SiteWizard';
 import SiteDrafts from './components/SiteDrafts';
+import RequestLog from './components/RequestLog';  // v1.11.0 — request/response log
 import { AuthProvider, useAuth } from './contexts/AuthContext';
 import { ClusterProvider } from './contexts/ClusterContext';
 import { ThemeProvider, useTheme } from './contexts/ThemeContext';
@@ -183,6 +185,18 @@ const { Text } = Typography;
       icon: <SecurityScanOutlined />,
       label: <Link to="/security">Security</Link>,
     },
+    // v1.11.0 — sits directly above Settings on purpose: the Request Log page
+    // and its retention/capture policy (Settings -> Request Log) are the two
+    // halves of one feature, so they are adjacent in the sidebar.
+    {
+      key: '/request-log',
+      icon: <FileSearchOutlined />,
+      label: (
+        <Tooltip placement="right" title="Request Log — every API call in, and every HTTP call this backend made out (ACME, DNS, agents)">
+          <Link to="/request-log">Request Log</Link>
+        </Tooltip>
+      ),
+    },
     {
       key: '/settings',
       icon: <SettingOutlined />,
@@ -469,6 +483,9 @@ function AppContent() {
               <Route path="/security" element={<Security />} />
               <Route path="/agents" element={<AgentManagement />} />
               <Route path="/configuration" element={<Configuration />} />
+              {/* v1.11.0 — the page gates itself on requestlog.read; the route
+                  is unconditional like every other route in this app. */}
+              <Route path="/request-log" element={<RequestLog />} />
               <Route path="/api-docs" element={<APIDocumentation />} />
               <Route path="/pools" element={<PoolManagement />} />
               <Route path="/settings" element={<Settings />} />

frontend/src/components/RequestLog.js

@@ -0,0 +1,606 @@
+/**
+ * v1.11.0 — Request Log.
+ *
+ * One timeline for both directions of HTTP traffic:
+ *   - inbound:  which user called which API endpoint, with what result
+ *   - outbound: which CA / DNS provider / agent this backend called, and what
+ *               came back
+ *
+ * Both live in the same table, so opening one inbound request shows the
+ * outbound calls it triggered (they share a request_id) — that "related" list
+ * is the whole point of the page. Bodies are captured redacted and size-capped
+ * by the backend; nothing is unredacted here.
+ *
+ * Pagination is SERVER-side (a first for this frontend — every other table
+ * filters an already-fetched array). The table can hold millions of rows, so
+ * fetching them to slice client-side is not an option.
+ */
+import React, { useCallback, useEffect, useMemo, useState } from 'react';
+import {
+  Alert, Button, Card, DatePicker, Descriptions, Empty, Input, Modal, Select,
+  Space, Spin, Statistic, Switch, Table, Tag, Tooltip, Typography, message, theme
+} from 'antd';
+import {
+  ApiOutlined, ClockCircleOutlined, CloudDownloadOutlined, DeleteOutlined,
+  EyeOutlined, ReloadOutlined, UserOutlined, WarningOutlined
+} from '@ant-design/icons';
+import axios from 'axios';
+
+import { extractApiError } from '../utils/apiError';
+import { useAuth } from '../contexts/AuthContext';
+
+const { Text, Paragraph } = Typography;
+
+const DIRECTION_OPTIONS = [
+  { label: 'Inbound (API calls to us)', value: 'inbound' },
+  { label: 'Outbound (calls we made)', value: 'outbound' },
+];
+
+const STATUS_OPTIONS = [
+  { label: '2xx Success', value: 2 },
+  { label: '3xx Redirect', value: 3 },
+  { label: '4xx Client error', value: 4 },
+  { label: '5xx Server error', value: 5 },
+  { label: 'No response (transport error)', value: 0 },
+];
+
+const METHOD_OPTIONS = ['GET', 'POST', 'PUT', 'PATCH', 'DELETE', 'HEAD'].map((m) => ({
+  label: m, value: m,
+}));
+
+// Mirrors utils/http_instrumentation.py's TARGET_* constants.
+const TARGET_LABELS = {
+  acme: "ACME / Let's Encrypt",
+  acme_diag: 'ACME diagnostics probe',
+  letsencrypt_ca: "Let's Encrypt CA chain",
+  dns_cloudflare: 'Cloudflare DNS',
+  dns_godaddy: 'GoDaddy DNS',
+  agent: 'HAProxy agent',
+  haproxy_stats: 'HAProxy stats',
+  settings_probe: 'ACME directory probe',
+};
+
+const statusColor = (statusClass) => {
+  if (statusClass === 2) return 'green';
+  if (statusClass === 3) return 'blue';
+  if (statusClass === 4) return 'orange';
+  if (statusClass >= 5) return 'red';
+  return 'red';
+};
+
+const isFailure = (row) => row?.status_class === 0 || row?.status_class >= 4;
+
+const formatTime = (value) => {
+  if (!value) return '—';
+  // created_at is TIMESTAMPTZ, so the ISO string already carries an offset —
+  // no manual 'Z' suffix needed here (unlike the naive-TIMESTAMP columns
+  // elsewhere in this app).
+  const d = new Date(value);
+  return Number.isNaN(d.getTime()) ? String(value) : d.toLocaleString();
+};
+
+const JsonBlock = ({ value, token }) => {
+  if (value === null || value === undefined) {
+    return <Text type="secondary">Not captured</Text>;
+  }
+  return (
+    <pre
+      style={{
+        fontSize: 11,
+        margin: 0,
+        maxHeight: 260,
+        overflow: 'auto',
+        whiteSpace: 'pre-wrap',
+        wordBreak: 'break-word',
+        background: token.colorFillQuaternary,
+        border: `1px solid ${token.colorBorderSecondary}`,
+        borderRadius: token.borderRadius,
+        padding: 8,
+      }}
+    >
+      {typeof value === 'string' ? value : JSON.stringify(value, null, 2)}
+    </pre>
+  );
+};
+
+const RequestLog = () => {
+  const { token } = theme.useToken();
+  const { hasPermission, isAdmin } = useAuth();
+
+  const canRead = hasPermission('requestlog', 'read') || isAdmin();
+  const canManage = hasPermission('requestlog', 'manage') || isAdmin();
+
+  const [rows, setRows] = useState([]);
+  const [total, setTotal] = useState(0);
+  const [totalIsEstimate, setTotalIsEstimate] = useState(false);
+  const [scopedToSelf, setScopedToSelf] = useState(false);
+  const [loading, setLoading] = useState(false);
+  const [loadError, setLoadError] = useState(null);
+
+  const [page, setPage] = useState(1);
+  const [pageSize, setPageSize] = useState(50);
+
+  const [direction, setDirection] = useState(undefined);
+  const [statusClass, setStatusClass] = useState(undefined);
+  const [methods, setMethods] = useState([]);
+  const [target, setTarget] = useState(undefined);
+  const [errorsOnly, setErrorsOnly] = useState(false);
+  const [range, setRange] = useState(null);
+  const [searchTyped, setSearchTyped] = useState('');
+  const [search, setSearch] = useState('');
+
+  const [stats, setStats] = useState(null);
+  const [purging, setPurging] = useState(false);
+
+  const [detailOpen, setDetailOpen] = useState(false);
+  const [detailLoading, setDetailLoading] = useState(false);
+  const [detailError, setDetailError] = useState(null);
+  const [detail, setDetail] = useState(null);
+
+  const params = useMemo(() => {
+    const p = { limit: pageSize, offset: (page - 1) * pageSize };
+    if (direction) p.direction = direction;
+    if (statusClass !== undefined && statusClass !== null) p.status_class = statusClass;
+    // The API takes one method; a single selection is the common case and
+    // keeps the query index-friendly.
+    if (methods.length === 1) p.method = methods[0];
+    if (target) p.target = target;
+    if (errorsOnly) p.errors_only = true;
+    if (search) p.q = search;
+    if (range && range[0]) p.since = range[0].toISOString();
+    if (range && range[1]) p.until = range[1].toISOString();
+    return p;
+  }, [page, pageSize, direction, statusClass, methods, target, errorsOnly, search, range]);
+
+  const fetchLogs = useCallback(async () => {
+    if (!canRead) return;
+    setLoading(true);
+    setLoadError(null);
+    try {
+      const res = await axios.get('/api/request-logs', { params });
+      setRows(res.data?.logs || []);
+      setTotal(res.data?.total || 0);
+      setTotalIsEstimate(Boolean(res.data?.total_is_estimate));
+      setScopedToSelf(Boolean(res.data?.scoped_to_self));
+    } catch (err) {
+      const msg = extractApiError(err, 'Failed to load request logs');
+      setLoadError(msg);
+      setRows([]);
+      setTotal(0);
+    } finally {
+      setLoading(false);
+    }
+  }, [canRead, params]);
+
+  const fetchStats = useCallback(async () => {
+    if (!canRead) return;
+    try {
+      const res = await axios.get('/api/request-logs/stats', { params: { hours: 24 } });
+      setStats(res.data || null);
+    } catch (err) {
+      // Stats are a nice-to-have header; a failure here must not hide the table.
+      setStats(null);
+    }
+  }, [canRead]);
+
+  useEffect(() => { fetchLogs(); }, [fetchLogs]);
+  useEffect(() => { fetchStats(); }, [fetchStats]);
+
+  const openDetail = useCallback(async (id) => {
+    setDetailOpen(true);
+    setDetailLoading(true);
+    setDetailError(null);
+    setDetail(null);
+    try {
+      const res = await axios.get(`/api/request-logs/${id}`);
+      setDetail(res.data || null);
+    } catch (err) {
+      setDetailError(extractApiError(err, 'Failed to load this request'));
+    } finally {
+      setDetailLoading(false);
+    }
+  }, []);
+
+  const runPurge = useCallback(() => {
+    Modal.confirm({
+      title: 'Apply retention now?',
+      icon: <DeleteOutlined />,
+      content:
+        'This runs the configured retention immediately instead of waiting for the next ' +
+        'scheduled pass. It removes rows that are already past their retention window or ' +
+        'beyond the row cap — it does not delete everything.',
+      okText: 'Run retention pass',
+      onOk: async () => {
+        setPurging(true);
+        try {
+          const res = await axios.post('/api/request-logs/purge');
+          const removed = res.data?.removed || {};
+          const count = (removed.success || 0) + (removed.error || 0) + (removed.overflow || 0);
+          message.success(`Retention pass completed — ${count} row(s) removed`);
+          fetchLogs();
+          fetchStats();
+        } catch (err) {
+          message.error(extractApiError(err, 'Retention pass failed'));
+        } finally {
+          setPurging(false);
+        }
+      },
+    });
+  }, [fetchLogs, fetchStats]);
+
+  const columns = useMemo(() => ([
+    {
+      title: 'Time',
+      dataIndex: 'created_at',
+      width: 180,
+      render: (v) => <Text style={{ fontSize: 12 }}>{formatTime(v)}</Text>,
+    },
+    {
+      title: 'Direction',
+      dataIndex: 'direction',
+      width: 110,
+      render: (v) => (
+        <Tag color={v === 'inbound' ? 'blue' : 'purple'}>{v === 'inbound' ? 'IN' : 'OUT'}</Tag>
+      ),
+    },
+    {
+      title: 'Method',
+      dataIndex: 'method',
+      width: 90,
+      render: (v) => <Tag>{v}</Tag>,
+    },
+    {
+      title: 'URL',
+      dataIndex: 'url',
+      ellipsis: true,
+      render: (v) => (
+        <Tooltip title={v} placement="topLeft">
+          <Text style={{ fontSize: 12 }} ellipsis>{v}</Text>
+        </Tooltip>
+      ),
+    },
+    {
+      title: 'Status',
+      dataIndex: 'status_code',
+      width: 100,
+      render: (v, row) => (
+        <Tag color={statusColor(row.status_class)}>{v ?? 'ERR'}</Tag>
+      ),
+    },
+    {
+      title: 'Duration',
+      dataIndex: 'duration_ms',
+      width: 110,
+      render: (v) => (
+        // 1000ms matches the backend's slow-request threshold, so "red here"
+        // means "logged as slow there".
+        <Text type={v > 1000 ? 'danger' : undefined} style={{ fontSize: 12 }}>{v} ms</Text>
+      ),
+    },
+    {
+      title: 'Who / Where',
+      key: 'who',
+      width: 200,
+      render: (_, row) => {
+        if (row.direction === 'inbound') {
+          return (
+            <Space size={4}>
+              <UserOutlined />
+              <Text style={{ fontSize: 12 }}>{row.username || (row.user_id ? `#${row.user_id}` : 'anonymous')}</Text>
+            </Space>
+          );
+        }
+        return (
+          <Tooltip title={TARGET_LABELS[row.target] || row.target}>
+            <Tag icon={<ApiOutlined />}>{row.target || '—'}</Tag>
+          </Tooltip>
+        );
+      },
+    },
+    {
+      title: 'Error',
+      dataIndex: 'error',
+      width: 200,
+      ellipsis: true,
+      render: (v) => (v ? (
+        <Tooltip title={v}><Text type="danger" style={{ fontSize: 12 }}>{v}</Text></Tooltip>
+      ) : <Text type="secondary">—</Text>),
+    },
+    {
+      title: '',
+      key: 'actions',
+      width: 90,
+      fixed: 'right',
+      render: (_, row) => (
+        <Button size="small" icon={<EyeOutlined />} onClick={() => openDetail(row.id)}>
+          Detail
+        </Button>
+      ),
+    },
+  ]), [openDetail]);
+
+  if (!canRead) {
+    return (
+      <Card>
+        <Alert
+          type="error"
+          showIcon
+          message="Access denied"
+          description="You need the requestlog.read permission to view the request log."
+        />
+      </Card>
+    );
+  }
+
+  const inboundStats = stats?.by_direction?.find((d) => d.direction === 'inbound');
+  const outboundStats = stats?.by_direction?.find((d) => d.direction === 'outbound');
+  const dropped = stats?.sink?.dropped || 0;
+
+  const detailRow = detail?.log;
+
+  return (
+    <div>
+      <Card
+        title={<Space><ClockCircleOutlined />Request Log</Space>}
+        extra={
+          <Space>
+            <Button icon={<ReloadOutlined />} onClick={() => { fetchLogs(); fetchStats(); }} loading={loading}>
+              Refresh
+            </Button>
+            {canManage && (
+              <Button icon={<DeleteOutlined />} onClick={runPurge} loading={purging}>
+                Apply retention now
+              </Button>
+            )}
+          </Space>
+        }
+      >
+        <Alert
+          type="info"
+          showIcon
+          style={{ marginBottom: 16 }}
+          message="Every API call in, and every HTTP call out"
+          description={
+            <>
+              Inbound rows show which user called which endpoint and what came back.
+              Outbound rows show which CA, DNS provider or agent this backend contacted.
+              Bodies are captured <strong>redacted and size-capped</strong> — credentials,
+              tokens, private keys and ACME signatures are never stored. Retention is
+              configured in <Text code>Settings → Request Log</Text>.
+            </>
+          }
+        />
+
+        {scopedToSelf && (
+          <Alert
+            type="warning"
+            showIcon
+            style={{ marginBottom: 16 }}
+            message="Showing your own requests only"
+            description="Seeing every user's traffic, and all outbound calls, requires the requestlog.manage permission."
+          />
+        )}
+
+        {dropped > 0 && (
+          <Alert
+            type="warning"
+            showIcon
+            icon={<WarningOutlined />}
+            style={{ marginBottom: 16 }}
+            message={`${dropped} row(s) dropped by this worker`}
+            description="The writer queue filled up. Lower the sampling rate, turn off body capture, or raise REQUEST_LOG_QUEUE_MAX."
+          />
+        )}
+
+        {stats && (
+          <Space size="large" wrap style={{ marginBottom: 16 }}>
+            <Statistic
+              title="Inbound (24h)"
+              value={inboundStats?.total || 0}
+              suffix={inboundStats?.errors ? <Text type="danger" style={{ fontSize: 14 }}>{`/ ${inboundStats.errors} failed`}</Text> : null}
+            />
+            <Statistic
+              title="Outbound (24h)"
+              value={outboundStats?.total || 0}
+              suffix={outboundStats?.errors ? <Text type="danger" style={{ fontSize: 14 }}>{`/ ${outboundStats.errors} failed`}</Text> : null}
+            />
+            <Statistic title="Rows stored" value={stats.total_rows || 0} />
+            <Statistic title="Oldest entry" valueRender={() => <span style={{ fontSize: 16 }}>{formatTime(stats.oldest_at)}</span>} />
+          </Space>
+        )}
+
+        <Space style={{ marginBottom: 16 }} wrap>
+          <Select
+            placeholder="All directions"
+            allowClear
+            style={{ width: 220 }}
+            value={direction}
+            onChange={(v) => { setDirection(v); setPage(1); }}
+            options={DIRECTION_OPTIONS}
+          />
+          <Select
+            placeholder="All statuses"
+            allowClear
+            style={{ width: 210 }}
+            value={statusClass}
+            onChange={(v) => { setStatusClass(v); setPage(1); }}
+            options={STATUS_OPTIONS}
+          />
+          <Select
+            placeholder="All methods"
+            allowClear
+            mode="multiple"
+            maxTagCount={1}
+            style={{ width: 180 }}
+            value={methods}
+            onChange={(v) => { setMethods(v); setPage(1); }}
+            options={METHOD_OPTIONS}
+          />
+          <Select
+            placeholder="All targets"
+            allowClear
+            style={{ width: 220 }}
+            value={target}
+            disabled={direction === 'inbound'}
+            onChange={(v) => { setTarget(v); setPage(1); }}
+            options={Object.entries(TARGET_LABELS).map(([value, label]) => ({ value, label }))}
+          />
+          <DatePicker.RangePicker
+            showTime
+            value={range}
+            onChange={(v) => { setRange(v); setPage(1); }}
+          />
+          <Input.Search
+            placeholder="Search URL…"
+            allowClear
+            enterButton
+            style={{ width: 300 }}
+            value={searchTyped}
+            onChange={(e) => setSearchTyped(e.target.value)}
+            onSearch={(v) => { setSearch(v); setPage(1); }}
+          />
+          <Space size={4}>
+            <Switch
+              checked={errorsOnly}
+              onChange={(v) => { setErrorsOnly(v); setPage(1); }}
+              checkedChildren="Errors"
+              unCheckedChildren="All"
+            />
+          </Space>
+        </Space>
+
+        {loadError && (
+          <Alert type="error" showIcon style={{ marginBottom: 16 }} message={loadError} />
+        )}
+
+        <Table
+          rowKey="id"
+          size="small"
+          loading={loading}
+          dataSource={rows}
+          columns={columns}
+          scroll={{ x: 1400 }}
+          rowClassName={(row) => (isFailure(row) ? 'request-log-error-row' : '')}
+          locale={{
+            emptyText: <Empty description="No requests match these filters" />,
+          }}
+          pagination={{
+            current: page,
+            pageSize,
+            total,
+            showSizeChanger: true,
+            pageSizeOptions: ['25', '50', '100', '200'],
+            showTotal: (t, r) => `${r[0]}-${r[1]} of ${totalIsEstimate ? `${t}+` : t} requests`,
+            onChange: (p, ps) => { setPage(p); setPageSize(ps); },
+          }}
+        />
+      </Card>
+
+      <Modal
+        open={detailOpen}
+        onCancel={() => setDetailOpen(false)}
+        width={960}
+        destroyOnClose
+        title="Request detail"
+        footer={<Button onClick={() => setDetailOpen(false)}>Close</Button>}
+      >
+        {detailLoading ? (
+          <div style={{ textAlign: 'center', padding: 48 }}><Spin /></div>
+        ) : detailError ? (
+          <Alert type="error" showIcon message={detailError} />
+        ) : !detailRow ? (
+          <Empty description="Nothing to show" />
+        ) : (
+          <Space direction="vertical" size="middle" style={{ width: '100%' }}>
+            <Descriptions size="small" column={2} bordered>
+              <Descriptions.Item label="Time">{formatTime(detailRow.created_at)}</Descriptions.Item>
+              <Descriptions.Item label="Direction">
+                <Tag color={detailRow.direction === 'inbound' ? 'blue' : 'purple'}>{detailRow.direction}</Tag>
+              </Descriptions.Item>
+              <Descriptions.Item label="Method"><Tag>{detailRow.method}</Tag></Descriptions.Item>
+              <Descriptions.Item label="Status">
+                <Tag color={statusColor(detailRow.status_class)}>{detailRow.status_code ?? 'no response'}</Tag>
+              </Descriptions.Item>
+              <Descriptions.Item label="URL" span={2}>
+                <Text copyable style={{ fontSize: 12 }}>{detailRow.url}</Text>
+              </Descriptions.Item>
+              <Descriptions.Item label="Duration">{detailRow.duration_ms} ms</Descriptions.Item>
+              <Descriptions.Item label="Target">
+                {detailRow.target ? (TARGET_LABELS[detailRow.target] || detailRow.target) : '—'}
+              </Descriptions.Item>
+              <Descriptions.Item label="User">
+                {detailRow.username || (detailRow.user_id ? `#${detailRow.user_id}` : 'anonymous')}
+              </Descriptions.Item>
+              <Descriptions.Item label="Client IP">{detailRow.client_ip || '—'}</Descriptions.Item>
+              <Descriptions.Item label="Request id" span={2}>
+                <Text code copyable style={{ fontSize: 11 }}>{detailRow.request_id}</Text>
+              </Descriptions.Item>
+              {detailRow.error && (
+                <Descriptions.Item label="Error" span={2}>
+                  <Text type="danger">{detailRow.error}</Text>
+                </Descriptions.Item>
+              )}
+            </Descriptions>
+
+            {detailRow.truncated && (
+              <Alert
+                type="warning"
+                showIcon
+                message="Body truncated"
+                description={
+                  `Only the first part of the body was captured (request ${detailRow.request_body_bytes} bytes, ` +
+                  `response ${detailRow.response_body_bytes} bytes on the wire). Raise the body cap in ` +
+                  `Settings → Request Log if you need more.`
+                }
+              />
+            )}
+
+            <Card size="small" title="Request headers">
+              <JsonBlock value={detailRow.request_headers} token={token} />
+            </Card>
+            <Card size="small" title="Request body">
+              <JsonBlock value={detailRow.request_body} token={token} />
+            </Card>
+            <Card size="small" title="Response headers">
+              <JsonBlock value={detailRow.response_headers} token={token} />
+            </Card>
+            <Card size="small" title="Response body">
+              <JsonBlock value={detailRow.response_body} token={token} />
+            </Card>
+
+            <Card
+              size="small"
+              title={<Space><CloudDownloadOutlined />Calls triggered by this request</Space>}
+            >
+              {detail?.related?.length ? (
+                <Table
+                  rowKey="id"
+                  size="small"
+                  pagination={false}
+                  dataSource={detail.related}
+                  columns={[
+                    { title: 'Dir', dataIndex: 'direction', width: 70,
+                      render: (v) => <Tag color={v === 'inbound' ? 'blue' : 'purple'}>{v === 'inbound' ? 'IN' : 'OUT'}</Tag> },
+                    { title: 'Target', dataIndex: 'target', width: 140, render: (v) => v || '—' },
+                    { title: 'Method', dataIndex: 'method', width: 80 },
+                    { title: 'URL', dataIndex: 'url', ellipsis: true },
+                    { title: 'Status', dataIndex: 'status_code', width: 80,
+                      render: (v, r) => <Tag color={statusColor(r.status_class)}>{v ?? 'ERR'}</Tag> },
+                    { title: '', key: 'go', width: 70,
+                      render: (_, r) => <Button size="small" type="link" onClick={() => openDetail(r.id)}>Open</Button> },
+                  ]}
+                />
+              ) : (
+                <Paragraph type="secondary" style={{ margin: 0 }}>
+                  No other calls share this request id.
+                </Paragraph>
+              )}
+            </Card>
+          </Space>
+        )}
+      </Modal>
+    </div>
+  );
+};
+
+export default RequestLog;

frontend/src/components/Settings.js

@@ -1,6 +1,6 @@
 import React, { useEffect, useState } from 'react';
 import { Card, Form, Switch, Button, InputNumber, message, Tabs, Input, Select, Collapse, Space, Alert, Tag, Spin, Tooltip } from 'antd';
-import { SafetyCertificateOutlined, ApiOutlined, CheckCircleOutlined, CloseCircleOutlined, InfoCircleOutlined } from '@ant-design/icons';
+import { SafetyCertificateOutlined, ApiOutlined, CheckCircleOutlined, CloseCircleOutlined, InfoCircleOutlined, FileSearchOutlined } from '@ant-design/icons';
 import { useSearchParams } from 'react-router-dom';
 import axios from 'axios';
 
@@ -43,6 +43,15 @@ const Settings = () => {
   const [testResult, setTestResult] = useState(null);
   const [testing, setTesting] = useState(false);
 
+  // v1.11.0 — request/response log retention. Read/written through
+  // /api/request-logs/settings, NOT the generic /api/settings/{category}: that
+  // endpoint stringifies values with str(), which turns True into 'True' and
+  // fails the ::jsonb cast.
+  const [rlForm] = Form.useForm();
+  const [rlLoading, setRlLoading] = useState(false);
+  const [rlSaving, setRlSaving] = useState(false);
+  const [rlDenied, setRlDenied] = useState(false);
+
   const onFinish = (values) => {
     try {
       localStorage.setItem('app_settings', JSON.stringify({
@@ -70,8 +79,62 @@ const Settings = () => {
 
   useEffect(() => {
     loadAcmeSettings();
+    loadRequestLogSettings();
   }, []);
 
+  const loadRequestLogSettings = async () => {
+    setRlLoading(true);
+    try {
+      const res = await axios.get('/api/request-logs/settings');
+      rlForm.setFieldsValue(res.data?.settings || {});
+      setRlDenied(false);
+    } catch (err) {
+      // A viewer can open Settings but has no requestlog.manage — show the tab
+      // read-only-with-explanation rather than a scary console error.
+      if (err?.response?.status === 403) {
+        setRlDenied(true);
+      } else {
+        console.error('Error loading request log settings:', err);
+      }
+    } finally {
+      setRlLoading(false);
+    }
+  };
+
+  const onRequestLogSave = async (values) => {
+    setRlSaving(true);
+    try {
+      const res = await axios.put('/api/request-logs/settings', {
+        ...values,
+        // Values come back from the InputNumber controls as numbers already;
+        // the endpoint is properly typed, so no per-value JSON.stringify here
+        // (unlike the ACME form above, which talks to the legacy endpoint).
+        exclude_paths: values.exclude_paths || [],
+      });
+      // Show what the server ACTUALLY applied, not what was typed. Values are
+      // clamped server-side, and clearing the exclude list does not mean "log
+      // everything": normalize_exclude_paths() falls back to the shipped
+      // defaults so the log viewer and the raw-body heartbeat stay excluded.
+      // Without this the form would keep displaying an empty list that is not
+      // in effect.
+      const applied = res?.data?.settings;
+      if (applied) {
+        rlForm.setFieldsValue(applied);
+        const typed = values.exclude_paths || [];
+        if (typed.length === 0 && (applied.exclude_paths || []).length > 0) {
+          message.warning(
+            'An empty exclude list is not applied as "log everything" — the shipped defaults were restored.'
+          );
+        }
+      }
+      message.success('Request log settings saved');
+    } catch (err) {
+      message.error(err?.response?.data?.detail || 'Failed to save request log settings');
+    } finally {
+      setRlSaving(false);
+    }
+  };
+
   const loadAcmeSettings = async () => {
     setAcmeLoading(true);
     try {
@@ -371,6 +434,158 @@ const Settings = () => {
         </Spin>
       ),
     },
+    {
+      key: 'requestlog',
+      label: (
+        <span><FileSearchOutlined /> Request Log</span>
+      ),
+      children: (
+        <Spin spinning={rlLoading}>
+          <Card>
+            <Alert
+              message="Request / Response Log"
+              description="Records every inbound API call and every outbound HTTP call this backend makes (ACME, DNS providers, agents), with redacted and size-capped request and response bodies. Browse it under Request Log in the sidebar."
+              type="info"
+              showIcon
+              icon={<FileSearchOutlined />}
+              style={{ marginBottom: 24 }}
+            />
+
+            {rlDenied && (
+              <Alert
+                message="Read-only"
+                description="Changing the request log policy requires the requestlog.manage permission."
+                type="warning"
+                showIcon
+                style={{ marginBottom: 24 }}
+              />
+            )}
+
+            <Form
+              form={rlForm}
+              layout="vertical"
+              onFinish={onRequestLogSave}
+              disabled={rlDenied}
+              initialValues={{
+                enabled: true,
+                capture_inbound: true,
+                capture_outbound: true,
+                capture_get: true,
+                capture_bodies: true,
+                max_body_bytes: 8192,
+                sample_rate: 1.0,
+                success_retention_days: 7,
+                error_retention_days: 30,
+                max_rows: 500000,
+                prune_interval_minutes: 60,
+                exclude_paths: [],
+              }}
+            >
+              <Card size="small" title="Capture" style={{ marginBottom: 24 }}>
+                <Form.Item
+                  name="enabled"
+                  label="Enable request log"
+                  valuePropName="checked"
+                  tooltip="Turning this off stops all capture immediately, without a restart. To remove the middleware entirely set REQUEST_LOG_ENABLED=false in the backend environment."
+                >
+                  <Switch />
+                </Form.Item>
+                <Form.Item name="capture_inbound" label="Log inbound API calls" valuePropName="checked">
+                  <Switch />
+                </Form.Item>
+                <Form.Item
+                  name="capture_outbound"
+                  label="Log outbound HTTP calls"
+                  valuePropName="checked"
+                  tooltip="Calls this backend makes to Let's Encrypt / ACME, Cloudflare, GoDaddy, agents and HAProxy stats."
+                >
+                  <Switch />
+                </Form.Item>
+                <Form.Item
+                  name="capture_get"
+                  label="Include GET requests"
+                  valuePropName="checked"
+                  tooltip="GETs are the bulk of the traffic. Turning this off keeps writes and errors only."
+                >
+                  <Switch />
+                </Form.Item>
+                <Form.Item
+                  name="capture_agent_success"
+                  label="Include successful agent polls"
+                  valuePropName="checked"
+                  tooltip="Each agent writes about 9,800 rows a day just saying nothing changed, so on a large fleet this fills the row cap in hours and shortens retention for everything else. FAILED agent calls are always logged regardless. Turn this on only while debugging a specific node, and turn it back off."
+                >
+                  <Switch />
+                </Form.Item>
+                <Form.Item
+                  name="capture_bodies"
+                  label="Capture bodies (redacted)"
+                  valuePropName="checked"
+                  tooltip="Passwords, tokens, API keys, private-key PEMs and ACME signatures are never stored, whatever this is set to."
+                >
+                  <Switch />
+                </Form.Item>
+                <Form.Item name="max_body_bytes" label="Maximum body size captured (bytes)">
+                  <InputNumber min={0} max={262144} step={1024} style={{ width: 200 }} />
+                </Form.Item>
+                <Form.Item
+                  name="sample_rate"
+                  label="Sampling rate for successful requests"
+                  tooltip="1.0 logs everything. Errors are always captured at 100%, whatever this is set to."
+                >
+                  <InputNumber min={0} max={1} step={0.05} style={{ width: 200 }} />
+                </Form.Item>
+              </Card>
+
+              <Card size="small" title="Retention" style={{ marginBottom: 24 }}>
+                <Alert
+                  type="warning"
+                  showIcon
+                  style={{ marginBottom: 16 }}
+                  message="Whichever limit is reached first wins"
+                  description="Rows are removed when they pass their retention window OR when the table exceeds the row cap — the cap is the backstop for a sudden traffic spike."
+                />
+                <Form.Item name="success_retention_days" label="Keep successful requests for (days)">
+                  <InputNumber min={1} max={365} style={{ width: 200 }} />
+                </Form.Item>
+                <Form.Item
+                  name="error_retention_days"
+                  label="Keep failed requests for (days)"
+                  tooltip="4xx, 5xx and calls that got no response at all. Usually set longer than the success window."
+                >
+                  <InputNumber min={1} max={365} style={{ width: 200 }} />
+                </Form.Item>
+                <Form.Item name="max_rows" label="Maximum stored rows">
+                  <InputNumber min={1000} max={50000000} step={10000} style={{ width: 200 }} />
+                </Form.Item>
+                <Form.Item name="prune_interval_minutes" label="Minimum interval between prune passes (minutes)">
+                  <InputNumber min={5} max={1440} style={{ width: 200 }} />
+                </Form.Item>
+              </Card>
+
+              <Card size="small" title="Excluded paths" style={{ marginBottom: 24 }}>
+                <Form.Item
+                  name="exclude_paths"
+                  label="Never log these path prefixes"
+                  tooltip="Health checks, the API docs, the ACME challenge endpoint and the agent heartbeat are excluded by default. The log viewer's own endpoints are always excluded and cannot be re-enabled."
+                >
+                  <Select
+                    mode="tags"
+                    tokenSeparators={[',', ' ']}
+                    placeholder="/api/health"
+                    style={{ width: '100%' }}
+                  />
+                </Form.Item>
+              </Card>
+
+              <Button type="primary" htmlType="submit" loading={rlSaving} disabled={rlDenied}>
+                Save Request Log Settings
+              </Button>
+            </Form>
+          </Card>
+        </Spin>
+      ),
+    },
   ];
 
   return (

frontend/src/components/UserManagement.js

@@ -216,6 +216,17 @@ const PERMISSION_TREE = [
       { title: 'Export Activity Logs', key: 'activity.export' }
     ]
   },
+  // v1.11.0 — the API does no server-side whitelist of permission strings, so
+  // this tree is the ONLY catalogue an admin can grant from. Without an entry
+  // here the permission exists but is unreachable for custom roles.
+  {
+    title: '🧾 Request Log',
+    key: 'requestlog',
+    children: [
+      { title: 'View Request/Response Logs', key: 'requestlog.read' },
+      { title: 'Manage Retention & Purge', key: 'requestlog.manage' }
+    ]
+  },
   {
     title: '⚙️ Settings',
     key: 'settings',

frontend/src/components/__tests__/RequestLog.test.js

@@ -0,0 +1,274 @@
+/**
+ * v1.11.0 regression tests for the Request Log page.
+ *
+ * Three properties matter here and none of them are visible from a snapshot:
+ *
+ *  1. Pagination is SERVER-side. Every other table in this app fetches once and
+ *     slices in the browser; this table can hold millions of rows, so changing
+ *     the page MUST issue a new request with a new offset. A client-side slice
+ *     would look identical on a two-row fixture and fall over in production.
+ *  2. The page gates itself on requestlog.read. The sidebar and the router are
+ *     unconditional (App.js has no hook access at module level), so this
+ *     component is the only gate — and it must not even call the API when the
+ *     caller lacks the permission.
+ *  3. Redacted values are what the modal shows. A test that only asserted the
+ *     modal opens would still pass if the UI un-redacted anything.
+ */
+import React from 'react';
+import { render, screen, fireEvent, waitFor, act } from '@testing-library/react';
+import axios from 'axios';
+import RequestLog from '../RequestLog';
+
+// Rendering the antd Table + filter row + modal is slow; keep the limit here so
+// plain `npm test` passes as shipped rather than needing --testTimeout.
+jest.setTimeout(30000);
+
+jest.mock('axios');
+
+let mockPermissions = { read: true, manage: true, admin: false };
+jest.mock('../../contexts/AuthContext', () => ({
+  useAuth: () => ({
+    hasPermission: (resource, action) =>
+      resource === 'requestlog' && Boolean(mockPermissions[action]),
+    isAdmin: () => mockPermissions.admin,
+  }),
+}));
+
+const INBOUND_ROW = {
+  id: 2,
+  request_id: 'abc123',
+  direction: 'inbound',
+  target: null,
+  method: 'POST',
+  url: '/api/letsencrypt/certificates',
+  path: '/api/letsencrypt/certificates',
+  status_code: 500,
+  status_class: 5,
+  duration_ms: 2431,
+  user_id: 1,
+  username: 'admin',
+  client_ip: '10.0.0.5',
+  error: null,
+  request_body_bytes: 120,
+  response_body_bytes: 88,
+  truncated: false,
+  created_at: '2026-08-11T09:00:00+00:00',
+};
+
+const OUTBOUND_ROW = {
+  id: 1,
+  request_id: 'abc123',
+  direction: 'outbound',
+  target: 'acme',
+  method: 'POST',
+  url: 'https://acme-v02.api.letsencrypt.org/acme/new-order',
+  path: '/acme/new-order',
+  status_code: 429,
+  status_class: 4,
+  duration_ms: 812,
+  user_id: null,
+  username: null,
+  client_ip: null,
+  error: null,
+  request_body_bytes: 0,
+  response_body_bytes: 210,
+  truncated: false,
+  created_at: '2026-08-11T09:00:01+00:00',
+};
+
+const LIST_RESPONSE = {
+  logs: [INBOUND_ROW, OUTBOUND_ROW],
+  total: 2,
+  total_is_estimate: false,
+  limit: 50,
+  offset: 0,
+  scoped_to_self: false,
+};
+
+const DETAIL_RESPONSE = {
+  log: {
+    ...INBOUND_ROW,
+    query_params: null,
+    request_headers: { 'content-type': 'application/json', authorization: '***REDACTED***' },
+    request_body: { domains: ['example.com'], eab_hmac_key: '***REDACTED***' },
+    response_headers: { 'content-type': 'application/json' },
+    response_body: { error: { message: 'ACME rate limited' } },
+  },
+  related: [OUTBOUND_ROW],
+};
+
+const STATS_RESPONSE = {
+  window_hours: 24,
+  by_direction: [
+    { direction: 'inbound', total: 120, errors: 3, avg_duration_ms: 45, max_duration_ms: 2431 },
+    { direction: 'outbound', total: 18, errors: 1, avg_duration_ms: 300, max_duration_ms: 812 },
+  ],
+  by_status_class: [],
+  by_target: [],
+  total_rows: 138,
+  oldest_at: '2026-08-04T09:00:00+00:00',
+  newest_at: '2026-08-11T09:00:01+00:00',
+  sink: { queued: 0, queue_capacity: 2000, written: 138, dropped: 0, failed_batches: 0, running: 1 },
+  retention: { success_retention_days: 7, error_retention_days: 30, max_rows: 500000 },
+};
+
+const listCalls = () => axios.get.mock.calls.filter((c) => c[0] === '/api/request-logs');
+
+beforeEach(() => {
+  jest.clearAllMocks();
+  mockPermissions = { read: true, manage: true, admin: false };
+  axios.get.mockImplementation((url) => {
+    if (url === '/api/request-logs') return Promise.resolve({ data: LIST_RESPONSE });
+    if (url === '/api/request-logs/stats') return Promise.resolve({ data: STATS_RESPONSE });
+    if (url.startsWith('/api/request-logs/')) return Promise.resolve({ data: DETAIL_RESPONSE });
+    return Promise.resolve({ data: {} });
+  });
+  axios.post.mockResolvedValue({ data: { removed: { success: 1, error: 0, overflow: 0 } } });
+});
+
+async function renderPage() {
+  render(<RequestLog />);
+  await waitFor(() => expect(listCalls().length).toBeGreaterThan(0));
+  await act(async () => {});
+}
+
+test('renders both directions of a request from the list endpoint', async () => {
+  await renderPage();
+
+  expect(await screen.findByText('/api/letsencrypt/certificates')).toBeInTheDocument();
+  expect(screen.getByText('https://acme-v02.api.letsencrypt.org/acme/new-order')).toBeInTheDocument();
+  // Inbound shows the user; outbound shows the target it called.
+  expect(screen.getByText('admin')).toBeInTheDocument();
+  expect(screen.getByText('acme')).toBeInTheDocument();
+  expect(screen.getByText('500')).toBeInTheDocument();
+  expect(screen.getByText('429')).toBeInTheDocument();
+});
+
+test('a caller without requestlog.read sees a denial and the API is never called', async () => {
+  mockPermissions = { read: false, manage: false, admin: false };
+
+  render(<RequestLog />);
+
+  expect(await screen.findByText('Access denied')).toBeInTheDocument();
+  expect(axios.get).not.toHaveBeenCalled();
+});
+
+test('an admin without an explicit grant still gets in', async () => {
+  mockPermissions = { read: false, manage: false, admin: true };
+  await renderPage();
+  expect(await screen.findByText('/api/letsencrypt/certificates')).toBeInTheDocument();
+});
+
+test('the errors-only switch is sent to the server, not applied in the browser', async () => {
+  await renderPage();
+  const before = listCalls().length;
+
+  await act(async () => {
+    fireEvent.click(document.querySelector('.ant-switch'));
+  });
+
+  await waitFor(() => expect(listCalls().length).toBeGreaterThan(before));
+  const params = listCalls()[listCalls().length - 1][1].params;
+  expect(params.errors_only).toBe(true);
+});
+
+test('the direction filter is sent as a query parameter', async () => {
+  await renderPage();
+  const before = listCalls().length;
+
+  await act(async () => {
+    fireEvent.mouseDown(document.querySelectorAll('.ant-select-selector')[0]);
+  });
+  await act(async () => {
+    const option = Array.from(document.querySelectorAll('.ant-select-item-option'))
+      .find((el) => el.textContent.includes('Outbound'));
+    fireEvent.click(option);
+  });
+
+  await waitFor(() => expect(listCalls().length).toBeGreaterThan(before));
+  const params = listCalls()[listCalls().length - 1][1].params;
+  expect(params.direction).toBe('outbound');
+});
+
+test('the first request asks for a bounded page, not the whole table', async () => {
+  await renderPage();
+  const params = listCalls()[0][1].params;
+  expect(params.limit).toBe(50);
+  expect(params.offset).toBe(0);
+});
+
+test('opening a row fetches the detail and shows the REDACTED body', async () => {
+  await renderPage();
+
+  await act(async () => {
+    fireEvent.click(screen.getAllByText('Detail')[0]);
+  });
+
+  await waitFor(() =>
+    expect(axios.get).toHaveBeenCalledWith(expect.stringMatching(/\/api\/request-logs\/\d+$/))
+  );
+
+  const modal = await waitFor(() => document.querySelector('.ant-modal-content'));
+  expect(modal.textContent).toContain('***REDACTED***');
+  // The redaction happens server-side; the UI must not attempt to show a raw value.
+  expect(modal.textContent).not.toContain('eab_hmac_key":"');
+  expect(modal.textContent).toContain('ACME rate limited');
+});
+
+test('the detail modal lists the outbound calls triggered by the same request', async () => {
+  await renderPage();
+
+  await act(async () => {
+    fireEvent.click(screen.getAllByText('Detail')[0]);
+  });
+
+  const modal = await waitFor(() => document.querySelector('.ant-modal-content'));
+  await waitFor(() => expect(modal.textContent).toContain('Calls triggered by this request'));
+  expect(modal.textContent).toContain('acme');
+});
+
+test('a self-scoped response explains why the list is narrower', async () => {
+  axios.get.mockImplementation((url) => {
+    if (url === '/api/request-logs') {
+      return Promise.resolve({ data: { ...LIST_RESPONSE, scoped_to_self: true } });
+    }
+    if (url === '/api/request-logs/stats') return Promise.resolve({ data: STATS_RESPONSE });
+    return Promise.resolve({ data: {} });
+  });
+
+  await renderPage();
+  expect(await screen.findByText('Showing your own requests only')).toBeInTheDocument();
+});
+
+test('dropped rows are surfaced as a warning', async () => {
+  axios.get.mockImplementation((url) => {
+    if (url === '/api/request-logs') return Promise.resolve({ data: LIST_RESPONSE });
+    if (url === '/api/request-logs/stats') {
+      return Promise.resolve({ data: { ...STATS_RESPONSE, sink: { ...STATS_RESPONSE.sink, dropped: 42 } } });
+    }
+    return Promise.resolve({ data: {} });
+  });
+
+  await renderPage();
+  expect(await screen.findByText(/42 row\(s\) dropped/)).toBeInTheDocument();
+});
+
+test('a failed load surfaces the server message instead of a blank table', async () => {
+  axios.get.mockImplementation((url) => {
+    if (url === '/api/request-logs') {
+      return Promise.reject({
+        response: { status: 500, data: { error: { message: 'Failed to list request logs' } } },
+      });
+    }
+    return Promise.resolve({ data: STATS_RESPONSE });
+  });
+
+  await renderPage();
+  expect(await screen.findByText('Failed to list request logs')).toBeInTheDocument();
+});
+
+test('the retention button is hidden without requestlog.manage', async () => {
+  mockPermissions = { read: true, manage: false, admin: false };
+  await renderPage();
+  expect(screen.queryByText('Apply retention now')).not.toBeInTheDocument();
+});